Sora users trick platform into creating racist celeb videos • The Register

Guardrails? What guardrails? Naughty netizens found a way to trick the Sora 2 video generator into producing deepfakes of public figures, including OpenAI CEO Sam Altman and billionaire Mark Cuban, that make it sound as though they’re spewing racial slurs. The trick works despite Sora’s built-in filters meant to block hateful language.

AI detection platform Copyleaks reported Wednesday that its review of the recently released Sora 2 app, with its improved video generation model, uncovered several videos using celebrity likenesses to recreate a 2020 incident in which a man wearing a Burger King crown was kicked off a JetBlue flight for a racist tirade. In place of the James May lookalike from the original incident, Sora users recreated the scene using Altman and Cuban, as well as popular streamers xQc, Amouranth, IDKSterling, and YouTuber Jake Paul. 

Sora 2 users weren’t able to perfectly recreate the incident, mind you, as OpenAI’s software does include guardrails to prevent the creation of content with epithets used in the original (i.e., the n-word). However, a simple homophone can be enough to sidestep those restrictions and make it sound as though public figures, including some who’ve opted into Sora’s Cameo feature, were uttering racist slurs, according to Copyleaks.

The so-called Cameo feature, added when Sora 2 launched last month, allows users to upload short clips of themselves that can then be inserted into Sora-generated videos. There’s no official list of celebrities and public figures available in Sora, but users have compiled their own that include a number of dead people and historical figures whose families have been less than happy about their inclusion, apparently granted without permission.  

“To evade platform filters meant to block hate speech, users appeared to prompt Sora with coded or phonetically similar terms—such as ‘knitter’—to generate audio that mimics a well-known racial slur,” Copyleaks said. “The digital double of Altman, for example, screams “I hate knitters” while being escorted from the plane.” 

The other videos that Copyleaks listed all include the word “knitters” in place of particular racist terminology, while the Jake Paul video instead uses the term “neck hurts.” While none of the posts drove a large volume of traffic on Sora itself, Copyleaks noted that many of the videos have been reposted to platforms like TikTok, where one of the Paul videos has been liked more than 168,000 times. A second example of content using Paul’s likeness had him explaining that he hated “juice” that alluded to Jewish people, not liquid pressed from fruit and vegetables.

“This behavior illustrates an unsurprising trend in prompt-based evasion, where users intentionally probe systems for weaknesses in content moderation,” Copyleaks noted. “When combined with the likenesses of recognizable individuals, these deepfakes become more viral and damaging—spreading quickly across and beyond the platform.” 

In short, copyright concerns aren’t even the tip of the iceberg when it comes to concerns of Sora 2’s capacity to distort reality, damage intellectual property, and harm reputations. 

“What makes the trend we’ve identified particularly concerning is the weaponization of recognizable public figures,” Copyleaks said of the risk behind this sort of content. “The combination of a familiar face with offensive content is jarring and drives engagement in ways that anonymous or fictional content doesn’t.”

Cuban doesn’t appear concerned, at least: He’s simply taken to deleting the videos when they’re posted on Sora, the entrepreneur said on X. Unfortunately, once they’re exported from Sora to the wider internet, that gets a lot more difficult, and they’re already out there. 

OpenAI didn’t respond to questions for this story, and the videos Copyleaks identified are still available on Sora as of writing. ®