A pro-Chinese organization allegedly used AI-powered avatars to spread propaganda online.
This was reported by Graphika, a company specializing in AI-based analysis of online communities. According to the report, the pro-Chinese organization "Spamouflage," which is allegedly specialized in spam operations, "almost certainly" first distributed video content with AI avatars on Facebook, Twitter, and YouTube, among other sites, in late November 2022. These videos were "almost certainly" created using AI video software from UK-based provider Synthesia.
Fake anchors in a fake news studio
Specifically, according to Graphika, Spamouflage distributed two videos featuring a male and female news anchor under the media brand "Wolf News" (see cover image). Spamouflage is said to have distributed content under this brand in the past.
The news anchors "Anna" and "Jason" are based on original recordings of actors, with the original voices replaced by synthetic voices in various languages, and the lips and facial expressions adapted to the language and statements using AI technology.
The actor and actress can be seen in numerous other online videos created by companies using the Synthesia service.
Deepfake propaganda
The propaganda videos use a composite of archival images and news footage from online sources and are narrated by synthetic voices. For example, the AI news anchors talk about the hypocritical repetition of empty rhetoric against gun violence in the U.S. and the importance of U.S.-China cooperation for the recovery of the global economy, including China's crucial role.
The videos violate Synthesia's rules, which state that the AI service is only available for commercial purposes with strict moderation. "Political, sexual, personal, criminal and discriminatory content is not tolerated or approved," the website states.
We believe the use of commercially-available AI products will allow influence operation actors to create increasingly high-quality deceptive content at greater scale and speed. In the weeks since we identified the activity described in this report, we have seen other actors move quickly to adopt the exact same tactics. Most recently, this involved unidentified actors using the same AI tools to create videos targeting online conversations in Burkina Faso.
Graphika
If Graphika's suspicions are correct, this would be the first confirmed use of deepfake videos for political propaganda. A deepfake video by Zelenskyy from last year was also suspected of being AI-manipulated, but this could not be verified so far.
Recently, OpenAI also warned against the misuse of AI for propaganda using large language models. Such campaigns are easier to scale with AI and can even operate in real time.
Graphika published its research in the report "Deepfake It Till You Make It". The deepfake videos described can be viewed online at The New York Times. Read our article to learn more about the history of deepfakes.
