From fabricating academic papers to plagiarizing artists, generative AI has its fair share of well-documented mistakes. Now, it appears to be popping up again in state influence operations. According to a new report from Massachusetts-based threat intelligence firm RecordedFuture, the latest campaign was "likely" aided by commercial artificial intelligence speech generation products, including publicly released technology from popular startup ElevenLabs.
The report describes a Russia-linked campaign aimed at undermining European support for Ukraine that it calls an "assassination operation" that prominently uses artificial intelligence-generated voiceovers in false or misleading "news" videos.
The videos were targeted at a European audience and, among other topics, attacked Ukrainian politicians for corruption or questioned the usefulness of military aid to Ukraine.
The report states that it is "highly likely" that video creators used speech-generating artificial intelligence, including technology from ElevenLabs, to make their content appear more legitimate. To test this, researchers at RecordedFuture submitted the clips to ElevenLabs' own AI speech classifier, which allows anyone to "detect whether an audio clip was created with ElevenLabs," and got a match;
ElevenLabs did not respond to a request for comment. While RecordedFuture noted that several commercial AI speech generation tools may be used, it did not name others besides ElevenLabs.
The influence movement's own masterminds inadvertently demonstrated the usefulness of artificial intelligence speech generation when they rather hastily released a number of videos featuring real-person voices with "unmistakable Russian accents." In contrast, the AI-generated dubbing was in a variety of European languages, such as English, French, German and Polish, without any foreign accents.
According to RecordedFuture, AI also allows misleading snippets to be quickly posted in multiple languages across Europe, such as English, German, French, Polish and Turkish (all supported by ElevenLabs).
RecordedFuture blamed the activity on SocialDesignAgency, a Russia-based organization that the U.S. government sanctioned in March for operating "a network of more than 60 websites that impersonated genuine European news organizations and then used fake social media accounts to amplify the misleading content of these spoofed sites." The State Department said at the time that all of this was being done "on behalf of the Russian government."
RecordedFuture concluded that the overall impact of the campaign on European opinion was minimal.
According to Bloomberg, a voice fraud detection company concluded that this is not the first time that ElevenLabs’ products have been accused of abuse. A voice fraud detection company has concluded that its technology played a role in a robocall impersonating U.S. President Joe Biden during a January 2024 primary election urging voters not to go out to vote. In response, ElevenLabs said it released new security features, such as automatically blocking politicians' voices.
ElevenLabs prohibits "unauthorized, harmful, or deceptive counterfeiting," saying it uses a variety of tools to enforce this, such as automated and human review;
ElevenLabs has experienced explosive growth since its founding in 2022. The company's ROE recently increased to $80 million, up from $25 million less than a year ago, and it could soon be valued at $3 billion. Its investors include Andreessen Horowitz and former Github CEO Nat Friedman.