Generative AI and Disinformation Report

Recent trends on Generative AI & disinformation

Generative AI (GenAI) has several legitimate uses across various fields, including content creation, healthcare, entertainment, education and others. However, the ability to create highly realistic content with minimal cost, experience, and time also poses risks of potential misuse. Specifically, GenAI can be and is often misused to create various forms of disinformation. 

• Deepfake videos
• Misleading synthetic images
• Fabricated news articles
• Synthetic social media posts (e.g. mimicking the style of celebrities and influencers)
• Phishing emails 

Α representative example of the shift from traditional disinformation methods to using GenAI is a case involving the South Korean first Lady Kim Keon-hee. In September 2022, a Facebook post shared a satirical image of 'South Korean first lady at flood cleanup' that misled Facebook users. While some users seemed to understand the doctored image was satirical, others appeared to be misled. A year later, Kim Keon became a victim of disinformation again, but this time it was through the use of AI methods. An AI-altered clip of South Korean first lady delivering English speech misleads online and an image of South Korean first lady 'on Vogue cover' is AI generated. Both cases were highly realistic and misleading.

Multiple stakeholders have recognised the problem, but have not sufficiently addressed it. Existing or forthcoming regulations and other instruments provide a basis for some actions. However, emerging cases of disinformation created with AI and platforms’ inadequate mitigation show that a more comprehensive approach is needed.

“Disinformation in 2024 is still a pressing concern, and tops lists of short-term risks of AI, such as the World Economic Forum Global Risks report.” 

A recent research on AI manipulation trends has been presented by vera.ai partner AFP at the GlobalFact, the 11th Global Fact-Checking Summit and shows the trends on deepfake intentions (Figure 1) and location (Figure 2).

An example of a disinformation case (see Figure 3)  using a fully synthetic image shows an ‘Image of Palestinian carrying children out of rubble’. 

Recent technical developments and the growing use of generative AI systems have vastly increased the disinformation challenges. AI is making it easier to modify and create fake texts, images, and audio clips that can look real. Despite offering opportunities for legitimate purposes (e.g., art or satire), AI content is also widely generated and disseminated across the Internet, causing – intentionally or not – harm and misperceptions. This recent study from Google's DeepMind ‘Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data’ is based on the analysis of 200 real incidents and sheds light on how GenAI models are being misused or exploited in practice so far (not just disinformation but also scams, fraud, profit, etc).

However, the risks do not only reside in creating realistic-looking content: the possibilities of production at scale and increased distribution are further challenges. The use of AI can also happen in the context of FIMI – Foreign Information Manipulation and Interference – campaigns, as documented in a recent OpenAI report.  In the documented cases, AI was mainly used to create and translate text and fake engagement, rather than to generate deepfakes. This technology allows the creation of tailored content that can better reach specific target audiences.  

AI-generated disinformation and synthetic media also open the door to plausible deniability of all types of content and allow true content to be challenged. It thus poses an almost philosophical challenge to the foundations of credibility. 

The impact of the information environment should also be researched. Recent research has shown for instance that Gen-AI media outlets are now recommended by Google services, jeopardizing the model of human curated and controlled news production.

Additional problems lie in the opacity of the data that feeds these systems, conditioning their outputs, and in the public's misunderstanding of this technology, which leads to it being used for inappropriate purposes, thus increasing confusion. Stakeholders are reacting at different paces to this challenge, and some attempts at collaboration have been made, but much remains to be done. Another challenge is that local solutions are not valid for a global issue.

More explicitly, research on Disinformation capabilities of Large Language Models (LLMs) showed that LLMs readily generate disinformation. To evaluate how LLMs behave in different contexts, disinformation news articles were generated with LLMs using five distinct categories of popular disinformation narratives: COVID-19, Russo-Ukrainian war, Health, US Elections and Regional. Summarizing the findings of Vykopal et al. 2023:

• Falcon is the only model that tends to disagree with the disinformation narratives.
• ChatGPT also behaves safely in some cases, but it is significantly less safe than Falcon.
• Vicuna and Davinci will readily generate disinformation.
• The model capacity impacts the quality and agreement.
• OPT-IML has the worst performance.
• Narrative abstracts in the prompt affect the results.

As an overall conclusion, LLMs exhibit capabilities to generate disinformation at will, and these capabilities are, to some extent, steerable via prompts. LLMs can be prompted to hallucinate novel supporting details and facts, such as names, events, or statistics. This is concerning as it might be difficult to fact-check a deluge of such "facts" in the future.

Regarding the vast and rapid dissemination of GenAI content, during the Digital Method Initiative Winter School, an investigation was conducted as part of vera.ai research, into four use cases of artificially generated images from the Israel war on Gaza. For each case, 1,000 URLs were collected from Google Fact Explorer and were filtered to contain valid images. The image related to each use case is presented in Table 1 as well as the number of valid URLs and unique images after filtering. An image is considered unique when it depicts the same content but is stored in a different file format (e.g., compressed). The instances of these images were spread out over a span of approximately two months for the first three cases and approximately 10 months for the fourth case. This indicates the rapid dissemination of misleading content and the sheer volume of misleading content. The same image has been circulated multiple times and altered, making it even more challenging to detect. Another observation from this research is that the images were shared on mainstream social media platforms like Facebook, Instagram, and Twitter/X, as well as on others like Tumblr, Stowe, and LinkedIn.

During the election season in India, AI tools have resurrected dead leaders, cloned voices and generated videos of dancing politicians. The overall effect is confounding, adding to a social media landscape already inundated with misinformation. The volume of online detritus is far too great for any election commission to track, let alone debunk (cf. A Small Army Combating a Flood of Deepfakes in India’s Election for more details).

Voice cloning is on the rise with several manipulated audio files being disseminated. Figure 4 presents the deepfake techniques on a dataset of 75 deepfakes retrieved from the Google Fact Check explorer showing that 68.1% are detected as voice cloning.

An election-related case involves a fake Joe Biden robocall urging New Hampshire voters not to participate in the Tuesday Democratic primary. Additionally, a Ukraine war-related disinformation case, disseminated in several languages, including Greek, shows Oleksiy Danilov, the former head of Ukraine's security council, appearing to joke about the attack and acknowledge his country's involvement.

The AFP Fact Check articles for the year 2023 include 107 out of 7447 fact checks labeled as AI content, which represents 1.44% of the total articles. For the year 2024, 80 out of 3834 articles (2.09%) have been labeled as AI content. This shows a significant relative 45% increase in articles related to AI content for year 2024 compared to 2023, indicating a growing focus on AI in the media.

The WIRED AI Election project is tracking every instance of AI’s use in and around the elections in 2024. In India and Indonesia, dead leaders are rising to throw their support behind their political successors; in South Africa, the rapper Eminem is endorsing opposition parties; and in the United States, President Joe Biden is telling voters in New Hampshire to stay home. 

A list of 99 synthetic images and 226 deepfake videos have been collected by veraAI project partners for evaluation purposes. The list contains images and videos intended for satire but also examples related to defamation of political figures and propaganda. Below we present indicative examples of a deepfake video and a synthetic image that received a lot of attention.

Ukrainian President Volodymyr Zelenskyy Deepfake

Description: In March 2022, a deepfake video of Ukrainian President Volodymyr Zelenskyy was spread, showing him purportedly surrendering to Russian forces. The video was intended to undermine Ukrainian morale and spread misinformation during the ongoing conflict.

Impact: The video showcased the potential of deepfakes to influence public perception in critical situations.

Fiery explosion outside the Pentagon on May 22, 2023

Description: Social media users are claiming an image shows a fiery explosion outside the Pentagon on May 22, 2023. The Defense Department confirmed to AFP that there was no such attack, and the picture of the supposed blast appears to have been generated using AI technology.

Impact: The earliest tweet AFP found sharing the image came from "CBKNEWS," a QAnon-promoting account that has previously shared other disinformation, though the original source of the image was not immediately known. The spread of the alleged photo appeared to cause a brief dip on Wall Street on May 22, with the S&P 500 stumbling by 0.29 percent before recovering. It also prompted live television coverage from an Indian news organization, according to journalists who shared recordings of the segment.

The same effect was attempted many years ago, in 2013, when the Associated Press Twitter account was hacked and reported that two explosions had occurred in the White House and Barack Obama had been injured. As CNBC had reported “the Dow plunged more than 140 points and bond yields fell. Within six minutes, the Dow recovered its losses and was trading with triple-digit gains. Reuters estimated that the temporary loss of market cap in the S&P 500 alone totaled $136.5 billion”. In 2024 a reputable account did not need to be hacked, as GenAI provided the necessary plausibility. 

The Database of known fakes (DBKF) is a tool that allows users to easily double-check whether a claim, image or video has already been debunked by trusted fact-checkers (IFCN signatories), including by whom, when and how. Leveraging text analysis, visual similarity and semantic technologies, DBKF provides powerful search functionalities and enables insights into disinformation campaigns. It consist of 140,400* debunks**. Searching with related keywords, statistics about existing debunks are presented. Searching for ‘midjourney’ 1,094 debunks are retrieved from 2018 to 2024 as presented in the following diagram.

* Accessed on 09.07.2024
** A Debunk combines all results of debunking misinformation, including the original claim, the sources where it appears, the description of the findings and the evidences used to draw the conclusions

More search results for:

• “chatgpt” in 29 debunks
• "fully synthetic" in 49 debunks
• "stable diffusion" OR "latent diffusion" in 25 debunks
• “genAI” in 875 debunks
• "deepfake" OR "deepfakes" in 426 debunks
• "faceswap" in 448 debunks

From simple posts and articles to Coordinated Behavior

Apart from simple posts and articles, GenAI media items are disseminated in a coordinated manner. An example of pro-Putin Coordinated groups is analysed by an automated alert workflow developed by vera.ai. The workflow retrieves posts three times a day from a set of public Facebook accounts known for frequently spreading content rated as false by Meta’s third-party fact-checkers. This monitoring process generates alerts for top-performing posts and coordinated links. Additionally, it detects new coordinated actors to keep the pool of monitored accounts updated over time.

Starting in October 2023, the alerts have identified over 6,000 top-performing posts and almost 10,000 coordinated links.

Among the different coordinated networks discovered (see Figure 5), this report focuses on a set of groups identified by the GPT-4 powered automatic labeling system as “Russian Language Pro-Putin Advocacy and Support.” The alerts currently rely on CrowdTangle data, and the analysis has been conducted using the Meta Content Library as the data source.

The network was discovered via eight posts. Indicatively four examples are presented in Figure 6. 

The core of this network consists of 15 Facebook public groups with an average membership of size of 45,127.27 members (max 83,200, min 18,902). The peak of posting activity was in February/March 2022 or the beginning of the full-scale invasion of Ukraine (see Figure 7).

Despite the size of the group membership base, some content circulated by these groups reached millions of Facebook users. In Figure 8,  top viewed content (original video from @iamgroot TikTok user) is presented where left is Putin's Inspection of Zapad 2021 Drills Garners Over 2.6M Views and Strong Reactions on Social Media. Zapad 2021, a large-scale joint military exercise between Russia and Belarus, showcased advanced military tactics and equipment, emphasizing the strategic partnership and defense capabilities of both nations. In Figure 8 right, coordinated effort is shown: Identical Meme of Zelensky with satirical caption 'Dad, don't leave me... Russia has won, I don't need you anymore' shared simultaneously across multiple pro-Putin Facebook pages, highlighting a strategic and synchronized campaign. The posts have accumulated a total of 1.18M views, showcasing their extensive reach and impact.

All these groups show a longstanding predilection for manipulated images often in the form of easily photoshopped images (cheap fakes). With the advent of generative AI they started posting AI generated content to support the same narratives with an even lower effort (see Figure 9).

In Figure 9, three images, posted on various Pro-Putin Facebook groups, showcase a strategic campaign to manipulate social media engagement. These posts exemplify the use of AI-generated images and synchronized sharing tactics. This reflects patterns observed in the Stanford Internet Observatory report, where spammers and scammers leverage AI-generated images and coordinated posting tactics to manipulate social media engagement.

Countermeasures

Technological Advances

While there has been intense academic research on the topic of synthetic media (often referred to as deepfake detection), there is typically a large gap between results reported in lab settings/public benchmark datasets and the actual performance in the wild. The MeVer Synthetic/Deepfake detector (developed by CERTH with the support of projects such as AI4Media and vera.ai) is a tool consisting of several detection models for images and videos that attempt to address the practical limitations of the problem. A list of the detection models includes:

• GAN-based image detector and variants
• Diffusion-based image detector and variants
• Video face swapping detector
• Video face reenactment detector

The detector is integrated into the Verification plugin, a highly impactful tool with 130,000 users worldwide that has attracted a large community of users such as fact-checkers and journalists, researchers, academics, media scholars teaching media literacy, OSINT investigators, law and enforcement agencies and NGOs. 

Deep learning is also advancing in local image manipulation, replacing traditional methods such as splicing, inpainting, and copy-move. TruFor (Guillaro et al 2023) is a forensic framework capable of detecting and localizing various image manipulations, effectively identifying both cheapfakes and deepfakes. In Figure 10, the image has been locally modified in two regions using the prompt ''insert trees''. The localization map clearly shows that the TruFor method can correctly identify the two areas as forged.

In contrast, in an effort to achieve generalization and robustness, a method has been developed to transition to a completely different approach. This involves training models exclusively on real videos, with the aim of identifying manipulated videos based on their abnormal behavior. This has proven to be particularly effective when the identification of genuine faces is grounded in semantic features, like soft biometrics, which leads to Person-of-Interest (POI) based detection (Cozzolino et al. 2023).

In an effort to bring together many stakeholders in the area, the Meet the Future of AI event was organised last year by the Horizon Europe projects veraAI, AI4Trust, and TITAN along with the H2020 AI4Media. The event focused on all relevant issues and challenges around generative AI and tackling disinformation. As an outcome of that event, a White Paper was released on Generative AI and Disinformation: Recent Advances, Challenges, and Opportunities, which summarized recent trends in the area and provided some critical view on the challenges ahead and ways forward.

This year, 2024, following the success of the 2023 edition, the event was co-organised by the above projects, and additionally the newly launched AI4Debunk and AI-CODE. This year the event was focusing on relevant issues and challenges around generative AI for the public good and democracy. 

Research directions beyond technological countermeasures

While technology-driven responses to the problem of Generative AI disinformation have largely focused on synthetic media detection, which is indeed a very important pillar of addressing the problem, there are more aspects and potential countermeasures that should be explored:

• The reception of AI generated content, especially in news videos, has triggered concerns about how audiences are reacting to AI altered content or AI altered distribution.
• Most solutions so far rely on labeling of AI-generated content (e.g. as adopted by many platforms), which opens questions on standards of labeling as well as, again, the reception of these efforts by audiences. While labeling may be desirable, it is not the only measure to solve the problem. 
• A more holistic approach of the problem would be welcome, e.g. by addressing its amplification by the recommender systems or the social reaction. 
• More looking at incidents: the creation of taxonomies of TTPs/attack patterns where GenAI is used is a good starting point to continue more harmonized research on the issue.
• Media literacy should be an important pillar: AI technology must be properly understood to use it best and avoid generating harmful content. This includes initiatives aiming at educating audiences on how AI is impacting the consumption of online information. Media literacy and digital literacy should also be applied to how AI can be used accurately, assess and evaluate the biases of models and tools to detect AI content and counter disinformation.

References

Cozzolino, D., Pianese, A., Nießner, M., & Verdoliva, L. (2023). Audio-visual person-of-interest deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 943-952).

Guillaro, F., Cozzolino, D., Sud, A., Dufour, N., & Verdoliva, L. (2023). Trufor: Leveraging all-round clues for trustworthy image forgery detection and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 20606-20615).

Vykopal, I., Pikuliak, M., Ostermann, S., & Šimko, M. (2024). Generative Large Language Models in Automated Fact-Checking: A Survey. arXiv preprint arXiv:2407.02351.

Authors: Symeon Papadopoulos (CERTH-ITI), Olga Papadopoulou (CERTH-ITI), Alexandre Alaphilippe (EUDL), Fabio Giglietto (UNIURB) with contributions from individuals of selected vera.ai consortium partners

Editor: Anna Schild (DW)