The persistence and resilience of misinformation in the face of fact checking

A project by vera.ai, entitled “The Persistence and Resilience of Misinformation in the Face of Fact Checking”, was facilitated by Fabio Giglietto, Massimo Terenzi (University of Urbino Carlo Bo) and Richard Rogers (University of Amsterdam) at the Digital Media Initiative (DMI) Winter School at the University of Amsterdam. We are pleased to share with you the results, that enrich the scientific work of vera.ai's work package 4.4 which focuses on the impact of disinformation and is led by UvA. These outcomes were first presented to a team of Masters, PhD students and postdoc researchers in a poster session on Friday, January 12th, 2024.

The study is based on a dataset of 38,099 Facebook stories shared between 2017 and 2022, identified as problematic by Meta's third-party fact-checkers. A first interactive map shows news items grouped by linguistic cluster. To identify the topics, OpenAI's GPT models and Mistral AI's embedding tools were used in a three-step process: The stories were translated into English using GPT-3.5-turbo and embeddings were generated for all the story titles and descriptions using Mistral's APIs. The embeddings were then clustered into 87 distinct topics using the k-means algorithm, grouping stories by thematic similarity. The clustering was performed using both OpenAI and MistralAI APIs, showing clear consistency in the results of the two models, as displayed in this graph. Finally, GPT-4 was used to label the clusters. The resulting map allowed for the identification of clusters for various topics.

The quality of these labels was assessed on a stratified sample of 400 stories, divided among three separate groups of participants, who individually evaluated the stories on a five-level Likert scale. On average, the labels were rated as “Very Accurate” or “Accurate” in 75% of the cases.

Three different thematic focuses were selected, picking the topics that emerged in the cluster map.

One of these research spotlights (fig. 1) concerned consumer fraud scams on Facebook. It revealed a surprising homogeneity in the language of the scams, highlighting how these fraudulent schemes are particularly concentrated in the retail sector and tend to occur in specific temporal spikes. It was interesting to note that despite fact-checking measures these scams continue to proliferate on the platform. 
 

Figure 1: Facebook scams caseMassimo Terenzi, Fabio Giglietto (UNIURB)

 

A second focus (fig. 2) was on misinformation regarding Emmanuel Macron and the French government. It was found that the false news concentrated on recurring themes such as corruption and political plans. The investigation allowed for a detailed picture of the various waves of misinformation and how they respond to or influence certain political and social events.

Figure 2: health misinformation caseMassimo Terenzi, Fabio Giglietto (UNIURB)

 

The third area of interest (fig. 3) focused on health-related stories, where the examination of misinformation titles led to their categorization into esoteric, sensationalist, and authoritative styles, with the study particularly highlighting a predominance of sensationalist titles.

Figure 3: health misinformation caseMassimo Terenzi, Fabio Giglietto (UNIURB)

 

Author: Massimo Terenzi (UNIURB)

Editor: Anna Schild (DW)

vera.ai is co-funded by the European Commission under grant agreement ID 101070093, and the UK and Swiss authorities. This website reflects the views of the vera.ai consortium and respective contributors. The EU cannot be held responsible for any use which may be made of the information contained herein.