Medical Misinformation in Polish on the World Wide Web During the COVID-19 Pandemic Period: Infodemiology Study.

BACKGROUND: Although researchers extensively study the rapid generation and spread of misinformation about the novel coronavirus during the pandemic, numerous other health-related topics are contaminating the internet with misinformation that have not received as much attention. OBJECTIVE: This study aims to gauge the reach of the most popular medical content on the World Wide Web, extending beyond the confines of the pandemic. We conducted evaluations of subject matter and credibility for the years 2021 and 2022, following the principles of evidence-based medicine with assessments performed by experienced clinicians. METHODS: We used 274 keywords to conduct web page searches through the BuzzSumo Enterprise Application. These keywords were chosen based on medical topics derived from surveys administered to medical practitioners. The search parameters were confined to 2 distinct date ranges: (1) January 1, 2021, to December 31, 2021; (2) January 1, 2022, to December 31, 2022. Our searches were specifically limited to web pages in the Polish language and filtered by the specified date ranges. The analysis encompassed 161 web pages retrieved in 2021 and 105 retrieved in 2022. Each web page underwent scrutiny by a seasoned doctor to assess its credibility, aligning with evidence-based medicine standards. Furthermore, we gathered data on social media engagements associated with the web pages, considering platforms such as Facebook, Pinterest, Reddit, and Twitter. RESULTS: In 2022, the prevalence of unreliable information related to COVID-19 saw a noteworthy decline compared to 2021. Specifically, the percentage of noncredible web pages discussing COVID-19 and general vaccinations decreased from 57% (43/76) to 24% (6/25) and 42% (10/25) to 30% (3/10), respectively. However, during the same period, there was a considerable uptick in the dissemination of untrustworthy content on social media pertaining to other medical topics. The percentage of noncredible web pages covering cholesterol, statins, and cardiology rose from 11% (3/28) to 26% (9/35) and from 18% (5/28) to 26% (6/23), respectively. CONCLUSIONS: Efforts undertaken during the COVID-19 pandemic to curb the dissemination of misinformation seem to have yielded positive results. Nevertheless, our analysis suggests that these interventions need to be consistently implemented across both established and emerging medical subjects. It appears that as interest in the pandemic waned, other topics gained prominence, essentially "filling the vacuum" and necessitating ongoing measures to address misinformation across a broader spectrum of health-related subjects.

Improving medical experts' efficiency of misinformation detection: an exploratory study.

Fighting medical disinformation in the era of the pandemic is an increasingly important problem. Today, automatic systems for assessing the credibility of medical information do not offer sufficient precision, so human supervision and the involvement of medical expert annotators are required. Our work aims to optimize the utilization of medical experts' time. We also equip them with tools for semi-automatic initial verification of the credibility of the annotated content. We introduce a general framework for filtering medical statements that do not require manual evaluation by medical experts, thus focusing annotation efforts on non-credible medical statements. Our framework is based on the construction of filtering classifiers adapted to narrow thematic categories. This allows medical experts to fact-check and identify over two times more non-credible medical statements in a given time interval without applying any changes to the annotation flow. We verify our results across a broad spectrum of medical topic areas. We perform quantitative, as well as exploratory analysis on our output data. We also point out how those filtering classifiers can be modified to provide experts with different types of feedback without any loss of performance.

Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training Data Sets for Machine Learning.

BACKGROUND: The spread of false medical information on the web is rapidly accelerating. Establishing the credibility of web-based medical information has become a pressing necessity. Machine learning offers a solution that, when properly deployed, can be an effective tool in fighting medical misinformation on the web. OBJECTIVE: The aim of this study is to present a comprehensive framework for designing and curating machine learning training data sets for web-based medical information credibility assessment. We show how to construct the annotation process. Our main objective is to support researchers from the medical and computer science communities. We offer guidelines on the preparation of data sets for machine learning models that can fight medical misinformation. METHODS: We begin by providing the annotation protocol for medical experts involved in medical sentence credibility evaluation. The protocol is based on a qualitative study of our experimental data. To address the problem of insufficient initial labels, we propose a preprocessing pipeline for the batch of sentences to be assessed. It consists of representation learning, clustering, and reranking. We call this process active annotation. RESULTS: We collected more than 10,000 annotations of statements related to selected medical subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, and food allergy testing) for less than US $7000 by employing 9 highly qualified annotators (certified medical professionals), and we release this data set to the general public. We developed an active annotation framework for more efficient annotation of noncredible medical statements. The application of qualitative analysis resulted in a better annotation protocol for our future efforts in data set creation. CONCLUSIONS: The results of the qualitative analysis support our claims of the efficacy of the presented method.