Can the crowd judge truthfulness? A longitudinal study on recent misinformation about COVID-19.

Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers' behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.

Leveraging Semantic Type Dependencies for Clinical Named Entity Recognition.

Previous work on clinical relation extraction from free-text sentences leveraged information about semantic types from clinical knowledge bases as a part of entity representations. In this paper, we exploit additional evidence by also making use of domain-specific semantic type dependencies. We encode the relation between a span of tokens matching a Unified Medical Language System (UMLS) concept and other tokens in the sentence. We implement our method and compare against different named entity recognition (NER) architectures (i.e., BiLSTM-CRF and BiLSTM-GCN-CRF) using different pre-trained clinical embeddings (i.e., BERT, BioBERT, UMLSBert). Our experimental results on clinical datasets show that in some cases NER effectiveness can be significantly improved by making use of domain-specific semantic type dependencies. Our work is also the first study generating a matrix encoding to make use of more than three dependencies in one pass for the NER task.

Novel insights into views towards H1N1 during the 2009 Pandemic: a thematic analysis of Twitter data.

BACKGROUND: Infectious disease outbreaks have the potential to cause a high number of fatalities and are a very serious public health risk. OBJECTIVES: Our aim was to utilise an indepth method to study a period of time where the H1N1 Pandemic of 2009 was at its peak. METHODS: A data set of n = 214 784 tweets was retrieved and filtered, and the method of thematic analysis was used to analyse the data. RESULTS: Eight key themes emerged from the analysis of data: emotion and feeling, health related information, general commentary and resources, media and health organisations, politics, country of origin, food, and humour and/or sarcasm. DISCUSSION: A major novel finding was that due to the name 'swine flu', Twitter users had the belief that pigs and pork could host and/or transmit the virus. Our paper also considered the methodological implications for the wider field of library and information science as well as specific implications for health information and library workers. CONCLUSIONS: Novel insights were derived on how users communicate about disease outbreaks on social media platforms. Our study also provides an innovative methodological contribution because it was found that by utilising an indepth method it was possible to extract greater insight into user communication.

Health Cards to Assist Decision Making in Consumer Health Search.

We investigate the effectiveness of health cards to assist decision making in Consumer Health Search (CHS). A health card is a concise presentation of a health concept shown along side search results to specific queries. We specifically focus on the decision making tasks of determining the health condition presented by a person and determining which action should be taken next with respect to the health condition. We explore two avenues for presenting health cards: a traditional single health card interface, and a novel multiple health cards interface. To validate the utility of health cards and their presentation interfaces, we conduct a laboratory user study where users are asked to solve the two decision making tasks for eight simulated scenarios. Our study makes the following contributions: (1) it proposes the novel multiple health card interface, which allows users to perform differential diagnoses, (2) it quantifies the impact of using health cards for assisting decision making in CHS, and (3) it determines the health card appraisal accuracy in the context of multiple health cards.