Automating hybrid collective intelligence in open-ended medical diagnostics.

Collective intelligence has emerged as a powerful mechanism to boost decision accuracy across many domains, such as geopolitical forecasting, investment, and medical diagnostics. However, collective intelligence has been mostly applied to relatively simple decision tasks (e.g., binary classifications). Applications in more open-ended tasks with a much larger problem space, such as emergency management or general medical diagnostics, are largely lacking, due to the challenge of integrating unstandardized inputs from different crowd members. Here, we present a fully automated approach for harnessing collective intelligence in the domain of general medical diagnostics. Our approach leverages semantic knowledge graphs, natural language processing, and the SNOMED CT medical ontology to overcome a major hurdle to collective intelligence in open-ended medical diagnostics, namely to identify the intended diagnosis from unstructured text. We tested our method on 1,333 medical cases diagnosed on a medical crowdsourcing platform: The Human Diagnosis Project. Each case was independently rated by ten diagnosticians. Comparing the diagnostic accuracy of single diagnosticians with the collective diagnosis of differently sized groups, we find that our method substantially increases diagnostic accuracy: While single diagnosticians achieved 46% accuracy, pooling the decisions of ten diagnosticians increased this to 76%. Improvements occurred across medical specialties, chief complaints, and diagnosticians' tenure levels. Our results show the life-saving potential of tapping into the collective intelligence of the global medical community to reduce diagnostic errors and increase patient safety.

A step-by-step researcher's guide to the use of an AI-based transformer in epidemiology: an exploratory analysis of ChatGPT using the STROBE checklist for observational studies.

Objective: This study aims at investigating how AI-based transformers can support researchers in designing and conducting an epidemiological study. To accomplish this, we used ChatGPT to reformulate the STROBE recommendations into a list of questions to be answered by the transformer itself. We then qualitatively evaluated the coherence and relevance of the transformer's outputs. Study design: Descriptive study. Methods: We first chose a study to be used as a basis for the simulation. We then used ChatGPT to transform each STROBE checklist's item into specific prompts. Each answer to the respective prompt was evaluated by independent researchers in terms of coherence and relevance. Results: The mean scores assigned to each prompt were heterogeneous. On average, for the coherence domain, the overall mean score was 3.6 out of 5.0, and for relevance it was 3.3 out of 5.0. The lowest scores were assigned to items belonging to the Methods section of the checklist. Conclusions: ChatGPT can be considered as a valuable support for researchers in conducting an epidemiological study, following internationally recognized guidelines and standards. It is crucial for the users to have knowledge on the subject and a critical mindset when evaluating the outputs. The potential benefits of AI in scientific research and publishing are undeniable, but it is crucial to address the risks, and the ethical and legal consequences associated with its use.

Using altmetrics for detecting impactful research in quasi-zero-day time-windows: the case of COVID-19.

On December 31st 2019, the World Health Organization China Country Office was informed of cases of pneumonia of unknown etiology detected in Wuhan City. The cause of the syndrome was a new type of coronavirus isolated on January 7th 2020 and named Severe Acute Respiratory Syndrome CoronaVirus 2 (SARS-CoV-2). SARS-CoV-2 is the cause of the coronavirus disease 2019 (COVID-19). Since January 2020 an ever increasing number of scientific works related to the new pathogen have appeared in literature. Identifying relevant research outcomes at very early stages is challenging. In this work we use COVID-19 as a use-case for investigating: (1) which tools and frameworks are mostly used for early scholarly communication; (2) to what extent altmetrics can be used to identify potential impactful research in tight (i.e. quasi-zero-day) time-windows. A literature review with rigorous eligibility criteria is performed for gathering a sample composed of scientific papers about SARS-CoV-2/COVID-19 appeared in literature in the tight time-window ranging from January 15th 2020 to February 24th 2020. This sample is used for building a knowledge graph that represents the knowledge about papers and indicators formally. This knowledge graph feeds a data analysis process which is applied for experimenting with altmetrics as impact indicators. We find moderate correlation among traditional citation count, citations on social media, and mentions on news and blogs. Additionally, correlation coefficients are not inflated by indicators associated with zero values, which are quite common at very early stages after an article has been published. This suggests there is a common intended meaning of the citational acts associated with aforementioned indicators. Then, we define a method, i.e. the Comprehensive Impact Score (CIS), that harmonises different indicators for providing a multi-dimensional impact indicator. CIS shows promising results as a tool for selecting relevant papers even in a tight time-window. Our results foster the development of automated frameworks aimed at helping the scientific community in identifying relevant work even in case of limited literature and observation time.

Adoption of Digital Technologies in Health Care During the COVID-19 Pandemic: Systematic Review of Early Scientific Literature.

BACKGROUND: The COVID-19 pandemic is favoring digital transitions in many industries and in society as a whole. Health care organizations have responded to the first phase of the pandemic by rapidly adopting digital solutions and advanced technology tools. OBJECTIVE: The aim of this review is to describe the digital solutions that have been reported in the early scientific literature to mitigate the impact of COVID-19 on individuals and health systems. METHODS: We conducted a systematic review of early COVID-19-related literature (from January 1 to April 30, 2020) by searching MEDLINE and medRxiv with appropriate terms to find relevant literature on the use of digital technologies in response to the pandemic. We extracted study characteristics such as the paper title, journal, and publication date, and we categorized the retrieved papers by the type of technology and patient needs addressed. We built a scoring rubric by cross-classifying the patient needs with the type of technology. We also extracted information and classified each technology reported by the selected articles according to health care system target, grade of innovation, and scalability to other geographical areas. RESULTS: The search identified 269 articles, of which 124 full-text articles were assessed and included in the review after screening. Most of the selected articles addressed the use of digital technologies for diagnosis, surveillance, and prevention. We report that most of these digital solutions and innovative technologies have been proposed for the diagnosis of COVID-19. In particular, within the reviewed articles, we identified numerous suggestions on the use of artificial intelligence (AI)-powered tools for the diagnosis and screening of COVID-19. Digital technologies are also useful for prevention and surveillance measures, such as contact-tracing apps and monitoring of internet searches and social media usage. Fewer scientific contributions address the use of digital technologies for lifestyle empowerment or patient engagement. CONCLUSIONS: In the field of diagnosis, digital solutions that integrate with traditional methods, such as AI-based diagnostic algorithms based both on imaging and clinical data, appear to be promising. For surveillance, digital apps have already proven their effectiveness; however, problems related to privacy and usability remain. For other patient needs, several solutions have been proposed, such as telemedicine or telehealth tools. These tools have long been available, but this historical moment may actually be favoring their definitive large-scale adoption. It is worth taking advantage of the impetus provided by the crisis; it is also important to keep track of the digital solutions currently being proposed to implement best practices and models of care in future and to adopt at least some of the solutions proposed in the scientific literature, especially in national health systems, which have proved to be particularly resistant to the digital transition in recent years.

Predicting the results of evaluation procedures of academics.

BACKGROUND: The 2010 reform of the Italian university system introduced the National Scientific Habilitation (ASN) as a requirement for applying to permanent professor positions. Since the CVs of the 59,149 candidates and the results of their assessments have been made publicly available, the ASN constitutes an opportunity to perform analyses about a nation-wide evaluation process. OBJECTIVE: The main goals of this paper are: (i) predicting the ASN results using the information contained in the candidates' CVs; (ii) identifying a small set of quantitative indicators that can be used to perform accurate predictions. APPROACH: Semantic technologies are used to extract, systematize and enrich the information contained in the applicants' CVs, and machine learning methods are used to predict the ASN results and to identify a subset of relevant predictors. RESULTS: For predicting the success in the role of associate professor, our best models using all and the top 15 predictors make accurate predictions (F-measure values higher than 0.6) in 88% and 88.6% of the cases, respectively. Similar results have been achieved for the role of full professor. EVALUATION: The proposed approach outperforms the other models developed to predict the results of researchers' evaluation procedures. CONCLUSIONS: Such results allow the development of an automated system for supporting both candidates and committees in the future ASN sessions and other scholars' evaluation procedures.