Application of a Comprehensive Evaluation Framework to COVID-19 Studies: Systematic Review of Translational Aspects of Artificial Intelligence in Health Care.

Background: Despite immense progress in artificial intelligence (AI) models, there has been limited deployment in health care environments. The gap between potential and actual AI applications is likely due to the lack of translatability between controlled research environments (where these models are developed) and clinical environments for which the AI tools are ultimately intended. Objective: We previously developed the Translational Evaluation of Healthcare AI (TEHAI) framework to assess the translational value of AI models and to support successful transition to health care environments. In this study, we applied the TEHAI framework to the COVID-19 literature in order to assess how well translational topics are covered. Methods: A systematic literature search for COVID-19 AI studies published between December 2019 and December 2020 resulted in 3830 records. A subset of 102 (2.7%) papers that passed the inclusion criteria was sampled for full review. The papers were assessed for translational value and descriptive data collected by 9 reviewers (each study was assessed by 2 reviewers). Evaluation scores and extracted data were compared by a third reviewer for resolution of discrepancies. The review process was conducted on the Covidence software platform. Results: We observed a significant trend for studies to attain high scores for technical capability but low scores for the areas essential for clinical translatability. Specific questions regarding external model validation, safety, nonmaleficence, and service adoption received failed scores in most studies. Conclusions: Using TEHAI, we identified notable gaps in how well translational topics of AI models are covered in the COVID-19 clinical sphere. These gaps in areas crucial for clinical translatability could, and should, be considered already at the model development stage to increase translatability into real COVID-19 health care environments.

Effectiveness of nutrition and dietary interventions for people with serious mental illness: systematic review and meta-analysis.

OBJECTIVE: To review recent published trials of nutrition and dietary interventions for people with serious mental illness; to assess their effectiveness in improving metabolic syndrome risk factors. STUDY DESIGN: Systematic review and meta-analysis of randomised and non-randomised controlled trials of interventions with a nutrition/diet-related component delivered to people with serious mental illness, published 1 January 2010 - 6 September 2021. Primary outcomes were weight, body mass index (BMI), and waist circumference. Secondary outcomes were total serum cholesterol, low-density lipoprotein (LDL) and high-density lipoprotein (HDL) cholesterol, triglyceride, and blood glucose levels. DATA SOURCES: MEDLINE, EMBASE, PsycINFO, CINAHL, and CENTRAL databases. In addition, reference lists of relevant publications were examined for further additional studies. DATA SYNTHESIS: Twenty-five studies encompassing 26 intervention arms were included in our analysis. Eight studies were at low or some risk of bias, seventeen were deemed to be at high risk. Eight of seventeen intervention arms found statistically significant intervention effects on weight, ten of 24 on BMI, and seven of seventeen on waist circumference. The pooled effects of nutrition interventions on metabolic syndrome risk factors were statistically non-significant. However, we identified small size effects on weight for interventions delivered by dietitians (five studies; 262 intervention, 258 control participants; standardised mean difference [SMD], -0.28; 95% CI, -0.51 to -0.04) and interventions consisting of individual sessions only (three studies; 141 intervention, 134 control participants; SMD, -0.30; 95% CI, -0.54 to -0.06). CONCLUSIONS: We found only limited evidence for nutrition interventions improving metabolic syndrome risk factors in people with serious mental illness. However, they may be more effective when delivered on an individual basis or by dietitians. PROSPERO REGISTRATION: CRD42021235979 (prospective).

Evaluation framework to guide implementation of AI systems into healthcare settings.

OBJECTIVES: To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components. METHODS: To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed 'Translational Evaluation of Healthcare AI (TEHAI)'. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert. RESULTS: TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system. DISCUSSION: One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems. CONCLUSION: The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.