A scoping review of reporting gaps in FDA-approved AI medical devices.

Machine learning and artificial intelligence (AI/ML) models in healthcare may exacerbate health biases. Regulatory oversight is critical in evaluating the safety and effectiveness of AI/ML devices in clinical settings. We conducted a scoping review on the 692 FDA-approved AI/ML-enabled medical devices approved from 1995-2023 to examine transparency, safety reporting, and sociodemographic representation. Only 3.6% of approvals reported race/ethnicity, 99.1% provided no socioeconomic data. 81.6% did not report the age of study subjects. Only 46.1% provided comprehensive detailed results of performance studies; only 1.9% included a link to a scientific publication with safety and efficacy data. Only 9.0% contained a prospective study for post-market surveillance. Despite the growing number of market-approved medical devices, our data shows that FDA reporting data remains inconsistent. Demographic and socioeconomic characteristics are underreported, exacerbating the risk of algorithmic bias and health disparity.

The TRIPOD-LLM Statement: A Targeted Guideline For Reporting Large Language Models Use.

Large Language Models (LLMs) are rapidly being adopted in healthcare, necessitating standardized reporting guidelines. We present TRIPOD-LLM, an extension of the TRIPOD+AI statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion. The guidelines introduce a modular format accommodating various LLM research designs and tasks, with 14 main items and 32 subitems applicable across all categories. Developed through an expedited Delphi process and expert consensus, TRIPOD-LLM emphasizes transparency, human oversight, and task-specific performance reporting. We also introduce an interactive website ( https://tripod-llm.vercel.app/ ) facilitating easy guideline completion and PDF generation for submission. As a living document, TRIPOD-LLM will evolve with the field, aiming to enhance the quality, reproducibility, and clinical applicability of LLM research in healthcare through comprehensive reporting. COI: DSB: Editorial, unrelated to this work: Associate Editor of Radiation Oncology, HemOnc.org (no financial compensation); Research funding, unrelated to this work: American Association for Cancer Research; Advisory and consulting, unrelated to this work: MercurialAI. DDF: Editorial, unrelated to this work: Associate Editor of JAMIA, Editorial Board of Scientific Data, Nature; Funding, unrelated to this work: the intramural research program at the U.S. National Library of Medicine, National Institutes of Health. JWG: Editorial, unrelated to this work: Editorial Board of Radiology: Artificial Intelligence, British Journal of Radiology AI journal and NEJM AI. All other authors declare no conflicts of interest.

Applied artificial intelligence for global child health: Addressing biases and barriers.

Given the potential benefits of artificial intelligence and machine learning (AI/ML) within healthcare, it is critical to consider how these technologies can be deployed in pediatric research and practice. Currently, healthcare AI/ML has not yet adapted to the specific technical considerations related to pediatric data nor adequately addressed the specific vulnerabilities of children and young people (CYP) in relation to AI. While the greatest burden of disease in CYP is firmly concentrated in lower and middle-income countries (LMICs), existing applied pediatric AI/ML efforts are concentrated in a small number of high-income countries (HICs). In LMICs, use-cases remain primarily in the proof-of-concept stage. This narrative review identifies a number of intersecting challenges that pose barriers to effective AI/ML for CYP globally and explores the shifts needed to make progress across multiple domains. Child-specific technical considerations throughout the AI/ML lifecycle have been largely overlooked thus far, yet these can be critical to model effectiveness. Governance concerns are paramount, with suitable national and international frameworks and guidance required to enable the safe and responsible deployment of advanced technologies impacting the care of CYP and using their data. An ambitious vision for child health demands that the potential benefits of AI/Ml are realized universally through greater international collaboration, capacity building, strong oversight, and ultimately diffusing the AI/ML locus of power to empower researchers and clinicians globally. In order that AI/ML systems that do not exacerbate inequalities in pediatric care, teams researching and developing these technologies in LMICs must ensure that AI/ML research is inclusive of the needs and concerns of CYP and their caregivers. A broad, interdisciplinary, and human-centered approach to AI/ML is essential for developing tools for healthcare workers delivering care, such that the creation and deployment of ML is grounded in local systems, cultures, and clinical practice. Decisions to invest in developing and testing pediatric AI/ML in resource-constrained settings must always be part of a broader evaluation of the overall needs of a healthcare system, considering the critical building blocks underpinning effective, sustainable, and cost-efficient healthcare delivery for CYP.

Single-Lesion Skin Cancer Risk Stratification Triage Pathway.

Importance: Access to timely dermatologic care remains a challenge, especially for patients with new skin lesions. Assessing the efficiency of new triage pathways may assist in better resource allocation and shorter time to care. Objective: To evaluate whether a rule-based triage system was associated with better skin cancer risk stratification of patients and reduced wait times. Design, Setting, and Participants: This retrospective quality improvement cohort study of patients referred to Stanford University dermatology clinics was conducted between November 2017 and January 2023. A rules-based triage system based on a priori-determined high-risk lesion characteristics was implemented. Exposures: Referral reasons and risk factors of patients provided by their primary care physicians. Main Outcomes and Measures: Biopsy results of patients (diagnosis of any skin cancer and melanoma) at their visit or within 6 months after the visit. Regression models were used to assess the association between risk factors at referral and (1) biopsy outcomes and (2) time to first visit, adjusting for sociodemographic factors. Results: Among 37 478 patients (mean [SD] age, 54 (18) years; 21 292 women [57%]), the rates of aggregate biopsy, malignant biopsy specimens, and melanoma were comparable across patients seen after (n = 12 302) and before (n = 25 176) the implementation of the new triage pathway. Patients seen through the lesion pathway had a higher risk of having malignant biopsy results (adjusted risk ratio [aRR], 1.6; 95% CI, 1.4-1.9) and melanoma (aRR, 2.0; 95% CI, 1.2-3.2) than those not seen through the pathway. Lesions that were concerning to referring clinicians for skin cancer were associated with an increased risk of skin cancer (all skin cancer: aRR, 2.8; 95% CI, 2.2-3.5; melanoma: aRR, 2.02; 95% CI, 1.1-3.7). Patients in the 3 high-risk lesion groups were seen faster in the new triage pathway (mean reduction, 26 days; 95% CI, 18-34 days). Conclusions and Relevance: In this study, a new automated, rules-based referral pathway was implemented that expedited care for patients with high-risk skin cancer. This reform may have contributed to improving patient stratification, reducing the time from referral to first encounter, and maintaining accuracy in identifying malignant lesions. The findings highlight the potential to optimize clinical resource allocation by better risk stratification of referred patients.

Transparent medical image AI via an image-text foundation model grounded in medical literature.

Building trustworthy and transparent image-based medical artificial intelligence (AI) systems requires the ability to interrogate data and models at all stages of the development pipeline, from training models to post-deployment monitoring. Ideally, the data and associated AI systems could be described using terms already familiar to physicians, but this requires medical datasets densely annotated with semantically meaningful concepts. In the present study, we present a foundation model approach, named MONET (medical concept retriever), which learns how to connect medical images with text and densely scores images on concept presence to enable important tasks in medical AI development and deployment such as data auditing, model auditing and model interpretation. Dermatology provides a demanding use case for the versatility of MONET, due to the heterogeneity in diseases, skin tones and imaging modalities. We trained MONET based on 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, competitively with supervised models built on previously concept-annotated dermatology datasets of clinical images. We demonstrate how MONET enables AI transparency across the entire AI system development pipeline, from building inherently interpretable models to dataset and model auditing, including a case study dissecting the results of an AI clinical trial.

Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis.

The development of diagnostic tools for skin cancer based on artificial intelligence (AI) is increasing rapidly and will likely soon be widely implemented in clinical use. Even though the performance of these algorithms is promising in theory, there is limited evidence on the impact of AI assistance on human diagnostic decisions. Therefore, the aim of this systematic review and meta-analysis was to study the effect of AI assistance on the accuracy of skin cancer diagnosis. We searched PubMed, Embase, IEE Xplore, Scopus and conference proceedings for articles from 1/1/2017 to 11/8/2022. We included studies comparing the performance of clinicians diagnosing at least one skin cancer with and without deep learning-based AI assistance. Summary estimates of sensitivity and specificity of diagnostic accuracy with versus without AI assistance were computed using a bivariate random effects model. We identified 2983 studies, of which ten were eligible for meta-analysis. For clinicians without AI assistance, pooled sensitivity was 74.8% (95% CI 68.6-80.1) and specificity was 81.5% (95% CI 73.9-87.3). For AI-assisted clinicians, the overall sensitivity was 81.1% (95% CI 74.4-86.5) and specificity was 86.1% (95% CI 79.2-90.9). AI benefitted medical professionals of all experience levels in subgroup analyses, with the largest improvement among non-dermatologists. No publication bias was detected, and sensitivity analysis revealed that the findings were robust. AI in the hands of clinicians has the potential to improve diagnostic accuracy in skin cancer diagnosis. Given that most studies were conducted in experimental settings, we encourage future studies to further investigate these potential benefits in real-life settings.

Current State of Dermatology Mobile Applications With Artificial Intelligence Features.

Importance: With advancements in mobile technology and artificial intelligence (AI) methods, there has been a substantial surge in the availability of direct-to-consumer mobile applications (apps) claiming to aid in the assessment and management of diverse skin conditions. Despite widespread patient downloads, these apps exhibit limited evidence supporting their efficacy. Objective: To identify and characterize current English-language AI dermatology mobile apps available for download, focusing on aspects such as purpose, supporting evidence, regulatory status, clinician input, data privacy measures, and use of image data. Evidence Review: In this cross-sectional study, both Apple and Android mobile app stores were systematically searched for dermatology-related apps that use AI algorithms. Each app's purpose, target audience, evidence-based claims, algorithm details, data availability, clinician input during development, and data usage privacy policies were evaluated. Findings: A total of 909 apps were initially identified. Following the removal of 518 duplicates, 391 apps remained. Subsequent review excluded 350 apps due to nonmedical nature, non-English languages, absence of AI features, or unavailability, ultimately leaving 41 apps for detailed analysis. The findings revealed several concerning aspects of the current landscape of AI apps in dermatology. Notably, none of the apps were approved by the US Food and Drug Administration, and only 2 of the apps included disclaimers for the lack of regulatory approval. Overall, the study found that these apps lack supporting evidence, input from clinicians and/or dermatologists, and transparency in algorithm development, data usage, and user privacy. Conclusions and Relevance: This cross-sectional study determined that although AI dermatology mobile apps hold promise for improving access to care and patient outcomes, in their current state, they may pose harm due to potential risks, lack of consistent validation, and misleading user communication. Addressing challenges in efficacy, safety, and transparency through effective regulation, validation, and standardized evaluation criteria is essential to harness the benefits of these apps while minimizing risks.

The Promises and Perils of Foundation Models in Dermatology.

Foundation models (FM), which are large-scale artificial intelligence (AI) models that can complete a range of tasks, represent a paradigm shift in AI. These versatile models encompass large language models, vision-language models, and multimodal models. Although these models are often trained for broad tasks, they have been applied either out of the box or after additional fine tuning to tasks in medicine, including dermatology. From addressing administrative tasks to answering dermatology questions, these models are poised to have an impact on dermatology care delivery. As FMs become more ubiquitous in health care, it is important for clinicians and dermatologists to have a basic understanding of how these models are developed, what they are capable of, and what pitfalls exist. In this paper, we present a comprehensive yet accessible overview of the current state of FMs and summarize their current applications in dermatology, highlight their limitations, and discuss future developments in the field.

Deep learning-aided decision support for diagnosis of skin disease across skin tones.

Although advances in deep learning systems for image-based medical diagnosis demonstrate their potential to augment clinical decision-making, the effectiveness of physician-machine partnerships remains an open question, in part because physicians and algorithms are both susceptible to systematic errors, especially for diagnosis of underrepresented populations. Here we present results from a large-scale digital experiment involving board-certified dermatologists (n = 389) and primary-care physicians (n = 459) from 39 countries to evaluate the accuracy of diagnoses submitted by physicians in a store-and-forward teledermatology simulation. In this experiment, physicians were presented with 364 images spanning 46 skin diseases and asked to submit up to four differential diagnoses. Specialists and generalists achieved diagnostic accuracies of 38% and 19%, respectively, but both specialists and generalists were four percentage points less accurate for the diagnosis of images of dark skin as compared to light skin. Fair deep learning system decision support improved the diagnostic accuracy of both specialists and generalists by more than 33%, but exacerbated the gap in the diagnostic accuracy of generalists across skin tones. These results demonstrate that well-designed physician-machine partnerships can enhance the diagnostic accuracy of physicians, illustrating that success in improving overall diagnostic accuracy does not necessarily address bias.

Large Language Models in Medicine: The Potentials and Pitfalls : A Narrative Review.

Large language models (LLMs) are artificial intelligence models trained on vast text data to generate humanlike outputs. They have been applied to various tasks in health care, ranging from answering medical examination questions to generating clinical reports. With increasing institutional partnerships between companies producing LLMs and health systems, the real-world clinical application of these models is nearing realization. As these models gain traction, health care practitioners must understand what LLMs are, their development, their current and potential applications, and the associated pitfalls in a medical setting. This review, coupled with a tutorial, provides a comprehensive yet accessible overview of these areas with the aim of familiarizing health care professionals with the rapidly changing landscape of LLMs in medicine. Furthermore, the authors highlight active research areas in the field that promise to improve LLMs' usability in health care contexts.

Black Box Warning: Large Language Models and the Future of Infectious Diseases Consultation.

Large language models (LLMs) are artificial intelligence systems trained by deep learning algorithms to process natural language and generate text responses to user prompts. Some approach physician performance on a range of medical challenges, leading some proponents to advocate for their potential use in clinical consultation and prompting some consternation about the future of cognitive specialties. However, LLMs currently have limitations that preclude safe clinical deployment in performing specialist consultations, including frequent confabulations, lack of contextual awareness crucial for nuanced diagnostic and treatment plans, inscrutable and unexplainable training data and methods, and propensity to recapitulate biases. Nonetheless, considering the rapid improvement in this technology, growing calls for clinical integration, and healthcare systems that chronically undervalue cognitive specialties, it is critical that infectious diseases clinicians engage with LLMs to enable informed advocacy for how they should-and shouldn't-be used to augment specialist care.