Performance of Large Language Models on Medical Oncology Examination Questions.

Importance: Large language models (LLMs) recently developed an unprecedented ability to answer questions. Studies of LLMs from other fields may not generalize to medical oncology, a high-stakes clinical setting requiring rapid integration of new information. Objective: To evaluate the accuracy and safety of LLM answers on medical oncology examination questions. Design, Setting, and Participants: This cross-sectional study was conducted between May 28 and October 11, 2023. The American Society of Clinical Oncology (ASCO) Oncology Self-Assessment Series on ASCO Connection, the European Society of Medical Oncology (ESMO) Examination Trial questions, and an original set of board-style medical oncology multiple-choice questions were presented to 8 LLMs. Main Outcomes and Measures: The primary outcome was the percentage of correct answers. Medical oncologists evaluated the explanations provided by the best LLM for accuracy, classified the types of errors, and estimated the likelihood and extent of potential clinical harm. Results: Proprietary LLM 2 correctly answered 125 of 147 questions (85.0%; 95% CI, 78.2%-90.4%; P < .001 vs random answering). Proprietary LLM 2 outperformed an earlier version, proprietary LLM 1, which correctly answered 89 of 147 questions (60.5%; 95% CI, 52.2%-68.5%; P < .001), and the best open-source LLM, Mixtral-8x7B-v0.1, which correctly answered 87 of 147 questions (59.2%; 95% CI, 50.0%-66.4%; P < .001). The explanations provided by proprietary LLM 2 contained no or minor errors for 138 of 147 questions (93.9%; 95% CI, 88.7%-97.2%). Incorrect responses were most commonly associated with errors in information retrieval, particularly with recent publications, followed by erroneous reasoning and reading comprehension. If acted upon in clinical practice, 18 of 22 incorrect answers (81.8%; 95% CI, 59.7%-94.8%) would have a medium or high likelihood of moderate to severe harm. Conclusions and Relevance: In this cross-sectional study of the performance of LLMs on medical oncology examination questions, the best LLM answered questions with remarkable performance, although errors raised safety concerns. These results demonstrated an opportunity to develop and evaluate LLMs to improve health care clinician experiences and patient care, considering the potential impact on capabilities and safety.

Multiplex imaging reveals spatially resolved DNA-damage response neighborhoods in TP53-mutated myelodysplastic neoplasms.

While increased DNA damage is a well-described feature of myelodysplastic syndrome (MDS) and acute myeloid leukemia (AML), it is unclear whether all lineages and all regions of the marrow are homogeneously affected. In this study, we performed immunohistochemistry on formalin-fixed, paraffin-embedded whole-section bone marrow biopsies using a well-established antibody to detect pH2A.X (phosphorylated histone variant H2A.X) that recognizes DNA double-strand breaks. Focusing on TP53-mutated and complex karyotype MDS/AML, we find a greater pH2A.X+ DNA damage burden compared to TP53 wild-type neoplastic cases and non-neoplastic controls. To understand how double-strand breaks vary between lineages and spatially in TP53-mutated specimens, we applied a low-multiplex immunofluorescence staining and spatial analysis protocol to visualize pH2A.X+ cells with p53 protein staining and lineage markers. pH2A.X marked predominantly mid- to late-stage erythroids, whereas early erythroids and CD34+ blasts were relatively spared. In a prototypical example, these pH2A.X+ erythroids were organized locally as distinct colonies, and each colony displayed pH2A.X+ puncta at a synchronous level. This highly coordinated immunophenotypic expression was also seen for p53 protein staining and among presumed early myeloid colonies. Neighborhood clustering analysis showed distinct marrow regions differentially enriched in pH2A.X+/p53+ erythroid or myeloid colonies, indicating spatial heterogeneity of DNA-damage response and p53 protein expression. The lineage and architectural context within which DNA damage phenotype and oncogenic protein are expressed is relevant to current therapeutic developments that leverage macrophage phagocytosis to remove leukemic cells in part due to irreparable DNA damage. © 2024 The Pathological Society of Great Britain and Ireland.

Impact of Automated Prognostication on Traumatic Brain Injury Care: A Focus Group Study.

BACKGROUND: Prognosticating outcomes for traumatic brain injury (TBI) patients is challenging due to the required specialized skills and variability among clinicians. Recent attempts to standardize TBI prognosis have leveraged machine learning (ML) methodologies. This study evaluates the necessity and influence of ML-assisted TBI prognostication through healthcare professionals' perspectives via focus group discussions. METHODS: Two virtual focus groups included ten key TBI care stakeholders (one neurosurgeon, two emergency clinicians, one internist, two radiologists, one registered nurse, two researchers in ML and healthcare and one patient representative). They answered six open-ended questions about their perceptions and potential ML use in TBI prognostication. Transcribed focus group discussions were thematically analyzed using qualitative data analysis software. RESULTS: The study captured diverse perceptions and interests in TBI prognostication across clinical specialties. Notably, certain clinicians who currently do not prognosticate expressed an interest in doing so independently provided they had access to ML support. Concerns included ML's accuracy and the need for proficient ML researchers in clinical settings. The consensus suggested using ML as a secondary consultation tool and promoting collaboration with internal or external research resources. Participants believed ML prognostication could enhance disposition planning and standardize care regardless of clinician expertise or injury severity. There was no evidence of perceived bias or interference during the discussions. CONCLUSION: Our findings revealed an overall positive attitude toward ML-based prognostication. Despite raising multiple concerns, the focus group discussions were particularly valuable in underscoring the potential of ML in democratizing and standardizing TBI prognosis practices.

Development of a Multimodal Machine Learning-Based Prognostication Model for Traumatic Brain Injury Using Clinical Data and Computed Tomography Scans: A CENTER-TBI and CINTER-TBI Study.

Computed tomography (CT) is an important imaging modality for guiding prognostication in patients with traumatic brain injury (TBI). However, because of the specialized expertise necessary, timely and dependable TBI prognostication based on CT imaging remains challenging. This study aimed to enhance the efficiency and reliability of TBI prognostication by employing machine learning (ML) techniques on CT images. A retrospective analysis was conducted on the Collaborative European NeuroTrauma Effectiveness Research in TBI (CENTER-TBI) data set (n = 1016). An ML-driven binary classifier was developed to predict favorable or unfavorable outcomes at 6 months post-injury. The prognostic performance was assessed using the area under the curve (AUC) over fivefold cross-validation and compared with conventional models that depend on clinical variables and CT scoring systems. An external validation was performed using the Comparative Indian Neurotrauma Effectiveness Research in Traumatic Brain Injury (CINTER-TBI) data set (n = 348). The developed model achieved superior performance without the necessity for manual CT assessments (AUC = 0.846 [95% CI: 0.843-0.849]) compared with the model based on the clinical and laboratory variables (AUC = 0.817 [95% CI: 0.814-0.820]) and established CT scoring systems requiring manual interpretations (AUC = 0.829 [95% CI: 0.826-0.832] for Marshall and 0.838 [95% CI: 0.835-0.841] for International Mission for Prognosis and Analysis of Clinical Trials in TBI [IMPACT]). The external validation demonstrated the prognostic capacity of the developed model to be significantly better (AUC = 0.859 [95% CI: 0.857-0.862]) than the model using clinical variables (AUC = 0.809 [95% CI: 0.798-0.820]). This study established an ML-based model that provides efficient and reliable TBI prognosis based on CT scans, with potential implications for earlier intervention and improved patient outcomes.

Machine learning in computational histopathology: Challenges and opportunities.

Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.

Automated screening of computed tomography using weakly supervised anomaly detection.

BACKGROUND: Current artificial intelligence studies for supporting CT screening tasks depend on either supervised learning or detecting anomalies. However, the former involves a heavy annotation workload owing to requiring many slice-wise annotations (ground truth labels); the latter is promising, but while it reduces the annotation workload, it often suffers from lower performance. This study presents a novel weakly supervised anomaly detection (WSAD) algorithm trained based on scan-wise normal and anomalous annotations to provide better performance than conventional methods while reducing annotation workload. METHODS: Based on surveillance video anomaly detection methodology, feature vectors representing each CT slice were trained on an AR-Net-based convolutional network using a dynamic multiple-instance learning loss and a center loss function. The following two publicly available CT datasets were retrospectively analyzed: the RSNA brain hemorrhage dataset (normal scans: 12,862; scans with intracranial hematoma: 8882) and COVID-CT set (normal scans: 282; scans with COVID-19: 95). RESULTS: Anomaly scores of each slice were successfully predicted despite inaccessibility to any slice-wise annotations. Slice-level area under the curve (AUC), sensitivity, specificity, and accuracy from the brain CT dataset were 0.89, 0.85, 0.78, and 0.79, respectively. The proposed method reduced the number of annotations in the brain dataset by 97.1% compared to an ordinary slice-level supervised learning method. CONCLUSION: This study demonstrated a significant annotation reduction in identifying anomalous CT slices compared to a supervised learning approach. The effectiveness of the proposed WSAD algorithm was verified through higher AUC than existing anomaly detection techniques.

Automated identification and quantification of traumatic brain injury from CT scans: Are we there yet?

BACKGROUND: The purpose of this study was to conduct a systematic review for understanding the availability and limitations of artificial intelligence (AI) approaches that could automatically identify and quantify computed tomography (CT) findings in traumatic brain injury (TBI). METHODS: Systematic review, in accordance with PRISMA 2020 and SPIRIT-AI extension guidelines, with a search of 4 databases (Medline, Embase, IEEE Xplore, and Web of Science) was performed to find AI studies that automated the clinical tasks for identifying and quantifying CT findings of TBI-related abnormalities. RESULTS: A total of 531 unique publications were reviewed, which resulted in 66 articles that met our inclusion criteria. The following components for identification and quantification regarding TBI were covered and automated by existing AI studies: identification of TBI-related abnormalities; classification of intracranial hemorrhage types; slice-, pixel-, and voxel-level localization of hemorrhage; measurement of midline shift; and measurement of hematoma volume. Automated identification of obliterated basal cisterns was not investigated in the existing AI studies. Most of the AI algorithms were based on deep neural networks that were trained on 2- or 3-dimensional CT imaging datasets. CONCLUSION: We identified several important TBI-related CT findings that can be automatically identified and quantified with AI. A combination of these techniques may provide useful tools to enhance reproducibility of TBI identification and quantification by supporting radiologists and clinicians in their TBI assessments and reducing subjective human factors.

Representation Learning Approaches to Detect False Arrhythmia Alarms from ECG Dynamics.

The high rate of intensive care unit false arrhythmia alarms can lead to disruption of care and slow response time due to desensitization of clinical staff. We study the use of machine learning models to detect false ventricular tachycardia (v-tach) alarms using ECG waveform recordings. We propose using a Supervised Denoising Autoencoder (SDAE) to detect false alarms using a low-dimensional representation of ECG dynamics learned by minimizing a combined reconstruction and classification loss. We evaluate our algorithms on the PhysioNet Challenge 2015 dataset, containing over 500 records (over 300 training and 200 testing) with v-tach alarms. Our results indicate that using the SDAE on Fast Fourier Transformed (FFT) ECG at a beat-by-beat level outperforms several competitive baselines on the task of v-tach false alarm classification. We show that it is important to exploit the underlying known physiological structure using beat-by-beat frequency distribution from multiple cardiac cycles of the ECG waveforms to obtain competitive results and improve over previous entries from the 2015 PhysioNet Challenge.