Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer.

Background The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. Purpose To compare the performance of the LLMs ChatGPT and GPT-4 in data mining and labeling oncologic phenotypes from free-text CT reports on lung cancer by using user-defined prompts. Materials and Methods This retrospective study included patients who underwent lung cancer follow-up CT between September 2021 and March 2023. A subset of 25 reports was reserved for prompt engineering to instruct the LLMs in extracting lesion diameters, labeling metastatic disease, and assessing oncologic progression. This output was fed into a rule-based natural language processing pipeline to match ground truth annotations from four radiologists and derive performance metrics. The oncologic reasoning of LLMs was rated on a five-point Likert scale for factual correctness and accuracy. The occurrence of confabulations was recorded. Statistical analyses included Wilcoxon signed rank and McNemar tests. Results On 424 CT reports from 424 patients (mean age, 65 years ± 11 [SD]; 265 male), GPT-4 outperformed ChatGPT in extracting lesion parameters (98.6% vs 84.0%, P < .001), resulting in 96% correctly mined reports (vs 67% for ChatGPT, P < .001). GPT-4 achieved higher accuracy in identification of metastatic disease (98.1% [95% CI: 97.7, 98.5] vs 90.3% [95% CI: 89.4, 91.0]) and higher performance in generating correct labels for oncologic progression (F1 score, 0.96 [95% CI: 0.94, 0.98] vs 0.91 [95% CI: 0.89, 0.94]) (both P < .001). In oncologic reasoning, GPT-4 had higher Likert scale scores for factual correctness (4.3 vs 3.9) and accuracy (4.4 vs 3.3), with a lower rate of confabulation (1.7% vs 13.7%) than ChatGPT (all P < .001). Conclusion When using user-defined prompts, GPT-4 outperformed ChatGPT in extracting oncologic phenotypes from free-text CT reports on lung cancer and demonstrated better oncologic reasoning with fewer confabulations. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Hafezi-Nejad and Trivedi in this issue.

AI-Supported Comprehensive Detection and Quantification of Biomarkers of Subclinical Widespread Diseases at Chest CT for Preventive Medicine.

Automated image analysis plays an increasing role in radiology in detecting and quantifying image features outside of the perception of human eyes. Common AI-based approaches address a single medical problem, although patients often present with multiple interacting, frequently subclinical medical conditions. A holistic imaging diagnostics tool based on artificial intelligence (AI) has the potential of providing an overview of multi-system comorbidities within a single workflow. An interdisciplinary, multicentric team of medical experts and computer scientists designed a pipeline, comprising AI-based tools for the automated detection, quantification and characterization of the most common pulmonary, metabolic, cardiovascular and musculoskeletal comorbidities in chest computed tomography (CT). To provide a comprehensive evaluation of each patient, a multidimensional workflow was established with algorithms operating synchronously on a decentralized Joined Imaging Platform (JIP). The results of each patient are transferred to a dedicated database and summarized as a structured report with reference to available reference values and annotated sample images of detected pathologies. Hence, this tool allows for the comprehensive, large-scale analysis of imaging-biomarkers of comorbidities in chest CT, first in science and then in clinical routine. Moreover, this tool accommodates the quantitative analysis and classification of each pathology, providing integral diagnostic and prognostic value, and subsequently leading to improved preventive patient care and further possibilities for future studies.

Deep Learning-based Assessment of Oncologic Outcomes from Natural Language Processing of Structured Radiology Reports.

Purpose: To train a deep natural language processing (NLP) model, using data mined structured oncology reports (SOR), for rapid tumor response category (TRC) classification from free-text oncology reports (FTOR) and to compare its performance with human readers and conventional NLP algorithms. Materials and Methods: In this retrospective study, databases of three independent radiology departments were queried for SOR and FTOR dated from March 2018 to August 2021. An automated data mining and curation pipeline was developed to extract Response Evaluation Criteria in Solid Tumors-related TRCs for SOR for ground truth definition. The deep NLP bidirectional encoder representations from transformers (BERT) model and three feature-rich algorithms were trained on SOR to predict TRCs in FTOR. Models' F1 scores were compared against scores of radiologists, medical students, and radiology technologist students. Lexical and semantic analyses were conducted to investigate human and model performance on FTOR. Results: Oncologic findings and TRCs were accurately mined from 9653 of 12 833 (75.2%) queried SOR, yielding oncology reports from 10 455 patients (mean age, 60 years ± 14 [SD]; 5303 women) who met inclusion criteria. On 802 FTOR in the test set, BERT achieved better TRC classification results (F1, 0.70; 95% CI: 0.68, 0.73) than the best-performing reference linear support vector classifier (F1, 0.63; 95% CI: 0.61, 0.66) and technologist students (F1, 0.65; 95% CI: 0.63, 0.67), had similar performance to medical students (F1, 0.73; 95% CI: 0.72, 0.75), but was inferior to radiologists (F1, 0.79; 95% CI: 0.78, 0.81). Lexical complexity and semantic ambiguities in FTOR influenced human and model performance, revealing maximum F1 score drops of -0.17 and -0.19, respectively. Conclusion: The developed deep NLP model reached the performance level of medical students but not radiologists in curating oncologic outcomes from radiology FTOR.Keywords: Neural Networks, Computer Applications-Detection/Diagnosis, Oncology, Research Design, Staging, Tumor Response, Comparative Studies, Decision Analysis, Experimental Investigations, Observer Performance, Outcomes Analysis Supplemental material is available for this article. © RSNA, 2022.

Automatic airway analysis on multidetector computed tomography in cystic fibrosis: correlation with pulmonary function testing.

PURPOSE: To evaluate the fully automatic quantification of airway dimensions on chest multidetector computed tomography (MDCT) performed in cystic fibrosis (CF) patients. Airflow indices including predicted forced expiratory volume in 1 second (FEV1%) were used to study the impact on regional lung function. MATERIALS AND METHODS: MDCT data of patients with CF (14 children and 23 adults) and of control patients (11 children and 22 adults) were used to compute total diameter (TD), lumen area (LA), and wall thickness (WT) using dedicated software. Pulmonary function testing including FEV1% was performed in parallel and correlated with MDCT parameters in a generation-based analysis. RESULTS: TD was largely increased in CF patients (third-generation to fourth-generation airways in children, first to ninth in adults; P<0.05). LA remained unchanged, but WT was also larger in CF compared with controls (third generation to sixth generation in children, first to eleventh in adults; P<0.05). In adult CF patients significant negative correlations for TD, LA, and WT with FEV1% were found for intermediate airways (fifth to seventh generation; r=-0.7 to -0.9) but not in pediatric CF patients and controls. CONCLUSIONS: Automatic airway analysis succeeded in quantifying specific pathologies such as airway dilatation and wall thickening in CF patients at different ages. Moreover, our results indicate a shift in main airflow resistance to intermediate airways in cases of chronic CF. The objective computational parameters TD, LA, and WT should be considered for assessment and follow-up of CF airway disease.