Evaluation of GPT Large Language Model Performance on RSNA 2023 Case of the Day Questions.

Background GPT-4V (GPT-4 with vision, ChatGPT; OpenAI) has shown impressive performance in several medical assessments. However, few studies have assessed its performance in interpreting radiologic images. Purpose To assess and compare the accuracy of GPT-4V in assessing radiologic cases with both images and textual context to that of radiologists and residents, to assess if GPT-4V assistance improves human accuracy, and to assess and compare the accuracy of GPT-4V with that of image-only or text-only inputs. Materials and Methods Seventy-two Case of the Day questions at the RSNA 2023 Annual Meeting were curated in this observer study. Answers from GPT-4V were obtained between November 26 and December 10, 2023, with the following inputs for each question: image only, text only, and both text and images. Five radiologists and three residents also answered the questions in an "open book" setting. For the artificial intelligence (AI)-assisted portion, the radiologists and residents were provided with the outputs of GPT-4V. The accuracy of radiologists and residents, both with and without AI assistance, was analyzed using a mixed-effects linear model. The accuracies of GPT-4V with different input combinations were compared by using the McNemar test. P < .05 was considered to indicate a significant difference. Results The accuracy of GPT-4V was 43% (31 of 72; 95% CI: 32, 55). Radiologists and residents did not significantly outperform GPT-4V in either imaging-dependent (59% and 56% vs 39%; P = .31 and .52, respectively) or imaging-independent (76% and 63% vs 70%; both P = .99) cases. With access to GPT-4V responses, there was no evidence of improvement in the average accuracy of the readers. The accuracy obtained by GPT-4V with text-only and image-only inputs was 50% (35 of 70; 95% CI: 39, 61) and 38% (26 of 69; 95% CI: 27, 49), respectively. Conclusion The radiologists and residents did not significantly outperform GPT-4V. Assistance from GPT-4V did not help human raters. GPT-4V relied on the textual context for its outputs. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Katz in this issue.

Plant growth-promoting rhizobacterial secondary metabolites in augmenting heavy metal(loid) phytoremediation: An integrated green in situ ecorestorative technology.

In recent times, increased geogenic and human-centric activities have caused significant heavy metal(loid) (HM) contamination of soil, adversely impacting environmental, plant, and human health. Phytoremediation is an evolving, cost-effective, environment-friendly, in situ technology that employs indigenous/exotic plant species as natural purifiers to remove toxic HM(s) from deteriorated ambient soil. Interestingly, the plant's rhizomicrobiome is pivotal in promoting overall plant nutrition, health, and phytoremediation. Certain secondary metabolites produced by plant growth-promoting rhizobacteria (PGPR) directly participate in HM bioremediation through chelation/mobilization/sequestration/bioadsorption/bioaccumulation, thus altering metal(loid) bioavailability for their uptake, accumulation, and translocation by plants. Moreover, the metallotolerance of the PGPR and the host plant is another critical factor for the successful phytoremediation of metal(loid)-polluted soil. Among the phytotechniques available for HM remediation, phytoextraction/phytoaccumulation (HM mobilization, uptake, and accumulation within the different plant tissues) and phytosequestration/phytostabilization (HM immobilization within the soil) have gained momentum in recent years. Natural metal(loid)-hyperaccumulating plants have the potential to assimilate increased levels of metal(loid)s, and several such species have already been identified as potential candidates for HM phytoremediation. Furthermore, the development of transgenic rhizobacterial and/or plant strains with enhanced environmental adaptability and metal(loid) uptake ability using genetic engineering might open new avenues in PGPR-assisted phytoremediation technologies. With the use of the Geographic Information System (GIS) for identifying metal(loid)-impacted lands and an appropriate combination of normal/transgenic (hyper)accumulator plant(s) and rhizobacterial inoculant(s), it is possible to develop efficient integrated phytobial remediation strategies in boosting the clean-up process over vast regions of HM-contaminated sites and eventually restore ecosystem health.

Weakly supervised detection of pheochromocytomas and paragangliomas in CT using noisy data.

Pheochromocytomas and Paragangliomas (PPGLs) are rare adrenal and extra-adrenal tumors that have metastatic potential. Management of patients with PPGLs mainly depends on the makeup of their genetic cluster: SDHx, VHL/EPAS1, kinase, and sporadic. CT is the preferred modality for precise localization of PPGLs, such that their metastatic progression can be assessed. However, the variable size, morphology, and appearance of these tumors in different anatomical regions can pose challenges for radiologists. Since radiologists must routinely track changes across patient visits, manual annotation of PPGLs is quite time-consuming and cumbersome to do across all axial slices in a CT volume. As such, PPGLs are only weakly annotated on axial slices by radiologists in the form of RECIST measurements. To ameliorate the manual effort spent by radiologists, we propose a method for the automated detection of PPGLs in CT via a proxy segmentation task. Weak 3D annotations (derived from 2D bounding boxes) were used to train both 2D and 3D nnUNet models to detect PPGLs via segmentation. We evaluated our approaches on an in-house dataset comprised of chest-abdomen-pelvis CTs of 255 patients with confirmed PPGLs. On a test set of 53 CT volumes, our 3D nnUNet model achieved a detection precision of 70% and sensitivity of 64.1%, and outperformed the 2D model that obtained a precision of 52.7% and sensitivity of 27.5% (p< 0.05). SDHx and sporadic genetic clusters achieved the highest precisions of 73.1% and 72.7% respectively. Our state-of-the art findings highlight the promising nature of the challenging task of automated PPGL detection.

A Comparison of CT-Based Pancreatic Segmentation Deep Learning Models.

RATIONALE AND OBJECTIVES: Pancreas segmentation accuracy at CT is critical for the identification of pancreatic pathologies and is essential for the development of imaging biomarkers. Our objective was to benchmark the performance of five high-performing pancreas segmentation models across multiple metrics stratified by scan and patient/pancreatic characteristics that may affect segmentation performance. MATERIALS AND METHODS: In this retrospective study, PubMed and ArXiv searches were conducted to identify pancreas segmentation models which were then evaluated on a set of annotated imaging datasets. Results (Dice score, Hausdorff distance [HD], average surface distance [ASD]) were stratified by contrast status and quartiles of peri-pancreatic attenuation (5 mm region around pancreas). Multivariate regression was performed to identify imaging characteristics and biomarkers (n = 9) that were significantly associated with Dice score. RESULTS: Five pancreas segmentation models were identified: Abdomen Atlas [AAUNet, AASwin, trained on 8448 scans], TotalSegmentator [TS, 1204 scans], nnUNetv1 [MSD-nnUNet, 282 scans], and a U-Net based model for predicting diabetes [DM-UNet, 427 scans]. These were evaluated on 352 CT scans (30 females, 25 males, 297 sex unknown; age 58 ± 7 years [ ± 1 SD], 327 age unknown) from 2000-2023. Overall, TS, AAUNet, and AASwin were the best performers, Dice= 80 ± 11%, 79 ± 16%, and 77 ± 18%, respectively (pairwise Sidak test not-significantly different). AASwin and MSD-nnUNet performed worse (for all metrics) on non-contrast scans (vs contrast, P < .001). The worst performer was DM-UNet (Dice=67 ± 16%). All algorithms except TS showed lower Dice scores with increasing peri-pancreatic attenuation (P < .01). Multivariate regression showed non-contrast scans, (P < .001; MSD-nnUNet), smaller pancreatic length (P = .005, MSD-nnUNet), and height (P = .003, DM-UNet) were associated with lower Dice scores. CONCLUSION: The convolutional neural network-based models trained on a diverse set of scans performed best (TS, AAUnet, and AASwin). TS performed equivalently to AAUnet and AASwin with only 13% of the training set size (8488 vs 1204 scans). Though trained on the same dataset, a transformer network (AASwin) had poorer performance on non-contrast scans whereas its convolutional network counterpart (AAUNet) did not. This study highlights how aggregate assessment metrics of pancreatic segmentation algorithms seen in other literature are not enough to capture differential performance across common patient and scanning characteristics in clinical populations.

Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports.

In radiology, Artificial Intelligence (AI) has significantly advanced report generation, but automatic evaluation of these AI-produced reports remains challenging. Current metrics, such as Conventional Natural Language Generation (NLG) and Clinical Efficacy (CE), often fall short in capturing the semantic intricacies of clinical contexts or overemphasize clinical details, undermining report clarity. To overcome these issues, our proposed method synergizes the expertise of professional radiologists with Large Language Models (LLMs), like GPT-3.5 and GPT-4. Utilizing In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning, our approach aligns LLM evaluations with radiologist standards, enabling detailed comparisons between human and AI-generated reports. This is further enhanced by a Regression model that aggregates sentence evaluation scores. Experimental results show that our "Detailed GPT-4 (5-shot)" model achieves a 0.48 score, outperforming the METEOR metric by 0.19, while our "Regressed GPT-4" model shows even greater alignment with expert evaluations, exceeding the best existing metric by a 0.35 margin. Moreover, the robustness of our explanations has been validated through a thorough iterative strategy. We plan to publicly release annotations from radiology experts, setting a new standard for accuracy in future assessments. This underscores the potential of our approach in enhancing the quality assessment of AI-driven medical reports.

MRISegmentator-Abdomen: A Fully Automated Multi-Organ and Structure Segmentation Tool for T1-weighted Abdominal MRI.

BACKGROUND: Segmentation of organs and structures in abdominal MRI is useful for many clinical applications, such as disease diagnosis and radiotherapy. Current approaches have focused on delineating a limited set of abdominal structures (13 types). To date, there is no publicly available abdominal MRI dataset with voxel-level annotations of multiple organs and structures. Consequently, a segmentation tool for multi-structure segmentation is also unavailable. METHODS: We curated a T1-weighted abdominal MRI dataset consisting of 195 patients who underwent imaging at National Institutes of Health (NIH) Clinical Center. The dataset comprises of axial pre-contrast T1, arterial, venous, and delayed phases for each patient, thereby amounting to a total of 780 series (69,248 2D slices). Each series contains voxel-level annotations of 62 abdominal organs and structures. A 3D nnUNet model, dubbed as MRISegmentator-Abdomen (MRISegmentator in short), was trained on this dataset, and evaluation was conducted on an internal test set and two large external datasets: AMOS22 and Duke Liver. The predicted segmentations were compared against the ground-truth using the Dice Similarity Coefficient (DSC) and Normalized Surface Distance (NSD). FINDINGS: MRISegmentator achieved an average DSC of 0.861$\pm$0.170 and a NSD of 0.924$\pm$0.163 in the internal test set. On the AMOS22 dataset, MRISegmentator attained an average DSC of 0.829$\pm$0.133 and a NSD of 0.908$\pm$0.067. For the Duke Liver dataset, an average DSC of 0.933$\pm$0.015 and a NSD of 0.929$\pm$0.021 was obtained. INTERPRETATION: The proposed MRISegmentator provides automatic, accurate, and robust segmentations of 62 organs and structures in T1-weighted abdominal MRI sequences. The tool has the potential to accelerate research on various clinical topics, such as abnormality detection, radiotherapy, disease classification among others.

Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification.

In chest X-ray (CXR) image analysis, rule-based systems are usually employed to extract labels from reports for dataset releases. However, there is still room for improvement in label quality. These labelers typically output only presence labels, sometimes with binary uncertainty indicators, which limits their usefulness. Supervised deep learning models have also been developed for report labeling but lack adaptability, similar to rule-based systems. In this work, we present MAPLEZ (Medical report Annotations with Privacy-preserving Large language model using Expeditious Zero shot answers), a novel approach leveraging a locally executable Large Language Model (LLM) to extract and enhance findings labels on CXR reports. MAPLEZ extracts not only binary labels indicating the presence or absence of a finding but also the location, severity, and radiologists' uncertainty about the finding. Over eight abnormalities from five test sets, we show that our method can extract these annotations with an increase of 3.6 percentage points (pp) in macro F1 score for categorical presence annotations and more than 20 pp increase in F1 score for the location annotations over competing labelers. Additionally, using the combination of improved annotations and multi-type annotations in classification supervision, we demonstrate substantial advancements in model quality, with an increase of 1.1 pp in AUROC over models trained with annotations from the best alternative approach. We share code and annotations.

Automated Classification of Body MRI Sequence Type Using Convolutional Neural Networks.

Multi-parametric MRI of the body is routinely acquired for the identification of abnormalities and diagnosis of diseases. However, a standard naming convention for the MRI protocols and associated sequences does not exist due to wide variations in imaging practice at institutions and myriad MRI scanners from various manufacturers being used for imaging. The intensity distributions of MRI sequences differ widely as a result, and there also exists information conflicts related to the sequence type in the DICOM headers. At present, clinician oversight is necessary to ensure that the correct sequence is being read and used for diagnosis. This poses a challenge when specific series need to be considered for building a cohort for a large clinical study or for developing AI algorithms. In order to reduce clinician oversight and ensure the validity of the DICOM headers, we propose an automated method to classify the 3D MRI sequence acquired at the levels of the chest, abdomen, and pelvis. In our pilot work, our 3D DenseNet-121 model achieved an F1 score of 99.5% at differentiating 5 common MRI sequences obtained by three Siemens scanners (Aera, Verio, Biograph mMR). To the best of our knowledge, we are the first to develop an automated method for the 3D classification of MRI sequences in the chest, abdomen, and pelvis, and our work has outperformed the previous state-of-the-art MRI series classifiers.

Weakly Supervised Detection of Pheochromocytomas and Paragangliomas in CT.

Pheochromocytomas and Paragangliomas (PPGLs) are rare adrenal and extra-adrenal tumors which have the potential to metastasize. For the management of patients with PPGLs, CT is the preferred modality of choice for precise localization and estimation of their progression. However, due to the myriad variations in size, morphology, and appearance of the tumors in different anatomical regions, radiologists are posed with the challenge of accurate detection of PPGLs. Since clinicians also need to routinely measure their size and track their changes over time across patient visits, manual demarcation of PPGLs is quite a time-consuming and cumbersome process. To ameliorate the manual effort spent for this task, we propose an automated method to detect PPGLs in CT studies via a proxy segmentation task. As only weak annotations for PPGLs in the form of prospectively marked 2D bounding boxes on an axial slice were available, we extended these 2D boxes into weak 3D annotations and trained a 3D full-resolution nnUNet model to directly segment PPGLs. We evaluated our approach on a dataset consisting of chest-abdomen-pelvis CTs of 255 patients with confirmed PPGLs. We obtained a precision of 70% and sensitivity of 64.1% with our proposed approach when tested on 53 CT studies. Our findings highlight the promising nature of detecting PPGLs via segmentation, and furthers the state-of-the-art in this exciting yet challenging area of rare cancer management.

Segmentation of pelvic structures in T2 MRI via MR-to-CT synthesis.

Segmentation of multiple pelvic structures in MRI volumes is a prerequisite for many clinical applications, such as sarcopenia assessment, bone density measurement, and muscle-to-fat volume ratio estimation. While many CT-specific datasets and automated CT-based multi-structure pelvis segmentation methods exist, there are few MRI-specific multi-structure segmentation methods in literature. In this pilot work, we propose a lightweight and annotation-free pipeline to synthetically translate T2 MRI volumes of the pelvis to CT, and subsequently leverage an existing CT-only tool called TotalSegmentator to segment 8 pelvic structures in the generated CT volumes. The predicted masks were then mapped back to the original MR volumes as segmentation masks. We compared the predicted masks against the expert annotations of the public TCGA-UCEC dataset and an internal dataset. Experiments demonstrated that the proposed pipeline achieved Dice measures ≥65% for 8 pelvic structures in T2 MRI. The proposed pipeline is an alternative method to obtain multi-organ and structure segmentations without being encumbered by time-consuming manual annotations. By exploiting the significant research progress in CTs, it is possible to extend the proposed pipeline to other MRI sequences in principle. Our research bridges the chasm between the current CT-based multi-structure segmentation and MRI-based segmentation. The manually segmented structures in the TCGA-UCEC dataset are publicly available.

Automated detection of incidental abdominal aortic aneurysms on computed tomography.

PURPOSE: To detect and assess abdominal aortic aneurysms (AAAs) on CT in a large asymptomatic adult patient population using fully-automated deep learning software. MATERIALS AND METHODS: The abdominal aorta was segmented using a fully-automated deep learning model trained on 66 manually-segmented abdominal CT scans from two datasets. The axial diameters of the segmented aorta were extracted to detect the presence of AAAs-maximum axial aortic diameter greater than 3 cm were labeled as AAA positive. The trained system was then externally-validated on CT colonography scans of 9172 asymptomatic outpatients (mean age, 57 years) referred for colorectal cancer screening. Using a previously-validated automated calcified atherosclerotic plaque detector, we correlated abdominal aortic Agatston and volume scores with the presence of AAA. RESULTS: The deep learning software detected AAA on the external validation dataset with a sensitivity, specificity, and AUC of 96%, (95% CI 89%, 100%), 96% (96%, 97%), and 99% (98%, 99%) respectively. The Agatston and volume scores of reported AAA-positive cases were statistically significantly greater than those of reported AAA-negative cases (p < 0.0001). Using plaque alone as a AAA detector, at a threshold Agatston score of 2871, the sensitivity and specificity were 84% (73%, 94%) and 87% (86%, 87%), respectively. CONCLUSION: Fully-automated detection and assessment of AAA on CT is feasible and accurate. There was a strong statistical association between the presence of AAA and the quantity of abdominal aortic calcified atherosclerotic plaque.

Longitudinal follow-up of incidental renal calculi on computed tomography.

RATIONALE AND OBJECTIVES: Measuring small kidney stones on CT is a time-consuming task often neglected. Volumetric assessment provides a better measure of size than linear dimensions. Our objective is to analyze the growth rate and prognosis of incidental kidney stones in asymptomatic patients on CT. MATERIALS AND METHODS: This retrospective study included 4266 scans from 2030 asymptomatic patients who underwent two or more nonenhanced CT scans for colorectal screening between 2004 and 2016. The DL software identified and measured the volume, location, and attenuation of 883 stones. The corresponding scans were manually evaluated, and patients without follow-up were excluded. At each follow-up, the stones were categorized as new, growing, persistent, or resolved. Stone size (volume and diameter), attenuation, and location were correlated with the outcome and growth rates of the stones. RESULTS: The stone cohort comprised 407 scans from 189 (M: 124, F: 65, median age: 55.4 years) patients. The median number of stones per scan was 1 (IQR: [1, 2]). The median stone volume was 17.1 mm3 (IQR: [7.4, 43.6]) and the median peak attenuation was 308 HU (IQR: [204, 532]. The 189 initial scans contained 291stones; 91 (31.3%) resolved, 142 (48.8%) grew, and 58 (19.9) remained persistent at the first follow-up. At the second follow-up (for 27 patients with 2 follow-ups), 14/44 (31.8%) stones had resolved, 19/44 (43.2%) grew and 11/44 (25%) were persistent. The median growth rate of growing stones was 3.3 mm3/year, IQR: [1.4,7.4]. Size and attenuation had a moderate correlation (Spearman rho 0.53, P < .001 for volume, and 0.50 P < .001 for peak attenuation) with the growth rate. Growing and persistent stones had significantly greater maximum axial diameter (2.7 vs 2.3 mm, P =.047) and peak attenuation (300 vs 258 HU, P =.031) CONCLUSION: We report a 12.7% prevalence of incidental kidney stones in asymptomatic adults, of which about half grew during follow-up with a median growth rate of about 3.3 mm3/year.

Feasibility of Using the Privacy-preserving Large Language Model Vicuna for Labeling Radiology Reports.

Background Large language models (LLMs) such as ChatGPT, though proficient in many text-based tasks, are not suitable for use with radiology reports due to patient privacy constraints. Purpose To test the feasibility of using an alternative LLM (Vicuna-13B) that can be run locally for labeling radiography reports. Materials and Methods Chest radiography reports from the MIMIC-CXR and National Institutes of Health (NIH) data sets were included in this retrospective study. Reports were examined for 13 findings. Outputs reporting the presence or absence of the 13 findings were generated by Vicuna by using a single-step or multistep prompting strategy (prompts 1 and 2, respectively). Agreements between Vicuna outputs and CheXpert and CheXbert labelers were assessed using Fleiss κ. Agreement between Vicuna outputs from three runs under a hyperparameter setting that introduced some randomness (temperature, 0.7) was also assessed. The performance of Vicuna and the labelers was assessed in a subset of 100 NIH reports annotated by a radiologist with use of area under the receiver operating characteristic curve (AUC). Results A total of 3269 reports from the MIMIC-CXR data set (median patient age, 68 years [IQR, 59-79 years]; 161 male patients) and 25 596 reports from the NIH data set (median patient age, 47 years [IQR, 32-58 years]; 1557 male patients) were included. Vicuna outputs with prompt 2 showed, on average, moderate to substantial agreement with the labelers on the MIMIC-CXR (κ median, 0.57 [IQR, 0.45-0.66] with CheXpert and 0.64 [IQR, 0.45-0.68] with CheXbert) and NIH (κ median, 0.52 [IQR, 0.41-0.65] with CheXpert and 0.55 [IQR, 0.41-0.74] with CheXbert) data sets, respectively. Vicuna with prompt 2 performed at par (median AUC, 0.84 [IQR, 0.74-0.93]) with both labelers on nine of 11 findings. Conclusion In this proof-of-concept study, outputs of the LLM Vicuna reporting the presence or absence of 13 findings on chest radiography reports showed moderate to substantial agreement with existing labelers. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Cai in this issue.

Performance of alternative manual and automated deep learning segmentation techniques for the prediction of benign and malignant lung nodules.

Purpose: We aim to evaluate the performance of radiomic biopsy (RB), best-fit bounding box (BB), and a deep-learning-based segmentation method called no-new-U-Net (nnU-Net), compared to the standard full manual (FM) segmentation method for predicting benign and malignant lung nodules using a computed tomography (CT) radiomic machine learning model. Materials and Methods: A total of 188 CT scans of lung nodules from 2 institutions were used for our study. One radiologist identified and delineated all 188 lung nodules, whereas a second radiologist segmented a subset (n=20) of these nodules. Both radiologists employed FM and RB segmentation methods. BB segmentations were generated computationally from the FM segmentations. The nnU-Net, a deep-learning-based segmentation method, performed automatic nodule detection and segmentation. The time radiologists took to perform segmentations was recorded. Radiomic features were extracted from each segmentation method, and models to predict benign and malignant lung nodules were developed. The Kruskal-Wallis and DeLong tests were used to compare segmentation times and areas under the curve (AUC), respectively. Results: For the delineation of the FM, RB, and BB segmentations, the two radiologists required a median time (IQR) of 113 (54 to 251.5), 21 (9.25 to 38), and 16 (12 to 64.25) s, respectively (p=0.04). In dataset 1, the mean AUC (95% CI) of the FM, RB, BB, and nnU-Net model were 0.964 (0.96 to 0.968), 0.985 (0.983 to 0.987), 0.961 (0.956 to 0.965), and 0.878 (0.869 to 0.888). In dataset 2, the mean AUC (95% CI) of the FM, RB, BB, and nnU-Net model were 0.717 (0.705 to 0.729), 0.919 (0.913 to 0.924), 0.699 (0.687 to 0.711), and 0.644 (0.632 to 0.657). Conclusion: Radiomic biopsy-based models outperformed FM and BB models in prediction of benign and malignant lung nodules in two independent datasets while deep-learning segmentation-based models performed similarly to FM and BB. RB could be a more efficient segmentation method, but further validation is needed.

SCOPE: predicting future diagnoses in office visits using electronic health records.

We propose an interpretable and scalable model to predict likely diagnoses at an encounter based on past diagnoses and lab results. This model is intended to aid physicians in their interaction with the electronic health records (EHR). To accomplish this, we retrospectively collected and de-identified EHR data of 2,701,522 patients at Stanford Healthcare over a time period from January 2008 to December 2016. A population-based sample of patients comprising 524,198 individuals (44% M, 56% F) with multiple encounters with at least one frequently occurring diagnosis codes were chosen. A calibrated model was developed to predict ICD-10 diagnosis codes at an encounter based on the past diagnoses and lab results, using a binary relevance based multi-label modeling strategy. Logistic regression and random forests were tested as the base classifier, and several time windows were tested for aggregating the past diagnoses and labs. This modeling approach was compared to a recurrent neural network based deep learning method. The best model used random forest as the base classifier and integrated demographic features, diagnosis codes, and lab results. The best model was calibrated and its performance was comparable or better than existing methods in terms of various metrics, including a median AUROC of 0.904 (IQR [0.838, 0.954]) over 583 diseases. When predicting the first occurrence of a disease label for a patient, the median AUROC with the best model was 0.796 (IQR [0.737, 0.868]). Our modeling approach performed comparably as the tested deep learning method, outperforming it in terms of AUROC (p < 0.001) but underperforming in terms of AUPRC (p < 0.001). Interpreting the model showed that the model uses meaningful features and highlights many interesting associations among diagnoses and lab results. We conclude that the multi-label model performs comparably with RNN based deep learning model while offering simplicity and potentially superior interpretability. While the model was trained and validated on data obtained from a single institution, its simplicity, interpretability and performance makes it a promising candidate for deployment.

Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports.

Despite the reduction in turn-around times in radiology reporting with the use of speech recognition software, persistent communication errors can significantly impact the interpretation of radiology reports. Pre-filling a radiology report holds promise in mitigating reporting errors, and despite multiple efforts in literature to generate comprehensive medical reports, there lacks approaches that exploit the longitudinal nature of patient visit records in the MIMIC-CXR dataset. To address this gap, we propose to use longitudinal multi-modal data, i.e., previous patient visit CXR, current visit CXR, and the previous visit report, to pre-fill the "findings" section of the patient's current visit. We first gathered the longitudinal visit information for 26,625 patients from the MIMIC-CXR dataset, and created a new dataset called Longitudinal-MIMIC. With this new dataset, a transformer-based model was trained to capture the multi-modal longitudinal information from patient visit records (CXR images + reports) via a cross-attention-based multi-modal fusion module and a hierarchical memory-driven decoder. In contrast to previous works that only uses current visit data as input to train a model, our work exploits the longitudinal information available to pre-fill the "findings" section of radiology reports. Experiments show that our approach outperforms several recent approaches. Code will be published at https://github.com/CelestialShine/Longitudinal-Chest-X-Ray.

Fully Automated Longitudinal Assessment of Renal Stone Burden on Serial CT Imaging Using Deep Learning.

Purpose: Use deep learning (DL) to automate the measurement and tracking of kidney stone burden over serial CT scans. Materials and Methods: This retrospective study included 259 scans from 113 symptomatic patients being treated for urolithiasis at a single medical center between 2006 and 2019. These patients underwent a standard low-dose noncontrast CT scan followed by ultra-low-dose CT scans limited to the level of the kidneys. A DL model was used to detect, segment, and measure the volume of all stones in both initial and follow-up scans. The stone burden was characterized by the total volume of all stones in a scan (SV). The absolute and relative change of SV, (SVA and SVR, respectively) over serial scans were computed. The automated assessments were compared with manual assessments using concordance correlation coefficient (CCC), and their agreement was visualized using Bland-Altman and scatter plots. Results: Two hundred twenty-eight out of 233 scans with stones were identified by the automated pipeline; per-scan sensitivity was 97.8% (95% confidence interval [CI]: 96.0-99.7). The per-scan positive predictive value was 96.6% (95% CI: 94.4-98.8). The median SV, SVA, and SVR were 476.5 mm3, -10 mm3, and 0.89, respectively. After removing outliers outside the 5th and 95th percentiles, the CCC measuring agreement on SV, SVA, and SVR were 0.995 (0.992-0.996), 0.980 (0.972-0.986), and 0.915 (0.881-0.939), respectively Conclusions: The automated DL-based measurements showed good agreement with the manual assessments of the stone burden and its interval change on serial CT scans.

Early Detection of Lung Cancer in the NLST Dataset.

Lung Cancer is the leading cause of cancer mortality in the U.S. The effectiveness of standard treatments, including surgery, chemotherapy or radiotherapy, depends on several factors like type and stage of cancer, with the survival rate being much worse for later cancer stages. The National Lung Screening Trial (NLST) established that patients screened using low-dose Computed Tomography (CT) had a 15 to 20 percent lower risk of dying from lung cancer than patients screened using chest X-rays. While CT excelled at detecting small early stage malignant nodules, a large proportion of patients (> 25%) screened positive and only a small fraction (< 10%) of these positive screens actually had or developed cancer in the subsequent years. We developed a model to distinguish between high and low risk patients among the positive screens, predicting the likelihood of having or developing lung cancer at the current time point or in subsequent years non-invasively, based on current and previous CT imaging data. However, most of the nodules in NLST are very small, and nodule segmentations or even precise locations are unavailable. Our model comprises two stages: the first stage is a neural network model trained on the Lung Image Database Consortium (LIDC-IDRI) cohort which detects nodules and assigns them malignancy scores. The second part of our model is a boosted tree which outputs a cancer probability for a patient based on the nodule information (location and malignancy score) predicted by the first stage. Our model, built on a subset of the NLST cohort (n = 1138) shows excellent performance, achieving an area under the receiver operating characteristics curve (ROC AUC) of 0.85 when predicting based on CT images from all three time points available in the NLST dataset.

Multimodal deep learning to predict prognosis in adult and pediatric brain tumors.

BACKGROUND: The introduction of deep learning in both imaging and genomics has significantly advanced the analysis of biomedical data. For complex diseases such as cancer, different data modalities may reveal different disease characteristics, and the integration of imaging with genomic data has the potential to unravel additional information than when using these data sources in isolation. Here, we propose a DL framework that combines these two modalities with the aim to predict brain tumor prognosis. METHODS: Using two separate glioma cohorts of 783 adults and 305 pediatric patients we developed a DL framework that can fuse histopathology images with gene expression profiles. Three strategies for data fusion were implemented and compared: early, late, and joint fusion. Additional validation of the adult glioma models was done on an independent cohort of 97 adult patients. RESULTS: Here we show that the developed multimodal data models achieve better prediction results compared to the single data models, but also lead to the identification of more relevant biological pathways. When testing our adult models on a third brain tumor dataset, we show our multimodal framework is able to generalize and performs better on new data from different cohorts. Leveraging the concept of transfer learning, we demonstrate how our pediatric multimodal models can be used to predict prognosis for two more rare (less available samples) pediatric brain tumors. CONCLUSIONS: Our study illustrates that a multimodal data fusion approach can be successfully implemented and customized to model clinical outcome of adult and pediatric brain tumors.

An increasing amount of complex patient data is generated when treating patients with cancer, including histopathology data (where the appearance of a tumor is examined under a microscope) and molecular data (such as analysis of a tumor's genetic material). Computational methods to integrate these data types might help us to predict outcomes in patients with cancer. Here, we propose a deep learning method which involves computer software learning from patterns in the data, to combine histopathology and molecular data to predict outcomes in patients with brain cancers. Using three cohorts of patients, we show that our method combining the different datasets performs better than models using one data type. Methods like ours might help clinicians to better inform patients about their prognosis and make decisions about their care.

Machine intelligence for radiation science: summary of the Radiation Research Society 67th annual meeting symposium.

The era of high-throughput techniques created big data in the medical field and research disciplines. Machine intelligence (MI) approaches can overcome critical limitations on how those large-scale data sets are processed, analyzed, and interpreted. The 67th Annual Meeting of the Radiation Research Society featured a symposium on MI approaches to highlight recent advancements in the radiation sciences and their clinical applications. This article summarizes three of those presentations regarding recent developments for metadata processing and ontological formalization, data mining for radiation outcomes in pediatric oncology, and imaging in lung cancer.