RESUMO
"Just Accepted" papers have undergone full peer review and have been accepted for publication in Radiology: Artificial Intelligence. This article will undergo copyediting, layout, and proof review before it is published in its final version. Please note that during production of the final copyedited article, errors may be discovered which could affect the content. Purpose To develop an artificial intelligence model that utilizes supervised contrastive learning to minimize bias in chest radiograph (CXR) diagnosis. Materials and Methods In this retrospective study, the proposed method was evaluated on two datasets: the Medical Imaging and Data Resource Center (MIDRC) dataset with 77,887 CXRs from 27,796 patients collected as of April 20, 2023 for COVID-19 diagnosis, and the NIH Chest x-ray 14 (NIH-CXR) dataset with 112,120 CXRs from 30,805 patients collected between 1992 and 2015. In the NIH-CXR dataset, thoracic abnormalities included atelectasis, cardiomegaly, effusion, infiltration, mass, nodule, pneumonia, pneumothorax, consolidation, edema, emphysema, fibrosis, pleural thickening, or hernia. The proposed method utilized supervised contrastive learning with carefully selected positive and negative samples to generate fair image embeddings, which were fine-tuned for subsequent tasks to reduce bias in CXR diagnosis. The method was evaluated using the marginal area under the receiver operating characteristic curve (AUC) difference (ΔmAUC). Results The proposed model showed a significant decrease in bias across all subgroups compared with the baseline models, as evidenced by a paired T-test (P < .001). The ΔmAUCs obtained by the proposed method were 0.01 (95% CI, 0.01-0.01), 0.21 (95% CI, 0.21-0.21), and 0.10 (95% CI, 0.10-0.10) for sex, race, and age subgroups, respectively, on MIDRC, and 0.01 (95% CI, 0.01-0.01) and 0.05 (95% CI, 0.05-0.05) for sex and age subgroups, respectively, on NIH-CXR. Conclusion Employing supervised contrastive learning can mitigate bias in CXR diagnosis, addressing concerns of fairness and reliability in deep learning-based diagnostic methods. ©RSNA, 2024.
RESUMO
De-identification of medical images intended for research is a core requirement for data sharing initiatives, particularly as the demand for data for artificial intelligence (AI) applications grows. The Center for Biomedical Informatics and Information Technology (CBIIT) of the United States National Cancer Institute (NCI) convened a two half-day virtual workshop with the intent of summarizing the state of the art in de-identification technology and processes and exploring interesting aspects of the subject. This paper summarizes the highlights of the second day of the workshop, the recordings and presentations of which are publicly available for review. The topics covered included pathology whole slide image de-identification, de-facing, the role of AI in image de-identification, and the NCI Medical Image De-Identification Initiative (MIDI) datasets and pipeline.
RESUMO
BACKGROUND AND OBJECTIVES: Integrating large language models (LLMs) such as GPT-4 Turbo into diagnostic imaging faces a significant challenge, with current misdiagnosis rates ranging from 30-50%. This study evaluates how prompt engineering and confidence thresholds can improve diagnostic accuracy in neuroradiology. METHODS: We analyze 751 neuroradiology cases from the American Journal of Neuroradiology using GPT-4 Turbo with customized prompts to improve diagnostic precision. RESULTS: Initially, GPT-4 Turbo achieved a baseline diagnostic accuracy of 55.1%. By reformatting responses to list five diagnostic candidates and applying a 90% confidence threshold, the highest precision of the diagnosis increased to 72.9%, with the candidate list providing the correct diagnosis at 85.9%, reducing the misdiagnosis rate to 14.1%. However, this threshold reduced the number of cases that responded. CONCLUSIONS: Strategic prompt engineering and high confidence thresholds significantly reduce misdiagnoses and improve the precision of the LLM diagnostic in neuroradiology. More research is needed to optimize these approaches for broader clinical implementation, balancing accuracy and utility.
RESUMO
Many real-world image recognition problems, such as diagnostic medical imaging exams, are "long-tailed" - there are a few common findings followed by many more relatively rare conditions. In chest radiography, diagnosis is both a long-tailed and multi-label problem, as patients often present with multiple findings simultaneously. While researchers have begun to study the problem of long-tailed learning in medical image recognition, few have studied the interaction of label imbalance and label co-occurrence posed by long-tailed, multi-label disease classification. To engage with the research community on this emerging topic, we conducted an open challenge, CXR-LT, on long-tailed, multi-label thorax disease classification from chest X-rays (CXRs). We publicly release a large-scale benchmark dataset of over 350,000 CXRs, each labeled with at least one of 26 clinical findings following a long-tailed distribution. We synthesize common themes of top-performing solutions, providing practical recommendations for long-tailed, multi-label medical image classification. Finally, we use these insights to propose a path forward involving vision-language foundation models for few- and zero-shot disease classification.
Assuntos
Radiografia Torácica , Humanos , Radiografia Torácica/métodos , Interpretação de Imagem Radiográfica Assistida por Computador/métodos , Doenças Torácicas/diagnóstico por imagem , Doenças Torácicas/classificação , AlgoritmosRESUMO
Background Generating radiologic findings from chest radiographs is pivotal in medical image analysis. The emergence of OpenAI's generative pretrained transformer, GPT-4 with vision (GPT-4V), has opened new perspectives on the potential for automated image-text pair generation. However, the application of GPT-4V to real-world chest radiography is yet to be thoroughly examined. Purpose To investigate the capability of GPT-4V to generate radiologic findings from real-world chest radiographs. Materials and Methods In this retrospective study, 100 chest radiographs with free-text radiology reports were annotated by a cohort of radiologists, two attending physicians and three residents, to establish a reference standard. Of 100 chest radiographs, 50 were randomly selected from the National Institutes of Health (NIH) chest radiographic data set, and 50 were randomly selected from the Medical Imaging and Data Resource Center (MIDRC). The performance of GPT-4V at detecting imaging findings from each chest radiograph was assessed in the zero-shot setting (where it operates without prior examples) and few-shot setting (where it operates with two examples). Its outcomes were compared with the reference standard with regards to clinical conditions and their corresponding codes in the International Statistical Classification of Diseases, Tenth Revision (ICD-10), including the anatomic location (hereafter, laterality). Results In the zero-shot setting, in the task of detecting ICD-10 codes alone, GPT-4V attained an average positive predictive value (PPV) of 12.3%, average true-positive rate (TPR) of 5.8%, and average F1 score of 7.3% on the NIH data set, and an average PPV of 25.0%, average TPR of 16.8%, and average F1 score of 18.2% on the MIDRC data set. When both the ICD-10 codes and their corresponding laterality were considered, GPT-4V produced an average PPV of 7.8%, average TPR of 3.5%, and average F1 score of 4.5% on the NIH data set, and an average PPV of 10.9%, average TPR of 4.9%, and average F1 score of 6.4% on the MIDRC data set. With few-shot learning, GPT-4V showed improved performance on both data sets. When contrasting zero-shot and few-shot learning, there were improved average TPRs and F1 scores in the few-shot setting, but there was not a substantial increase in the average PPV. Conclusion Although GPT-4V has shown promise in understanding natural images, it had limited effectiveness in interpreting real-world chest radiographs. © RSNA, 2024 Supplemental material is available for this article.
Assuntos
Radiografia Torácica , Humanos , Radiografia Torácica/métodos , Estudos Retrospectivos , Feminino , Masculino , Pessoa de Meia-Idade , Interpretação de Imagem Radiográfica Assistida por Computador/métodos , Idoso , AdultoRESUMO
The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.
Assuntos
Inteligência Artificial , Radiologia , Humanos , Diagnóstico por Imagem/métodos , Sociedades Médicas , América do NorteRESUMO
Purpose: The development and evaluation of machine learning models that automatically identify the body part(s) imaged, axis of imaging, and the presence of intravenous contrast material of a CT series of images. Methods: This retrospective study included 6955 series from 1198 studies (501 female, 697 males, mean age 56.5 years) obtained between January 2010 and September 2021. Each series was annotated by a trained board-certified radiologist with labels consisting of 16 body parts, 3 imaging axes, and whether an intravenous contrast agent was used. The studies were randomly assigned to the training, validation and testing sets with a proportion of 70%, 20% and 10%, respectively, to develop a 3D deep neural network for each classification task. External validation was conducted with a total of 35,272 series from 7 publicly available datasets. The classification accuracy for each series was independently assessed for each task to evaluate model performance. Results: The accuracies for identifying the body parts, imaging axes, and the presence of intravenous contrast were 96.0% (95% CI: 94.6%, 97.2%), 99.2% (95% CI: 98.5%, 99.7%), and 97.5% (95% CI: 96.4%, 98.5%) respectively. The generalizability of the models was demonstrated through external validation with accuracies of 89.7 - 97.8%, 98.6 - 100%, and 87.8 - 98.6% for the same tasks. Conclusions: The developed models demonstrated high performance on both internal and external testing in identifying key aspects of a CT series.
Assuntos
Aprendizado Profundo , Masculino , Humanos , Feminino , Pessoa de Meia-Idade , Estudos Retrospectivos , Corpo Humano , Aprendizado de Máquina , Tomografia Computadorizada por Raios X/métodos , Meios de ContrasteRESUMO
RATIONALE AND OBJECTIVES: Following autosomal dominant polycystic kidney disease (ADPKD) progression by measuring organ volumes requires low measurement variability. The objective of this study is to reduce organ volume measurement variability on MRI of ADPKD patients by utilizing all pulse sequences to obtain multiple measurements which allows outlier analysis to find errors and averaging to reduce variability. MATERIALS AND METHODS: In order to make measurements on multiple pulse sequences practical, a 3D multi-modality multi-class segmentation model based on nnU-net was trained/validated using T1, T2, SSFP, DWI and CT from 413 subjects. Reproducibility was assessed with test-re-test methodology on ADPKD subjects (n = 19) scanned twice within a 3-week interval correcting outliers and averaging the measurements across all sequences. Absolute percent differences in organ volumes were compared to paired students t-test. RESULTS: Dice similarlity coefficient > 97%, Jaccard Index > 0.94, mean surface distance < 1 mm and mean Hausdorff Distance < 2 cm for all three organs and all five sequences were found on internal (n = 25), external (n = 37) and test-re-test reproducibility assessment (38 scans in 19 subjects). When averaging volumes measured from five MRI sequences, the model automatically segmented kidneys with test-re-test reproducibility (percent absolute difference between exam 1 and exam 2) of 1.3% which was better than all five expert observers. It reliably stratified ADPKD into Mayo Imaging Classification (area under the curve=100%) compared to radiologist. CONCLUSION: 3D deep learning measures organ volumes on five MRI sequences leveraging the power of outlier analysis and averaging to achieve 1.3% total kidney test-re-test reproducibility.
Assuntos
Aprendizado Profundo , Rim Policístico Autossômico Dominante , Humanos , Rim Policístico Autossômico Dominante/diagnóstico por imagem , Tamanho do Órgão , Reprodutibilidade dos Testes , Rim/diagnóstico por imagem , Imageamento por Ressonância Magnética/métodosRESUMO
Many real-world image recognition problems, such as diagnostic medical imaging exams, are "long-tailed" - there are a few common findings followed by many more relatively rare conditions. In chest radiography, diagnosis is both a long-tailed and multi-label problem, as patients often present with multiple findings simultaneously. While researchers have begun to study the problem of long-tailed learning in medical image recognition, few have studied the interaction of label imbalance and label co-occurrence posed by long-tailed, multi-label disease classification. To engage with the research community on this emerging topic, we conducted an open challenge, CXR-LT, on long-tailed, multi-label thorax disease classification from chest X-rays (CXRs). We publicly release a large-scale benchmark dataset of over 350,000 CXRs, each labeled with at least one of 26 clinical findings following a long-tailed distribution. We synthesize common themes of top-performing solutions, providing practical recommendations for long-tailed, multi-label medical image classification. Finally, we use these insights to propose a path forward involving vision-language foundation models for few- and zero-shot disease classification.
RESUMO
Pruning has emerged as a powerful technique for compressing deep neural networks, reducing memory usage and inference time without significantly affecting overall performance. However, the nuanced ways in which pruning impacts model behavior are not well understood, particularly for long-tailed, multi-label datasets commonly found in clinical settings. This knowledge gap could have dangerous implications when deploying a pruned model for diagnosis, where unexpected model behavior could impact patient well-being. To fill this gap, we perform the first analysis of pruning's effect on neural networks trained to diagnose thorax diseases from chest X-rays (CXRs). On two large CXR datasets, we examine which diseases are most affected by pruning and characterize class "forgettability" based on disease frequency and co-occurrence behavior. Further, we identify individual CXRs where uncompressed and heavily pruned models disagree, known as pruning-identified exemplars (PIEs), and conduct a human reader study to evaluate their unifying qualities. We find that radiologists perceive PIEs as having more label noise, lower image quality, and higher diagnosis difficulty. This work represents a first step toward understanding the impact of pruning on model behavior in deep long-tailed, multi-label medical image classification. All code, model weights, and data access instructions can be found at https://github.com/VITA-Group/PruneCXR.
RESUMO
Deep learning has become a popular tool for computer-aided diagnosis using medical images, sometimes matching or exceeding the performance of clinicians. However, these models can also reflect and amplify human bias, potentially resulting inaccurate missed diagnoses. Despite this concern, the problem of improving model fairness in medical image classification by deep learning has yet to be fully studied. To address this issue, we propose an algorithm that leverages the marginal pairwise equal opportunity to reduce bias in medical image classification. Our evaluations across four tasks using four independent large-scale cohorts demonstrate that our proposed algorithm not only improves fairness in individual and intersectional subgroups but also maintains overall performance. Specifically, the relative change in pairwise fairness difference between our proposed model and the baseline model was reduced by over 35%, while the relative change in AUC value was typically within 1%. By reducing the bias generated by deep learning models, our proposed approach can potentially alleviate concerns about the fairness and reliability of image-based computer-aided diagnosis.
Assuntos
Algoritmos , Diagnóstico por Computador , Humanos , Reprodutibilidade dos Testes , Diagnóstico por Computador/métodos , ComputadoresRESUMO
Pruning has emerged as a powerful technique for compressing deep neural networks, reducing memory usage and inference time without significantly affecting overall performance. However, the nuanced ways in which pruning impacts model behavior are not well understood, particularly for long-tailed, multi-label datasets commonly found in clinical settings. This knowledge gap could have dangerous implications when deploying a pruned model for diagnosis, where unexpected model behavior could impact patient well-being. To fill this gap, we perform the first analysis of pruning's effect on neural networks trained to diagnose thorax diseases from chest X-rays (CXRs). On two large CXR datasets, we examine which diseases are most affected by pruning and characterize class "forgettability" based on disease frequency and co-occurrence behavior. Further, we identify individual CXRs where uncompressed and heavily pruned models disagree, known as pruning-identified exemplars (PIEs), and conduct a human reader study to evaluate their unifying qualities. We find that radiologists perceive PIEs as having more label noise, lower image quality, and higher diagnosis difficulty. This work represents a first step toward understanding the impact of pruning on model behavior in deep long-tailed, multi-label medical image classification. All code, model weights, and data access instructions can be found at https://github.com/VITA-Group/PruneCXR.
RESUMO
Total kidney volume measured on MRI is an important biomarker for assessing the progression of autosomal dominant polycystic kidney disease and response to treatment. However, we have noticed that there can be substantial differences in the kidney volume measurements obtained from the various pulse sequences commonly included in an MRI exam. Here we examine kidney volume measurement variability among five commonly acquired MRI pulse sequences in abdominal MRI exams in 105 patients with ADPKD. Right and left kidney volumes were independently measured by three expert observers using model-assisted segmentation for axial T2, coronal T2, axial single-shot fast spin echo (SSFP), coronal SSFP, and axial 3D T1 images obtained on a single MRI from ADPKD patients. Outlier measurements were analyzed for data acquisition errors. Most of the outlier values (88%) were due to breathing during scanning causing slice misregistration with gaps or duplication of imaging slices (n = 35), slice misregistration from using multiple breath holds during acquisition (n = 25), composing of two overlapping acquisitions (n = 17), or kidneys not entirely within the field of view (n = 4). After excluding outlier measurements, the coefficient of variation among the five measurements decreased from 4.6% pre to 3.2%. Compared to the average of all sequences without errors, TKV measured on axial and coronal T2 weighted imaging were 1.2% and 1.8% greater, axial SSFP was 0.4% greater, coronal SSFP was 1.7% lower and axial T1 was 1.5% lower than the mean, indicating intrinsic measurement biases related to the different MRI contrast mechanisms. In conclusion, MRI data acquisition errors are common but can be identified using outlier analysis and excluded to improve organ volume measurement consistency. Bias toward larger volume measurements on T2 sequences and smaller volumes on axial T1 sequences can also be mitigated by averaging data from all error-free sequences acquired.
Assuntos
Rim Policístico Autossômico Dominante , Humanos , Rim , Imageamento por Ressonância Magnética , Controle de QualidadeRESUMO
Large chest X-rays (CXR) datasets have been collected to train deep learning models to detect thorax pathology on CXR. However, most CXR datasets are from single-center studies and the collected pathologies are often imbalanced. The aim of this study was to automatically construct a public, weakly-labeled CXR database from articles in PubMed Central Open Access (PMC-OA) and to assess model performance on CXR pathology classification by using this database as additional training data. Our framework includes text extraction, CXR pathology verification, subfigure separation, and image modality classification. We have extensively validated the utility of the automatically generated image database on thoracic disease detection tasks, including Hernia, Lung Lesion, Pneumonia, and pneumothorax. We pick these diseases due to their historically poor performance in existing datasets: the NIH-CXR dataset (112,120 CXR) and the MIMIC-CXR dataset (243,324 CXR). We find that classifiers fine-tuned with additional PMC-CXR extracted by the proposed framework consistently and significantly achieved better performance than those without (e.g., Hernia: 0.9335 vs 0.9154; Lung Lesion: 0.7394 vs. 0.7207; Pneumonia: 0.7074 vs. 0.6709; Pneumothorax 0.8185 vs. 0.7517, all in AUC with p< 0.0001) for CXR pathology detection. In contrast to previous approaches that manually submit the medical images to the repository, our framework can automatically collect figures and their accompanied figure legends. Compared to previous studies, the proposed framework improved subfigure segmentation and incorporates our advanced self-developed NLP technique for CXR pathology verification. We hope it complements existing resources and improves our ability to make biomedical image data findable, accessible, interoperable, and reusable.
Assuntos
Pneumonia , Pneumotórax , Doenças Torácicas , Humanos , Pneumotórax/diagnóstico por imagem , Radiografia Torácica/métodos , Raios X , Acesso à Informação , Pneumonia/diagnóstico por imagemRESUMO
BACKGROUND: Total kidney volume (TKV) is an important biomarker for assessing kidney function, especially for autosomal dominant polycystic kidney disease (ADPKD). However, TKV measurements from a single MRI pulse sequence have limited reproducibility, ± ~5%, similar to ADPKD annual kidney growth rates. PURPOSE: To improve TKV measurement reproducibility on MRI by extending artificial intelligence algorithms to automatically segment kidneys on T1-weighted, T2-weighted, and steady state free precession (SSFP) sequences in axial and coronal planes and averaging measurements. STUDY TYPE: Retrospective training, prospective testing. SUBJECTS: Three hundred ninety-seven patients (356 with ADPKD, 41 without), 75% for training and 25% for validation, 40 ADPKD patients for testing and 17 ADPKD patients for assessing reproducibility. FIELD STRENGTH/SEQUENCE: T2-weighted single-shot fast spin echo (T2), SSFP, and T1-weighted 3D spoiled gradient echo (T1) at 1.5 and 3T. ASSESSMENT: 2D U-net segmentation algorithm was trained on images from all sequences. Five observers independently measured each kidney volume manually on axial T2 and using model-assisted segmentations on all sequences and image plane orientations for two MRI exams in two sessions separated by 1-3 weeks to assess reproducibility. Manual and model-assisted segmentation times were recorded. STATISTICAL TESTS: Bland-Altman, Schapiro-Wilk (normality assessment), Pearson's chi-squared (categorical variables); Dice similarity coefficient, interclass correlation coefficient, and concordance correlation coefficient for analyzing TKV reproducibility. P-value < 0.05 was considered statistically significant. RESULTS: In 17 ADPKD subjects, model-assisted segmentations of axial T2 images were significantly faster than manual segmentations (2:49 minute vs. 11:34 minute), with no significant absolute percent difference in TKV (5.9% vs. 5.3%, P = 0.88) between scans 1 and 2. Absolute percent differences between the two scans for model-assisted segmentations on other sequences were 5.5% (axial T1), 4.5% (axial SSFP), 4.1% (coronal SSFP), and 3.2% (coronal T2). Averaging measurements from all five model-assisted segmentations significantly reduced absolute percent difference to 2.5%, further improving to 2.1% after excluding an outlier. DATA CONCLUSION: Measuring TKV on multiple MRI pulse sequences in coronal and axial planes is practical with deep learning model-assisted segmentations and can improve TKV measurement reproducibility more than 2-fold in ADPKD. EVIDENCE LEVEL: 2 TECHNICAL EFFICACY: Stage 1.
Assuntos
Rim Policístico Autossômico Dominante , Humanos , Rim Policístico Autossômico Dominante/diagnóstico por imagem , Estudos Retrospectivos , Estudos Prospectivos , Reprodutibilidade dos Testes , Inteligência Artificial , Rim/diagnóstico por imagem , Imageamento por Ressonância Magnética/métodosRESUMO
This paper aims to identify uncommon cardiothoracic diseases and patterns on chest X-ray images. Training a machine learning model to classify rare diseases with multi-label indications is challenging without sufficient labeled training samples. Our model leverages the information from common diseases and adapts to perform on less common mentions. We propose to use multi-label few-shot learning (FSL) schemes including neighborhood component analysis loss, generating additional samples using distribution calibration and fine-tuning based on multi-label classification loss. We utilize the fact that the widely adopted nearest neighbor-based FSL schemes like ProtoNet are Voronoi diagrams in feature space. In our method, the Voronoi diagrams in the features space generated from multi-label schemes are combined into our geometric DeepVoro Multi-label ensemble. The improved performance in multi-label few-shot classification using the multi-label ensemble is demonstrated in our experiments (The code is publicly available at https://github.com/Saurabh7/Few-shot-learning-multilabel-cxray).