Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Performance of the Winning Algorithms of the RSNA 2022 Cervical Spine Fracture Detection Challenge.

Purpose To evaluate and report the performance of the winning algorithms of the Radiological Society of North America Cervical Spine Fracture AI Challenge. Materials and Methods The competition was open to the public on Kaggle from July 28 to October 27, 2022. A sample of 3112 CT scans with and without cervical spine fractures (CSFx) were assembled from multiple sites (12 institutions across six continents) and prepared for the competition. The test set had 1093 scans (private test set: n = 789; mean age, 53.40 years ± 22.86 [SD]; 509 males; public test set: n = 304; mean age, 52.51 years ± 20.73; 189 males) and 847 fractures. The eight top-performing artificial intelligence (AI) algorithms were retrospectively evaluated, and the area under the receiver operating characteristic curve (AUC) value, F1 score, sensitivity, and specificity were calculated. Results A total of 1108 contestants composing 883 teams worldwide participated in the competition. The top eight AI models showed high performance, with a mean AUC value of 0.96 (95% CI: 0.95, 0.96), mean F1 score of 90% (95% CI: 90%, 91%), mean sensitivity of 88% (95% Cl: 86%, 90%), and mean specificity of 94% (95% CI: 93%, 96%). The highest values reported for previous models were an AUC of 0.85, F1 score of 81%, sensitivity of 76%, and specificity of 97%. Conclusion The competition successfully facilitated the development of AI models that could detect and localize CSFx on CT scans with high performance outcomes, which appear to exceed known values of previously reported models. Further study is needed to evaluate the generalizability of these models in a clinical environment. Keywords: Cervical Spine, Fracture Detection, Machine Learning, Artificial Intelligence Algorithms, CT, Head/Neck Supplemental material is available for this article. © RSNA, 2024.

GenomeMUSter mouse genetic variation service enables multitrait, multipopulation data integration and analysis.

Hundreds of inbred mouse strains and intercross populations have been used to characterize the function of genetic variants that contribute to disease. Thousands of disease-relevant traits have been characterized in mice and made publicly available. New strains and populations including consomics, the collaborative cross, expanded BXD, and inbred wild-derived strains add to existing complex disease mouse models, mapping populations, and sensitized backgrounds for engineered mutations. The genome sequences of inbred strains, along with dense genotypes from others, enable integrated analysis of trait-variant associations across populations, but these analyses are hampered by the sparsity of genotypes available. Moreover, the data are not readily interoperable with other resources. To address these limitations, we created a uniformly dense variant resource by harmonizing multiple data sets. Missing genotypes were imputed using the Viterbi algorithm with a data-driven technique that incorporates local phylogenetic information, an approach that is extendable to other model organisms. The result is a web- and programmatically accessible data service called GenomeMUSter, comprising single-nucleotide variants covering 657 strains at 106.8 million segregating sites. Interoperation with phenotype databases, analytic tools, and other resources enable a wealth of applications, including multitrait, multipopulation meta-analysis. We show this in cross-species comparisons of type 2 diabetes and substance use disorder meta-analyses, leveraging mouse data to characterize the likely role of human variant effects in disease. Other applications include refinement of mapped loci and prioritization of strain backgrounds for disease modeling to further unlock extant mouse diversity for genetic and genomic studies in health and disease.

The RSNA Cervical Spine Fracture CT Dataset.

This dataset is composed of cervical spine CT images with annotations related to fractures; it is available at https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/.

GenomeMUSter mouse genetic variation service enables multi-trait, multi-population data integration and analyses.

Hundreds of inbred laboratory mouse strains and intercross populations have been used to functionalize genetic variants that contribute to disease. Thousands of disease relevant traits have been characterized in mice and made publicly available. New strains and populations including the Collaborative Cross, expanded BXD and inbred wild-derived strains add to set of complex disease mouse models, genetic mapping resources and sensitized backgrounds against which to evaluate engineered mutations. The genome sequences of many inbred strains, along with dense genotypes from others could allow integrated analysis of trait - variant associations across populations, but these analyses are not feasible due to the sparsity of genotypes available. Moreover, the data are not readily interoperable with other resources. To address these limitations, we created a uniformly dense data resource by harmonizing multiple variant datasets. Missing genotypes were imputed using the Viterbi algorithm with a data-driven technique that incorporates local phylogenetic information, an approach that is extensible to other model organism species. The result is a web- and programmatically-accessible data service called GenomeMUSter ( https://muster.jax.org ), comprising allelic data covering 657 strains at 106.8M segregating sites. Interoperation with phenotype databases, analytic tools and other resources enable a wealth of applications including multi-trait, multi-population meta-analysis. We demonstrate this in a cross-species comparison of the meta-analysis of Type 2 Diabetes and of substance use disorders, resulting in the more specific characterization of the role of human variant effects in light of mouse phenotype data. Other applications include refinement of mapped loci and prioritization of strain backgrounds for disease modeling to further unlock extant mouse diversity for genetic and genomic studies in health and disease.

Mouse phenome database: curated data repository with interactive multi-population and multi-trait analyses.

The Mouse Phenome Database continues to serve as a curated repository and analysis suite for measured attributes of members of diverse mouse populations. The repository includes annotation to community standard ontologies and guidelines, a database of allelic states for 657 mouse strains, a collection of protocols, and analysis tools for flexible, interactive, user directed analyses that increasingly integrates data across traits and populations. The database has grown from its initial focus on a standard set of inbred strains to include heterogeneous mouse populations such as the Diversity Outbred and mapping crosses and well as Collaborative Cross, Hybrid Mouse Diversity Panel, and recombinant inbred strains. Most recently the system has expanded to include data from the International Mouse Phenotyping Consortium. Collectively these data are accessible by API and provided with an interactive tool suite that enables users' persistent selection, storage, and operation on collections of measures. The tool suite allows basic analyses, advanced functions with dynamic visualization including multi-population meta-analysis, multivariate outlier detection, trait pattern matching, correlation analyses and other functions. The data resources and analysis suite provide users a flexible environment in which to explore the basis of phenotypic variation in health and disease across the lifespan.

Variation in histone configurations correlates with gene expression across nine inbred strains of mice.

The Diversity Outbred (DO) mice and their inbred founders are widely used models of human disease. However, although the genetic diversity of these mice has been well documented, their epigenetic diversity has not. Epigenetic modifications, such as histone modifications and DNA methylation, are important regulators of gene expression and, as such, are a critical mechanistic link between genotype and phenotype. Therefore, creating a map of epigenetic modifications in the DO mice and their founders is an important step toward understanding mechanisms of gene regulation and the link to disease in this widely used resource. To this end, we performed a strain survey of epigenetic modifications in hepatocytes of the DO founders. We surveyed four histone modifications (H3K4me1, H3K4me3, H3K27me3, and H3K27ac), as well as DNA methylation. We used ChromHMM to identify 14 chromatin states, each of which represents a distinct combination of the four histone modifications. We found that the epigenetic landscape is highly variable across the DO founders and is associated with variation in gene expression across strains. We found that epigenetic state imputed into a population of DO mice recapitulated the association with gene expression seen in the founders, suggesting that both histone modifications and DNA methylation are highly heritable mechanisms of gene expression regulation. We illustrate how DO gene expression can be aligned with inbred epigenetic states to identify putative cis-regulatory regions. Finally, we provide a data resource that documents strain-specific variation in the chromatin state and DNA methylation in hepatocytes across nine widely used strains of laboratory mice.

The Ontology of Biological Attributes (OBA)-computational traits for the life sciences.

Existing phenotype ontologies were originally developed to represent phenotypes that manifest as a character state in relation to a wild-type or other reference. However, these do not include the phenotypic trait or attribute categories required for the annotation of genome-wide association studies (GWAS), Quantitative Trait Loci (QTL) mappings or any population-focussed measurable trait data. The integration of trait and biological attribute information with an ever increasing body of chemical, environmental and biological data greatly facilitates computational analyses and it is also highly relevant to biomedical and clinical applications. The Ontology of Biological Attributes (OBA) is a formalised, species-independent collection of interoperable phenotypic trait categories that is intended to fulfil a data integration role. OBA is a standardised representational framework for observable attributes that are characteristics of biological entities, organisms, or parts of organisms. OBA has a modular design which provides several benefits for users and data integrators, including an automated and meaningful classification of trait terms computed on the basis of logical inferences drawn from domain-specific ontologies for cells, anatomical and other relevant entities. The logical axioms in OBA also provide a previously missing bridge that can computationally link Mendelian phenotypes with GWAS and quantitative traits. The term components in OBA provide semantic links and enable knowledge and data integration across specialised research community boundaries, thereby breaking silos.

A genetic locus complements resistance to Bordetella pertussis-induced histamine sensitization.

Histamine plays pivotal role in normal physiology and dysregulated production of histamine or signaling through histamine receptors (HRH) can promote pathology. Previously, we showed that Bordetella pertussis or pertussis toxin can induce histamine sensitization in laboratory inbred mice and is genetically controlled by Hrh1/HRH1. HRH1 allotypes differ at three amino acid residues with P263-V313-L331 and L263-M313-S331, imparting sensitization and resistance respectively. Unexpectedly, we found several wild-derived inbred strains that carry the resistant HRH1 allotype (L263-M313-S331) but exhibit histamine sensitization. This suggests the existence of a locus modifying pertussis-dependent histamine sensitization. Congenic mapping identified the location of this modifier locus on mouse chromosome 6 within a functional linkage disequilibrium domain encoding multiple loci controlling sensitization to histamine. We utilized interval-specific single-nucleotide polymorphism (SNP) based association testing across laboratory and wild-derived inbred mouse strains and functional prioritization analyses to identify candidate genes for this modifier locus. Atg7, Plxnd1, Tmcc1, Mkrn2, Il17re, Pparg, Lhfpl4, Vgll4, Rho and Syn2 are candidate genes within this modifier locus, which we named Bphse, enhancer of Bordetella pertussis induced histamine sensitization. Taken together, these results identify, using the evolutionarily significant diversity of wild-derived inbred mice, additional genetic mechanisms controlling histamine sensitization.

Mouse Phenome Database: towards a more FAIR-compliant and TRUST-worthy data repository and tool suite for phenotypes and genotypes.

The Mouse Phenome Database (MPD; https://phenome.jax.org; RRID:SCR_003212), supported by the US National Institutes of Health, is a Biomedical Data Repository listed in the Trans-NIH Biomedical Informatics Coordinating Committee registry. As an increasingly FAIR-compliant and TRUST-worthy data repository, MPD accepts phenotype and genotype data from mouse experiments and curates, organizes, integrates, archives, and distributes those data using community standards. Data are accompanied by rich metadata, including widely used ontologies and detailed protocols. Data are from all over the world and represent genetic, behavioral, morphological, and physiological disease-related characteristics in mice at baseline or those exposed to drugs or other treatments. MPD houses data from over 6000 strains and populations, representing many reproducible strain types and heterogenous populations such as the Diversity Outbred where each mouse is unique but can be genotyped throughout the genome. A suite of analysis tools is available to aggregate, visualize, and analyze these data within and across studies and populations in an increasingly traceable and reproducible manner. We have refined existing resources and developed new tools to continue to provide users with access to consistent, high-quality data that has translational relevance in a modernized infrastructure that enables interaction with a suite of bioinformatics analytic and data services.

Mortality Prediction Analysis among COVID-19 Inpatients Using Clinical Variables and Deep Learning Chest Radiography Imaging Features.

The emergence of the COVID-19 pandemic over a relatively brief interval illustrates the need for rapid data-driven approaches to facilitate clinical decision making. We examined a machine learning process to predict inpatient mortality among COVID-19 patients using clinical and chest radiographic data. Modeling was performed with a de-identified dataset of encounters prior to widespread vaccine availability. Non-imaging predictors included demographics, pre-admission clinical history, and past medical history variables. Imaging features were extracted from chest radiographs by applying a deep convolutional neural network with transfer learning. A multi-layer perceptron combining 64 deep learning features from chest radiographs with 98 patient clinical features was trained to predict mortality. The Local Interpretable Model-Agnostic Explanations (LIME) method was used to explain model predictions. Non-imaging data alone predicted mortality with an ROC-AUC of 0.87 ± 0.03 (mean ± SD), while the addition of imaging data improved prediction slightly (ROC-AUC: 0.91 ± 0.02). The application of LIME to the combined imaging and clinical model found HbA1c values to contribute the most to model prediction (17.1 ± 1.7%), while imaging contributed 8.8 ± 2.8%. Age, gender, and BMI contributed 8.7%, 8.2%, and 7.1%, respectively. Our findings demonstrate a viable explainable AI approach to quantify the contributions of imaging and clinical data to COVID mortality predictions.

The RSNA Pulmonary Embolism CT Dataset.

Supplemental material is available for this article.

Electronic decision support and diarrhoeal disease guideline adherence (mHDM): a cluster randomised controlled trial.

BACKGROUND: Acute diarrhoeal disease management often requires rehydration alone without antibiotics. However, non-indicated antibiotics are frequently ordered and this is an important driver of antimicrobial resistance. The mHealth Diarrhoea Management (mHDM) trial aimed to establish whether electronic decision support improves rehydration and antibiotic guideline adherence in resource-limited settings. METHODS: A cluster randomised controlled trial was done at ten district hospitals in Bangladesh. Inclusion criteria were patients aged 2 months or older with uncomplicated acute diarrhoea. Admission orders were observed without intervention in the pre-intervention period, followed by randomisation to electronic (rehydration calculator) or paper formatted WHO guidelines for the intervention period. The primary outcome was rate of intravenous fluid ordered as a binary variable. Generalised linear mixed-effect models, accounting for hospital clustering, served as the analytical framework; the analysis was intention to treat. The trial is registered with ClinicalTrials.gov (NCT03154229) and is completed. FINDINGS: From March 11 to Sept 10, 2018, 4975 patients (75·6%) of 6577 screened patients were enrolled. The intervention effect for the primary outcome showed no significant differences in rates of intravenous fluids ordered as a function of decision-support type. Intravenous fluid orders decreased by 0·9 percentage points for paper electronic decision support and 4·2 percentage points for electronic decision support, with a 4·2-point difference between decision-support types in the intervention period (paper 98·7% [95% CI 91·8-99·8] vs electronic 94·5% [72·2-99·1]; pinteraction=0·31). Adverse events such as complications and mortality events were uncommon and could not be statistically estimated. INTERPRETATION: Although intravenous fluid orders did not change, electronic decision support was associated with increases in the volume of intravenous fluid ordered and decreases in antibiotics ordered, which are consistent with WHO guidelines. FUNDING: US National Institutes of Health.

CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV.

Tuberculosis (TB) is the leading cause of preventable death in HIV-positive patients, and yet often remains undiagnosed and untreated. Chest x-ray is often used to assist in diagnosis, yet this presents additional challenges due to atypical radiographic presentation and radiologist shortages in regions where co-infection is most common. We developed a deep learning algorithm to diagnose TB using clinical information and chest x-ray images from 677 HIV-positive patients with suspected TB from two hospitals in South Africa. We then sought to determine whether the algorithm could assist clinicians in the diagnosis of TB in HIV-positive patients as a web-based diagnostic assistant. Use of the algorithm resulted in a modest but statistically significant improvement in clinician accuracy (p = 0.002), increasing the mean clinician accuracy from 0.60 (95% CI 0.57, 0.63) without assistance to 0.65 (95% CI 0.60, 0.70) with assistance. However, the accuracy of assisted clinicians was significantly lower (p < 0.001) than that of the stand-alone algorithm, which had an accuracy of 0.79 (95% CI 0.77, 0.82) on the same unseen test cases. These results suggest that deep learning assistance may improve clinician accuracy in TB diagnosis using chest x-rays, which would be valuable in settings with a high burden of HIV/TB co-infection. Moreover, the high accuracy of the stand-alone algorithm suggests a potential value particularly in settings with a scarcity of radiological expertise.

Erratum: Author Correction: PENet-a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging.

[This corrects the article DOI: 10.1038/s41746-020-0266-y.].

PENet-a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging.

Pulmonary embolism (PE) is a life-threatening clinical problem and computed tomography pulmonary angiography (CTPA) is the gold standard for diagnosis. Prompt diagnosis and immediate treatment are critical to avoid high morbidity and mortality rates, yet PE remains among the diagnoses most frequently missed or delayed. In this study, we developed a deep learning model-PENet, to automatically detect PE on volumetric CTPA scans as an end-to-end solution for this purpose. The PENet is a 77-layer 3D convolutional neural network (CNN) pretrained on the Kinetics-600 dataset and fine-tuned on a retrospective CTPA dataset collected from a single academic institution. The PENet model performance was evaluated in detecting PE on data from two different institutions: one as a hold-out dataset from the same institution as the training data and a second collected from an external institution to evaluate model generalizability to an unrelated population dataset. PENet achieved an AUROC of 0.84 [0.82-0.87] on detecting PE on the hold out internal test set and 0.85 [0.81-0.88] on external dataset. PENet also outperformed current state-of-the-art 3D CNN models. The results represent successful application of an end-to-end 3D CNN model for the complex task of PE diagnosis without requiring computationally intensive and time consuming preprocessing and demonstrates sustained performance on data from an external institution. Our model could be applied as a triage tool to automatically identify clinically important PEs allowing for prioritization for diagnostic radiology interpretation and improved care pathways via more efficient diagnosis.

Dose-Response Relationship Between Intracoronary Acetylcholine and Minimal Lumen Diameter in Coronary Endothelial Function Testing of Women and Men With Angina and No Obstructive Coronary Artery Disease.

BACKGROUND: Intracoronary acetylcholine (Ach) provocation testing is the gold standard for assessing coronary endothelial function. However, dosing regimens of Ach are quite varied in the literature, and there are limited data evaluating the optimal dose. We evaluated the dose-response relationship between Ach and minimal lumen diameter (MLD) by sex and studied whether incremental intracoronary Ach doses given during endothelial function testing improve its diagnostic utility. METHODS: We evaluated 65 men and 212 women with angina and no obstructive coronary artery disease who underwent endothelial function testing using the highest tolerable dose of intracoronary Ach, up to 200 µg. Epicardial endothelial dysfunction was defined as a decrease in MLD >20% after intracoronary Ach by quantitative coronary angiography. We used a linear mixed effects model to evaluate the dose-response relationship. Deming regression analysis was done to compare the %MLD constriction after incremental doses of intracoronary Ach. RESULTS: The mean age was 53.5 years. Endothelial dysfunction was present in 186 (68.1%). Among men with endothelial dysfunction, there was a significant decrease in MLD/10 µg of Ach at doses above 50 µg and 100 µg, while this decrease in MLD was not observed in women (P<0.001). The %MLD constriction at 20 µg versus 50 µg and 50 µg versus 100 µg were not equivalent while the %MLD constriction at 100 µg versus 200 µg were equivalent. CONCLUSIONS: Women and men appear to have different responses to Ach during endothelial function testing. In addition to having a greater response to intracoronary Ach at all doses, men also demonstrate an Ach-MLD dose-response relationship with doses up to 200 µg, while women have minimal change in MLD with doses above 50 µg. An incremental dosing regimen during endothelial function testing appears to improve the diagnostic utility of the test and should be adjusted based on the sex of the patient.

Impact of a deep learning assistant on the histopathologic classification of liver cancer.

Artificial intelligence (AI) algorithms continue to rival human performance on a variety of clinical tasks, while their actual impact on human diagnosticians, when incorporated into clinical workflows, remains relatively unexplored. In this study, we developed a deep learning-based assistant to help pathologists differentiate between two subtypes of primary liver cancer, hepatocellular carcinoma and cholangiocarcinoma, on hematoxylin and eosin-stained whole-slide images (WSI), and evaluated its effect on the diagnostic performance of 11 pathologists with varying levels of expertise. Our model achieved accuracies of 0.885 on a validation set of 26 WSI, and 0.842 on an independent test set of 80 WSI. Although use of the assistant did not change the mean accuracy of the 11 pathologists (p = 0.184, OR = 1.281), it significantly improved the accuracy (p = 0.045, OR = 1.499) of a subset of nine pathologists who fell within well-defined experience levels (GI subspecialists, non-GI subspecialists, and trainees). In the assisted state, model accuracy significantly impacted the diagnostic decisions of all 11 pathologists. As expected, when the model's prediction was correct, assistance significantly improved accuracy (p = 0.000, OR = 4.289), whereas when the model's prediction was incorrect, assistance significantly decreased accuracy (p = 0.000, OR = 0.253), with both effects holding across all pathologist experience levels and case difficulty levels. Our results highlight the challenges of translating AI models into the clinical setting, and emphasize the importance of taking into account potential unintended negative consequences of model assistance when designing and testing medical AI-assistance tools.

Asymmetric dimethylarginine predicts impaired epicardial coronary vasomotion in patients with angina in the absence of obstructive coronary artery disease.

BACKGROUND: Impaired epicardial coronary vasomotion is a potential mechanism of angina and a predictor of adverse cardiovascular outcomes in patients without angiographic evidence of obstructive coronary artery disease (CAD). We sought to evaluate the association of asymmetric dimethylarginine (ADMA)-a marker of nitric oxide-mediated vascular dysfunction-with epicardial coronary vasomotor dysfunction in this select population. METHODS: Invasive testing for epicardial vasomotor dysfunction was performed using intracoronary acetylcholine in the left anterior descending coronary artery. Impaired vasomotor response was defined as a luminal constriction of >20% on quantitative coronary angiography. Plasma ADMA levels were measured using high performance liquid chromatography. A robust multivariate linear mixed-effect model approach and Akaike information criterion were used to determine predictors of vasomotor dysfunction. RESULTS: In 191 patients with angina in the absence of obstructive CAD, abnormal epicardial vasomotion was observed in 137 (71.7%) patients. Median ADMA rose as the extent of impairment progressed: none (0.48 [0.44-0.59] µM), any (0.51 [0.46-0.60] µM, p = 0.12), focal (0.54 [0.49,0.61] µM, p = 0.17), and diffuse (0.55 [0.49,0.63] µM, p = 0.02). In unadjusted analysis, ADMA was highly predictive of vasomotor dysfunction (χ2=15.1, p = 0.002). Notably, ADMA remained a significant predictor even after adjusting for other factors in the best fit model (χ2=10.0, p = 0.02). CONCLUSIONS: ADMA is an independent predictor of epicardial coronary vasomotor dysfunction in patients with angina in the absence of obstructive CAD. These data support a very early mechanistic role of ADMA in the continuum of atherosclerotic heart disease.