Improving reference standards for validation of AI-based radiography.

OBJECTIVE: Demonstrate the importance of combining multiple readers' opinions, in a context-aware manner, when establishing the reference standard for validation of artificial intelligence (AI) applications for, e.g. chest radiographs. By comparing individual readers, majority vote of a panel, and panel-based discussion, we identify methods which maximize interobserver agreement and label reproducibility. METHODS: 1100 frontal chest radiographs were evaluated for 6 findings: airspace opacity, cardiomegaly, pulmonary edema, fracture, nodules, and pneumothorax. Each image was reviewed by six radiologists, first individually and then via asynchronous adjudication (web-based discussion) in two panels of three readers to resolve disagreements within each panel. We quantified the reproducibility of each method by measuring interreader agreement. RESULTS: Panel-based majority vote improved agreement relative to individual readers for all findings. Most disagreements were resolved with two rounds of adjudication, which further improved reproducibility for some findings, particularly reducing misses. Improvements varied across finding categories, with adjudication improving agreement for cardiomegaly, fractures, and pneumothorax. CONCLUSION: The likelihood of interreader agreement, even within panels of US board-certified radiologists, must be considered before reads can be used as a reference standard for validation of proposed AI tools. Agreement and, by extension, reproducibility can be improved by applying majority vote, maximum sensitivity, or asynchronous adjudication for different findings, which supports the development of higher quality clinical research. ADVANCES IN KNOWLEDGE: A panel of three experts is a common technique for establishing reference standards when ground truth is not available for use in AI validation. The manner in which differing opinions are resolved is shown to be important, and has not been previously explored.

Chest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted Evaluation.

BackgroundDeep learning has the potential to augment the use of chest radiography in clinical radiology, but challenges include poor generalizability, spectrum bias, and difficulty comparing across studies.PurposeTo develop and evaluate deep learning models for chest radiograph interpretation by using radiologist-adjudicated reference standards.Materials and MethodsDeep learning models were developed to detect four findings (pneumothorax, opacity, nodule or mass, and fracture) on frontal chest radiographs. This retrospective study used two data sets. Data set 1 (DS1) consisted of 759 611 images from a multicity hospital network and ChestX-ray14 is a publicly available data set with 112 120 images. Natural language processing and expert review of a subset of images provided labels for 657 954 training images. Test sets consisted of 1818 and 1962 images from DS1 and ChestX-ray14, respectively. Reference standards were defined by radiologist-adjudicated image review. Performance was evaluated by area under the receiver operating characteristic curve analysis, sensitivity, specificity, and positive predictive value. Four radiologists reviewed test set images for performance comparison. Inverse probability weighting was applied to DS1 to account for positive radiograph enrichment and estimate population-level performance.ResultsIn DS1, population-adjusted areas under the receiver operating characteristic curve for pneumothorax, nodule or mass, airspace opacity, and fracture were, respectively, 0.95 (95% confidence interval [CI]: 0.91, 0.99), 0.72 (95% CI: 0.66, 0.77), 0.91 (95% CI: 0.88, 0.93), and 0.86 (95% CI: 0.79, 0.92). With ChestX-ray14, areas under the receiver operating characteristic curve were 0.94 (95% CI: 0.93, 0.96), 0.91 (95% CI: 0.89, 0.93), 0.94 (95% CI: 0.93, 0.95), and 0.81 (95% CI: 0.75, 0.86), respectively.ConclusionExpert-level models for detecting clinically relevant chest radiograph findings were developed for this study by using adjudicated reference standards and with population-level performance estimation. Radiologist-adjudicated labels for 2412 ChestX-ray14 validation set images and 1962 test set images are provided.© RSNA, 2019Online supplemental material is available for this article.See also the editorial by Chang in this issue.

Scalable and accurate deep learning with electronic health records.

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two US academic medical centers with 216,221 adult patients hospitalized for at least 24 h. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting: in-hospital mortality (area under the receiver operator curve [AUROC] across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed traditional, clinically-used predictive models in all cases. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios. In a case study of a particular prediction, we demonstrate that neural networks can be used to identify relevant information from the patient's chart.

Characterization and prediction of the mechanism of action of antibiotics through NMR metabolomics.

BACKGROUND: The emergence of antibiotic resistant pathogenic bacteria has reduced our ability to combat infectious diseases. At the same time the numbers of new antibiotics reaching the market have decreased. This situation has created an urgent need to discover novel antibiotic scaffolds. Recently, the application of pattern recognition techniques to identify molecular fingerprints in 'omics' studies, has emerged as an important tool in biomedical research and laboratory medicine to identify pathogens, to monitor therapeutic treatments or to develop drugs with improved metabolic stability, toxicological profile and efficacy. Here, we hypothesize that a combination of metabolic intracellular fingerprints and extracellular footprints would provide a more comprehensive picture about the mechanism of action of novel antibiotics in drug discovery programs. RESULTS: In an attempt to integrate the metabolomics approach as a classification tool in the drug discovery processes, we have used quantitative (1)H NMR spectroscopy to study the metabolic response of Escherichia coli cultures to different antibiotics. Within the frame of our study the effects of five different and well-known antibiotic classes on the bacterial metabolome were investigated both by intracellular fingerprint and extracellular footprint analysis. The metabolic fingerprints and footprints of bacterial cultures were affected in a distinct manner and provided complementary information regarding intracellular and extracellular targets such as protein synthesis, DNA and cell wall. While cell cultures affected by antibiotics that act on intracellular targets showed class-specific fingerprints, the metabolic footprints differed significantly only when antibiotics that target the cell wall were applied. In addition, using a training set of E. coli fingerprints extracted after treatment with different antibiotic classes, the mode of action of streptomycin, tetracycline and carbenicillin could be correctly predicted. CONCLUSION: The metabolic profiles of E. coli treated with antibiotics with intracellular and extracellular targets could be separated in fingerprint and footprint analysis, respectively and provided complementary information. Based on the specific fingerprints obtained for different classes of antibiotics, the mode of action of several antibiotics could be predicted. The same classification approach should be applicable to studies of other pathogenic bacteria.

Metabolic profiling of serum samples by 1H nuclear magnetic resonance spectroscopy as a potential diagnostic approach for septic shock.

OBJECTIVES: To determine whether a nuclear magnetic resonance-based metabolomics approach can be useful for the early diagnosis and prognosis of septic shock in ICUs. DESIGN: Laboratory-based study. SETTING: University research laboratory. SUBJECTS: Serum samples from septic shock patients and ICU controls (ICU patients with systemic inflammatory response syndrome but not suspected of having an infection) were collected within 24 hours of admittance to the ICU. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: H nuclear magnetic resonance spectra of septic shock and ICU control samples were analyzed and quantified using a targeted profiling approach. By applying multivariate statistical analysis (e.g., orthogonal partial least squares discriminant analysis), we were able to distinguish the patient groups and detect specific metabolic patterns. Some of the metabolites were found to have a significant impact on the separation between septic shock and control samples. These metabolites could be interpreted in terms of a biological human response to septic shock and they might serve as a biomarker pattern for septic shock in ICUs. Additionally, nuclear magnetic resonance-based metabolomics was evaluated in order to detect metabolic variation between septic shock survivors and nonsurvivors and to predict patient outcome. The area under the receiver operating characteristic curve indicated an excellent predictive ability for the constructed orthogonal partial least squares discriminant analysis models (septic shock vs ICU controls: area under the receiver operating characteristic curve = 0.98; nonsurvivors vs survivors: area under the receiver operating characteristic curve = 1). CONCLUSIONS: Our results indicate that nuclear magnetic resonance-based metabolic profiling could be used for diagnosis and mortality prediction of septic shock in the ICU.

Effect of aluminium and copper on biofilm development of Pseudomonas pseudoalcaligenes KF707 and P. fluorescens as a function of different media compositions.

Bioremediation efforts worldwide are faced with the problem of metals interfering with the degradation of organic pollutants. There has been little systematic investigation into how the important environmental factors of media composition, buffering agent, and carbon source affect the exertion of metal toxicity on bacteria. This study aimed to systematically separate and investigate the influence of these factors by examining planktonic and biofilm establishment and growth. Two Pseudomonads were chosen, the PCB degrader P. pseudoalcaligenes KF707 and P. fluorescens. The two strains were grown in the presence of Al(3+) and Cu(2+) under different media conditions of carbon source (Lysogeny broth, biphenyl, succinate, aspartic acid, butyric acid, oxaloacetic acid, putrescine and benzoic acid) and under different buffering conditions (high and low phosphate or MOPS). These experiments allowed for the elucidation of an effect of different metabolic conditions and metal speciation on planktonic bacteria growth and biofilm establishment and development under metal stress. Here we show that the nature of bacterial growth (planktonic and biofilm development) is dramatically affected by the interplay between toxic metals, carbon source and media composition. The capacity of a media to bind toxic metals as well as quality of carbon source greatly influences the amount of metal that bacteria can tolerate, depending on both the bacterium and metal. Future studies evaluating metal ion toxicity should consider these effects, as well as their interactions with specific environments into account in order to improve clean-up success.

Metabolic profiling of vitamin C deficiency in Gulo-/- mice using proton NMR spectroscopy.

Nutrient deficiencies are an ongoing problem in many populations and ascorbic acid is a key vitamin whose mild or acute absence leads to a number of conditions including the famously debilitating scurvy. As such, the biochemical effects of ascorbate deficiency merit ongoing scrutiny, and the Gulo knockout mouse provides a useful model for the metabolomic examination of vitamin C deficiency. Like humans, these animals are incapable of synthesizing ascorbic acid but with dietary supplements are otherwise healthy and grow normally. In this study, all vitamin C sources were removed after weaning from the diet of Gulo-/- mice (n = 7) and wild type controls (n = 7) for 12 weeks before collection of serum. A replicate study was performed with similar parameters but animals were harvested pre-symptomatically after 2-3 weeks. The serum concentration of 50 metabolites was determined by quantitative profiling of 1D proton NMR spectra. Multivariate statistical models were used to describe metabolic changes as compared to control animals; replicate study animals were used for external validation of the resulting models. The results of the study highlight the metabolites and pathways known to require ascorbate for proper flux.

Metabolomic response to exercise training in lean and diet-induced obese mice.

Exercise training is a common therapeutic approach known to antagonize the metabolic consequences of obesity. The aims of the present study were to examine 1) whether short-term, moderate-intensity exercise training alters the basal metabolite profile and 2) if 10 days of mild exercise training can correct obesity-induced shifts in metabolic spectra. After being weaned, male C57BL/6J littermates were randomly divided into two diet groups: low fat (LF) or high fat (HF). After 12 wk of dietary manipulation, HF animals were obese and hyperglycemic compared with LF animals. Mice from each group were further divided into sedentary or exercise treatments. Exercise training consisted of wheel running exercise (2 h/day, 10 days, 5.64 m/min). After exercise training, animals were rested (36 h) and fasted (6 h) before serum collection. Samples were analyzed by high-resolution one-dimensional proton NMR. Fifty high- and medium-concentration metabolites were identified. Pattern recognition algorithms and multivariate modeling were used to identify and isolate significant metabolites changing in response to HF and exercise training. The results showed that while exercise can mitigate some of the abnormal patterns in metabolic spectra induced by HF diet feeding, they cannot negate it. In fact, when the effects of diet and exercise were compared, diet was a stronger predictor and had the larger influence on the metabolic profile. External validation of models showed that diet could be correctly classified with an accuracy of 89%, whereas exercise training could be classified 73% of the time. The results demonstrate metabolomics to effectively characterize obesity-induced perturbations in metabolism and support the concept that exercise is beneficial for this condition.

HMDB: the Human Metabolome Database.

The Human Metabolome Database (HMDB) is currently the most complete and comprehensive curated collection of human metabolite and human metabolism data in the world. It contains records for more than 2180 endogenous metabolites with information gathered from thousands of books, journal articles and electronic databases. In addition to its comprehensive literature-derived data, the HMDB also contains an extensive collection of experimental metabolite concentration data compiled from hundreds of mass spectra (MS) and Nuclear Magnetic resonance (NMR) metabolomic analyses performed on urine, blood and cerebrospinal fluid samples. This is further supplemented with thousands of NMR and MS spectra collected on purified, reference metabolites. Each metabolite entry in the HMDB contains an average of 90 separate data fields including a comprehensive compound description, names and synonyms, structural information, physico-chemical data, reference NMR and MS spectra, biofluid concentrations, disease associations, pathway information, enzyme data, gene sequence data, SNP and mutation data as well as extensive links to images, references and other public databases. Extensive searching, relational querying and data browsing tools are also provided. The HMDB is designed to address the broad needs of biochemists, clinical chemists, physicians, medical geneticists, nutritionists and members of the metabolomics community. The HMDB is available at: www.hmdb.ca.

Human chromosome 7: DNA sequence and biology.

DNA sequence and annotation of the entire human chromosome 7, encompassing nearly 158 million nucleotides of DNA and 1917 gene structures, are presented. To generate a higher order description, additional structural features such as imprinted genes, fragile sites, and segmental duplications were integrated at the level of the DNA sequence with medical genetic data, including 440 chromosome rearrangement breakpoints associated with disease. This approach enabled the discovery of candidate genes for developmental diseases including autism.