Use and misuse of random forest variable importance metrics in medicine: demonstrations through incident stroke prediction.

BACKGROUND: Machine learning tools such as random forests provide important opportunities for modeling large, complex modern data generated in medicine. Unfortunately, when it comes to understanding why machine learning models are predictive, applied research continues to rely on 'out of bag' (OOB) variable importance metrics (VIMPs) that are known to have considerable shortcomings within the statistics community. After explaining the limitations of OOB VIMPs - including bias towards correlated features and limited interpretability - we describe a modern approach called 'knockoff VIMPs' and explain its advantages. METHODS: We first evaluate current VIMP practices through an in-depth literature review of 50 recent random forest manuscripts. Next, we recommend organized and interpretable strategies for analysis with knockoff VIMPs, including computing them for groups of features and considering multiple model performance metrics. To demonstrate methods, we develop a random forest to predict 5-year incident stroke in the Sleep Heart Health Study and compare results based on OOB and knockoff VIMPs. RESULTS: Nearly all papers in the literature review contained substantial limitations in their use of VIMPs. In our demonstration, using OOB VIMPs for individual variables suggested two highly correlated lung function variables (forced expiratory volume, forced vital capacity) as the best predictors of incident stroke, followed by age and height. Using an organized analytic approach that considered knockoff VIMPs of both groups of features and individual features, the largest contributions to model sensitivity were medications (especially cardiovascular) and measured medical risk factors, while the largest contributions to model specificity were age, diastolic blood pressure, self-reported medical risk factors, polysomnography features, and pack-years of smoking. Thus, we reach very different conclusions about stroke risk factors using OOB VIMPs versus knockoff VIMPs. CONCLUSIONS: The near-ubiquitous reliance on OOB VIMPs may provide misleading results for researchers who use such methods to guide their research. Given the rapid pace of scientific inquiry using machine learning, it is essential to bring modern knockoff VIMPs that are interpretable and unbiased into widespread applied practice to steer researchers using random forest machine learning toward more meaningful results.

Machine Learning Classification of False-Positive Human Immunodeficiency Virus Screening Results.

BACKGROUND: Human immunodeficiency virus (HIV) screening has improved significantly in the past decade as we have implemented tests that include antigen detection of p24. Incorporation of p24 detection narrows the window from 4 to 2 weeks between infection acquisition and ability to detect infection, reducing unintentional spread of HIV. The fourth- and fifth-generation HIV (HIV5G) screening tests in low prevalence populations have high numbers of false-positive screens and it is unclear if orthogonal testing improves diagnostic and public health outcomes. METHODS: We used a cohort of 60,587 HIV5G screening tests with molecular and clinical correlates collected from 2016 to 2018 and applied machine learning to generate a classifier that could predict likely true and false positivity. RESULTS: The best classification was achieved by using support vector machines and transformation of results with principle component analysis. The final classifier had an accuracy of 94% for correct classification of false-positive screens and an accuracy of 92% for classification of true-positive screens. CONCLUSIONS: Implementation of this classifier as a screening method for all HIV5G reactive screens allows for improved workflow with likely true positives reported immediately to reduce infection spread and initiate follow-up testing and treatment and likely false positives undergoing orthogonal testing utilizing the same specimen already drawn to reduce distress and follow-up visits. Application of machine learning to the clinical laboratory allows for workflow improvement and decision support to provide improved patient care and public health.

Pediatric decision limits for serologic screening of Lyme disease.

BACKGROUND: Laboratory diagnosis of Lyme disease (LD) relies on a two-tier protocol. We have observed disproportionate equivocal serologies in children requiring reflex western blot (WB) using manufacturer-provided ranges based on adult studies. We aimed to determine appropriate ranges for our pediatric population. METHODS: We performed a one-year retrospective institutional review of all 2755 children with LD testing with the Vidas® Lyme IgM II/IgG II immunoassays with reflex to WB for equivocal/positive serologies. Results were assessed by frequency distributions, optimization via percent agreement analysis, and clinical adjudication. RESULTS: The proposed ranges for IgM (negative ≤0.20, equivocal ≥0.21 to <0.32, positive ≥0.32) and IgG (negative ≤0.50, positive >0.50) allowed for a decrease in the IgM equivocal rate (7% to 2%) and IgG positive rate (15% to 13%). There was a decrease in the positive percent agreement between tiers (95% to 83% and 98% to 95%) with increase in the negative (32% to 63% and 70% to 81%) and overall (65% to 73% and 85% to 88%) percent agreements for IgM and IgG, respectively. Of 15 IgM serologies reclassified as negative with a positive WB and not positive for IgG, 8 were clinically negative, 5 were clinically positive, and two had insufficient history. Of the 10 IgG serologies reclassified as negative with a positive WB 3 were clinically positive, 6 were clinically negative and one had insufficient history. CONCLUSIONS: Our modified ranges are more suitable for our pediatric population while reducing overdiagnosis, unnecessary treatment, diagnostic uncertainty, and turnaround time.

Variable Performance in 6 Commercial SARS-CoV-2 Antibody Assays May Affect Convalescent Plasma and Seroprevalence Screening.

OBJECTIVES: Serologic detection of prior severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection is needed for definition of convalescent plasma donors, for confounding SARS-CoV-2 presentation, and for seroprevalence studies. Reliable serologic assays with independent validation are required. METHODS: Six SARS-CoV-2 antibody assays from Beckman Coulter, Euroimmun (IgG, IgA), Roche, and Siemens (Centaur, Vista) were assessed for specificity (n = 184), sensitivity (n = 154), and seroconversion in a defined cohort with clinical correlates and molecular SARS-CoV-2 results. RESULTS: Assay specificity was 99% or greater for all assays except the Euroimmun IgA (95%). Sensitivity at more than 21 days from symptom onset was 84%, 95%, 72%, 98%, 67%, and 96% for Beckman Coulter, Centaur, Vista, Roche, Euroimmun IgA, and Euroimmun IgG, respectively. Average day of seroconversion was similar between assays (8-10 d), with 2 patients not producing nucleocapsid antibodies during hospitalization. CONCLUSIONS: SARS-CoV-2 nucleocapsid antibodies may be less reliably produced early in disease than spike protein antibodies. Assessment of convalescent plasma donors at more than 30 days from symptom onset and seroprevalence studies should use assays with defined sensitivity at time points of interest because not all assays detected antibodies reliably at more than 30 days.

Multiplex assessment of SARS-CoV-2 antibodies improves assay sensitivity and correlation with neutralizing antibodies.

OBJECTIVES: Detection of antibodies to multiple SARS-CoV-2 antigens in a single assay could increase diagnostic accuracy, differentiate vaccination from natural disease, and aid in retrospective exposure determination. Correlation of binding antibody assessment in clinical assays with neutralizing antibodies is needed to better understand the humoral response to SARS-CoV-2 infection and establish of correlates of protection. METHODS: A cohort of 752 samples was used to assess specificity, sensitivity, and comparison to 6 other Conformitè Europëenne serologic assays for the BioRad SARS-CoV-2 IgG multiplex assay which measures receptor binding domain IgG (RBD), spike-S1 IgG (S1), spike-S2 IgG (S2), and nucleocapsid IgG (N). A subset of serial specimens from 14 patients was also tested for neutralizing antibodies (n = 61). RESULTS: Specificity for RBD and S1 IgG was 99.4% (n = 170) and 100% for S2 and N IgG (n = 170) in a cohort selected for probable interference. Overall assay concordance with other assays was >93% for IgG and total antibody assays and reached 100% sensitivity for clinical concordance at >14 days as a multiplex assay. RBD and S1 binding antibody positivity demonstrated 79-95% agreement with the presence of neutralizing antibodies. CONCLUSIONS: The BioRad SARS-CoV-2 IgG assay is comparable to existing assays, and achieved 100% sensitivity when all markers were included. The ability to measure antibodies against spike and nucleocapsid proteins simultaneously may be advantageous for complex clinical presentations, epidemiologic research, and in decisions regarding infection prevention strategies. Additional independent validations are needed to further determine binding antibody and neutralizing antibody correlations.