Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper From the College of American Pathologists.

CONTEXT.­: Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. OBJECTIVE.­: To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. DATA SOURCES.­: An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. CONCLUSIONS.­: Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

Development, Evaluation, and Multisite Deployment of a Machine Learning Decision Tree Algorithm To Optimize Urinalysis Parameters for Predicting Urine Culture Positivity.

PittUDT, a recursive partitioning decision tree algorithm for predicting urine culture (UC) positivity based on macroscopic and microscopic urinalysis (UA) parameters, was developed in support of a broader system-wide diagnostic stewardship initiative to increase appropriateness of UC testing. Reflex algorithm training utilized results from 19,511 paired UA and UC cases (26.8% UC positive); the average patient age was 57.4 years, and 70% of samples were from female patients. Receiver operating characteristic (ROC) analysis identified urine white blood cells (WBCs), leukocyte esterase, and bacteria as the best predictors of UC positivity, with areas under the ROC curve of 0.79, 0.78, and 0.77, respectively. Using the held-out test data set (9,773 cases; 26.3% UC positive), the PittUDT algorithm met the prespecified target of a negative predictive value above 90% and resulted in a 30 to 60% total negative proportion (true-negative plus false-negative predictions). These data show that a supervised rule-based machine learning algorithm trained on paired UA and UC data has adequate predictive ability for triaging urine specimens by identifying low-risk urine specimens, which are unlikely to grow pathogenic organisms, with a false-negative proportion under 5%. The decision tree approach also generates human-readable rules that can be easily implemented across multiple hospital sites and settings. Our work demonstrates how a data-driven approach can be used to optimize UA parameters for predicting UC positivity in a reflex protocol, with the intent of improving antimicrobial stewardship and UC utilization, a potential avenue for cost savings.

Neurovascular injury with complement activation and inflammation in COVID-19.

The underlying mechanisms by which severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) leads to acute and long-term neurological manifestations remains obscure. We aimed to characterize the neuropathological changes in patients with coronavirus disease 2019 and determine the underlying pathophysiological mechanisms. In this autopsy study of the brain, we characterized the vascular pathology, the neuroinflammatory changes and cellular and humoral immune responses by immunohistochemistry. All patients died during the first wave of the pandemic from March to July 2020. All patients were adults who died after a short duration of the infection, some had died suddenly with minimal respiratory involvement. Infection with SARS-CoV-2 was confirmed on ante-mortem or post-mortem testing. Descriptive analysis of the pathological changes and quantitative analyses of the infiltrates and vascular changes were performed. All patients had multifocal vascular damage as determined by leakage of serum proteins into the brain parenchyma. This was accompanied by widespread endothelial cell activation. Platelet aggregates and microthrombi were found adherent to the endothelial cells along vascular lumina. Immune complexes with activation of the classical complement pathway were found on the endothelial cells and platelets. Perivascular infiltrates consisted of predominantly macrophages and some CD8+ T cells. Only rare CD4+ T cells and CD20+ B cells were present. Astrogliosis was also prominent in the perivascular regions. Microglial nodules were predominant in the hindbrain, which were associated with focal neuronal loss and neuronophagia. Antibody-mediated cytotoxicity directed against the endothelial cells is the most likely initiating event that leads to vascular leakage, platelet aggregation, neuroinflammation and neuronal injury. Therapeutic modalities directed against immune complexes should be considered.

Neuropathology of Pediatric SARS-CoV-2 Infection in the Forensic Setting: Novel Application of Ex Vivo Imaging in Analysis of Brain Microvasculature.

Two years into the COVID-19 pandemic, there are few published accounts of postmortem SARS-CoV-2 pathology in children. We report 8 such cases (4 infants aged 7-36 weeks, 4 children aged 5-15 years). Four underwent ex vivo magnetic resonance neuroimaging, to assist in identification of subtle lesions related to vascular compromise. All infants were found unresponsive (3 in unsafe sleeping conditions); all but 1 had recent rhinitis and/or influenza-like illness (ILI) in the family; 1 had history of sickle cell disease. Ex vivo neuroimaging in 1 case revealed white matter (WM) signal hyperintensity and diffuse exaggeration of perivascular spaces, corresponding microscopically to WM mineralization. Neurohistology in the remaining 3 infants variably encompassed WM gliosis and mineralization; brainstem gliosis; perivascular vacuolization; perivascular lymphocytes and brainstem microglia. One had ectopic hippocampal neurons (with pathogenic variant in DEPDC5). Among the children, 3 had underlying conditions (e.g., obesity, metabolic disease, autism) and all presented with ILI. Three had laboratory testing suggesting multisystem inflammatory syndrome (MIS-C). Two were hospitalized for critical care including mechanical ventilation and extracorporeal membrane oxygenation (ECMO); one (co-infected with adenovirus) developed right carotid stroke ipsilateral to the ECMO cannula and the other required surgery for an ingested foreign body. Autopsy findings included: acute lung injury in 3 (1 with microthrombi); and one each with diabetic ketoacidosis and cardiac hypertrophy; coronary and cerebral arteritis and aortitis, resembling Kawasaki disease; and neuronal storage and enlarged fatty liver. All 4 children had subtle meningoencephalitis, focally involving the brainstem. On ex vivo neuroimaging, 1 had focal pontine susceptibility with corresponding perivascular inflammation/expanded perivascular spaces on histopathology. Results suggest SARS-CoV-2 in infants may present as sudden unexpected infant death, while in older children, signs and symptoms point to severe disease. Underlying conditions may predispose to fatal outcomes. As in adults, the neuropathologic changes may be subtle, with vascular changes such as perivascular vacuolization and gliosis alongside sparse perivascular lymphocytes. Detection of subtle vascular pathology is enhanced by ex vivo neuroimaging. Additional analysis of the peripheral/autonomic nervous system and investigation of co-infection in children with COVID-19 is necessary to understand risk for cardiovascular collapse/sudden death.

Benchmarking the centralized urgent plasma exchange service for patients admitted with a diagnosis of suspected acquired thrombotic thrombocytopenic purpura at a large healthcare system.

BACKGROUND: Consensus guidelines recommend that therapeutic plasma exchange (TPE) should be started within 4 to 8 hours after the diagnosis of suspected acquired thrombotic thrombocytopenic purpura (aTTP). This study aimed to audit the steps from diagnosis to initiation of plasma exchange at a centralized apheresis service. METHODS: A retrospective review of the electronic medical record and laboratory information systems from January 1, 2014 to August 31, 2017 was conducted to identify all patients with suspected aTTP undergoing TPE. Demographics, comorbidities, pertinent laboratory tests, and temporal TPE procedural data were collected. RESULTS: The median (5th-95th percentile) time from request to initiation of TPE was 5.4 (3.2-10.6) hours. TPE was initiated within 8 hours in 94 of the 108 patients (87.0%). The median (5th-95th percentile) time from request to central venous access was 2.5 (0.5-6.9) hours and from request to plasma product issuance from the blood bank was 3.4 (1.6-8.1) hours. aTTP patients in whom TPE was initiated greater than 6 hours from request did not have worse outcomes compared to those with TPE initiation within 6 hours: in-hospital mortality (2/14 [14.3%] vs 2/21 [9.5%], P = 0.66), median length of stay (9.0 [4.7-44.1] vs 8.3 [3.9-27.0] days, P = 0.76), and median number of days to durable platelet count recovery (4.5 [2.0-9.0] vs 4.0 [2.0-18.0] days, P = 0.66). CONCLUSIONS: The 4 to 8-hour target window from TPE request to initiation appears feasible for a centralized apheresis program servicing a large healthcare system.

The risk to future pregnancies of transfusing Rh(D)-negative females of childbearing potential with Rh(D)-positive red blood cells during trauma resuscitation is dependent on their age at transfusion.

BACKGROUND: A risk assessment model for predicting the risk of haemolytic disease of the fetus and newborn (HDFN) in future pregnancies following the transfusion of Rh(D)-positive red blood cell (RBC)-containing products to females of childbearing potential (FCP) was developed, accounting for the age that the FCP is transfused in various countries. METHODS: The HDFN risk prediction model included the following inputs: risk of FCP death in trauma, Rh(D) alloimmunization rate following Rh(D)-positive RBC transfusion, expected number of live births following resuscitation, probability of carrying an Rh(D)-positive fetus, the probability of HDFN in an Rh(D)-positive fetus carried by an alloimmunized mother. The model was implemented in Microsoft R Open, and one million FCPs of each age between 18 and 49 years old were simulated. Published data from eight countries, including the United States, were utilized to generate country-specific HDFN risk estimates. RESULTS: The risk predictions showed similar characteristics for each country in that the overall risk of having a pregnancy affected by HDFN was higher if the FCP was younger when she received her Rh(D)-positive transfusion than if she was older. In the United States, the overall risk of HDFN if the FCP was transfused at age 18 was 3·4% (mild: 1·20%, moderate: 0·45%; severe: 1·15%; IUFD: 0·57%); the risk was approximately 0% if the FCP was 43 years or older at the time of transfusion. CONCLUSION: This model can be used to predict HDFN outcomes when establishing transfusion policies as it relates to the administration of Rh(D)-positive products for massively bleeding FCPs.

Introduction to Artificial Intelligence and Machine Learning for Pathology.

CONTEXT.­: Recent developments in machine learning have stimulated intense interest in software that may augment or replace human experts. Machine learning may impact pathology practice by offering new capabilities in analysis, interpretation, and outcomes prediction using images and other data. The principles of operation and management of machine learning systems are unfamiliar to pathologists, who anticipate a need for additional education to be effective as expert users and managers of the new tools. OBJECTIVE.­: To provide a background on machine learning for practicing pathologists, including an overview of algorithms, model development, and performance evaluation; to examine the current status of machine learning in pathology and consider possible roles and requirements for pathologists in local deployment and management of machine learning systems; and to highlight existing challenges and gaps in deployment methodology and regulation. DATA SOURCES.­: Sources include the biomedical and engineering literature, white papers from professional organizations, government reports, electronic resources, and authors' experience in machine learning. References were chosen when possible for accessibility to practicing pathologists without specialized training in mathematics, statistics, or software development. CONCLUSIONS.­: Machine learning offers an array of techniques that in recent published results show substantial promise. Data suggest that human experts working with machine learning tools outperform humans or machines separately, but the optimal form for this combination in pathology has not been established. Significant questions related to the generalizability of machine learning systems, local site verification, and performance monitoring remain to be resolved before a consensus on best practices and a regulatory environment can be established.

In silico model of the dilutional effects of conventional component therapy versus whole blood in the management of massively bleeding adult trauma patients.

BACKGROUND: There are multiple approaches to the blood product and fluid resuscitation of a bleeding trauma patient. An in silico model of different trauma resuscitation strategies was constructed to predict their effects on the volumes of the different body fluid compartments and on several important hemostatic factors. STUDY DESIGN AND METHODS: This multicompartment dynamic deterministic model comprised four interconnected modules (hemostatic, resuscitation, body fluid compartment, and dilutional coagulopathy). The model was divided into five resuscitation phases with simulations using six different resuscitation strategies: whole blood (WB) only, conventional component therapy (CCT) only or 10 units of WB followed by CCT, with either 1 L of crystalloid or 1.5 units of WB or red blood cells in the prehospital phase. RESULTS: At the end of the simulations using 1 L of crystalloid fluids in the prehospital resuscitation phase, the use of WB led to a 1.4 g/dL higher hemoglobin concentration, 32 mg/dL higher fibrinogen concentration, and 0.9 L lower total extracellular fluid volume compared to CCT. Prehospital blood product transfusion in place of crystalloid resulted in higher hemoglobin and fibrinogen concentrations and a lower international normalized ratio throughout the resuscitation regardless of the resuscitation strategy used. Throughout both the prehospital crystalloid and prehospital blood product transfusion simulations, the hemoglobin and fibrinogen concentrations and platelet counts were higher, and the international normalized ratio was lower, when WB was used compared to CCT. CONCLUSIONS: This model predicted improved hemostatic factor levels and a smaller total extracellular fluid volume volume when WB was transfused instead of CCT to bleeding trauma patients.

Application of a recursive partitioning decision tree algorithm for the prediction of massive transfusion in civilian trauma: the MTPitt prediction tool.

BACKGROUND: A supervised machine learning algorithm was used to generate decision trees for the prediction of massive transfusion at a Level 1 trauma center. METHODS: Trauma patients who received at least one unit of RBCs and/or low-titer group O whole blood between January 1, 2015, and December 31, 2017, were included. Massive transfusion was defined as the transfusion of 10 or more units of RBCs and/or low-titer group O whole blood in the first 24 hours of admission. A recursive partitioning algorithm was used to generate two decision trees for prediction of massive transfusion using a training data set (n = 550): the first, MTPitt, was based on demographic and clinical parameters, and the second, MTPitt+Labs, also included laboratory data. Decision tree performance was compared with the Assessment of Blood Consumption score and the Trauma Associated Severe Hemorrhage score. RESULTS: The incidence of massive transfusion in the validation data set (n = 199) was 7.5%. The MTPitt decision tree had a higher balanced accuracy (81.4%) and sensitivity (86.7%) compared to an Assessment of Blood Consumption Score of 2 or higher (77.9% and 66.7%, respectively) and a Trauma Associated Severe Hemorrhage score of 9 or higher (75.0% and 73.3%, respectively), although the 95% confidence intervals overlapped. Addition of laboratory data to the MTPitt decision tree (MTPitt+Labs) resulted in a higher specificity and balanced accuracy compared to MTPitt without an increase in sensitivity. CONCLUSIONS: The MTPitt decisions trees are highly sensitive tools for identifying patients who received a massive transfusion and do not require computational resources to be implemented in the trauma setting.

Herpes Simplex Virus-1 qPCR in the Diagnosis of Lower Respiratory Tract Infections in Organ Transplant Recipients and Critically Ill Patients.

OBJECTIVES: To determine a quantitative herpes simplex virus (HSV) DNA threshold in lower respiratory tract specimens that correlates with positive viral culture and clinical outcomes. METHODS: Bronchoalveolar lavage and bronchial wash samples from 53 HSV culture-positive and 61 culture-negative matched controls were tested using HSV-1 and HSV-2 quantitative polymerase chain reaction (qPCR). RESULTS: Median viral culture turnaround time was 21.8 days and 9.9 days for culture-negative and culture-positive specimens, respectively. Using an HSV-1 viral load threshold of 1.62 × 103 copies/mL, there was 93% agreement with viral culture. An HSV-1 viral load ≥1.3 × 104 copies/mL was associated with worse clinical outcome compared to a viral load <1.3 × 104 copies/mL (hazard ratio [HR] = 4.27, P = .017), and there was a trend of worse outcome compared to patients with undetectable HSV-1 DNA (HR = 1.60, P = .056). CONCLUSIONS: qPCR has clinical utility for rapid accurate identification of HSV-1 in lower respiratory tract specimens.