Validating, Implementing, and Monitoring Machine Learning Solutions in the Clinical Laboratory Safely and Effectively.

BACKGROUND: Machine learning solutions offer tremendous promise for improving clinical and laboratory operations in pathology. Proof-of-concept descriptions of these approaches have become commonplace in laboratory medicine literature, but only a scant few of these have been implemented within clinical laboratories, owing to the often substantial barriers in validating, implementing, and monitoring these applications in practice. This mini-review aims to highlight the key considerations in each of these steps. CONTENT: Effective and responsible applications of machine learning in clinical laboratories require robust validation prior to implementation. A comprehensive validation study involves a critical evaluation of study design, data engineering and interoperability, target label definition, metric selection, generalizability and applicability assessment, algorithmic fairness, and explainability. While the main text highlights these concepts in broad strokes, a supplementary code walk-through is also provided to facilitate a more practical understanding of these topics using a real-world classification task example, the detection of saline-contaminated chemistry panels.Following validation, the laboratorian's role is far from over. Implementing machine learning solutions requires an interdisciplinary effort across several roles in an organization. We highlight the key roles, responsibilities, and terminologies for successfully deploying a validated solution into a live production environment. Finally, the implemented solution must be routinely monitored for signs of performance degradation and updated if necessary. SUMMARY: This mini-review aims to bridge the gap between theory and practice by highlighting key concepts in validation, implementation, and monitoring machine learning solutions effectively and responsibly in the clinical laboratory.

How do experts determine where to intervene on test ordering? An interview study.

OBJECTIVES: Lab testing is a high-volume activity that is often overused, leading to wasted resources and inappropriate care. Improving test ordering practices in tertiary care involves deciding where to focus scarce intervention resources, but clear guidance on how to optimize these resources is lacking. We aimed to explore context-sensitive factors and processes that inform individual decisions about laboratory stewardship interventions by speaking to key interest holders in this area. METHODS: We conducted semi-structured interviews with test-ordering intervention development experts and authors of test-ordering guidance documents to explore five broad topics: 1) processes used to prioritize tests for intervention; 2) factors considered when deciding which tests to target; 3) measurement of these factors; 4) interventions selected; 5) suggestions for a framework to support these decisions. Transcripts were double coded using directed-content and thematic analysis. RESULTS: We interviewed 14 intervention development experts. Experts noted they frequently consider test volume, test value, and patient care when deciding on a test to target. Experts indicated that quantifying many relevant factors was challenging. Processes to support these decisions often involved examining local data, obtaining buy-in, and relying on an existing guideline. Suggestions for building a framework emphasized the importance of collaboration, consideration of context and resources, and starting with "easy wins" to gain support and experience. CONCLUSIONS: Our study provides insight into the factors and processes experts consider when deciding which tests to target for intervention and can inform the development of a framework to guide the selection of tests for intervention and guideline development.

Short Timeframe Prediction of Kidney Failure among Patients with Advanced Chronic Kidney Disease.

BACKGROUND: Development of a short timeframe (6-12 months) kidney failure risk prediction model may serve to improve transitions from advanced chronic kidney disease (CKD) to kidney failure and reduce rates of unplanned dialysis. The optimal model for short timeframe kidney failure risk prediction remains unknown. METHODS: This retrospective study included 1757 consecutive patients with advanced CKD (mean age 66 years, estimated glomerular filtration rate 18 mL/min/1.73 m2). We compared the performance of Cox regression models using (a) baseline variables alone, (b) time-varying variables and machine learning models, (c) random survival forest, (d) random forest classifier in the prediction of kidney failure over 6/12/24 months. Performance metrics included area under the receiver operating characteristic curve (AUC-ROC) and maximum precision at 70% recall (PrRe70). Top-performing models were applied to 2 independent external cohorts. RESULTS: Compared to the baseline Cox model, the machine learning and time-varying Cox models demonstrated higher 6-month performance [Cox baseline: AUC-ROC 0.85 (95% CI 0.84-0.86), PrRe70 0.53 (95% CI 0.51-0.55); Cox time-varying: AUC-ROC 0.88 (95% CI 0.87-0.89), PrRe70 0.62 (95% CI 0.60-0.64); random survival forest: AUC-ROC 0.87 (95% CI 0.86-0.88), PrRe70 0.61 (95% CI 0.57-0.64); random forest classifier AUC-ROC 0.88 (95% CI 0.87-0.89), PrRe70 0.62 (95% CI 0.59-0.65)]. These trends persisted, but were less pronounced, at 12 months. The random forest classifier was the highest performing model at 6 and 12 months. At 24 months, all models performed similarly. Model performance did not significantly degrade upon external validation. CONCLUSIONS: When predicting kidney failure over short timeframes among patients with advanced CKD, machine learning incorporating time-updated data provides enhanced performance compared with traditional Cox models.

Choosing which in-hospital laboratory tests to target for intervention: a scoping review.

INTRODUCTION: Some laboratory testing practices may be of low value, leading to wasted resources and potential patient harm. Our scoping review investigated factors and processes that developers report using to inform decisions about what tests to target for practice improvement. METHODS: We searched Medline on May 30th, 2019 and June 28th, 2021 and included guidelines, recommendation statements, or empirical studies related to test ordering practices. Studies were included if they were conducted in a tertiary care setting, reported making a choice about a specific test requiring intervention, and reported at least one factor informing that choice. We extracted descriptive details, tests chosen, processes used to make the choice, and factors guiding test choice. RESULTS: From 114 eligible studies, we identified 30 factors related to test choice including clinical value, cost, prevalence of test, quality of test, and actionability of test results. We identified nine different processes used to inform decisions regarding where to spend intervention resources. CONCLUSIONS: Intervention developers face difficult choices when deciding where to put scarce resources intended to improve test utilization. Factors and processes identified here can be used to inform a framework to help intervention developers make choices relevant to improving testing practices.

Rh(D) immune globulin administration in pregnancy: Retrospective audit of patient safety events followed by targeted educational intervention with Bayesian analysis.

OBJECTIVES: To examine local patient safety events related to the administration of anti-Rh(D) immune globin (RhIG) during pregnancy, and to follow-up with targeted educational intervention to improve knowledge of this process. BACKGROUND: Administration RhIG is established treatment for the prevention of haemolytic disease of the foetus and newborn (HDFN). However, patient safety events in relation to its correct use continue to occur. METHODS: A retrospective audit of patient safety events related to RhIG administration during pregnancy was performed. Targeted educational intervention in the form of PowerPoint® presentation were given to nursing staff, laboratory staff and physicians and evaluated with pre- and post-tests using multiple-choice questions given immediately before and after the presentation. RESULTS: An annual incidence of 0.24% of patient safety events related to the administration of RhIG during pregnancy was found. These events were mostly in the preanalytical phase, for example mislabelled samples or samples for D-rosette/Kleihauer-Betke testing drawn from the baby, not the mother. Using Bayesian analysis, the probability of positive effect for the targeted educational intervention was 100% with a median improved score of 29%. This was compared with a control group using standard curriculum education intervention based on the current curriculum for nursing, laboratory and medical students which showed a median improved score of only 4.4%. CONCLUSIONS: Administration of RhIG during pregnancy is a multistep process involving health care professionals of several disciplines providing opportunities to enhance the curriculum for nursing, laboratory and medical students and to ensure on-going education.

Transformation of Sequential Hospital and Outpatient Laboratory Data into Between-Day Reference Change Values.

BACKGROUND: Serial differences between intrapatient consecutive measurements can be transformed into Taylor series of variation vs time with the intersection at time = 0 (y0) equal to the total variation (analytical + biological + preanalytical). With small preanalytical variation, y0, expressed as a percentage of the mean, is equal to the variable component of the reference change value (RCV) calculation: (CVA2 + CVI2)1/2. METHODS: We determined the between-day RCV of patient data for 17 analytes and compared them to healthy participants' RCVs. We analyzed 653 consecutive days of Dartmouth-Hitchcock Roche Modular general chemistry data (4.2 million results: 60% inpatient, 40% outpatient). The serial patient values of 17 analytes were transformed into 95% 2-sided RCV (RCVAlternate), and 3 sets of RCVhealthy were calculated from 3 Roche Modular analyzers' quality control summaries and CVI derived from biological variation (BV) studies using healthy participants. RESULTS: The RCVAlternate values are similar to RCVhealthy derived from known components of variation. For sodium, chloride, bicarbonate calcium, magnesium, phosphate, alanine aminotransferase, albumin, and total protein, the RCVs are equivalent. As expected, increased variation was found for glucose, aspartate aminotransferase, creatinine, and potassium. Direct bilirubin and urea demonstrated lower variation. CONCLUSIONS: Our RCVAlternate values integrate known and unknown components of analytic, biologic, and preanalytic variation, and depict the variations observed by clinical teams that make medical decisions based on the test values. The RCVAlternate values are similar to the RCVhealthy values derived from known components of variation and suggest further studies to better understand the results being generated on actual patients tested in typical laboratory environments.

Limited Evidence for Use of a Black Race Modifier in eGFR Calculations: A Systematic Review.

BACKGROUND: Commonly used estimated glomerular filtration rate (eGFR) equations include a Black race modifier (BRM) that was incorporated during equation derivation. Race is a social construct, and a poorly characterized variable that is applied inconsistently in clinical settings. The BRM results in higher eGFR for any creatinine concentration, implying fundamental differences in creatinine production or excretion in Black individuals compared to other populations. Equations without inclusion of the BRM have the potential to detect kidney disease earlier in patients at the greatest risk of chronic kidney disease (CKD), but also has the potential to over-diagnose CKD or impact downstream clinical interventions. The purpose of this study was to use an evidence-based approach to systematically evaluate the literature relevant to the performance of the eGFR equations with and without the BRM and to examine the clinical impact of the use or removal. CONTENT: PubMed and Embase databases were searched for studies comparing measured GFR to eGFR in racially diverse adult populations using the Modification of Diet in Renal Disease or the 2009-Chronic Kidney Disease Epidemiology Collaboration-creatinine equations based on standardized creatinine measurements. Additionally, we searched for studies comparing clinical use of eGFR calculated with and without the BRM. Here, 8632 unique publications were identified; an additional 3 studies were added post hoc. In total, 96 studies were subjected to further analysis and 44 studies were used to make a final assessment. SUMMARY: There is limited published evidence to support the use of a BRM in eGFR equations.

Performance of the 2021 Race-Free CKD-EPI Creatinine- and Cystatin C-Based Estimated GFR Equations Among Kidney Transplant Recipients.

RATIONALE & OBJECTIVE: Race-free estimated glomerular filtration rate (eGFR) equations incorporating creatinine with and without cystatin C were recently developed and recommended for routine use. However, the performance of these equations among kidney transplant recipients (KTRs) remains unknown. STUDY DESIGN: Cross-sectional study to validate the 2021 race-free Chronic Kidney Disease (CKD) Epidemiology Collaboration (CKD-EPI) eGFR equation based on creatinine alone (eGFRcr) or based on creatinine and cystatin C (eGFRcr-cys) among KTRs. SETTING & PARTICIPANTS: KTRs in stable condition (N = 415) from Canada and New Zealand with same-day measurements of creatinine, cystatin C, and glomerular filtration rate (GFR) using radiolabeled diethylenetriaminepentaacetic acid. TESTS COMPARED: The 2009 CKD-EPI eGFRcr, 2021 CKD-EPI eGFRcr, 2012 CKD-EPI eGFRcr-cys, 2021 CKD-EPI eGFRcr-cys, 2012 CKD-EPI eGFRcys, and Modification of Diet in Renal Disease (MDRD) Study eGFR equations were compared with measured GFR. OUTCOMES: Bias, precision, accuracy, and correct classification by CKD stage. Bias was defined as the difference between estimated and measured GFR. Precision was represented by the interquartile range. Accuracy was defined as the percentages of participants with eGFRs within 10%/20%/30% (P10/P20/P30) of measured GFR, root mean square error, and mean absolute error. RESULTS: 87% of patients studied were White, 3% Black, and 10% other races. Mean measured GFR was 53 ± 19 (SD) mL/min/1.73 m2. The 2009 and 2021 CKD-EPI eGFRcr equations demonstrated similar median bias (-2.3 vs -0.2 mL/min/1.73 m2, respectively), precision (14.5 vs 14.9 mL/min/1.73 m2), and accuracy (P10/P20/P30, 32%/65%/84% vs 33%/63%/84%). The 2012 and 2021 CKD-EPI eGFRcr-cys equations also demonstrated similar median bias (-3.6 vs 0.3 mL/min/1.73 m2, respectively), precision (13.3 vs 14.3 mL/min/1.73 m2), and accuracy (P10/P20/P30, 32%/63%/80% vs 32%/67%/83%). No clear difference in performance was detected between the 2021 CKD-EPI eGFRcr and eGFRcr-cys equations among KTRs. The proportion of correct classification by CKD stage was similar across all eGFR equations. LIMITATIONS: Moderate sample size, few patients had a GFR <30 mL/min/1.73 m2, and the large majority of patients were White. CONCLUSIONS: Among KTRs, the 2021 race-free CKD-EPI eGFR equations perform similarly to the previous CKD-EPI equations that included race correction terms. No significant difference in performance was observed between the 2021 CKD-EPI eGFRcr and eGFRcr-cys equations in the kidney transplant population.

Evaluation of the protein gap for detection of abnormal serum gammaglobulin level: an imperfect predictor.

OBJECTIVES: The value of the serum protein gap (PG, difference between total protein and albumin) in the detection of hyper- or hypogammaglobulinemia is not well established. We assessed the performance of PG for the detection of hyper- or hypogammaglobulinemia in a large sample of patients. METHODS: We reviewed all paired measurements of serum total protein, albumin, quantitative immunoglobulins, and serum protein electrophoresis tested between March 2014 and June 2017 at the Eastern Ontario Regional Laboratory Association. Sensitivity, specificity, positive predictive value, negative predictive value and likelihood ratios of PG at thresholds between 18 and 44 g/L for the detection of hyper- and hypogammaglobulinemia were assessed. RESULTS: There were 19,575 and 5,426 simultaneous paired data points to assess hyper- and hypogammaglobulinemia identified by serum protein electrophoresis (SPE) and nephelometry, respectively. The mean PG was 36.3 g/L (SD 8.6). The prevalence of hypergammaglobulinemia (>16 g/L by SPE) and hypogammaglobulinemia (IgG <7 g/L) was 21.9 and 5.5%, respectively. High PG (≥38 g/L) had sensitivity and specificity of 76.2 and 71.5% respectively for hypergammaglobulinemia. PG ≥38 g/L had a negative predictive value (NPV) of 93.1% for monoclonal, and 96.9% for polyclonal gammopathy. A PG threshold of ≤18 g/L had of sensitivity of 0.4%, specificity of 100%, PPV of 100% and NPV of 80.1% to detect hypogammaglobulinemia (IgG <7 g/L). CONCLUSIONS: High and low PG values were not sensitive in detecting hyper- or hypogammaglobulinemia, although negative predictive values were high for both. Performance of PG should be further evaluated prospectively in specific populations at risk of for abnormal IgG levels.

An international multi-center serum protein electrophoresis accuracy and M-protein isotyping study. Part II: limit of detection and follow-up of patients with small M-proteins.

Background Electrophoretic methods to detect, characterize and quantify M-proteins play an important role in the management of patients with monoclonal gammopathies (MGs). Significant uncertainty in the quantification and limit of detection (LOD) is documented when M-proteins are <10 g/L. Using spiked sera, we aimed to assess the variability in intact M-protein quantification and LOD across 16 laboratories. Methods Sera with normal, hypo- or hyper-gammaglobulinemia were spiked with daratumumab or elotuzumab, with concentrations from 0.125 to 10 g/L (n = 62) along with a beta-migrating sample (n = 9). Laboratories blindly analyzed samples according to their serum protein electrophoresis (SPEP)/isotyping standard operating procedures. LOD and intra-laboratory percent coefficient of variation (%CV) were calculated and further specified with regard to the method (gel/capillary electrophoresis [CZE]), gating strategy (perpendicular drop [PD]/tangent skimming [TS]), isotyping (immunofixation/immunosubtraction [ISUB]) and manufacturer (Helena/Sebia). Results All M-proteins ≥1 g/L were detected by SPEP. With isotyping the LOD was moderately more sensitive than with SPEP. The intensity of polyclonal background had the biggest negative impact on LOD. Independent of the method used, the intra-laboratory imprecision of M-protein quantification was small (mean CV = 5.0%). Low M-protein concentration and high polyclonal background had the strongest negative impact on intra-laboratory precision. All laboratories were able to follow trend of M-protein concentrations down to 1 g/L. Conclusions In this study, we describe a large variation in the reported LOD for both SPEP and isotyping; overall LOD is most affected by the polyclonal immunoglobulin background. Satisfactory intra-laboratory precision was demonstrated. This indicates that the quantification of small M-proteins to monitor patients over time is appropriate, when subsequent testing is performed within the same laboratory.

An international multi-center serum protein electrophoresis accuracy and M-protein isotyping study. Part I: factors impacting limit of quantitation of serum protein electrophoresis.

Background Serum protein electrophoresis (SPEP) is used to quantify the serum monoclonal component or M-protein, for diagnosis and monitoring of monoclonal gammopathies. Significant imprecision and inaccuracy pose challenges in reporting small M-proteins. Using therapeutic monoclonal antibody-spiked sera and a pooled beta-migrating M-protein, we aimed to assess SPEP limitations and variability across 16 laboratories in three continents. Methods Sera with normal, hypo- or hypergammaglobulinemia were spiked with daratumumab, Dara (cathodal migrating), or elotuzumab, Elo (central-gamma migrating), with concentrations from 0.125 to 10 g/L (n = 62) along with a beta-migrating sample (n = 9). Provided with total protein (reverse biuret, Siemens), laboratories blindly analyzed samples according to their SPEP and immunofixation (IFE) or immunosubtraction (ISUB) standard operating procedures. Sixteen laboratories reported the perpendicular drop (PD) method of gating the M-protein, while 10 used tangent skimming (TS). A mean percent recovery range of 80%-120% was set as acceptable. The inter-laboratory %CV was calculated. Results Gamma globulin background, migration pattern and concentration all affect the precision and accuracy of quantifying M-proteins by SPEP. As the background increases, imprecision increases and accuracy decreases leading to overestimation of M-protein quantitation especially evident in hypergamma samples, and more prominent with PD. Cathodal migrating M-proteins were associated with less imprecision and higher accuracy compared to central-gamma migrating M-proteins, which is attributed to the increased gamma background contribution in M-proteins migrating in the middle of the gamma fraction. There is greater imprecision and loss of accuracy at lower M-protein concentrations. Conclusions This study suggests that quantifying exceedingly low concentrations of M-proteins, although possible, may not yield adequate accuracy and precision between laboratories.

A Survey of Cerebrospinal Fluid Total Protein Upper Limits in Canada: Time for an Update?

BACKGROUND: The antiquated standard reference range of 0.15-0.45 g/L for cerebrospinal fluid total protein (CSF-TP) is well entrenched in medical literature and laboratory operating procedures across the world. METHODS: We conducted a web-based survey with a response rate of 34.9% through the listserv of the Canadian Neurological Sciences Federation. Additional laboratory reference data were collated by telephone interview of hospital laboratory technologists across Canada. RESULTS: A total of 142 site responses were obtained: 64.1% from academic/tertiary hospitals and 35.9% from community hospitals. A strong majority (80.4%) of both types of institutions reported using a CSF-TP upper reference limit of 0.45 g/L or less. As a rule, no age adjustments were implemented in CSF-TP-level interpretation. CONCLUSIONS: Recent well-powered laboratory reference studies have documented CSF-TP upper reference limits that are above 0.6 g/L starting at age 50, with incremental limits partitioned by subsequent decades of age. The conventional 0.45 g/L limit could lead to false positive results. Our survey suggests there is a need to consider a wide adoption of data-driven, rather than historical, reference values.

Cerebrospinal Fluid Total Protein Reference Intervals Derived from 20 Years of Patient Data.

BACKGROUND: Reference intervals are vital for interpretation of laboratory results. Many existing reference intervals for cerebrospinal fluid total protein (CSF-TP) are derived from old literature because of the invasive nature of sampling. The objective of this study was to determine reference intervals for CSF-TP using available patient data. METHODS: Twenty years of hospital database information was mined for previously reported CSF-TP results. Associated demographic, laboratory, and clinical diagnosis (International Classification of Diseases 9/10 codes) details were extracted. CSF-TP results included 3 different analytical platforms: the Siemens Vista 1500, Beckman Lx20, and Roche Hitachi 917. From an initial data set of 19591 samples, the following exclusion criteria were applied: incomplete data, white blood cells (WBCs) >5 × 106/L, red blood cells (RBCs) >50 × 106/L, and glucose <2.5 mmol/L. Patient charts were reviewed in detail to exclude 60 different conditions for which increases in CSF-TP would be expected. A total of 6068 samples were included; 63% of the samples were from females. Continuous reference intervals were determined using quantile regression. Age- and sex-partitioned intervals were established using the quantile regression equation and splitting age-groups into 5-year bins. RESULTS: CSF-TP showed a marked age dependence, and males had significantly higher CSF-TP than females across all ages. CSF-TP results from the 3 different instruments and manufacturers showed small (approximately 0.04 g/L), but statistically significant, differences. CSF-TP showed weak, but again statistically significant, correlation with WBC and RBC but was independent of serum total protein and creatinine. CONCLUSIONS: The age dependence of CSF-TP supports that age-partitioned reference intervals will be more accurate than a single cutoff, particularly in patients with advancing age.

Monitoring multiple myeloma patients treated with daratumumab: teasing out monoclonal antibody interference.

BACKGROUND: Monoclonal antibodies are promising anti-myeloma treatments. As immunoglobulins, monoclonal antibodies have the potential to be identified by serum protein electrophoresis (SPE) and immunofixation electrophoresis (IFE). Therapeutic antibody interference with standard clinical SPE and IFE can confound the use of these tests for response assessment in clinical trials and disease monitoring. METHODS: To discriminate between endogenous myeloma protein and daratumumab, a daratumumab-specific immunofixation electrophoresis reflex assay (DIRA) was developed using a mouse anti-daratumumab antibody. To evaluate whether anti-daratumumab bound to and shifted the migration pattern of daratumumab, it was spiked into daratumumab-containing serum and resolved by IFE/SPE. The presence (DIRA positive) or absence (DIRA negative) of residual M-protein in daratumumab-treated patient samples was evaluated using predetermined assessment criteria. DIRA was evaluated for specificity, limit of sensitivity, and reproducibility. RESULTS: In all of the tested samples, DIRA distinguished between daratumumab and residual M-protein in commercial serum samples spiked with daratumumab and in daratumumab-treated patient samples. The DIRA limit of sensitivity was 0.2 g/L daratumumab, using spiking experiments. Results from DIRA were reproducible over multiple days, operators, and assays. The anti-daratumumab antibody was highly specific for daratumumab and did not shift endogenous M-protein. CONCLUSIONS: As the treatment of myeloma evolves to incorporate novel monoclonal antibodies, additional solutions will be needed for clinical monitoring of patient responses to therapeutic regimens. In the interim, assays such as DIRA can inform clinical outcomes by distinguishing daratumumab from endogenous M-protein by IFE.