Artificial Intelligence in the Clinical Laboratory: An Overview with Frequently Asked Questions.

This article provides an overview of machine learning fundamentals and some applications of machine learning to clinical laboratory diagnostics and patient management. A key goal of this article is to provide a basic foundation in clinical machine learning for readers with clinical laboratory experience that will set them up for more in-depth study of the topic and/or to become a better collaborator with computational colleagues in the development and deployment of machine learning-based solutions.

Real evidence to assess clinical testing interference risk (REACTIR): A strategy using real world data to assess the prevalence of interfering substances in patients undergoing clinical laboratory testing.

INTRODUCTION: Laboratory test interferences can cause spurious test results and patient harm. Knowing the frequency of various interfering substances in patient populations likely to be tested with a particular laboratory assay may inform test development, test utilization and strategies to mitigate interference risk. METHODS: We developed REACTIR (Real Evidence to Assess Clinical Testing Interference Risk), an approach using real world data to assess the prevalence of various interfering substances in patients tested with a particular type of assay. REACTIR uses administrative real world data to identify and subgroup patient cohorts tested with a particular laboratory test and evaluate interference risk. RESULTS: We demonstrate the application REACTIR to point of care (POC) blood glucose testing. We found that exposure to several substances with the potential to interfere in POC blood glucose tests, including N-acetyl cysteine (NAC) and high dose vitamin C was uncommon in most patients undergoing POC glucose tests with several key exceptions, such as burn patients receiving high dose IV-vitamin C or acetaminophen overdose patients receiving NAC. CONCLUSIONS: Findings from REACTIR may support risk mitigation strategies including targeted clinician education and clinical decision support. Likewise, adaptations of REACTIR to premarket assay development may inform optimal assay design and assessment.

Use of machine learning to predict clinical decision support compliance, reduce alert burden, and evaluate duplicate laboratory test ordering alerts.

OBJECTIVES: While well-designed clinical decision support (CDS) alerts can improve patient care, utilization management, and population health, excessive alerting may be counterproductive, leading to clinician burden and alert fatigue. We sought to develop machine learning models to predict whether a clinician will accept the advice provided by a CDS alert. Such models could reduce alert burden by targeting CDS alerts to specific cases where they are most likely to be effective. MATERIALS AND METHODS: We focused on a set of laboratory test ordering alerts, deployed at 8 hospitals within the Partners Healthcare System. The alerts notified clinicians of duplicate laboratory test orders and advised discontinuation. We captured key attributes surrounding 60 399 alert firings, including clinician and patient variables, and whether the clinician complied with the alert. Using these data, we developed logistic regression models to predict alert compliance. RESULTS: We identified key factors that predicted alert compliance; for example, clinicians were less likely to comply with duplicate test alerts triggered in patients with a prior abnormal result for the test or in the context of a nonvisit-based encounter (eg, phone call). Likewise, differences in practice patterns between clinicians appeared to impact alert compliance. Our best-performing predictive model achieved an area under the receiver operating characteristic curve (AUC) of 0.82. Incorporating this model into the alerting logic could have averted more than 1900 alerts at a cost of fewer than 200 additional duplicate tests. CONCLUSIONS: Deploying predictive models to target CDS alerts may substantially reduce clinician alert burden while maintaining most or all the CDS benefit.

Development of a "meta-model" to address missing data, predict patient-specific cancer survival and provide a foundation for clinical decision support.

OBJECTIVE: Like most real-world data, electronic health record (EHR)-derived data from oncology patients typically exhibits wide interpatient variability in terms of available data elements. This interpatient variability leads to missing data and can present critical challenges in developing and implementing predictive models to underlie clinical decision support for patient-specific oncology care. Here, we sought to develop a novel ensemble approach to addressing missing data that we term the "meta-model" and apply the meta-model to patient-specific cancer prognosis. MATERIALS AND METHODS: Using real-world data, we developed a suite of individual random survival forest models to predict survival in patients with advanced lung cancer, colorectal cancer, and breast cancer. Individual models varied by the predictor data used. We combined models for each cancer type into a meta-model that predicted survival for each patient using a weighted mean of the individual models for which the patient had all requisite predictors. RESULTS: The meta-model significantly outperformed many of the individual models and performed similarly to the best performing individual models. Comparisons of the meta-model to a more traditional imputation-based method of addressing missing data supported the meta-model's utility. CONCLUSIONS: We developed a novel machine learning-based strategy to underlie clinical decision support and predict survival in cancer patients, despite missing data. The meta-model may more generally provide a tool for addressing missing data across a variety of clinical prediction problems. Moreover, the meta-model may address other challenges in clinical predictive modeling including model extensibility and integration of predictive algorithms trained across different institutions and datasets.

Iron studies and transferrin, a source of test ordering confusion highly amenable to clinical decision support.

INTRODUCTION: An important cause of laboratory test misordering and overutilization is clinician confusion between tests with similar sounding names or similar indications. We identified an area of test ordering confusion with iron studies that involves total iron binding capacity (TIBC), transferrin, and transferrin saturation. We observed concurrent ordering of direct transferrin along with TIBC at many hospitals within our health system and suspected this was unnecessary. METHODS: We extracted patient test results for transferrin, TIBC and other biomarkers. Using these data, we evaluated both patterns of test utilization and test result concordance. We implemented a clinical decision support (CDS) alert to discourage unnecessary orders for direct transferrin. RESULTS: Using linear regression, we were able to predict transferrin from either TIBC alone or TIBC with other analytes with a high degree of accuracy, demonstrating that in most cases, direct transferrin in combination with TIBC provides little if any additional diagnostic information beyond TIBC alone. The CDS alert proved highly effective in reducing transferrin test utilization at four different hospitals. CONCLUSIONS: Concurrent ordering of direct transferrin and TIBC should usually be avoided. Removal of transferrin or TIBC from the test menu or implementation of CDS may improve utilization of these tests.

Machine Learning Models Improve the Diagnostic Yield of Peripheral Blood Flow Cytometry.

OBJECTIVES: Peripheral blood flow cytometry (PBFC) is useful for evaluating circulating hematologic malignancies (HM) but has limited diagnostic value for screening. We used machine learning to evaluate whether clinical history and CBC/differential parameters could improve PBFC utilization. METHODS: PBFC cases with concurrent/recent CBC/differential were split into training (n = 626) and test (n = 159) cohorts. We classified PBFC results with abnormal blast/lymphoid populations as positive and used two models to predict results. RESULTS: Positive PBFC results were seen in 58% and 21% of training cases with and without prior HM (P < .001). % neutrophils, absolute lymphocyte count, and % blasts/other cells differed significantly between positive and negative PBFC groups (areas under the curve [AUC] > 0.7). Among test cases, a decision tree model achieved 98% sensitivity and 65% specificity (AUC = 0.906). A logistic regression model achieved 100% sensitivity and 54% specificity (AUC = 0.919). CONCLUSIONS: We outline machine learning-based triaging strategies to decrease unnecessary utilization of PBFC by 35% to 40%.

Validation and Implementation of an Ordering Alert to Improve the Efficiency of Monoclonal Gammopathy Evaluation.

OBJECTIVES: To evaluate the use of a provider ordering alert to improve laboratory efficiency and reduce costs. METHODS: We conducted a retrospective study to assess the use of an institutional reflex panel for monoclonal gammopathy evaluation. We then created a clinical decision support (CDS) alert to educate and encourage providers to change their less-efficient orders to the reflex panel. RESULTS: Our retrospective analysis demonstrated that an institutional reflex panel could be safely substituted for a less-efficient and higher-cost panel. The implemented CDS alert resulted in 79% of providers changing their high-cost order panel to an order panel based on the reflex algorithm. CONCLUSIONS: The validated decision support alert demonstrated high levels of provider acceptance and directly led to operational and cost savings within the laboratory. Furthermore, these studies highlight the value of laboratory involvement with CDS efforts to provide agile and targeted provider ordering assistance.

Machine Learning and Other Emerging Decision Support Tools.

Emerging applications of machine learning and artificial intelligence offer the opportunity to discover new clinical knowledge through secondary exploration of existing patient medical records. This new knowledge may in turn offer a foundation to build new types of clinical decision support (CDS) that provide patient-specific insights and guidance across a wide range of clinical questions and settings. This article will provide an overview of these emerging approaches to CDS, discussing both existing technologies as well as challenges that health systems and informaticists will need to address to allow these emerging approaches to reach their full potential.

Order Indication Solicitation to Assess Clinical Laboratory Test Utilization: D-Dimer Order Patterns as an Illustrative Case.

BACKGROUND: A common challenge in the development of laboratory clinical decision support (CDS) and laboratory utilization management (UM) initiatives stems from the fact that many laboratory tests have multiple potential indications, limiting the ability to develop context-specific alerts. As a potential solution, we designed a CDS alert that asks the ordering clinician to provide the indication for testing, using D-dimer as an exemplar. Using data collected over a nearly 3-year period, we sought to determine whether the indication capture was a useful feature within the CDS alert and whether it provided actionable intelligence to guide the development of an UM strategy. METHODS: We extracted results and ordering data for D-dimer testing performed in our laboratory over a 35-month period. We analyzed order patterns by clinical indication, hospital service, and length of hospitalization. RESULTS: Our final data set included 13,971 result-order combinations and indeed provided actionable intelligence regarding test utilization patterns. For example, pulmonary embolism was the most common emergency department indication (86%), while disseminated intravascular coagulation was the most common inpatient indication (56%). D-dimer positivity rates increased with the duration of hospitalization and our data suggested limited utility for ordering this test in the setting of suspected venous thromboembolic disease in admitted patients. In addition, we found that D-dimer was ordered for unexpected indications including the assessment of stroke, dissection, and extracorporeal membrane oxygenation. CONCLUSIONS: Indication capture within a CDS alert and correlation with result data can provide insight into order patterns which can be used to develop future CDS strategies to guide appropriate test use by clinical indication.

Using Machine Learning-Based Multianalyte Delta Checks to Detect Wrong Blood in Tube Errors.

OBJECTIVES: An unfortunate reality of laboratory medicine is that blood specimens collected from one patient occasionally get mislabeled with identifiers from a different patient, resulting in so-called "wrong blood in tube" (WBIT) errors and potential patient harm. Here, we sought to develop a machine learning-based, multianalyte delta check algorithm to detect WBIT errors and mitigate patient harm. METHODS: We simulated WBIT errors within sets of routine inpatient chemistry test results to develop, train, and evaluate five machine learning-based WBIT detection algorithms. RESULTS: The best-performing WBIT detection algorithm we developed was based on a support vector machine and incorporated changes in test results between consecutive collections across 11 analytes. This algorithm achieved an area under the curve of 0.97 and considerably outperformed traditional single-analyte delta checks. CONCLUSIONS: Machine learning-based multianalyte delta checks may offer a practical strategy to identify WBIT errors prior to test reporting and improve patient safety.

Creation and Use of an Electronic Health Record Reporting Database to Improve a Laboratory Test Utilization Program.

OBJECTIVES: Laboratory-based utilization management programs typically rely primarily on data derived from the laboratory information system to analyze testing volumes for trends and utilization concerns. We wished to examine the ability of an electronic health record (EHR) laboratory orders database to improve a laboratory utilization program. METHODS: We obtained a daily file from our EHR containing data related to laboratory test ordering. We then used an automated process to import this file into a database to facilitate self-service queries and analysis. RESULTS: The EHR laboratory orders database has proven to be an important addition to our utilization management program. We provide three representative examples of how the EHR laboratory orders database has been used to address common utilization issues. We demonstrate that analysis of EHR laboratory orders data has been able to provide unique insights that cannot be obtained by review of laboratory information system data alone. Further, we provide recommendations on key EHR data fields of importance to laboratory utilization efforts. CONCLUSION: We demonstrate that an EHR laboratory orders database may be a useful tool in the monitoring and optimization of laboratory testing. We recommend that health care systems develop and maintain a database of EHR laboratory orders data and integrate this data with their laboratory utilization programs.

3D-MICE: integration of cross-sectional and longitudinal imputation for multi-analyte longitudinal clinical data.

Objective: A key challenge in clinical data mining is that most clinical datasets contain missing data. Since many commonly used machine learning algorithms require complete datasets (no missing data), clinical analytic approaches often entail an imputation procedure to "fill in" missing data. However, although most clinical datasets contain a temporal component, most commonly used imputation methods do not adequately accommodate longitudinal time-based data. We sought to develop a new imputation algorithm, 3-dimensional multiple imputation with chained equations (3D-MICE), that can perform accurate imputation of missing clinical time series data. Methods: We extracted clinical laboratory test results for 13 commonly measured analytes (clinical laboratory tests). We imputed missing test results for the 13 analytes using 3 imputation methods: multiple imputation with chained equations (MICE), Gaussian process (GP), and 3D-MICE. 3D-MICE utilizes both MICE and GP imputation to integrate cross-sectional and longitudinal information. To evaluate imputation method performance, we randomly masked selected test results and imputed these masked results alongside results missing from our original data. We compared predicted results to measured results for masked data points. Results: 3D-MICE performed significantly better than MICE and GP-based imputation in a composite of all 13 analytes, predicting missing results with a normalized root-mean-square error of 0.342, compared to 0.373 for MICE alone and 0.358 for GP alone. Conclusions: 3D-MICE offers a novel and practical approach to imputing clinical laboratory time series data. 3D-MICE may provide an additional tool for use as a foundation in clinical predictive analytics and intelligent clinical decision support.

Analysis of Daily Laboratory Orders at a Large Urban Academic Center: A Multifaceted Approach to Changing Test Ordering Patterns.

OBJECTIVES: We sought to address concerns regarding recurring inpatient laboratory test order practices (daily laboratory tests) through a multifaceted approach to changing ordering patterns. METHODS: We engaged in an interdepartmental collaboration to foster mindful test ordering through clinical policy creation, electronic clinical decision support, and continuous auditing and feedback. RESULTS: Annualized daily order volumes decreased from approximately 25,000 to 10,000 during a 33-month postintervention review. This represented a significant change from preintervention order volumes (95% confidence interval, 0.61-0.64; P < 10-16). Total inpatient test volumes were not affected. CONCLUSIONS: Durable changes to inpatient order practices can be achieved through a collaborative approach to utilization management that includes shared responsibility for establishing clinical guidelines and electronic decision support. Our experience suggests auditing and continued feedback are additional crucial components to changing ordering behavior. Curtailing daily orders alone may not be a sufficient strategy to reduce in-laboratory costs.

Utilization Management of High-Cost Imaging in an Outpatient Setting in a Large Stable Patient and Provider Cohort over 7 Years.

Purpose To quantify the effect of a comprehensive, long-term, provider-led utilization management (UM) program on high-cost imaging (computed tomography, magnetic resonance imaging, nuclear imaging, and positron emission tomography) performed on an outpatient basis. Materials and Methods This retrospective, 7-year cohort study included all patients regularly seen by primary care physicians (PCPs) at an urban academic medical center. The main outcome was the number of outpatient high-cost imaging examinations per patient per year ordered by the patient's PCP or by any specialist. The authors determined the probability of a patient undergoing any high-cost imaging procedure during a study year and the number of examinations per patient per year (intensity) in patients who underwent high-cost imaging. Risk-adjusted hierarchical models were used to directly quantify the physician component of variation in probability and intensity of high-cost imaging use, and clinicians were provided with regular comparative feedback on the basis of the results. Observed trends in high-cost imaging use and provider variation were compared with the same measures for outpatient laboratory studies because laboratory use was not subject to UM during this period. Finally, per-member per-year high-cost imaging use data were compared with statewide high-cost imaging use data from a major private payer on the basis of the same claim set. Results The patient cohort steadily increased in size from 88 959 in 2007 to 109 823 in 2013. Overall high-cost imaging utilization went from 0.43 examinations per year in 2007 to 0.34 examinations per year in 2013, a decrease of 21.33% (P < .0001). At the same time, similarly adjusted routine laboratory study utilization decreased by less than half that rate (9.4%, P < .0001). On the basis of unadjusted data, outpatient high-cost imaging utilization in this cohort decreased 28%, compared with a 20% decrease in statewide utilization (P = .0023). Conclusion Analysis of high-cost imaging utilization in a stable cohort of patients cared for by PCPs during a 7-year period showed that comprehensive UM can produce a significant and sustained reduction in risk-adjusted per-patient year outpatient high-cost imaging volume. © RSNA, 2017.

Health Care Infrastructure for Financially Sustainable Clinical Genomics.

Next-generation sequencing has evolved technically and economically into the method of choice for interrogating the genome in cancer and inherited disorders. The introduction of procedural code sets for whole-exome and genome sequencing is a milestone toward financially sustainable clinical implementation; however, achieving reimbursement is currently a major challenge. As part of a prospective quality-improvement initiative to implement the new code sets, we adopted Agile, a development methodology originally devised in software development. We implemented eight functionally distinct modules (request review, cost estimation, preauthorization, accessioning, prebilling, testing, reporting, and reimbursement consultation) and obtained feedback via an anonymous survey. We managed 50 clinical requests (January to June 2015). The fraction of pursued-to-requested cases (n = 15/50; utilization management fraction, 0.3) aimed for a high rate of preauthorizations. In 13 of 15 patients the insurance plan required preauthorization, which we obtained in 70% and ultimately achieved reimbursement in 50%. Interoperability enabled assessment of 12 different combinations of modules that underline the importance of an adaptive workflow and policy tailoring to achieve higher yields of reimbursement. The survey confirmed a positive attitude toward self-organizing teams. We acknowledge the individuals and their interactions and termed the infrastructure: human pipeline. Nontechnical barriers currently are limiting the scope and availability of clinical genomic sequencing. The presented human pipeline is one approach toward long-term financial sustainability of clinical genomics.