A Machine Learning Model for the Routine Detection of "Wrong Blood in Complete Blood Count Tube" Errors.

BACKGROUND: Current laboratory procedures may fail to detect wrong blood in tube (WBIT) errors. Machine learning models have the potential to improve WBIT error detection, as demonstrated by proof-of-concept studies. The models developed so far, however, are not appropriate for routine use because they are unable to handle missing values and have low positive predictive value (PPV). In this study, a machine learning model suitable for routine use was developed. METHODS: A model was trained and a preliminary evaluation performed on a retrospective data set of 135 128 current and previous patient complete blood count (CBC) results. The model was then applied prospectively to routine samples tested in a public hospital laboratory over a period of 22 weeks. Each week, the 5 samples identified by the model as most likely to be WBIT errors underwent further investigation by testing blood group and red cell phenotype. The study assessed the number of WBIT errors that were missed by current procedures but detected by the model, as well as the PPV of the model. RESULTS: The model was applied prospectively to 38 187 CBC results that had passed routine laboratory checks. One hundred and ten samples were identified for further testing and 12 WBIT errors were detected. The PPV of the model was 10.9%. CONCLUSION: A machine learning model suitable for routine use was able to identify WBIT errors missed by the laboratory's current procedures. Machine learning models are valuable for the identification of WBIT errors, and their validation and deployment in clinical laboratories would improve patient safety.

Explainability does not improve biochemistry staff trust in artificial intelligence-based decision support.

BACKGROUND: Explainability, the aspect of artificial intelligence-based decision support (ADS) systems that allows users to understand why predictions are made, offers many potential benefits. One common claim is that explainability increases user trust, yet this has not been established in healthcare contexts. For advanced algorithms such as artificial neural networks, the generation of explanations is not trivial, but requires the use of a second algorithm. The assumption of improved user trust should therefore be investigated to determine if it justifies the additional complexity. METHODS: Biochemistry staff completed a wrong blood in tube (WBIT) error identification task with the help of an ADS system. One-half of the volunteers were provided with both ADS predictions and explanations for those predictions, while the other half received predictions alone. The two groups were compared in terms of their rate of agreement with ADS predictions, as an index of user trust, and WBIT error detection performance. Since the AI model used to generate predictions was known to out-perform laboratory staff, increased trust was expected to improve user performance. RESULTS: Volunteers reviewed 1590 sets of results. The volunteers provided with explanations demonstrated no difference in their rate of agreement with the ADS system compared to volunteers receiving predictions alone (83.3% versus 81.8%, p = 0.46). The two volunteer groups were also equivalent in accuracy, sensitivity and specificity for WBIT error identification (p-values >0.78). CONCLUSIONS: For a WBIT error identification task, there was no evidence to justify the additional complexity of explainability on the grounds of increased user trust.

Machine learning models outperform manual result review for the identification of wrong blood in tube errors in complete blood count results.

INTRODUCTION: Wrong blood in tube (WBIT) errors are a significant patient-safety issue encountered by clinical laboratories. This study assessed the performance of machine learning models for the identification of WBIT errors affecting complete blood count (CBC) results against the benchmark of manual review of results by laboratory staff. METHODS: De-identified current and previous (within seven days) CBC results were used in the computer simulation of WBIT errors. 101 015 sets of samples were used to develop machine learning models using artificial neural network, extreme gradient boosting, support vector machine, random forest, logistic regression, decision trees (one complex and one simple) and k-nearest neighbours algorithms. The performance of these models, and of manual review by laboratory staff, was assessed on a separate data set of 1940 samples. RESULTS: Volunteers manually reviewing results identified WBIT errors with an accuracy of 85.7%, sensitivity of 80.1% and specificity of 92.1%. All machine learning models exceeded human-level performance (p-values for all metrics were <.001). The artificial neural network model was the most accurate (99.1%), and the simple decision tree was the least accurate (96.8%). Sensitivity for the machine learning models varied from 95.7% to 99.3%, and specificity varied from 96.3% to 98.9%. CONCLUSION: This study provides preliminary evidence supporting the value of machine learning for detecting WBIT errors affecting CBC results. Although further work addressing practical issues is required, substantial patient-safety benefits await the successful deployment of machine learning models for WBIT error detection.

Decision support or autonomous artificial intelligence? The case of wrong blood in tube errors.

OBJECTIVES: Artificial intelligence (AI) models are increasingly being developed for clinical chemistry applications, however, it is not understood whether human interaction with the models, which may occur once they are implemented, improves or worsens their performance. This study examined the effect of human supervision on an artificial neural network trained to identify wrong blood in tube (WBIT) errors. METHODS: De-identified patient data for current and previous (within seven days) electrolytes, urea and creatinine (EUC) results were used in the computer simulation of WBIT errors at a rate of 50%. Laboratory staff volunteers reviewed the AI model's predictions, and the EUC results on which they were based, before making a final decision regarding the presence or absence of a WBIT error. The performance of this approach was compared to the performance of the AI model operating without human supervision. RESULTS: Laboratory staff supervised the classification of 510 sets of EUC results. This workflow identified WBIT errors with an accuracy of 81.2%, sensitivity of 73.7% and specificity of 88.6%. However, the AI model classifying these samples autonomously was superior on all metrics (p-values<0.05), including accuracy (92.5%), sensitivity (90.6%) and specificity (94.5%). CONCLUSIONS: Human interaction with AI models can significantly alter their performance. For computationally complex tasks such as WBIT error identification, best performance may be achieved by autonomously functioning AI models.

Identifying mislabelled samples: Machine learning models exceed human performance.

BACKGROUND: It is difficult for clinical laboratories to identify samples that are labelled with the details of an incorrect patient. Many laboratories screen for these errors with delta checks, with final decision-making based on manual review of results by laboratory staff. Machine learning models have been shown to outperform delta checks for identifying these errors. However, a comparison of machine learning models to human-level performance has not yet been made. METHODS: Deidentified data for current and previous (within seven days) electrolytes, urea and creatinine results was used in the computer simulation of mislabelled samples. Eight different machine learning models were developed on 127,256 sets of results using different algorithms: artificial neural network, extreme gradient boosting, support vector machine, random forest, logistic regression, k-nearest neighbours and two decision trees (one complex and one simple). A separate test data-set (n = 14,140) was used to evaluate the performance of these models as well as laboratory staff volunteers, who manually reviewed a random subset of this data (n = 500). RESULTS: The best performing machine learning model was the artificial neural network (92.1% accuracy), with the simple decision tree demonstrating the poorest accuracy (86.5%). The accuracy of laboratory staff for identifying mislabelled samples was 77.8%. CONCLUSIONS: The results of this preliminary investigation suggest that even relatively simple machine learning models can exceed human performance for identifying mislabelled samples. Machine learning techniques should be considered for implementation in clinical laboratories to assist with error identification.

Indirect Reference Intervals: Harnessing the Power of Stored Laboratory Data.

Reference intervals are relied upon by clinicians when interpreting their patients' test results. Therefore, laboratorians directly contribute to patient care when they report accurate reference intervals. The traditional approach to establishing reference intervals is to perform a study on healthy volunteers. However, the practical aspects of the staff time and cost required to perform these studies make this approach difficult for clinical laboratories to routinely use. Indirect methods for deriving reference intervals, which utilise patient results stored in the laboratory's database, provide an alternative approach that is quick and inexpensive to perform. Additionally, because large amounts of patient data can be used, the approach can provide more detailed reference interval information when multiple partitions are required, such as with different age-groups. However, if the indirect approach is to be used to derive accurate reference intervals, several considerations need to be addressed. The laboratorian must assess whether the assay and patient population were stable over the study period, whether data 'clean-up' steps should be used prior to data analysis and, often, how the distribution of values from healthy individuals should be modelled. The assumptions and potential pitfalls of the particular indirect technique chosen for data analysis also need to be considered. A comprehensive understanding of all aspects of the indirect approach to establishing reference intervals allows the laboratorian to harness the power of the data stored in their laboratory database and ensure the reference intervals they report are accurate.

Parathyroid hormone: Data mining for age-related reference intervals in adults.

OBJECTIVE: Age-related changes in parathyroid hormone (PTH) have been previously documented in adults. However, because of the limitations of traditional approaches to establishing reference intervals, age-related reference intervals have not been defined. We sought to use a data mining approach to derive age-related PTH reference intervals. DESIGN AND PARTICIPANTS: Results from patients undergoing PTH testing over a 4-year period were extracted from the database of a private pathology laboratory in New South Wales, Australia. Patients were included in the study if they were 18 years or older and had simultaneous determination of PTH, serum calcium, estimated glomerular filtration rate and 25-hydroxyvitamin D (25-OHD). Patients with abnormalities of serum calcium or renal function were excluded. MEASUREMENTS: Bhattacharya analysis of log-transformed data was used to derive age-related PTH reference intervals across adulthood. RESULTS: Results were available for 33 652 subjects. Among patients with optimal 25-OHD status, older age was associated with higher PTH concentrations. Age-related reference intervals were derived and showed a 63% increase in the upper and lower reference limits between the youngest (18-29 years of age) and the oldest (80 years of age or older) age partitions. The appropriateness of using a single reference interval for patients of all ages was evaluated against objective criteria and was found to be unsatisfactory. CONCLUSIONS: Data mining was demonstrated to be a useful tool for establishing age-related PTH reference intervals. The technique demonstrated that increasing age is associated with higher PTH concentrations and that age-related reference intervals are important for accurate result interpretation.

Assessment of vitamin D status - a changing landscape.

In recent years it has been shown that vitamin D deficiency is associated with an increased incidence as well as the progression of a broad range of diseases including osteoporosis, rickets, cardiovascular disease, autoimmune disease, multiple sclerosis and cancer. Consequently, requests for the assessment of vitamin D status have increased dramatically. Despite significant progress in the analysis of vitamin D metabolites and an expansion of our pathophysiological knowledge of vitamin D, the assessment of vitamin D status remains a challenging and partially unresolved issue. Current guidelines from scientific bodies recommend the measurement of 25-hydroxy vitamin D (25-OHD) in blood as the preferred test. However, growing evidence indicates significant limitations of this test, including analytical aspects and interpretation of results. In addition, the relationships between 25-OHD and various clinical indices, such as bone mineral density and fracture risk, are rather weak and not consistent across races. Recent studies have systematically investigated new markers of vitamin D status including the vitamin D metabolite ratio (VMR) (ratio between 25-OHD and 24,25-dihydroxy vitamin D), bioavailable 25-OHD [25-OHD not bound to vitamin D binding protein (DBP)], and free 25-OHD [circulating 25-OHD bound to neither DBP nor albumin (ALB)]. These parameters may potentially change how we will assess vitamin D status in the future. Although these new biomarkers have expanded our knowledge about vitamin D metabolism, a range of unresolved issues regarding their measurement and the interpretation of results prevent their use in daily practice. It can be expected that some of these issues will be overcome in the near future so that they may be considered for routine use (at least in specialized centers). In addition, genetic studies have revealed several polymorphisms in key proteins of vitamin D metabolism that affect the circulating concentrations of vitamin D metabolites. The affected proteins include DBP, 7-dehydrocholesterol synthase and the vitamin D receptor (VDR). Here we aim to review existing knowledge regarding the biochemistry, physiology and measurement of vitamin D. We will also provide an overview of current and emerging biomarkers for the assessment of vitamin D status, with particular attention methodological aspects and their usefulness in clinical practice.

Serum indices: managing assay interference.

Clinical laboratories frequently encounter samples showing significant haemolysis, icterus or lipaemia. Technical advances, utilizing spectrophotometric measurements on automated chemistry analysers, allow rapid and accurate identification of such samples. However, accurate quantification of haemolysis, icterus and lipaemia interference is of limited value if laboratories do not set rational alert limits, based on sound interference testing experiments. Furthermore, in the context of increasing consolidation of laboratories and the formation of laboratory networks, there is an increasing requirement for harmonization of the handling of haemolysis, icterus and lipaemia-affected samples across different analytical platforms. Harmonization may be best achieved by considering both the analytical aspects of index measurement and the possible variations in the effects of haemolysis, icterus and lipaemia interferences on assays from different manufacturers. Initial verification studies, followed up with ongoing quality control testing, can help a laboratory ensure the accuracy of haemolysis, icterus and lipaemia index results, as well as assist in managing any biases in index results from analysers from different manufacturers. Similarities, and variations, in the effect of haemolysis, icterus and lipaemia interference in assays from different manufacturers can often be predicted from the mechanism of interference. Nevertheless, interference testing is required to confirm expected similarities or to quantify differences. It is important that laboratories are familiar with a number of interference testing protocols and the particular strengths and weaknesses of each. A rigorous approach to all aspects of haemolysis, icterus and lipaemia interference testing allows the analytical progress in index measurement to be translated into improved patient care.

The role of telomeres and vitamin D in cellular aging and age-related diseases.

Aging is a complex biological process characterized by a progressive decline of organ functions leading to an increased risk of age-associated diseases and death. Decades of intensive research have identified a range of molecular and biochemical pathways contributing to aging. However, many aspects regarding the regulation and interplay of these pathways are insufficiently understood. Telomere dysfunction and genomic instability appear to be of critical importance for aging at a cellular level. For example, age-related diseases and premature aging syndromes are frequently associated with telomere shortening. Telomeres are repetitive nucleotide sequences that together with the associated sheltrin complex protect the ends of chromosomes and maintain genomic stability. Recent studies suggest that micronutrients, such as vitamin D, folate and vitamin B12, are involved in telomere biology and cellular aging. In particular, vitamin D is important for a range of vital cellular processes including cellular differentiation, proliferation and apoptosis. As a result of the multiple functions of vitamin D it has been speculated that vitamin D might play a role in telomere biology and genomic stability. Here we review existing knowledge about the link between telomere biology and cellular aging with a focus on the role of vitamin D. We searched the literature up to November 2014 for human studies, animal models and in vitro experiments that addressed this topic.

Impact of assay design on test performance: lessons learned from 25-hydroxyvitamin D.

BACKGROUND: Current automated immunoassays vary significantly in many aspects of their design. This study sought to establish if the theoretical advantages and disadvantages associated with different design formats of automated 25-hydroxyvitamin D (25-OHD) assays are translated into variations in assay performance in practice. METHODS: 25-OHD was measured in 1236 samples using automated assays from Abbott, DiaSorin, Roche and Siemens. A subset of 362 samples had up to three liquid chromatography-tandem mass spectrometry 25-OHD analyses performed. 25-OHD2 recovery, dilution recovery, human anti-animal antibody (HAAA) interference, 3-epi-25-OHD3 cross-reactivity and precision of the automated assays were evaluated. RESULTS: The assay that combined release of 25-OHD with analyte capture in a single step showed the most accurate 25-OHD2 recovery and the best dilution recovery. The use of vitamin D binding protein (DBP) as the capture moiety was associated with 25-OHD2 under-recovery, a trend consistent with 3-epi-25-OHD3 cross-reactivity and immunity to HAAA interference. Assays using animal-derived antibodies did not show 3-epi-25-OHD3 cross-reactivity but were variably susceptible to HAAA interference. Not combining 25-OHD release and capture in one step and use of biotin-streptavidin interaction for solid phase separation were features of the assays with inferior accuracy for diluted samples. The assays that used a backfill assay format showed the best precision at high concentrations but this design did not guarantee precision at low 25-OHD concentrations. CONCLUSIONS: Variations in design among automated 25-OHD assays influence their performance characteristics. Consideration of the details of assay design is therefore important when selecting and validating new assays.

Determination of vitamin D and its metabolites.

The demand for analysis of 25-hydroxyvitamin D has increased dramatically throughout the world over the past decade. As a consequence, a number of new automated assays have been introduced for 25-hydroxyvitamin D measurement. Automated assays have shown variable ability to meet the technical challenges associated with 25-hydroxyvitamin D measurement. Assays are able to meet performance goals for precision at high concentrations but fail to do so at low concentrations of 25-hydroxyvitamin D. The overall accuracy of automated methods has improved over recent years and generally shows good overall agreement with reference methods; however, discrepancies persist for individual samples. Liquid chromatography-tandem mass spectrometry is used by some routine laboratories for 25-hydroxyvitamin D analysis but its widespread use is hampered by limited sample throughput. 1,25-Dihydroxyvitamin D is an important analyte in specific clinical situations, which remains in the hands of specialised laboratories using manual analytical methods.

Red cell or serum folate: what to do in clinical practice?

Folate deficiency has been linked to diverse clinical manifestations and despite the importance of accurate assessment of folate status, the best test for routine use is uncertain. Both serum and red cell folate assays are widely available in clinical laboratories; however, red cell folate is the more time-consuming and costly test. This review sought to evaluate whether the red cell assay demonstrated superior performance characteristics to justify these disadvantages. Red cell folate, but not serum folate, measurements demonstrated analytical variation due to sample pre-treatment parameters, oxygen saturation of haemoglobin and haematocrit. Neither marker was clearly superior in characterising deficiency but serum folate more frequently showed the higher correlation with homocysteine, a sensitive marker of deficiency. Similarly, both serum and red cell folate were shown to increase in response to folic acid supplementation. However, serum folate generally gave the greater response and was able to distinguish different supplementation doses. The C677T polymorphism of methylenetetrahydrofolate reductase alters the distribution of folate forms in red cells and may thereby cause further analytical variability in routine red cell folate assays. Overall, serum folate is cheaper and faster to perform than red cell folate, is influenced by fewer analytical variables and provides an assessment of folate status that may be superior to red cell folate.

State-of-the-art vitamin D assays: a comparison of automated immunoassays with liquid chromatography-tandem mass spectrometry methods.

BACKGROUND: Vitamin D testing is increasing worldwide. Recently several diagnostic manufacturers including Abbott and Siemens have launched automated 25-hydroxy vitamin D (25OH-D) immunoassays. Furthermore, preexisting assays from DiaSorin and Roche have recently been modified. We compared the performance of 5 automated immunoassays, an RIA and 2 liquid chromatography-tandem mass spectrometry (LC-MS/MS) methods. METHODS: Aliquots of 170 randomly selected patient samples were prepared and 25OH-D was measured by 2 LC-MS/MS methods, an RIA (DiaSorin), and automated immunoassays from Abbott (Architect), DiaSorin (LIAISON), IDS (ISYS), Roche (E170, monoclonal 25OH-D(3) assay), and Siemens (Centaur). Within-run and between-run imprecision were evaluated by measurement of 5 replicates of 2 serum pools on 5 consecutive days. RESULTS: The LC-MS/MS methods agreed, with a concordance correlation coefficient (CCC) of 0.99 and bias of 0.56 µg/L (1.4 nmol/L). The RIA assay showed a performance comparable to LC-MS/MS, with a CCC of 0.97 and a mean bias of 1.1 µg/L (2.7 nmo/L). All immunoassays measured total 25OH-D (including D(3) and D(2)), with the exception of the Roche assay (D(3) only). Among the immunoassays detecting total 25OH-D, the CCCs varied between 0.85 (Abbott) to 0.95 (LIAISON). The mean bias ranged between 0.2 µg/L (0.5 nmol/L) (LIAISON) and 4.56 µg/L (11.4 nmol/L) (Abbott). The Roche 25OH-D(3) assay demonstrated small mean bias [-2.7 µg/L (-6.7 nmol/L)] [-2.7 µg/L (-6.7 nmol/L)] but a low CCC of just 0.66. Most assays demonstrated good intra- and interassay precision, with CV <10%. CONCLUSIONS: Automated immunoassays demonstrated variable performance and not all tests met our minimum performance goals. It is important that laboratories be aware of the limitations of their assay.