Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper From the College of American Pathologists.

CONTEXT.­: Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. OBJECTIVE.­: To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. DATA SOURCES.­: An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. CONCLUSIONS.­: Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

Diagnostic quality model (DQM): an integrated framework for the assessment of diagnostic quality when using AI/ML.

BACKGROUND: Laboratory medicine has reached the era where promises of artificial intelligence and machine learning (AI/ML) seem palpable. Currently, the primary responsibility for risk-benefit assessment in clinical practice resides with the medical director. Unfortunately, there is no tool or concept that enables diagnostic quality assessment for the various potential AI/ML applications. Specifically, we noted that an operational definition of laboratory diagnostic quality - for the specific purpose of assessing AI/ML improvements - is currently missing. METHODS: A session at the 3rd Strategic Conference of the European Federation of Laboratory Medicine in 2022 on "AI in the Laboratory of the Future" prompted an expert roundtable discussion. Here we present a conceptual diagnostic quality framework for the specific purpose of assessing AI/ML implementations. RESULTS: The presented framework is termed diagnostic quality model (DQM) and distinguishes AI/ML improvements at the test, procedure, laboratory, or healthcare ecosystem level. The operational definition illustrates the nested relationship among these levels. The model can help to define relevant objectives for implementation and how levels come together to form coherent diagnostics. The affected levels are referred to as scope and we provide a rubric to quantify AI/ML improvements while complying with existing, mandated regulatory standards. We present 4 relevant clinical scenarios including multi-modal diagnostics and compare the model to existing quality management systems. CONCLUSIONS: A diagnostic quality model is essential to navigate the complexities of clinical AI/ML implementations. The presented diagnostic quality framework can help to specify and communicate the key implications of AI/ML solutions in laboratory diagnostics.

Recommendations for the Use of in Silico Approaches for Next-Generation Sequencing Bioinformatic Pipeline Validation: A Joint Report of the Association for Molecular Pathology, Association for Pathology Informatics, and College of American Pathologists.

In silico approaches for next-generation sequencing (NGS) data modeling have utility in the clinical laboratory as a tool for clinical assay validation. In silico NGS data can take a variety of forms, including pure simulated data or manipulated data files in which variants are inserted into existing data files. In silico data enable simulation of a range of variants that may be difficult to obtain from a single physical sample. Such data allow laboratories to more accurately test the performance of clinical bioinformatics pipelines without sequencing additional cases. For example, clinical laboratories may use in silico data to simulate low variant allele fraction variants to test the analytical sensitivity of variant calling software or simulate a range of insertion/deletion sizes to determine the performance of insertion/deletion calling software. In this article, the Working Group reviews the different types of in silico data with their strengths and limitations, methods to generate in silico data, and how data can be used in the clinical molecular diagnostic laboratory. Survey data indicate how in silico NGS data are currently being used. Finally, potential applications for which in silico data may become useful in the future are presented.

The Ethics of Artificial Intelligence in Pathology and Laboratory Medicine: Principles and Practice.

Growing numbers of artificial intelligence applications are being developed and applied to pathology and laboratory medicine. These technologies introduce risks and benefits that must be assessed and managed through the lens of ethics. This article describes how long-standing principles of medical and scientific ethics can be applied to artificial intelligence using examples from pathology and laboratory medicine.

A Survey of LOINC Code Selection Practices Among Participants of the College of American Pathologists Coagulation (CGL) and Cardiac Markers (CRT) Proficiency Testing Programs.

CONTEXT.­: Biomedical terminologies such as Logical Observation Identifiers, Names, and Codes (LOINC) were developed to enable interoperability of health care data between disparate health information systems to improve patient outcomes, public health, and research activities. OBJECTIVE.­: To ascertain the utilization rate and accuracy of LOINC terminology mapping to 10 commonly ordered tests by participants of the College of American Pathologists (CAP) Proficiency Testing program. DESIGN.­: Questionnaires were sent to 1916 US and Canadian laboratories participating in the 2018 CAP coagulation (CGL) and/or cardiac markers (CRT) surveys requesting information on practice setting, instrument(s) and test method(s), and LOINC code selection and usage in the laboratory and electronic health records. RESULTS.­: Ninety of 1916 CGL and/or CRT participants (4.7%) responded to the questionnaire. Of the 275 LOINC codes reported, 54 (19.6%) were incorrect: 2 codes (5934-2 and 12345-1) (0.7%) did not exist in the LOINC database and the highest error rates were observed in the property (27 of 275, 9.8%), system (27 of 275, 9.8%), and component (22 of 275, 8.0%) LOINC axes. Errors in LOINC code selection included selection of the incorrect component (eg, activated clotting time instead of activated partial thromboplastin time); selection of panels that can never be used to obtain an individual analyte (eg, prothrombin time panel instead of international normalized ratio); and selection of an incorrect specimen type. CONCLUSIONS.­: These findings of real-world LOINC code implementation across a spectrum of laboratory settings should raise concern about the reliability and utility of using LOINC for clinical research or to aggregate data.

A Model Information Management Plan for Molecular Pathology Sequence Data Using Standards: From Sequencer to Electronic Health Record.

Incorporating genetic variant data into the electronic health record (EHR) in discrete computable fashion has vexed the informatics community for years. Genetic sequence test results are typically communicated by the molecular laboratory and stored in the EHR as textual documents. Although text documents are useful for human readability and initial use, they are not conducive for data retrieval and reuse. As a result, clinicians often struggle to find historical gene sequence results on a series of oncology patients within the EHR that might influence the care of the current patient. Second, identification of patients with specific mutation results in the EHR who are now eligible for new and/or changing therapy is not easily accomplished. Third, the molecular laboratory is challenged to monitor its sequencing processes for nonrandom process variation and other quality metrics. A novel approach to address each of these issues is presented and demonstrated. The authors use standard Health Level 7 laboratory result message formats in conjunction with international standards, Systematized Nomenclature of Medicine Clinical Terms and Human Genome Variant Society nomenclature, to represent, communicate, and store discrete gene sequence data within the EHR in a scalable fashion. This information management plan enables the support of the clinician at the point of care, enhances population management, and facilitates audits for maintaining laboratory quality.

Bone Marrow Synoptic Reporting for Hematologic Neoplasms: Guideline From the College of American Pathologists Pathology and Laboratory Quality Center.

CONTEXT: -There is ample evidence from the solid tumor literature that synoptic reporting improves accuracy and completeness of relevant data. No evidence-based guidelines currently exist for synoptic reporting for bone marrow samples. OBJECTIVE: -To develop evidence-based recommendations to standardize the basic components of a synoptic report template for bone marrow samples. DESIGN: -The College of American Pathologists Pathology and Laboratory Quality Center convened a panel of experts in hematopathology to develop recommendations. A systematic evidence review was conducted to address 5 key questions. Recommendations were derived from strength of evidence, open comment feedback, and expert panel consensus. RESULTS: -Nine guideline statements were established to provide pathology laboratories with a framework by which to develop synoptic reporting templates for bone marrow samples. The guideline calls for specific data groups in the synoptic section of the pathology report; provides a list of evidence-based parameters for key, pertinent elements; and addresses ancillary testing. CONCLUSION: -A framework for bone marrow synoptic reporting will improve completeness of the final report in a manner that is clear, succinct, and consistent among institutions.

Ordo ab Chao: framework for an integrated disease report.

CONTEXT: The volume of information that must be assimilated to appropriately manage patients with complex or chronic disease can make this task difficult because of the number of data points, their variable temporal availability, and the fact that they may reside in different systems or even institutions. OBJECTIVE .- To outline a framework for building an integrated disease report (IDR) that takes advantage of the capabilities of electronic reporting to create a single, succinct, interpretative report comprising all disease pertinent data. DESIGN: Disease pertinent data of an IDR include pathology results, laboratory and radiology data, pathologic correlations, risk profiles, and therapeutic implications. We used cancer herein as a representative process for proposing what is, to our knowledge, the first example of standardized guidelines for such a report. The IDR was defined as a modular, dynamic, electronic summary of the most current state of a patient in regard to a particular illness such as lung cancer or diabetes, which includes all information relevant for patient management. RESULTS: We propose the following 11 core data concepts that an IDR should include: patient identification; patient demographics; disease, diagnosis, and prognosis; tumor board dispositions and decisions; graphic timeline; preresection workup and therapy; resection workup; interpretative comment summarizing pertinent findings; biobanking data; postresection workup; and disease and patient status at follow-up. CONCLUSIONS: A well-executed IDR should improve patient care and efficiency for health care team members. It would demonstrate the added value of pathology interpretation and likely contribute to a reduction in errors and improved patient safety by decreasing the risk that important data will be overlooked.

Assessment of erythroid dysplasia by "difference from normal" in routine clinical flow cytometry workup.

INTRODUCTION: While multidimensional flow cytometry (MDF) has great utility in diagnostic workups of patients with suspected myelodysplastic syndromes (MDS), only the myeloid lineage has demonstrated reproducible abnormalities from multiple laboratories. With the effects of ammonium chloride (NH4 Cl) lysis on erythroid progenitors previously described, we applied this protocol to a patient cohort with diagnosed MDS to investigate phenotypic abnormalities that indicate erythroid dysplasia. METHOD: Bone marrow specimens [39 MDS, 9 acute myeloid leukemia (AML), 7 JAK2(V617F) positive myeloproliferative neoplasms (MPN), and 5 nutritional deficiencies] were processed by NH4 Cl lysis and Ficoll preparation and evaluated by MDF using a difference from normal algorithm. RESULTS: For the MDS cohort, phenotypic abnormalities on the mature erythroid progenitors were frequent for CD71 and CD36 (36% for each antigen); abnormalities for CD235a (8%) were observed. Among immature erythroid progenitors, abnormal maturation patterns (≤5%), and increased CD105 intensity (9%) were seen. Increased frequency of CD105 bright cells was observed (18%). While antigenic abnormalities correlated between NH4 Cl lysis and Ficoll preparation, the lysis method demonstrated the most consistent quantitative antigen intensities. Mean erythroid phenotypic abnormalities and prognostic cytogenetic subgroups correlated strongly. Morphologic and erythroid phenotypic abnormalities correlated, as did increasing FCSS and number of erythroid abnormalities, albeit without further increase for AML patients. DISCUSSION: These data expand the understanding of erythropoiesis and define immunophenotypic abnormalities that indicate dyserythropoiesis in MDS using a lysis protocol practical for routine implementation in clinical flow cytometric workup. Preliminary studies also indicate strong correlation between phenotypic erythroid dysplasia and poor prognosis, as classified cytogenetically.

Detection of clonal evolution in hematopoietic malignancies by combining comparative genomic hybridization and single nucleotide polymorphism arrays.

BACKGROUND: Array comparative genomic hybridization (aCGH) has become a powerful tool for analyzing hematopoietic neoplasms and identifying genome-wide copy number changes in a single assay. aCGH also has superior resolution compared with fluorescence in situ hybridization (FISH) or conventional cytogenetics. Integration of single nucleotide polymorphism (SNP) probes with microarray analysis allows additional identification of acquired uniparental disomy, a copy neutral aberration with known potential to contribute to tumor pathogenesis. However, a limitation of microarray analysis has been the inability to detect clonal heterogeneity in a sample. METHODS: This study comprised 16 samples (acute myeloid leukemia, myelodysplastic syndrome, chronic lymphocytic leukemia, plasma cell neoplasm) with complex cytogenetic features and evidence of clonal evolution. We used an integrated manual peak reassignment approach combining analysis of aCGH and SNP microarray data for characterization of subclonal abnormalities. We compared array findings with results obtained from conventional cytogenetic and FISH studies. RESULTS: Clonal heterogeneity was detected in 13 of 16 samples by microarray on the basis of log2 values. Use of the manual peak reassignment analysis approach improved resolution of the sample's clonal composition and genetic heterogeneity in 10 of 13 (77%) patients. Moreover, in 3 patients, clonal disease progression was revealed by array analysis that was not evident by cytogenetic or FISH studies. CONCLUSIONS: Genetic abnormalities originating from separate clonal subpopulations can be identified and further characterized by combining aCGH and SNP hybridization results from 1 integrated microarray chip by use of the manual peak reassignment technique. Its clinical utility in comparison to conventional cytogenetic or FISH studies is demonstrated.

Assessment of erythroid dysplasia by "difference from normal" in routine clinical flow cytometry work-up.

Introduction: While multidimensional flow cytometry (MDF) has great utility in diagnostic work-ups of patients with suspected myelodysplastic syndromes (MDS), only the myeloid lineage has demonstrated reproducible abnormalities from multiple laboratories. With the effects of ammonium chloride (NH4 Cl) lysis on erythroid progenitors previously described, we applied this protocol to a patient cohort with diagnosed MDS to investigate phenotypic abnormalities that indicate erythroid dysplasia. Method: Bone marrow specimens [39 MDS, 9 acute myeloid leukemia (AML), 7 JAK2V617F positive myeloproliferative neoplasms (MPN), 5 nutritional deficiencies] were processed by NH4 Cl lysis and Ficoll preparation and evaluated by MDF using a difference from normal algorithm. Results: For the MDS cohort, phenotypic abnormalities on the mature erythroid progenitors were frequent for CD71 and CD36 (36% for each antigen); abnormalities for CD235a (8%) were observed. Among immature erythroid progenitors, abnormal maturation patterns (≤5%) and increased CD105 intensity (9%) were seen. Increased frequency of CD105 bright cells was observed (18%). While antigenic abnormalities correlated between NH4 Cl lysis and Ficoll preparation, the lysis method demonstrated the most consistent quantitative antigen intensities. Mean erythroid phenotypic abnormalities and prognostic cytogenetic subgroups correlated strongly. Morphologic and erythroid phenotypic abnormalities correlated, as did increasing FCSS and number of erythroid abnormalities, albeit without further increase for AML patients. Discussion: These data expand the understanding of erythropoiesis and define immunophenotypic abnormalities that indicate dyserythropoiesis in MDS utilizing a lysis protocol practical for routine implementation in clinical flow cytometric work-up. Preliminary studies also indicate strong correlation between phenotypic erythroid dysplasia and poor prognosis, as classified cytogenetically. © 2014 Clinical Cytometry Society.

Intraclonal heterogeneity in concomitant monoclonal lymphocyte and plasma cell populations: combining flow cytometric cell sorting with molecular monoclonality profiling.

Flow cytometric cell sorting combined with molecular gene rearrangement analysis can assist in further characterizing simultaneously occurring, phenotypically distinct, monoclonal B-lymphoid and monoclonal plasma cell populations that express immunoglobulin of the same light chain. We previously established monoclonality profiles for lymphoid and plasma cell populations of lymphoplasmacytic lymphoma (LPL) bone marrow aspirates by using flow cytometric cell sorting and subsequent monoclonal gene rearrangement analysis. Our findings demonstrated that related genetic processes are less likely than unrelated genetic processes. Here, we demonstrated the utility of cell sorting combined with gene rearrangement (both immunoglobulin IgH and IgK) and IgVH sequence analysis as well as plasma cell targeted fluorescence in situ hybridization analysis in clinical cases of presumed Waldenström macroglobulinemia/LPL in which multiple distinct B-cell and plasma cell populations were identified. Combining cell sorting with subsequent molecular analysis can provide proof of identical monoclonal genotype for Waldenström macroglobulinemia/LPL and nonidentical distinct lymphoid and plasma cell populations in the clinical setting. Understanding how many clonal processes (molecular profiles) are present can help guide patient monitoring throughout treatment and potentially identify patients with worse outcomes.

Array-based karyotyping in plasma cell neoplasia after plasma cell enrichment increases detection of genomic aberrations.

The discovery of genomic abnormalities present in monoclonal plasma cells has diagnostic, prognostic, and disease-monitoring implications in plasma cell neoplasms (PCNs). However, technical and disease-related limitations hamper the detection of these abnormalities using cytogenetic analysis or fluorescence in situ hybridization (FISH). In this study, 28 bone marrow specimens with known PCNs were examined for the presence of genomic abnormalities using microarray analysis after plasma cell enrichment. Cytogenetic analysis was performed on 15 of 28 samples, revealing disease-related genomic aberrations in only 3 (20%) of 15 cases. FISH analysis was performed on enriched plasma cells and detected aberrations in 84.6% of specimens while array comparative genomic hybridization (aCGH) detected abnormalities in 89.3% of cases. Furthermore, aCGH revealed additional abnormalities in 24 cases compared with FISH alone. We conclude that aCGH after plasma cell enrichment, in combination with FISH, is a valuable approach for routine clinical use in achieving a more complete genetic characterization of patients with PCN.

Detection of genomic abnormalities in multiple myeloma: the application of FISH analysis in combination with various plasma cell enrichment techniques.

Multiple myeloma (MM) is a hematopoietic neoplasm characterized by malignant plasma cells (PCs) that accumulate in the bone marrow. A number of different genomic abnormalities are associated with MM; however, detection of these by fluorescence in situ hybridization (FISH) can be limited by the percentage of PCs in the specimen. In this study, we tested 20 bone marrow specimens with known MM and a low concentration of monoclonal PCs for the presence of genomic abnormalities using FISH in combination with various PC enrichment techniques: magnetic cell sorting, targeted manual scoring, and automated image analysis. In addition, flow cytometric cell sorting of PCs in combination with FISH analysis was also tested for minimal residual disease applications. Different parameters were evaluated when assessing the detection efficiency of each approach. FISH results are highly dependent on the chosen enrichment method. We describe the evaluation of different techniques applicable for various laboratory settings and specimen parameters.

Phenotypic abnormalities strongly reflect genotype in patients with unexplained cytopenias.

BACKGROUND: In patients with unexplained cytopenias, abnormal karyotyping studies can be found with inconclusive light microscopic findings. Multidimensional flow cytometry (FCM) can identify myelomonocytic cells with aberrant phenotypes often not seen by standard morphology. METHODS: In 431 patients presenting with unexplained cytopenia(s) FCM results were compared to abnormal karyotyping and FISH results recognized as associated with myelodysplastic syndrome (MDS) in the 2008 WHO classification, to assess the degree of and types of phenotypic abnormalities observed using a previously reported flow cytometric scoring system (FCSS). Fluorescence activated cell sorting was also used to identify subpopulations of abnormal maturing myelomonocytic cells that carry the genotypic abnormality. RESULTS: For marrows with complex (three or more karyotypic abnormalities), two abnormalities, isolated chromosome seven anomalies, del(5q) or del(13q), 100% of cases were positive when using a FCSS cutoff of ≥ 2. Trisomy 8, del(20 q), and minus Y had flow scores ≥ 2 in 72, 60, and 18%, respectively, but in some cases the flow score was high, indicating myeloid dysplasia. Most patients (16/22) with high myeloid progenitor cells (MyPC) (> 20%) also exhibited maturing myeloid cell abnormalities by FCM. Morphology was negative in the maturing myeloid cells in many cases with phenotypically abnormal myeloid cells. CONCLUSIONS: The high correlation between genotypic and phenotypic abnormalities suggests a possible increased utility of flow cytometry in the diagnosis of patients with unexplained cytopenias and may be useful in future clinical studies and in the classification by the WHO, using the FCSS rather than simple counting of flow cytometric abnormalities.