Accurately identifying incident cases of venous thromboembolism in the electronic health record: Performance of a novel phenotyping algorithm.

BACKGROUND: Accurate identification of incident venous thromboembolism (VTE) for quality improvement and health services research is challenging. The purpose of this study was to evaluate the performance of a novel incident VTE phenotyping algorithm defined using standard terminologies, requiring three key indicators documented in the electronic health record (EHR): VTE diagnostic code, VTE-related imaging procedure code, and anticoagulant medication code. METHODS: Retrospective chart reviews were conducted to assess the performance of the algorithm using a random sample of phenotype(+) and phenotype(-) diagnostic encounters from primary care practices and acute care sites affiliated with five hospitals across a large integrated care delivery system in Massachusetts. The performance of the algorithm was evaluated by calculating the positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity, using the phenotype(+) and phenotype(-) diagnostic encounters sample and target population data. RESULTS: Based on gold-standard manual chart review, the algorithm had a PPV of 95.2 % (95 % CI: 93.1-96.8 %), NPV of 97.1 % (95 % CI: 95.3-98.4 %), sensitivity of 91.7 % (95 % CI: 90.8-92.6 %), and specificity of 98.4 % (95 % CI: 98.1-98.6 %). The algorithm systematically misclassified a low number of specific types of encounters, highlighting potential areas for improvement. CONCLUSIONS: This novel phenotyping algorithm offers an accurate approach for identifying incident VTE in general populations using EHR data and standard terminologies, and accurately identifies the specific encounter and date of diagnosis of the incident VTE. This approach can be used for measurement of incident VTE to drive quality improvement, research to expand the evidence, and development of quality metrics and clinical decision support to improve the diagnostic process.

Improved accuracy and efficiency of primary care fall risk screening of older adults using a machine learning approach.

BACKGROUND: While many falls are preventable, they remain a leading cause of injury and death in older adults. Primary care clinics largely rely on screening questionnaires to identify people at risk of falls. Limitations of standard fall risk screening questionnaires include suboptimal accuracy, missing data, and non-standard formats, which hinder early identification of risk and prevention of fall injury. We used machine learning methods to develop and evaluate electronic health record (EHR)-based tools to identify older adults at risk of fall-related injuries in a primary care population and compared this approach to standard fall screening questionnaires. METHODS: Using patient-level clinical data from an integrated healthcare system consisting of 16-member institutions, we conducted a case-control study to develop and evaluate prediction models for fall-related injuries in older adults. Questionnaire-derived prediction with three questions from a commonly used fall risk screening tool was evaluated. We then developed four temporal machine learning models using routinely available longitudinal EHR data to predict the future risk of fall injury. We also developed a fall injury-prevention clinical decision support (CDS) implementation prototype to link preventative interventions to patient-specific fall injury risk factors. RESULTS: Questionnaire-based risk screening achieved area under the receiver operating characteristic curve (AUC) up to 0.59 with 23% to 33% similarity for each pair of three fall injury screening questions. EHR-based machine learning risk screening showed significantly improved performance (best AUROC = 0.76), with similar prediction performance between 6-month and one-year prediction models. CONCLUSIONS: The current method of questionnaire-based fall risk screening of older adults is suboptimal with redundant items, inadequate precision, and no linkage to prevention. A machine learning fall injury prediction method can accurately predict risk with superior sensitivity while freeing up clinical time for initiating personalized fall prevention interventions. The developed algorithm and data science pipeline can impact routine primary care fall prevention practice.

Predicting COVID-19 severity: Challenges in reproducibility and deployment of machine learning methods.

The increasing use of electronic health records (EHR) based computable phenotypes in clinical research is providing new opportunities for development of data-driven medical applications. Adopted widely in the United States and globally, EHRs facilitate systematic collection of patients' longitudinal information, which serves as one of the important foundations for artificial intelligence applications in medicine. Harmonization of input variables and outcome definitions is critically important for wider clinical applicability of artificial intelligence (AI) methodologies. In this review, we focused on Coronavirus Disease 2019 (COVID-19) severity machine learning prediction models and explored the pipeline for standardizing future disease severity model development using EHR information. We identified 2,967 studies published between 01/01/2020 and 02/15/2022 and selected 135 independent studies that had built machine learning prediction models to predict severity related outcomes of COVID-19 patients based on EHR data for the final review. These 135 studies spanning across 27 counties covered a broad range of severity related prediction outcomes. We observed substantial inconsistency in COVID-19 severity phenotype definitions among models in these studies. Moreover, there was a gap between the outcome of these models and clinician-recognized clinical concepts. Accordingly, we recommend that robust clinical input metrics, with outcome definitions which eliminate ambiguity in interpretation, to reduce algorithmic bias, mitigate model brittleness and improve generalizability of a universal model for COVID-19 severity. This framework can potentially be extended to broader clinical application.

Testing of an Electronic Clinical Quality Measure for Diagnostic Delay of Venous Thromboembolism (DOVE) in Primary Care.

Venous Thromboembolism (VTE) is a serious, preventable public health problem that requires timely treatment. Because signs and symptoms are non-specific, patients often present to primary care providers with VTE symptoms prior to diagnosis. Today there are no federal measurement tools in place to track delayed diagnosis of VTE. We developed and tested an electronic clinical quality measure (eCQM) to quantify Diagnostic Delay of Venous Thromboembolism (DOVE); the rate of avoidable delayed VTE events occurring in patients with a VTE who had reported VTE symptoms in primary care within 30 days of diagnosis. DOVE uses routinely collected EHR data without contributing to documentation burden. DOVE was tested in two geographically distant healthcare systems. Overall DOVE rates were 72.60% (site 1) and 77.14% (site 2). This novel, data-driven eCQM could inform healthcare providers and facilities about opportunities to improve care, strengthen incentives for quality improvement, and ultimately improve patient safety.

Cost Savings Associated With Implementing 4 Total Joint Replacement Electronic Clinical Quality Measures Nationally: 2020-2040.

BACKGROUND: Digital transformation using widely available electronic data is a key component to improving health outcomes and customer choice and decreasing cost and measurement burden. Despite these benefits, existing information on the potential cost savings from electronic clinical quality measures (eCQMs) is limited. METHODS: We assessed the costs of implementing 4 eCQMs related to total hip and/or total knee arthroplasty into electronic health record systems across healthcare systems in the United States. We used published literature and technical expert panel consultation to calculate low-, mid-, and high-range hip and knee arthroplasty surgery projections, and used empirical testing, literature, and technical expert panel consultation to develop an economic model to assess projected cost savings of eCQMs when implemented nationally. RESULTS: Low-, mid-, and high-range projected cost savings for year's 2020, 2030, and 2040 were calculated for 4 orthopedic eCQMs. Mid-range projected cost savings for 2020 ranged from $7.9 to $31.9 million per measure per year. A breakeven of between 0.5% and 5.1% of adverse events (measure dependent) must be averted for cost savings to outweigh implementation costs. CONCLUSIONS: All measures demonstrated potential cost savings. These findings suggest that eCQMs have the potential to lower healthcare costs and improve patient outcomes without adding to physician documentation burden. The Centers for Medicare and Medicaid Services' investment in eCQMs is an opportunity to reduce adverse outcomes and excess costs in orthopedics.

Opioid2MME: Standardizing opioid prescriptions to morphine milligram equivalents from electronic health records.

BACKGROUND: The national increase in opioid use and misuse has become a public health crisis in the U.S. To tackle this crisis, the systematic evaluation and monitoring of opioid prescribing patterns is necessary. Thus, opioid prescriptions from electronic health records (EHRs) must be standardized to morphine milligram equivalent (MME) to facilitate monitoring and surveillance. While most studies report MMEs to describe opioid prescribing patterns, there is a lack of transparency regarding their data pre-processing and conversion processes for replication or comparison purposes. METHODS: In this work, we developed Opioid2MME, a SQL-based open-source framework, to convert opioid prescriptions to MMEs using EHR prescription data. The MME conversions were validated internally using F-measures through manual chart review; were compared with two existing tools, as MedEx and MedXN; and the framework was tested in an external academic EHR system. RESULTS: We identified 232,913 prescriptions for 49,060 unique patients in the EHRs, 2008-2019. We manually annotated a sample of prescriptions to assess the performance of the framework. The internal evaluation for medication information extraction achieved F-measures from 0.98 to 1.00 for each piece of the extracted information, outperforming MedEx and MedXN (F-Scores 0.98 and 0.94, respectively). MME values in the internal EHR system obtained a F-measure of 0.97 and identified 3% of the data as outliers and 7% missing values. The MME conversion in the external EHR system obtained 78.3% agreement between the MME values obtained with the development site. CONCLUSIONS: The results demonstrated that the framework is replicable and capable of converting opioid prescriptions to MMEs across different medical institutions. In summary, this work sets the groundwork for the systematic evaluation and monitoring of opioid prescribing patterns across healthcare systems.

Multi-Site Testing of an Opioid Prescribing Electronic Clinical Quality Measure Following Elective Primary Total Hip and/or Total Knee Arthroplasties.

As the United States faces the third wave of the ongoing opioid epidemic, development of measures which report on prolonged opioid prescribing (POP) rates, specifically following orthopedic surgeries, are needed to better understand and improve prescribing practices at the clinician group level. Brigham and Women's Hospital (BWH) has been contracted by the Centers for Medicare and Medicaid Services (CMS) to create a novel electronic clinical quality measure (eCQM) to quantify the prolonged opioid prescribing rate of opioid episodes lasting > 42 days in patients aged 18+ years following elective primary total hip arthroplasties (THA) and/or total knee arthroplasties (TKA) for use in the Merit-Based Incentive Payment System (MIPS). When this measure was tested on two geographically distinct sites, it was found that the THA rate was 3.80% and 16.07% at sites 1 and 2, respectively, and that the TKA rate is 7.65% and 24.15% at sites 1 and 2, respectively. This manuscript reports on the testing of this eCQM between these two sites, highlighting differences in state and organizational level policies regarding opioid prescribing and documentation practices.

Testing a Novel Inpatient Respiratory Depression Electronic Clinical Quality Measure (eCQM) for Orthopedic Practice in Two Large U.S. Health Systems.

The objective of this study was to assess the feasibility of using an electronic clinical quality measure (eCQM) to assess inpatient respiratory depression rates following elective primary total hip or total knee arthroplasty using data routinely collected in electronic health records. Measure testing was conducted at two large urban, academic health systems - Mass General Brigham and a geographically distant system in southern U.S. The risk-adjusted inpatient respiratory depression rates were 3.83 and 2.73% for the two health systems, respectively. Clinician group rates ranged from 1.40 to 4.35%, demonstrating opportunity for improvement. Both the data and measure specifications showed strong reliability and validity to allow for calculation of accurate and comparable rates of inpatient respiratory depression.

Predicting hospitalization of COVID-19 positive patients using clinician-guided machine learning methods.

OBJECTIVES: The coronavirus disease 2019 (COVID-19) is a resource-intensive global pandemic. It is important for healthcare systems to identify high-risk COVID-19-positive patients who need timely health care. This study was conducted to predict the hospitalization of older adults who have tested positive for COVID-19. METHODS: We screened all patients with COVID test records from 11 Mass General Brigham hospitals to identify the study population. A total of 1495 patients with age 65 and above from the outpatient setting were included in the final cohort, among which 459 patients were hospitalized. We conducted a clinician-guided, 3-stage feature selection, and phenotyping process using iterative combinations of literature review, clinician expert opinion, and electronic healthcare record data exploration. A list of 44 features, including temporal features, was generated from this process and used for model training. Four machine learning prediction models were developed, including regularized logistic regression, support vector machine, random forest, and neural network. RESULTS: All 4 models achieved area under the receiver operating characteristic curve (AUC) greater than 0.80. Random forest achieved the best predictive performance (AUC = 0.83). Albumin, an index for nutritional status, was found to have the strongest association with hospitalization among COVID positive older adults. CONCLUSIONS: In this study, we developed 4 machine learning models for predicting general hospitalization among COVID positive older adults. We identified important clinical factors associated with hospitalization and observed temporal patterns in our study cohort. Our modeling pipeline and algorithm could potentially be used to facilitate more accurate and efficient decision support for triaging COVID positive patients.

Testing of a Risk-Standardized Complication Rate Electronic Clinical Quality Measure (eCQM) for Total Hip and/or Total Knee Arthroplasty.

Supported by the Centers for Medicare & Medicaid Services (CMS), Brigham and Women's Hospital (BWH) has retooled the existing claims-based measures NQF1550 and NQF3493 into an electronic clinical quality measure (eCQM) to assess the risk-standardized complication rate (RSCR) following elective primary total hip (THA) and knee arthroplasty (TKA) at the clinician group level. This novel eCQM includes risk-adjustment for social determinants of health, includes all adult patients from all payers, leverages electronic health records (EHRs) rather than claims-based data, and includes both inpatient and outpatient procedures and complications which offers benefits compared to existing metrics. Following testing in two geographically different healthcare systems, the overall risk-standardized complication rate within 90 days following THA and TKA at the two sites was 3.60% (Site 1) and 3.70% (Site 2). This measure is designed for use in the Merit-Based Incentive Payment System (MIPS).

Testing of a Risk-Standardized Major Bleeding and Venous Thromboembolism Electronic Clinical Quality Measure for Elective Total Hip and/or Knee Arthroplasties.

Brigham and Women's Hospital has received funding from the Centers for Medicare and Medicaid Services to develop a novel electronic clinical quality measure to assess the risk-standardized major bleeding and venous thromboembolism (VTE) rate following elective total hip and/or knee arthroplasty. There are currently no existing measures that evaluate both the bleeding and VTE events following joint arthroplasty (TJA). Our novel composite measure was tested within two academic health systems with 17 clinician groups meeting the inclusion criteria. Following risk adjustment, the overall adjusted bleeding rate was 3.87% and ranged between 1.99% - 5.66%. The unadjusted VTE rate was 0.39% and ranged between 0% - 2.65%. The overall VTE/Bleeding composite score was 2.15 and ranged between 1.15 - 3.19. This measure seeks to provide clinician groups with a tool to assess their patient bleeding and VTE rates and compare them to their peers, ultimately providing an evidence-based quality metric assessing orthopedic practices.

Development of four electronic clinical quality measures (eCQMs) for use in the Merit-based Incentive Payment System (MIPS) following elective primary total hip and knee arthroplasty.

The Centers for Medicare & Medicaid Services (CMS) supported Brigham and Women's Hospital (BWH) Center for Patient Safety, Research, and Practice to retool one existing National Quality Forum (NQF) endorsed clinical quality measure (CQM) measure into an electronic clinical quality measure (eCQM) and develop three new eCQMs related to orthopedic care. This manuscript details the iterative process of measure development through environmental scans and stakeholder feedback prior to testing at two geographically different sites. The four measures under development are the: Risk Standardized Complication Rate (RSCR), Risk Standardized Venous Thromboembolism and Major Bleeding Rate (VTE/Bleeding), Risk Standardized Prolonged Opioid Prescribing Rate (POP), and the Risk Standardized Inpatient Respiratory Depression Rate (IRD).

Development and Alpha Testing of Specifications for a Prolonged Opioid Prescribing Electronic Clinical Quality Measure (eCQM).

Brigham and Women's Hospital (BWH) has received funding from the Centers for Medicare and Medicaid Services (CMS) to design and implement an electronic clinical quality measure (eCQM) assessing the rate of prolonged opioid prescribing practices following Total Hip Arthroplasty (THA) and Total Knee Arthroplasty (TKA). Utilizing an existing guideline, 'prolonged prescribing' has been defined as opioid prescriptions that exceed 42 days (6 weeks) following surgery. This measure was tested on 12,803 Partners' Healthcare (PHS) patients. Findings demonstrated that after 42 days, meeting the criteria for 'prolonged prescribing' as defined by the proposed measure, 3.7% of THA patients and 12.1% of TKA patients were still receiving opioids. With a better understanding of how specific clinician group post-operative prescribing practices compare with their peers and incorporating monetary incentives through the MIPS participation pathway of the Quality Payment Program (QPP), this measure will motivate orthopedic practices to improve their prescribing patterns, ultimately driving evidence-based quality improvement.

Re-tooling an Existing Clinical Quality Measure for Chronic Opioid Use to an Electronic Clinical Quality Measure (eCQM) for Post-Operative Opioid Prescribing: Development and Testing of Draft Specifications.

The objective was to re-tool the existing claims-based measure NQF2940 "Use of Opioids at High Dosage in Persons Without Cancer" to an electronic clinical quality measure (eCQM) for use by orthopedic practices to assess potentially inappropriate high-dose post-operative opioid prescribing practices. Measure specifications were revised based on stakeholder feedback, initial testing and a targeted review of the literature. The eCQM was developed and alpha tested on 9,108 opioid-naive patients who received an elective primary total hip or total knee arthroplasty at Mass General Brigham formerly Partners HealthCare System) from 2016 to 2018. Thirty-eight percent of patients were prescribed high doses (defined as an average daily dose ≥90 morphine milligram equivalents) for the duration of their post-operative opioid prescriptions, demonstrating that this is a meaningful performance measure with substantial opportunity for improvement. National implementation and reporting of this eCQM could be used to facilitate quality improvement to deliver standardized, safe and high-quality care.