Development and Validation of a Machine Learning Algorithm Using Clinical Pages to Predict Imminent Clinical Deterioration.

BACKGROUND: Early detection of clinical deterioration among hospitalized patients is a clinical priority for patient safety and quality of care. Current automated approaches for identifying these patients perform poorly at identifying imminent events. OBJECTIVE: Develop a machine learning algorithm using pager messages sent between clinical team members to predict imminent clinical deterioration. DESIGN: We conducted a large observational study using long short-term memory machine learning models on the content and frequency of clinical pages. PARTICIPANTS: We included all hospitalizations between January 1, 2018 and December 31, 2020 at Vanderbilt University Medical Center that included at least one page message to physicians. Exclusion criteria included patients receiving palliative care, hospitalizations with a planned intensive care stay, and hospitalizations in the top 2% longest length of stay. MAIN MEASURES: Model classification performance to identify in-hospital cardiac arrest, transfer to intensive care, or Rapid Response activation in the next 3-, 6-, and 12-hours. We compared model performance against three common early warning scores: Modified Early Warning Score, National Early Warning Score, and the Epic Deterioration Index. KEY RESULTS: There were 87,783 patients (mean [SD] age 54.0 [18.8] years; 45,835 [52.2%] women) who experienced 136,778 hospitalizations. 6214 hospitalized patients experienced a deterioration event. The machine learning model accurately identified 62% of deterioration events within 3-hours prior to the event and 47% of events within 12-hours. Across each time horizon, the model surpassed performance of the best early warning score including area under the receiver operating characteristic curve at 6-hours (0.856 vs. 0.781), sensitivity at 6-hours (0.590 vs. 0.505), specificity at 6-hours (0.900 vs. 0.878), and F-score at 6-hours (0.291 vs. 0.220). CONCLUSIONS: Machine learning applied to the content and frequency of clinical pages improves prediction of imminent deterioration. Using clinical pages to monitor patient acuity supports improved detection of imminent deterioration without requiring changes to clinical workflow or nursing documentation.

Impact of inpatient addiction psychiatry consultation on opioid use disorder outcomes.

BACKGROUND AND OBJECTIVES: Addiction consultation services provide access to specialty addiction care during general hospital admission. This study assessed opioid use disorder (OUD) outcomes associated with addiction consultation. METHODS: Retrospective cohort study of individuals with OUD admitted to an academic medical center between 2018 and 2023. The exposure was addiction consultation. Outcomes included initiating medication for OUD (MOUD), hospital length of stay, before-medically-advised (BMA) discharge, and 30- and 90-day postdischarge acute care utilization. RESULTS: Of 26,766 admissions (10,501 patients) with OUD, 2826 addiction consultations were completed. Consultation cohort was more likely to be young, male, and White than controls. Consultation was associated with greater MOUD initiation (adjusted odds ratio [aOR], 5.07; 95% confidence interval [CI], 4.41-5.82), fewer emergency department visits at 30 (aOR, 0.78; 95% CI, 0.67-0.92) and 90 (aOR, 0.79; 95% CI, 0.69-0.89) days, and fewer hospitalizations at 30 (aOR, 0.65; 95% CI, 0.56 to 0.76) and 90 (aOR, 0.67; 95% CI, 0.59-0.76) days. Additionally, consultation patients were more likely to have a longer hospital stay and leave BMA. CONCLUSIONS AND SCIENTIFIC SIGNIFICANCE: Addiction consultation was associated with increased MOUD initiation and reduced postdischarge acute care utilization. This is the largest study to date showing a significant association between addiction psychiatry consultation and improved OUD outcomes when compared to controls. The observed reduction in postdischarge acute care utilization remains even after adjusting for MOUD initiation. Disparities in access to addiction consultation warrant further study.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

A systematic review of theoretical constructs in CDS literature.

BACKGROUND: Studies that examine the adoption of clinical decision support (CDS) by healthcare providers have generally lacked a theoretical underpinning. The Unified Theory of Acceptance and Use of Technology (UTAUT) model may provide such a theory-based explanation; however, it is unknown if the model can be applied to the CDS literature. OBJECTIVE: Our overall goal was to develop a taxonomy based on UTAUT constructs that could reliably characterize CDS interventions. METHODS: We used a two-step process: (1) identified randomized controlled trials meeting comparative effectiveness criteria, e.g., evaluating the impact of CDS interventions with and without specific features or implementation strategies; (2) iteratively developed and validated a taxonomy for characterizing differential CDS features or implementation strategies using three raters. RESULTS: Twenty-five studies with 48 comparison arms were identified. We applied three constructs from the UTAUT model and added motivational control to characterize CDS interventions. Inter-rater reliability was as follows for model constructs: performance expectancy (κ = 0.79), effort expectancy (κ = 0.85), social influence (κ = 0.71), and motivational control (κ = 0.87). CONCLUSION: We found that constructs from the UTAUT model and motivational control can reliably characterize features and associated implementation strategies. Our next step is to examine the quantitative relationships between constructs and CDS adoption.

Iterative heuristic design of temporal graphic displays with clinical domain experts.

Conventional electronic health record information displays are not optimized for efficient information processing. Graphical displays that integrate patient information can improve information processing, especially in data-rich environments such as critical care. We propose an adaptable and reusable approach to patient information display with modular graphical components (widgets). We had two study objectives. First, reduce numerous widget prototype alternatives to preferred designs. Second, derive widget design feature recommendations. Using iterative human-centered design methods, we interviewed experts to hone design features of widgets displaying frequently measured data elements, e.g., heart rate, for acute care patient monitoring and real-time clinical decision-making. Participant responses to design queries were coded to calculate feature-set agreement, average prototype score, and prototype agreement. Two iterative interview cycles covering 64 design queries and 86 prototypes were needed to reach consensus on six feature sets. Interviewers agreed that line graphs with a smoothed or averaged trendline, 24-h timeframe, and gradient coloring for urgency were useful and informative features. Moreover, users agreed that widgets should include key functions: (1) adjustable reference ranges, (2) expandable timeframes, and (3) access to details on demand. Participants stated graphical widgets would be used to identify correlating patterns and compare abnormal measures across related data elements at a specific time. Combining theoretical principles and validated design methods was an effective and reproducible approach to designing widgets for healthcare displays. The findings suggest our widget design features and recommendations match critical care clinician expectations for graphical information display of continuous and frequently updated patient data.

Medical and genetic correlates of long-term buprenorphine treatment in the electronic health records.

Despite the benefits associated with longer buprenorphine treatment duration (i.e., >180 days) (BTD) for opioid use disorder (OUD), retention remains poor. Research on the impact of co-occurring psychiatric issues on BTD has yielded mixed results. It is also unknown whether the genetic risk in the form of polygenic scores (PGS) for OUD and other comorbid conditions, including problematic alcohol use (PAU) are associated with BTD. We tested the association between somatic and psychiatric comorbidities and long BTD and determined whether PGS for OUD-related conditions was associated with BTD. The study included 6686 individuals with a buprenorphine prescription that lasted for less than 6 months (short-BTD) and 1282 individuals with a buprenorphine prescription that lasted for at least 6 months (long-BTD). Recorded diagnosis of substance addiction and disorders (Odds Ratio (95% CI) = 22.14 (21.88-22.41), P = 2.8 × 10-116), tobacco use disorder (OR (95% CI) = 23.4 (23.13-23.68), P = 4.5 × 10-111), and bipolar disorder (OR(95% CI) = 9.70 (9.48-9.92), P = 1.3 × 10-91), among others, were associated with longer BTD. The PGS of OUD and several OUD co-morbid conditions were associated with any buprenorphine prescription. A higher PGS for OUD (OR per SD increase in PGS (95%CI) = 1.43(1.16-1.77), P = 0.0009), loneliness (OR(95% CI) = 1.39(1.13-1.72), P = 0.002), problematic alcohol use (OR(95%CI) = 1.47(1.19-1.83), P = 0.0004), and externalizing disorders (OR(95%CI) = 1.52(1.23 to 1.89), P = 0.0001) was significantly associated with long-BTD. Associations between BTD and the PGS of depression, chronic pain, nicotine dependence, cannabis use disorder, and bipolar disorder did not survive correction for multiple testing. Longer BTD is associated with diagnoses of psychiatric and somatic conditions in the EHR, as is the genetic score for OUD, loneliness, problematic alcohol use, and externalizing disorders.

Provider Perceptions toward Extended-Release Buprenorphine for Treatment of Opioid Use Disorder.

OBJECTIVES: The persistence of the opioid crisis and the proliferation of synthetic fentanyl have heightened the demand for the implementation of novel delivery mechanisms of pharmacotherapy for the treatment of opioid use disorder, including injectable extended-release buprenorphine (buprenorphine-ER). The purpose of this study was to understand provider-level barriers to prescribing buprenorphine in order to facilitate targeted strategies to improve implementation for patients who would benefit from this novel formulation. METHODS: Using an interview template adapted from the Consolidated Framework for Implementation Research (CFIR), we conducted structured focus group interviews with 20 providers in an outpatient addiction clinic across 4 sessions to assess providers' perceptions of buprenorphine-ER. Ninety-four unique comments were identified and deductively coded using standardized CFIR constructs. RESULTS: Providers expressed mixed receptivity and confidence in using buprenorphine-ER. Although providers could identify a number of theoretical advantages to the injectable formulation over sublingual buprenorphine, many expressed reservations about using it due to inexperience, negative patient experiences, uncertainties about patient candidacy, cost, and logistical constraints. CONCLUSIONS: Provider concerns about buprenorphine-ER may limit utilization. Some concerns may be mitigated through improved education, research, and logistical support. Given the putative benefits of buprenorphine-ER, future research should target barriers to implementation, in part based on hypotheses generated by these findings.

Factors Associated with Acute Telemental Health Consultations in Older Veterans.

Introduction: The United States Veterans Health Administration is a leader in the use of telemental health (TMH) to enhance access to mental healthcare amidst a nationwide shortage of mental health professionals. The Tennessee Valley Veterans Affairs (VA) Health System piloted TMH in its emergency department (ED) and urgent care clinic (UCC) in 2019, with full 24/7 availability beginning March 1, 2020. Following implementation, preliminary data demonstrated that veterans ≥65 years old were less likely to receive TMH than younger patients. We sought to examine factors associated with older veterans receiving TMH consultations in acute, unscheduled, outpatient settings to identify limitations in the current process. Methods: This was a retrospective cohort study conducted within the Tennessee Valley VA Health System. We included veterans ≥55 years who received a mental health consultation in the ED or UCC from April 1, 2020-September 30, 2022. Telemental health was administered by a mental health clinician (attending physician, resident physician, nurse practitioner, or physician assistant) via iPad, whereas in-person evaluations were performed in the ED. We examined the influence of patient demographics, visit timing, chief complaint, and psychiatric history on TMH, using multivariable logistic regression. Results: Of the 254 patients included in this analysis, 177 (69.7%) received TMH. Veterans with high-risk chief complaints (suicidal ideation, homicidal ideation, or agitation) were less likely to receive TMH consultation (adjusted odds ratio [AOR]: 0.47, 95% confidence interval [CI] 0.24-0.95). Compared to attending physicians, nurse practitioners and physician assistants were associated with increased TMH use (AOR 4.81, 95% CI 2.04-11.36), whereas consultation by resident physicians was associated with decreased TMH use (AOR 0.04, 95% CI 0.00-0.59). The UCC used TMH for all but one encounter. Patient characteristics including their visit timing, gender, additional medical complaints, comorbidity burden, and number of psychoactive medications did not influence use of TMH. Conclusion: High-risk chief complaints, location, and type of mental health clinician may be key determinants of telemental health use in older adults. This may help expand mental healthcare access to areas with a shortage of mental health professionals and prevent potentially avoidable transfers in low-acuity situations. Further studies and interventions may optimize TMH for older patients to ensure safe, equitable mental health care.

Implementable Prediction of Pressure Injuries in Hospitalized Adults: Model Development and Validation.

Background: Numerous pressure injury prediction models have been developed using electronic health record data, yet hospital-acquired pressure injuries (HAPIs) are increasing, which demonstrates the critical challenge of implementing these models in routine care. Objective: To help bridge the gap between development and implementation, we sought to create a model that was feasible, broadly applicable, dynamic, actionable, and rigorously validated and then compare its performance to usual care (ie, the Braden scale). Methods: We extracted electronic health record data from 197,991 adult hospital admissions with 51 candidate features. For risk prediction and feature selection, we used logistic regression with a least absolute shrinkage and selection operator (LASSO) approach. To compare the model with usual care, we used the area under the receiver operating curve (AUC), Brier score, slope, intercept, and integrated calibration index. The model was validated using a temporally staggered cohort. Results: A total of 5458 HAPIs were identified between January 2018 and July 2022. We determined 22 features were necessary to achieve a parsimonious and highly accurate model. The top 5 features included tracheostomy, edema, central line, first albumin measure, and age. Our model achieved higher discrimination than the Braden scale (AUC 0.897, 95% CI 0.893-0.901 vs AUC 0.798, 95% CI 0.791-0.803). Conclusions: We developed and validated an accurate prediction model for HAPIs that surpassed the standard-of-care risk assessment and fulfilled necessary elements for implementation. Future work includes a pragmatic randomized trial to assess whether our model improves patient outcomes.

Leveraging natural language processing to identify eligible lung cancer screening patients with the electronic health record.

OBJECTIVE: To develop and validate an approach that identifies patients eligible for lung cancer screening (LCS) by combining structured and unstructured smoking data from the electronic health record (EHR). METHODS: We identified patients aged 50-80 years who had at least one encounter in a primary care clinic at Vanderbilt University Medical Center (VUMC) between 2019 and 2022. We fine-tuned an existing natural language processing (NLP) tool to extract quantitative smoking information using clinical notes collected from VUMC. Then, we developed an approach to identify patients who are eligible for LCS by combining smoking information from structured data and clinical narratives. We compared this method with two approaches to identify LCS eligibility only using smoking information from structured EHR. We used 50 patients with a documented history of tobacco use for comparison and validation. RESULTS: 102,475 patients were included. The NLP-based approach achieved an F1-score of 0.909, and accuracy of 0.96. The baseline approach could identify 5,887 patients. Compared to the baseline approach, the number of identified patients using all structured data and the NLP-based algorithm was 7,194 (22.2 %) and 10,231 (73.8 %), respectively. The NLP-based approach identified 589 Black/African Americans, a significant increase of 119 %. CONCLUSION: We present a feasible NLP-based approach to identify LCS eligible patients. It provides a technical basis for the development of clinical decision support tools to potentially improve the utilization of LCS and diminish healthcare disparities.

Impact of notification policy on patient-before-clinician review of immediately released test results.

The 21st Century Cures Act mandates immediate availability of test results upon request. The Cures Act does not require that patients be informed of results, but many organizations send notifications when results become available. Our medical center implemented 2 sequential policies: immediate notifications for all results, and notifications only to patients who opt in. We used over 2 years of data from Vanderbilt University Medical Center to measure the effect of these policies on rates of patient-before-clinician result review and patient-initiated messaging using interrupted time series analysis. When releasing test results with immediate notification, the proportion of patient-before-clinician review increased 4-fold and the proportion of patients who sent messages rose 3%. After transition to opt-in notifications, patient-before-clinician review decreased 2.4% and patient-initiated messaging decreased 0.4%. Replacing automated notifications with an opt-in policy provides patients flexibility to indicate their preferences but may not substantially alleviate clinicians' messaging workload.

POINT: Pipeline for Offline Conversion and Integration of Geocodes and Neighborhood Data.

OBJECTIVES: Geocoding, the process of converting addresses into precise geographic coordinates, allows researchers and health systems to obtain neighborhood-level estimates of social determinants of health. This information supports opportunities to personalize care and interventions for individual patients based on the environments where they live. We developed an integrated offline geocoding pipeline to streamline the process of obtaining address-based variables, which can be integrated into existing data processing pipelines. METHODS: POINT is a web-based, containerized, application for geocoding addresses that can be deployed offline and made available to multiple users across an organization. Our application supports use through both a graphical user interface and application programming interface to query geographic variables, by census tract, without exposing sensitive patient data. We evaluated our application's performance using two datasets: one consisting of 1 million nationally representative addresses sampled from Open Addresses, and the other consisting of 3,096 previously geocoded patient addresses. RESULTS: A total of 99.4 and 99.8% of addresses in the Open Addresses and patient addresses datasets, respectively, were geocoded successfully. Census tract assignment was concordant with reference in greater than 90% of addresses for both datasets. Among successful geocodes, median (interquartile range) distances from reference coordinates were 52.5 (26.5-119.4) and 14.5 (10.9-24.6) m for the two datasets. CONCLUSION: POINT successfully geocodes more addresses and yields similar accuracy to existing solutions, including the U.S. Census Bureau's official geocoder. Addresses are considered protected health information and cannot be shared with common online geocoding services. POINT is an offline solution that enables scalability to multiple users and integrates downstream mapping to neighborhood-level variables with a pipeline that allows users to incorporate additional datasets as they become available. As health systems and researchers continue to explore and improve health equity, it is essential to quickly and accurately obtain neighborhood variables in a Health Insurance Portability and Accountability Act (HIPAA)-compliant way.

The effect of telemental versus in-person mental health consults in the emergency department on 30-day utilization and processes of care.

OBJECTIVES: We sought to characterize how telemental health (TMH) versus in-person mental health consults affected 30-day postevaluation utilization outcomes and processes of care in Veterans presenting to the emergency department (ED) and urgent care clinic (UCC) with acute psychiatric complaints. METHODS: This exploratory retrospective cohort study was conducted in an ED and UCC located in a single Veterans Affairs system. A mental health provider administered TMH via iPad. The primary outcome was a composite of return ED/UCC visits, rehospitalizations, or death within 30 days. The following processes of care were collected during the index visit: changes to home psychiatric medications, admission, involuntary psychiatric hold placement, parenteral benzodiazepine or antipsychotic medication use, and physical restraints or seclusion. Data were abstracted from the Veterans Affairs electronic health record and the Clinical Data Warehouse. Multivariable logistic regression was performed. Adjusted odds ratios (aORs) with their 95% confidence intervals (95% CIs) were reported. RESULTS: Of the 496 Veterans in this analysis, 346 (69.8%) received TMH, and 150 (30.2%) received an in-person mental health evaluation. There was no significant difference in the primary outcome of 30-day return ED/UCC, rehospitalization, or death (aOR 1.47, 95% CI 0.87-2.49) between the TMH and in-person groups. TMH was significantly associated with increased ED/UCC length of stay (aOR 1.46, 95% CI 1.03-2.06) and decreased use of involuntary psychiatric holds (aOR 0.42, 95% CI 0.23-0.75). There were no associations between TMH and the other processes-of-care outcomes. CONCLUSIONS: TMH was not significantly associated with the 30-day composite outcome of return ED/UCC visits, rehospitalizations, and death compared with traditional in-person mental health evaluations. TMH was significantly associated with increased ED/UCC length of stay and decreased odds of placing an involuntary psychiatric hold. Future studies are required to confirm these findings and, if confirmed, explore the potential mechanisms for these associations.

Perspectives of Patients About Immediate Access to Test Results Through an Online Patient Portal.

Importance: The 21st Century Cures Act Final Rule mandates the immediate electronic availability of test results to patients, likely empowering them to better manage their health. Concerns remain about unintended effects of releasing abnormal test results to patients. Objective: To assess patient and caregiver attitudes and preferences related to receiving immediately released test results through an online patient portal. Design, Setting, and Participants: This large, multisite survey study was conducted at 4 geographically distributed academic medical centers in the US using an instrument adapted from validated surveys. The survey was delivered in May 2022 to adult patients and care partners who had accessed test results via an online patient portal account between April 5, 2021, and April 4, 2022. Exposures: Access to test results via a patient portal between April 5, 2021, and April 4, 2022. Main Outcomes and Measures: Responses to questions related to demographics, test type and result, reaction to result, notification experience and future preferences, and effect on health and well-being were aggregated. To evaluate characteristics associated with patient worry, logistic regression and pooled random-effects models were used to assess level of worry as a function of whether test results were perceived by patients as normal or not normal and whether patients were precounseled. Results: Of 43 380 surveys delivered, there were 8139 respondents (18.8%). Most respondents were female (5129 [63.0%]) and spoke English as their primary language (7690 [94.5%]). The median age was 64 years (IQR, 50-72 years). Most respondents (7520 of 7859 [95.7%]), including 2337 of 2453 individuals (95.3%) who received nonnormal results, preferred to immediately receive test results through the portal. Few respondents (411 of 5473 [7.5%]) reported that reviewing results before they were contacted by a health care practitioner increased worry, though increased worry was more common among respondents who received abnormal results (403 of 2442 [16.5%]) than those whose results were normal (294 of 5918 [5.0%]). The result of the pooled model for worry as a function of test result normality was statistically significant (odds ratio [OR], 2.71; 99% CI, 1.96-3.74), suggesting an association between worry and nonnormal results. The result of the pooled model evaluating the association between worry and precounseling was not significant (OR, 0.70; 99% CI, 0.31-1.59). Conclusions and Relevance: In this multisite survey study of patient attitudes and preferences toward receiving immediately released test results via a patient portal, most respondents preferred to receive test results via the patient portal despite viewing results prior to discussion with a health care professional. This preference persisted among patients with nonnormal results.

Using intervention mapping to design and implement a multicomponent intervention to improve antibiotic and NSAID prescribing.

Successfully changing prescribing behavior to reduce inappropriate antibiotic and nonsteroidal anti-inflammatory drug (NSAID) prescriptions often requires combining components into a multicomponent intervention. However, multicomponent interventions often fail because of development and implementation complexity. To increase the likelihood of successfully changing prescribing behavior, we applied a systematic process to design and implement a multicomponent intervention. We used Intervention Mapping to create a roadmap for a multicomponent intervention in unscheduled outpatient care settings in the Veterans Health Administration. Intervention Mapping is a systematic process consisting of six steps that we grouped into three phases: (i) understand behavioral determinants and barriers to implementation, (ii) develop the intervention, and (iii) define evaluation plan and implementation strategies. A targeted literature review, combined with 25 prescriber and 25 stakeholder interviews, helped identify key behavioral determinants to inappropriate prescribing (e.g. perceived social pressure from patients to prescribe). We targeted three desired prescriber behaviors: (i) review guideline-concordant prescribing and patient outcomes, (ii) manage diagnostic and treatment uncertainty, and (iii) educate patients and caregivers. The intervention consisted of components for academic detailing, prescribing feedback, and alternative prescription order sets. Implementation strategies consisted of preparing clinical champions, conducting readiness assessments, and incentivizing use of the intervention. We chose a mixed-method study design with a commonly used evaluation framework to assess effectiveness and implementation outcomes in a subsequent trial. This study furthers knowledge about causes of inappropriate antibiotic and NSAID prescribing and demonstrates how theoretical, empirical, and practical information can be systematically applied to develop a multicomponent intervention to help address these causes.

Reducing adverse drug events from antibiotics and nonsteroidal anti-inflammatory drugs (NSAIDs) is a patient safety priority. Successfully changing prescribing behavior to reduce inappropriate prescriptions can require combining intervention components, each with different mechanisms for behavior change, into a multicomponent intervention. However, multicomponent interventions often fail because of development and implementation complexity. To increase the chance of successfully changing antibiotic and NSAID prescribing, the objective this study was to apply a systematic process to design and implement a multicomponent intervention. Three desired prescriber behaviors were targeted: (i) review guideline-concordant prescribing and patient outcomes, (ii) manage diagnostic and treatment uncertainty, and (iii) educate patients and caregivers. The designed intervention consisted of components for prescribing feedback, academic detailing, and alternative prescription order sets. Strategies to improve use of the intervention consisted of preparing clinical champions, conducting readiness assessments prior to study onset, and incentivizing use of the intervention. We chose a mixed-method study design with a commonly used evaluation framework to assess effectiveness and implementation outcomes of the multicomponent intervention in a subsequent trial.

Rapid-cycle designs to adapt interventions for COVID-19 in safety-net healthcare systems.

Racial/ethnic minority, low socioeconomic status, and rural populations are disproportionately affected by COVID-19. Developing and evaluating interventions to address COVID-19 testing and vaccination among these populations are crucial to improving health inequities. The purpose of this paper is to describe the application of a rapid-cycle design and adaptation process from an ongoing trial to address COVID-19 among safety-net healthcare system patients. The rapid-cycle design and adaptation process included: (a) assessing context and determining relevant models/frameworks; (b) determining core and modifiable components of interventions; and (c) conducting iterative adaptations using Plan-Do-Study-Act (PDSA) cycles. PDSA cycles included: Plan. Gather information from potential adopters/implementers (e.g., Community Health Center [CHC] staff/patients) and design initial interventions; Do. Implement interventions in single CHC or patient cohort; Study. Examine process, outcome, and context data (e.g., infection rates); and, Act. If necessary, refine interventions based on process and outcome data, then disseminate interventions to other CHCs and patient cohorts. Seven CHC systems with 26 clinics participated in the trial. Rapid-cycle, PDSA-based adaptations were made to adapt to evolving COVID-19-related needs. Near real-time data used for adaptation included data on infection hot spots, CHC capacity, stakeholder priorities, local/national policies, and testing/vaccine availability. Adaptations included those to study design, intervention content, and intervention cohorts. Decision-making included multiple stakeholders (e.g., State Department of Health, Primary Care Association, CHCs, patients, researchers). Rapid-cycle designs may improve the relevance and timeliness of interventions for CHCs and other settings that provide care to populations experiencing health inequities, and for rapidly evolving healthcare challenges such as COVID-19.

Racial/ethnic minority, low socioeconomic status, and rural populations experience a disproportionate burden of COVID-19. Finding ways to address COVID-19 among these populations is crucial to improving health inequities. The purpose of this paper is to describe the rapid-cycle design process for a research project to address COVID-19 testing and vaccination among safety-net healthcare system patients. The project used real-time information on changes in COVID-19 policy (e.g., vaccination authorization), local case rates, and the capacity of safety-net healthcare systems to iteratively change interventions to ensure interventions were relevant and timely for patients. Key changes that were made to interventions included a change to the study design to include vaccination as a focus of the interventions after the vaccine was authorized; change in intervention content according to the capacity of local Community Health Centers to provide testing to patients; and changes to intervention cohorts such that priority groups of patients were selected for intervention based on characteristics including age, residency in an infection "hot spot," or race/ethnicity. Iteratively improving interventions based on real-time data collection may increase intervention relevance and timeliness, and rapid-cycle adaptions can be successfully implemented in resource constrained settings like safety-net healthcare systems.

Conceptualizing clinical decision support as complex interventions: a meta-analysis of comparative effectiveness trials.

OBJECTIVES: Complex interventions with multiple components and behavior change strategies are increasingly implemented as a form of clinical decision support (CDS) using native electronic health record functionality. Objectives of this study were, therefore, to (1) identify the proportion of randomized controlled trials with CDS interventions that were complex, (2) describe common gaps in the reporting of complexity in CDS research, and (3) determine the impact of increased complexity on CDS effectiveness. MATERIALS AND METHODS: To assess CDS complexity and identify reporting gaps for characterizing CDS interventions, we used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting tool for complex interventions. We evaluated the effect of increased complexity using random-effects meta-analysis. RESULTS: Most included studies evaluated a complex CDS intervention (76%). No studies described use of analytical frameworks or causal pathways. Two studies discussed use of theory but only one fully described the rationale and put it in context of a behavior change. A small but positive effect (standardized mean difference, 0.147; 95% CI, 0.039-0.255; P < .01) in favor of increasing intervention complexity was observed. DISCUSSION: While most CDS studies should classify interventions as complex, opportunities persist for documenting and providing resources in a manner that would enable CDS interventions to be replicated and adapted. Unless reporting of the design, implementation, and evaluation of CDS interventions improves, only slight benefits can be expected. CONCLUSION: Conceptualizing CDS as complex interventions may help convey the careful attention that is needed to ensure these interventions are contextually and theoretically informed.

The potential for leveraging machine learning to filter medication alerts.

OBJECTIVE: To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view. MATERIALS AND METHODS: We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision. RESULTS: A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001). CONCLUSIONS: Machine learning potentially enables the intelligent filtering of medication alerts.

New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record.

OBJECTIVE: To develop and test an accurate deep learning model for predicting new onset delirium in hospitalized adult patients. METHODS: Using electronic health record (EHR) data extracted from a large academic medical center, we developed a model combining long short-term memory (LSTM) and machine learning to predict new onset delirium and compared its performance with machine-learning-only models (logistic regression, random forest, support vector machine, neural network, and LightGBM). The labels of models were confusion assessment method (CAM) assessments. We evaluated models on a hold-out dataset. We calculated Shapley additive explanations (SHAP) measures to gauge the feature impact on the model. RESULTS: A total of 331 489 CAM assessments with 896 features from 34 035 patients were included. The LightGBM model achieved the best performance (AUC 0.927 [0.924, 0.929] and F1 0.626 [0.618, 0.634]) among the machine learning models. When combined with the LSTM model, the final model's performance improved significantly (P = .001) with AUC 0.952 [0.950, 0.955] and F1 0.759 [0.755, 0.765]. The precision value of the combined model improved from 0.497 to 0.751 with a fixed recall of 0.8. Using the mean absolute SHAP values, we identified the top 20 features, including age, heart rate, Richmond Agitation-Sedation Scale score, Morse fall risk score, pulse, respiratory rate, and level of care. CONCLUSION: Leveraging LSTM to capture temporal trends and combining it with the LightGBM model can significantly improve the prediction of new onset delirium, providing an algorithmic basis for the subsequent development of clinical decision support tools for proactive delirium interventions.