A pragmatic, stepped-wedge, hybrid type II trial of interoperable clinical decision support to improve venous thromboembolism prophylaxis for patients with traumatic brain injury.

BACKGROUND: Venous thromboembolism (VTE) is a preventable medical condition which has substantial impact on patient morbidity, mortality, and disability. Unfortunately, adherence to the published best practices for VTE prevention, based on patient centered outcomes research (PCOR), is highly variable across U.S. hospitals, which represents a gap between current evidence and clinical practice leading to adverse patient outcomes. This gap is especially large in the case of traumatic brain injury (TBI), where reluctance to initiate VTE prevention due to concerns for potentially increasing the rates of intracranial bleeding drives poor rates of VTE prophylaxis. This is despite research which has shown early initiation of VTE prophylaxis to be safe in TBI without increased risk of delayed neurosurgical intervention or death. Clinical decision support (CDS) is an indispensable solution to close this practice gap; however, design and implementation barriers hinder CDS adoption and successful scaling across health systems. Clinical practice guidelines (CPGs) informed by PCOR evidence can be deployed using CDS systems to improve the evidence to practice gap. In the Scaling AcceptabLE cDs (SCALED) study, we will implement a VTE prevention CPG within an interoperable CDS system and evaluate both CPG effectiveness (improved clinical outcomes) and CDS implementation. METHODS: The SCALED trial is a hybrid type 2 randomized stepped wedge effectiveness-implementation trial to scale the CDS across 4 heterogeneous healthcare systems. Trial outcomes will be assessed using the RE2-AIM planning and evaluation framework. Efforts will be made to ensure implementation consistency. Nonetheless, it is expected that CDS adoption will vary across each site. To assess these differences, we will evaluate implementation processes across trial sites using the Exploration, Preparation, Implementation, and Sustainment (EPIS) implementation framework (a determinant framework) using mixed-methods. Finally, it is critical that PCOR CPGs are maintained as evidence evolves. To date, an accepted process for evidence maintenance does not exist. We will pilot a "Living Guideline" process model for the VTE prevention CDS system. DISCUSSION: The stepped wedge hybrid type 2 trial will provide evidence regarding the effectiveness of CDS based on the Berne-Norwood criteria for VTE prevention in patients with TBI. Additionally, it will provide evidence regarding a successful strategy to scale interoperable CDS systems across U.S. healthcare systems, advancing both the fields of implementation science and health informatics. TRIAL REGISTRATION: Clinicaltrials.gov - NCT05628207. Prospectively registered 11/28/2022, https://classic. CLINICALTRIALS: gov/ct2/show/NCT05628207 .

Building to learn: Information technology innovations to enable rapid pragmatic evaluation in a learning health system.

Background: Learning health systems (LHSs) iteratively generate evidence that can be implemented into practice to improve care and produce generalizable knowledge. Pragmatic clinical trials fit well within LHSs as they combine real-world data and experiences with a degree of methodological rigor which supports generalizability. Objectives: We established a pragmatic clinical trial unit ("RapidEval") to support the development of an LHS. To further advance the field of LHS, we sought to further characterize the role of health information technology (HIT), including innovative solutions and challenges that occur, to improve LHS project delivery. Methods: During the period from December 2021 to February 2023, eight projects were selected out of 51 applications to the RapidEval program, of which five were implemented, one is currently in pilot testing, and two are in planning. We evaluated pre-study planning, implementation, analysis, and study closure approaches across all RapidEval initiatives to summarize approaches across studies and identify key innovations and learnings by gathering data from study investigators, quality staff, and IT staff, as well as RapidEval staff and leadership. Implementation Results: Implementation approaches spanned a range of HIT capabilities including interruptive alerts, clinical decision support integrated into order systems, patient navigators, embedded micro-education, targeted outpatient hand-off documentation, and patient communication. Study approaches include pre-post with time-concordant controls (1), randomized stepped-wedge (1), cluster randomized across providers (1) and location (3), and simple patient level randomization (2). Conclusions: Study selection, design, deployment, data collection, and analysis required close collaboration between data analysts, informaticists, and the RapidEval team.

The Role of Tumor Location on Endoscopic and Surgical Management of Malignant Colon Polyps.

BACKGROUND: Endoscopic polypectomy could be an appropriate, definitive treatment for pathologic T1 (pT1) colon polyps without high-risk features. Prior studies suggested worse prognosis for proximal versus distal advanced-stage colon cancers following curative treatment. However, there is limited evidence on the prognostic impact of tumor location for pT1s. PATIENTS AND METHODS: This was a retrospective cohort study using the Surveillance, Epidemiology, and End Results database to identify adults with T1NxMx or T1N0-3M0/x colon adenocarcinoma from 2000 to 2019. RESULTS: A total of 3398 patients underwent endoscopic polypectomy (17% proximal) and 28,334 had a partial colectomy (49% proximal) for pT1 adenocarcinoma. Following endoscopic polypectomy, 5-year overall and cancer-specific survival rates were 64% and 91% for proximal versus 83% and 96% for distal polyps, compared with 82% and 95% for proximal versus 88% and 97% for distal tumors after colectomy. In multivariable models, there was a greater difference in overall survival between proximal and distal polyps for those who underwent endoscopic versus surgical resection [hazard ratio (HR) 1.73, 95% confidence interval (CI) 1.49-2.02 vs. HR 1.13, 95% CI 1.08-1.18]. Patients with proximal versus distal polyps who underwent polypectomy also exhibited increased cancer-specific mortality (HR 1.94, 95% CI 1.37-2.75). However, cancer-specific survival variations based on tumor location were no longer observed in patients undergoing partial colectomy (HR 1.09, 95% CI 0.98-1.21). CONCLUSIONS: Proximal tumor location was independently associated with worse overall and cancer-specific survival following endoscopic polypectomy. However, after colectomy, the cancer-specific disparity based on tumor laterality was mitigated. These findings suggest that proximal location may be considered a high-risk feature in endoscopic polypectomy.

Consensus modeling: Safer transfer learning for small health systems.

Predictive modeling is becoming an essential tool for clinical decision support, but health systems with smaller sample sizes may construct suboptimal or overly specific models. Models become over-specific when beside true physiological effects, they also incorporate potentially volatile site-specific artifacts. These artifacts can change suddenly and can render the model unsafe. To obtain safer models, health systems with inadequate sample sizes may adopt one of the following options. First, they can use a generic model, such as one purchased from a vendor, but often such a model is not sufficiently specific to the patient population and is thus suboptimal. Second, they can participate in a research network. Paradoxically though, sites with smaller datasets contribute correspondingly less to the joint model, again rendering the final model suboptimal. Lastly, they can use transfer learning, starting from a model trained on a large data set and updating this model to the local population. This strategy can also result in a model that is over-specific. In this paper we present the consensus modeling paradigm, which uses the help of a large site (source) to reach a consensus model at the small site (target). We evaluate the approach on predicting postoperative complications at two health systems with 9,044 and 38,045 patients (rare outcomes at about 1% positive rate), and conduct a simulation study to understand the performance of consensus modeling relative to the other three approaches as a function of the available training sample size at the target site. We found that consensus modeling exhibited the least over-specificity at either the source or target site and achieved the highest combined predictive performance.

Selection and Implementation of Virtual Scribe Solutions to Reduce Documentation Burden: A Mixed Methods Pilot.

Electronic health record (EHR) documentation is a leading reason for clinician burnout. While technology-enabled solutions like virtual and digital scribes aim to improve this, there is limited evidence of their effectiveness and minimal guidance for healthcare systems around solution selection and implementation. A transdisciplinary approach, informed by clinician interviews and other considerations, was used to evaluate and select a virtual scribe solution to pilot in a rapid iterative sprint over 12 weeks. Surveys, interviews, and EHR metadata were analyzed over a staggered 30 day implementation with live and asynchronous virtual scribe solutions. Among 16 pilot clinicians, documentation burden metrics decreased for some but not all. Some clinicians had highly positive comments, and others had concerns regarding scribe training and quality. Our findings demonstrate that virtual scribes may reduce documentation burden for some clinicians and describe a method for a collaborative and iterative technology selection process for digital tools in practice.

Exploring Large Language Models for Acronym, Symbol Sense Disambiguation, and Semantic Similarity and Relatedness Assessment.

Acronyms, abbreviations, and symbols play a significant role in clinical notes. Acronym and symbol sense disambiguation are crucial natural language processing (NLP) tasks that ensure the clarity and consistency of clinical notes and downstream NLP processing. Previous studies using traditional machine learning methods have been relatively successful in tackling this issue. In our research, we conducted an evaluation of large language models (LLMs), including ChatGPT 3.5 and 4, as well as other open LLMs, and BERT-based models, across three NLP tasks: acronym and symbol sense disambiguation, semantic similarity, and relatedness. Our findings emphasize ChatGPT's remarkable ability to distinguish between senses with minimal or zero-shot training. Additionally, open source LLM Mixtrial-8x7B exhibited high accuracy for acronyms with fewer senses, and moderate accuracy for symbol sense accuracy. BERT-based models outperformed previous machine learning approaches, achieving an impressive accuracy rate of over 95%, showcasing their effectiveness in addressing the challenge of acronym and symbol sense disambiguation. Furthermore, ChatGPT exhibited a strong correlation, surpassing 70%, with human gold standards when evaluating similarity and relatedness.

Evaluation of a Clinical Decision Support Tool to Guide Adoption of the American Heart Association Telemetry Monitoring Practice Standards.

BACKGROUND: The objectives of this study were to (1) evaluate telemetry use pre- and postimplementation of clinical decision support tools to support American Heart Association practice standards for telemetry monitoring and (2) understand the factors that may contribute to variation of telemetry monitoring in practice. METHODS AND RESULTS: First, we captured overall variability in telemetry use pre- and postimplementation of the clinical decision support intervention. We then conducted semistructured interviews with telemetry-ordering providers to identify key barriers and facilitators to adoption. During the study period, 399 physicians met criteria for inclusion and were divided into excessive and nonexcessive orderers. Distribution of telemetry use was bimodal. Among nonexcessive users, 24.4% of patient days were with telemetry compared with 51.6% among excessive users. On average, both excessive (6.1% reduction) and nonexcessive users (2.8% reduction) decreased telemetry use postimplementation, and these reductions were sustained over a 16-month period. Sixteen interviews were conducted. Physicians believed that the tool was successful because it caused them to more closely consider if telemetry was indicated for each patient. Physicians also voiced frustration with interruptions to their workflow, and some noted that they commonly use telemetry outside of practice standards to monitor patients who were acutely but not critically ill. CONCLUSIONS: Embedding telemetry practice standards into the electronic health record in the form of clinical decision support is effective at reducing excess telemetry use. Although the intervention was well received, there are persistent barriers, such as preexisting views on telemetry and existing workflow habits, that may inhibit higher adoption of standards.

Implementing Telestroke in the Inpatient Setting: Identifying Factors for Success.

BACKGROUND: Inpatient telestroke programs have emerged as a solution to provide timely stroke care in underserved areas, but their successful implementation and factors influencing their effectiveness remain underexplored. This study aimed to qualitatively evaluate the perspectives of inpatient clinicians located at spoke hospitals participating in a newly established inpatient telestroke program to identify implementation barriers and facilitators. METHODS: This was a formative evaluation relying on semistructured qualitative interviews with 16 inpatient providers (physicians and nurse practitioners) at 5 spoke sites of a hub-and-spoke inpatient telestroke program. The Integrated-Promoting Action on Research Implementation in Health Services framework guided data analysis, focusing on the innovation, recipients, context, and facilitation aspects of implementation. Interviews were transcribed and coded using thematic analysis. RESULTS: Fifteen themes were identified in the data and mapped to the Integrated-Promoting Action on Research Implementation in Health Services framework. Themes related to the innovation (the telestroke program) included easy access to stroke specialists, the benefits of limiting patient transfers, concerns about duplicating tests, and challenges of timing inpatient telestroke visits and notes to align with discharge workflow. Themes pertaining to recipients (care team members and patients) were communication gaps between teams, concern about the supervision of inpatient telestroke advanced practice providers and challenges with nurse empowerment. With regard to the context (hospital and system factors), providers highlighted familiarity with telehealth technologies as a facilitator to implementing inpatient telestroke, yet highlighted resource limitations in smaller facilities. Facilitation (program implementation) was recognized as crucial for education, standardization, and buy-in. CONCLUSIONS: Understanding barriers and facilitators to implementation is crucial to determining where programmatic changes may need to be made to ensure the success and sustainment of inpatient telestroke services.

A population-based analysis on the incidence of metachronous colon cancer after endoscopic resection of advanced adenomas with high-grade dysplasia: does location matter?

BACKGROUND: Advanced adenomas (AAs) with high-grade dysplasia (HGD) represent a risk factor for metachronous neoplasia, with guidelines recommending short-interval surveillance. Although the worse prognosis of proximal (vs distal) colon cancers (CCs) is established, there is paucity of evidence on the impact of laterality on the risk of subsequent neoplasia for these AAs. METHODS: Adults with HGD adenomas undergoing polypectomy were identified in the Surveillance, Epidemiology, and End Results database (2000-2019). Cumulative incidence of malignancy was estimated using the Kaplan-Meier method. Fine-Gray models assessed the effect of patient and disease characteristics on CC incidence. RESULTS: Of 3199 patients, 26% had proximal AAs. A total of 65 cases of metachronous adenocarcinoma were identified after polypectomy of 35 proximal and 30 distal adenomas with HGD. The 10-year cumulative incidence of CC was 2.3%; when stratified by location, it was 4.8% for proximal vs 1.4% for distal adenomas. Proximal location was significantly associated with increased incidence of metachronous cancer (adjusted hazard ratio, 3.32; 95% CI, 2.05-5.38). CONCLUSION: Proximal location of AAs with HGD was associated with >3-fold increased incidence of metachronous CC and shorter time to diagnosis. These data suggest laterality should be considered in the treatment and follow-up of these patients.

Decreasing Opioid Addiction and Diversion Using Behavioral Economics Applied Through a Digital Engagement Solution: Protocol for a Randomized Controlled Trial.

BACKGROUND: Despite strong and growing interest in ending the ongoing opioid health crisis, there has been limited success in reducing the prevalence of opioid addiction and the number of deaths associated with opioid overdoses. Further, 1 explanation for this is that existing interventions target those who are opiate-dependent but do not prevent opioid-naïve patients from becoming addicted. OBJECTIVE: Leveraging behavioral economics at the patient level could help patients successfully use, discontinue, and dispose of their opioid medications in an acute pain setting. The primary goal of this project is to evaluate the effect of the 3 versions of the Opioid Management for You (OPY) tool on measures of opioid use relative to the standard of care by leveraging a pragmatic randomized controlled trial (RCT). METHODS: A team of researchers from the Center for Learning Health System Sciences (CLHSS) at the University of Minnesota partnered with M Health Fairview to design, build, and test the 3 versions of the OPY tool: social influence, precommitment, and testimonial version. The tool is being built using the Epic Care Companion (Epic Inc) platform and interacts with the patient through their existing MyChart (Epic Systems Corporation) personal health record account, and Epic patient portal, accessed through a phone app or the MyChart website. We have demonstrated feasibility with pilot data of the social influence version of the OPY app by targeting our pilot to a specific cohort of patients undergoing upper-extremity procedures. This study will use a group sequential RCT design to test the impact of this important health system initiative. Patients who meet OPY inclusion criteria will be stratified into low, intermediate, and high risk of opiate use based on their type of surgery. RESULTS: This study is being funded and supported by the CLHSS Rapid Prospective Evaluation and Digital Technology Innovation Programs, and M Health Fairview. Support and coordination provided by CLHSS include the structure of engagement, survey development, data collection, statistical analysis, and dissemination. The project was initially started in August 2022. The pilot was launched in February 2023 and is still running, with the data last counted in August 2023. The actual RCT is planned to start by early 2024. CONCLUSIONS: Through this RCT, we will test our hypothesis that patient opioid use and diverted prescription opioid availability can both be improved by information delivery applied through a behavioral economics lens via sending nudges directly to the opioid users through their personal health record. TRIAL REGISTRATION: ClinicalTrials.gov NCT06124079; https://clinicaltrials.gov/study/NCT06124079. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/52882.

A Symptom-Based Natural Language Processing Surveillance Pipeline for Post-COVID-19 Patients.

Post-acute sequelae of SARS CoV-2 (PASC) are a group of conditions in which patients previously infected with COVID-19 experience symptoms weeks/months post-infection. PASC has substantial societal burden, including increased healthcare costs and disabilities. This study presents a natural language processing (NLP) based pipeline for identification of PASC symptoms and demonstrates its ability to estimate the proportion of suspected PASC cases. A manual case review to obtain this estimate indicated our sample incidence of PASC (13%) was representative of the estimated population proportion (95% CI: 19±6.22%). However, the high number of cases classified as indeterminate demonstrates the challenges in classifying PASC even among experienced clinicians. Lastly, this study developed a dashboard to display views of aggregated PASC symptoms and measured its utility using the System Usability Scale. Overall comments related to the dashboard's potential were positive. This pipeline is crucial for monitoring post-COVID-19 patients with potential for use in clinical settings.

Evolving Interoperability Across a State Public Health Immunization Registry and Electronic Health Records.

The critical need for system interoperability and robust information infrastructure in public health was highlighted during the COVID-19 pandemic. An assessment of the evolving interoperability between immunization information system (IIS) in a state-based public health agency and electronic health records (EHRs) including pandemic-driven evolution/use was conducted. The Minnesota Immunization Information Connection (MIIC), the IIS for Minnesota (US) supports interoperability with EHRs using HL7v2.5.1 standards-based queries. Structured interviews were conducted with 28 experts across 12 healthcare systems and public health clinics (n=286 sites) between April - July 2022. Though all reported use of MIIC, most (83%) had MIIC integration within their EHRs, and high EHR queries to MIIC (∼6 million/month), numerous organizational/technical barriers were identified including standard vaccine-naming need in EHRs, app access issues, limited resources and informatics-staff shortage in public health. Results underscore vital role of IIS, on-going interoperability evaluation to address issues and promote standards-based bi-directional EHR-IIS data exchanges.

Promoting Learning Health System Cycles by Optimizing EHR Data Clinical Concept Encoding Processes.

Electronic health records (EHRs) and other real-world data (RWD) are critical to accelerating and scaling care improvement and transformation. To efficiently leverage it for secondary uses, EHR/RWD should be optimally managed and mapped to industry standard concepts (ISCs). Inherent challenges in concept encoding usually result in inefficient and costly workflows and resultant metadata representation structures outside the EHR. Using three related projects to map data to ISCs, we describe the development of standard, repeatable processes for precisely and unambiguously representing EHR data using appropriate ISCs within the EHR platform lifecycle and mappings specific to SNOMED-CT for Demographics, Specialty and Services. Mappings in these 3 areas resulted in ISC mappings of 779 data elements requiring 90 new concept requests to SNOMED-CT and 738 new ISCs mapped into the workflow within an accessible, enterprise-wide EHR resource with supporting processes.

Machine Learning for Risk Prediction of Recurrent AKI in Adult Patients After Hospital Discharge.

Recurrent AKI has been found common among hospitalized patients after discharge, and early prediction may allow timely intervention and optimized post-discharge treatment [1]. There are significant gaps in the literature regarding the risk prediction on the post-AKI population, and most current works only included a limited number of pre-selected variables [2]. In this study, we built and compared machine learning models using both knowledge-based and data-driven features in predicting the risk of recurrent AKI within 1-year of discharge. Our results showed that the additional use of data-driven features statistically improved the model performances, with best AUC=0.766 by using logistic regression.

Post-Implementation Outcomes of a Remote Patient Monitoring Program After Emergency Department Discharge.

To better communicate and improve post-visit outcomes, a remote patient monitoring (RPM) program was implemented for patients discharged from emergency departments (ED) across 10 hospitals. The solution was offered to patients at the time of ED discharge and staffed by a group of care coordinators to respond to questions/urgent needs. Of 107,477 consecutive patients offered RPM, 28,425 patients (26.4%) engaged with the program. Activated patients with RPM were less likely to return to the ED within 90 days of their index visit [19.8% compared to 23.6%, p<.001]. While activation rates were modest, we observed fewer return visits to the ED in patients using RPM, with a 16.2% lower hazard of returning in the next year. Future research is needed to understand methods to improve RPM activation, any causal effects of RPM activation on return ED visits, and external validation of these findings.

Advanced Care Planning Content Encoding with Natural Language Processing.

While advanced care planning (ACP) is an essential practice for ensuring patient-centered care, its adoption remains poor and the completeness of its documentation variable. Natural language processing (NLP) approaches hold promise for supporting ACP, including its use for decision support to improve ACP gaps at the point of care. ACP themes were annotated on palliative care notes across four annotators (Fleiss kappa = 0.753) and supervised models trained (Huggingface models bert-base-uncased and Bio_ClinicalBERT) using 5-fold cross validation (F1=0.8, precision=0.75, recall=0.86, any theme). When applied across the full note corpus of 12,711 notes, we observed variability in documentation of ACP information. Our findings demonstrate the promise of NLP approaches for informatics-based approaches for ACP and patient-centered care.

Development and Usability Evaluation of an Opioid Management App.

We describe the development and usability evaluation of a novel patient engagement tool (OPY) in its early stage from perspectives of both experts and end-users. The tool is aimed at engaging patients in positive behaviors surrounding the use, weaning, and disposal of opioid medications in the post-surgical setting. The messaging and design of the application were created through a behavioral economics lens. Expert-based heuristic analysis and user testing were conducted and demonstrated that while patients found the tool to be easy to use and subjectively somewhat useful, additional work to enhance the user interface and features is needed in close partnership with developers and stakeholders.

Large Language Model-Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures.

Importance: Informed consent is a critical component of patient care before invasive procedures, yet it is frequently inadequate. Electronic consent forms have the potential to facilitate patient comprehension if they provide information that is readable, accurate, and complete; it is not known if large language model (LLM)-based chatbots may improve informed consent documentation by generating accurate and complete information that is easily understood by patients. Objective: To compare the readability, accuracy, and completeness of LLM-based chatbot- vs surgeon-generated information on the risks, benefits, and alternatives (RBAs) of common surgical procedures. Design, Setting, and Participants: This cross-sectional study compared randomly selected surgeon-generated RBAs used in signed electronic consent forms at an academic referral center in San Francisco with LLM-based chatbot-generated (ChatGPT-3.5, OpenAI) RBAs for 6 surgical procedures (colectomy, coronary artery bypass graft, laparoscopic cholecystectomy, inguinal hernia repair, knee arthroplasty, and spinal fusion). Main Outcomes and Measures: Readability was measured using previously validated scales (Flesh-Kincaid grade level, Gunning Fog index, the Simple Measure of Gobbledygook, and the Coleman-Liau index). Scores range from 0 to greater than 20 to indicate the years of education required to understand a text. Accuracy and completeness were assessed using a rubric developed with recommendations from LeapFrog, the Joint Commission, and the American College of Surgeons. Both composite and RBA subgroup scores were compared. Results: The total sample consisted of 36 RBAs, with 1 RBA generated by the LLM-based chatbot and 5 RBAs generated by a surgeon for each of the 6 surgical procedures. The mean (SD) readability score for the LLM-based chatbot RBAs was 12.9 (2.0) vs 15.7 (4.0) for surgeon-generated RBAs (P = .10). The mean (SD) composite completeness and accuracy score was lower for surgeons' RBAs at 1.6 (0.5) than for LLM-based chatbot RBAs at 2.2 (0.4) (P < .001). The LLM-based chatbot scores were higher than the surgeon-generated scores for descriptions of the benefits of surgery (2.3 [0.7] vs 1.4 [0.7]; P < .001) and alternatives to surgery (2.7 [0.5] vs 1.4 [0.7]; P < .001). There was no significant difference in chatbot vs surgeon RBA scores for risks of surgery (1.7 [0.5] vs 1.7 [0.4]; P = .38). Conclusions and Relevance: The findings of this cross-sectional study suggest that despite not being perfect, LLM-based chatbots have the potential to enhance informed consent documentation. If an LLM were embedded in electronic health records in a manner compliant with the Health Insurance Portability and Accountability Act, it could be used to provide personalized risk information while easing documentation burden for physicians.