Design and implementation of electronic health record-based tools to support a weight management program in primary care.

Objectives: This paper reports on a mixed methods formative evaluation to support the design and implementation of information technology (IT) tools for a primary care weight management intervention delivered through the patient portal using primary care staff as coaches. Methods: We performed a qualitative needs assessment, designed the IT tools to support the weight management program, and developed implementation tracking metrics. Implementation tracking metrics were designed to use real world electronic health record (EHR) data. Results: The needs assessment revealed IT requirements as well as barriers and facilitators to implementation of EHR-based weight management interventions in primary care. We developed implementation metrics for the IT tools. These metrics were used in weekly project team calls to make sure that project resources were allocated to areas of need. Conclusion: This study identifies the important role of IT in supporting weight management through patient identification, weight and activity tracking in the patient portal, and the use of the EHR as a population management tool. An intensive multi-level implementation approach is required for successful primary care-based weight management interventions including well-designed IT tools, comprehensive involvement of clinic leadership, and implementation tracking metrics to guide the process of workflow integration. This study helps to bridge the gap between informatics and implementation by using socio-technical formative evaluation methods early in order to support the implementation of IT tools. Trial registration: clinicaltrials.gov, NCT04420936. Registered June 9, 2020.

Implementing weight maintenance with existing staff and electronic health record tools in a primary care setting: Baseline results from the MAINTAIN PRIME trial.

BACKGROUND: There is need for interventions that can assist with long-term maintenance of healthy body weight and be sustainably integrated into existing primary care teams. The goal of MAINTAIN PRIME (Promoting Real (World) IMplEmentation) is to evaluate whether a successful electronic health record (EHR)-based weight maintenance intervention can be adapted to a new clinical setting with primary care staff serving as coaches. METHODS: EHR tools include tracking tools, standardized surveys, and standardized "SmartPhrases" for coaching. Inclusion criteria were age 18-75 years, voluntary 5% weight loss in the past 2 years with prior BMI ≥ 25 kg/m2, and no bariatric procedures in past 2 years. Participants were randomized 1:1 to tailored online coaching with EHR tracking tools (coaching) or EHR tracking tools alone (tracking). RESULTS: We screened 405 individuals between September 2021 and April 2023; 269 participants enrolled (134 coaching; 135 tracking). The most common reason for not enrolling was ineligibility (55%). At baseline, participants were 50.3 (SD 15.02) years old, 66.4% female, and 84% White; 83.7% reported moderate physical activity. Average weight and BMI at baseline were 205.0 (SD 48.9) lbs. and 33.2 (6.8) kg/m2, respectively. Participants lost an average of 10.7% (SD 5.2) of their body weight before enrolling. We recruited 39 primary care coaches over the same period. Conclusion The study successfully identified and recruited primary care patients with recent intentional weight loss for participation in a weight maintenance program that uses EHR-based tools. We also successfully recruited and trained primary care staff as coaches.

One Health: Insights from Organizational & Social, Technology Assessment and Human Factors Perspectives.

OBJECTIVES: To offer diverse but complementary perspectives on how biomedical and health informatics can be informed by and help to achieve the vision of One Health. METHODS: Overview of key considerations and critical discussion of common themes, barriers and opportunities, based on collaborative review by International Medical Informatics Association (IMIA) working group members active in related fields. RESULTS: Health and care systems are complex sociotechnical systems that need explicit design and implementation strategies to align with the goals of One Health. The evidence-based health informatics paradigm and associated frameworks for evaluation of digital health technologies need to broaden their scope to take full account of the One Health approach. Informatics has specific contributions to make to One Health, for example by improved user experience reducing energy consumption and effective app design enhancing medication adherence. CONCLUSIONS: One Health is inherently intertwined with ergonomic, sociotechnical and evaluation perspectives in biomedical and health informatics. Health is a planetary issue that requires interdisciplinary collaborative action. The theories and principles of biomedical and health informatics offer many opportunities to transform digital health technology to better serve the One Health agenda.

Considerations for using predictive models that include race as an input variable: The case study of lung cancer screening.

Indiscriminate use of predictive models incorporating race can reinforce biases present in source data and lead to an exacerbation of health disparities. In some countries, such as the United States, there is therefore a push to remove race from prediction models; however, there are still many prediction models that use race as an input. Biomedical informaticists who are given the responsibility of using these predictive models in healthcare environments are likely to be faced with questions like how to deal with race covariates in these models. Thus, there is a need for a pragmatic framework to help model users think through how to include race in their chosen model so as to avoid inadvertently exacerbating disparities. In this paper, we use the case study of lung cancer screening to propose a simple framework to guide how model users can approach the use (or non-use) of race inputs in the predictive models they are tasked with leveraging in electronic health records and clinical workflows.

The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Implementation of Lung Cancer Screening in Primary Care and Pulmonary Clinics: Pragmatic Clinical Trial of Electronic Health Record-Integrated Everyday Shared Decision-Making Tool and Clinician-Facing Prompts.

BACKGROUND: Although low-dose CT (LDCT) scan imaging lung cancer screening (LCS) can reduce lung cancer mortality, it remains underused. Shared decision-making (SDM) is recommended to assess the balance of benefits and harms for each patient. RESEARCH QUESTION: Do clinician-facing electronic health record (EHR) prompts and an EHR-integrated everyday SDM tool designed to support routine incorporation of SDM into primary care improve LDCT scan imaging ordering and completion? STUDY DESIGN AND METHODS: A preintervention and postintervention analysis was conducted in 30 primary care and four pulmonary clinics for visits with patients who met United States Preventive Services Task Force criteria for LCS. Propensity scores were used to adjust for covariates. Subgroup analyses were conducted based on the expected benefit from screening (high benefit vs intermediate benefit), pulmonologist involvement (ie, whether the patient was seen in a pulmonary clinic in addition to a primary care clinic), sex, and race and ethnicity. RESULTS: In the 12-month preintervention phase among 1,090 eligible patients, 77 patients (7.1%) had LDCT scan imaging orders and 48 patients (4.4%) completed screenings. In the 9-month intervention phase among 1,026 eligible patients, 280 patients (27.3%) had LDCT scan imaging orders and 182 patients (17.7%) completed screenings. Adjusted ORs were 4.9 (95% CI, 3.4-6.9; P < .001) and 4.7 (95% CI, 3.1-7.1; P < .001) for LDCT imaging ordering and completion, respectively. Subgroup analyses showed increases in ordering and completion for all patient subgroups. In the intervention phase, the SDM tool was used by 23 of 102 ordering providers (22.5%) and for 69 of 274 patients (25.2%) for whom LDCT scan imaging was ordered and who needed SDM at the time of ordering. INTERPRETATION: Clinician-facing EHR prompts and an EHR-integrated everyday SDM tool are promising approaches to improving LCS in the primary care setting. However, room for improvement remains. As such, further research is warranted. TRIAL REGISTRY: ClinicalTrials.gov; No.: NCT04498052; URL: www. CLINICALTRIALS: gov.

Impact of electronic health record-coaching features in weight change: A secondary analysis from the MAINTAIN-pc randomized trial.

OBJECTIVE: This study aimed to evaluate whether coaching features were successfully transmitted via electronic health record (EHR) communication and to evaluate their relationship with weight change in a previously tested EHR-based coaching intervention. METHODS: A secondary analysis from the Maintaining Activity and Nutrition through Technology-Assisted Innovation in Primary Care (MAINTAIN-pc) study randomized clinical trial was conducted in nine primary care practices and one specialty practice (endocrinology) affiliated with the University of Pittsburgh Medical Center. Eligibility included age 18 to 75 years, intentional 5% weight loss in the previous 2 years, access to an internet-connected computer, and receipt of care from a University of Pittsburgh Medical Center primary care provider. Survey data were collected during the randomized clinical trial. RESULTS: Participants content with intervention delivery via the EHR and those who felt a strong connection to their coach had significantly less weight regain (p = 0.013 and p < 0.01, respectively). Participants who had needs unmet by the intervention (e.g., "in-person" support in a group setting or individual settings) regained more weight (p < 0.01). CONCLUSIONS: The data suggest heterogeneity in the patient population regarding preference for in-person versus EHR-based coaching formats. Such heterogeneity might result in the differential success of EHR-based interventions.

Electronic Health Record-Integrated Clinical Decision Support for Clinicians Serving Populations Facing Health Care Disparities: Literature Review.

OBJECTIVES: To review current studies about designing and implementing clinician-facing clinical decision support (CDS) integrated or interoperable with an electronic health record (EHR) to improve health care for populations facing disparities. METHODS: We searched PubMed to identify studies published between January 1, 2011 and October 22, 2021 about clinician-facing CDS integrated or interoperable with an EHR. We screened abstracts and titles and extracted study data from articles using a protocol developed by team consensus. Extracted data included patient population characteristics, clinical specialty, setting, EHR, clinical problem, CDS type, reported user-centered design, implementation strategies, and outcomes. RESULTS: There were 28 studies (36 articles) included. Most studies were performed at safety net institutions (14 studies) or Indian Health Service sites (6 studies). CDS tools were implemented in primary care outpatient settings in 24 studies (86%) for screening or treatment. CDS included point-of-care alerts (93%), order facilitators (46%), workflow support (39%), relevant information display (36%), expert systems (11%), and medication dosing support (7%). Successful outcomes were reported in 19 of 26 studies that reported outcomes (73%). User-centered design was reported during CDS planning (39%), development (32%), and implementation phase (25%). Most frequent implementation strategies were education (89%) and consensus facilitation (50%). CONCLUSIONS: CDS tools may improve health equity and outcomes for patients who face disparities. The present review underscores the need for high-quality analyses of CDS-associated health outcomes, reporting of user-centered design and implementation strategies used in low-resource settings, and methods to disseminate CDS created to improve health equity.

Assessing perceived effectiveness of career development efforts led by the women in American Medical Informatics Association Initiative.

OBJECTIVE: We sought to ascertain perceived factors affecting women's career development efforts in the American Medical Informatics Association (AMIA) and to provide recommendations for improvements. MATERIALS AND METHODS: Data were collected using a 27-item survey administered via the AMIA newsletter and other social channels. Survey questions comprised 3 demographics, 15 Likert-scale, and 9 open-ended items. Likert-scale responses were summarized across respondent ages, career stages, and career domains, and open-ended responses were thematically analyzed. RESULTS: We received survey responses from 109 AMIA women members. Our findings demonstrate that AMIA had made strides in promoting career development, and the most effective AMIA efforts included social events (83%), panel discussions (80%), and scientific sessions (79%). However, despite these efforts, women members perceived that gender-specific challenges persisted within AMIA, and recognized the need for increased networking opportunities (96%), raising awareness of gender-specific challenges (95%), and encouraging gender proportional representation in leadership (92%). DISCUSSION: International and national biomedical informatics professional communities have put forth efforts to address gender-specific issues in career development. Yet, our study identified that some of these, including the deep-rooted gender power hierarchy and bias, are still perceived as profound in AMIA. CONCLUSION: Even though existing career development efforts for women are highly effective, important perceived gender-specific career development issues require further attention and investigation to improve existing AMIA activities.

The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals.

OBJECTIVES: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them. METHODS: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings. RESULTS: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time. CONCLUSIONS: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Association of Mobile Workstations and Rounding-in-Flow with Resident Efficiency: A Controlled Study at an Academic Internal Medicine Department.

BACKGROUND: Residents commonly use a batched workflow to round on hospitalized patients, creating redundancy and decreasing efficiency. OBJECTIVE: To improve resident efficiency through a novel workflow using mobile laptops and modified rounding-in-flow. DESIGN, SETTING, PARTICIPANTS: A controlled experimental study conducted at an academic medical center for 3 months. One internal medicine team served as the intervention group, and two other teams served as a control group; 34 interns and 20 senior residents participated. INTERVENTION: Residents in the intervention group were provided a novel workflow and a mobile laptop to allow them to round "in-flow." Control group residents rounded as usual (batched workflow without laptops). MAIN MEASURES: Fourteen interns were monitored for a time-motion study. Time-stamped electronic medical record (EMR) data were used to assess percentage of progress notes and orders placed during rounds (9 a.m.-12:30 p.m.) and percentage of discharge summaries signed within 24 h of discharge. A post-intervention survey measured perceived efficiency. RESULTS: A time-motion study showed non-significant differences between time in the intervention group and that in the control group: communication time with patients (128 min vs 105 min, p = 0.37) and computer time (289 min vs 306 min, p = 0.71). EMR data for 664 visits in the control group and 374 in the intervention group showed that rounding-in-flow was associated with an odds ratio (OR) of 1.5 for placing progress notes during rounds (95% CI: 1.2-1.7, p < 0.001), an OR of 1.1 for placing non-discharge orders during rounds (95% CI: 1.0-1.2, p = 0.01), and an OR of 3.9 for signing discharge summaries within 24 h of discharge (95% CI: 2.3-7.2, p < 0.001). Post-intervention survey, completed by 23 of 34 interns, showed that interns in the intervention group perceived that orders were completed during rounds more often than the control group (OR 7.8; 95% CI: 1.3-60.1, p = 0.03). CONCLUSIONS: Using mobile laptops with modified rounding-in-flow was associated with earlier completion of residents' work, suggesting improved efficiency.

Using CDS Hooks to increase SMART on FHIR app utilization: a cluster-randomized trial.

OBJECTIVE: HL7 SMART on FHIR apps have the potential to improve healthcare delivery and EHR usability, but providers must be aware of the apps and use them for these potential benefits to be realized. The HL7 CDS Hooks standard was developed in part for this purpose. The objective of this study was to determine if contextually relevant CDS Hooks prompts can increase utilization of a SMART on FHIR medical reference app (MDCalc for EHR). MATERIALS AND METHODS: We conducted a 7-month, provider-randomized trial with 70 providers in a single emergency department. The intervention was a collection of CDS Hooks prompts suggesting the use of 6 medical calculators in a SMART on FHIR medical reference app. The primary outcome was the percentage of provider-patient interactions in which the app was used to view a recommended calculator. Secondary outcomes were app usage stratified by individual calculators. RESULTS: Intervention group providers viewed a study calculator in the app in 6.0% of interactions compared to 2.6% in the control group (odds ratio = 2.45, 95% CI, 1.2-5.2, P value .02), an increase of 130%. App use was significantly greater for 2 of 6 calculators. DISCUSSION AND CONCLUSION: Contextually relevant CDS Hooks prompts led to a significant increase in SMART on FHIR app utilization. This demonstrates the potential of using CDS Hooks to guide appropriate use of SMART on FHIR apps and was a primary motivation for the development of the standard. Future research may evaluate potential impacts on clinical care decisions and outcomes.

Inaccuracies in electronic health records smoking data and a potential approach to address resulting underestimation in determining lung cancer screening eligibility.

OBJECTIVE: The US Preventive Services Task Force (USPSTF) requires the estimation of lifetime pack-years to determine lung cancer screening eligibility. Leading electronic health record (EHR) vendors calculate pack-years using only the most recently recorded smoking data. The objective was to characterize EHR smoking data issues and to propose an approach to addressing these issues using longitudinal smoking data. MATERIALS AND METHODS: In this cross-sectional study, we evaluated 16 874 current or former smokers who met USPSTF age criteria for screening (50-80 years old), had no prior lung cancer diagnosis, and were seen in 2020 at an academic health system using the Epic® EHR. We described and quantified issues in the smoking data. We then estimated how many additional potentially eligible patients could be identified using longitudinal data. The approach was verified through manual review of records from 100 subjects. RESULTS: Over 80% of evaluated records had inaccuracies, including missing packs-per-day or years-smoked (42.7%), outdated data (25.1%), missing years-quit (17.4%), and a recent change in packs-per-day resulting in inaccurate lifetime pack-years estimation (16.9%). Addressing these issues by using longitudinal data enabled the identification of 49.4% more patients potentially eligible for lung cancer screening (P < .001). DISCUSSION: Missing, outdated, and inaccurate smoking data in the EHR are important barriers to effective lung cancer screening. Data collection and analysis strategies that reflect changes in smoking habits over time could improve the identification of patients eligible for screening. CONCLUSION: The use of longitudinal EHR smoking data could improve lung cancer screening.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Implementing lung cancer screening in primary care: needs assessment and implementation strategy design.

Lung cancer screening with low-dose computed tomography (CT) could help avert thousands of deaths each year. Since the implementation of screening is complex and underspecified, there is a need for systematic and theory-based strategies. Explore the implementation of lung cancer screening in primary care, in the context of integrating a decision aid into the electronic health record. Design implementation strategies that target hypothesized mechanisms of change and context-specific barriers. The study had two phases. The Qualitative Analysis phase included semi-structured interviews with primary care physicians to elicit key task behaviors (e.g., ordering a low-dose CT) and understand the underlying behavioral determinants (e.g., social influence). The Implementation Strategy Design phase consisted of defining implementation strategies and hypothesizing causal pathways to improve screening with a decision aid. Three key task behaviors and four behavioral determinants emerged from 14 interviews. Implementation strategies were designed to target multiple levels of influence. Strategies included increasing provider self-efficacy toward performing shared decision making and using the decision aid, improving provider performance expectancy toward ordering a low-dose CT, increasing social influence toward performing shared decision making and using the decision aid, and addressing key facilitators to using the decision aid. This study contributes knowledge about theoretical determinants of key task behaviors associated with lung cancer screening. We designed implementation strategies according to causal pathways that can be replicated and tested at other institutions. Future research is needed to evaluate the effectiveness of these strategies and to determine the contexts in which they can be effectively applied.

Establishing a multidisciplinary initiative for interoperable electronic health record innovations at an academic medical center.

OBJECTIVE: To establish an enterprise initiative for improving health and health care through interoperable electronic health record (EHR) innovations. MATERIALS AND METHODS: We developed a unifying mission and vision, established multidisciplinary governance, and formulated a strategic plan. Key elements of our strategy include establishing a world-class team; creating shared infrastructure to support individual innovations; developing and implementing innovations with high anticipated impact and a clear path to adoption; incorporating best practices such as the use of Fast Healthcare Interoperability Resources (FHIR) and related interoperability standards; and maximizing synergies across research and operations and with partner organizations. RESULTS: University of Utah Health launched the ReImagine EHR initiative in 2016. Supportive infrastructure developed by the initiative include various FHIR-related tooling and a systematic evaluation framework. More than 10 EHR-integrated digital innovations have been implemented to support preventive care, shared decision-making, chronic disease management, and acute clinical care. Initial evaluations of these innovations have demonstrated positive impact on user satisfaction, provider efficiency, and compliance with evidence-based guidelines. Return on investment has included improvements in care; over $35 million in external grant funding; commercial opportunities; and increased ability to adapt to a changing healthcare landscape. DISCUSSION: Key lessons learned include the value of investing in digital innovation initiatives leveraging FHIR; the importance of supportive infrastructure for accelerating innovation; and the critical role of user-centered design, implementation science, and evaluation. CONCLUSION: EHR-integrated digital innovation initiatives can be key assets for enhancing the EHR user experience, improving patient care, and reducing provider burnout.

Leveraging Artificial Intelligence to Improve Chronic Disease Care: Methods and Application to Pharmacotherapy Decision Support for Type-2 Diabetes Mellitus.

OBJECTIVES: Artificial intelligence (AI), including predictive analytics, has great potential to improve the care of common chronic conditions with high morbidity and mortality. However, there are still many challenges to achieving this vision. The goal of this project was to develop and apply methods for enhancing chronic disease care using AI. METHODS: Using a dataset of 27,904 patients with diabetes, an analytical method was developed and validated for generating a treatment pathway graph which consists of models that predict the likelihood of alternate treatment strategies achieving care goals. An AI-driven clinical decision support system (CDSS) integrated with the electronic health record (EHR) was developed by encapsulating the prediction models in an OpenCDS Web service module and delivering the model outputs through a SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources) web-based dashboard. This CDSS enables clinicians and patients to review relevant patient parameters, select treatment goals, and review alternate treatment strategies based on prediction results. RESULTS: The proposed analytical method outperformed previous machine-learning algorithms on prediction accuracy. The CDSS was successfully integrated with the Epic EHR at the University of Utah. CONCLUSION: A predictive analytics-based CDSS was developed and successfully integrated with the EHR through standards-based interoperability frameworks. The approach used could potentially be applied to many other chronic conditions to bring AI-driven CDSS to the point of care.

A Pathway for Community-Acquired Pneumonia With Rapid Conversion to Oral Therapy Improves Health Care Value.

BACKGROUND: Evidence supports streamlined approaches for inpatients with community-acquired pneumonia (CAP) including early transition to oral antibiotics and shorter therapy. Uptake of these approaches is variable, and the best approaches to local implementation of infection-specific guidelines are unknown. Our objective was to evaluate the impact of a clinical decision support (CDS) tool linked with a clinical pathway on CAP care. METHODS: This is a retrospective, observational pre-post intervention study of inpatients with pneumonia admitted to a single academic medical center. Interventions were introduced in 3 sequential 6-month phases; Phase 1: education alone; Phase 2: education and a CDS-driven CAP pathway coupled with active antimicrobial stewardship and provider feedback; and Phase 3: education and a CDS-driven CAP pathway without active stewardship. The 12 months preceding the intervention were used as a baseline. Primary outcomes were length of intravenous antibiotic therapy and total length of antibiotic therapy. Clinical, process, and cost outcomes were also measured. RESULTS: The study included 1021 visits. Phase 2 was associated with significantly lower length of intravenous and total antibiotic therapy, higher procalcitonin lab utilization, and a 20% cost reduction compared with baseline. Phase 3 was associated with significantly lower length of intravenous antibiotic therapy and higher procalcitonin lab utilization compared with baseline. CONCLUSIONS: A CDS-driven CAP pathway supplemented by active antimicrobial stewardship review led to the most robust improvements in antibiotic use and decreased costs with similar clinical outcomes.