Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Breastfeeding Attitudes and Practices in a Rural Utah Navajo Community.

OBJECTIVES: Despite known health benefits of breastfeeding, the Navajo have low reported frequency of breastfeeding initiation and support. We evaluated breastfeeding frequencies and practices in the predominately Navajo community of rural San Juan County, Utah, to identify factors that affect breastfeeding decisions and duration. METHODS: We performed retrospective chart review for 135 infants aged 0 to 12 months, and surveys of 85 mothers of infants aged 0 to 2 years, and eight primary care providers. We characterized demographic factors using counts/percentages and medians/inter-quartile ranges, and compared mothers who breastfed for 6 months or less versus greater than 6 months. RESULTS: In 96 infants with complete feeding documentation, 86 infants (90%) received some breast milk and 36 infants (38%) were exclusively breastfed at age 2 months. In 67 infants with complete feeding documentation at ≥ 6 months, 22 infants (33%) were exclusively breastfed 6 months. Most mothers knew about breastfeeding benefits. In 56 mothers whose infants were aged ≥ 6 months at the time of the survey, breastfeeding for more than 6 months had been planned by 44 mothers (79%) but performed by only 29 mothers (52%). Mothers who breastfed for > 6 months were more likely to have been influenced by WIC and less likely to have introduced formula at an early age. Barriers to breastfeeding included maternal pain, latch difficulties, and concerns about inadequate milk supply. Primary care providers reported limited confidence in providing breastfeeding support but would support telehealth-driven interventions. CONCLUSIONS FOR PRACTICE: Practical, culturally sensitive interventions, including telehealth and improved provider education, may improve breastfeeding outcomes and community health in this underserved population.

Modes of failure of hip hemiarthroplasty for femoral neck fracture.

BACKGROUND: Hemiarthroplasty is a common treatment for displaced femoral neck fractures, but limited Canadian data are available about hemiarthroplasty failure. We evaluated the frequency and predictors of hemiarthroplasty failure in Manitoba. METHODS: In this retrospective multicentre province-wide study, billing and joint registry databases showed 4693 patients who had hemiarthroplasty for treatment of femoral neck fracture in Manitoba over an 11-year period (2005-2015), including 155 hips with subsequent reoperations (open or closed) for treatment of hemiarthroplasty failure. Hospital records were reviewed to identify modes of hemiarthroplasty failure, comorbidities and reoperations. Data were analyzed using χ2 test and Poisson and γ regression models. RESULTS: During our study period, 155 hips (154 patients [3%]) underwent 230 reoperations. Of these, 131 hips (85%) initially had an uncemented unipolar modular implant. Indications for first-time reoperation included periprosthetic femur fracture (49 hips [32%]), dislocation (45 hips [29%]), acetabular wear (28 hips [18%]) and infection (26 hips [17%]). There were 46 hips (30%) that had 2 or more reoperations. Reoperation for dislocation was associated with presence of dementia; acetabular wear was associated with absence of dementia. Time from hemiarthroplasty to reoperation was associated inversely with age at hemiarthroplasty, dislocation and dementia and was directly associated with acetabular wear. The risk of having 2 or more reoperations was associated independently with dislocation, infection, and alcohol abuse. CONCLUSION: Hemiarthroplasty for femoral neck fracture in Manitoba had a low frequency of failure. Risk factors for multiple reoperations included dislocation, infection and alcohol abuse.

Improvement Initiative: End-Tidal Carbon Monoxide Measurement in Newborns Receiving Phototherapy.

OBJECTIVES: To determine, as part of our Utah Newborn Nursery Bilirubin Management Program, whether end-tidal carbon monoxide concentration (ETCOc) measurements in all newborns in our nursery receiving phototherapy were associated with outcomes related to the management of hyperbilirubinemia, including time (hours after birth) when phototherapy was initiated, total duration of phototherapy during the nursery stay, repeat phototherapy treatments, and hospital readmission for phototherapy. STUDY DESIGN: We performed a planned interim analysis of a component of our program in which we measured ETCOc noninvasively using CoSense on each newborn in our nursery receiving phototherapy and recorded specific outcomes related to phototherapy management. RESULTS: Of 1856 newborns admitted to our nursery in a 6-month period in 2020, 170 (9.8%) were treated with phototherapy. An ETCOc reading was successfully obtained in 145 of 151 attempts (96%). Higher ETCOc values were associated with earlier institution of phototherapy and longer duration of phototherapy. For every 1-ppm increase in ETCOc, phototherapy was started 9 hours earlier (95% CI, 3.3-14.8; P = .002) and was administered for an additional 9.3 hours (95% CI, 4.1-14.6; P < .001). Three newborns were readmitted to the hospital for intensive phototherapy; while in the nursery, all 3 had an elevated ETCOc (2.2, 2.6, and 2.9 ppm). CONCLUSIONS: Our findings provide answers to questions raised in the 2004 American Academy of Pediatrics bilirubin guidelines. In our neonatal nursery, measuring ETCOc in all phototherapy recipients was feasible and safe, and the results were associated with multiple aspects of phototherapy management. Higher ETCOc values predicted earlier and longer phototherapy courses.

Real-Time Breastfeeding Documentation: Timing of Breastfeeding Initiation and Outpatient Duration.

BACKGROUND: Current breastfeeding guidelines promote initiating breastfeeding ≤1 h after birth to establish long-term breastfeeding. Previous studies dichotomized initiation to ≤1 h versus subsequent hours combined. There are limited data evaluating the effect of initiation in each subsequent hour on breastfeeding duration. Our objective was to evaluate the association between breastfeeding initiated at ≤1 h versus the subsequent 23 hours after birth and outpatient breastfeeding duration. METHODS: In this retrospective cohort study, we analyzed real-time, discretely documented electronic health record (EHR) breastfeeding data for 3315 infants born at a university center and followed to age ≥12 mo at 27 university primary care clinics. The primary outcome was breastfeeding duration. The exposure variable was hour of breastfeeding initiation within 24 h postnatally. Data were analyzed by univariable and multivariable linear regression separately for infants born by vaginal versus cesarean delivery. RESULTS: In adjusted models, initiating breastfeeding during each hour from age >1 to ≤6 h and during ages >6 to ≤24 h was not associated with decreased breastfeeding duration versus initiating breastfeeding at ≤1 h after birth for infants born via vaginal or cesarean delivery. CONCLUSIONS: Delaying breastfeeding initiation to >1 to ≤24 h after birth is not associated with decreased breastfeeding duration compared with initiating breastfeeding at ≤1 h after birth. Integration of breastfeeding measures into inpatient and outpatient EHR discrete data fields may clarify best practices that support long-term breastfeeding as a public health imperative.

Electronic Health Record-Integrated Clinical Decision Support for Clinicians Serving Populations Facing Health Care Disparities: Literature Review.

OBJECTIVES: To review current studies about designing and implementing clinician-facing clinical decision support (CDS) integrated or interoperable with an electronic health record (EHR) to improve health care for populations facing disparities. METHODS: We searched PubMed to identify studies published between January 1, 2011 and October 22, 2021 about clinician-facing CDS integrated or interoperable with an EHR. We screened abstracts and titles and extracted study data from articles using a protocol developed by team consensus. Extracted data included patient population characteristics, clinical specialty, setting, EHR, clinical problem, CDS type, reported user-centered design, implementation strategies, and outcomes. RESULTS: There were 28 studies (36 articles) included. Most studies were performed at safety net institutions (14 studies) or Indian Health Service sites (6 studies). CDS tools were implemented in primary care outpatient settings in 24 studies (86%) for screening or treatment. CDS included point-of-care alerts (93%), order facilitators (46%), workflow support (39%), relevant information display (36%), expert systems (11%), and medication dosing support (7%). Successful outcomes were reported in 19 of 26 studies that reported outcomes (73%). User-centered design was reported during CDS planning (39%), development (32%), and implementation phase (25%). Most frequent implementation strategies were education (89%) and consensus facilitation (50%). CONCLUSIONS: CDS tools may improve health equity and outcomes for patients who face disparities. The present review underscores the need for high-quality analyses of CDS-associated health outcomes, reporting of user-centered design and implementation strategies used in low-resource settings, and methods to disseminate CDS created to improve health equity.

Improving Autism Screening and Referrals With Electronic Support and Evaluations in Primary Care.

BACKGROUND: Universal screening for autism promotes early evidence-based treatment. However, many children are not screened, and screened children are often not referred for autism evaluation. METHODS: We implemented process changes in 3 phases: phase 1, changing the screening instrument and adding decision support; phase 2, adding automatic reminders; and phase 3, adding a referral option for autism evaluations in primary care. We analyzed the proportion of visits with autism screening at 2 intervention clinics before and after implementation of process changes versus 27 community clinics (which received only automatic reminders in phase 2) with χ2 test and interrupted time series. We evaluated changes in referral for autism evaluation by calculating the rate ratio for referral. RESULTS: In 12 233 visits over 2 years (baseline and phased improvements), autism screening increased by 52% in intervention clinics (58.6%-88.8%; P < .001) and 21% in community clinics (43.4%-52.4%; P < .001). In phase 1, interrupted time series trend for screening in intervention clinics increased by 2% per week (95% confidence interval [CI]: 1.1% to 2.9%) and did not increase in community clinics. In phase 2, screening in the community clinics increased by 0.46% per week (95% CI: 0.03% to 0.89%). In phase 3, the intervention clinic providers referred patients for diagnostic evaluation 3.4 times more frequently (95% CI: 2.0 to 5.8) than at baseline. CONCLUSIONS: We improved autism screening and referrals by changing the screening instrument, adding decision support, using automatic reminders, and offering autism evaluation in primary care in intervention clinics. Automatic reminders alone improved screening in community clinics.

Establishing a multidisciplinary initiative for interoperable electronic health record innovations at an academic medical center.

OBJECTIVE: To establish an enterprise initiative for improving health and health care through interoperable electronic health record (EHR) innovations. MATERIALS AND METHODS: We developed a unifying mission and vision, established multidisciplinary governance, and formulated a strategic plan. Key elements of our strategy include establishing a world-class team; creating shared infrastructure to support individual innovations; developing and implementing innovations with high anticipated impact and a clear path to adoption; incorporating best practices such as the use of Fast Healthcare Interoperability Resources (FHIR) and related interoperability standards; and maximizing synergies across research and operations and with partner organizations. RESULTS: University of Utah Health launched the ReImagine EHR initiative in 2016. Supportive infrastructure developed by the initiative include various FHIR-related tooling and a systematic evaluation framework. More than 10 EHR-integrated digital innovations have been implemented to support preventive care, shared decision-making, chronic disease management, and acute clinical care. Initial evaluations of these innovations have demonstrated positive impact on user satisfaction, provider efficiency, and compliance with evidence-based guidelines. Return on investment has included improvements in care; over $35 million in external grant funding; commercial opportunities; and increased ability to adapt to a changing healthcare landscape. DISCUSSION: Key lessons learned include the value of investing in digital innovation initiatives leveraging FHIR; the importance of supportive infrastructure for accelerating innovation; and the critical role of user-centered design, implementation science, and evaluation. CONCLUSION: EHR-integrated digital innovation initiatives can be key assets for enhancing the EHR user experience, improving patient care, and reducing provider burnout.

Improving the Bilirubin Management Program in the Newborn Nursery: Background, Aims, and Protocol.

BACKGROUND: Practices to detect and manage hyperbilirubinemia in newborn nurseries are highly variable. American Academy of Pediatrics guidelines in 1999, 2004, and 2009 have generated, perhaps unintentionally, divergent practices that might not all be of equivalent value. Evidence-based progress is needed to define less invasive, less expensive, uniform, and safe methods to reduce ER visits and hospital readmissions for jaundice treatment and bilirubin encephalopathy. OBJECTIVES: This research briefing is intended to inform readers of a new prospective quality improvement program aimed at testing the value of specific changes in newborn nursery hyperbilirubinemia detection and management. This new program includes predetermined means of assessing those specific changes, which relate to diagnosis, safety, outcomes, and cost. METHODS: In this briefing, we present the perceived problems in our present bilirubin management system, as voiced by stakeholders. We report our proposed means to test minimization of those problems utilizing already acquired data on approximately 400,000 well babies in the Intermountain Healthcare system of hospitals in the western USA. We then describe our methods of assessing specific outcomes in a pre- versus postpractice change analysis. RESULTS AND CONCLUSIONS: The University of Utah Newborn Nursery will implement a quality improvement project in bilirubin management during 2020 to test the feasibility and effectiveness of several changes to our current bilirubin management program. We maintain that the improved understanding generated by this project will be a step toward new evidence-based strategies for reducing ER visits and hospital readmissions for jaundice treatment and preventing bilirubin encephalopathy.

Early-Onset Sepsis Risk Calculator Integration Into an Electronic Health Record in the Nursery.

BACKGROUND AND OBJECTIVES: An early-onset sepsis (EOS) risk calculator tool to guide evaluation and treatment of infants at risk for sepsis has reduced antibiotic use without increased adverse outcomes. We performed an electronic health record (EHR)-driven quality improvement intervention to increase calculator use for infants admitted to a newborn nursery and reduce antibiotic treatment of infants at low risk for sepsis. METHODS: This 2-phase intervention included programming (1) an EHR form containing calculator fields that were external to the infant's admission note, with nonautomatic access to the calculator, education for end-users, and reviewing risk scores in structured bedside rounds and (2) discrete data entry elements into the EHR admission form with a hyperlink to the calculator Web site. We used statistical process control to assess weekly entry of risk scores and antibiotic orders and interrupted time series to assess trend of antibiotic orders. RESULTS: During phase 1 (duration, 14 months), a mean 59% of infants had EOS calculator scores entered. There was wide variability around the mean, with frequent crossing of weekly means beyond the 3σ control lines, indicating special-cause variation. During phase 2 (duration, 2 years), mean frequency of EOS calculator use increased to 85% of infants, and variability around the mean was within the 3σ control lines. The frequency of antibiotic orders decreased from preintervention (7%) to the final 6 months of phase 2 (1%, P < .001). CONCLUSIONS: An EHR-driven quality improvement intervention increased EOS calculator use and reduced antibiotic orders, with no increase in adverse events.

Association of an Electronic Health Record Add-on App for Neonatal Bilirubin Management With Physician Efficiency and Care Quality.

Importance: The usefulness of electronic health record (EHR) systems could be significantly enhanced by innovative, third-party EHR add-on apps. Objective: To evaluate whether an EHR add-on app for neonatal bilirubin management can save clinicians time and improve patient care. Design, Setting, and Participants: This quality improvement study was conducted at the University of Utah Health Well Baby nursery and outpatient clinics and consisted of 4 substudies: (1) time savings were estimated in an experimental task-timing study comparing the time required for physicians to manage newborns' bilirubin levels with and without the add-on app, (2) app use was estimated from app logs, (3) health care use measures and guideline compliance were compared retrospectively before and after the intervention, and (4) clinician-perceived usability was measured through System Usability Scale surveys. The study took place between April 1, 2016, and September 3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019. Interventions: At baseline, clinicians used a manual approach to ensure compliance with an evidence-based clinical guideline for neonatal bilirubin management. To facilitate guideline compliance, an EHR add-on app that automatically retrieves, organizes, and visualizes relevant patient data was developed. The app provides patient-specific assessments and recommendations, including the risk of rebound hyperbilirubinemia following phototherapy based on a predictive model. The add-on app was integrated with the University of Utah Health EHR on April 12, 2017. Main Outcomes and Measures: Clinician time savings, app use, health care use measures, guideline-compliant phototherapy ordering, and perceived usability as measured by the System Usability Scale survey. The survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100, with ratings described as worst imaginable (mean System Usability Scale score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9). Results: In 2018, the application was used 20 516 times by clinicians for 91.84% of eligible newborns. Use of the app saved 66 seconds for bilirubin management tasks compared with a commonly used tool (95% CI, 53-79 seconds; P < .001). Following the intervention, health care use rates remained stable, while orders for clinically appropriate phototherapy during hospitalization increased for newborns with bilirubin levels above the guideline-recommended threshold (odds ratio, 1.84; 95% CI, 1.16-2.90; P = .009). Surveys indicated excellent usability (System Usability Scale score, 83.90; 95% CI, 81.49-86.31). Conclusions and Relevance: Well-designed EHR add-on apps may save clinicians time and improve patient care. If time-saving apps, such as the bilirubin app, were implemented widely across institutions and care domains, the potential association with improved patient care and clinician efficiency could be significant. The University of Utah Health bilirubin app is being prepared for release into EHR app stores as free-to-use software.

Quality Improvement Intervention for Universal Lipid Screening in Children Aged 9 to 11 Years.

We performed a quality improvement intervention to increase universal lipid screening in well-child visits (age 9 to 11 years): 12-month preintervention; phase 1 (8 months) with provider education, group monthly chart review with feedback, and electronic health record cues to order lipids; and phase 2 (16 months) with electronic health record cues and examination room phlebotomy. Outcomes were compared with clinics having no intervention. In phase 1, immediate treatment effect on the regression line for provider behavior (proportion of visits with lipids ordered) showed 34% increase in intervention and 7% decrease in comparison clinics; patient behavior (phlebotomy completed) showed 19% increase in intervention and 5% decrease in comparison clinics. At the beginning of phase 2, the intervention clinic had average 44% orders entered and 33% phlebotomy completed per well-child visit, and these proportions were maintained. Provider education and chart review with feedback were associated with the greatest gains in outcomes.

Financial Analysis of Pediatric Resident Physician Primary Care Longitudinal Outpatient Experience.

OBJECTIVE: To determine whether residency training represents a net positive or negative cost to academic medical centers, we analyzed the cost of a residency program and clinical productivity of residents and faculty in an outpatient primary care practice with or without residents. METHODS: Patient volume and revenue data (Current Procedural Terminology codes) from an academic primary care general pediatric clinic were evaluated for faculty clinics (faculty only) and resident teaching clinics (longitudinal outpatient experience [LOE]) with 1 to 4 residents per faculty. A detailed cost per resident was determined using a departmental financial model that included salary, benefits, faculty and administrative staff effort, nonpersonnel costs, and institutional graduate medical education support. RESULTS: The LOE clinics had a greater mean number of patient visits (11.6 vs 6.8) than faculty clinics per faculty member. In the LOE clinic, the number of patient visits per clinic was directly proportional to the number of residents per faculty. The cost for each resident was $250 per clinic ($112 per resident, $88 per medical assistant per resident, and $50 per room per resident). When factoring in clinic costs and faculty supervision time, the LOE clinics (average 3.5 residents with 1 supervising faculty) had greater average cost (+$687.00) and revenue (+$319.45) and lower operating margin (revenue minus cost, -$367.55) than the faculty clinics (1 faculty member). CONCLUSIONS: Pediatric resident LOE clinics had a greater average number of patient visits and revenue per faculty member but higher costs and lower operating margins than faculty clinics.

A Pragmatic Guide to Establishing Clinical Decision Support Governance and Addressing Decision Support Fatigue: a Case Study.

There is limited guidance available in the literature for establishing clinical decision support (CDS) governance and improving CDS effectiveness in a pragmatic, resource-efficient manner. Here, we describe how University of Utah Health established enterprise CDS governance in 2015 leveraging existing resources. Key components of the governance include a multi-stakeholder CDS Committee that vets new requests and reviews existing content; a requirement that proposed CDS is actually desired by intended recipients; coordination with other governance bodies; basic data analytics to identify high-frequency, low-value CDS and monitor progress; active solicitation of user issues; the transition of alert and reminder content to other, more appropriate areas in the electronic health record; and the judicious use of experimental designs to guide decision-making regarding CDS effectiveness. In the three years since establishing this governance, new CDS has been continuously added while the overall burden of clinician-facing alerts and reminders has been reduced by 53.8%.