Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Implementation and Continuous Monitoring of an Electronic Health Record Embedded Readmissions Clinical Decision Support Tool.

Unplanned hospital readmissions represent a significant health care value problem with high costs and poor quality of care. A significant percentage of readmissions could be prevented if clinical inpatient teams were better able to predict which patients were at higher risk for readmission. Many of the current clinical decision support models that predict readmissions are not configured to integrate closely with the electronic health record or alert providers in real-time prior to discharge about a patient's risk for readmission. We report on the implementation and monitoring of the Epic electronic health record-"Unplanned readmission model version 1"-over 2 years from 1/1/2018-12/31/2019. For patients discharged during this time, the predictive capability to discern high risk discharges was reflected in an AUC/C-statistic at our three hospitals of 0.716-0.760 for all patients and 0.676-0.695 for general medicine patients. The model had a positive predictive value ranging from 0.217-0.248 for all patients. We also present our methods in monitoring the model over time for trend changes, as well as common readmissions reduction strategies triggered by the score.

Unrealized potential and residual consequences of electronic prescribing on pharmacy workflow in the outpatient pharmacy.

INTRODUCTION: Electronic prescribing systems have often been promoted as a tool for reducing medication errors and adverse drug events. Recent evidence has revealed that adoption of electronic prescribing systems can lead to unintended consequences such as the introduction of new errors. The purpose of this study is to identify and characterize the unrealized potential and residual consequences of electronic prescribing on pharmacy workflow in an outpatient pharmacy. METHODS: A multidisciplinary team conducted direct observations of workflow in an independent pharmacy and semi-structured interviews with pharmacy staff members about their perceptions of the unrealized potential and residual consequences of electronic prescribing systems. We used qualitative methods to iteratively analyze text data using a grounded theory approach, and derive a list of major themes and subthemes related to the unrealized potential and residual consequences of electronic prescribing. RESULTS: We identified the following five themes: Communication, workflow disruption, cost, technology, and opportunity for new errors. These contained 26 unique subthemes representing different facets of our observations and the pharmacy staff's perceptions of the unrealized potential and residual consequences of electronic prescribing. DISCUSSION: We offer targeted solutions to improve electronic prescribing systems by addressing the unrealized potential and residual consequences that we identified. These recommendations may be applied not only to improve staff perceptions of electronic prescribing systems but also to improve the design and/or selection of these systems in order to optimize communication and workflow within pharmacies while minimizing both cost and the potential for the introduction of new errors.

Adverse drug events caused by serious medication administration errors.

OBJECTIVE: To determine how often serious or life-threatening medication administration errors with the potential to cause harm (potential adverse drug events) result in actual harm (adverse drug events (ADEs)) in the hospital setting. DESIGN: Retrospective chart review of clinical events following observed medication administration errors. BACKGROUND: Medication errors are common at the medication administration stage for inpatients. While many errors can cause harm, it is unclear exactly how often. METHODS: In a previous study where 14 041 medication administrations were directly observed, 1271 medication administration errors were discovered, of which 133 had the potential to cause serious or life-threatening harm and were considered serious or life-threatening potential adverse drug events. As a follow-up, clinical reviewers conducted detailed chart review of serious or life-threatening potential ADEs to determine if they caused an ADE. Reviewers assessed severity of the ADE and attribution to the error. RESULTS: Ten (7.5% (95% CI 6.98 to 8.01)) actual ADEs resulted from the 133 serious and life-threatening potential ADEs, of which 6 resulted in significant, three in serious, and one life threatening injury. Therefore 4 (3% (95% CI 2.12 to 3.6)) of serious or life threatening potential ADEs led to serious or life threatening ADEs. Half of the ADEs were caused by dosage or monitoring errors for anti-hypertensives. CONCLUSIONS: Unintercepted potential ADEs at the medication administration stage can cause serious patient harm. At hospitals where 6 million doses are administered per year, about 4000 preventable ADEs would be attributable to medication administration errors annually.

Ability to generate patient registries among practices with and without electronic health records.

BACKGROUND: The ability to generate registries of patients with particular clinical attributes, such as diagnoses or medications taken, is central to measuring and improving the quality of health care. However, it is not known how many providers have the ability to generate such registries. OBJECTIVES: To assess the proportion of physician practices that can construct registries of patients with specific diagnoses, laboratory results, or medications, and to determine the relationship between electronic health record (EHR) usage and the ability to perform registry functions. METHODS: We conducted a mail survey of a stratified random sample of physician practices in Massachusetts in the northeastern United States (N = 1884). The survey included questions about the physicians' ability to generate diagnosis, laboratory result, and medication registries; the presence of EHR; and usage of specific EHR features. RESULTS: The response rate was 71% (1345/1884). Overall, 79.8% of physician practices reported being able to generate registries of patients by diagnosis; 56.1% by laboratory result; and 55.8% by medication usage. In logistic regression analyses, adjusting for urban/rural location, practice size and ownership, teaching status, hospital affiliation, and specialty, physician practices with an EHR were more likely to be able to construct diagnosis registries (adjusted odds ratio [OR] 1.53, 95% confidence interval [CI] 1.25 - 1.86), laboratory registries (OR 1.42, 95% CI 1.22 - 1.66), and medication registries (OR 2.30, 95% CI 1.96 - 2.70). CONCLUSIONS: Many physician practices were able to generate registries, but this capability is far from universal. Adoption of EHRs appears to be a useful step toward this end, and practices with EHRs are considerably more likely to be able to carry out registry functions. Because practices need registries to perform broad-based quality improvement, they should consider adopting EHRs that have built-in registry functionality.

Electronic results management in pediatric ambulatory care: qualitative assessment.

BACKGROUND: Electronic results management may improve the reliability and efficiency of test results management, but few studies have investigated this topic in pediatrics. METHODS: We conducted semi-structured, key informant interviews before and after implementation of electronic results management at 8 pediatric ambulatory care practices. We also surveyed all pediatricians at 18 practices (10 additional practices). All practices were members of Partners Healthcare and had been using an electronic health record when they were offered electronic results management. We assessed baseline processes for results management, barriers to electronic results management adoption, and the perceived impact of electronic results management on quality, efficiency, and provider satisfaction. RESULTS: From interviews, we found a range of processes in place to manage test results, but all practices reported losing some results and no practice tracked all test results from the time of ordering to parent/patient notification. Practices that fully adopted electronic results management reported gains in efficiency, reliability, timeliness, and provider satisfaction, whereas some partial adopters reported decreased efficiency and increased risk of lost test results. Barriers to electronic results management adoption included lack of inclusion of all ordered tests in the electronic results management system, user interface design issues, and lack of sufficient pediatric customization. Survey results (response rate: 62%) indicated that pediatricians thought electronic results management improved the quality and efficiency of care, with 72% of pediatricians reporting safer care and 63% reporting more-effective care. CONCLUSIONS: We found that pediatric practices have room for improvement in the management of test results, and electronic results management may be an effective method for improving the efficiency and safety of test results management. However, partial adoption of electronic results management may decrease efficiency and pose a threat to patient safety.

Formative evaluation: a critical component in EHR implementation.

This Viewpoint paper has grown out of a presentation at the American College of Medical Informatics 2007 Winter Symposium, the resulting discussion, and several activities that have coalesced around an issue that most informaticians accept as true but is not commonly considered during the implementation of Electronic Health Records (EHR) outside of academia or research institutions. Successful EHR implementation is facilitated and sometimes determined by formative evaluation, usually focusing on process rather than outcomes. With greater federal funding for the implementation of electronic health record systems in health care organizations unfamiliar with research protocols, the need for formative evaluation assistance is growing. Such assistance, in the form of tools and protocols necessary to do formative evaluation and resulting in successful EHR implementations, should be provided by practicing medical informaticians.

Cost-benefit analysis of a hospital pharmacy bar code solution.

BACKGROUND: Bar coding can reduce hospital pharmacy dispensing errors, but it is unclear if the benefits of this technology justify its costs. The purpose of this study was to assess the costs and benefits and determine the return on investment at the institutional level for implementing a pharmacy bar code system. METHODS: We performed a cost-benefit analysis of a bar code-assisted medication-dispensing system within a large, academic, nonprofit tertiary care hospital pharmacy. We took the implementing hospital's perspective for a 5-year horizon. The primary outcome was the net financial cost and benefit after 5 years. The secondary outcome was the time until total benefits equaled total costs. Single-variable, 2-variable, and multiple-variable Monte Carlo sensitivity analyses were performed to test the stability of the outcomes. RESULTS: In inflation- and time value-adjusted 2005 dollars, total costs during 5 years were $2.24 million ($1.31 million in 1-time costs during the initial 3.5 years and $342 000 per year in recurring costs starting in year 3). The primary benefit was a decrease in adverse drug events from dispensing errors (517 events annually), resulting in an annual savings of $2.20 million. The net benefit after 5 years was $3.49 million. The break-even point for the hospital's investment occurred within 1 year after becoming fully operational. A net benefit was achieved within 10 years under almost all sensitivity scenarios. In the Monte Carlo simulation, the net benefit during 5 years was $3.2 million (95% confidence interval, -$1.2 million to $12.1 million), and the break-even point for return on investment occurred after 51 months (95% confidence interval, 30 to 180 months). CONCLUSION: Implementation of a bar code-assisted medication-dispensing system in hospital pharmacies can result in a positive financial return on investment for the health care organization.

The extent and importance of unintended consequences related to computerized provider order entry.

BACKGROUND: Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research. OBJECTIVE: To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals. DESIGN, SETTING, AND PARTICIPANTS: Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE. MEASUREMENTS: Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE. RESULTS We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites. CONCLUSIONS: The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one's perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.

The costs of a national health information network.

BACKGROUND: The use of information technology may result in a safer and more efficient health care system. However, consensus does not exist about the structure or costs of a national health information network (NHIN). OBJECTIVES: To describe the potential structure and estimate the costs of an NHIN. DESIGN: Cost estimates of an NHIN model developed by an expert panel. SETTING: U.S. health care system. MEASUREMENTS: An expert panel estimated the existing and the expected prevalence in 5 years of critical information technology functionalities. They then developed a model of an achievable NHIN by defining key providers, functionalities, and interoperability functions. By using these data and published cost estimates, the authors determined the cost of achieving this model NHIN in 5 years given the current state of information technology infrastructure. RESULTS: To achieve an NHIN would cost 156 billion dollars in capital investment over 5 years and 48 billion dollars in annual operating costs. Approximately two thirds of the capital costs would be required for acquiring functionalities and one third for interoperability. Ongoing costs would be more evenly divided between functionality and interoperability. If the current trajectory continues, the health care system will spend 24 billion dollars on functionalities over the next 5 years or about one quarter of the cost for functionalities of a model NHIN. LIMITATIONS: Because of a lack of primary data, the authors relied on expert estimates. CONCLUSIONS: While an NHIN will be expensive, 156 billion dollars is equivalent to 2% of annual health care spending for 5 years. Assessments such as this one may assist policymakers in determining the level of investment that the United States should make in an NHIN.

Communication factors in the follow-up of abnormal mammograms.

OBJECTIVE: To identify the communication factors that are significantly associated with appropriate short-term follow-up of abnormal mammograms. DESIGN: Prospective longitudinal study involving medical record review and patient survey. SETTING: Ten academically affiliated ambulatory medical practices in the Boston metropolitan area. PARTICIPANTS: One hundred twenty-six women with abnormal mammograms requiring short-term (6 months) follow-up imaging. MEASUREMENTS: Proportion of women in the study who received appropriate follow-up care. RESULTS: Eighty-one (64%) of the women with abnormal mammograms requiring short-term follow-up imaging received the appropriate follow-up care. After adjusting for patients' age and insurance status, 2 communication factors were found to be independently associated with the delivery of appropriate follow-up care: 1). physicians' documentation of a follow-up plan in the medical record (adjusted odds ratio, 2.79; 95% confidence interval, 1.11 to 6.98; P =.029); and 2). patients' understanding of the need for follow-up (adjusted odds ratio, 3.86; 95% confidence interval, 1.50 to 9.96; P =.006). None of the patients' clinical or psychological characteristics were associated with the delivery of appropriate follow-up care. CONCLUSIONS: Follow-up care for women with abnormal mammograms requiring short-term follow-up imaging is suboptimal. Documentation of the follow-up plan by the physician and understanding of the follow-up plan by the patient are important factors that are correlated with the receipt of appropriate follow-up care for these women. Interventions designed to improve the quality of result follow-up in the outpatient setting should address these issues in patient-doctor communication.

Overcoming the barriers to the implementing computerized physician order entry systems in US hospitals: perspectives from senior management.

We sought to identify the barriers to CPOE implementation and the strategies for overcoming them. By analyzing 57 transcripts of interviews with management officials at 25 US hospitals, we identified costs and physician resistance as the two most significant barriers. Hospitals often overcome the high cost of CPOE implementation by placing patient safety at the top of their agenda. Other hospitals manage physician resistance by leveraging strong leadership, external influence, vendor commitment and the presence of house staff and hospitalists. Efforts to promote the adoption of CPOE should therefore focus on these strategies.