Engaging the next generation of physician-informaticians through early exposure to the field: successes and challenges associated with starting a novel clinical informatics interest group.

Clinical informatics remains underappreciated among medical students in part due to a lack of integration into undergraduate medical education (UME). New developments in the study and practice of medicine are traditionally introduced via formal integration into undergraduate medical curricula. While this path has certain advantages, curricular changes are slow and may fail to showcase the breadth of clinical informatics activities. Less formal and more flexible approaches can circumvent these drawbacks. Interest groups (IGs), which are organized through the Association of American Medical College Careers in Medicine (CiM) program, exemplify the informal approach. CiM IGs are student-led groups that provide exposure to different specialty options, acting as an adjunct to the traditional medical curriculum. While the primary purpose of these groups is to assist students applying to residency programs, we took a novel approach of using an IG to increase student exposure to an area of medicine that had not yet been formally integrated at our institution. IGs provide unique advantages to formal integration into a curriculum as they can be more easily setup and can quickly respond to student interests. Furthermore, IGs can act synergistically with UME, acting as proving grounds for ideas that can lead to new courses. We believe that the lessons and takeaways from our experience can act as a guide for those interested in starting similar organizations at their own schools.

Domains, tasks, and knowledge for health informatics practice: results of a practice analysis.

OBJECTIVE: To develop a comprehensive and current description of what health informatics (HI) professionals do and what they need to know. MATERIALS AND METHODS: Six independent subject-matter expert panels drawn from and representative of HI professionals contributed to the development of a draft HI delineation of practice (DoP). An online survey was distributed to HI professionals to validate the draft DoP. A total of 1011 HI practitioners completed the survey. Survey respondents provided domain, task, knowledge and skill (KS) ratings, qualitative feedback on the completeness of the DoP, and detailed professional background and demographic information. RESULTS: This practice analysis resulted in a validated, comprehensive, and contemporary DoP comprising 5 domains, 74 tasks, and 144 KS statements. DISCUSSION: The HI practice analysis defined "health informatics professionals" to include practitioners with clinical (eg, dentistry, nursing, pharmacy), public health, and HI or computer science training. The affirmation of the DoP by reviewers and survey respondents reflects the emergence of a core set of tasks performed and KSs used by informaticians representing a broad spectrum of those currently practicing in the field. CONCLUSION: The HI practice analysis represents the first time that HI professionals have been surveyed to validate a description of their practice. The resulting HI DoP is an important milestone in the maturation of HI as a profession and will inform HI certification, accreditation, and education activities.

Next generation pathways into biomedical informatics: lessons from 10 years of the Vanderbilt Biomedical Informatics Summer Internship Program.

OBJECTIVES: To examine roles for summer internship programs in expanding pathways into biomedical informatics, based on 10 years of the Vanderbilt Department of Biomedical Informatics (DBMI) Summer Research Internship Program. MATERIALS AND METHODS: Vanderbilt DBMI's internship program is a research-intensive paid 8-10 week program for high school, undergraduate, and graduate students. The program is grounded in a "Windows, Mirrors, and Open Doors" educational framework, and is guided by an evolving set of design principles, including providing meaningful research experiences, applying a multi-factor approach to diversity, and helping interns build peer connections. RESULTS: Over 10 years, 90 individuals have participated in the internship program, with nine students participating for more than one summer. Of 90 participants, 38 were women and 52 were men. Participants represented a range of racial/ethnic groups. A total of 39 faculty members have served as primary mentor for one or more interns. Five key lessons emerged from our program experience: Festina Lente ("Make haste slowly"), The Power of Community, Learning by Doing, Thoughtful Partnerships Lead to Innovation, and The Whole is More Than the Sum of Its Parts. DISCUSSION: Based on our experience, we suggest that internship programs should become a core element of the biomedical informatics educational ecosystem. Continued development and growth of this important educational outreach approach requires stable funding sources and building connections between programs to share best practices. CONCLUSION: Internship programs can play a substantial role in the biomedical informatics educational ecosystem, helping introduce individuals to the field earlier in their educational trajectories.

Creating advanced health informatics certification.

In 2005, AMIA leaders and members concluded that certification of advanced health informatics professionals would offer value to individual practitioners, organizations that hire them, and society at large. AMIA's work to create advanced informatics certification began by leading a successful effort to create the clinical informatics subspecialty for American Board of Medical Specialties board-certified physicians. Since 2012, AMIA has been working to establish advanced health informatics certification (AHIC) for all health informatics practitioners regardless of their primary discipline. In November 2015, AMIA completed the first of 3 key tasks required to establish AHIC, with the AMIA Board of Directors' endorsement of proposed eligibility requirements. This AMIA Board white paper describes efforts to establish AHIC, reports on the current status of AHIC components, and provides a context for the proposed AHIC eligibility requirements.

Eligibility requirements for advanced health informatics certification.

AMIA is leading the effort to strengthen the health informatics profession by creating an advanced health informatics certification (AHIC) for individuals whose informatics work directly impacts the practice of health care, public health, or personal health. The AMIA Board of Directors has endorsed a set of proposed AHIC eligibility requirements that will be presented to the future AHIC certifying entity for adoption. These requirements specifically establish who will be eligible to sit for the AHIC examination and more generally signal the depth and breadth of knowledge and experience expected from certified individuals. They also inform the development of the accreditation process and provide guidance to graduate health informatics programs as well as individuals interested in pursuing AHIC. AHIC eligibility will be determined by practice focus, education in primary field and health informatics, and significant health informatics experience.

Perceived Reasons for High and Low Quality Observational HIV Research Data.

Audits of data quality in a Latin America HIV research network revealed that study sites collected weight measurements, laboratory results, and medication data of inconsistent quality. We surveyed site personnel about perceived drivers of their high or low quality data. Most sites reported their research teams contained no data specialists and that missing data stemmed primarily from incomplete patient assessments at the point of care rather than inconsistent data recording. The root causes of data errors resulted from limited clinic resources (e.g., broken scales, limited record storage space), workflow complications, or the indifference of external participants towards research activities. Understanding these factors supports targeted quality improvement processes.

A novel approach to supporting relationship-centered care through electronic health record ergonomic training in preclerkship medical education.

PROBLEM: How can physicians incorporate the electronic health record (EHR) into clinical practice in a relationship-enhancing fashion ("EHR ergonomics")? APPROACH: Three convenience samples of 40 second-year medical students with varying levels of EHR ergonomic training were compared in the 2012 spring semester. All participants first received basic EHR training and completed a presurvey. Two study groups were then instructed to use the EHR during the standardized patient (SP) encounter in each of four regularly scheduled Doctoring (clinical skills) course sessions. One group received additional ergonomic training in each session. Ergonomic assessment data were collected from students, faculty, and SPs in each session. A postsurvey was administered to all students, and data were compared across all three groups to assess the impact of EHR use and ergonomic training. OUTCOMES: There was a significant positive effect of EHR ergonomics skills training on students' relationship-centered EHR use (P<.005). Students who received training reported that they were able to use the EHR to engage with patients more effectively, better articulate the benefits of using the EHR, better address patient concerns, more appropriately position the EHR device, and more effectively integrate the EHR into patient encounters. Additionally, students' self-assessments were strongly corroborated by SP and faculty assessments. A minimum of three ergonomic training sessions were needed to see an overall improvement in EHR use. NEXT STEPS: In addition to replication of these results, further effectiveness studies of this educational intervention need to be carried out in GME, practice, and other environments.

Bridging organizational divides in health care: an ecological view of health information exchange.

BACKGROUND: The fragmented nature of health care delivery in the United States leads to fragmented health information and impedes patient care continuity and safety. Technologies to support interorganizational health information exchange (HIE) are becoming more available. Understanding how HIE technology changes health care delivery and affects people and organizations is crucial to long-term successful implementation. OBJECTIVE: Our study investigated the impacts of HIE technology on organizations, health care providers, and patients through a new, context-aware perspective, the Regional Health Information Ecology. METHODS: We conducted more than 180 hours of direct observation, informal interviews during observation, and 9 formal semi-structured interviews. Data collection focused on workflow and information flow among health care team members and patients and on health care provider use of HIE technology. RESULTS: We structured the data analysis around five primary information ecology components: system, locality, diversity, keystone species, and coevolution. Our study identified three main roles, or keystone species, involved in HIE: information consumers, information exchange facilitators, and information repositories. The HIE technology impacted patient care by allowing providers direct access to health information, reducing time to obtain health information, and increasing provider awareness of patient interactions with the health care system. Developing the infrastructure needed to support HIE technology also improved connections among information technology support groups at different health care organizations. Despite the potential of this type of technology to improve continuity of patient care, HIE technology adoption by health care providers was limited. CONCLUSIONS: To successfully build a HIE network, organizations had to shift perspectives from an ownership view of health data to a continuity of care perspective. To successfully integrate external health information into clinical work practices, health care providers had to move toward understanding potential contributions of external health information. Our study provides a foundation for future context-aware development and implementation of HIE technology. Integrating concepts from the Regional Health Information Ecology into design and implementation may lead to wider diffusion and adoption of HIE technology into clinical work.

Mediation of adoption and use: a key strategy for mitigating unintended consequences of health IT implementation.

OBJECTIVE: Without careful attention to the work of users, implementation of health IT can produce new risks and inefficiencies in care. This paper uses the technology use mediation framework to examine the work of a group of nurses who serve as mediators of the adoption and use of a barcode medication administration (BCMA) system in an inpatient setting. MATERIALS AND METHODS: The study uses ethnographic methods to explore the mediators' work. Data included field notes from observations, documents, and email communications. This variety of sources enabled triangulation of findings between activities observed, discussed in meetings, and reported in emails. RESULTS: Mediation work integrated the BCMA tool with nursing practice, anticipating and solving implementation problems. Three themes of mediation work include: resolving challenges related to coordination, integrating the physical aspects of BCMA into everyday practice, and advocacy work. DISCUSSION: Previous work suggests the following factors impact mediation effectiveness: proximity to the context of use, understanding of users' practices and norms, credibility with users, and knowledge of the technology and users' technical abilities. We describe three additional factors observed in this case: 'influence on system developers,' 'influence on institutional authorities,' and 'understanding the network of organizational relationships that shape the users' work.' CONCLUSION: Institutionally supported clinicians who facilitate adoption and use of health IT systems can improve the safety and effectiveness of implementation through the management of unintended consequences. Additional research on technology use mediation can advance the science of implementation by providing decision-makers with theoretically durable, empirically grounded evidence for designing implementations.

Measuring the quality of observational study data in an international HIV research network.

Observational studies of health conditions and outcomes often combine clinical care data from many sites without explicitly assessing the accuracy and completeness of these data. In order to improve the quality of data in an international multi-site observational cohort of HIV-infected patients, the authors conducted on-site, Good Clinical Practice-based audits of the clinical care datasets submitted by participating HIV clinics. Discrepancies between data submitted for research and data in the clinical records were categorized using the audit codes published by the European Organization for the Research and Treatment of Cancer. Five of seven sites had error rates >10% in key study variables, notably laboratory data, weight measurements, and antiretroviral medications. All sites had significant discrepancies in medication start and stop dates. Clinical care data, particularly antiretroviral regimens and associated dates, are prone to substantial error. Verifying data against source documents through audits will improve the quality of databases and research and can be a technique for retraining staff responsible for clinical data collection. The authors recommend that all participants in observational cohorts use data audits to assess and improve the quality of data and to guide future data collection and abstraction efforts at the point of care.

The financial impact of health information exchange on emergency department care.

OBJECTIVE: To examine the financial impact health information exchange (HIE) in emergency departments (EDs). MATERIALS AND METHODS: We studied all ED encounters over a 13-month period in which HIE data were accessed in all major emergency departments Memphis, Tennessee. HIE access encounter records were matched with similar encounter records without HIE access. Outcomes studied were ED-originated hospital admissions, admissions for observation, laboratory testing, head CT, body CT, ankle radiographs, chest radiographs, and echocardiograms. Our estimates employed generalized estimating equations for logistic regression models adjusted for admission type, length of stay, and Charlson co-morbidity index. Marginal probabilities were used to calculate changes in outcome variables and their financial consequences. RESULTS: HIE data were accessed in approximately 6.8% of ED visits across 12 EDs studied. In 11 EDs directly accessing HIE data only through a secure Web browser, access was associated with a decrease in hospital admissions (adjusted odds ratio (OR)=0.27; p<0001). In a 12th ED relying more on print summaries, HIE access was associated with a decrease in hospital admissions (OR=0.48; p<0001) and statistically significant decreases in head CT use, body CT use, and laboratory test ordering. DISCUSSION: Applied only to the study population, HIE access was associated with an annual cost savings of $1.9 million. Net of annual operating costs, HIE access reduced overall costs by $1.07 million. Hospital admission reductions accounted for 97.6% of total cost reductions. CONCLUSION: Access to additional clinical data through HIE in emergency department settings is associated with net societal saving.

Barriers and facilitators to the use of computer-based intensive insulin therapy.

PURPOSE: Computerized clinical decision support systems (CDSSs) for intensive insulin therapy (IIT) are increasingly common. However, recent studies question IIT's safety and mortality benefit. Researchers have identified factors influencing IIT performance, but little is known about how workflow affects computer-based IIT. We used ethnographic methods to evaluate IIT CDSS with respect to other clinical information systems and care processes. METHODS: We conducted direct observation of and unstructured interviews with nurses using IIT CDSS in the surgical and trauma intensive care units at an academic medical center. We observed 49h of intensive care unit workflow including 49 instances of nurses using IIT CDSS embedded in a provider order entry system. Observations focused on the interaction of people, process, and technology. By analyzing qualitative field note data through an inductive approach, we identified barriers and facilitators to IIT CDSS use. RESULTS: Barriers included (1) workload tradeoffs between computer system use and direct patient care, especially related to electronic nursing documentation, (2) lack of IIT CDSS protocol reminders, (3) inaccurate user interface design assumptions, and (4) potential for error in operating medical devices. Facilitators included (1) nurse trust in IIT CDSS combined with clinical judgment, (2) nurse resilience, and (3) paper serving as an intermediary between patient bedside and IIT CDSS. CONCLUSION: This analysis revealed sociotechnical interactions affecting IIT CDSS that previous studies have not addressed. These issues may influence protocol performance at other institutions. Findings have implications for IIT CDSS user interface design and alerts, and may contribute to nascent general CDSS theory.

User perspectives on the usability of a regional health information exchange.

OBJECTIVE: We assessed the usability of a health information exchange (HIE) in a densely populated metropolitan region. This grant-funded HIE had been deployed rapidly to address the imminent needs of the patient population and the need to draw wider participation from regional entities. DESIGN: We conducted a cross-sectional survey of individuals given access to the HIE at participating organizations and examined some of the usability and usage factors related to the technology acceptance model. MEASUREMENTS: We probed user perceptions using the Questionnaire for User Interaction Satisfaction, an author-generated Trust scale, and user characteristic questions (eg, age, weekly system usage time). RESULTS: Overall, users viewed the system favorably (ratings for all usability items were greater than neutral (one-sample Wilcoxon test, p<0.0014, Bonferroni-corrected for 35 tests). System usage was regressed on usability, trust, and demographic and user characteristic factors. Three usability factors were positively predictive of system usage: overall reactions (p<0 0.01), learning (p<0.05), and system functionality (p<0.01). Although trust is an important component in collaborative relationships, we did not find that user trust of other participating healthcare entities was significantly predictive of usage. An analysis of respondents' comments revealed ways to improve the HIE. CONCLUSION: We used a rapid deployment model to develop an HIE and found that perceptions of system usability were positive. We also found that system usage was predicted well by some aspects of usability. Results from this study suggest that a rapid development approach may serve as a viable model for developing usable HIEs serving communities with limited resources.

Characteristics and effects of nurse dosing over-rides on computer-based intensive insulin therapy protocol performance.

OBJECTIVE: To determine characteristics and effects of nurse dosing over-rides of a clinical decision support system (CDSS) for intensive insulin therapy (IIT) in critical care units. DESIGN: Retrospective analysis of patient database records and ethnographic study of nurses using IIT CDSS. MEASUREMENTS: The authors determined the frequency, direction-greater than recommended (GTR) and less than recommended (LTR)- and magnitude of over-rides, and then compared recommended and over-ride doses' blood glucose (BG) variability and insulin resistance, two measures of IIT CDSS associated with mortality. The authors hypothesized that rates of hypoglycemia and hyperglycemia would be greater for recommended than over-ride doses. Finally, the authors observed and interviewed nurse users. RESULTS: 5.1% (9075) of 179,452 IIT CDSS doses were over-rides. 83.4% of over-ride doses were LTR, and 45.5% of these were ≥ 50% lower than recommended. In contrast, 78.9% of GTR doses were ≤ 25% higher than recommended. When recommended doses were administered, the rate of hypoglycemia was higher than the rate for GTR (p = 0.257) and LTR (p = 0.033) doses. When recommended doses were administered, the rate of hyperglycemia was lower than the rate for GTR (p = 0.003) and LTR (p < 0.001) doses. Estimates of patients' insulin requirements were higher for LTR doses than recommended and GTR doses. Nurses reported trusting IIT CDSS overall but appeared concerned about recommendations when administering LTR doses. CONCLUSION: When over-riding IIT CDSS recommendations, nurses overwhelmingly administered LTR doses, which emphasized prevention of hypoglycemia but interfered with hyperglycemia control, especially when BG was >150 mg/dl. Nurses appeared to consider the amount of a recommended insulin dose, not a patient's trend of insulin resistance, when administering LTR doses overall. Over-rides affected IIT CDSS protocol performance.

Desiderata for a computer-assisted audit tool for clinical data source verification audits.

Clinical data auditing often requires validating the contents of clinical research databases against source documents available in health care settings. Currently available data audit software, however, does not provide features necessary to compare the contents of such databases to source data in paper medical records. This work enumerates the primary weaknesses of using paper forms for clinical data audits and identifies the shortcomings of existing data audit software, as informed by the experiences of an audit team evaluating data quality for an international research consortium. The authors propose a set of attributes to guide the development of a computer-assisted clinical data audit tool to simplify and standardize the audit process.

A computerized provider order entry intervention for medication safety during acute kidney injury: a quality improvement report.

BACKGROUND: Frequently, prescribers fail to account for changing kidney function when prescribing medications. We evaluated the use of a computerized provider order entry intervention to improve medication management during acute kidney injury. STUDY DESIGN: Quality improvement report with time series analyses. SETTING & PARTICIPANTS: 1,598 adult inpatients with a minimum 0.5-mg/dL increase in serum creatinine level over 48 hours after an order for at least one of 122 nephrotoxic or renally cleared medications. QUALITY IMPROVEMENT PLAN: Passive noninteractive warnings about increasing serum creatinine level appeared within the computerized provider order entry interface and on printed rounding reports. For contraindicated or high-toxicity medications that should be avoided or adjusted, an interruptive alert within the system asked providers to modify or discontinue the targeted orders, mark the current dosing as correct and to remain unchanged, or defer the alert to reappear in the next session. OUTCOMES & MEASUREMENTS: Intervention effect on drug modification or discontinuation, time to modification or discontinuation, and provider interactions with alerts. RESULTS: The modification or discontinuation rate per 100 events for medications included in the interruptive alert within 24 hours of increasing creatinine level improved from 35.2 preintervention to 52.6 postintervention (P < 0.001); orders were modified or discontinued more quickly (P < 0.001). During the postintervention period, providers initially deferred 78.1% of interruptive alerts, although 54% of these eventually were modified or discontinued before patient death, discharge, or transfer. The response to passive alerts about medications requiring review did not significantly change compared with baseline. LIMITATIONS: Single tertiary-care academic medical center; provider actions were not independently adjudicated for appropriateness. CONCLUSIONS: A computerized provider order entry-based alerting system to support medication management after acute kidney injury significantly increased the rate and timeliness of modification or discontinuation of targeted medications.

Effects of blood glucose transcription mismatches on a computer-based intensive insulin therapy protocol.

PURPOSE: Computerized clinical decision support systems (CDSS) for intensive insulin therapy (IIT) generate recommendations using blood glucose (BG) values manually transcribed from testing devices to computers, a potential source of error. We quantified the frequency and effect of blood glucose transcription mismatches on IIT protocol performance. METHODS: We examined 38 months of retrospective data for patients treated with CDSS IIT in two intensive care units at one teaching hospital. A manually transcribed BG value not equal to a corresponding device value was deemed mismatched. For mismatches we recalculated CDSS recommendations using device BG values. We compared matched and mismatched data in terms of CDSS alerts, blood glucose variability, and dosing. RESULTS: Of 189,499 CDSS IIT instances, 5.3% contained mismatched BG values. Mismatched data triggered 93 false alerts and failed to issue 170 alerts for nurses to notify physicians. Four of six BG variability measures differed between matched and mismatched data. Overall insulin dose was greater for matched than mismatched [matched 3.8 (1.6-6.0), median (interquartile range, IQR), versus 3.6 (1.6-5.7); p < 0.001], but recalculated and actual dose were similar. In mismatches preceding hypoglycemia, recalculated insulin dose was significantly lower than actual dose [recalculated 2.7 (0.4-5.0), median (IQR), versus 3.5 (1.4-5.6)]. In mismatches preceding hyperglycemia, recalculated insulin dose was significantly greater than actual dose [recalculated 4.7 (3.3-6.2), median (IQR), versus 3.3 (2.4-4.3); p < 0.001]. Administration of recalculated doses might have prevented blood glucose excursions. CONCLUSIONS: Mismatched blood glucose values can influence CDSS IIT protocol performance.

Social, organizational, and contextual characteristics of clinical decision support systems for intensive insulin therapy: a literature review and case study.

INTRODUCTION: Evaluations of computerized clinical decision support systems (CDSS) typically focus on clinical performance changes and do not include social, organizational, and contextual characteristics explaining use and effectiveness. Studies of CDSS for intensive insulin therapy (IIT) are no exception, and the literature lacks an understanding of effective computer-based IIT implementation and operation. RESULTS: This paper presents (1) a literature review of computer-based IIT evaluations through the lens of institutional theory, a discipline from sociology and organization studies, to demonstrate the inconsistent reporting of workflow and care process execution and (2) a single-site case study to illustrate how computer-based IIT requires substantial organizational change and creates additional complexity with unintended consequences including error. DISCUSSION: Computer-based IIT requires organizational commitment and attention to site-specific technology, workflow, and care processes to achieve intensive insulin therapy goals. The complex interaction between clinicians, blood glucose testing devices, and CDSS may contribute to workflow inefficiency and error. Evaluations rarely focus on the perspective of nurses, the primary users of computer-based IIT whose knowledge can potentially lead to process and care improvements. CONCLUSION: This paper addresses a gap in the literature concerning the social, organizational, and contextual characteristics of CDSS in general and for intensive insulin therapy specifically. Additionally, this paper identifies areas for future research to define optimal computer-based IIT process execution: the frequency and effect of manual data entry error of blood glucose values, the frequency and effect of nurse overrides of CDSS insulin dosing recommendations, and comprehensive ethnographic study of CDSS for IIT.

Peers, regulators, and professions: the influence of organizations in health information technology adoption.

According to the U.S. National Research Council, current health information technology (HIT) efforts are insufficient and arguably detrimental to healthcare transformation. Many hospitals have already implemented HIT, and federal stimulus funding will further adoption efforts. Organizations become more similar through the adoption of innovations like HIT, but the effects of the changes do not necessarily improve efficiency. This view from sociology and organizational studies, called institutional isomorphism, suggests that organizations pursue changes endorsed by peers, regulators, and professional societies through mimetic, coercive, and normative mechanisms, respectively, that improve legitimacy, a socially constructed value that determines an organization's ability to obtain resources and survive. In this paper we examine mimetic, coercive, and normative influences in the adoption of three HIT innovations as well as evidence of resulting inefficiency. Institutional isomorphism provides a useful framework for researchers and practitioners to examine variation in HIT adoption.

Forecasting emergency department crowding: an external, multicenter evaluation.

STUDY OBJECTIVE: We apply a previously described tool to forecast emergency department (ED) crowding at multiple institutions and assess its generalizability for predicting the near-future waiting count, occupancy level, and boarding count. METHODS: The ForecastED tool was validated with historical data from 5 institutions external to the development site. A sliding-window design separated the data for parameter estimation and forecast validation. Observations were sampled at consecutive 10-minute intervals during 12 months (n=52,560) at 4 sites and 10 months (n=44,064) at the fifth. Three outcome measures-the waiting count, occupancy level, and boarding count-were forecast 2, 4, 6, and 8 hours beyond each observation, and forecasts were compared with observed data at corresponding times. The reliability and calibration were measured following previously described methods. After linear calibration, the forecasting accuracy was measured with the median absolute error. RESULTS: The tool was successfully used for 5 different sites. Its forecasts were more reliable, better calibrated, and more accurate at 2 hours than at 8 hours. The reliability and calibration of the tool were similar between the original development site and external sites; the boarding count was an exception, which was less reliable at 4 of 5 sites. Some variability in accuracy existed among institutions; when forecasting 4 hours into the future, the median absolute error of the waiting count ranged between 0.6 and 3.1 patients, the median absolute error of the occupancy level ranged between 9.0% and 14.5% of beds, and the median absolute error of the boarding count ranged between 0.9 and 2.8 patients. CONCLUSION: The ForecastED tool generated potentially useful forecasts of input and throughput measures of ED crowding at 5 external sites, without modifying the underlying assumptions. Noting the limitation that this was not a real-time validation, ongoing research will focus on integrating the tool with ED information systems.