Development and Evaluation of a Machine Learning Model for the Early Identification of Patients at Risk for Sepsis.

STUDY OBJECTIVE: The Third International Consensus Definitions (Sepsis-3) Task Force recommended the use of the quick Sequential [Sepsis-related] Organ Failure Assessment (qSOFA) score to screen patients for sepsis outside of the ICU. However, subsequent studies raise concerns about the sensitivity of qSOFA as a screening tool. We aim to use machine learning to develop a new sepsis screening tool, the Risk of Sepsis (RoS) score, and compare it with a slate of benchmark sepsis-screening tools, including the Systemic Inflammatory Response Syndrome, Sequential Organ Failure Assessment (SOFA), qSOFA, Modified Early Warning Score, and National Early Warning Score. METHODS: We used retrospective electronic health record data from adult patients who presented to 49 urban community hospital emergency departments during a 22-month period (N=2,759,529). We used the Rhee clinical surveillance criteria as our standard definition of sepsis and as the primary target for developing our model. The data were randomly split into training and test cohorts to derive and then evaluate the model. A feature selection process was carried out in 3 stages: first, we reviewed existing models for sepsis screening; second, we consulted with local subject matter experts; and third, we used a supervised machine learning called gradient boosting. Key metrics of performance included alert rate, area under the receiver operating characteristic curve, sensitivity, specificity, and precision. Performance was assessed at 1, 3, 6, 12, and 24 hours after an index time. RESULTS: The RoS score was the most discriminant screening tool at all time thresholds (area under the receiver operating characteristic curve 0.93 to 0.97). Compared with the next most discriminant benchmark (Sequential Organ Failure Assessment), RoS was significantly more sensitive (67.7% versus 49.2% at 1 hour and 84.6% versus 80.4% at 24 hours) and precise (27.6% versus 12.2% at 1 hour and 28.8% versus 11.4% at 24 hours). The sensitivity of qSOFA was relatively low (3.7% at 1 hour and 23.5% at 24 hours). CONCLUSION: In this retrospective study, RoS was more timely and discriminant than benchmark screening tools, including those recommend by the Sepsis-3 Task Force. Further study is needed to validate the RoS score at independent sites.

Development and Evaluation of an Automated Machine Learning Algorithm for In-Hospital Mortality Risk Adjustment Among Critical Care Patients.

OBJECTIVES: Risk adjustment algorithms for ICU mortality are necessary for measuring and improving ICU performance. Existing risk adjustment algorithms are not widely adopted. Key barriers to adoption include licensing and implementation costs as well as labor costs associated with human-intensive data collection. Widespread adoption of electronic health records makes automated risk adjustment feasible. Using modern machine learning methods and open source tools, we developed and evaluated a retrospective risk adjustment algorithm for in-hospital mortality among ICU patients. The Risk of Inpatient Death score can be fully automated and is reliant upon data elements that are generated in the course of usual hospital processes. SETTING: One hundred thirty-one ICUs in 53 hospitals operated by Tenet Healthcare. PATIENTS: A cohort of 237,173 ICU patients discharged between January 2014 and December 2016. DESIGN: The data were randomly split into training (36 hospitals), and validation (17 hospitals) data sets. Feature selection and model training were carried out using the training set while the discrimination, calibration, and accuracy of the model were assessed in the validation data set. MEASUREMENTS AND MAIN RESULTS: Model discrimination was evaluated based on the area under receiver operating characteristic curve; accuracy and calibration were assessed via adjusted Brier scores and visual analysis of calibration curves. Seventeen features, including a mix of clinical and administrative data elements, were retained in the final model. The Risk of Inpatient Death score demonstrated excellent discrimination (area under receiver operating characteristic curve = 0.94) and calibration (adjusted Brier score = 52.8%) in the validation dataset; these results compare favorably to the published performance statistics for the most commonly used mortality risk adjustment algorithms. CONCLUSIONS: Low adoption of ICU mortality risk adjustment algorithms impedes progress toward increasing the value of the healthcare delivered in ICUs. The Risk of Inpatient Death score has many attractive attributes that address the key barriers to adoption of ICU risk adjustment algorithms and performs comparably to existing human-intensive algorithms. Automated risk adjustment algorithms have the potential to obviate known barriers to adoption such as cost-prohibitive licensing fees and significant direct labor costs. Further evaluation is needed to ensure that the level of performance observed in this study could be achieved at independent sites.

Impact of Palliative Care Screening and Consultation in the ICU: A Multihospital Quality Improvement Project.

CONTEXT: There are few multicenter studies that examine the impact of systematic screening for palliative care and specialty consultation in the intensive care unit (ICU). OBJECTIVE: To determine the outcomes of receiving palliative care consultation (PCC) for patients who screened positive on palliative care referral criteria. METHODS: In a prospective quality assurance intervention with a retrospective analysis, the covariate balancing propensity score method was used to estimate the conditional probability of receiving a PCC and to balance important covariates. For patients with and without PCCs, outcomes studied were as follows: 1) change to "do not resuscitate" (DNR), 2) discharge to hospice, 3) 30-day readmission, 4) hospital length of stay (LOS), 5) total direct hospital costs. RESULTS: In 405 patients with positive screens, 161 (40%) who received a PCC were compared to 244 who did not. Patients receiving PCCs had higher rates of DNR-adjusted odds ratio (AOR) = 7.5; 95% CI 5.6-9.9) and hospice referrals-(AOR = 7.6; 95% CI 5.0-11.7). They had slightly lower 30-day readmissions-(AOR = 0.7; 95% CI 0.5-1.0); no overall difference in direct costs or LOS was found between the two groups. When patients receiving PCCs were stratified by time to PCC initiation, early consultation-by Day 4 of admission-was associated with reductions in LOS (1.7 days [95% CI -3.1, -1.2]) and average direct variable costs (-$1815 [95% CI -$3322, -$803]) compared to those who received no PCC. CONCLUSION: Receiving a PCC in the ICUs was significantly associated with more frequent DNR code status and hospice referrals, but not 30-day readmissions or hospital utilization. Early PCC was associated with significant LOS and direct cost reductions. Providing PCC early in the ICU should be considered.

Health information technology: an updated systematic review with a focus on meaningful use.

BACKGROUND: Incentives offered by the U.S. government have spurred marked increases in use of health information technology (IT). PURPOSE: To update previous reviews and examine recent evidence that relates health IT functionalities prescribed in meaningful use regulations to key aspects of health care. DATA SOURCES: English-language articles in PubMed from January 2010 to August 2013. STUDY SELECTION: 236 studies, including pre-post and time-series designs and clinical trials that related the use of health IT to quality, safety, or efficiency. DATA EXTRACTION: Two independent reviewers extracted data on functionality, study outcomes, and context. DATA SYNTHESIS: Fifty-seven percent of the 236 studies evaluated clinical decision support and computerized provider order entry, whereas other meaningful use functionalities were rarely evaluated. Fifty-six percent of studies reported uniformly positive results, and an additional 21% reported mixed-positive effects. Reporting of context and implementation details was poor, and 61% of studies did not report any contextual details beyond basic information. LIMITATION: Potential for publication bias, and evaluated health IT systems and outcomes were heterogeneous and incompletely described. CONCLUSION: Strong evidence supports the use of clinical decision support and computerized provider order entry. However, insufficient reporting of implementation and context of use makes it impossible to determine why some health IT implementations are successful and others are not. The most important improvement that can be made in health IT evaluations is increased reporting of the effects of implementation and context. PRIMARY FUNDING SOURCE: Office of the National Coordinator.

The value of health information technology: filling the knowledge gap.

Despite rapid growth in the rate of adoption of health information technology (HIT), and in the volume of evaluation studies, the existing knowledge base for the value of HIT is not advancing at a similar rate. Most evaluation articles are limited in that they use incomplete measures of value and fail to report the important contextual and implementation characteristics that would allow for an adequate understanding of how the study results were achieved. To address these deficiencies, we present a conceptual framework for measuring HIT value and we propose a checklist of characteristics that should be considered in HIT evaluation studies. The framework consists of 3 key principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder's perspective is used. Through examples, we show how these principles can be used to guide and improve HIT evaluation studies. The checklist includes a list of contextual and implementation characteristics that are important for interpretation of results. These improvements will make future studies more useful for policy makers and more relevant to the current needs of the healthcare system.

The Future of Health Care in the Kurdistan Region - Iraq: Toward an Effective, High-Quality System with an Emphasis on Primary Care.

At the request of the Kurdistan Regional Government (KRG), RAND researchers undertook a yearlong analysis of the health care system in the Kurdistan Region of Iraq, with a focus on primary care. RAND staff reviewed available literature on the Kurdistan Region and information relevant to primary care; interviewed a wide range of policy leaders, health practitioners, patients, and government officials to gather information and understand their priorities; collected and studied all available data related to health resources, services, and conditions; and projected future supply and demand for health services in the Kurdistan Region; and laid out the health financing challenges and questions. In this volume, the authors describe the strengths of the health care system in the Kurdistan Region as well as the challenges it faces. The authors suggest that a primary care-oriented health care system could help the KRG address many of these challenges. The authors discuss how such a system might be implemented and financed, and they make recommendations for better utilizing resources to improve the quality, access, effectiveness, and efficiency of primary care.

Primary care technicians: a solution to the primary care workforce gap.

Efforts to close the primary care workforce gap typically employ one of three basic strategies: train more primary care physicians; boost the supply of nurse practitioners or physician assistants, or both; or use community health workers to extend the reach of primary care physicians. In this article we briefly review each strategy and the barriers to its success. We then propose a new approach adapted from the widely accepted model of emergency medical services. Translating this model to primary care and leveraging the capabilities of modern health information technology, it should be possible to create primary care technicians who can dramatically expand the impact and reach of patient-centered medical homes by providing basic preventive, minor illness, and stable chronic disease care in rural and resource-deprived communities.

What it will take to achieve the as-yet-unfulfilled promises of health information technology.

A team of RAND Corporation researchers projected in 2005 that rapid adoption of health information technology (IT) could save the United States more than $81 billion annually. Seven years later the empirical data on the technology's impact on health care efficiency and safety are mixed, and annual health care expenditures in the United States have grown by $800 billion. In our view, the disappointing performance of health IT to date can be largely attributed to several factors: sluggish adoption of health IT systems, coupled with the choice of systems that are neither interoperable nor easy to use; and the failure of health care providers and institutions to reengineer care processes to reap the full benefits of health IT. We believe that the original promise of health IT can be met if the systems are redesigned to address these flaws by creating more-standardized systems that are easier to use, are truly interoperable, and afford patients more access to and control over their health data. Providers must do their part by reengineering care processes to take full advantage of efficiencies offered by health IT, in the context of redesigned payment models that favor value over volume.

Reduction of peripherally inserted central catheter-associated DVT.

BACKGROUND: As peripherally inserted central catheter (PICC) use has increased, so has the upper extremity DVT rate. PICC diameter may pose the most modifiable risk for PICC-associated DVT. METHODS: A 3-year, prospective, observational study of all PICC insertions by a specially trained and certified team using a consistent and replicable approach was conducted at a 456-bed, level I trauma and tertiary referral hospital during January 1, 2008, through December 31, 2010. An intensified effort by the PICC team in 2010 was introduced to discuss and reach interdisciplinary consensus on the need for each lumen of the PICC and a change to smaller diameter 5F triple-lumen PICC. RESULTS: Significantly more 4F single-lumen PICCs were used during 2010 (n = 470) compared with 2008 and 2009 (n = 338, 382; P < .0001). DVT rates were similar with the use of 5F triple-lumen PICCs in 2010 as 5F double-lumen PICCs and lower rates than 6F triple-lumen catheters used in 2008 and 2009. The PICC-associated DVT rate was significantly lower (1.9% vs 3.0%, P < .04) in 2010 compared with 2008 and 2009. The cost and length of stay attributable to PICC-associated DVT were $15,973 and 4.6 days. CONCLUSIONS: A significant increase in the use of single-lumen PICCs in addition to the institutional adoption of smaller 5F triple-lumen PICCs was associated with a significant decrease in the rate of PICC-associated DVT.

CMS Innovation Center Health Care Innovation Awards: Evaluation Plan.

The Center for Medicare and Medicaid Innovation within the Centers for Medicare & Medicaid Services (CMS) has funded 108 Health Care Innovation Awards, funded through the Affordable Care Act, for applicants who proposed compelling new models of service delivery or payment improvements that promise to deliver better health, better health care, and lower costs through improved quality of care for Medicare, Medicaid, and Children's Health Insurance Program enrollees. CMS is also interested in learning how new models would affect subpopulations of beneficiaries (e.g., those eligible for Medicare and Medicaid and complex patients) who have unique characteristics or health care needs that could be related to poor outcomes. In addition, the initiative seeks to identify new models of workforce development and deployment, as well as models that can be rapidly deployed and have the promise of sustainability. This article describes a strategy for evaluating the results. The goal for the evaluation design process is to create standardized approaches for answering key questions that can be customized to similar groups of awardees and that allow for rapid and comparable assessment across awardees. The evaluation plan envisions that data collection and analysis will be carried out on three levels: at the level of the individual awardee, at the level of the awardee grouping, and as a summary evaluation that includes all awardees. Key dimensions for the evaluation framework include implementation effectiveness, program effectiveness, workforce issues, impact on priority populations, and context. The ultimate goal is to identify strategies that can be employed widely to lower cost while improving care.

Health information exchange, Health Information Technology use, and hospital readmission rates.

The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 offers significant financial incentives to hospitals that can demonstrate "meaningful use" of EHRs. Reduced hospital readmissions are an expected outcome of improved care coordination. Increased use of HIT, and in particular participation in HIE are touted as ways to improve coordination of care. In a 2007 national sample of US hospitals, we evaluated the association between hospitals' HIE and HIT use and 30-day risk adjusted readmission rates for acute myocardial infarction (AMI), heart failure, and pneumonia. We found that hospital participation in HIE was not associated with lower hospital readmission rates; however, high levels of electronic documentation (an aspect of HIT use) were associated with modest reductions in readmission for heart failure (24.6% vs. 24.1%, P=.02) and pneumonia (18.4% vs. 17.9%, P=.003). More detailed data on participation in HIE are necessary to conduct more robust assessment of the relationship between HIE and hospital readmission rates.

Today's 'meaningful use' standard for medication orders by hospitals may save few lives; later stages may do more.

The federal government is currently offering bonus payments through Medicare and Medicaid to hospitals, physicians, and other eligible health professionals who meet new standards for "meaningful use" of health information technology. Whether these incentives will improve care, reduce errors, and improve patient safety as intended remains uncertain. We sought to partially fill this knowledge gap by evaluating the relationship between the use of electronic medication order entry and hospital mortality. Our results suggest that the initial meaningful-use threshold for hospitals-which requires using electronic orders for at least 30 percent of eligible patients-is probably too low to have a significant impact on deaths from heart failure and heart attack among hospitalized Medicare beneficiaries. However, the proposed threshold for the next stage of the program-using the orders for at least 60 percent of patients, a rate some stakeholders have said is too high-is more consistently associated with lower mortality. Our results suggest that the higher standard that will probably follow in the second stage of meaningful-use regulations would be more likely than the first-stage standard to produce the improved patient outcomes at the heart of the federal health information technology initiative.

The Role of Consumer-Controlled Personal Health Management Systems in the Evolution of Employer-Based Health Care Benefits.

The passage of the Patient Protection and Affordable Care Act has piqued employers' interest in new benefit designs because it includes numerous provisions that favor cost-reducing strategies, such as workplace wellness programs, value-based insurance design (VBID), and consumer-directed health plans (CDHPs). Consumer-controlled personal health management systems (HMSs) are a class of tools that provide encouragement, data, and decision support to individuals. Their functionalities fall into the following three categories: health information management, promotion of wellness and healthy lifestyles, and decision support. In this study, we review the evidence for many of the possible components of an HMS, including personal health records, web-based health risk assessments, integrated remote monitoring data, personalized health education and messaging, nutrition solutions and physical activity monitoring, diabetes-management solutions, medication reminders, vaccination and preventive-care applications, integrated incentive programs, social-networking tools, comparative data on price and value of providers, telehealth consultations, virtual coaching, and an integrated nurse hotline. The value of the HMS will be borne out as employers begin to adopt and implement these emerging technologies, enabling further assessment as their benefits and costs become better understood.

Electronic health record adoption and quality improvement in US hospitals.

OBJECTIVE: To estimate the relationship between quality improvement and electronic health record (EHR) adoption in US hospitals. STUDY DESIGN: National cohort study based on primary survey data about hospital EHR capability collected in 2003 and 2006 and on publicly reported hospital quality data for 2004 and 2007. METHODS: Difference-in-differences regression analysis to assess the relationship between EHR adoption and quality improvement for acute myocardial infarction, heart failure, and pneumonia care. RESULTS: Availability of a basic EHR was associated with a significant increase in quality improvement for heart failure (additional improvement, 2.6%; 95% confidence interval [CI], 1.0%-4.1%). However, adoption of advanced EHR capabilities was associated with significant decreases in quality improvement for acute myocardial infarction and heart failure. We observed 0.9% (95% CI, -1.7% to -0.1%) less improvement for acute myocardial infarction quality scores and 3.0% (95% CI, -5.2% to -0.8%) less improvement for heart failure quality scores among hospitals that newly adopted an advanced EHR, and 1.2% (95% CI, -2.0% to -0.3%) less improvement for acute myocardial infarction quality scores and 2.8% (95% CI, -5.4% to -0.3%) less improvement for heart failure quality scores among hospitals that upgraded their basic EHR. CONCLUSIONS: Mixed results suggest that current practices for implementation and use of EHRs have had a limited effect on quality improvement in US hospitals. However, potential "ceiling effects" limit the ability of existing measures to assess the effect that EHRs have had on hospital quality. In addition to the development of standard criteria for EHR functionality and use, standard measures of the effect of EHRs on quality are needed.

Forecasting emergency department crowding: an external, multicenter evaluation.

STUDY OBJECTIVE: We apply a previously described tool to forecast emergency department (ED) crowding at multiple institutions and assess its generalizability for predicting the near-future waiting count, occupancy level, and boarding count. METHODS: The ForecastED tool was validated with historical data from 5 institutions external to the development site. A sliding-window design separated the data for parameter estimation and forecast validation. Observations were sampled at consecutive 10-minute intervals during 12 months (n=52,560) at 4 sites and 10 months (n=44,064) at the fifth. Three outcome measures-the waiting count, occupancy level, and boarding count-were forecast 2, 4, 6, and 8 hours beyond each observation, and forecasts were compared with observed data at corresponding times. The reliability and calibration were measured following previously described methods. After linear calibration, the forecasting accuracy was measured with the median absolute error. RESULTS: The tool was successfully used for 5 different sites. Its forecasts were more reliable, better calibrated, and more accurate at 2 hours than at 8 hours. The reliability and calibration of the tool were similar between the original development site and external sites; the boarding count was an exception, which was less reliable at 4 of 5 sites. Some variability in accuracy existed among institutions; when forecasting 4 hours into the future, the median absolute error of the waiting count ranged between 0.6 and 3.1 patients, the median absolute error of the occupancy level ranged between 9.0% and 14.5% of beds, and the median absolute error of the boarding count ranged between 0.9 and 2.8 patients. CONCLUSION: The ForecastED tool generated potentially useful forecasts of input and throughput measures of ED crowding at 5 external sites, without modifying the underlying assumptions. Noting the limitation that this was not a real-time validation, ongoing research will focus on integrating the tool with ED information systems.

Patient-level analysis of outcomes using structured labor and delivery data.

This paper presents methods for identifying and analyzing associations among nursing care processes, patient attributes, and patient outcomes using unit-level and patient-level representations of care derived from computerized nurse documentation. The retrospective, descriptive analysis included documented nursing events for 900 Labor and Delivery patients at three hospitals over the 2-month period of January and February 2006. Two models were used to produce quantified measurements of nursing care received by each patient. The first model considered only the hourly census of nurses and patients. The second model considered the size of nurses' patient loads as represented by computerized nurse-entered documentation. Significant relationships were identified between durations of labor and nursing care scores generated by the second model. In addition to the clinical associations identified, the study demonstrated an approach with global application for representing the amount of nursing care received at the individual patient level in analyses of patient outcomes.

A multivariate time series approach to modeling and forecasting demand in the emergency department.

STUDY OBJECTIVE: The goals of this investigation were to study the temporal relationships between the demands for key resources in the emergency department (ED) and the inpatient hospital, and to develop multivariate forecasting models. METHODS: Hourly data were collected from three diverse hospitals for the year 2006. Descriptive analysis and model fitting were carried out using graphical and multivariate time series methods. Multivariate models were compared to a univariate benchmark model in terms of their ability to provide out-of-sample forecasts of ED census and the demands for diagnostic resources. RESULTS: Descriptive analyses revealed little temporal interaction between the demand for inpatient resources and the demand for ED resources at the facilities considered. Multivariate models provided more accurate forecasts of ED census and of the demands for diagnostic resources. CONCLUSION: Our results suggest that multivariate time series models can be used to reliably forecast ED patient census; however, forecasts of the demands for diagnostic resources were not sufficiently reliable to be useful in the clinical setting.

An agent based simulation tool for scheduling emergency department physicians.

Emergency department overcrowding is a problem that threatens the public health of communities and compromises the quality of care given to individual patients. The Institute of Medicine recommends that hospitals employ information technology and operations research methods to reduce overcrowding. This paper describes the development of an agent based simulation tool that has been designed to evaluate the impact of various physician staffing configurations on patient waiting times in the emergency department. We evaluate the feasibility of this tool at a single hospital emergency department.