Impact of using data from electronic protocols in nursing performance management: A qualitative interview study.

AIM: To explore the impact of using electronic data in performance management to improve nursing compliance with a protocol. BACKGROUND: Electronic data are increasingly used to monitor protocol compliance but little is known about the impact on nurses' practice in hospital wards. METHOD: Seventeen acute hospital nursing staff participated in semi-structured interviews about compliance with an early warning score (EWS) protocol delivered by a bedside electronic handheld device. RESULTS: Before electronic EWS data was used to monitor compliance, staff combined protocol-led actions with clinical judgement. However, some observations were missed to reduce noise and disruption at night. After compliance monitoring was introduced, observations were sometimes covertly omitted using a loophole. Interviewees described a loss of autonomy but acknowledged the EWS system sometimes flagged unexpected patient deterioration. CONCLUSIONS: Introducing automated electronic systems to support nursing tasks can decrease nursing burden but remove the ability to record legitimate reasons for missing observations. This can result in covert resistance that could reduce patient safety. IMPLICATIONS FOR NURSING MANAGEMENT: Providing the ability to log legitimate reasons for missing observations would allow nurses to balance professional judgement with the use of electronic data in performance management of protocol compliance.

National Early Warning Score Accurately Discriminates the Risk of Serious Adverse Events in Patients With Liver Disease.

BACKGROUND & AIMS: The National Early Warning Score (NEWS) is used to identify deteriorating adult hospital inpatients. However, it includes physiological parameters frequently altered in patients with cirrhosis. We aimed to assess the performance of the NEWS in acute and chronic liver diseases. METHODS: We collected vital signs, recorded in real time, from completed consecutive admissions of patients 16 years or older to a large acute-care hospital in Southern England, from January 1, 2010, through October 31, 2014. Using International Classification of Diseases, 10th revision, codes, we categorized patients as having primary liver disease, secondary liver disease, or none. For patients with liver disease, 2 analysis groups were developed: the first was based on clinical group (such as acute or chronic, alcohol-induced, or associated with portal hypertension), and the second was based on a summary of liver-related, hospital-level mortality indicator diagnoses. For each, we compared the abilities of the NEWS and 34 other early warning scores to discriminate 24-hour mortality, cardiac arrest, or unanticipated admission to the intensive care unit using the area under the receiver operating characteristic (AUROC) curve and early warning score efficiency curve analyses. RESULTS: The NEWS identified patients with primary, nonprimary, and no diagnoses of liver disease with AUROC values of 0.873 (95% CI, 0.860-0.886), 0.898 (95% CI, 0.891-0.905), and 0.879 (95% CI, 0.877-0.881), respectively. High AUROC values were also obtained for all clinical subgroups; the NEWS identified patients with alcohol-related liver disease with an AUROC value of 0.927 (95% CI, 0.912-0.941). The NEWS identified patients with liver diseases with higher AUROC values than other early warning scoring systems. CONCLUSIONS: The NEWS accurately discriminates patients at risk of death, admission to the intensive care unit, or cardiac arrest within a 24-hour period for a range of liver-related diagnoses. Its widespread use provides a ready-made, easy-to-use option for identifying patients with liver disease who require early assessment and intervention, without the need to modify parameters, weightings, or escalation criteria.

A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.

OBJECTIVES: The Sepsis-3 task force recommended the quick Sequential (Sepsis-Related) Organ Failure Assessment score for identifying patients with suspected infection who are at greater risk of poor outcomes, but many hospitals already use the National Early Warning Score to identify high-risk patients, irrespective of diagnosis. We sought to compare the performance of quick Sequential (Sepsis-Related) Organ Failure Assessment and National Early Warning Score in hospitalized, non-ICU patients with and without an infection. DESIGN: Retrospective cohort study. SETTING: Large U.K. General Hospital. PATIENTS: Adults hospitalized between January 1, 2010, and February 1, 2016. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We applied the quick Sequential (Sepsis-Related) Organ Failure Assessment score and National Early Warning Score to 5,435,344 vital signs sets (241,996 hospital admissions). Patients were categorized as having no infection, primary infection, or secondary infection using International Classification of Diseases, 10th Edition codes. National Early Warning Score was significantly better at discriminating in-hospital mortality, irrespective of infection status (no infection, National Early Warning Score 0.831 [0.825-0.838] vs quick Sequential [Sepsis-Related] Organ Failure Assessment 0.688 [0.680-0.695]; primary infection, National Early Warning Score 0.805 [0.799-0.812] vs quick Sequential [Sepsis-Related] Organ Failure Assessment 0.677 [0.670-0.685]). Similarly, National Early Warning Score performed significantly better in all patient groups (all admissions, emergency medicine admissions, and emergency surgery admissions) for all outcomes studied. Overall, quick Sequential (Sepsis-Related) Organ Failure Assessment performed no better, and often worse, in admissions with infection than without. CONCLUSIONS: The National Early Warning Score outperforms the quick Sequential (Sepsis-Related) Organ Failure Assessment score, irrespective of infection status. These findings suggest that quick Sequential (Sepsis-Related) Organ Failure Assessment should be reevaluated as the system of choice for identifying non-ICU patients with suspected infection who are at greater risk of poor outcome.

A fundamental conflict of care: Nurses' accounts of balancing patients' sleep with taking vital sign observations at night.

AIMS AND OBJECTIVES: To explore why adherence to vital sign observations scheduled by an early warning score protocol reduces at night. BACKGROUND: Regular vital sign observations can reduce avoidable deterioration in hospital. early warning score protocols set the frequency of these observations by the severity of a patient's condition. Vital sign observations are taken less frequently at night, even with an early warning score in place, but no literature has explored why. DESIGN: A qualitative interpretative design informed this study. METHODS: Seventeen semi-structured interviews with nursing staff working on wards with varying levels of adherence to scheduled vital sign observations. A thematic analysis approach was used. RESULTS: At night, nursing teams found it difficult to balance the competing care goals of supporting sleep with taking vital sign observations. The night-time frequency of these observations was determined by clinical judgement, ward-level expectations of observation timing and the risk of disturbing other patients. Patients with COPD or dementia could be under-monitored, while patients nearing the end of life could be over-monitored. CONCLUSION: In this study, we found an early warning score algorithm focused on deterioration prevention did not account for long-term management or palliative care trajectories. Nurses were therefore less inclined to wake such patients to take vital sign observations at night. However, the perception of widespread exceptions and lack of evidence regarding optimum frequency risks delegitimising the early warning score approach. This may pose a risk to patient safety, particularly patients with dementia or chronic conditions. RELEVANCE TO CLINICAL PRACTICE: Nurses should document exceptions and discuss these with the wider team. Hospitals should monitor why vital sign observations are missed at night, identify which groups are under-monitored and provide guidance on prioritising competing expectations. early warning score protocols should take account of different care trajectories.

Universal screening for alcohol misuse in acute medical admissions is feasible and identifies patients at high risk of liver disease.

BACKGROUND & AIMS: Many people who die from alcohol related liver disease (ARLD) have a history of recurrent admissions to hospital, representing potential missed opportunities for intervention. Universal screening for alcohol misuse has been advocated but it is not known if this is achievable or effective at detecting individuals at high risk of ARLD. METHODS: We systematically screened all admissions to the Acute Medical Unit (AMU) of a large acute hospital using an electronic data capture system in real time. Patients at an increasing risk of alcohol harm were referred for either brief intervention (BI) or further assessment by an Alcohol Specialist Nursing Service (ASNS). Additional data were recorded on admission diagnoses, alcohol unit consumption, previous attendances, previous admissions, length of stay and mortality. RESULTS: Between July 2011 and March 2014, there were 53,165 admissions and 48,211 (90.68%) completed screening. Of these, 1,122 (2.3%) were classified as "increasing", and 1,921 (4.0%) as "high" risk of alcohol harm. High risk patients had more hospital admissions in the three previous years (average 4.74) than the low (3.00) and increasing (2.92) risk groups (p<0.001). The high risk patients also had more frequent emergency department (ED) attendances (7.68) than the lower (2.64) and increasing (3.81) groups (p<0.001 for both). A total of 1,396 (72.6%) of the high risk group were seen by the ASNS and 1,135 (81.2%) had an Alcohol Use Disorders Identification Test (AUDIT) score over 20 with 527 (37.8%) recording the maximum value of 40. Compared to the other groups, high risk patients had a distinct profile of admissions with the most common diagnoses being mental health disorders, gastro-intestinal bleeding, poisoning and liver disease. CONCLUSIONS: Universal screening of admissions for alcohol misuse is feasible and identifies a cohort with frequent ED attendances, recurrent admissions and an elevated risk of ARLD. An additional group of patients at an increasing risk of alcohol harm can be identified in a range of common presentations. These patients can be targeted with interventions to reduce the burden of alcohol related harm. Lay summary: Many people who die from alcohol related liver disease (ARLD) have a recent history of recurrent admissions to hospital. These admissions may represent missed opportunities to intervene earlier and offer effective therapies for alcohol misuse. Unfortunately, we know that patients are often missed because medical staff may not routinely ask about alcohol consumption. In our study of over 50,000 admissions, we have demonstrated the feasibility of offering screening for alcohol misuse to all medical admissions to hospital and delivered this 24hours a day, 7days a week, with automatic referral to treatment services. We have shown that it is possible to identify those people who are at the highest risk of dependency, those who have attended the emergency department the most and those who are at an increased risk of ARLD. We hope this study will lead to improved detection and management of alcohol problems in acute hospitals.

A Comparison of the Ability of the Physiologic Components of Medical Emergency Team Criteria and the U.K. National Early Warning Score to Discriminate Patients at Risk of a Range of Adverse Clinical Outcomes.

OBJECTIVE: To compare the ability of medical emergency team criteria and the National Early Warning Score to discriminate cardiac arrest, unanticipated ICU admission and death within 24 hours of a vital signs measurement, and to quantify the associated workload. DESIGN: Retrospective cohort study. SETTING: A large U.K. National Health Service District General Hospital. PATIENTS: Adults hospitalized from May 25, 2011, to December 31, 2013. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We applied the National Early Warning Score and 44 sets of medical emergency team criteria to a database of 2,245,778 vital signs sets (103,998 admissions). The National Early Warning Score's performance was assessed using the area under the receiver-operating characteristic curve and compared with sensitivity/specificity for different medical emergency team criteria. Area under the receiver-operating characteristic curve (95% CI) for the National Early Warning Score for the combined outcome (i.e., death, cardiac arrest, or unanticipated ICU admission) was 0.88 (0.88-0.88). A National Early Warning Score value of 7 had sensitivity/specificity values of 44.5% and 97.4%, respectively. For the 44 sets of medical emergency team criteria studied, sensitivity ranged from 19.6% to 71.2% and specificity from 71.5% to 98.5%. For all outcomes, the position of the National Early Warning Score receiver-operating characteristic curve was above and to the left of all medical emergency team criteria points, indicating better discrimination. Similarly, the positions of all medical emergency team criteria points were above and to the left of the National Early Warning Score efficiency curve, indicating higher workloads (trigger rates). CONCLUSIONS: When medical emergency team systems are compared to a National Early Warning Score value of greater than or equal to 7, some medical emergency team systems have a higher sensitivity than National Early Warning Score values of greater than or equal to 7. However, all of these medical emergency team systems have a lower specificity and would generate greater workloads.

Early warning scores: unravelling detection and escalation.

PURPOSE: The purpose of this paper is to increase understanding of how patient deterioration is detected and how clinical care escalates when early warning score (EWS) systems are used. DESIGN/METHODOLOGY/APPROACH: The authors critically review a recent National Early Warning Score paper published in IJHCQA using personal experience and EWS-related publications, and debate the difference between detection and escalation. FINDINGS: Incorrect EWS choice or poorly understood EWS escalation may result in unnecessary workloads forward and responding staff. PRACTICAL IMPLICATIONS: EWS system implementers may need to revisit their guidance materials; medical and nurse educators may need to expand the curriculum to improve EWS system understanding and use. ORIGINALITY/VALUE: The paper raises the EWS debate and alerts EWS users that scrutiny is required.

A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study.

AIMS: To compare the ability of the National Early Warning Score (NEWS) and the National Early Warning Score 2 (NEWS2) to identify patients at risk of in-hospital mortality and other adverse outcomes. METHODS: We undertook a multi-centre retrospective observational study at five acute hospitals from two UK NHS Trusts. Data were obtained from completed adult admissions who were not fit enough to be discharged alive on the day of admission. Diagnostic coding and oxygen prescriptions were used to identify patients with type II respiratory failure (T2RF). The primary outcome was in-hospital mortality within 24 h of a vital signs observation. Secondary outcomes included unanticipated intensive care unit admission or cardiac arrest within 24 h of a vital signs observation. Discrimination was assessed using the c-statistic. RESULTS: Among 251,266 adult admissions, 48,898 were identified to be at risk of T2RF by diagnostic coding. In this group, NEWS2 showed statistically significant lower discrimination (c-statistic, 95% CI) for identifying in-hospital mortality within 24 h (0.860, 0.857-0.864) than NEWS (0.881, 0.878-0.884). For 1394 admissions with documented T2RF, discrimination was similar for both systems: NEWS2 (0.841, 0.827-0.855), NEWS (0.862, 0.848-0.875). For all secondary endpoints, NEWS2 showed no improvements in discrimination. CONCLUSIONS: NEWS2 modifications to NEWS do not improve discrimination of adverse outcomes in patients with documented T2RF and decrease discrimination in patients at risk of T2RF. Further evaluation of the relationship between SpO2 values, oxygen therapy and risk should be investigated further before wide-scale adoption of NEWS2.

Nurse staffing, nursing assistants and hospital mortality: retrospective longitudinal cohort study.

OBJECTIVE: To determine the association between daily levels of registered nurse (RN) and nursing assistant staffing and hospital mortality. DESIGN: This is a retrospective longitudinal observational study using routinely collected data. We used multilevel/hierarchical mixed-effects regression models to explore the association between patient outcomes and daily variation in RN and nursing assistant staffing, measured as hours per patient per day relative to ward mean. Analyses were controlled for ward and patient risk. PARTICIPANTS: 138 133 adult patients spending >1 days on general wards between 1 April 2012 and 31 March 2015. OUTCOMES: In-hospital deaths. RESULTS: Hospital mortality was 4.1%. The hazard of death was increased by 3% for every day a patient experienced RN staffing below ward mean (adjusted HR (aHR) 1.03, 95% CI 1.01 to 1.05). Relative to ward mean, each additional hour of RN care available over the first 5 days of a patient's stay was associated with 3% reduction in the hazard of death (aHR 0.97, 95% CI 0.94 to 1.0). Days where admissions per RN exceeded 125% of the ward mean were associated with an increased hazard of death (aHR 1.05, 95% CI 1.01 1.09). Although low nursing assistant staffing was associated with increases in mortality, high nursing assistant staffing was also associated with increased mortality. CONCLUSION: Lower RN staffing and higher levels of admissions per RN are associated with increased risk of death during an admission to hospital. These findings highlight the possible consequences of reduced nurse staffing and do not give support to policies that encourage the use of nursing assistants to compensate for shortages of RNs.

Review and performance evaluation of aggregate weighted 'track and trigger' systems.

OBJECTIVES: There is no up-to-date literature review of physiologically based, aggregate weighted 'track and trigger' systems (AWTTS) and few data on their predictive ability for serious adverse outcomes. The aim of this study was to describe the AWTTS in clinical use and assess their ability to discriminate between survivors and non-survivors of hospital admission, based on an initial set of vital signs. MATERIALS AND METHODS: A systematic review of the literature was performed, to describe the AWTTS, their components and their differences. Their ability to discriminate between survivors and non-survivors was evaluated using the area under the receiver-operating characteristics (AUROC) curve, and a database of 9987 vital signs datasets. RESULTS: A total of 33 unique AWTTS were identified with AUROC (+/-95% CI) ranging from 0.657 (0.636-0.678) to 0.782 (0.767-0.797). 12 AWTTS (36%) discriminated reasonably well between survivors and non-survivors, the top four performing AWTTS incorporated age as a component (AUROCs ranging from 0.722 to 0.782). The top two systems also incorporated temperature. CONCLUSIONS: There is a wide range of unique, but very similar, AWTTS in clinical use. There is no consistency regarding their physiological components, but the majority differ only in minor variations in the weightings for physiological derangement and/or the cut-off points between physiological weighting bands. The performance of most systems tested was poor when used to discriminate between survivors and non-survivors, although 36% discriminated reasonably well. Our results suggest that physiology can be used to predict outcome, but that further work is required to improve the AWTTS models.

Should age be included as a component of track and trigger systems used to identify sick adult patients?

AIM OF STUDY: Few published "track and trigger systems" used to identify sick adult patients incorporate patient age as a variable. We investigated the relationship between vital signs, patient age and in-hospital mortality and investigated the impact of patient age on the function as predictors of in-hospital mortality of the two most commonly used track and trigger systems. MATERIALS AND METHODS: Using a database of 9987 vital signs datasets, we studied the relationship between admission vital signs and in-hospital mortality for a range of selected vital signs, grouped by patient age. We also used the vital signs data set to study the impact of patient age on the relationship between patient triggers using the "MET criteria" and "MEWS", and in-hospital mortality. RESULTS: At hospital discharge, there were 9152 (91.6%) survivors and 835 (8.4%) non-survivors. As admission vital signs worsened, mortality increased for each age range. Where groups of patients had triggered a certain MET criterion, mortality was higher as patient age increased. Mortality varied significantly with age (p<0.05; Fishers exact test) for breathing rate >36breathsmin(-1), systolic BP<90mmHg and decreased conscious level. For each age group, mortality also increased as total MEWS score increased. As the number of simultaneously occurring MEWS abnormalities, or simultaneously occurring MET criteria, increased, mortality increased for each age range. CONCLUSIONS: Age has a significant impact on in-hospital mortality. Our data suggest that the inclusion of age as a component of these systems could be advantageous in improving their function.

A review, and performance evaluation, of single-parameter "track and trigger" systems.

OBJECTIVES: There is no up-to-date literature review of physiologically-based, single-parameter weighted "track and trigger" systems (SPTTS) and little data on their sensitivity and specificity to predict adverse outcomes. The aim of this study was to describe the SPTTS in clinical use and measure their sensitivity and specificity when using admission vital signs data for predicting in-hospital mortality. MATERIALS AND METHODS: We performed a systematic review of the literature to describe the SPTTS, their components and their differences. We measured their sensitivity and specificity for predicting in-hospital mortality when using a database of 9987 admission vital signs datasets. RESULTS: We identified 39 unique classes of SPTTS, of which 30 were evaluated. There was considerable variation in the physiological variables used, together with significant variation in the physiological values used to trigger a medical emergency or critical care outreach team. There was marked variation in sensitivity (7.3-52.8%), specificity (69.1-98.1%), positive predictive values (13.5-26.1%), negative predictive values (92.1-94.2%) and the potential number of calls triggered (234-3271). CONCLUSIONS: There is a wide range of unique, but very similar, SPTTS in clinical use. Although specificities were high, sensitivities were too low to provide institutions with confidence that these SPTTS could identify patients at risk of in-hospital death using admission vital signs. Institutions may wish to consider these data when selecting which, if any, single-parameter track and trigger systems to introduce.

Aggregate National Early Warning Score (NEWS) values are more important than high scores for a single vital signs parameter for discriminating the risk of adverse outcomes.

INTRODUCTION: The Royal College of Physicians (RCPL) National Early Warning Score (NEWS) escalates care to a doctor at NEWS values of ≥5 and when the score for any single vital sign is 3. METHODS: We calculated the 24-h risk of serious clinical outcomes for vital signs observation sets with NEWS values of 3, 4 and 5, separately determining risks when the score did/did not include a single score of 3. We compared workloads generated by the RCPL's escalation protocol and for aggregate NEWS value alone. RESULTS: Aggregate NEWS values of 3 or 4 (n=142,282) formed 15.1% of all vital signs sets measured; those containing a single vital sign scoring 3 (n=36,207) constituted 3.8% of all sets. Aggregate NEWS values of either 3 or 4 with a component score of 3 have significantly lower risks (OR: 0.26 and 0.53) than an aggregate value of 5 (OR: 1.0). Escalating care to a doctor when any single component of NEWS scores 3 compared to when aggregate NEWS values ≥5, would have increased doctors' workload by 40% with only a small increase in detected adverse outcomes from 2.99 to 3.08 per day (a 3% improvement in detection). CONCLUSIONS: The recommended NEWS escalation protocol produces additional work for the bedside nurse and responding doctor, disproportionate to a modest benefit in increased detection of adverse outcomes. It may have significant ramifications for efficient staff resource allocation, distort patient safety focus and risk alarm fatigue. Our findings suggest that the RCPL escalation guidance warrants review.

Impact of introducing an electronic physiological surveillance system on hospital mortality.

BACKGROUND: Avoidable hospital mortality is often attributable to inadequate patient vital signs monitoring, and failure to recognise or respond to clinical deterioration. The processes involved with vital sign collection and charting; their integration, interpretation and analysis; and the delivery of decision support regarding subsequent clinical care are subject to potential error and/or failure. OBJECTIVE: To determine whether introducing an electronic physiological surveillance system (EPSS), specifically designed to improve the collection and clinical use of vital signs data, reduced hospital mortality. METHODS: A pragmatic, retrospective, observational study of seasonally adjusted in-hospital mortality rates in three main hospital specialties was undertaken before, during and after the sequential deployment and ongoing use of a hospital-wide EPSS in two large unconnected acute general hospitals in England. The EPSS, which uses wireless handheld computing devices, replaced a paper-based vital sign charting and clinical escalation system. RESULTS: During EPSS implementation, crude mortality fell from a baseline of 7.75% (2168/27,959) to 6.42% (1904/29,676) in one hospital (estimated 397 fewer deaths), and from 7.57% (1648/21,771) to 6.15% (1614/26,241) at the second (estimated 372 fewer deaths). At both hospitals, multiyear statistical process control analyses revealed abrupt and sustained mortality reductions, coincident with the deployment and increasing use of the system. The cumulative total of excess deaths reduced in all specialties with increasing use of the system across the hospital. CONCLUSIONS: The use of technology specifically designed to improve the accuracy, reliability and availability of patients' vital signs and early warning scores, and thereby the recognition of and response to patient deterioration, is associated with reduced mortality in this study.

Can binary early warning scores perform as well as standard early warning scores for discriminating a patient's risk of cardiac arrest, death or unanticipated intensive care unit admission?

INTRODUCTION: Although the weightings to be summed in an early warning score (EWS) calculation are small, calculation and other errors occur frequently, potentially impacting on hospital efficiency and patient care. Use of a simpler EWS has the potential to reduce errors. METHODS: We truncated 36 published 'standard' EWSs so that, for each component, only two scores were possible: 0 when the standard EWS scored 0 and 1 when the standard EWS scored greater than 0. Using 1564,153 vital signs observation sets from 68,576 patient care episodes, we compared the discrimination (measured using the area under the receiver operator characteristic curve--AUROC) of each standard EWS and its truncated 'binary' equivalent. RESULTS: The binary EWSs had lower AUROCs than the standard EWSs in most cases, although for some the difference was not significant. One system, the binary form of the National Early Warning System (NEWS), had significantly better discrimination than all standard EWSs, except for NEWS. Overall, Binary NEWS at a trigger value of 3 would detect as many adverse outcomes as are detected by NEWS using a trigger of 5, but would require a 15% higher triggering rate. CONCLUSIONS: The performance of Binary NEWS is only exceeded by that of standard NEWS. It may be that Binary NEWS, as a simplified system, can be used with fewer errors. However, its introduction could lead to significant increases in workload for ward and rapid response team staff. The balance between fewer errors and a potentially greater workload needs further investigation.

Are observation selection methods important when comparing early warning score performance?

INTRODUCTION: Sicker patients generally have more vital sign assessments, particularly immediately before an adverse outcome, and especially if the vital sign monitoring schedule is driven by an early warning score (EWS) value. This lack of independence could influence the measured discriminatory performance of an EWS. METHODS: We used a population of 1564,143 consecutive vital signs observation sets collected as a routine part of patients' care. We compared 35 published EWSs for their discrimination of the risk of death within 24h of an observation set using (1) all observations in our dataset, (2) one observation per patient care episode, chosen at random and (3) one observation per patient care episode, chosen as the closest to a randomly selected point in time in each episode. We compared the area under the ROC curve (AUROC) as a measure of discrimination for each of the 35 EWSs under each observation selection method and looked for changes in their rank order. RESULTS: There were no significant changes in rank order of the EWSs based on AUROC between the different observation selection methods, except for one EWS that included age among its components. Whichever method of observation selection was used, the National Early Warning Score (NEWS) showed the highest discrimination of risk of death within 24h. AUROCs were higher when only one observation set was used per episode of care (significantly higher for many EWSs, including NEWS). CONCLUSIONS: Vital sign measurements can be treated as if they are independent - multiple observations can be used from each episode of care--when comparing the performance and ranking of EWSs, provided no EWS includes age.