Accuracy of automated computer-aided risk scoring systems to estimate the risk of COVID-19: a retrospective cohort study.

BACKGROUND: In the UK National Health Service (NHS), the patient's vital signs are monitored and summarised into a National Early Warning Score (NEWS) score. A set of computer-aided risk scoring systems (CARSS) was developed and validated for predicting in-hospital mortality and sepsis in unplanned admission to hospital using NEWS and routine blood tests results. We sought to assess the accuracy of these models to predict the risk of COVID-19 in unplanned admissions during the first phase of the pandemic. METHODS: Adult ( > = 18 years) non-elective admissions discharged (alive/deceased) between 11-March-2020 to 13-June-2020 from two acute hospitals with an index NEWS electronically recorded within ± 24 h of admission. We identified COVID-19 admission based on ICD-10 code 'U071' which was determined by COVID-19 swab test results (hospital or community). We assessed the performance of CARSS (CARS_N, CARS_NB, CARM_N, CARM_NB) for predicting the risk of COVID-19 in terms of discrimination (c-statistic) and calibration (graphically). RESULTS: The risk of in-hospital mortality following emergency medical admission was 8.4% (500/6444) and 9.6% (620/6444) had a diagnosis of COVID-19. For predicting COVID-19 admissions, the CARS_N model had the highest discrimination 0.73 (0.71 to 0.75) and calibration slope 0.81 (0.72 to 0.89) compared to other CARSS models: CARM_N (discrimination:0.68 (0.66 to 0.70) and calibration slope 0.47 (0.41 to 0.54)), CARM_NB (discrimination:0.68 (0.65 to 0.70) and calibration slope 0.37 (0.31 to 0.43)), and CARS_NB (discrimination:0.68 (0.66 to 0.70) and calibration slope 0.56 (0.47 to 0.64)). CONCLUSIONS: The CARS_N model is reasonably accurate for predicting the risk of COVID-19. It may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned admissions because it requires no additional data collection and is readily automated.

Development and validation of automated computer-aided risk scores to predict in-hospital mortality for emergency medical admissions with COVID-19: a retrospective cohort development and validation study.

OBJECTIVES: There are no established mortality risk equations specifically for unplanned emergency medical admissions which include patients with SARS-19 (COVID-19). We aim to develop and validate a computer-aided risk score (CARMc19) for predicting mortality risk by combining COVID-19 status, the first electronically recorded blood test results and the National Early Warning Score (NEWS2). DESIGN: Logistic regression model development and validation study. SETTING: Two acute hospitals (York Hospital-model development data; Scarborough Hospital-external validation data). PARTICIPANTS: Adult (aged ≥16 years) medical admissions discharged over a 24-month period with electronic NEWS and blood test results recorded on admission. We used logistic regression modelling to predict the risk of in-hospital mortality using two models: (1) CARMc19_N: age+sex+NEWS2 including subcomponents+COVID19; (2) CARMc19_NB: CARMc19_N in conjunction with seven blood test results and acute kidney injury score. Model performance was evaluated according to discrimination (c-statistic), calibration (graphically) and clinical usefulness at NEWS2 thresholds of 4+, 5+, 6+. RESULTS: The risk of in-hospital mortality following emergency medical admission was similar in development and validation datasets (8.4% vs 8.2%). The c-statistics for predicting mortality for CARMc19_NB is better than CARMc19_N in the validation dataset (CARMc19_NB=0.88 (95% CI 0.86 to 0.90) vs CARMc19_N=0.86 (95% CI 0.83 to 0.88)). Both models had good calibration (CARMc19_NB=1.01 (95% CI 0.88 to 1.14) and CARMc19_N:0.95 (95% CI 0.83 to 1.06)). At all NEWS2 thresholds (4+, 5+, 6+) model, CARMc19_NB had better sensitivity and similar specificity. CONCLUSIONS: We have developed a validated CARMc19 scores with good performance characteristics for predicting the risk of in-hospital mortality. Since the CARMc19 scores place no additional data collection burden on clinicians, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

Predictive accuracy of enhanced versions of the on-admission National Early Warning Score in estimating the risk of COVID-19 for unplanned admission to hospital: a retrospective development and validation study.

BACKGROUND: The novel coronavirus SARS-19 produces 'COVID-19' in patients with symptoms. COVID-19 patients admitted to the hospital require early assessment and care including isolation. The National Early Warning Score (NEWS) and its updated version NEWS2 is a simple physiological scoring system used in hospitals, which may be useful in the early identification of COVID-19 patients. We investigate the performance of multiple enhanced NEWS2 models in predicting the risk of COVID-19. METHODS: Our cohort included unplanned adult medical admissions discharged over 3 months (11 March 2020 to 13 June 2020 ) from two hospitals (YH for model development; SH for external model validation). We used logistic regression to build multiple prediction models for the risk of COVID-19 using the first electronically recorded NEWS2 within ± 24 hours of admission. Model M0' included NEWS2; model M1' included NEWS2 + age + sex, and model M2' extends model M1' with subcomponents of NEWS2 (including diastolic blood pressure + oxygen flow rate + oxygen scale). Model performance was evaluated according to discrimination (c statistic), calibration (graphically), and clinical usefulness at NEWS2 ≥ 5. RESULTS: The prevalence of COVID-19 was higher in SH (11.0 %=277/2520) than YH (8.7 %=343/3924) with a higher first NEWS2 scores ( SH 3.2 vs YH 2.8) but similar in-hospital mortality (SH 8.4 % vs YH 8.2 %). The c-statistics for predicting the risk of COVID-19 for models M0',M1',M2' in the development dataset were: M0': 0.71 (95 %CI 0.68-0.74); M1': 0.67 (95 %CI 0.64-0.70) and M2': 0.78 (95 %CI 0.75-0.80)). For the validation datasets the c-statistics were: M0' 0.65 (95 %CI 0.61-0.68); M1': 0.67 (95 %CI 0.64-0.70) and M2': 0.72 (95 %CI 0.69-0.75) ). The calibration slope was similar across all models but Model M2' had the highest sensitivity (M0' 44 % (95 %CI 38-50 %); M1' 53 % (95 %CI 47-59 %) and M2': 57 % (95 %CI 51-63 %)) and specificity (M0' 75 % (95 %CI 73-77 %); M1' 72 % (95 %CI 70-74 %) and M2': 76 % (95 %CI 74-78 %)) for the validation dataset at NEWS2 ≥ 5. CONCLUSIONS: Model M2' appears to be reasonably accurate for predicting the risk of COVID-19. It may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned hospital admissions.

Use of the first National Early Warning Score recorded within 24 hours of admission to estimate the risk of in-hospital mortality in unplanned COVID-19 patients: a retrospective cohort study.

OBJECTIVES: Although the National Early Warning Score (NEWS) and its latest version NEWS2 are recommended for monitoring deterioration in patients admitted to hospital, little is known about their performance in COVID-19 patients. We aimed to compare the performance of the NEWS and NEWS2 in patients with COVID-19 versus those without during the first phase of the pandemic. DESIGN: A retrospective cross-sectional study. SETTING: Two acute hospitals (Scarborough and York) are combined into a single dataset and analysed collectively. PARTICIPANTS: Adult (≥18 years) non-elective admissions discharged between 11 March 2020 and 13 June 2020 with an index or on-admission NEWS2 electronically recorded within ±24 hours of admission to predict mortality at four time points (in-hospital, 24 hours, 48 hours and 72 hours) in COVID-19 versus non-COVID-19 admissions. RESULTS: Out of 6480 non-elective admissions, 620 (9.6%) had a diagnosis of COVID-19. They were older (73.3 vs 67.7 years), more often male (54.7% vs 50.1%), had higher index NEWS (4 vs 2.5) and NEWS2 (4.6 vs 2.8) scores and higher in-hospital mortality (32.1% vs 5.8%). The c-statistics for predicting in-hospital mortality in COVID-19 admissions was significantly lower using NEWS (0.64 vs 0.74) or NEWS2 (0.64 vs 0.74), however, these differences reduced at 72hours (NEWS: 0.75 vs 0.81; NEWS2: 0.71 vs 0.81), 48 hours (NEWS: 0.78 vs 0.81; NEWS2: 0.76 vs 0.82) and 24hours (NEWS: 0.84 vs 0.84; NEWS2: 0.86 vs 0.84). Increasing NEWS2 values reflected increased mortality, but for any given value the absolute risk was on average 24% higher (eg, NEWS2=5: 36% vs 9%). CONCLUSIONS: The index or on-admission NEWS and NEWS2 offers lower discrimination for COVID-19 admissions versus non-COVID-19 admissions. The index NEWS2 was not proven to be better than the index NEWS. For each value of the index NEWS/NEWS2, COVID-19 admissions had a substantially higher risk of mortality than non-COVID-19 admissions which reflects the increased baseline mortality risk of COVID-19.

A comparison of logistic regression models with alternative machine learning methods to predict the risk of in-hospital mortality in emergency medical admissions via external validation.

We compare the performance of logistic regression with several alternative machine learning methods to estimate the risk of death for patients following an emergency admission to hospital based on the patients' first blood test results and physiological measurements using an external validation approach. We trained and tested each model using data from one hospital (n = 24,696) and compared the performance of these models in data from another hospital (n = 13,477). We used two performance measures - the calibration slope and area under the receiver operating characteristic curve. The logistic model performed reasonably well - calibration slope: 0.90, area under the receiver operating characteristic curve: 0.847 compared to the other machine learning methods. Given the complexity of choosing tuning parameters of these methods, the performance of logistic regression with transformations for in-hospital mortality prediction was competitive with the best performing alternative machine learning methods with no evidence of overfitting.

Performance of externally validated enhanced computer-aided versions of the National Early Warning Score in predicting mortality following an emergency admission to hospital in England: a cross-sectional study.

OBJECTIVES: In the English National Health Service, the patient's vital signs are monitored and summarised into a National Early Warning Score (NEWS) to support clinical decision making, but it does not provide an estimate of the patient's risk of death. We examine the extent to which the accuracy of NEWS for predicting mortality could be improved by enhanced computer versions of NEWS (cNEWS). DESIGN: Logistic regression model development and external validation study. SETTING: Two acute hospitals (YH-York Hospital for model development; NH-Northern Lincolnshire and Goole Hospital for external model validation). PARTICIPANTS: Adult (≥16 years) medical admissions discharged over a 24-month period with electronic NEWS (eNEWS) recorded on admission are used to predict mortality at four time points (in-hospital, 24 hours, 48 hours and 72 hours) using the first electronically recorded NEWS (model M0) versus a cNEWS model which included age+sex (model M1) +subcomponents of NEWS (including diastolic blood pressure) (model M2). RESULTS: The risk of dying in-hospital following emergency medical admission was 5.8% (YH: 2080/35 807) and 5.4% (NH: 1900/35 161). The c-statistics for model M2 in YH for predicting mortality (in-hospital=0.82, 24 hours=0.91, 48 hours=0.88 and 72 hours=0.88) was higher than model M0 (in-hospital=0.74, 24 hours=0.89, 48 hours=0.86 and 72 hours=0.85) with higher Positive Predictive Value (PPVs) for in-hospital mortality (M2 19.3% and M0 16.6%). Similar findings were seen in NH. Model M2 performed better than M0 in almost all major disease subgroups. CONCLUSIONS: An externally validated enhanced computer-aided NEWS model (cNEWS) incrementally improves on the performance of a NEWS only model. Since cNEWS places no additional data collection burden on clinicians and is readily automated, it may now be carefully introduced and evaluated to determine if it can improve care in hospitals that have eNEWS systems.

Computer-aided National Early Warning Score to predict the risk of sepsis following emergency medical admission to hospital: a model development and external validation study.

BACKGROUND: In hospitals in England, patients' vital signs are monitored and summarized into the National Early Warning Score (NEWS); this score is more accurate than the Quick Sepsis-related Organ Failure Assessment (qSOFA) score at identifying patients with sepsis. We investigated the extent to which the accuracy of the NEWS is enhanced by developing and comparing 3 computer-aided NEWS (cNEWS) models (M0 = NEWS alone, M1 = M0 + age + sex, M2 = M1 + subcomponents of NEWS + diastolic blood pressure) to predict the risk of sepsis. METHODS: We included all emergency medical admissions of patients 16 years of age and older discharged over 24 months from 2 acute care hospital centres (York Hospital [YH] for model development and a combined data set from 2 hospitals [Diana, Princess of Wales Hospital and Scunthorpe General Hospital] in the Northern Lincolnshire and Goole National Health Service Foundation Trust [NH] for external model validation). We used a validated Canadian method for defining sepsis from administrative hospital data. RESULTS: The prevalence of sepsis was lower in YH (4.5%, 1596/35 807) than in NH (8.5%, 2983/35 161). The C statistic increased across models (YH: M0 0.705, M1 0.763, M2 0.777; NH: M0 0.708, M1 0.777, M2 0.791). For NEWS of 5 or higher, sensitivity increased (YH: 47.24% v. 50.56% v. 52.69%; NH: 37.91% v. 43.35% v. 48.07%), the positive likelihood ratio increased (YH: 2.77 v. 2.99 v. 3.06; NH: 3.18 v. 3.32 v. 3.45) and the positive predictive value increased (YH: 11.44% v. 12.24% v. 12.49%; NH: 22.75% v. 23.55% v. 24.21%). INTERPRETATION: From the 3 cNEWS models, model M2 is the most accurate. Given that it places no additional burden of data collection on clinicians and can be automated, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

The inclusion of delirium in version 2 of the National Early Warning Score will substantially increase the alerts for escalating levels of care: findings from a retrospective database study of emergency medical admissions in two hospitals .

BACKGROUND: The National Early Warning Score (NEWS) is being replaced with NEWS2 which adds 3 points for new confusion or delirium. We estimated the impact of adding delirium on the number of medium/high level alerts that are triggers to escalate care. METHODS: Analysis of emergency medical admissions in two acute hospitals (York Hospital (YH) and Northern Lincolnshire and Goole NHS Foundation Trust hospitals (NH)) in England. Twenty per cent were randomly assigned to have delirium. RESULTS: The number of emergency admissions (YH: 35584; NH: 35795), mortality (YH: 5.7%; NH: 5.5%), index NEWS (YH: 2.5; NH: 2.1) and numbers of NEWS recorded (YH: 879193; NH: 884072) were similar in each hospital. The mean number of patients with medium level alerts per day increased from 55.3 (NEWS) to 69.5 (NEWS2), a 25.7% increase in YH and 64.1 (NEWS) to 77.4 (NEWS2), a 20.7% increase in NH. The mean number of patients with high level alerts per day increased from 27.3 (NEWS) to 34.4 (NEWS2), a 26.0% increase in YH and 29.9 (NEWS) to 37.7 (NEWS2), a 26.1% increase in NH. CONCLUSIONS: The addition of delirium in NEWS2 will have a substantial increase in medium and high level alerts in hospitalised emergency medical patients. Rigorous evaluation of NEWS2 is required before widespread implementation because the extent to which staff can cope with this increase without adverse consequences remains unknown.

Development and validation of a novel computer-aided score to predict the risk of in-hospital mortality for acutely ill medical admissions in two acute hospitals using their first electronically recorded blood test results and vital signs: a cross-sectional study.

OBJECTIVES: There are no established mortality risk equations specifically for emergency medical patients who are admitted to a general hospital ward. Such risk equations may be useful in supporting the clinical decision-making process. We aim to develop and externally validate a computer-aided risk of mortality (CARM) score by combining the first electronically recorded vital signs and blood test results for emergency medical admissions. DESIGN: Logistic regression model development and external validation study. SETTING: Two acute hospitals (Northern Lincolnshire and Goole NHS Foundation Trust Hospital (NH)-model development data; York Hospital (YH)-external validation data). PARTICIPANTS: Adult (aged ≥16 years) medical admissions discharged over a 24-month period with electronic National Early Warning Score(s) and blood test results recorded on admission. RESULTS: The risk of in-hospital mortality following emergency medical admission was 5.7% (NH: 1766/30 996) and 6.5% (YH: 1703/26 247). The C-statistic for the CARM score in NH was 0.87 (95% CI 0.86 to 0.88) and was similar in an external hospital setting YH (0.86, 95% CI 0.85 to 0.87) and the calibration slope included 1 (0.97, 95% CI 0.94 to 1.00). CONCLUSIONS: We have developed a novel, externally validated CARM score with good performance characteristics for estimating the risk of in-hospital mortality following an emergency medical admission using the patient's first, electronically recorded, vital signs and blood test results. Since the CARM score places no additional data collection burden on clinicians and is readily automated, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

The National Early Warning Score and its subcomponents recorded within ±24 h of emergency medical admission are poor predictors of hospital-acquired acute kidney injury.

Hospital-acquired acute kidney injury (H-AKI) is a common cause of avoidable morbidity and mortality. Therefore, in the current study, we investigated whether vital signs data from patients, as defined by a National Early Warning Score (NEWS), can predict H-AKI following emergency admission to hospital. We analysed all emergency admissions (n=33,608) to York Hospital with NEWS data over a 24-month period. Here, we report the area under the curve (AUC) for logistic regression models that used the index NEWS (model A0), plus age and sex (A1), plus subcomponents of NEWS (A2) and two-way interactions (A3), and similarly for maximum NEWS (models B0,B1,B2,B3). Of the total emergency admissions, 4.05% (1,361/33,608) had H-AKI. Models using the index NEWS had lower AUCs (0.59-0.68) than models using the maximum NEWS AUCs (0.75-0.77). The maximum NEWS model (B3) was more sensitive than the index NEWS model (A0) (67.60% vs 19.84%) but identified twice as many cases as being at risk of H-AKI (9581 vs 4099) at a NEWS of 5. Based on these results, we suggest that the index NEWS is a poor predictor of H-AKI. The maximum NEWS is a better predictor but appears to be unfeasible because it is only knowable in retrospect and is associated with a substantial increase in workload, albeit with improved sensitivity.

Development and External Validation of an Automated Computer-Aided Risk Score for Predicting Sepsis in Emergency Medical Admissions Using the Patient's First Electronically Recorded Vital Signs and Blood Test Results.

OBJECTIVES: To develop a logistic regression model to predict the risk of sepsis following emergency medical admission using the patient's first, routinely collected, electronically recorded vital signs and blood test results and to validate this novel computer-aided risk of sepsis model, using data from another hospital. DESIGN: Cross-sectional model development and external validation study reporting the C-statistic based on a validated optimized algorithm to identify sepsis and severe sepsis (including septic shock) from administrative hospital databases using International Classification of Diseases, 10th Edition, codes. SETTING: Two acute hospitals (York Hospital - development data; Northern Lincolnshire and Goole Hospital - external validation data). PATIENTS: Adult emergency medical admissions discharged over a 24-month period with vital signs and blood test results recorded at admission. INTERVENTIONS: None. MAIN RESULTS: The prevalence of sepsis and severe sepsis was lower in York Hospital (18.5% = 4,861/2,6247; 5.3% = 1,387/2,6247) than Northern Lincolnshire and Goole Hospital (25.1% = 7,773/30,996; 9.2% = 2,864/30,996). The mortality for sepsis (York Hospital: 14.5% = 704/4,861; Northern Lincolnshire and Goole Hospital: 11.6% = 899/7,773) was lower than the mortality for severe sepsis (York Hospital: 29.0% = 402/1,387; Northern Lincolnshire and Goole Hospital: 21.4% = 612/2,864). The C-statistic for computer-aided risk of sepsis in York Hospital (all sepsis 0.78; sepsis: 0.73; severe sepsis: 0.80) was similar in an external hospital setting (Northern Lincolnshire and Goole Hospital: all sepsis 0.79; sepsis: 0.70; severe sepsis: 0.81). A cutoff value of 0.2 gives reasonable performance. CONCLUSIONS: We have developed a novel, externally validated computer-aided risk of sepsis, with reasonably good performance for estimating the risk of sepsis for emergency medical admissions using the patient's first, electronically recorded, vital signs and blood tests results. Since computer-aided risk of sepsis places no additional data collection burden on clinicians and is automated, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

Impact of the level of sickness on higher mortality in emergency medical admissions to hospital at weekends.

Objective Routine administrative data have been used to show that patients admitted to hospitals over the weekend appear to have a higher mortality compared to weekday admissions. Such data do not take the severity of sickness of a patient on admission into account. Our aim was to incorporate a standardized vital signs physiological-based measure of sickness known as the National Early Warning Score to investigate if weekend admissions are: sicker as measured by their index National Early Warning Score; have an increased mortality; and experience longer delays in the recording of their index National Early Warning Score. Methods We extracted details of all adult emergency medical admissions during 2014 from hospital databases and linked these with electronic National Early Warning Score data in four acute hospitals. We analysed 47,117 emergency admissions after excluding 1657 records, where National Early Warning Score was missing or the first (index) National Early Warning Score was recorded outside ±24 h of the admission time. Results Emergency medical admissions at the weekend had higher index National Early Warning Score (weekend: 2.53 vs. weekday: 2.30, p < 0.001) with a higher mortality (weekend: 706/11,332 6.23% vs. weekday: 2039/35,785 5.70%; odds ratio = 1.10, 95% CI 1.01 to 1.20, p = 0.04) which was no longer seen after adjusting for the index National Early Warning Score (odds ratio = 0.99, 95% CI 0.90 to 1.09, p = 0.87). Index National Early Warning Score was recorded sooner (-0.45 h, 95% CI -0.52 to -0.38, p < 0.001) for weekend admissions. Conclusions Emergency medical admissions at the weekend with electronic National Early Warning Score recorded within 24 h are sicker, have earlier clinical assessments, and after adjusting for the severity of their sickness, do not appear to have a higher mortality compared to weekday admissions. A larger definitive study to confirm these findings is needed.

Quality review of an adverse incident reporting system and root cause analysis of serious adverse surgical incidents in a teaching hospital of Scotland.

BACKGROUND: A significant proportion of surgical patients are unintentionally harmed during their hospital stay. Root Cause Analysis (RCA) aims to determine the aetiology of adverse incidents that lead to patient harm and produce a series of recommendations, which would minimise the risk of recurrence of similar events, if appropriately applied to clinical practice. A review of the quality of the adverse incident reporting system and the RCA of serious adverse incidents at the Department of Surgery of Ninewells hospital, in Dundee, United Kingdom was performed. METHODS: The Adverse Incident Management (AIM) database of the Department of Surgery of Ninewells Hospital was retrospectively reviewed. Details of all serious (red, sentinel) incidents recorded between May 2004 and December 2009, including the RCA reports and outcomes, where applicable, were reviewed. Additional related information was gathered by interviewing the involved members of staff. RESULTS: The total number of reported surgical incidents was 3142, of which 81 (2.58%) cases had been reported as red or sentinel. 19 of the 81 incidents (23.4%) had been inappropriately reported as red. In 31 reports (38.2%) vital information with regards to the details of the adverse incidents had not been recorded. In 12 cases (14.8%) the description of incidents was of poor quality. RCA was performed for 47 cases (58%) and only 12 cases (15%) received recommendations aiming to improve clinical practice. CONCLUSION: The results of our study demonstrate the need for improvement in the quality of incident reporting. There are enormous benefits to be gained by this time and resource consuming process, however appropriate staff training on the use of this system is a pre-requisite. Furthermore, sufficient support and resources are required for the implementation of RCA recommendations in clinical practice.

Interventional radiology in acute pancreatitis: friend or foe?

CONTEXT: Peripancreatic fluid collections are known complications of pancreatitis. The majority of fluid collections can be drained percutaneously under radiological guidance. Although radiological percutaneous drainage is regarded as safe, here it resulted in catastrophic haemorrhage from the colon due to an iatrogenic injury. CASE REPORT: We present a case of a 70-year-old man who presented with acute alcohol-related severe necrotizing pancreatitis and an associated massive peripancreatic fluid collection. The drainage of this collection was attempted under computed tomography (CT) scan guidance. During the procedure the splenic artery and the splenic flexure of the colon were inadvertently damaged leading to life threatening per rectal bleeding requiring emergency angiographic embolisation of the splenic artery. CONCLUSION: Radiological drainage of peripancreatic fluid collections is generally regarded as having lower rates of complications compared to surgical necrosectomy. However, in this case it leads to a life threatening per rectal bleed requiring emergency splenic artery embolisation.