Comparison of two prognostic models in trauma outcome.

BACKGROUND: The Trauma Audit and Research Network (TARN) in the UK publicly reports hospital performance in the management of trauma. The TARN risk adjustment model uses a fractional polynomial transformation of the Injury Severity Score (ISS) as the measure of anatomical injury severity. The Trauma Mortality Prediction Model (TMPM) is an alternative to ISS; this study compared the anatomical injury components of the TARN model with the TMPM. METHODS: Data from the National Trauma Data Bank for 2011-2015 were analysed. Probability of death was estimated for the TARN fractional polynomial transformation of ISS and compared with the TMPM. The coefficients for each model were estimated using 80 per cent of the data set, selected randomly. The remaining 20 per cent of the data were used for model validation. TMPM and TARN were compared using calibration curves, measures of discrimination (area under receiver operating characteristic curves; AUROC), proximity to the true model (Akaike information criterion; AIC) and goodness of model fit (Hosmer-Lemeshow test). RESULTS: Some 438 058 patient records were analysed. TMPM demonstrated preferable AUROC (0·882 for TMPM versus 0·845 for TARN), AIC (18 204 versus 21 163) and better fit to the data (32·4 versus 153·0) compared with TARN. CONCLUSION: TMPM had greater discrimination, proximity to the true model and goodness-of-fit than the anatomical injury component of TARN. TMPM should be considered for the injury severity measure for the comparative assessment of trauma centres.

Comparing outcomes of coronary artery bypass surgery: Is the New York Cardiac Surgery Reporting System model sensitive to changes in case mix?

OBJECTIVE: To assess the validity of using the standardized mortality ratio (SMR), based on the New York State Cardiac Surgery Reporting System (CSRS) prediction model to compare coronary artery bypass grafting (CABG) outcomes between hospitals. DESIGN: The study was designed as a retrospective study based on a database containing all patients undergoing isolated CABG surgery in New York State hospitals in 1996 (n = 20,078). In the first part of this study, a computer simulation was used to assess the impact of case mix variation on the SMR. A computer-intensive algorithm was used to create 5,000 random case mixes from the patients in the CSRS database. The SMR associated with each of the 5,000 case mixes was calculated using a resampling algorithm. The second part of this study was designed to determine whether the identity of quality outliers among all of the 32 hospitals in the CSRS database would change after adjusting for the effects of case mix on the SMR. The SMR associated with the case mix of each hospital in the CSRS database (the hospital case mix SMR) was obtained using a resampling algorithm. The hospital SMR (as well as 95% confidence interval) was then calculated using bootstrapping for each of the 32 hospitals within the CSRS database. An adjusted SMR was then derived for each hospital by dividing the hospital SMR by the case mix SMR for that hospital. SETTING: Thirty-two hospitals in New York State performing CABG surgery. INTERVENTIONS: None. RESULTS: Changes in patient case mix are associated with statistically significant changes in the SMR. However, there was no difference in the identity of quality outliers in the New York State CSRS database when using either the SMR or the SMR adjusted for the effects of case mix. CONCLUSION: Risk-adjusted measures of outcomes in CABG patients may be potentially biased by differences in case mix between institutions because of the influence of case mix on the process of risk adjustment. There was, however, no evidence of bias in the specific application of the CSRS model to the hospitals in the CSRS database.

Effect of mortality rate on the performance of the Acute Physiology and Chronic Health Evaluation II: a simulation study.

OBJECTIVE: To evaluate the impact of case mix variation on the performance of the Acute Physiology and Chronic Health Evaluation (APACHE) II using measures of calibration and discrimination. DESIGN: APACHE II data were collected prospectively at the surgical intensive care unit of the University of Vermont on all adult admissions over an 8-yr period (excluding cardiac surgical patients, burn patients, and patients < 16 yrs of age). The original case mix was systematically varied to create 2,000 different case mixes ranging in mortality between 5% and 18% using a computer-intensive resampling algorithm. The area under the receiver operating characteristic curve and the Hosmer-Lemeshow C statistic were derived for each of the simulated case mixes with bootstrapping. SETTING: The surgical intensive care unit at a 450-bed teaching hospital. PATIENTS: A group of 6,806 adult surgical patients excluding cardiac surgical patients and burn patients. MEASUREMENTS AND RESULTS: Simulated data sets were created from a database of patients treated at a single institution to test the hypothesis that the performance of APACHE II is stable across a clinically reasonable range of mortality rates. The discrimination and calibration of APACHE II varied with case mix. CONCLUSION: The discrimination of APACHE II is not independent of case mix. However, the variability of the Hosmer-Lemeshow statistic as a function of the case mix may simply reflect the limitations of this goodness of fit statistic to assess model calibration. Because the discrimination of APACHE II is a function of case mix, caution should be exercised when using APACHE II-based adjusted mortality rates to compare intensive care units with widely divergent case mixes.

Effect of varying the case mix on the standardized mortality ratio and W statistic: A simulation study.

OBJECTIVE: To evaluate the validity of using the standardized mortality ratio (SMR) and the W statistic as risk-adjusted measures of hospital mortality to judge ICU performance. DESIGN: APACHE (acute physiology and chronic health evaluation) II data were collected prospectively from the surgical ICU (SICU) at a single institution using all adult admissions (n = 6806) over an 8-year period (excluding cardiac surgical patients, burn patients, and patients under 16 years of age). Using a computer simulation technique, virtual ICUs (VICUs) with mortality rates between 5% and 16% were constructed. After first dividing the original data set into deciles of risk, each VICU was constructed by randomly resampling between 10 and 680 patients from each decile. The SMR, W statistic, and Z statistic were calculated for 10,000 different case mixes. SETTING: The SICU at a 450-bed teaching hospital. PATIENTS: A group of 6,806 adult patient admissions, excluding cardiac surgical patients and burn patients. MEASUREMENTS AND RESULTS: VICUs were created from a data set of actual patients treated at one institution in order to test the hypothesis that the SMR and W statistic would remain invariant when applied to subsets of patients from a single institution. Instead, the SMR and W statistic were found to be very sensitive to changes in case mix. The SMR and W statistic were linear functions of the simulated ICU mortality rate. CONCLUSION: This simulation demonstrates that the SMR and the W statistic based on APACHE II cannot be used to compare outcomes of ICUs. We have proposed a revision of the SMR that eliminates the effect of case mix and allows for more accurate comparisons of ICU performance.

The cost effectiveness of anesthesia workforce models: a simulation approach using decision-analysis modeling.

UNLABELLED: The objective of this study was to evaluate the incremental cost effectiveness of anesthesia workforce staffing scenarios, as a function of skill mix, by using the technique of decision analysis. A decision tree model was constructed to compare the incremental cost effectiveness of alternative delivery systems for anesthesia care from the perspective of the payer. Five different staffing scenarios, ranging from physician-intensive to nurse-intensive, were modeled. In the nurse-intensive model, low- and intermediate-risk patients were cared for by solo certified registered nurse anesthetists (CRNAs) and high-risk patients were cared for by physicians. In the physician-intensive model, physicians anesthetized all patients. In the first-, second-, and third-team models, all high-risk patients were cared for by physicians working alone, and all intermediate-risk patients were cared for using an anesthesia care team approach with a ratio of one physician to two CRNAs. The low-risk patients were managed by using an anesthesia care team approach with physician to CRNA ratios of 1:2, 1:4, and 1:8 in the first-, second-, and third-team models, respectively. The findings of this decision-analysis model suggest that physician-only anesthesia is not cost effective. However, the third-team model is cost effective when compared with the nurse-intensive model. IMPLICATIONS: An anesthesia care-team approach with a physician to certified registered nurse anesthetist (CRNA) ratio of 1:2 is the preferred staffing scenario for intermediate-risk patients. Although medical direction of CRNAs caring for low-risk patients is cost-effective, the small improvement in outcome resulting from increasing the physician to CRNA ratio from 1:8 to 1:4 may not be justified by the added cost.

Selective preoperative cardiac screening improves five-year survival in patients undergoing major vascular surgery: a cost-effectiveness analysis.

OBJECTIVE: To evaluate the long-term outcomes (5-year survival) and cost-effectiveness of selective coronary revascularization before major vascular surgery. DESIGN: A decision-tree model was constructed to compare the cost-effectiveness of four preoperative screening strategies from the perspective of the health care system. SETTING: Based on patient mortality, morbidity, and cost data from a literature review. PARTICIPANTS: Hypothetical cohort of patients scheduled for elective abdominal aortic aneurysm repair followed up over a 5-year period. INTERVENTIONS: Patients either proceeded directly to surgery or were screened using one of three possible preoperative screening strategies. In the first strategy, all patients were screened with a dipyridamole-thallium test. In the second strategy, all patients underwent coronary angiography. The third strategy, selective screening, first divided patients into high-, intermediate-, and low-risk groups using clinical criteria. High-risk patients underwent preoperative angiography. Intermediate-risk patients were screened noninvasively, and low-risk patients proceeded directly to surgery without further testing. MEASUREMENT AND MAIN RESULTS: Proceeding directly to vascular surgery resulted in the poorest 5-year survival rate (77.4%) compared with preoperative risk stratification followed by selective coronary revascularization, routine noninvasive testing (86.1%), selective testing (86.0%), and routine angiography (87.9%; p = 0.00). The incremental cost-effectiveness ratio for selective testing was significantly lower than for routine angiography ($44,800/years of life saved (YLS) v $93,300/YLS; p < 0.02). Routine noninvasive testing was not cost-effective. Thirty-day mortality was the same for all four strategies (p = 0.84). CONCLUSION: Selective screening before vascular surgery may improve 5-year survival and be cost-effective. Neither routine noninvasive testing nor routine angiography appears to be cost-effective compared with currently accepted medical therapies.

Intensive care unit prognostic scoring systems to predict death: a cost-effectiveness analysis.

OBJECTIVE: To evaluate the cost-effectiveness, using the technique of decision analysis, of withdrawing care from patients in the intensive care unit (ICU) who are predicted to have a high probability of death (>90%) after 48 hrs using a mortality risk estimate based on daily Acute Physiology and Chronic Health Evaluation (APACHE) III scores. MATERIALS AND METHODS: A decision tree model was constructed to compare the cost-effectiveness of two clinical strategies. In the first strategy, patients receive ICU care until they were discharged, died, or had care withdrawn based on subjective clinical criteria. In the second strategy, patients remained in the ICU until they were either discharged, died, or had life-sustaining care withdrawn based on subjective criteria or if they were predicted to have a >90% risk of mortality after 48 hrs by a prognostic scoring system. Transition probabilities were based on a retrospective data analysis of 4,106 noncardiac ICU patients admitted to a tertiary surgical ICU over a 9-yr period. Cost estimates were based on daily Therapeutic Intervention Scoring System (TISS) scores from our database and using published data on the estimated production cost for a TISS point. The sensitivity (16.6%) and specificity (99.6%) of the mortality risk estimate at 48 hrs (using the >90% decision point) based on daily APACHE III scores were derived from published data. RESULTS: In the base case analysis, we assumed that the sensitivity and specificity of the prognostic risk estimate are unchanged when exported to a new environment. Not using a prognostic scoring system as the basis for withdrawing care resulted in a slightly higher survival rate (87.2% vs. 86.85%) at a cost-per-death prevented (CPDP) of $263,700. Since prognostic scoring systems have not been shown to retain the same predictive power when exported to new databases, we chose to explore the effect of varying the specificity of the scoring system on CPDP. Decreasing the specificity from .996 (baseline) to .98 causes the CPDP to drop to $53,300. Changing the specificity to .95 results in a CPDP prevented of $21,700. Using one-way sensitivity analysis, the CPDP is shown to be relatively insensitive to delaying the decision point from ICU day 3 to day 7. Sensitivity analysis also indicates that CPDP increases rapidly with hospital death rate. For a death rate of 30%, the CPDP increases to $768,600 (in the base case, the death rate is 12.8%); when the specificity is decreased to .95, the CPDP drops to $62,100. CONCLUSION: Unless daily mortality risk estimates based on APACHE III can be shown to retain the same level of predictive power in ICUs outside the development database, it is unlikely that the incremental cost-effectiveness gained by using them as the basis to withdraw care is sufficient to justify their use in this manner.

Predicting survival, length of stay, and cost in the surgical intensive care unit: APACHE II versus ICISS.

BACKGROUND: Risk stratification of patients in the intensive care unit (ICU) is an important tool because it permits comparison of patient populations for research and quality control. Unfortunately, currently available scoring systems were developed primarily in medical ICUs and have only mediocre performance in surgical ICUs. Moreover, they are very expensive to purchase and use. We conceived a simple risk-stratification tool for the surgical ICU that uses readily available International Classification of Diseases, Ninth Revision, codes to predict outcome. Called ICISS (International Classification of Disease Illness Severity Score), our score is the product of the survival risk ratios (obtained from an independent data set) for all International Classification of Diseases, Ninth Revision, diagnosis codes. METHODS: A total of 5,322 noncardiac patients admitted to a surgical ICU during an 8-year period had their Acute Physiology and Chronic Health Evaluation (APACHE) II scores compared with their ICISS as predictors of outcome (survival/nonsurvival, length of stay, and charges). RESULTS: ICISS proved to be a much better predictor of survival than APACHE (receiver operating characteristic (ROC) APACHE = 0.806; Hosmer-Lemeshow (HL) APACHE = 22.56; ROC ICISS = 0.892; HL ICISS = 12.06) or the APACHE survival probability (ROC = 0.836; HL = 34.47). These differences were highly statistically significant (p < 0.001). ICISS was also better correlated with ICU length of stay (APACHE R2 = 0.06; ICISS R2 = 0.32) and ICU charges (APACHE R2 = 0.07; ICISS R2 = 0.39). When combined in a logistic model with ICISS, APACHE II added slightly to the predictive power of ICISS alone (combined ROC = 0.903) but degraded the calibration of the model (combined HL = 16.29; p = 0.038). CONCLUSION: Because ICISS is both more accurate and much less expensive to calculate than APACHE II score, ICISS should replace APACHE II score as the standard risk stratification tool in surgical ICUs.