Extended-age Out-of-sample Validation of Risk Stratification Index 3.0 Models Using Commercial All-payer Claims.

BACKGROUND: The authors previously reported a broad suite of individualized Risk Stratification Index 3.0 (Health Data Analytics Institute, Inc., USA) models for various meaningful outcomes in patients admitted to a hospital for medical or surgical reasons. The models used International Classification of Diseases, Tenth Revision, trajectories and were restricted to information available at hospital admission, including coding history in the previous year. The models were developed and validated in Medicare patients, mostly age 65 yr or older. The authors sought to determine how well their models predict utilization outcomes and adverse events in younger and healthier populations. METHODS: The authors' analysis was based on All Payer Claims for surgical and medical hospital admissions from Utah and Oregon. Endpoints included unplanned hospital admissions, in-hospital mortality, acute kidney injury, sepsis, pneumonia, respiratory failure, and a composite of major cardiac complications. They prospectively applied previously developed Risk Stratification Index 3.0 models to the younger and healthier 2017 Utah and Oregon state populations and compared the results to their previous out-of-sample Medicare validation analysis. RESULTS: In the Utah dataset, there were 55,109 All Payer Claims admissions across 40,710 patients. In the Oregon dataset, there were 21,213 admissions from 16,951 patients. Model performance on the two state datasets was similar or better than in Medicare patients, with an average area under the curve of 0.83 (0.71 to 0.91). Model calibration was reasonable with an R2 of 0.93 (0.84 to 0.97) for Utah and 0.85 (0.71 to 0.91) for Oregon. The mean sensitivity for the highest 5% risk population was 28% (17 to 44) for Utah and 37% (20 to 56) for Oregon. CONCLUSIONS: Predictive analytical modeling based on administrative claims history provides individualized risk profiles at hospital admission that may help guide patient management. Similar predictive performance in Medicare and in younger and healthier populations indicates that Risk Stratification Index 3.0 models are valid across a broad range of adult hospital admissions.

Risk Stratification Index 3.0, a Broad Set of Models for Predicting Adverse Events during and after Hospital Admission.

BACKGROUND: Risk stratification helps guide appropriate clinical care. Our goal was to develop and validate a broad suite of predictive tools based on International Classification of Diseases, Tenth Revision, diagnostic and procedural codes for predicting adverse events and care utilization outcomes for hospitalized patients. METHODS: Endpoints included unplanned hospital admissions, discharge status, excess length of stay, in-hospital and 90-day mortality, acute kidney injury, sepsis, pneumonia, respiratory failure, and a composite of major cardiac complications. Patient demographic and coding history in the year before admission provided features used to predict utilization and adverse events through 90 days after admission. Models were trained and refined on 2017 to 2018 Medicare admissions data using an 80 to 20 learn to test split sample. Models were then prospectively tested on 2019 out-of-sample Medicare admissions. Predictions based on logistic regression were compared with those from five commonly used machine learning methods using a limited dataset. RESULTS: The 2017 to 2018 development set included 9,085,968 patients who had 18,899,224 inpatient admissions, and there were 5,336,265 patients who had 9,205,835 inpatient admissions in the 2019 validation dataset. Model performance on the validation set had an average area under the curve of 0.76 (range, 0.70 to 0.82). Model calibration was strong with an average R 2 for the 99% of patients at lowest risk of 1.00. Excess length of stay had a root-mean-square error of 0.19 and R 2 of 0.99. The mean sensitivity for the highest 5% risk population was 19.2% (range, 11.6 to 30.1); for positive predictive value, it was 37.2% (14.6 to 87.7); and for lift (enrichment ratio), it was 3.8 (2.3 to 6.1). Predictive accuracies from regression and machine learning techniques were generally similar. CONCLUSIONS: Predictive analytical modeling based on administrative claims history can provide individualized risk profiles at hospital admission that may help guide patient management. Similar results from six different modeling approaches suggest that we have identified both the value and ceiling for predictive information derived from medical claims history.

Comparison of an Updated Risk Stratification Index to Hierarchical Condition Categories.

BACKGROUND: The Risk Stratification Index and the Hierarchical Condition Categories model baseline risk using comorbidities and procedures. The Hierarchical Condition categories are rederived yearly, whereas the Risk Stratification Index has not been rederived since 2010. The two models have yet to be directly compared. The authors thus rederived the Risk Stratification Index using recent data and compared their results to contemporaneous Hierarchical Condition Categories. METHODS: The authors reimplemented procedures used to derive the original Risk Stratification Index derivation using the 2007 to 2011 Medicare Analysis and Provider review file. The Hierarchical Condition Categories were constructed on the entire data set using software provided by the Center for Medicare and Medicaid Services. C-Statistics were used to compare discrimination between the models. After calibration, accuracy for each model was evaluated by plotting observed against predicted event rates. RESULTS: Discrimination of the Risk Stratification Index improved after rederivation. The Risk Stratification Index discriminated considerably better than the Hierarchical Condition Categories for in-hospital, 30-day, and 1-yr mortality and for hospital length-of-stay. Calibration plots for both models demonstrated linear predictive accuracy, but the Risk Stratification Index predictions had less variance. CONCLUSIONS: Risk Stratification discrimination and minimum-variance predictions make it superior to Hierarchical Condition Categories. The Risk Stratification Index provides a solid basis for care-quality metrics and for provider comparisons.

Validation and Calibration of the Risk Stratification Index.

BACKGROUND: The Risk Stratification Index was developed from 35 million Medicare hospitalizations from 2001 to 2006 but has yet to be externally validated on an independent large national data set, nor has it been calibrated. Finally, the Medicare Analysis and Provider Review file now allows 25 rather than 9 diagnostic codes and 25 rather than 6 procedure codes and includes present-on-admission flags. The authors sought to validate the index on new data, test the impact of present-on-admission codes, test the impact of the expansion to 25 diagnostic and procedure codes, and calibrate the model. METHODS: The authors applied the original index coefficients to 39,753,036 records from the 2007-2012 Medicare Analysis data set and calibrated the model. The authors compared their results with 25 diagnostic and 25 procedure codes, with results after restricting the model to the first 9 diagnostic and 6 procedure codes and to codes present on admission. RESULTS: The original coefficients applied to the 2007-2012 data set yielded C statistics of 0.83 for 1-yr mortality, 0.84 for 30-day mortality, 0.94 for in-hospital mortality, and 0.86 for median length of stay-values nearly identical to those originally reported. Calibration equations performed well against observed outcomes. The 2007-2012 model discriminated similarly when codes were restricted to nine diagnostic and six procedure codes. Present-on-admission models were about 10% less predictive for in-hospital mortality and hospital length of stay but were comparably predictive for 30-day and 1-yr mortality. CONCLUSIONS: Risk stratification performance was largely unchanged by additional diagnostic and procedure codes and only slightly worsened by restricting analysis to codes present on admission. The Risk Stratification Index, after calibration, thus provides excellent discrimination and calibration for important health services outcomes and thus appears to be a good basis for making hospital comparisons.

Elucidating the association between regional variation in diagnostic frequency with risk-adjusted mortality through analysis of claims data of medicare inpatients: a cross-sectional study.

OBJECTIVE: The validity of risk-adjustment methods based on administrative data has been questioned because hospital referral regions with higher diagnosis frequencies report lower case-fatality rates, implying that diagnoses do not track the underlying health risk. The objective of this study is to test the hypothesis that regional variation of diagnostic frequency in inpatient records is not associated with different coding practices but a reflection of the underlying health risks. DESIGN: We applied two stratification methods to Medicare Analysis and Provider Review data from 2009 through 2014: (1) the number of chronic conditions; and, (2) quartiles of Risk Stratification Index (RSI)-defined risk. After sorting hospital referral regions into quintiles of diagnostic frequency, we examined all-cause mortality. SETTING: Medicare Analysis and Provider Review administrative database. PARTICIPANTS: 18 126 301 hospitalised Medicare fee-for-service beneficiaries aged 65 or older who had at least one hospital-based procedure between 2009 and 2014. EXPOSURE: Coding frequency and baseline regional population risk factors by hospital referral region. PRIMARY AND SECONDARY OUTCOMES AND MEASURES: One year all-cause mortality in patients having the same number of chronic conditions or within the same RSI score quartile across US health referral regions, grouped by diagnostic frequency. RESULTS: No consistent relationship between diagnostic frequency and mortality in the risk stratum defined by number of chronic conditions was detected. In the strata defined by RSI quartile, there was no decrease in mortality as a function of diagnostic frequency. Instead, adjusted mortality was positively correlated with socioeconomic risk factors. CONCLUSIONS: Using present-on-admission codes only, diagnostic frequency among inpatients with at least one hospital-based procedure appears to be consequent to differences in baseline population health status, rather than diagnostic coding practices. In this population, claims-based risk-adjustment using RSI appears to be useful for assessing hospital outcomes and performance.