NSQIP Quality Benchmarking and Evaluation of Potential Bias Associated with Higher- or Lower-Risk Operation Case Mix.

BACKGROUND: To ensure validity and acceptance of NSQIP risk-adjusted benchmarking, it is important that adjustments adequately control for hospitals that vary in their proportions of lower- or higher-risk operations (combined risk for procedure and patient). This issue was addressed in separate empirical and simulation studies. STUDY DESIGN: For the empirical study, potential miscalibration bias favoring hospitals that do lower-risk operations or disfavoring hospitals that do higher-risk operations was evaluated for 14 modeled outcomes using NSQIP data. A determination was also made as to whether there was a relationship between mean hospital operation risk and benchmarking results (log odds ratio). In the simulation study of the same 14 outcomes, hospital benchmarked performance was evaluated when sampled cases were reconstituted to include either a larger proportion of lower-risk operations or a larger proportion of higher-risk operations. RESULTS: Miscalibration favoring either lower- or higher-risk operations was absent, as were important associations between operative risk and hospital log odds ratios (most model R 2 less than 0.01). In the simulation, there were no substantial changes in log odds ratios when greater percentages of either lower- or higher-risk operations were included in a hospital's sample (nonsignificant p values and effect sizes less than 0.1). CONCLUSIONS: These results should enhance NSQIP participants' confidence in the adequacy of NSQIP patient and procedure risk-adjustment methods. NSQIP participants may rely on benchmarking findings, and implement quality improvement efforts based on them, without concern that they are biased by a preponderance of lower or higher risk operations.

Effect of wound classification on risk adjustment in American College of Surgeons NSQIP.

BACKGROUND: Surgical wound classification has been used in risk-adjustment models. However, it can be subjective and could potentially improperly bias hospital quality comparisons. The objective is to examine the effect of wound classification on hospital performance risk-adjustment models. STUDY DESIGN: Retrospective review of the 2011 American College of Surgeons NSQIP database was conducted for the following wound classification categories: clean, clean-contaminated, contaminated, and dirty-infected. To assess the influence of wound classification on risk adjustment, 2 models were developed for all outcomes: 1 including and 1 excluding wound classification. For each model, hospital postoperative complications were estimated using hierarchical multivariable regression methods. Absolute changes in hospital rank, correlations of odds ratios, and outlier status agreement between models were examined. RESULTS: Of the 442,149 cases performed in 315 hospitals: 53.6% were classified as clean; 34.2% as clean-contaminated; 6.7% as contaminated; and 5.5% as dirty-infected. The surgical site infection rate was highest in dirty-infected (8.5%) and lowest in clean (1.8%) cases. For overall surgical site infection, the absolute change in risk-adjusted hospital performance rank between models, including vs excluding wound classification, was minimal (mean 4.5 of 315 positions). The correlations between odds ratios of the 2 performance models were nearly perfect (R = 0.9976, p < 0.0001), and outlier status agreement was excellent (κ = 0.95ss08, p < 0.0001). Similar findings were observed in models of subgroups of surgical site infections and other postoperative outcomes. CONCLUSIONS: In circumstances where alternate information is available for risk adjustment, there appear to be minimal differences in performance models that include vs exclude wound classification. Therefore, the American College of Surgeons NSQIP is critically evaluating the continued use of wound classification in hospital performance risk-adjustment models.