National Multi-Institutional Validation of a Surgical Transfusion Risk Prediction Model.

BACKGROUND: Accurate estimation of surgical transfusion risk is important for many aspects of surgical planning, yet few methods for estimating are available for estimating such risk. There is a need for reliable validated methods for transfusion risk stratification to support effective perioperative planning and resource stewardship. STUDY DESIGN: This study was conducted using the American College of Surgeons NSQIP datafile from 2019. S-PATH performance was evaluated at each contributing hospital, with and without hospital-specific model tuning. Linear regression was used to assess the relationship between hospital characteristics and area under the receiver operating characteristic (AUROC) curve. RESULTS: A total of 1,000,927 surgical cases from 414 hospitals were evaluated. Aggregate AUROC was 0.910 (95% CI 0.904 to 0.916) without model tuning and 0.925 (95% CI 0.919 to 0.931) with model tuning. AUROC varied across individual hospitals (median 0.900, interquartile range 0.849 to 0.944), but no statistically significant relationships were found between hospital-level characteristics studied and model AUROC. CONCLUSIONS: S-PATH demonstrated excellent discriminative performance, although there was variation across hospitals that was not well-explained by hospital-level characteristics. These results highlight the S-PATH's viability as a generalizable surgical transfusion risk prediction tool.

Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program: Bariatric Surgical Risk/Benefit Calculator: 1-year comorbidity remission.

BACKGROUND: Clinical calculators can provide patient-personalized estimates of treatment risks and health outcomes. The American College of Surgeons Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program (MBSAQIP) set out to create a publicly available tool to assess both short-term postoperative risk and long-term benefits for prospective adult patients eligible for 1 of 4 primary bariatric procedures. The calculator is comprised of multiple prediction elements: (1) 30-day postoperative risk, (2) 1-year body mass index projections, and (3) 1-year comorbidity remission. OBJECTIVES: To assess the performance of the 1-year comorbidity remission prediction feature of the calculator. SETTING: Not-for-profit organization clinical data registry. METHODS: MBSAQIP data across 4.5 years from 240,227 total patients indicating at least 1 comorbidity of interest present preoperatively and who had a 1-year follow-up record documenting their comorbidity status were included. Six models were constructed, stratified by the presence of the respective preoperative comorbidity: hypertension, hyperlipidemia, gastroesophageal reflux disease, sleep apnea, non-insulin-dependent diabetes, and insulin-dependent diabetes. A multinomial logistic regression model was used to predict 1-year remission (total, partial, or no remission) of insulin-dependent diabetes. All other outcomes were binary (yes or no at 1 yr), and ordinary logistic regression models were used. RESULTS: All models showed adequate discrimination (C statistics ranging from .58 to .68). Plots of observed versus predicted remission (%) showed excellent calibration across all models. CONCLUSION: All remission models were well calibrated with sufficient discrimination. The MBSAQIP Bariatric Surgical Risk/Benefit Calculator is a publicly available tool intended for integration into clinical practice to enhance patient-clinician discussions and informed consent.

The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators.

Objective: To compare the performance of the ACS NSQIP "universal" risk calculator (N-RC) to operation-specific RCs. Background: Resources have been directed toward building operation-specific RCs because of an implicit belief that they would provide more accurate risk estimates than the N-RC. However, operation-specific calculators may not provide sufficient improvements in accuracy to justify the costs in development, maintenance, and access. Methods: For the N-RC, a cohort of 5,020,713 NSQIP patient records were randomly divided into 80% for machine learning algorithm training and 20% for validation. Operation-specific risk calculators (OS-RC) and OS-RCs with operation-specific predictors (OSP-RC) were independently developed for each of 6 operative groups (colectomy, whipple pancreatectomy, thyroidectomy, abdominal aortic aneurysm (open), hysterectomy/myomectomy, and total knee arthroplasty) and 14 outcomes using the same 80%/20% rule applied to the appropriate subsets of the 5M records. Predictive accuracy was evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and Hosmer-Lemeshow (H-L) P values, for 13 binary outcomes, and mean squared error for the length of stay outcome. Results: The N-RC was found to have greater AUROC (P = 0.002) and greater AUPRC (P < 0.001) compared to the OS-RC. No other statistically significant differences in accuracy, across the 3 risk calculator types, were found. There was an inverse relationship between the operation group sample size and magnitude of the difference in AUROC (r = -0.278; P = 0.014) and in AUPRC (r = -0.425; P < 0.001) between N-RC and OS-RC. The smaller the sample size, the greater the superiority of the N-RC. Conclusions: While operation-specific RCs might be assumed to have advantages over a universal RC, their reliance on smaller datasets may reduce their ability to accurately estimate predictor effects. In the present study, this tradeoff between operation specificity and accuracy, in estimating the effects of predictor variables, favors the N-R, though the clinical impact is likely to be negligible.

The past, present and future of ACS NSQIP-Pediatric: Evolution from a quality registry to a comparative quality performance platform.

Quality and process improvement (QI/PI) in children's surgical care require reliable data across the care continuum. Since 2012, the American College of Surgeons' (ACS) National Surgical Quality Improvement Program-Pediatric (NSQIP-Pediatric) has supported QI/PI by providing participating hospitals with risk-adjusted, comparative data regarding postoperative outcomes for multiple surgical specialties. To advance this goal over the past decade, iterative changes have been introduced to case inclusion and data collection, analysis and reporting. New datasets for specific procedures, such as appendectomy, spinal fusion for scoliosis, vesicoureteral reflux procedures, and tracheostomy in children less than 2 years old, have incorporated additional risk factors and outcomes to enhance the clinical relevance of data, and resource utilization to consider healthcare value. Recently, process measures for urgent surgical diagnoses and surgical antibiotic prophylaxis variables have been developed to promote timely and appropriate care. While a mature program, NSQIP-Pediatric remains dynamic and responsive to meet the needs of the surgical community. Future directions include introduction of variables and analyses to address patient-centered care and healthcare equity.

Influence of Perioperative COVID-19 Infection on Surgical Outcomes and Risk Adjustment for Quality Assessment in the American College of Surgeons NSQIP.

BACKGROUND: Surgical patients with perioperative coronavirus disease 19 (COVID-19) infection experience higher rates of adverse events than those without COVID-19, which may lead to imprecision in hospital-level quality assessment. Our objectives were to quantify differences in COVID-19-associated adverse events in a large national sample and examine distortions in surgical quality benchmarking if COVID-19 status is not considered. STUDY DESIGN: Data included 793,280 patient records from the American College of Surgeons NSQIP from April 1, 2020, to March 31, 2021. Models predicting 30-day mortality, morbidity, pneumonia, and ventilator dependency greater than 48 hours, and unplanned intubation were constructed. Risk adjustment variables were selected for these models from standard NSQIP predictors and perioperative COVID-19 status. RESULTS: A total of 5,878 (0.66%) had preoperative COVID-19, and 5,215 (0.58%) had postoperative COVID-19. COVID-19 rates demonstrated some consistency across hospitals (median preoperative 0.84%, interquartile range 0.14% to 0.84%; median postoperative 0.50%, interquartile range 0.24% to 0.78%). Postoperative COVID-19 was always associated with increased adverse events. For postoperative COVID-19 among all cases, there was nearly a 6-fold increase in mortality (1.07% to 6.37%) and15-fold increase in pneumonia (0.92% to 13.57%), excluding the diagnosis of COVID-19 itself. The effects of preoperative COVID-19 were less consistent. Inclusion of COVID-19 in risk-adjustment models had minimal effects on surgical quality assessments. CONCLUSIONS: Perioperative COVID-19 was associated with a dramatic increase in adverse events. However, quality benchmarking was minimally affected. This may be the result of low overall COVID-19 rates or balance in rates established across hospitals during the 1-year observational period. There remains limited evidence for restructuring ACS NSQIP risk-adjustment for the time-limited effects of the COVID-19 pandemic.

American College of Surgeons NSQIP Risk Calculator Accuracy Using a Machine Learning Algorithm Compared with Regression.

BACKGROUND: The American College of Surgeons NSQIP risk calculator (RC) uses regression to make predictions for fourteen 30-day surgical outcomes. While this approach provides accurate (discrimination and calibration) risk estimates, they might be improved by machine learning (ML). To investigate this possibility, accuracy for regression-based risk estimates were compared to estimates from an extreme gradient boosting (XGB)-ML algorithm. STUDY DESIGN: A cohort of 5,020,713 million NSQIP patient records was randomly divided into 80% for model construction and 20% for validation. Risk predictions using regression and XGB-ML were made for 13 RC binary 30-day surgical complications and one continuous outcome (length of stay [LOS]). For the binary outcomes, discrimination was evaluated using the area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC), and calibration was evaluated using Hosmer-Lemeshow statistics. Mean squared error and a calibration curve analog were evaluated for the continuous LOS outcome. RESULTS: For every binary outcome, discrimination (AUROC and AUPRC) was slightly greater for XGB-ML than for regression (mean [across the outcomes] AUROC was 0.8299 vs 0.8251, and mean AUPRC was 0.1558 vs 0.1476, for XGB-ML and regression, respectively). For each outcome, miscalibration was greater (larger Hosmer-Lemeshow values) with regression; there was statistically significant miscalibration for all regression-based estimates, but only for 4 of 13 when XGB-ML was used. For LOS, mean squared error was lower for XGB-ML. CONCLUSIONS: XGB-ML provided more accurate risk estimates than regression in terms of discrimination and calibration. Differences in calibration between regression and XGB-ML were of substantial magnitude and support transitioning the RC to XGB-ML.

The Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program Bariatric Surgical Risk/Benefit Calculator: 1-year weight.

BACKGROUND: Data-driven tools can be designed to provide patient-personalized estimates of health outcomes. Clinical calculators are commonly built to assess risk, but potential benefits of treatment should be equally considered. The American College of Surgeons Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program (MBSAQIP) sought to create a risk and benefit calculator for adult patients considering primary metabolic and bariatric surgery with multiple prediction features: (1) 30-day risk, (2) 1-year body mass index (BMI) projections, and (3) 1-year co-morbidity remission. OBJECTIVE: To assess the performance of the 1-year BMI projections feature of this tool. SETTING: Not-for-profit organization, clinical data registry. METHODS: MBSAQIP data from 596,024 cases across 4.5 years from 882 centers with ∼2.5 million records through 18 months postoperatively were included. A generalized estimating equation model was used to estimate BMI over time for 4 primary procedures: laparoscopic adjustable gastric band, laparoscopic sleeve gastrectomy, laparoscopic Roux-en-Y gastric bypass, and biliopancreatic diversion with duodenal switch. RESULTS: The mean absolute error (MAE) in BMI predictions through postoperative month 12 was 1.68 units; overall correlation of actual and predicted BMI was .94. MAE of postoperative BMI estimates (1-12 mo) was lowest for laparoscopic sleeve gastrectomy (1.64) and highest for biliopancreatic diversion with duodenal switch (1.99). BMI predictions at 12 months showed MAE = 2.99 units. CONCLUSIONS: Predicted BMI closely aligned with actual BMI values across the 12-month postoperative period. The MBSAQIP Bariatric Surgical Risk/Benefit Calculator is publicly available with the intent to facilitate patient-clinician communication and guide surgical decision making. This tool can aid in evaluating postoperative risk as well as benefits and long-term expectations.

Feasibility and Acceptability of a Preoperative Multimodal Mobile Health Assessment in Spine Surgery Candidates.

BACKGROUND: Rapid growth in smartphone use has expanded opportunities to use mobile health (mHealth) technology to collect real-time patient-reported and objective biometric data. These data may have important implication for personalized treatments of degenerative spine disease. However, no large-scale study has examined the feasibility and acceptability of these methods in spine surgery patients. OBJECTIVE: To evaluate the feasibility and acceptability of a multimodal preoperative mHealth assessment in patients with degenerative spine disease. METHODS: Adults undergoing elective spine surgery were provided with Fitbit trackers and sent preoperative ecological momentary assessments (EMAs) assessing pain, disability, mood, and catastrophizing 5 times daily for 3 weeks. Objective adherence rates and a subjective acceptability survey were used to evaluate feasibility of these methods. RESULTS: The 77 included participants completed an average of 82 EMAs each, with an average completion rate of 86%. Younger age and chronic pulmonary disease were significantly associated with lower EMA adherence. Seventy-two (93%) participants completed Fitbit monitoring and wore the Fitbits for an average of 247 hours each. On average, participants wore the Fitbits for at least 12 hours per day for 15 days. Only worse mood scores were independently associated with lower Fitbit adherence. Most participants endorsed positive experiences with the study protocol, including 91% who said they would be willing to complete EMAs to improve their preoperative surgical guidance. CONCLUSION: Spine fusion candidates successfully completed a preoperative multimodal mHealth assessment with high acceptability. The intensive longitudinal data collected may provide new insights that improve patient selection and treatment guidance.

Correlation Between Postoperative Antimicrobial Prophylaxis Use and Surgical Site Infection in Children Undergoing Nonemergent Surgery.

Importance: Use of postoperative antimicrobial prophylaxis is common in pediatric surgery despite consensus guidelines recommending discontinuation following incision closure. The association between postoperative prophylaxis use and surgical site infection (SSI) in children undergoing surgical procedures remains poorly characterized. Objective: To evaluate whether use of postoperative surgical prophylaxis is correlated with SSI rates in children undergoing nonemergent surgery. Design, Setting, and Participants: This is a multicenter cohort study using 30-day postoperative SSI data from the American College of Surgeons' Pediatric National Surgical Quality Improvement Program (ACS NSQIP-Pediatric) augmented with antibiotic-use data obtained through supplemental medical record review from June 2019 to June 2021. This study took place at 93 hospitals participating in the ACS NSQIP-Pediatric Surgical Antibiotic Prophylaxis Stewardship Collaborative. Participants were children (<18 years of age) undergoing nonemergent surgical procedures. Exclusion criteria included antibiotic allergies, conditions associated with impaired immune function, and preexisting infections requiring intravenous antibiotics at time of surgery. Exposures: Continuation of antimicrobial prophylaxis beyond time of incision closure. Main Outcomes and Measures: Thirty-day postoperative rate of incisional or organ space SSI. Hierarchical regression was used to estimate hospital-level odds ratios (ORs) for SSI rates and postoperative prophylaxis use. SSI measures were adjusted for differences in procedure mix, patient characteristics, and comorbidity profiles, while use measures were adjusted for clinically related procedure groups. Pearson correlations were used to examine the associations between hospital-level postoperative prophylaxis use and SSI measures. Results: Forty thousand six hundred eleven patients (47.3% female; median age, 7 years) were included, of which 41.6% received postoperative prophylaxis (hospital range, 0%-71.2%). Odds ratios (ORs) for postoperative prophylaxis use ranged 190-fold across hospitals (OR, 0.10-19.30) and ORs for SSI rates ranged 4-fold (OR, 0.55-1.90). No correlation was found between use of postoperative prophylaxis and SSI rates overall (r = 0.13; P = .20), and when stratified by SSI type (incisional SSI, r = 0.08; P = .43 and organ space SSI, r = 0.13; P = .23), and surgical specialty (general surgery, r = 0.02; P = .83; urology, r = 0.05; P = .64; plastic surgery, r = 0.11; P = .35; otolaryngology, r = -0.13; P = .25; orthopedic surgery, r = 0.05; P = .61; and neurosurgery, r = 0.02; P = .85). Conclusions and Relevance: Use of postoperative surgical antimicrobial prophylaxis was not correlated with SSI rates at the hospital level after adjusting for differences in procedure mix and patient characteristics.

NSQIP Quality Benchmarking and Evaluation of Potential Bias Associated with Higher- or Lower-Risk Operation Case Mix.

BACKGROUND: To ensure validity and acceptance of NSQIP risk-adjusted benchmarking, it is important that adjustments adequately control for hospitals that vary in their proportions of lower- or higher-risk operations (combined risk for procedure and patient). This issue was addressed in separate empirical and simulation studies. STUDY DESIGN: For the empirical study, potential miscalibration bias favoring hospitals that do lower-risk operations or disfavoring hospitals that do higher-risk operations was evaluated for 14 modeled outcomes using NSQIP data. A determination was also made as to whether there was a relationship between mean hospital operation risk and benchmarking results (log odds ratio). In the simulation study of the same 14 outcomes, hospital benchmarked performance was evaluated when sampled cases were reconstituted to include either a larger proportion of lower-risk operations or a larger proportion of higher-risk operations. RESULTS: Miscalibration favoring either lower- or higher-risk operations was absent, as were important associations between operative risk and hospital log odds ratios (most model R 2 less than 0.01). In the simulation, there were no substantial changes in log odds ratios when greater percentages of either lower- or higher-risk operations were included in a hospital's sample (nonsignificant p values and effect sizes less than 0.1). CONCLUSIONS: These results should enhance NSQIP participants' confidence in the adequacy of NSQIP patient and procedure risk-adjustment methods. NSQIP participants may rely on benchmarking findings, and implement quality improvement efforts based on them, without concern that they are biased by a preponderance of lower or higher risk operations.

Personalized Surgical Transfusion Risk Prediction Using Machine Learning to Guide Preoperative Type and Screen Orders.

BACKGROUND: Accurate estimation of surgical transfusion risk is essential for efficient allocation of blood bank resources and for other aspects of anesthetic planning. This study hypothesized that a machine learning model incorporating both surgery- and patient-specific variables would outperform the traditional approach that uses only procedure-specific information, allowing for more efficient allocation of preoperative type and screen orders. METHODS: The American College of Surgeons National Surgical Quality Improvement Program Participant Use File was used to train four machine learning models to predict the likelihood of red cell transfusion using surgery-specific and patient-specific variables. A baseline model using only procedure-specific information was created for comparison. The models were trained on surgical encounters that occurred at 722 hospitals in 2016 through 2018. The models were internally validated on surgical cases that occurred at 719 hospitals in 2019. Generalizability of the best-performing model was assessed by external validation on surgical cases occurring at a single institution in 2020. RESULTS: Transfusion prevalence was 2.4% (73,313 of 3,049,617), 2.2% (23,205 of 1,076,441), and 6.7% (1,104 of 16,053) across the training, internal validation, and external validation cohorts, respectively. The gradient boosting machine outperformed the baseline model and was the best- performing model. At a fixed 96% sensitivity, this model had a positive predictive value of 0.06 and 0.21 and recommended type and screens for 36% and 30% of the patients in internal and external validation, respectively. By comparison, the baseline model at the same sensitivity had a positive predictive value of 0.04 and 0.144 and recommended type and screens for 57% and 45% of the patients in internal and external validation, respectively. The most important predictor variables were overall procedure-specific transfusion rate and preoperative hematocrit. CONCLUSIONS: A personalized transfusion risk prediction model was created using both surgery- and patient-specific variables to guide preoperative type and screen orders and showed better performance compared to the traditional procedure-centric approach.

Reproducibility of Hospital Rankings Based on Centers for Medicare & Medicaid Services Hospital Compare Measures as a Function of Measure Reliability.

Importance: Unreliable performance measures can mask poor-quality care and distort financial incentives in value-based purchasing. Objective: To examine the association between test-retest reliability and the reproducibility of hospital rankings. Design, Setting, and Participants: In a cross-sectional design, Centers for Medicare & Medicaid Services Hospital Compare data were analyzed for the 2017 (based on 2014-2017 data) and 2018 (based on 2015-2018 data) reporting periods. The study was conducted from December 13, 2020, to September 30, 2021. This analysis was based on 28 measures, including mortality (acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery bypass grafting), readmissions (acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery bypass grafting), and surgical complications (postoperative acute kidney failure, postoperative respiratory failure, postoperative sepsis, and failure to rescue). Exposures: Measure reliability based on test-retest reliability testing. Main Outcomes and Measures: The reproducibility of hospital rankings was quantified by calculating the reclassification rate across the 2017 and 2018 reporting periods after categorizing the hospitals into terciles, quartiles, deciles, and statistical outliers. Linear regression analysis was used to examine the association between the reclassification rate and the intraclass correlation coefficient for each of the classification systems. Results: The analytic cohort consisted of 28 measures from 4452 hospitals with a median of 2927 (IQR, 2378-3160) hospitals contributing data for each measure. The hospitals participating in the Inpatient Prospective Payment System (n = 3195) had a median bed size of 141 (IQR, 69-261), average daily census of 70 (IQR, 24-155) patients, and a median disproportionate share hospital percentage of 38.2% (IQR, 18.7%-36.6%). The median intraclass correlation coefficient was 0.78 (IQR, 0.72-0.81), ranging between 0.50 and 0.85. The median reclassification rate was 70% (IQR, 62%-71%) when hospitals were ranked by deciles, 43% (IQR, 39%-45%) when ranked by quartiles, 34% (IQR, 31%-36%) when ranked by terciles, and 3.8% (IQR, 2.0%-6.2%) when ranked by outlier status. Increases in measure reliability were not associated with decreases in the reclassification rate. Each 0.1-point increase in the intraclass correlation coefficient was associated with a 6.80 (95% CI, 2.28-11.30; P = .005) percentage-point increase in the reclassification rate when hospitals were ranked into performance deciles, 4.15 (95% CI, 1.16-7.14; P = .008) when ranked into performance quartiles, 1.47 (95% CI, 1.84, 4.77; P = .37) when ranked into performance terciles, and 3.70 (95% CI, 1.30-6.09; P = .004) when ranked by outlier status. Conclusions and Relevance: In this study, more reliable measures were not associated with lower rates of reclassifying hospitals using test-retest reliability testing. These findings suggest that measure reliability should not be assessed with test-retest reliability testing.

Comparison of cost and complication rates for profiling hospital performance in lumbar fusion for spondylolisthesis.

BACKGROUND CONTEXT: There is growing interest among payers in profiling hospital value and quality-of-care, including both the cost and safety of common surgeries, such as lumbar fusion. Nonetheless, there is sparse evidence describing the statistical reliability of such measures when applied to lumbar fusion for spondylolisthesis. PURPOSE: To evaluate the reliability of 90-day inpatient hospital costs, overall complications, and rates of serious complications for profiling hospital performance in lumbar fusion surgery for spondylolisthesis. STUDY DESIGN/SETTING: Data for this analysis came from State Inpatient Databases from nine states made available through the Healthcare Cost and Utilization Project. PATIENT SAMPLE: Patients undergoing elective lumbar spine fusion for spondylolisthesis from 2010 to 2017 in participating states. OUTCOME MEASURES: Statistical reliability, defined as the ability to distinguish true performance differences across hospitals relative to statistical noise. Reliability was assessed separately for 90-day inpatient costs (standardized across years to 2019 dollars), overall complications, and serious complication rates. METHODS: Statistical reliability was measured as the amount of variation between hospitals relative to the total amount of variation for each measure. Total variation includes both between-hospital variation ("signal") and within-hospital variation ("statistical noise"). Thus, reliability equals signal over (signal plus noise) and ranges from 0 to 1. To adjust for differences in patient-level risk and procedural characteristics, hierarchical linear and logistic regression models were created for the cost and complication outcomes. Random hospital intercepts were used to assess between-hospital variation. We evaluated the reliability of each measure by study year and examined the number of hospitals meeting different thresholds of reliability by year. RESULTS: We included a total of 66,571 elective lumbar fusion surgeries for spondylolisthesis performed at 244 hospitals during the study period. The mean 90-day hospital cost was $30,827 (2019 dollars). 12.0% of patients experienced a complication within 90 days of surgery, including 7.8% who had a serious complication. The median reliability of 90-day cost ranged from 0.97to 0.99 across study years, and there was a narrow distribution of reliability values. By comparison, the median reliability for the overall complication metric ranged from 0.22 to 0.44, and the reliability of the serious complication measure ranged from 0.30 to 0.49 across the study years. At least 96% of hospitals had high (> 0.7) reliability for cost in any year, whereas only 0-9% and 0-11% of hospitals reached this cutoff for the overall and serious complication rate in any year, respectively. By comparison, 10%-69% of hospitals per year achieved a more moderate threshold of 0.4 reliability for overall complications, compared to 21%-80% of hospitals who achieved this lower reliability threshold for serious complications. CONCLUSIONS: 90-day inpatient costs are highly reliable for assessing variation across hospitals, whereas overall and serious complications are only moderately reliable for profiling performance. These results support the viability of emerging bundled payment programs that assume true differences in costs of care exist across hospitals.

Administrative Data Are Unreliable for Ranking Hospital Performance Based on Serious Complications After Spine Fusion.

STUDY DESIGN: Retrospective analysis of administrative billing data. OBJECTIVE: To evaluate the extent to which a metric of serious complications determined from administrative data can reliably profile hospital performance in spine fusion surgery. SUMMARY OF BACKGROUND DATA: While payers are increasingly focused on implementing pay-for-performance measures, quality metrics must reliably reflect true differences in performance among the hospitals profiled. METHODS: We used State Inpatient Databases from nine states to characterize serious complications after elective cervical and thoracolumbar fusion. Hierarchical logistic regression was used to risk-adjust differences in case mix, along with variability from low case volumes. The reliability of this risk-stratified complication rate (RSCR) was assessed as the variation between hospitals that was not due to chance alone, calculated separately by fusion type and year. Finally, we estimated the proportion of hospitals that had sufficient case volumes to obtain reliable (>0.7) complication estimates. RESULTS: From 2010 to 2017 we identified 154,078 cervical and 213,133 thoracolumbar fusion surgeries. 4.2% of cervical fusion patients had a serious complication, and the median RSCR increased from 4.2% in 2010 to 5.5% in 2017. The reliability of the RSCR for cervical fusion was poor and varied substantially by year (range 0.04-0.28). Overall, 7.7% of thoracolumbar fusion patients experienced a serious complication, and the RSCR varied from 6.8% to 8.0% during the study period. Although still modest, the RSCR reliability was higher for thoracolumbar fusion (range 0.16-0.43). Depending on the study year, 0% to 4.5% of hospitals had sufficient cervical fusion case volume to report reliable (>0.7) estimates, whereas 15% to 36% of hospitals reached this threshold for thoracolumbar fusion. CONCLUSION: A metric of serious complications was unreliable for benchmarking cervical fusion outcomes and only modestly reliable for thoracolumbar fusion. When assessed using administrative datasets, these measures appear inappropriate for high-stakes applications, such as public reporting or pay-for-performance.Level of Evidence: 3.

The Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program bariatric surgical risk/benefit calculator: 30-day risk.

BACKGROUND: There is increasing demand for data-driven tools that provide accurate and clearly communicated patient-specific information. These can aid discussions between practitioners and patients, promote shared decision-making, and enhance informed consent. The American College of Surgeons Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program (MBSAQIP) sought to create a risk calculator for adult patients considering primary metabolic and bariatric surgery, with multiple prediction features: (1) 30-day risk; (2) 1-year body mass index projections; and (3) 1-year co-morbidity remission. OBJECTIVES: To evaluate the 30-day risk estimation feature of this tool. SETTING: Not-for-profit organization, international bariatric surgery clinical data registry. METHODS: MBSAQIP data across 5.5 years, 925 hospitals, and 775,291 cases were used to develop the 30-day risk feature. Logistic regression models were employed to estimate postoperative risks for 9 outcomes across 4 procedures: laparoscopic Roux-en-Y gastric bypass, laparoscopic sleeve gastrectomy, laparoscopic adjustable gastric band, and biliopancreatic diversion with duodenal switch. RESULTS: The tool showed good discrimination for mortality and surgical site infection models (c-statistics, .80 and .70, respectively), and was slightly less accurate for the 7 other complications (.62-.69). Graphical representations showed excellent calibration for all 9 outcomes. CONCLUSIONS: Overall, the 30-day risk models were accurate and well calibrated, with acceptable discrimination. The MBSAQIP bariatric surgical risk/benefit calculator is publicly available, with the intent to be integrated into healthcare practice to guide bariatric surgical decision-making and care planning, and to enhance communication between patients and their surgical care team.