Acceptability of the decision support for safer surgery tool.

BACKGROUND: We examined providers' perceptions of the Decision Support for Safer Surgery (DS3) tool, which provided preoperative patient-level risk estimates of postoperative adverse events. METHODS: The DS3 tool was evaluated at 2 academic medical centers. During the validation study, surgeons provided usefulness ratings of the DS3 tool for each patient before surgery. At the end of the study, providers' perceptions of the DS3 tool were assessed via questionnaire. Data were analyzed using descriptive statistics and independent samples t tests. RESULTS: During the trial, 23 surgeons completed usefulness ratings of the DS3 tool for 1,006 patients. Surgeons rated the tool as "very useful" or "moderately useful" in 251 (25%) of the cases, "neutral" in 469 (46.6%) of the cases, and "moderately unuseful" or "not useful" in 286 (28.4%) cases. At the end of the trial, 32 providers completed the questionnaire; perceptions were relatively neutral, although several aspects were rated quite favorably. CONCLUSION: The DS3 tool may be most useful for achieving particular tasks (eg, training novice surgeons, increasing patient engagement) or encouraging specific processes (eg, team-based care) in surgical care settings.

Comparison of prospective risk estimates for postoperative complications: human vs computer model.

BACKGROUND: Surgical quality improvement tools such as NSQIP are limited in their ability to prospectively affect individual patient care by the retrospective audit and feedback nature of their design. We hypothesized that statistical models using patient preoperative characteristics could prospectively provide risk estimates of postoperative adverse events comparable to risk estimates provided by experienced surgeons, and could be useful for stratifying preoperative assessment of patient risk. STUDY DESIGN: This was a prospective observational cohort. Using previously developed models for 30-day postoperative mortality, overall morbidity, cardiac, thromboembolic, pulmonary, renal, and surgical site infection (SSI) complications, model and surgeon estimates of risk were compared with each other and with actual 30-day outcomes. RESULTS: The study cohort included 1,791 general surgery patients operated on between June 2010 and January 2012. Observed outcomes were mortality (0.2%), overall morbidity (8.2%), and pulmonary (1.3%), cardiac (0.3%), thromboembolism (0.2%), renal (0.4%), and SSI (3.8%) complications. Model and surgeon risk estimates showed significant correlation (p < 0.0001) for each outcome category. When surgeons perceived patient risk for overall morbidity to be low, the model-predicted risk and observed morbidity rates were 2.8% and 4.1%, respectively, compared with 10% and 18% in perceived high risk patients. Patients in the highest quartile of model-predicted risk accounted for 75% of observed mortality and 52% of morbidity. CONCLUSIONS: Across a broad range of general surgical operations, we confirmed that the model risk estimates are in fairly good agreement with risk estimates of experienced surgeons. Using these models prospectively can identify patients at high risk for morbidity and mortality, who could then be targeted for intervention to reduce postoperative complications.

Toward prospective identification of high-risk surgical patients.

The purpose of this study was to explore the feasibility of prospectively identifying patients at high risk for surgical complications using automatable methods focused on patient characteristics. We used data from the Michigan Surgical Quality Collaborative (60,411 elective surgeries) performed between 2003 and 2008. Regression models for postoperative mortality, overall morbidity, cardiac, thromboembolic, pulmonary, renal, and surgical site infection complications were developed using preoperative patient and planned procedure data. Risk was categorized by quartiles of predicted probability: "low" risk corresponding to the bottom quartile, "average" to the middle two quartiles, and "high" to the top quartile. C-indices were calculated to measure discrimination; model validity was assessed by cross-validation. Models were repeated using only patient characteristics. Risk category was closely related to event rates; 80 to 90 per cent of mortality and cardiac, renal, and pulmonary complications occurred among the 25 per cent of "high-risk" patients. Although thromboembolisms and surgical site infections were less predictable, 60 to 70 per cent of events occurred among high-risk patients. Cross-validation results were consistent and only slightly attenuated when predictors were restricted to patient characteristics alone. Adverse postoperative events are concentrated among patients identifiable preoperatively as high risk. Preoperative risk assessment could allow for efficient interventions targeted to high-risk patients.

Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program.

BACKGROUND: Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure. STUDY DESIGN: We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year. RESULTS: Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables. CONCLUSIONS: The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.