Development and Assessment of the Wisconsin Surgical Coaching Rubric.

Importance: Surgical coaching continues to gain momentum as an innovative method for continuous professional development. A tool to measure the performance of a surgical coach is needed to provide formative feedback to coaches for continued skill development and to assess the fidelity of a coaching intervention for future research and dissemination. Objective: To evaluate the validity of the Wisconsin Surgical Coaching Rubric (WiSCoR), a novel tool to assess the performance of a peer surgical coach. Design, Setting, and Participants: Surgical coaching sessions from November 2014 through February 2018 conducted by 2 statewide peer surgical coaching programs were audio recorded and transcribed. Twelve raters used WiSCoR to rate the performance of the surgical coach for each session. The study included peer surgical coaches in the Wisconsin Surgical Coaching Program (n = 8) and the Michigan Bariatric Surgery Collaborative coaching program (n = 15). The data were analyzed in 2019. Interventions or Exposures: Use of WiSCoR to rate peer surgical coaching sessions. Main Outcomes and Measures: There were 282 WiSCoR ratings from the 106 coaching sessions included in the study. WiSCoR was evaluated using a framework, including inter-rater reliability assessed with Gwet weighted agreement coefficent. Descriptive statistics of WiSCoR were calculated. Results: Eight coaches (35%) and 11 coachees (29%) were from the Wisconsin Surgical Program and 15 coaches (65%) and 27 coachees (71%) were from the Michigan Bariatric Surgery Collaborative. The validity of WiSCoR is supported by high interrater reliability (Gwet weighted agreement coefficient, 0.87) as well as a weakly positive correlation of WiSCoR to coachee ratings of coaches (r = 0.22; P = .04), rigorous content development, consistent rater training, and the association of WiSCoR with coach and coaching program development. The mean (SD) overall coach performance rating using WiSCoR was 3.23 (0.82; range, 1-5). Conclusions and Relevance: WiSCoR is a reliable measure that can assess the performance of a surgical coach, inform fidelity to coaching principles, and provide formative feedback to surgical coaches. While coachee ratings may reflect coachee satisfaction, they are not able to determine the quality of a coach.

Modeling Surgical Technical Skill Using Expert Assessment for Automated Computer Rating.

OBJECTIVE: Computer vision was used to predict expert performance ratings from surgeon hand motions for tying and suturing tasks. SUMMARY BACKGROUND DATA: Existing methods, including the objective structured assessment of technical skills (OSATS), have proven reliable, but do not readily discriminate at the task level. Computer vision may be used for evaluating distinct task performance throughout an operation. METHODS: Open surgeries was videoed and surgeon hands were tracked without using sensors or markers. An expert panel of 3 attending surgeons rated tying and suturing video clips on continuous scales from 0 to 10 along 3 task measures adapted from the broader OSATS: motion economy, fluidity of motion, and tissue handling. Empirical models were developed to predict the expert consensus ratings based on the hand kinematic data records. RESULTS: The predicted versus panel ratings for suturing had slopes from 0.73 to 1, and intercepts from 0.36 to 1.54 (Average R2 = 0.81). Predicted versus panel ratings for tying had slopes from 0.39 to 0.88, and intercepts from 0.79 to 4.36 (Average R2 = 0.57). The mean square error among predicted and expert ratings was consistently less than the mean squared difference among individual expert ratings and the eventual consensus ratings. CONCLUSIONS: The computer algorithm consistently predicted the panel ratings of individual tasks, and were more objective and reliable than individual assessment by surgical experts.

Safety considerations in learning new procedures: a survey of surgeons.

BACKGROUND: There exists a tension between surgical innovation and safety. The learning curve associated with the introduction of new procedures/technologies has been associated with preventable patient harm. Surgeon's perceptions regarding the safety of methods for learning new procedures/technologies are largely uncharacterized. MATERIALS AND METHODS: A survey was designed to evaluate surgeons' perceptions related to learning new procedures/technologies. This included clinical vignettes across two domains: (1) experience with an operation (e.g., colectomy) and (2) experience with a technology (e.g., laparoscopy). This study also focuses on a surgeon's perceptions of existing credentialing/privileging requirements. Participants were faculty surgeons (n = 150) at two large Midwestern academic health centers. RESULTS: Survey response rate was 77% (116/150). 69% of respondents believed the processes of credentialing/privileging is "far too relaxed" or "too relaxed" for ensuring patient safety. Surgeons most commonly indicated a mini-fellowship is required to learn a new laparoscopic procedure. However, that requirement differed based on a surgeon's prior experience with laparoscopy. For example, to learn laparoscopic colectomy, 35% of respondents felt a surgeon with limited laparoscopic experience should complete a mini-fellowship, whereas 3% felt this was necessary if the surgeon had extensive laparoscopic experience. In the latter scenario, most respondents felt a surgeon should scrub in cases performed by an expert (38%) or perform cases under a proctor's supervision (33%) when learning laparoscopic colectomy. CONCLUSIONS: Many surgeons believe existing hospital credentialing/privileging practices may be too relaxed. Moreover, surgeons believe the "one-size-fits-all" approach for training practicing surgeons may not protect patients from unsafe introduction of new procedures/technologies.

Institutional variation in surgical care for early-stage breast cancer at community hospitals.

BACKGROUND: There is significant institutional variation in the surgical care of breast cancer, and this may reflect access to services and resultant physician practice patterns. In previous studies, specialty care has been associated with variation in the operative treatment of breast cancer but has not been evaluated in a community setting. This study investigates these issues in a cohort of 59 community hospitals in the United States. MATERIALS AND METHODS: Data on patients receiving an operation for breast cancer (2006-2009) in a large, geographically diverse cohort of hospitals were obtained. Administrative data, autoabstracted cancer-specific variables from free text, and multiple other data sets were combined. Polymotous logistic regression with multilevel outcomes identified associations between these variables and surgical treatment. RESULTS: At 59 community hospitals, 4766 patients underwent breast conserving surgery (BCS), mastectomy, or mastectomy with reconstruction. The older patients were most likely to receive mastectomy alone, whereas the younger age group underwent more reconstruction (age <50), and BCS was most likely in patients aged 50-65. Surgical procedure also varied according to tumor characteristics. BCS was more likely at smaller hospitals, those with ambulatory surgery centers, and those located in nonmetropolitan areas. The likelihood of reconstruction doubled when there were more reconstructive surgeons in the health services area (P = 0.02). BCS was more likely when radiation oncology services were available within the hospital or network (P = 0.04). CONCLUSIONS: Interpretation of these results for practice redesign is not straightforward. Although access to specialty care is statistically associated with type of breast surgical procedure, clinical impact is limited. It may be more effective to target other aspects of care to ensure each patient receives treatment consistent with her individual preferences.