Practical Guide to Survey Research in Surgical Education.

This Guide to Statistics and Methods describes aspects of methods of survey research in surgical education, important considerations, and pitfalls and limitations.

Resident Readiness for Senior Level Decision Making: Identifying the Domains for Formative Assessment and Feedback.

OBJECTIVE: To establish expert consensus regarding the domains and topics for senior surgery residents (PGY-4) to make critical decisions and assume senior-level responsibilities, and to develop the formative American College of Surgeons Senior Resident Readiness Assessment (ACS SRRA) Program. DESIGN: The American College of Surgeons (ACS) education leadership team conducted a focus group with surgical experts to identify the content for an assessment tool to evaluate senior residents' readiness for their increased levels of responsibility. After the focus group, national experts were recruited to develop consensus on the topics through three rounds of surveys using Delphi methodology. The Delphi participants rated topics using Likert-type scales and their comments were incorporated into subsequent rounds. Consensus was defined as ≥ 80% agreement with internal-consistency reliability (Cronbach's alpha) ≥ 0.8. In a stepwise fashion, topics that did not achieve consensus for inclusion were removed from subsequent survey rounds. SETTING: The surveys were administered via an online questionnaire. PARTICIPANTS: Twelve program directors and assistant program directors made up the focus group. The 39 Delphi participants represented seven different surgical subspecialties and were from diverse practice settings. The median length of experience in general surgery resident education was 20 years (IQR 14.3-30.0) with 64% of the experts being either current or past general surgery residency program directors. RESULTS: The response rate was 100% and Cronbach's alpha was ≥ 0.9 for each round. The Delphi participants contributed a large number of comments. Of the 201 topics that were evaluated initially, 120 topics in 25 core clinical areas were included to create the final domains of ACS SRRA. CONCLUSIONS: National consensus on the domain of the ACS SRRA has been achieved via the modified Delphi method among expert surgeon educators. ACS SRRA will identify clinical topics and areas in which each senior resident needs improvement and provide data to residents and residency programs to develop individualized learning plans. This would help in preparing the senior residents to assume their responsibilities and support their readiness for future fellowship training or surgical practice.

A comparative analysis of video-based surgical assessment: is evaluation of the entire critical portion of the operation necessary?

BACKGROUND: Previous studies of video-based operative assessments using crowd sourcing have established the efficacy of non-expert evaluations. Our group sought to establish the equivalence of abbreviating video content for operative assessment. METHODS: A single institution video repository of six core general surgery operations was submitted for evaluation. Each core surgery included three unique surgical performances, totaling 18 unique operative videos. Each video was edited using four different protocols based on the critical portion of the operation: (1) custom edited critical portion (2) condensed critical portion (3) first 20 s of every minute of the critical portion, and (4) first 10 s of every minute of the critical portion. In total, 72 individually edited operative videos were submitted to the C-SATS (Crowd-Sourced Assessment of Technical Skills) platform (C-SATS) for evaluation. Aggregate score for study protocol was compared using the Kruskal-Wallis test. A multivariable, multilevel mixed-effects model was constructed to predict total skill assessment scores. RESULTS: Median video lengths for each protocol were: custom, 6:20 (IQR 5:27-7:28); condensed, 10:35 (8:50-12:06); 10 s, 4:35 (2:11-6:09); and 20 s, 9:09 (4:20-12:14). There was no difference in aggregate median score among the four study protocols: custom, 15.7 (14.4-16.2); condensed, 15.8 (15.2-16.4); 10 s, 15.8 (15.3-16.1); 20 s, 16.0 (15.1-16.3); χ2 = 1.661, p = 0.65. Regression modeling demonstrated a significant, but minimal effect of the 10 s and 20 s editing protocols compared to the custom method on individual video score: condensed, + 0.33 (- 0.05-0.70), p = 0.09; 10 s, + 0.29 (0.04-0.55), p = 0.03; 20 s, + 0.40 (0.15-0.66), p = 0.002. CONCLUSION: A standardized protocol for video editing abbreviated surgical performances yields reproducible assessment of surgical aptitude when assessed by non-experts.

Crowd-Sourced and Attending Assessment of General Surgery Resident Operative Performance Using Global Ratings Scales.

OBJECTIVE: We sought to assess the extent to which both crowd and intraoperative attending ratings using objective structured assessment of technical skill (OSATS) or global objective assessment of laparoscopic skills (GOALS) would correlate with the system for improving procedural learning (SIMPL) Zwisch and Performance scales. DESIGN: Comparison of directly observed versus crowd sourced review of operative video. SETTING: Operative video captured at 2 institutions. PARTICIPANTS: Six (6) core general surgery procedures, 3 open and 3 laparoscopic, were selected from the American Board of Surgery's Resident Assessments list. Thirty-two cases performed by General Surgery residents across all training levels at 2 institutions were filmed. Videos were condensed using a standardized protocol to include the critical portion of the procedure.  Condensed videos were then submitted to crowd-sourced assessment of technical skills (C-SATS), an online crowd source-driven assessment service, for assessment using the appropriate resident assessment form (GOALS or OSATS) as well as with the SIMPL Zwisch and Performance scales. Crowd workers watched an educational tutorial on how to use the Zwisch and SIMPL Performance rating scales prior to participating. Attendings scored residents using the same tools immediately after the shared operative experience. Statistical analysis was performed using Pearson's correlation coefficient. RESULTS: Crowd raters evaluated 32 procedures using GOALS/OSATS, Zwisch and Performance (35-50 ratings per video). Attendings also evaluated all 32 procedures using GOALS/OSATS and 26 of the procedures using SIMPL Zwisch and Performance. Pearson correlation coefficients with 95% confidence intervals for crowd ratings were: GOALS and Zwisch -0.40 [-0.73 to 0.10], OSATS and Zwisch 0.11 [-0.41 to 0.57], GOALS and Performance -0.06 [-0.44 to 0.35], and OSATS and Performance 0.22 [-0.46 to 0.20]. Pearson correlation coefficients for attendings were: GOALS and Zwisch (0.77), OSATS and Zwisch (0.65), GOALS and Performance (0.93), and OSATS and Performance (0.59). CONCLUSIONS: Overall, correlations between crowd-sourced ratings using GOALS/OSATS and SIMPL global operative performance ratings tools were weak, yet for attendings, they were strong. Direct attending assessment may be required for evaluation of global performance while crowd sourcing may be more suitable for technical assessment.  Further studies are needed to see if more extensive crowd training would result in improved ability for global performance evaluation.

Providing complex GI surgical care with minimally invasive approaches: a survey of the practice patterns of Fellowship Council alumni.

INTRODUCTION: The Fellowship Council (FC) oversees 172 non-ACGME surgical fellowships offering 211 fellowship positions per year. These training programs cover multiple specialties including Advanced gastrointestinal (GI), Advanced GI/MIS, Bariatric, Hepatopancreaticobiliary (HPB), Flexible Endoscopy, Colorectal, and Thoracic Surgery. Although some data have been published detailing the practice environments (i.e., urban vs. rural) and yearly total case volumes of FC alumni, there is a lack of granular data regarding the practice patterns of FC graduates. The aim of this study was to gather detailed data on the specific case types performed and surgical approaches employed by recent FC alumni. METHODS: A 21-item survey covering 64 data points was emailed to 835 FC alumni who completed their fellowship between 2013 and 2017. Email addresses were obtained from FC program directors and FC archives. RESULTS: We received 327 responses (39% response rate). HPB, Advanced Colorectal, and Advanced Thoracic alumni appear to establish practices focused on their respective fields. Graduates from Advanced GI, Adv GI/MIS, and Bariatric programs appear to build practices with a mix of several complex GI case types including bariatrics, colorectal, foregut, HPB, and hernia cases. CONCLUSIONS: This is the first large data set to provide granular information on the practice patterns of FC alumni. FC trained surgeons perform impressive volumes of complex procedures, and minimally invasive approaches are extremely prevalent in these practices. Further, many graduates carve out practices with large footprints in robotics and endoscopy.

Training for a Career in Rural and Nonmetropolitan Surgery-A Practical Needs Assessment.

OBJECTIVE: The current shortage of surgeons in rural and smaller communities is predicted to get worse. In this study, we solicited practicing rural surgeons' opinions about the skill set needed in a rural practice in order to inform curriculum development for general surgery residents who intend to embark on rural careers. DESIGN: We developed an online survey consisting of demographic questions and closed- and open-ended questions regarding current practice environment and scope of practice. Priorities for training were identified using descriptive analyses of both the quantitative and qualitative data, including frequency of responses regarding specific skills training. PARTICIPANTS: We surveyed currently practicing surgeons who subscribe to the American College of Surgeons Rural Surgery listserv. RESULTS: 237 surgeons from 49 states and 1 Canadian territory responded; 60% of participants had been in practice for 20 or more years, and 70% did not pursue subspecialty training. Valuable skills identified for rural surgeons were: endoscopy, advanced laparoscopy, and basic non-general surgery subspecialty procedures. Regardless of years of practice or setting, respondents felt that rural experience during residency was highly valuable (82%) and overwhelmingly supported training future rural surgeons at residency programs with broad general surgery experiences and high case volumes with no or few fellows. CONCLUSIONS: Practicing rural surgeons identify endoscopy, basic non-general surgery subspecialty procedures, and advanced laparoscopy as key components of their current practice. These skills may not be strongly emphasized in traditional general surgery training programs. Surgical educators should focus on developing curricula that emphasize these areas in order to prepare residents for careers in rural surgery.

Initiation of adjuvant therapy following surgical resection of pancreatic ductal adenocarcinoma (PDAC): Are patients from rural, remote areas disadvantaged?

BACKGROUND AND OBJECTIVES: Although race and socioeconomic status have been shown to affect outcomes in pancreatic ductal adenocarcinoma (PDAC), the impact of rural residence on the delivery of adjuvant therapy (AT) has not been studied. METHODS: Patients with resected PDAC were identified using the National Cancer Database (NCDB). Individuals were classified as living in a metro area, urban/rural adjacent to a metro area (URA), and urban/rural remote (URR) area. Multivariate logistic regression was used to assess geographic inhabitance as a predictor of receiving AT. RESULTS: A total of 32 521 individuals who underwent pancreatectomy for PDAC were identified. Univariate analysis demonstrated individuals in URR areas were less likely to receive adjuvant chemotherapy (ACT) than those living in URA or metro areas (55.3% vs 55.6% vs 58.8%, P = 0.011). However on multivariate analysis URR inhabitance was no longer a predictor of ACT (OR = 0.911 P = 0.125) or ART (OR = 0.953 P = 0.462). Cox proportional hazard modeling demonstrated URR inhabitance remained independently associated with poor OS (HR 1.076; 95% CI [1.008, 1.149], P < 0.029). CONCLUSIONS: URR inhabitance does not impact access to AT, however it is independently associated with a decreased OS. Attention must be focused on optimizing oncologic care to patients with disparate access to healthcare.

Concerns of Quality and Safety in Public Domain Surgical Education Videos: An Assessment of the Critical View of Safety in Frequently Used Laparoscopic Cholecystectomy Videos.

BACKGROUND: Online videos are among the most common resources for case preparation. Using crowd sourcing, we evaluated the relationship between operative quality and viewing characteristics of online laparoscopic cholecystectomy videos. STUDY DESIGN: We edited 160 online videos of laparoscopic cholecystectomy to 60 seconds or less. Crowd workers (CW) rated videos using Global Objective Assessment of Laparoscopic Skills (GOALS), the critical view of safety (CVS) criteria, and assigned overall pass/fail ratings if CVS was achieved; linear mixed effects models derived average ratings. Views, likes, dislikes, subscribers, and country were recorded for subset analysis of YouTube videos. Spearman correlation coefficient (SCC) assessed correlation between performance measures. RESULTS: One video (0.06%) achieved a passing CVS score of ≥5; 23%, ≥4; 44%, ≥3; 79%, ≥2; and 100% ≥1. Pass/fail ratings correlated to CVS, SCC 0.95 (p < 0.001) and to GOALS, SCC 0.79 (p < 0.001). YouTube videos (n = 139) with higher views, likes, or subscribers did not correlate with better quality. The average CVS and GOALS scores were no different for videos with >20,000 views (22%) compared with those with <20,000 (78%). CONCLUSIONS: There is an incredibly low frequency of CVS and average GOALS technical performance in frequently used online surgical videos of LC. Favorable characteristics, such as number of views or likes, do not translate to higher quality.

Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy.

BACKGROUND: Faculty experts (FE) and crowd workers (CW) can assess technical skill, but assessment of operative technique has not been explored. We sought to evaluate if CW could be taught to assess completion of the critical view of safety (CVS) in laparoscopic cholecystectomy. METHODS: We prepared 160 blinded, surgical videos of laparoscopic cholecystectomy from public domain websites. Videos were edited to ≤60 s, ending when a structure was cut/clipped. CW analyzed videos using Global Objective Assessment of Laparoscopic Skills (GOALS) and CVS criteria assessment tools after watching an instructional tutorial. Ten videos were randomly selected from each performance quartile based on GOALS. Five FE rated the 40 videos using GOALS and CVS. Linear mixed effects models derived average CW and FE ratings for GOALS and CVS for each video. Spearman correlation coefficients (SCC) were used to assess the degree of correlation between performance measures. Satisfactory completion of the CVS was defined as scoring an average CVS ≥ 5. Videos with an average GOALS ≥ 15 were considered top technical performers. RESULTS: A high degree of correlation was seen between all performance measures: CVS ratings between CW and FE, SCC 0.89 (p < 0.001); GOALS and CVS ratings SCC 0.77 (p < 0.001) for CW, and SCC 0.71 (p < 0.001) for FE. Sixteen videos were assigned top technical performer ratings by both CW and FE but the average CVS was inadequate (3.8 and 3.6, respectively), and the percentage of satisfactory CVS ≥ 5 was 12.5%. CONCLUSIONS: A high degree of correlation was found between CW and FE in assessment of the CVS. However, in this video analysis, high technical performers did not achieve a complete CVS in most cases. Educating CW to assess operative technique for the identification of low or average performers is feasible and may broaden the application of this assessment and feedback tool.

General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors.

OBJECTIVE: To assess readiness of general surgery graduate trainees entering accredited surgical subspecialty fellowships in North America. METHODS: A multidomain, global assessment survey designed by the Fellowship Council research committee was electronically sent to all subspecialty program directors. Respondents spanned minimally invasive surgery, bariatric, colorectal, hepatobiliary, and thoracic specialties. There were 46 quantitative questions distributed across 5 domains and 1 or more reflective qualitative questions/domains. RESULTS: There was a 63% response rate (n = 91/145). Of respondent program directors, 21% felt that new fellows arrived unprepared for the operating room, 38% demonstrated lack of patient ownership, 30% could not independently perform a laparoscopic cholecystectomy, and 66% were deemed unable to operate for 30 unsupervised minutes of a major procedure. With regard to laparoscopic skills, 30% could not atraumatically manipulate tissue, 26% could not recognize anatomical planes, and 56% could not suture. Furthermore, 28% of fellows were not familiar with therapeutic options and 24% were unable to recognize early signs of complications. Finally, it was felt that the majority of new fellows were unable to conceive, design, and conduct research/academic projects. Thematic clustering of qualitative data revealed deficits in domains of operative autonomy, progressive responsibility, longitudinal follow-up, and scholarly focus after general surgery education.