Impact of a coaching program on resident perceptions of communication confidence and feedback quality.

BACKGROUND: While communication is an essential skill for providing effective medical care, it is infrequently taught or directly assessed, limiting targeted feedback and behavior change. We sought to evaluate the impact of a multi-departmental longitudinal residency communication coaching program. We hypothesized that program implementation would result in improved confidence in residents' communication skills and higher-quality faculty feedback. METHODS: The program was implemented over a 3-year period (2019-2022) for surgery and neurology residents at a single institution. Trained faculty coaches met with assigned residents for coaching sessions. Each session included an observed clinical encounter, self-reflection, feedback, and goal setting. Eligible residents completed baseline and follow-up surveys regarding their perceptions of feedback and communication. Quantitative responses were analyzed using paired t-tests; qualitative responses were analyzed using content analysis. RESULTS: The baseline and follow-up survey response rates were 90.0% (126/140) and 50.5% (46/91), respectively. In a paired analysis of 40 respondents, residents reported greater confidence in their ability to communicate with patients (inpatient: 3.7 vs. 4.3, p < 0.001; outpatient: 3.5 vs. 4.2, p < 0.001), self-reflect (3.3 vs. 4.3, p < 0.001), and set goals (3.6 vs. 4.3, p < 0.001), as measured on a 5-point scale. Residents also reported greater usefulness of faculty feedback (3.3 vs. 4.2, p = 0.001). The content analysis revealed helpful elements of the program, challenges, and opportunities for improvement. Receiving mentorship, among others, was indicated as a core program strength, whereas solving session coordination and scheduling issues, as well as lowering the coach-resident ratio, were suggested as some of the improvement areas. CONCLUSIONS: These findings suggest that direct observation of communication in clinical encounters by trained faculty coaches can facilitate long-term trainee growth across multiple core competencies. Future studies should evaluate the impact on patient outcomes and workplace-based assessments.

The domino effect: the impact of gender on operative self-efficacy.

BACKGROUND: Studies suggest that there are key differences in operative experience based on a trainee's gender. A large-scale self-efficacy (SE) survey, distributed to general surgery residents after the American Board of Surgery In-Training Examination in 2020, found that female gender was associated with decreased SE in graduating PGY5 residents for all 4 laparoscopic procedures included on the survey (cholecystectomy, appendectomy, right hemicolectomy, and diagnostic laparoscopy). We sought to determine whether these differences were reflected at the case level when considering operative performance and supervision using an operative assessment tool (SIMPL OR). METHODS: Supervision and performance data reported through the SIMPL OR platform for the same 4 laparoscopic procedures included in the SE survey were aggregated for residents who were PGY5s in 2020. Independent t-tests and multiple linear regression were used to determine the relationship between trainee gender and supervision/performance ratings. RESULTS: For laparoscopic cases in aggregate (n = 2708), male residents rated their performance higher than females (3.57 vs. 3.26, p < 0.001, 1 = critical deficiency, 5 = exceptional performance) and reported less supervision (3.15 vs. 2.85, p < 0.001, 1 = show and tell, 4 = supervision only); similar findings were seen when looking at attending reports of resident supervision and performance. A multiple linear regression model showed that attending gender did not significantly predict resident-reported supervision or performance levels, while case complexity and trainee gender significantly affected both supervision and performance (p < 0.001). DISCUSSION: Female residents perceive themselves to be less self-efficacious at core laparoscopic procedures compared to their male colleagues. Comparison to more case-specific data confirm that female residents receive more supervision and lower performance ratings. This may create a domino effect in which female residents receive less operative independence, preventing the opportunity to establish SE. Further research should identify opportunities to break this cycle and consider gender identity beyond the male/female construct.

What are essential laparoscopic skills these days? Results of the SAGES Fundamentals of Laparoscopic Surgery (FLS) Committee technical skills survey.

INTRODUCTION: The Fundamentals of Laparoscopic Surgery (FLS) program tests basic knowledge and skills required to perform laparoscopic surgery. Educational experiences in laparoscopic training and development of associated competencies have evolved since FLS inception, making it important to review the definition of fundamental laparoscopic skills. The Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) assigned an FLS Technical Skills Working Group to characterize technical skills used in basic laparoscopic surgery in current practice contexts and their possible application to future FLS tests. METHODS: A group of subject matter experts defined an inventory of 65 laparoscopic skills using a Nominal Group Technique. From these, a survey was developed rating these items for importance, frequency of use, and priority for testing for FLS certification. This survey was distributed to SAGES members, recent recipients of FLS certification, and members of the Association of Program Directors in Surgery (APDS). Results were collected using a secure web-based survey platform. RESULTS: Complete data were available for 1742 surveys. Of these, 1143 comprised results for post-residency participants who performed advanced procedures. Seventeen competencies were identified for FLS testing prioritization by determining the proportion of respondents who identified them of highest priority, at median (50th percentile) of the maximum survey scale rating. These included basic peritoneal access, laparoscope and instrument use, tissue manipulation, and specific problem management skills. Sixteen could be used to show appropriateness of the domain construct by confirmatory factor analysis. Of these 8 could be characterized as manipulative tasks. Of these 5 mapped to current FLS tasks. CONCLUSIONS: This survey-identified competencies, some of which are currently assessed in FLS, with a high level of priority for testing. Further work is needed to determine if this should prompt consideration of changes or additions to the FLS technical skills test component.

Developing an Inpatient Relationship Centered Communication Curriculum (I-RCCC) rounding framework for surgical teams.

BACKGROUND: Morning rounds by an acute care surgery (ACS) service at a level one trauma center are uniquely demanding, given the fast pace, high acuity, and increased patient volume. These demands notwithstanding, communication remains integral to the success of surgical teams. Yet there are limited published curricula that address trauma inpatient communication needs. Observations at our institution confirmed that the surgical team lacked a shared mental model for communication. We hypothesized that creating a relationship-centered rounding conceptual framework model would enhance the provider-patient experience. STUDY DESIGN: A mixed-methods approach was used for this study. A multi-pronged needs assessment was conducted. Provider communion items for Press Ganey and Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) surveys were used to measure patients' expressed needs. Faculty with experience in relationship-centered communication observed morning rounds and documented demonstrated behaviors. A five-hour workshop was designed based on the identified needs. A pre-and post-course Assessment and course evaluation were conducted. Provider-related patient satisfaction items were measured six months before the course and six months after the workshop. RESULTS: Needs assessment revealed a lack of a shared communication framework and a lack of leadership skills for senior trauma residents. Barriers included: time constraints, patient load, and interruptions during rounds. The curriculum was very well received. The self-reflected behaviors that demonstrated the most dramatic change between the pre and post-workshop surveys were: I listened without interrupting; I spoke clearly and at a moderate pace; I repeated key points; and I checked that the patient understood. All these changed from being performed by 50% of respondents "about half of the time" to 100% of them "always". Press Ganey top box likelihood to recommend (LTR) and provider-related top box items showed a trend towards improvement after implementing the training with a percentage difference of up to 20%. CONCLUSION: The Inpatient Relationship Centered Communication Curriculum (I-RCCC) targeting senior residents and Nurse Practitioners (NP) was feasible, practical, and well-received by participants. There was a trend of an increase in LTRs and provider-specific patient satisfaction items. This curriculum will be refined based on the study results and potentially scalable to other surgical specialties.

Do Individual Surgeon Preferences Affect Procedural Outcomes?

OBJECTIVES: Surgeon preferences such as instrument and suture selection and idiosyncratic approaches to individual procedure steps have been largely viewed as minor differences in the surgical workflow. We hypothesized that idiosyncratic approaches could be quantified and shown to have measurable effects on procedural outcomes. METHODS: At the American College of Surgeons (ACS) Clinical Congress, experienced surgeons volunteered to wear motion tracking sensors and be videotaped while evaluating a loop of porcine intestines to identify and repair 2 preconfigured, standardized enterotomies. Video annotation was used to identify individual surgeon preferences and motion data was used to quantify surgical actions. χ 2 analysis was used to determine whether surgical preferences were associated with procedure outcomes (bowel leak). RESULTS: Surgeons' (N=255) preferences were categorized into 4 technical decisions. Three out of the 4 technical decisions (repaired injuries together, double-layer closure, corner-stitches vs no corner-stitches) played a significant role in outcomes, P <0.05. Running versus interrupted did not affect outcomes. Motion analysis revealed significant differences in average operative times (leak: 6.67 min vs no leak: 8.88 min, P =0.0004) and work effort (leak-path length=36.86 cm vs no leak-path length=49.99 cm, P =0.001). Surgeons who took the riskiest path but did not leak had better bimanual dexterity (leak=0.21/1.0 vs no leak=0.33/1.0, P =0.047) and placed more sutures during the repair (leak=4.69 sutures vs no leak=6.09 sutures, P =0.03). CONCLUSIONS: Our results show that individual preferences affect technical decisions and play a significant role in procedural outcomes. Future analysis in more complex procedures may make major contributions to our understanding of contributors to procedure outcomes.

Surgical endoscopy education research: how are we doing?

BACKGROUND: Surgical endoscopy (SE), the official journal of the Society of American Gastrointestinal and Endoscopic Surgeons and the European Association for Endoscopic Surgery, is an important source of new evidence pertaining to surgical education in the field. However, qualitative deficiencies in medical education research have prompted medical education leaders to advocate for increased methodological rigor. The purpose of this study is to review the quality of education-focused research published through SE. METHODS: A PubMed search examining all SE articles categorized as education-related research from 2010 to 2019 was conducted; studies not meeting inclusion criteria were excluded. Remaining publications were independently reviewed, classified, and scored by 7 raters using the medical education research study quality instrument (MERSQI). Intraclass correlation was calculated and data were examined with descriptive statistics. RESULTS: A total of 227 studies met inclusion criteria. There was no significant difference in number of publications by year (average 25.88 [SD 5.6]); 60% were conducted outside of the United States, and 47% (n = 106) were funded. The average MERSQI was 12.5 (SD 2). Most studies used two-group non-random (42%, n = 96) or post/cross-sectional designs (29%, n = 65). Thirty-six (16%) were randomized controlled trials. Multi-institutional studies comprised 24% (n = 54). Of the manuscripts, 96% (n = 217) reported at least one measure of validity evidence and 28% (n = 67) described three levels of validity evidence. Studies primarily reported changes in skills or knowledge (45%, n = 103) or satisfaction or general facts (44%, n = 99), while patient-related outcomes encompassed 3% (n = 6) of studies. ICC between raters was 0.93 (CI 0.90-0.93, p < 0.001). CONCLUSIONS: Based on publications to date, this journal's peer review process appears to facilitate the dissemination of education-related studies of moderate to good quality. However, there were uncovered deficits, ranging from validity evidence to study designs and level of outcomes. This journal's breadth of viewership offers a potential venue to advance education-related research.

The fundamentals of laparoscopic surgery in general surgery residency: fundamental for junior residents' self-efficacy.

BACKGROUND: Implementation of the Fundamentals of Laparoscopic Surgery (FLS) by the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) has served a need for educational structure for laparoscopic skill within General Surgery training since 2004. This study looks at how FLS affects resident self-efficacy (SE) with laparoscopic procedures. METHODS: We conducted a national survey, linked to the 2020 American Board of Surgery In-Training Examination (ABSITE), in which 9275 residents from 325 US General Surgery Training Programs participated. The online survey included multimodal questions that analyzed whether participants felt they could perform the most commonly-logged laparoscopic operations among residents [Laparoscopic Appendectomy (LA), Laparoscopic Cholecystectomy (LC), Laparoscopic Right Hemicolectomy (LRH), Diagnostic Laparoscopy (DL)] without faculty assistance. This used a 5-point scaled assessment, ranging from "not able to" to "definitely able to." Multivariate analyses determined if completion of FLS made a difference for resident self-efficacy, stratified by post-graduate year (PGY). RESULTS: At the time of the survey, 2300 reported completion of FLS. The percentage of FLS completion increased from PGY1 to PGY5 (4.2% n = 59 vs 85.8% n = 893). PGY1 residents who completed FLS, from 48 diverse institutions, demonstrated the most significant increases in SE (p < 0.05) with significantly higher perceived self-efficacy in LA (p = 0.001) and LRH (p = 0.012). PGY2 and PGY3 residents indicated increased SE in DL (p = 0.037, p = 0.015, respectively), based on FLS completion. These FLS effects were less evident in the more senior classes. CONCLUSIONS: Completion of FLS arguably has the greatest benefits for more junior residents, as it establishes a foundation of laparoscopic knowledge and skill, upon which further residency training can build. Successful completion of the curriculum and assessment offered by the Fundamentals of Laparoscopic Surgery leads to greater sense of ability in early trainees.

Developing a multi-departmental residency communication coaching program.

Background: Local needs assessments in our institution's surgery and neurology residency programs identified barriers to effective communication, such as no shared communication framework and limited feedback on nontechnical clinical skills. Residents identified faculty-led coaching as a desired educational intervention to improve communication skills. Three university departments (Surgery, Neurology, and Pediatrics) and health-care system leaders collaborated closely to develop an innovative communication coaching initiative generalizable to other residency programs. Innovation: Coaching program development involved several layers of collaboration between health-care system leaders, faculty educators, and departmental communication champions. The efforts included: (1) creating and delivering communication skills training to faculty and residents; (2) hosting frequent meetings among various stakeholders to develop program strategy, discuss opportunities and learnings, and engage other medical educators interested in coaching; (3) obtaining funding to implement the coaching initiative; (4) selecting coaches and providing salary and training support. Evaluation: A multi-phased mixed-methods study utilized online surveys and virtual semi-structured interviews to assess the program's quality and impact on the communication culture and the satisfaction and communication skills of residents. Quantitative and qualitative data have been integrated during data collection and analysis using embedding, building, and merging strategies. Discussion and Implications: Establishing a multi-departmental coaching program may be feasible and can be adapted by other programs if similar resources and focus are present. We found that stakeholders' buy-in, financial support, protected faculty time, flexible approach, and rigorous evaluation are crucial factors in successfully implementing and sustaining such an initiative.

From Listening to Action: Academic Surgical Departmental Response to Social Injustice Through Curricular Development.

OBJECTIVE: The aim of this study was to describe the development and evaluation of a structured department wide cultural competency curriculum. SUMMARY BACKGROUND DATA: Despite numerous organizational policies and statements, social injustice and bias still exist. Our department committed to assist individuals of the entire department to develop foundational knowledge and skills to combat implicit bias and systemic racism through the creation of a cultural competency curriculum. The purpose of this manuscript is to detail our curriculum and the evaluation of its effectiveness. METHODS: Using a well-established curriculum development framework, a cultural competency curriculum was developed focusing on knowledge, skills and attitudes at the individual level, for all members of the department. The curriculum was implemented through 6-hour-long sessions over a 9-week period. Effectiveness was assessed through a post curriculum survey. RESULTS: Twenty percent of the respondents had experienced bias based on race, ethnicity, or sexual orientation in the past 12 months, whereas 30% had experienced bias based on sex. Seventy-one percent independently explored related topics. The curriculum was overall well received and generally achieved the goals and objectives. CONCLUSION: Using a standard curriculum development framework, an effective department-wide cultural competency curriculum can be developed and implemented.

Situating Artificial Intelligence in Surgery: A Focus on Disease Severity.

OBJECTIVES: Artificial intelligence (AI) has numerous applications in surgical quality assurance. We assessed AI accuracy in evaluating the critical view of safety (CVS) and intraoperative events during laparoscopic cholecystectomy. We hypothesized that AI accuracy and intraoperative events are associated with disease severity. METHODS: One thousand fifty-one laparoscopic cholecystectomy videos were annotated by AI for disease severity (Parkland Scale), CVS achievement (Strasberg Criteria), and intraoperative events. Surgeons performed focused video review on procedures with ≥1 intraoperative events (n = 335). AI versus surgeon annotation of CVS components and intraoperative events were compared. For all cases (n = 1051), intraoperative-event association with CVS achievement and severity was examined using ordinal logistic regression. RESULTS: Using AI annotation, surgeons reviewed 50 videos/hr. CVS was achieved in ≤10% of cases. Hepatocystic triangle and cystic plate visualization was achieved more often in low-severity cases (P < 0.03). AI-surgeon agreement for all CVS components exceeded 75%, with higher agreement in high-severity cases (P < 0.03). Surgeons agreed with 99% of AI-annotated intraoperative events. AI-annotated intraoperative events were associated with both disease severity and number of CVS components not achieved. Intraoperative events occurred more frequently in high-severity versus low-severity cases (0.98 vs 0.40 events/case, P < 0.001). CONCLUSIONS: AI annotation allows for efficient video review and is a promising quality assurance tool. Disease severity may limit its use and surgeon oversight is still required, especially in complex cases. Continued refinement may improve AI applicability and allow for automated assessment.

Simulation-based mastery learning for endoscopy using the endoscopy training system: a strategy to improve endoscopic skills and prepare for the fundamentals of endoscopic surgery (FES) manual skills exam.

INTRODUCTION: The fundamentals of endoscopic surgery (FES) program has considerable validity evidence for its use in measuring the knowledge, skills, and abilities required for competency in endoscopy. Beginning in 2018, the American Board of Surgery will require all candidates to have taken and passed the written and performance exams in the FES program. Recent work has shown that the current ACGME/ABS required case volume may not be enough to ensure trainees pass the FES skills exam. The aim of this study was to investigate the feasibility of a simulation-based mastery-learning curriculum delivered on a novel physical simulation platform to prepare trainees to pass the FES manual skills exam. METHODS: The newly developed endoscopy training system (ETS) was used as the training platform. Seventeen PGY 1 (10) and PGY 2 (7) general surgery residents completed a pre-training assessment consisting of all 5 FES tasks on the GI Mentor II. Subjects then trained to previously determined expert performance benchmarks on each of 5 ETS tasks. Once training benchmarks were reached for all tasks, a post-training assessment was performed with all 5 FES tasks. RESULTS: Two subjects were lost to follow-up and never returned for training or post-training assessment. One additional subject failed to complete any portion of the curriculum, but did return for post-training assessment. The group had minimal endoscopy experience (median 0, range 0-67) and minimal prior simulation experience. Three trainees (17.6%) achieved a passing score on the pre-training FES assessment. Training consisted of an average of 48 ± 26 repetitions on the ETS platform distributed over 5.1 ± 2 training sessions. Seventy-one percent achieved proficiency on all 5 ETS tasks. There was dramatic improvement demonstrated on the mean post-training FES assessment when compared to pre-training (74.0 ± 8 vs. 50.4 ± 16, p < 0.0001, effect size = 2.4). The number of ETS tasks trained to proficiency correlated moderately with the score on the post-training assessment (r = 0.57, p = 0.028). Fourteen (100%) subjects who trained to proficiency on at least one ETS task passed the post-training FES manual skills exam. CONCLUSIONS: This simulation-based mastery learning curriculum using the ETS is feasible for training novices and allows for the acquisition of the technical skills required to pass the FES manual skills exam. This curriculum should be strongly considered by programs wishing to ensure that trainees are prepared for the FES exam.

SAGES University MASTERS Program: a structured curriculum for deliberate, lifelong learning.

BACKGROUND: Postgraduate training has been haphazard to date. Surgeons have relied on attendance to annual meetings and multiple choice study guides to demonstrate maintenance of certification and continuing medical education. METHODS: SAGES held a retreat to develop the concept and scope of the Masters Program. Surveys were sent to SAGES members to guide curriculum development and selection of anchoring operations. RESULTS: SAGES has developed an educational curriculum across eight domains (Acute Care, Biliary, Bariatric, Colorectal, Hernia, Foregut, Flex Endoscopy, and Robotic Surgery) incorporating SAGES educational materials and guidelines, social media, coaching and mentoring. CONCLUSIONS: Deliberate, lifelong learning should be a better way to teach and learn.

Using Simulation to Improve Systems-Based Practices.

BACKGROUND: Ensuring the safe, effective management of patients requires efficient processes of care within a smoothly operating system in which highly reliable teams of talented, skilled health care providers are able to use the vast array of high-technology resources and intensive care techniques available. Simulation can play a unique role in exploring and improving the complex perioperative system by proactively identifying latent safety threats and mitigating their damage to ensure that all those who work in this critical health care environment can provide optimal levels of patient care. METHODS: A panel of five experts from a wide range of institutions was brought together to discuss the added value of simulation-based training for improving systems-based aspects of the perioperative service line. Panelists shared the way in which simulation was demonstrated at their institutions. The themes discussed by each panel member were delineated into four avenues through which simulation-based techniques have been used. RESULTS: Simulation-based techniques are being used in (1) testing new clinical workspaces and facilities before they open to identify potential latent conditions; (2) practicing how to identify the deteriorating patient and escalate care in an effective manner; (3) performing prospective root cause analyses to address system weaknesses leading to sentinel events; and (4) evaluating the efficiency and effectiveness of the electronic health record in the perioperative setting. CONCLUSION: This focused review of simulation-based interventions to test and improve components of the perioperative microsystem, which includes literature that has emerged since the panel's presentation, highlights the broad-based utility of simulation-based technologies in health care.

Design of vessel ligation simulator for deliberate practice.

BACKGROUND: Surgical residents develop technical skills at variable rates, often based on random chance of cases encountered. One such skill is tying secure knots without exerting excessive force. This study describes the design of a simulator using a force sensor to measure instantaneous forces exerted on a blood vessel analog during vessel ligation and the development of expert-derived performance goals. MATERIALS AND METHODS: Vessel ligations were performed on Silastic tubing at an offset from a Vernier Force Sensor. Nine experts (surgical faculty and senior residents) and 10 novices (junior residents) were recruited to each perform 10 vessel ligations (two square knots each) with two-handed and one-handed techniques. Internal consistency for the series of vessel ligations was tested with Cronbach alpha. Maximum forces exerted by novices and experts were compared using Student t-test. RESULTS: Internal consistency across the 10 ligations on the simulator was excellent (Cronbach alpha = 0.91). The expert group on average exerted a significantly lower maximum force when compared with novices while performing two-handed (0.76 ± 0.39 N versus 1.12 ± 0.49 N, P < 0.01) and one-handed (0.84 ± 0.32 N versus 1.36 ± 0.44 N, P < 0.01) vessel ligations. CONCLUSIONS: Although the expert group performed vessel ligations with significantly lower peak force than the novice group, there were novices who performed at the expert level. This is consistent with the conceptual framework of milestones and suggests that the skill of gentle knot-tying can be measured and develops at different chronologic levels of training in different individuals. This simulator can be used as part of a deliberate practice curriculum with instantaneous visual feedback.

Camera navigation and cannulation: validity evidence for new educational tasks to complement the Fundamentals of Laparoscopic Surgery program.

BACKGROUND: Experts identified camera navigation and cannulation as important skills that are not assessed by the Fundamentals of Laparoscopic Surgery (FLS) hands-on examination. The purpose of this study was to create metrics for and evaluate the validity for two new tasks: camera navigation (N) and cannulation (C), and to explore the potential value of adding these tasks to the FLS program. METHODS: Participants were assessed by two raters during performance of N and C in addition to the five standard FLS tasks. They also completed a questionnaire regarding the educational value of the new tasks. Validity evidence was assessed by comparing performance between Novice (PGY 1 and 2) and Experienced (PGY 3 and higher) participants, and by correlating new task scores with standard FLS scores. The ability to predict level of training using scores was evaluated by regression analysis. RESULTS: Sixty subjects participated from five North American centers. Inter-rater reliabilities for both tasks were 0.99. Novice and Experienced participants scored 74 ± 17.8 versus 85 ± 8.3 (p < 0.01) and 21 ± 17.3 versus 39 ± 20.1 (p < 0.01) on N and C tasks, respectively. Correlations with total FLS scores for N and C were 0.39 and 0.53, respectively. Prediction of training level using the combination of all seven tasks was 52.6 % (R (2) = 0.526, p < 0.01), adding an additional 2.2 % to the five FLS tasks. Of 55 participants with laparoscopic experience, 51 % reported N to be similar in difficulty to reality. Of 28 participants who perform intraoperative cholangiograms, 43 % found C to be more difficult than reality. Most (70 %) participants thought the new tasks added value to FLS. CONCLUSIONS: This study provides preliminary validity evidence for the metrics of these new tasks. The value of adding these tasks to the FLS manual skills assessment is marginal in terms of predicting level of training.

Establishing technical performance norms for general surgery residents.

BACKGROUND: Surgery residents are required to achieve performance milestones to advance in their residency. Level-specific, technical performance norms that could be used as milestones, however, do not currently exist. Our aim was to develop level-specific, technical performance norms for general surgery residents on select simulated tasks across multiple institutions. STUDY DESIGN: An IRB-approved, prospective, multi-institutional collaborative study with voluntary participation of residents was undertaken at the start of the 2011-2012 academic year. General surgery residents (PGY I-V) from seven institutions were tested on three laparoscopic and five open simulated surgical tasks, and their performance was assessed based on task time and errors. Means and standard deviations of performance for each resident level were calculated and compared. Residents with performance 1 standard deviation below the mean were considered outliers. RESULTS: A total of 147 residents were evaluated. Mean resident age was 28 ± 3 years; 42 % were female; and they had attended 74 different medical schools. Senior residents (PGY III-V) had more clinical and simulator experience than junior residents (PGY I-II) (p < 0.001). Resident performance scores progressively increased in all tasks reaching a plateau at a lower PGY level for open tasks. Depending on the task, 0-18 % of residents were outliers. When surveyed, 66 % of residents agreed that national performance norms for residents should exist. CONCLUSIONS: Performance norms were established for select tasks in a representative sample of US surgery residents. Such performance norms allow a more informed assessment of resident skill through comparison to national data and enable the identification of outliers who may benefit from additional training.

Fundamentals of Endoscopic Surgery cognitive examination: development and validity evidence.

BACKGROUND: Flexible endoscopy is an integral part of surgical care. Exposure to endoscopic procedures varies greatly in surgical training. The Society of American Gastrointestinal and Endoscopic Surgeons has developed the Fundamentals of Endoscopic Surgery (FES), which serves to teach and assess the fundamental knowledge and skills required to practice flexible endoscopy of the gastrointestinal tract. This report describes the validity evidence in the development of the FES cognitive examination. METHODS: Core areas in the practice of gastrointestinal endoscopy were identified through facilitated expert focus groups to establish validity evidence for the test content. Test items then were developed based on the content areas. Prospective enrollment of participants at various levels of training and experience was used for beta testing. Two FES cognitive test versions then were developed based on beta testing data. The Angoff and contrasting group methods were used to determine the passing score. Validity evidence was established through correlation of experience level with examination score. RESULTS: A total of 220 test items were developed in accordance with the defined test blueprint and formulated into two versions of 120 questions each. The versions were administered randomly to 363 participants. The correlation between test scores and training level was high (r = 0.69), with similar results noted for contrasting groups based on endoscopic rotation and endoscopic procedural experience. Items then were selected for two test forms of 75 items each, and a passing score was established. CONCLUSIONS: The FES cognitive examination is the first test with validity evidence to assess the basic knowledge needed to perform flexible endoscopy. Combined with the hands-on skills examination, this assessment tool is a key component for FES certification.

Fundamentals of endoscopic surgery: creation and validation of the hands-on test.

BACKGROUND: The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score. METHODS: Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach's alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves. RESULTS: A total of 160 participants (17 % GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants' level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach's alpha) was 0.82. Test-retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21. CONCLUSIONS: The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.

Using simulation to improve root cause analysis of adverse surgical outcomes.

OBJECTIVE: The purpose of this study was to develop and test a simulation method of conducting investigation of the causality of adverse surgical outcomes. DESIGN: Six hundred and thirty-one closed claims of a major medical malpractice insurance company were reviewed. Each case had undergone conventional root cause analysis (RCA). Claims were categorized by comparing the predominant underlying cause documented in the case files. Three cases were selected for simulation. SETTING: All records (medical and legal) were analyzed. Simulation scenarios were developed by abstracting data from the records and then developing paper and electronic medical records, choosing appropriate STUDY PARTICIPANTS: including test subjects and confederates, scripting the simulation and choosing the appropriate simulated environment. INTERVENTION: In a simulation center, each case simulation was run 6-7 times and recorded, with participants debriefed at the conclusion. MAIN OUTCOME MEASURES: Sources of error identified during simulation were compared with those noted in the closed claims. Test subject decision-making was assessed qualitatively. RESULTS: Simulation of adverse outcomes (SAOs) identified more system errors and revealed the way complex decisions were made by test subjects. Compared with conventional RCA, SAO identified root causes less focused on errors by individuals and more on systems-based error. CONCLUSIONS: The use of simulation for investigation of adverse surgical outcomes is feasible and identifies causes that may be more amenable to effective systems changes than conventional RCA. The information that SAO provides may facilitate the implementation of corrective measures, decreasing the risk of recurrence and improving patient safety.