Practical Guide to Surgical Simulation Research.

This Guide to Statistics and Methods provides an overview of simulation-based education and its impact on surgeon performance.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test.

BACKGROUND: Situational judgment tests (SJT) are hypothetical but realistic scenario-based assessments that allow residency programs to measure judgment and decision-making among future trainees. A surgery-specific SJT was created to identify highly valued competencies among residency applicants. We aim to demonstrate a stepwise process for validation of this assessment for applicant screening through exploration of two often-overlooked sources of validity evidence - relations with other variables and consequences. METHODS: This was a prospective multi-institutional study involving 7 general surgery residency programs. All applicants completed the SurgSJT, a 32-item test aimed to measure 10 core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Performance on the SJT was compared to application data, including race, ethnicity, gender, medical school, and USMLE scores. Medical school rankings were determined based on the 2022 U.S. News & World Report rankings. RESULTS: In total, 1491 applicants across seven residency programs were invited to complete the SJT. Of these, 1454 (97.5%) candidates completed the assessment. Applicants were predominantly White (57.5%), Asian (21.6%), Hispanic (9.7%), Black (7.3%), and 52% female. A total of 208 medical schools were represented, majority were allopathic (87.1%) and located in United States (98.7%). Less than a quarter of applicants (22.8%; N=337) were from a top 25 school based on U.S. News & World Report rankings for primary care, surgery, or research. Average USMLE Step 1 score was 235 (SD 37) and Step 2 score was 250 (SD 29). Sex, race, ethnicity, and medical school ranking did not significantly impact performance on the SJT. There was no relationship between SJT score and USMLE scores and medical school rankings. CONCLUSIONS: We demonstrate the process of validity testing and importance of two specific sources of evidence-consequences and relations with other variables, in implementing future educational assessments.

Video-based assessment for laparoscopic fundoplication: initial development of a robust tool for operative performance assessment.

INTRODUCTION: While better technical performance correlates with improved outcomes, there is a lack of procedure-specific tools to perform video-based assessment (VBA). SAGES is developing a series of VBA tools with enough validity evidence to allow reliable measurement of surgeon competence. A task force was established to develop a VBA tool for laparoscopic fundoplication using an evidence-based process that can be replicated for additional procedures. The first step in this process was to seek content validity evidence. METHODS: Forty-two subject matter experts (SME) in laparoscopic fundoplication were interviewed to obtain consensus on procedural steps, identify potential variations in technique, and to generate an inventory of required skills and common errors. The results of these interviews were used to inform creation of a task inventory questionnaire (TIQ) that was delivered to a larger SME group (n = 188) to quantify the criticality and difficulty of the procedural steps, the impact of potential errors associated with each step, the technical skills required to complete the procedure, and the likelihood that future techniques or technologies may change the presence or importance of any of these factors. Results of the TIQ were used to generate a list of steps, skills, and errors with strong validity evidence. RESULTS: Initial SMEs interviewed included fellowship program directors (45%), recent fellows (24%), international surgeons (19%), and highly experienced super SMEs with quality outcomes data (12%). Qualitative analysis of interview data identified 6 main procedural steps (visualization, hiatal dissection, fundus mobilization, esophageal mobilization, hiatal repair, and wrap creation) each with 2-5 sub steps. Additionally, the TIQ identified 5-10 potential errors for each step and 11 key technical skills required to perform the procedure. Based on the TIQ, the mean criticality and difficulty scores for the 11/21 sub steps included in the final scoring rubric is 4.66/5 (5 = absolutely essential for patient outcomes) and 3.53/5 (5 = difficulty level requires significant experience and use of alternative strategies to accomplish consistently), respectively. The mean criticality and frequency scores for the 9/11 technical skills included is 4.51/5 and 4.51/5 (5 = constantly used ≥ 80% of the time), respectively. The mean impact score of the 42/47 errors incorporated into the final rubric is 3.85/5 (5 = significant error that is unrecoverable, or even if recovered, likely to have a negative impact on patient outcome). CONCLUSIONS: A rigorous, multi-method process has documented the content validity evidence for the SAGES video-based assessment tool for laparoscopic fundoplication. Work is ongoing to pilot the assessment tool on recorded fundoplication procedures to establish reliability and further validity evidence.

Pursuing Excellence: The Power of Selection Science to Provide Meaningful Data and Enhance Efficiency in Selecting Surgical Trainees.

INTRODUCTION: As current screening methods for selecting surgical trainees are receiving increasing scrutiny, development of a more efficient and effective selection system is needed. We describe the process of creating an evidence-based selection system and examine its impact on screening efficiency, faculty perceptions, and improving representation of underrepresented minorities. METHODS: The program partnered with an expert in organizational science to identify fellowship position requirements and associated competencies. Situational judgment tests, personality profiles, structured interviews, and technical skills assessments were used to measure these competencies. The situational judgment test and personality profiles were administered online and used to identify candidates to invite for on-site structured interviews and skills testing. A final rank list was created based on all data points and their respective importance. All faculty completed follow-up surveys regarding their perceptions of the process. Candidate demographic and experience data were pulled from the application website. RESULTS: Fifty-five of 72 applicants met eligibility requirements and were invited to take the online assessment, with 50 (91%) completing it. Average time to complete was 42 ±â€Š12 minutes. Eighteen applicants (35%) were invited for on-site structured interviews and skills testing-a greater than 50% reduction in number of invites compared to prior years. Time estimates reveal that the process will result in a time savings of 68% for future iterations, compared to traditional methodologies. Fellowship faculty (N = 5) agreed on the value and efficiency of the process. Underrepresented minority candidates increased from an initial 70% to 92% being invited for an interview and ranked using the new screening tools. DISCUSSION: Applying selection science to the process of choosing surgical trainees is feasible, efficient, and well-received by faculty for making selection decisions.

Informing the Revolution: A Needs Assessment of Ultrasound Knowledge and Skills Among Graduating Physician Assistant Students.

PURPOSE: We performed a needs assessment to understand how existing physician assistant (PA) program curricula and clinical training affect students' ultrasound knowledge, skills, and competence and prepare students for performing ultrasound techniques in clinical practice. METHODS: Students graduating from a PA program completed a 23-item questionnaire examining their ultrasound training experiences, their self-assessment of competency, and their demographics. Students also completed a 15-item ultrasound knowledge assessment. RESULTS: Thirty-eight of 39 students (97%) completed the survey. Students received little hands-on ultrasound training, with the most hands-on training being offered during emergency medicine (44.7%), obstetrics and gynecology (42.1%), and inpatient internal medicine (39.5%) rotations. This lack of preparedness was reflected in a mean score of 47.1% (±16.4%) on the ultrasound knowledge assessment. Most students (84.2%) indicated that the ultrasound instruction they received during clinical rotations was insufficient to prepare them for clinical practice, and 84.2% desired a formal ultrasound training program in the PA program curriculum. CONCLUSIONS: Existing PA program curricula are insufficient for developing critical skills related to ultrasonography.

Can We Get Faculty Interviewers on the Same Page? An Examination of a Structured Interview Course for Surgeons.

INTRODUCTION: Guidance on how to train faculty to conduct structured interviews and implement them into current screening processes is lacking. The goal of this study is to describe a structured interview training program designed specifically for surgeons and examine its effectiveness. METHODS: Faculty involved in advanced surgical fellowship interviews completed a 20-item knowledge assessment and video-based applicant interview ratings before taking a half-day course on conducting structured interviews. The course consisted of evidence-based strategies and methods for conducting structured interviews, asking questions, and rating applicants in a highly interactive format. After the course, faculty again completed the knowledge assessment and provided ratings for 3 video-based applicant interviews. RESULTS: All faculty members (N = 5) responsible for selecting fellows in minimally invasive and bariatric surgery completed the training. Faculty had an average of 15.8 ± 9.12 years in practice. Average performance on the precourse knowledge assessment was 35% ± 6.12% and the group was unable to achieve acceptable agreement for applicant interview scores for any of the competencies assessed. After the course, faculty demonstrated significant improvements (p < 0.01) on the knowledge assessment, more than doubling their scores on the pretest with average scores of 80% ± 9.35%. Faculty also improved their interrater agreement of applicant competency, with 80% of the applicant interview ratings within 2 points of each other. CONCLUSIONS: Implementation of a half-day course designed to teach principles and skills around structured interviewing and assessment demonstrated significant improvements in both interviewing knowledge and interrater agreement. These findings support the time and resources required to develop and implement a structured interview training program for surgeons for the postgraduate admissions process.

Passing the fundamentals of endoscopic surgery (FES) exam: linking specialty choice and attitudes about endoscopic surgery to success.

INTRODUCTION: Previous work has shown that up to 30% of graduating surgery residents fail the fundamentals of endoscopic surgery (FES) exam. This study investigated the extent to which FES pass rates differ in a specific sample of individuals who have chosen a career in GI surgery and to examine the relationships between FES performance and confidence in performing flexible endoscopy. METHODS: Fellows attending the 2016 SAGES Flexible Endoscopy Course were invited to complete the FES manual skills examination. Participants also provided survey responses examining demographics, fellowship type, endoscopy curricula in residency, previous endoscopic case volume, confidence in performing endoscopy, and future practice plans. RESULTS: Twenty-nine (age: 32.24 ± 3.24; 72% men) fellows completed the FES skills examination. Reported fellowships were MIS/Bariatric (41.4%), MIS (24.1%), bariatric (13.8%), flexible endoscopy (6.9%), Advanced GI (6.9%), and MIS/bariatric/flexible endoscopy (6.9%). Almost half (41.4%) had previously participated in a simulation curricula, with 20.7% completing a didactic endoscopy curriculum. Fellows reported performing an average of 110 ± 109.48 EGDs and 77.44 ± 58.80 colonoscopies. The majority (96.4%) indicated that they will perform endoscopy at least occasionally in practice. Overall pass rate was 60%. Previous endoscopy experience did not correlate with overall FES examination scores. However, confidence performing EGDs (r = 0.57, p < 0.01), colonoscopies (r = 0.45, p < 0.05), polypectomy (r = 0.52, p < 0.01), and PEGs (r = 0.46, p < 0.05) did. CONCLUSIONS: These data support existing research suggesting that current flexible endoscopy training in residency may be insufficient for trainees to pass the FES examination, and that failure rates hold true even for this select group of trainees who have chosen a profession in GI surgery and intend to use endoscopy in practice.

What Do Faculty in Health Professions Need to be Competent Educators? Results from a School-Wide Needs Assessment.

BACKGROUND: Teaching may be the one responsibility most distinctly identified with being a faculty member, but may be the one for which faculty is least prepared. We performed a needs assessment to understand the educational needs of health professions faculty. METHODS: Faculty completed a survey examining the extent to which they valued and felt competent in 36 education-related skills within six categories, including instructional methods, curriculum development, assessment, instructional design, learners and learning, and diversity and inclusion. Participants rated each item using a 1 (not at all valuable/competent) to 5 (extremely valuable/competent) scale. RESULTS: Ninety-five percent of faculty (n=19/20) responded. Value means per topic included diversity and inclusion (4.61±0.65), instructional methods (4.60±0.28), learners and learning (4.60±0.34), assessment (4.54±0.31), instructional design (4.27±0.50), and curriculum development (4.21±0.28). Self-rated competence was highest for diversity and inclusion (4.18±0.76), followed by learners and learning (3.76±0.75), instructional methods (3.73±0.65), assessment (3.22±0.95), curriculum development (3.07±0.88), and instructional design (2.71±0.77). All value/competency comparisons found the value of these skills skills to be significantly greater than competence in that category (p<0.001 for each), with largest discrepancies in instructional design and assessment. CONCLUSIONS: Opportunities exist to improve alignment between valued educator skills and their perceived competency levels in these skills.

Using Simulation to Improve Systems-Based Practices.

BACKGROUND: Ensuring the safe, effective management of patients requires efficient processes of care within a smoothly operating system in which highly reliable teams of talented, skilled health care providers are able to use the vast array of high-technology resources and intensive care techniques available. Simulation can play a unique role in exploring and improving the complex perioperative system by proactively identifying latent safety threats and mitigating their damage to ensure that all those who work in this critical health care environment can provide optimal levels of patient care. METHODS: A panel of five experts from a wide range of institutions was brought together to discuss the added value of simulation-based training for improving systems-based aspects of the perioperative service line. Panelists shared the way in which simulation was demonstrated at their institutions. The themes discussed by each panel member were delineated into four avenues through which simulation-based techniques have been used. RESULTS: Simulation-based techniques are being used in (1) testing new clinical workspaces and facilities before they open to identify potential latent conditions; (2) practicing how to identify the deteriorating patient and escalate care in an effective manner; (3) performing prospective root cause analyses to address system weaknesses leading to sentinel events; and (4) evaluating the efficiency and effectiveness of the electronic health record in the perioperative setting. CONCLUSION: This focused review of simulation-based interventions to test and improve components of the perioperative microsystem, which includes literature that has emerged since the panel's presentation, highlights the broad-based utility of simulation-based technologies in health care.

Is current surgery resident and GI fellow training adequate to pass FES?

BACKGROUND: The purpose of this study was to assess the adequacy of current surgical residency and gastroenterology (GI) fellowship flexible endoscopy training as measured by performance on the FES examination. METHODS: Fifth-year general surgery residents and GI fellows across six institutions were invited to participate. All general surgery residents had met ACGME/ABS case volume requirements as well as additional institution-specific requirements for endoscopy. All participants completed FES testing at the end of their respective academic year. Procedure volumes were obtained from ACGME case logs. Curricular components for each specialty and institution were recorded. RESULTS: Forty-eight (28 surgery and 20 GI) trainees completed the examination. Average case numbers for residents were 76 ± 26 colonoscopies and 45 ± 12 EGDs. Among GI fellows, PGY4 s (N = 10) reported 99 ± 64 colonoscopies and 147 ± 79 EGDs. PGY5 s (N = 3) reported 462 ± 307 colonoscopies and 411 ± 260 EGDs. PGY6 GI fellows (N = 7) reported 515 ± 111 colonoscopies and 418 ± 146 EGDs. The overall pass rate for all participants was 75 %, with 68 % of residents and 85 % of fellows passing both the cognitive and skills components. For surgery residents, pass rates were 75 % for manual skills and 85.7 % for cognitive. On the skills examination, Task 2 (loop reduction) was associated with the lowest performance. Skills scores correlated with both colonoscopy (r = 0.46, p < 0.001) and EGD experience (r = 0.46, p < 0.001). Receiver operating characteristics curves were examined among the resident cohort. The minimum number of total cases associated with passing the FES skills component was 103. Significant variability existed in curricular components across institutions. DISCUSSION: These data suggest that current flexible endoscopy training may not be sufficient for all trainees to pass the examination. Implementing additional components of the FEC may prove beneficial in achieving more uniform pass rates on the FES examination.

Changing attitudes and improving skills: demonstrating the value of the SAGES flexible endoscopy course for fellows.

BACKGROUND: The purpose of this study was to examine the effectiveness of the SAGES flexible endoscopy course in improving fellows' attitudes, confidence, and skills related to implementing endoscopy in practice. METHODS: Fellows participated in a 2-day course consisting of case presentations, expert panels, and hands-on laboratory training. Before and after the course, fellows completed a questionnaire assessing demographics, experiences in residency, practice plans, plans to implement flexible endoscopy in practice, and level of confidence performing 15 endoscopic procedures. Half of the fellows were randomly assigned to complete pre- and post-skills testing using a previously validated endoscopic targeting model. RESULTS: Fifty-four fellows (90 %; age 33.5 ± 2.8; 58 % male) completed the pre- and post-questionnaire. All MIS fellowship types were represented. Almost half (48 %) reported none or very little flexible endoscopy in their current fellowship. The average prior case volume among those completing an ACGME-approved residency (42/54) was 76 upper and 75 lower endoscopies with one-third reporting no experience in therapeutic EGD (33 %) or polypectomy (31 %). Intentions to implement flexible endoscopy in practice significantly improved after the course overall (3.72 ± .85-3.92 ± .69, p < 0.05; 1 = never; 5 = very frequently). Prior to the course, 39 % of fellows reported plans to use endoscopy in practice "occasionally" or "rarely." After, this decreased to 28 with 72 % planning to implement "frequently" or "very frequently." Mean levels of confidence performing all 15 endoscopic tasks improved significantly after the course. Skills performance for the 27 fellows improved significantly as well; participants decreased their time to perform the targeting task by 40 % (222.3 ± 119.8-133.0 ± 70.1 s; p < 0.001) and decreased errors by 49 % (2.9 ± 1.7-1.5 ± 1.5; p < 0.001). CONCLUSIONS: These results indicate that the SAGES flexible endoscopy course increases fellow confidence to implement endoscopic techniques, expands the ways in which they plan to include endoscopy in practice, and enhances their endoscopic skills.

Frame-of-reference training for simulation-based intraoperative communication assessment.

BACKGROUND: The purpose of this study was to examine the impact of frame-of-reference (FOR) training on assessments of intraoperative communication skills and identify areas of need to inform curricular efforts. METHODS: Simulation instructors (M.D., Ph.D., Research Fellow, Simulation Technician) underwent a 2-hour FOR training session with the operating room communication instrument. They then independently rated communication skills of 19 PGY1s who participated in a team-based simulation. Residents completed self-assessments via video review of the scenario. Intraclass correlation coefficients were used to examine inter-rater reliability. Relationships between trained raters and resident scores were assessed with Pearson correlation coefficients and paired sample t tests. RESULTS: Inter-reliability after FOR training was .91. The correlation between trained rater scores and resident evaluations was nonsignificant. Residents significantly underestimated their intraoperative communication skills (P < .05). Use of names, closed loop communication, and sharing information with team members demonstrated consistently low ratings among all residents. CONCLUSIONS: These findings reveal that a number of individuals can be trained to reliably rate resident intraoperative communication performance and that residents tend to under-rate their communication skills.

Characteristics and Core Curricular Elements of Medical Simulation Fellowships in North America.

Background In the past few years, there has been rapid growth in the number of simulation fellowships for physicians in the United States and Canada, with the objective of producing faculty with expertise and leadership training in medical simulation. Relatively little is known about the collective content and structure of these new fellowship opportunities. Objective We sought to identify a common set of core curricular elements among existing simulation fellowships and to obtain demographic background information on participants and leadership. Methods We designed a web-based survey and circulated it to simulation fellowship directors in the United States and Canada. The questions explored aspects of the fellowship curriculum. A grounded theory approach was used to qualitatively analyze fellowship goals and objectives. Results Of the 29 program directors surveyed, 23 responded (79%). The most commonly listed goals and objectives were to increase skills in simulation curriculum development, simulation operations and training environment setup, research, educational theory, administration, and debriefing. The majority of the responding fellowship directors (17 of 22, 77%) indicated that a set of consensus national guidelines would benefit their fellowship program. Conclusions Simulation fellowships are experiencing a period of rapid growth. Development of a common set of program guidelines is a widely shared objective among fellowship directors.

What do residents need to be competent laparoscopic and endoscopic surgeons?

BACKGROUND: Despite numerous efforts to ensure that surgery residents are adequately trained in the areas of laparoscopy and flexible endoscopy, there remain significant concerns that graduates are not comfortable performing many of these procedures. METHODS: Online surveys were sent to surgery residents (98 items, PGY1-5 Categorical) and faculty (78 items, general surgery, and gastrointestinal specialties) at seven institutions. De-identified data were analyzed under an IRB-approved protocol. RESULTS: Ninety-five faculty and 121 residents responded, with response rates of 65 and 52 %, respectively. Seventy-three percent of faculty indicated that competency of their graduating residents were dramatically or slightly worse than previous graduates. Only 29 % of graduating residents felt very comfortable performing advanced laparoscopic (AL) cases and 5 % performing therapeutic endoscopy (TE) cases immediately after graduation. Over half of interns expressed a need for fellowship to feel comfortable performing AL and TE procedures, and this need did not decrease as residents neared graduation. For these procedures, residents receive only "little to some" autonomy, as reported by both faculty and PGY5s. Residents reported that current curricula for laparoscopy and endoscopy consist primarily of clinical experience. Both residents and faculty, though, reported considerable value in other training modalities, including simulations, live animal laboratories, cadavers, and additional didactics. CONCLUSIONS: These data indicate that both residents and faculty perceive significant competency gaps for both laparoscopy and flexible endoscopy, with the most notable shortcomings for advanced and therapeutic cases, respectively. Improvement in resident training methods in these areas is warranted.

Developing a comprehensive resident education evaluation system in the era of milestone assessment.

OBJECTIVES: In an effort to move training programs toward competency-based education, the Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System (NAS), which organizes specific milestones regarding resident skills, knowledge, and abilities along a continuum. In order to foster innovation and creativity, the ACGME has provided programs with minimal guidelines regarding the optimal way to approach these milestones. METHODS: The education team at UT Southwestern embraced the milestones and developed a process in which performance assessment methods were critically evaluated, mapped onto an extrapolated performance list corresponding to the areas required by the ACGME milestones, and filled gaps in the previous system by modifying evaluation tools and creating new program components. RESULTS: Although the authors are early in the evolution of applying the new milestones system, this approach has thus far allowed them to comprehensively evaluate the residents and the program in an efficient and effective fashion, with notable improvements compared to the prior approach. CONCLUSIONS: The authors hope that these experiences can inform others embarking upon similar journeys with the milestones.

The association for surgical education CESERT grant program: the first 15 years.

BACKGROUND: The Association for Surgical Education established the Center for Excellence in Surgical Education Research and Training (CESERT) program in 1999 to support innovative research and education projects and programs that will advance surgical education. We sought to examine scholarly outcomes of the projects during the first 15 years of the program. METHODS: E-mail surveys were sent to the 24 researchers who were awarded CESERT grants. Data collection focused on recipient professional experiences, publications, and presentations stemming from CESERT-funded research and impact on career development. For grant recipients who were unable to complete the survey, we obtained publication information on studies authored by the grant recipients that described the same grant-funded topic, described similar methods, and fell within the study timeline. RESULTS: Complete survey data were obtained from 18 of the 24 grant recipients. Grants were most commonly awarded to General Surgeons (40%) and Education and Psychology PhDs (24%). Overall, 23 of the 25 projects had reached completion at the time of contact, and from these, 70 articles were published or in press. Abstract presentations were more common, with respondents documenting 84 projects locally, nationally, and internationally. Grant awards ranged from $8,122 to $97,000, with an average of $39,026. In total, the Association for Surgical Education Foundation distributed $988,000 in grant funding from 1999 to 2013. Respondents reported that CESERT funding was critical to their career as it legitimized their pursuit of educational research, helped them establish multidisciplinary and multi-institutional collaborations, provided greater visibility for their research, and helped them develop an understanding of educational principles and grant writing skills. CONCLUSIONS: Overall, the CESERT program has produced remarkable results. The careful review process and monitoring protocol have ensured that high-potential studies are funded and successful. These data support continued efforts to garner resources such that CESERT grants may be awarded to fund high-quality, high-impact projects.