Going Above and Beyond With SJTs: Impact of Applicant Characteristics on Open Response SJT Participation.

BACKGROUND: Situational judgment tests (SJT) have gained popularity as a standardized assessment of nontechnical competencies for applicants to medical school and residency. SJT formats range from rating the effectiveness of potential response options to solely open response. We investigated differences in test-taking patterns between responders and nonresponders to optional open response SJT questions during the application process. METHODS: This was a prospective multi-institutional study of general surgery applicants to seven residency programs. Applicants completed a 32-item SJT designed to measure ten core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Each SJT item included an optional, nonscored, open response space for applicants to provide a behavioral response if they desired. Trends in applicant gender, race, ethnicity, medical school ranking, and USMLE scores were examined between the responder versus nonresponder group. RESULTS: In total, 1491 general surgery applicants were invited to complete the surgery-specific SJT. Of these, 1454 (97.5%) candidates completed the assessment and 1177 (78.9%) provided additional responses to at least one of the 32 SJT scenario sets. There were no differences in overall SJT performance, USMLE scores (Step 1: 235, SD 14, Step 2: 250, SD 11), race and/or ethnicity between the responder and nonresponder groups. Responders were more likely to be from a top 25 medical school (p < 0.05) compared to the nonresponder group. Among applicants who completed any open response questions, women completed a significantly higher number of questions compared to men (7.21 vs 6.07, p = 0.003). The number of open responses provided correlated with higher scores on SJT items measuring dependability (r = 0.07, p = 0.007). CONCLUSIONS: SJT design and format has the potential to impact test-taker response patterns. SJT developers and adopters should ensure test format and design have no unintended consequences prior to implementation.

Predicting Surgical Resident Performance With Situational Judgment Tests.

PURPOSE: Situational judgment tests (SJTs) have been proposed as an efficient, effective, and equitable approach to residency program applicant selection. This study examined how SJTs can predict milestone performance during early residency. METHOD: General surgery residency program applicants during 3 selection cycles (2018-2019, 2019-2020, 2020-2021) completed SJTs. Accreditation Council for Graduate Medical Education milestone performance data from selected applicants were collected in March and April 2019, 2020, and 2021 and from residents in March 2020, August 2020, March 2021, September 2021, and March 2022. Descriptive statistics and correlations were computed and analysis of variance tests performed to examine differences among 4 SJT performance groups: green, top 10% to 25%; yellow, next 25% to 50%; red, bottom 50%; and unknown, did not complete the SJT. RESULTS: Data were collected for 70 residents from 7 surgery residency programs. Differences were found for patient care (F3,189 = 3.19, P = .03), medical knowledge (F3,176 = 3.22, P = .02), practice-based learning and improvement (F3,189 = 3.18, P = .04), professionalism (F3,189 = 3.82, P = .01), interpersonal and communication skills (F3,190 = 3.35, P = .02), and overall milestone score (F3,189 = 3.44, P = .02). The green group performed better on patient care, medical knowledge, practice-based learning and improvement, professionalism, and overall milestone score. The yellow group performed better than the red group on professionalism and overall milestone score, better than the green group on interpersonal and communication skills, and better than the unknown group on all but practice-based learning and improvement. The red group outperformed the unknown group on all but professionalism and outperformed the green group on medical knowledge. CONCLUSIONS: Situational judgment tests demonstrate promise for assessing important noncognitive attributes in residency applicants and align with national efforts to review candidates more holistically and minimize potential biases.

Mapping the Terrain of Faculty Development for Simulation: A Scoping Review.

ABSTRACT: Understanding what interventions and approaches are currently being used to improve the knowledge, skills, and effectiveness of instructors in simulation-based education is an integral step for carving out the future of simulation. The current study is a scoping review on the topic, to uncover what is known about faculty development for simulation-based education.We screened 3259 abstracts and included 35 studies in this scoping review. Our findings reveal a clear image that the landscape of faculty development in simulation is widely diverse, revealing an array of foundations, terrains, and peaks even within the same zone of focus. As the field of faculty development in simulation continues to mature, we would hope that greater continuity and cohesiveness across the literature would continue to grow as well. Recommendations provided here may help provide the pathway toward that aim.

Practical Guide to Surgical Simulation Research.

This Guide to Statistics and Methods provides an overview of simulation-based education and its impact on surgeon performance.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test.

BACKGROUND: Situational judgment tests (SJT) are hypothetical but realistic scenario-based assessments that allow residency programs to measure judgment and decision-making among future trainees. A surgery-specific SJT was created to identify highly valued competencies among residency applicants. We aim to demonstrate a stepwise process for validation of this assessment for applicant screening through exploration of two often-overlooked sources of validity evidence - relations with other variables and consequences. METHODS: This was a prospective multi-institutional study involving 7 general surgery residency programs. All applicants completed the SurgSJT, a 32-item test aimed to measure 10 core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Performance on the SJT was compared to application data, including race, ethnicity, gender, medical school, and USMLE scores. Medical school rankings were determined based on the 2022 U.S. News & World Report rankings. RESULTS: In total, 1491 applicants across seven residency programs were invited to complete the SJT. Of these, 1454 (97.5%) candidates completed the assessment. Applicants were predominantly White (57.5%), Asian (21.6%), Hispanic (9.7%), Black (7.3%), and 52% female. A total of 208 medical schools were represented, majority were allopathic (87.1%) and located in United States (98.7%). Less than a quarter of applicants (22.8%; N=337) were from a top 25 school based on U.S. News & World Report rankings for primary care, surgery, or research. Average USMLE Step 1 score was 235 (SD 37) and Step 2 score was 250 (SD 29). Sex, race, ethnicity, and medical school ranking did not significantly impact performance on the SJT. There was no relationship between SJT score and USMLE scores and medical school rankings. CONCLUSIONS: We demonstrate the process of validity testing and importance of two specific sources of evidence-consequences and relations with other variables, in implementing future educational assessments.

Getting on the Same Page: The Impact of Interviewer Education and Structured Interviews on Interrater Agreement in Residency Interviews.

INTRODUCTION: We explored the impact of implementing structured interviews and associated interviewer education on interrater agreement within a large academic residency program. METHODS: Faculty and senior resident interviewers from a large academic residency program participated in a 3-hour structured interview course. Before and after the course, participants completed a 15-item assessment pertaining to the characteristics, logistics, and guidelines associated with structured interviews. Along with interviewer training, interview day logistics also changed from an unstructured format (no specific questions, one overall 1-9 rating scale) to a structured interview format, including incorporation of behavioral-based competency questions that would be asked of every applicant and behavioral anchored rating scales (1-10; 10 = highest). Interrater agreement was assessed via intraclass correlation coefficients (ICC1) for the 2 years before and 2 years after incorporation of the structured interview format. RESULTS: A total of 45 faculty and resident interviewers participated in the course in 2018. Participant knowledge significantly increased from an average of 36% to 79% after the course (p < 0.01). Prior to the intervention, overall interrater agreement was "poor" to "fair," with an ICC1 of 0.51 in 2016 and 0.49 in 2017. After the structured interview intervention, overall agreement increased to the "good" level with an ICC1 of 0.71 in 2018 and 0.66 in 2019. The proportion of applicants who received interview scores with at least 2 ratings more than 2 points apart significantly decreased from 59% to 47% after the intervention (p < 0.01). CONCLUSIONS: Incorporating an interviewer educational session and a structured interview format into residency selection can help increase agreement in ratings between interviewers. However, these data suggest that ongoing refresher trainings may be needed to maintain acceptable levels of interrater agreement.

Is Beauty in the Eye of the Beholder? Differences in Residency Training Program Preferences Among Women and Minorities.

INTRODUCTION: Training programs are now more than ever seeking ways to promote recruitment and retention of a diverse resident workforce. The goal of this study was to examine how gender and ethnic identities affect applicant attraction to surgery training programs. METHODS: Applicants to general surgery residency in 2018 to 2019 completed a 31-item assessment measuring preferences for training program characteristics and attributes. Differences in preferences across candidate gender and ethnicity were investigated. Factor analyses and analysis of variance (ANOVA) were used to explore these differences. RESULTS: 1491 unique applicants to 7 residency programs completed the assessment, representing 67% of all applicants to general surgery during the 2018 to 2019 season. Women preferred training programs that had high levels of social support (p < 0.001), were less traditional (p < 0.001), and with less turbulence (p < 0.05). Non-white candidates reported greater preference for programs with higher levels of established academics (p < 0.001), clinical experiences (p < 0.001), social support (p < 0.05), traditionalism (p < 0.001), flexibility (p < 0.001), and innovation (p < 0.001). CONCLUSIONS: Organizational efforts to attract and retain a diverse workforce may benefit from considering the aspects of work that align with female and underrepresented minority preferences.

Measuring the fire in their hearts: assessing passion for the profession among students pursuing surgical careers.

Background: Leaders in surgery have posited that passion for the surgery profession is diminishing among entering trainees, and that its scarcity is related to the high levels of attrition observed in general surgery training. This study explores trends in passion for the profession among applicants to general surgery training. Methods: Applicants to a large midwestern academic general surgery program were invited to complete a voluntary, anonymous 12-item Passion for Surgery Index (PSI) as part of their supplementary application package during the 2020-2021 and 2021-2022 residency selection seasons. The PSI is adapted from a generic work-related passion index and is based on the dualistic model of passion, organizing scores into harmonious passion and consuming passion. Applicants completed the index on a stand-alone website which automatically generated results pertaining to overall passion, harmonious passion, and consuming passion for the surgery profession. Applicants were provided with their results and provided feedback. Results: Sixty-one percent (871/1428) of invited applicants completed the PSI. Approximately 67.4% (N = 587) of these applicants reported an overall high level of passion for surgery, while 31.1% (N = 271) reported a moderate level and the remaining 1.5% (N = 13) reported a low level. When comparing the two different types of passion, the vast majority of applicants (92.8%; N = 808) reported a high level of harmonious passion and only 7.1% (N = 62) reported a moderate level of harmonious passion. The results for consuming passion were much more varied, with 36.9% (N = 321) reporting a high level, 47.5% (N = 414) reporting a moderate level, and 15.6% (N = 136) reporting a low level of consuming passion for the profession. Discussion: These results suggest that there is substantial variation in passion for the profession among those pursuing a career in surgery. While the majority of applicants reported a high level of harmonious passion for surgery, less than half of applicants reported a high level of consuming passion for surgery. This variability in consuming passion among entering trainees is concerning, as individuals with low or only moderate passion for the profession may not have the motivation or drive to persist in demanding training environments.

The Relationship Between Surgeon Faculty Emotional Intelligence and Medical Student Evaluations.

OBJECTIVE: We sought to measure the emotional intelligence (EI) of surgical faculty and the relationship between faculty EI and medical student (MS) evaluations of faculty. DESIGN: Faculty completed the Emotional Intelligence Appraisal. Aggregate, anonymous MS evaluations were collected from the Program Director's office. Parametric and nonparametric tests were used for analysis. SETTING: This study was first performed in a single surgical division at 1 center which informed an expanded study including the entire General Surgery Department at a single academic institution. PARTICIPANTS: A pilot study was conducted in 1 surgical division which was then expanded to all clinical faculty in the Department of Surgery. All clinical faculty in the Department of Surgery were eligible for enrollment. RESULTS: Pilot study faculty EI scores were positively correlated with MS evaluations (r = 0.92, p < 0.001). The follow-up study enrolled 41 surgeons with a median age of 48 (inter-quartile range 12). The sample was mostly white (70.7%). Mean EI for the group was 76 (standard deviation ± 7.8). Total faculty EI scores were not significantly correlated with MS evaluations (r = 0.30, p = 0.06). CONCLUSIONS: MS evaluations of surgeon faculty were not related to EI in the larger sample. However, EI did correlate to MS evaluations in 2 surgical specialties. Further exploration into the utility of EI training in surgical departments should be conducted to determine the true value of such endeavors.

Faculty Emotional Intelligence Matters for Resident Education.

OBJECTIVE: This study aimed to determine the emotional intelligence (EI) of surgical faculty and evaluate its relationship with resident evaluations of faculty behaviors. DESIGN: This study retrospectively collected faculty EI scores as well as general surgery resident evaluations of faculty. Parametric and nonparametric tests were used for statistical analysis. SETTING: The study was conducted at the University of Texas Southwestern in the Department of Surgery in Dallas, Texas. This is an academic, tertiary care center. PARTICIPANTS: Surgical faculty members at a single institution in 2018 completed the Emotional Intelligence Appraisal, a 28-item, electronic assessment with possible scores ranging from zero to 100. Aggregate, anonymous resident evaluations of faculty members were collected from the program director's office. Faculty with fewer than 8 resident evaluations were excluded. RESULTS: In total, 59 faculty members participated (89%). The sample was mostly white (69.2%), male (63.5%), with an average of 47 ± 10 years of age, 12.2 ± 10 years in practice, and 44 ± 24 evaluations per faculty member. The group's mean EI score was 76 ± 7.7. Faculty EI scores were found to have a moderate, positive correlation with resident rotational evaluations of faculty (r(51) = 0.52, p < 0.001). Faculty EI scores did not significantly correlate with resident evaluations of faculty intraoperative behaviors. CONCLUSIONS: The results of this study suggest that a majority of our faculty are competent with regard to EI. Furthermore, faculty EI is an important factor in the clinical learning environment and correlates with resident rotational evaluations of teaching behaviors.

Getting better all the time? Facilitating accurate team self-assessments through simulation.

Introduction: Our study explores the extent to which teams are accurate assessors of their own performance and teamwork, and how simulation can help this critical skill develop over time. Methods: Surgery residents in teams of three completed five daily simulations. After each scenario, each team reviewed their performance and jointly completed a scenario-specific team performance evaluation and a 17-item Communication and Teamwork Skills tool. Videos were rated to obtain discrepancy values. Paired-samples t-tests and mean comparisons were used to examine changes in team self-assessment accuracy and comparisons between high-performing and low-performing teams. Results: Resident (n=30) teams rated team performance higher than faculty across the first 3 days (p<0.01), but provided similar ratings thereafter. Agreement of team performance from day 1 to 5 significantly improved (p<0.001). Teams rated their teamwork higher than faculty across all days (p<0.01). Top performing teams provided more accurate self-assessments for both teamwork (average discrepancy 8% vs 39%) and team performance (average discrepancy 12% vs 23%). Conclusion: Teams that continue to work together over time may become more accurate judges of their own performance, but do not become more accurate assessors of teamwork competencies.

Medical and Physician Assistant Student Competence in Basic Life Support: Opportunities to Improve Cardiopulmonary Resuscitation Training.

INTRODUCTION: Medical and physician assistant (PA) students are often required to have Basic Life Support (BLS) education prior to engaging in patient care. Given the potential role of students in resuscitations, it is imperative to ensure that current BLS training prepares students to provide effective cardiopulmonary resuscitation (CPR). The objective of this study was to assess whether current BLS training produces student providers who can deliver BLS in an American Heart Association (AHA) guideline-adherent manner. METHODS: Students at a US medical school were recruited by convenience sampling. BLS performance immediately following a standard AHA BLS training course was evaluated during a two-minute CPR cycle using manikins. We also collected information on demographics, previous BLS training attendance, perceived comfort in providing CPR, and prior experiences in healthcare and providing or observing CPR. RESULTS: Among 80 participants, we found that compression rate, depth, and inter-compression recoil were AHA guideline-adherent for 90.0%, 68.8%, and 79.3% of total compression time, respectively. Mean hands-off time was also within AHA guidelines. Mean number of unsuccessful ventilations per cycle was 2.2. Additionally, 44.3% of ventilations delivered were of adequate tidal volume, 12.2% were excessive, and 41.0% were inadequate. Past BLS course attendance, prior healthcare certification, and previous provision of real-life CPR were associated with improved performance. CONCLUSION: Following BLS training, medical and PA students met a majority of AHA compressions guidelines, but not ventilations guidelines, for over 70% of CPR cycles. Maintaining compression depth and providing appropriate ventilation volumes represent areas of improvement. Conducting regular practice and involving students in real-life CPR may improve performance.

Video-based assessment for laparoscopic fundoplication: initial development of a robust tool for operative performance assessment.

INTRODUCTION: While better technical performance correlates with improved outcomes, there is a lack of procedure-specific tools to perform video-based assessment (VBA). SAGES is developing a series of VBA tools with enough validity evidence to allow reliable measurement of surgeon competence. A task force was established to develop a VBA tool for laparoscopic fundoplication using an evidence-based process that can be replicated for additional procedures. The first step in this process was to seek content validity evidence. METHODS: Forty-two subject matter experts (SME) in laparoscopic fundoplication were interviewed to obtain consensus on procedural steps, identify potential variations in technique, and to generate an inventory of required skills and common errors. The results of these interviews were used to inform creation of a task inventory questionnaire (TIQ) that was delivered to a larger SME group (n = 188) to quantify the criticality and difficulty of the procedural steps, the impact of potential errors associated with each step, the technical skills required to complete the procedure, and the likelihood that future techniques or technologies may change the presence or importance of any of these factors. Results of the TIQ were used to generate a list of steps, skills, and errors with strong validity evidence. RESULTS: Initial SMEs interviewed included fellowship program directors (45%), recent fellows (24%), international surgeons (19%), and highly experienced super SMEs with quality outcomes data (12%). Qualitative analysis of interview data identified 6 main procedural steps (visualization, hiatal dissection, fundus mobilization, esophageal mobilization, hiatal repair, and wrap creation) each with 2-5 sub steps. Additionally, the TIQ identified 5-10 potential errors for each step and 11 key technical skills required to perform the procedure. Based on the TIQ, the mean criticality and difficulty scores for the 11/21 sub steps included in the final scoring rubric is 4.66/5 (5 = absolutely essential for patient outcomes) and 3.53/5 (5 = difficulty level requires significant experience and use of alternative strategies to accomplish consistently), respectively. The mean criticality and frequency scores for the 9/11 technical skills included is 4.51/5 and 4.51/5 (5 = constantly used ≥ 80% of the time), respectively. The mean impact score of the 42/47 errors incorporated into the final rubric is 3.85/5 (5 = significant error that is unrecoverable, or even if recovered, likely to have a negative impact on patient outcome). CONCLUSIONS: A rigorous, multi-method process has documented the content validity evidence for the SAGES video-based assessment tool for laparoscopic fundoplication. Work is ongoing to pilot the assessment tool on recorded fundoplication procedures to establish reliability and further validity evidence.

Can Better Selection Tools Help Us Achieve Our Diversity Goals in Postgraduate Medical Education? Comparing Use of USMLE Step 1 Scores and Situational Judgment Tests at 7 Surgical Residencies.

PURPOSE: Use of the United States Medical Licensing Examination (USMLE) for residency selection has been criticized for its inability to predict clinical performance and potential bias against underrepresented minorities (URMs). This study explored the impact of altering traditional USMLE cutoffs and adopting more evidence-based applicant screening tools on inclusion of URMs in the surgical residency selection process. METHOD: Multimethod job analyses were conducted at 7 U.S. general surgical residency programs during the 2018-2019 application cycle to gather validity evidence for developing selection assessments. Unique situational judgment tests (SJTs) and scoring algorithms were created to assess applicant competencies and fit. Programs lowered their traditional USMLE Step 1 cutoffs and invited candidates to take their unique SJT. URM status (woman, racial/ethnic minority) of candidates who would have been considered for interview using traditional USMLE Step 1 cutoffs was compared with the candidate pool considered based on SJT performance. RESULTS: A total of 2,742 general surgery applicants were invited to take an online SJT by at least 1 of the 7 programs. Approximately 35% of applicants who were invited to take the SJT would not have met traditional USMLE Step 1 cutoffs. Comparison of USMLE-driven versus SJT-driven assessment results demonstrated statistically different percentages of URMs recommended, and including the SJT allowed an average of 8% more URMs offered an interview invitation (P < .01). CONCLUSIONS: Reliance on USMLE Step 1 as a primary screening tool precludes URMs from being considered for residency positions at higher rate than non-URMs. Developing screening tools to measure a wider array of candidate competencies can help create a more equitable surgical workforce.

Great Expectations? Future Competency Requirements Among Candidates Entering Surgery Training.

INTRODUCTION: We describe a multimethod, multi-institutional approach documenting future competencies required for entry into surgery training. METHODS: Five residency programs involved in a statewide collaborative each provided 12 to 15 subject matter experts (SMEs) to participate. These SMEs participated in a 1-hour semistructured interview with organizational psychologists to discuss program culture and expectations, and rated the importance of 20 core competencies derived from the literature for candidates entering general surgery training within the next 3 to 5 years (1 = importance decreases significantly; 3 = importance stays the same; 5 = importance increases significantly). RESULTS: Seventy-three SMEs across 5 programs were interviewed (77% faculty; 23% resident). All competencies were rated to be more important in the next 3 to 5 years, with team orientation (3.87 ± 0.81), communication (3.82 ± 0.79), team leadership (3.81 ± 0.82), feedback receptivity (3.79 ± 0.76), and professionalism (3.76 ± 0.89) rated most highly. CONCLUSIONS: These findings suggest that the competencies desired and required among future surgery residents are likely to change in the near future.

Priorities in surgical simulation research: What do the experts say?

INTRODUCTION: Progressing the field of surgical simulation research cohesively requires organization. The purpose of this study was to establish contemporary research priorities utilizing Delphi methodology. METHODS: Surgical researchers with expertise in simulation-based research were invited to submit important questions for the field according to an organized framework. Thematic analysis was used to collapse questions into unique questions. In a second round, experts rated the importance of questions. In a third round, experts re-rated the importance of questions. A prioritized agenda was then created. RESULTS: Eighteen experts submitted 80 questions in round one, which were collapsed into 43. In the final round, experts rated the following question as the most important priority: "Does demonstrated competency in the simulation lab translate to clinical competency (OR etc.)?" CONCLUSIONS: Our systematic approach identified multiple important questions to advance the field that may guide researchers and funding agencies alike.

If You Build It, Will They Come? Candidate Completion of Preinterview Screening Assessments.

INTRODUCTION: Residency applicant screening practices are inefficient and costly. However, programs may not consider using alternative assessments for fear that candidates will be "turned off" by additional hurdles in the application process. This study explores the relationship between candidate completion of preinterview screening assessments, applicant examination scores, and program factors. METHODS: Applicants to any of 7 general surgery residency programs were invited to take a preinterview online assessment. Program characteristics and applicant United States Medical Licensing Exams scores were considered in relation to each program's assessment completion rate. RESULTS: A total of 2960 applicants were invited to take the assessment and 97% (2870/2960) completed it. Program completion rates ranged from 95% to 98%. There was no correlation between program characteristics and applicant completion rates. Candidates who did not complete the assessment had significantly lower United States Medical Licensing Exams scores. CONCLUSIONS: Incorporating preinterview assessments to objectively measure candidate competencies and fit will not detract applicants from a general surgery program.