Predicting Surgical Resident Performance With Situational Judgment Tests.

PURPOSE: Situational judgment tests (SJTs) have been proposed as an efficient, effective, and equitable approach to residency program applicant selection. This study examined how SJTs can predict milestone performance during early residency. METHOD: General surgery residency program applicants during 3 selection cycles (2018-2019, 2019-2020, 2020-2021) completed SJTs. Accreditation Council for Graduate Medical Education milestone performance data from selected applicants were collected in March and April 2019, 2020, and 2021 and from residents in March 2020, August 2020, March 2021, September 2021, and March 2022. Descriptive statistics and correlations were computed and analysis of variance tests performed to examine differences among 4 SJT performance groups: green, top 10% to 25%; yellow, next 25% to 50%; red, bottom 50%; and unknown, did not complete the SJT. RESULTS: Data were collected for 70 residents from 7 surgery residency programs. Differences were found for patient care (F3,189 = 3.19, P = .03), medical knowledge (F3,176 = 3.22, P = .02), practice-based learning and improvement (F3,189 = 3.18, P = .04), professionalism (F3,189 = 3.82, P = .01), interpersonal and communication skills (F3,190 = 3.35, P = .02), and overall milestone score (F3,189 = 3.44, P = .02). The green group performed better on patient care, medical knowledge, practice-based learning and improvement, professionalism, and overall milestone score. The yellow group performed better than the red group on professionalism and overall milestone score, better than the green group on interpersonal and communication skills, and better than the unknown group on all but practice-based learning and improvement. The red group outperformed the unknown group on all but professionalism and outperformed the green group on medical knowledge. CONCLUSIONS: Situational judgment tests demonstrate promise for assessing important noncognitive attributes in residency applicants and align with national efforts to review candidates more holistically and minimize potential biases.

Practical Guide to Surgical Simulation Research.

This Guide to Statistics and Methods provides an overview of simulation-based education and its impact on surgeon performance.

Mapping the Terrain of Faculty Development for Simulation: A Scoping Review.

ABSTRACT: Understanding what interventions and approaches are currently being used to improve the knowledge, skills, and effectiveness of instructors in simulation-based education is an integral step for carving out the future of simulation. The current study is a scoping review on the topic, to uncover what is known about faculty development for simulation-based education.We screened 3259 abstracts and included 35 studies in this scoping review. Our findings reveal a clear image that the landscape of faculty development in simulation is widely diverse, revealing an array of foundations, terrains, and peaks even within the same zone of focus. As the field of faculty development in simulation continues to mature, we would hope that greater continuity and cohesiveness across the literature would continue to grow as well. Recommendations provided here may help provide the pathway toward that aim.

Tolerability and Acceptability of Autonomous Immersive Virtual Reality Incorporating Digital Twin Technology for Mass Training in Healthcare.

INTRODUCTION: As part of onboarding and systems testing for a clinical expansion, immersive virtual reality (VR) incorporating digital twin technology was used. While digital twin technology has been leveraged by industry, its use in health care has been limited with no prior application for onboarding or training. The tolerability and acceptability of immersive VR for use by a large population of healthcare staff were unknown. METHODS: A prospective, observational study of an autonomous immersive VR onboarding experience to a new clinical space was conducted from May to September 2021. Participants were healthcare staff from several critical care and acute care units. Primary outcomes were tolerance and acceptability measured by reported adverse effects and degree of immersion. Secondary outcomes were attitudes toward the efficacy of VR compared with standard onboarding experiences. RESULTS: A total of 1522 healthcare staff participated. Rates of adverse effects were low and those with prior VR experience were more likely to report no adverse effects. Odds of reporting immersion were high across all demographic groups, though decreased with increasing age. The preference for VR over low-fidelity methods was high across all demographics; however, preferences were mixed when compared with traditional simulation and real-time clinical care. CONCLUSIONS: Large-scale VR onboarding is feasible, tolerable, and acceptable to a diverse population of healthcare staff when using digital twin technology. This study also represents the largest VR onboarding experience to date and may address preconceived notions that VR-based training in health care is not ready for widespread adoption.

Interprofessional staff perspectives on the adoption of or black box technology and simulations to improve patient safety: a multi-methods survey.

INTRODUCTION: Medical errors still plague healthcare. Operating Room Black Box (ORBB) and ORBB-simulation (ORBBSIM) are innovative emerging technologies which continuously capture as well as categorize intraoperative data, team information, and audio-visual files, in effort to improve objective quality measures. ORBB and ORBBSIM have an opportunity to improve patient safety, yet a paucity of implementation literature exists. Overcoming implementation barriers is critical. This study sought to obtain rich insights while identifying facilitators and barriers to adoption of ORBB and ORBBSIM in alignment with Donabedian's model of health services and healthcare quality. Enrichment themes included translational performance improvement and real-world examples to develop sessions. METHODS: Interprofessional OR staff were invited to complete two surveys assessing staff's perceptions using TeamSTEPPS's validated Teamwork Perceptions Questionnaire (T-TPQ) and open-ended questions. Descriptive statistics were calculated for quantitative variables, and inductive phenomenological content analysis was used for qualitative. RESULTS: Survey 1 captured 71 responses from 334 invited (RR 21%) while survey 2 captured 47 responses from 157 (RR 29.9%). The T-TPQ score was 65.2, with Communication (70.4) the highest construct and Leadership (58.0) the lowest. Quality Improvement (QI), Patient Safety, and Objective Case Review were the most common perceived ORBB benefits. Trends suggested a reciprocal benefit of dual ORBB and ORBBSIM adoption. Trends also suggested that dual implementation can promote Psychological Safety, culture, trust, and technology comfort. The need for an implementation plan built on change management principles and a constructive culture were key findings. CONCLUSIONS: Findings supported ORBB implementation themes from previous literature and deepened our understanding through the exploration of team culture. This blueprint provides a model to help organizations adopt ORBB and ORBBSIM. Outcomes can establish an empirical paradigm for future studies.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test.

BACKGROUND: Situational judgment tests (SJT) are hypothetical but realistic scenario-based assessments that allow residency programs to measure judgment and decision-making among future trainees. A surgery-specific SJT was created to identify highly valued competencies among residency applicants. We aim to demonstrate a stepwise process for validation of this assessment for applicant screening through exploration of two often-overlooked sources of validity evidence - relations with other variables and consequences. METHODS: This was a prospective multi-institutional study involving 7 general surgery residency programs. All applicants completed the SurgSJT, a 32-item test aimed to measure 10 core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Performance on the SJT was compared to application data, including race, ethnicity, gender, medical school, and USMLE scores. Medical school rankings were determined based on the 2022 U.S. News & World Report rankings. RESULTS: In total, 1491 applicants across seven residency programs were invited to complete the SJT. Of these, 1454 (97.5%) candidates completed the assessment. Applicants were predominantly White (57.5%), Asian (21.6%), Hispanic (9.7%), Black (7.3%), and 52% female. A total of 208 medical schools were represented, majority were allopathic (87.1%) and located in United States (98.7%). Less than a quarter of applicants (22.8%; N=337) were from a top 25 school based on U.S. News & World Report rankings for primary care, surgery, or research. Average USMLE Step 1 score was 235 (SD 37) and Step 2 score was 250 (SD 29). Sex, race, ethnicity, and medical school ranking did not significantly impact performance on the SJT. There was no relationship between SJT score and USMLE scores and medical school rankings. CONCLUSIONS: We demonstrate the process of validity testing and importance of two specific sources of evidence-consequences and relations with other variables, in implementing future educational assessments.

Autonomy lessons: Tips and tricks for building self-determination through the lens of a first time principal investigator.

Early career faculty face many challenges, including establishing a career direction, building skills, balancing work and personal life demands, finding mentors, and establishing collegial relationships within their departments. Early career funding has been shown to augment future success in academia; less is known about the impact of early career funding on the social, emotional, and professional identity aspects of work life. One theoretical perspective to examine this issue is self-determination theory, a broad psychological paradigm explaining motivation, well-being, and development. Self-determination theory is predicated on the idea that fulfillment of three basic needs leads to the achievement of integrated well-being. Optimizing autonomy (a sense of choice and control), competence (sense of mastery), and relatedness (a sense of belonging) accompanies greater motivation, productivity, and perceived success. The authors share how applying for and implementing an early career grant affected these three constructs. Early career funding manifested challenges and beneficial outcomes in relationship to each of the three psychological needs and led to important lessons that may be generally applicable to faculty across a wide range of disciplines. The authors offer broad principles as well as specific grant-related strategies for optimizing autonomy, competence, and relatedness while applying for and executing a grant.[Box: see text].

Getting on the Same Page: The Impact of Interviewer Education and Structured Interviews on Interrater Agreement in Residency Interviews.

INTRODUCTION: We explored the impact of implementing structured interviews and associated interviewer education on interrater agreement within a large academic residency program. METHODS: Faculty and senior resident interviewers from a large academic residency program participated in a 3-hour structured interview course. Before and after the course, participants completed a 15-item assessment pertaining to the characteristics, logistics, and guidelines associated with structured interviews. Along with interviewer training, interview day logistics also changed from an unstructured format (no specific questions, one overall 1-9 rating scale) to a structured interview format, including incorporation of behavioral-based competency questions that would be asked of every applicant and behavioral anchored rating scales (1-10; 10 = highest). Interrater agreement was assessed via intraclass correlation coefficients (ICC1) for the 2 years before and 2 years after incorporation of the structured interview format. RESULTS: A total of 45 faculty and resident interviewers participated in the course in 2018. Participant knowledge significantly increased from an average of 36% to 79% after the course (p < 0.01). Prior to the intervention, overall interrater agreement was "poor" to "fair," with an ICC1 of 0.51 in 2016 and 0.49 in 2017. After the structured interview intervention, overall agreement increased to the "good" level with an ICC1 of 0.71 in 2018 and 0.66 in 2019. The proportion of applicants who received interview scores with at least 2 ratings more than 2 points apart significantly decreased from 59% to 47% after the intervention (p < 0.01). CONCLUSIONS: Incorporating an interviewer educational session and a structured interview format into residency selection can help increase agreement in ratings between interviewers. However, these data suggest that ongoing refresher trainings may be needed to maintain acceptable levels of interrater agreement.

Is Beauty in the Eye of the Beholder? Differences in Residency Training Program Preferences Among Women and Minorities.

INTRODUCTION: Training programs are now more than ever seeking ways to promote recruitment and retention of a diverse resident workforce. The goal of this study was to examine how gender and ethnic identities affect applicant attraction to surgery training programs. METHODS: Applicants to general surgery residency in 2018 to 2019 completed a 31-item assessment measuring preferences for training program characteristics and attributes. Differences in preferences across candidate gender and ethnicity were investigated. Factor analyses and analysis of variance (ANOVA) were used to explore these differences. RESULTS: 1491 unique applicants to 7 residency programs completed the assessment, representing 67% of all applicants to general surgery during the 2018 to 2019 season. Women preferred training programs that had high levels of social support (p < 0.001), were less traditional (p < 0.001), and with less turbulence (p < 0.05). Non-white candidates reported greater preference for programs with higher levels of established academics (p < 0.001), clinical experiences (p < 0.001), social support (p < 0.05), traditionalism (p < 0.001), flexibility (p < 0.001), and innovation (p < 0.001). CONCLUSIONS: Organizational efforts to attract and retain a diverse workforce may benefit from considering the aspects of work that align with female and underrepresented minority preferences.

Measuring the fire in their hearts: assessing passion for the profession among students pursuing surgical careers.

Background: Leaders in surgery have posited that passion for the surgery profession is diminishing among entering trainees, and that its scarcity is related to the high levels of attrition observed in general surgery training. This study explores trends in passion for the profession among applicants to general surgery training. Methods: Applicants to a large midwestern academic general surgery program were invited to complete a voluntary, anonymous 12-item Passion for Surgery Index (PSI) as part of their supplementary application package during the 2020-2021 and 2021-2022 residency selection seasons. The PSI is adapted from a generic work-related passion index and is based on the dualistic model of passion, organizing scores into harmonious passion and consuming passion. Applicants completed the index on a stand-alone website which automatically generated results pertaining to overall passion, harmonious passion, and consuming passion for the surgery profession. Applicants were provided with their results and provided feedback. Results: Sixty-one percent (871/1428) of invited applicants completed the PSI. Approximately 67.4% (N = 587) of these applicants reported an overall high level of passion for surgery, while 31.1% (N = 271) reported a moderate level and the remaining 1.5% (N = 13) reported a low level. When comparing the two different types of passion, the vast majority of applicants (92.8%; N = 808) reported a high level of harmonious passion and only 7.1% (N = 62) reported a moderate level of harmonious passion. The results for consuming passion were much more varied, with 36.9% (N = 321) reporting a high level, 47.5% (N = 414) reporting a moderate level, and 15.6% (N = 136) reporting a low level of consuming passion for the profession. Discussion: These results suggest that there is substantial variation in passion for the profession among those pursuing a career in surgery. While the majority of applicants reported a high level of harmonious passion for surgery, less than half of applicants reported a high level of consuming passion for surgery. This variability in consuming passion among entering trainees is concerning, as individuals with low or only moderate passion for the profession may not have the motivation or drive to persist in demanding training environments.

Dissecting a department of surgery: Exploring organizational culture and competency expectations.

INTRODUCTION: In order to recruit high-potential trainees, surgery residency and fellowship programs must first understand what competencies and attributes are required for success in their respective programs. This study performed a systematic analysis to define organizational culture and competency expectations across training programs within one academic surgery department. METHODS: Subject matter experts rated the importance and frequency of 22 competencies and completed a 44-item organizational culture inventory along 1 to 5 Likert-type scales. RESULTS: Importance and frequency attributions of competencies varied significantly among programs (p < .05 by ANOVA), but there was substantial agreement on organizational culture; self-directed (x̄ = 3.8), perfectionist (x̄ = 3.7) and social (x̄ = 3.7) attributes were most representative of the program, while oppositional (x̄ = 1.8), competitive (x̄ = 2.5) and hierarchical (x̄ = 2.7) characteristics were least representative. CONCLUSIONS: Residency and fellowship programs within the same department have shared perceptions of the culture and values of their institution, but seek different competencies among entering trainees.

The Relationship Between Surgeon Faculty Emotional Intelligence and Medical Student Evaluations.

OBJECTIVE: We sought to measure the emotional intelligence (EI) of surgical faculty and the relationship between faculty EI and medical student (MS) evaluations of faculty. DESIGN: Faculty completed the Emotional Intelligence Appraisal. Aggregate, anonymous MS evaluations were collected from the Program Director's office. Parametric and nonparametric tests were used for analysis. SETTING: This study was first performed in a single surgical division at 1 center which informed an expanded study including the entire General Surgery Department at a single academic institution. PARTICIPANTS: A pilot study was conducted in 1 surgical division which was then expanded to all clinical faculty in the Department of Surgery. All clinical faculty in the Department of Surgery were eligible for enrollment. RESULTS: Pilot study faculty EI scores were positively correlated with MS evaluations (r = 0.92, p < 0.001). The follow-up study enrolled 41 surgeons with a median age of 48 (inter-quartile range 12). The sample was mostly white (70.7%). Mean EI for the group was 76 (standard deviation ± 7.8). Total faculty EI scores were not significantly correlated with MS evaluations (r = 0.30, p = 0.06). CONCLUSIONS: MS evaluations of surgeon faculty were not related to EI in the larger sample. However, EI did correlate to MS evaluations in 2 surgical specialties. Further exploration into the utility of EI training in surgical departments should be conducted to determine the true value of such endeavors.

Faculty Emotional Intelligence Matters for Resident Education.

OBJECTIVE: This study aimed to determine the emotional intelligence (EI) of surgical faculty and evaluate its relationship with resident evaluations of faculty behaviors. DESIGN: This study retrospectively collected faculty EI scores as well as general surgery resident evaluations of faculty. Parametric and nonparametric tests were used for statistical analysis. SETTING: The study was conducted at the University of Texas Southwestern in the Department of Surgery in Dallas, Texas. This is an academic, tertiary care center. PARTICIPANTS: Surgical faculty members at a single institution in 2018 completed the Emotional Intelligence Appraisal, a 28-item, electronic assessment with possible scores ranging from zero to 100. Aggregate, anonymous resident evaluations of faculty members were collected from the program director's office. Faculty with fewer than 8 resident evaluations were excluded. RESULTS: In total, 59 faculty members participated (89%). The sample was mostly white (69.2%), male (63.5%), with an average of 47 ± 10 years of age, 12.2 ± 10 years in practice, and 44 ± 24 evaluations per faculty member. The group's mean EI score was 76 ± 7.7. Faculty EI scores were found to have a moderate, positive correlation with resident rotational evaluations of faculty (r(51) = 0.52, p < 0.001). Faculty EI scores did not significantly correlate with resident evaluations of faculty intraoperative behaviors. CONCLUSIONS: The results of this study suggest that a majority of our faculty are competent with regard to EI. Furthermore, faculty EI is an important factor in the clinical learning environment and correlates with resident rotational evaluations of teaching behaviors.

Educating physician-assistant students as agents of lifestyle medicine.

BACKGROUND: The burden of chronic disease in the USA necessitates a shift in medicine and medical education from disease treatment to health and wellness promotion. At the forefront of this shift is the field of lifestyle medicine - the evidence-based use of lifestyle modification to prevent, treat and reverse disease. Health care providers, including physician assistants (PAs), can be the vehicle of behaviour change for patients, families and communities, and must receive adequate training to practice lifestyle medicine. METHODS: We describe a pilot lifestyle medicine curriculum implemented through a PA training programme run in the USA in the academic year 2016/2017. The curriculum included four taught modules on lifestyle medicine, two assessment activities and a survey of self-perceived competency, measured before and after the programme. RESULTS: Forty students participated in the curriculum. Results showed a significant increase in self-perceived competencies in seven of nine lifestyle medicine areas (p < 0.001). After the curriculum, all students were successfully able to provide a written lifestyle medicine prescription for a mock patient, with the most common prescription topics including nutrition-related prescriptions (41%), followed by physical activity (26%), addressing substance use (10%), mental health or stresses (10%) and sleep (7%). Furthermore, on average students were able to correctly identify 70% of the desired recommendations for a mock patient. CONCLUSION: Lifestyle medicine curricula can be successfully integrated into existing PA curricula, with demonstrated increases in self-assessments of competency and practical skills.

Medical and Physician Assistant Student Competence in Basic Life Support: Opportunities to Improve Cardiopulmonary Resuscitation Training.

INTRODUCTION: Medical and physician assistant (PA) students are often required to have Basic Life Support (BLS) education prior to engaging in patient care. Given the potential role of students in resuscitations, it is imperative to ensure that current BLS training prepares students to provide effective cardiopulmonary resuscitation (CPR). The objective of this study was to assess whether current BLS training produces student providers who can deliver BLS in an American Heart Association (AHA) guideline-adherent manner. METHODS: Students at a US medical school were recruited by convenience sampling. BLS performance immediately following a standard AHA BLS training course was evaluated during a two-minute CPR cycle using manikins. We also collected information on demographics, previous BLS training attendance, perceived comfort in providing CPR, and prior experiences in healthcare and providing or observing CPR. RESULTS: Among 80 participants, we found that compression rate, depth, and inter-compression recoil were AHA guideline-adherent for 90.0%, 68.8%, and 79.3% of total compression time, respectively. Mean hands-off time was also within AHA guidelines. Mean number of unsuccessful ventilations per cycle was 2.2. Additionally, 44.3% of ventilations delivered were of adequate tidal volume, 12.2% were excessive, and 41.0% were inadequate. Past BLS course attendance, prior healthcare certification, and previous provision of real-life CPR were associated with improved performance. CONCLUSION: Following BLS training, medical and PA students met a majority of AHA compressions guidelines, but not ventilations guidelines, for over 70% of CPR cycles. Maintaining compression depth and providing appropriate ventilation volumes represent areas of improvement. Conducting regular practice and involving students in real-life CPR may improve performance.

Getting better all the time? Facilitating accurate team self-assessments through simulation.

Introduction: Our study explores the extent to which teams are accurate assessors of their own performance and teamwork, and how simulation can help this critical skill develop over time. Methods: Surgery residents in teams of three completed five daily simulations. After each scenario, each team reviewed their performance and jointly completed a scenario-specific team performance evaluation and a 17-item Communication and Teamwork Skills tool. Videos were rated to obtain discrepancy values. Paired-samples t-tests and mean comparisons were used to examine changes in team self-assessment accuracy and comparisons between high-performing and low-performing teams. Results: Resident (n=30) teams rated team performance higher than faculty across the first 3 days (p<0.01), but provided similar ratings thereafter. Agreement of team performance from day 1 to 5 significantly improved (p<0.001). Teams rated their teamwork higher than faculty across all days (p<0.01). Top performing teams provided more accurate self-assessments for both teamwork (average discrepancy 8% vs 39%) and team performance (average discrepancy 12% vs 23%). Conclusion: Teams that continue to work together over time may become more accurate judges of their own performance, but do not become more accurate assessors of teamwork competencies.

Great Expectations? Future Competency Requirements Among Candidates Entering Surgery Training.

INTRODUCTION: We describe a multimethod, multi-institutional approach documenting future competencies required for entry into surgery training. METHODS: Five residency programs involved in a statewide collaborative each provided 12 to 15 subject matter experts (SMEs) to participate. These SMEs participated in a 1-hour semistructured interview with organizational psychologists to discuss program culture and expectations, and rated the importance of 20 core competencies derived from the literature for candidates entering general surgery training within the next 3 to 5 years (1 = importance decreases significantly; 3 = importance stays the same; 5 = importance increases significantly). RESULTS: Seventy-three SMEs across 5 programs were interviewed (77% faculty; 23% resident). All competencies were rated to be more important in the next 3 to 5 years, with team orientation (3.87 ± 0.81), communication (3.82 ± 0.79), team leadership (3.81 ± 0.82), feedback receptivity (3.79 ± 0.76), and professionalism (3.76 ± 0.89) rated most highly. CONCLUSIONS: These findings suggest that the competencies desired and required among future surgery residents are likely to change in the near future.

Video-based assessment for laparoscopic fundoplication: initial development of a robust tool for operative performance assessment.

INTRODUCTION: While better technical performance correlates with improved outcomes, there is a lack of procedure-specific tools to perform video-based assessment (VBA). SAGES is developing a series of VBA tools with enough validity evidence to allow reliable measurement of surgeon competence. A task force was established to develop a VBA tool for laparoscopic fundoplication using an evidence-based process that can be replicated for additional procedures. The first step in this process was to seek content validity evidence. METHODS: Forty-two subject matter experts (SME) in laparoscopic fundoplication were interviewed to obtain consensus on procedural steps, identify potential variations in technique, and to generate an inventory of required skills and common errors. The results of these interviews were used to inform creation of a task inventory questionnaire (TIQ) that was delivered to a larger SME group (n = 188) to quantify the criticality and difficulty of the procedural steps, the impact of potential errors associated with each step, the technical skills required to complete the procedure, and the likelihood that future techniques or technologies may change the presence or importance of any of these factors. Results of the TIQ were used to generate a list of steps, skills, and errors with strong validity evidence. RESULTS: Initial SMEs interviewed included fellowship program directors (45%), recent fellows (24%), international surgeons (19%), and highly experienced super SMEs with quality outcomes data (12%). Qualitative analysis of interview data identified 6 main procedural steps (visualization, hiatal dissection, fundus mobilization, esophageal mobilization, hiatal repair, and wrap creation) each with 2-5 sub steps. Additionally, the TIQ identified 5-10 potential errors for each step and 11 key technical skills required to perform the procedure. Based on the TIQ, the mean criticality and difficulty scores for the 11/21 sub steps included in the final scoring rubric is 4.66/5 (5 = absolutely essential for patient outcomes) and 3.53/5 (5 = difficulty level requires significant experience and use of alternative strategies to accomplish consistently), respectively. The mean criticality and frequency scores for the 9/11 technical skills included is 4.51/5 and 4.51/5 (5 = constantly used ≥ 80% of the time), respectively. The mean impact score of the 42/47 errors incorporated into the final rubric is 3.85/5 (5 = significant error that is unrecoverable, or even if recovered, likely to have a negative impact on patient outcome). CONCLUSIONS: A rigorous, multi-method process has documented the content validity evidence for the SAGES video-based assessment tool for laparoscopic fundoplication. Work is ongoing to pilot the assessment tool on recorded fundoplication procedures to establish reliability and further validity evidence.