A Randomized Crossover Trial Evaluating the Impact of Cultural Dexterity Training on Surgical Residents' Knowledge, Cross-Cultural Care, Skills, and Beliefs: The Provider Awareness and Cultural dexterity Training for Surgeons (PACTS) Trial.

OBJECTIVES: This trial examines the impact of the Provider Awareness and Cultural dexterity Toolkit for Surgeons (PACTS) curriculum on surgical residents' knowledge, cross-cultural care, skills, and beliefs. SUMMARY BACKGROUND DATA: Cross-cultural training of providers may reduce healthcare outcome disparities, but its effectiveness in surgical trainees is unknown. METHODS: PACTS focuses on developing skills needed for building trust, working with patients with limited English proficiency, optimizing informed consent, and managing pain. The PACTS trial was a randomized crossover trial of 8 academic general surgery programs in the United States: The Early group ("Early") received PACTS between Periods 1 and 2, while the Delayed group ("Delayed") received PACTS between Periods 2 and 3. Residents were assessed pre- and post-intervention on Knowledge, Cross-Cultural Care, Self-Assessed Skills, and Beliefs. Chi-square and Fisher's exact tests were conducted to evaluate within- and between-intervention group differences. RESULTS: Of 406 residents enrolled, 315 were exposed to the complete PACTS curriculum. Early residents' Cross-Cultural Care (79.6% to 88.2%, P<0.0001), Self-Assessed Skills (74.5% to 85.0%, P<0.0001), and Beliefs (89.6% to 92.4%, P=0.0028) improved after PACTS; Knowledge scores (71.3% to 74.3%, P=0.0661) were unchanged. Delayed resident scores pre- to post-PACTS showed minimal improvements in all domains. When comparing the two groups at Period 2, Early residents had modest improvement in all 4 assessment areas, with statistically significant increase in Beliefs (92.4% vs 89.9%, P=0.0199). CONCLUSION: The PACTS curriculum is a comprehensive tool that improved surgical residents' knowledge, preparedness, skills, and beliefs, which will help with caring for diverse patient populations.

Hiatal hernia reporting: time to remove subjectivity?

INTRODUCTION: The size of a hiatal hernia (HH) is a key determinant of the approach for surgical repair. However, endoscopists will often utilize subjective terms, such as "small," "medium," and "large," without any standardized objective correlations. The aim of this study was to identify HHs described using objective axial length measurements versus subjective size allocations and compare them to their corresponding manometry and barium swallow studies. METHODS AND PROCEDURES: Retrospective chart reviews were conducted on 93 patients diagnosed endoscopically with HHs between 2017 and 2021 at Newton-Wellesley Hospital. Information was collected regarding their HH subjective size assessment, axial length measurement (cm), manometry results, and barium swallow readings. Linear regression models were used to analyze the correlation between the objective endoscopic axial length measurements and manometry measurements. Ordered logistic regression models were used to correlate the ordinal endoscopic and barium swallow subjective size allocations with the continuous axial length measurements and manometry measurements. RESULTS: Of the 93 endoscopy reports, 42 included a subjective size estimate, 38 had axial length measurement, and 12 gave both. Of the 34 barium swallow reads, only one gave an objective HH size measurement. Axial length measurements were significantly correlated with the manometry measurements (R2 = 0.0957, p = 0.049). The endoscopic subjective size estimates were also closely related to the manometry measurements (R2 = 0.0543, p = 0.0164). Conversely, the subjective size estimates from barium swallow reads were not significantly correlated with the endoscopic axial length measurements (R2 = 0.0143, p = 0.366), endoscopic subjective size estimates (R2 = 0.0481, p = 0.0986), or the manometry measurements (R2 = 0.0418, p = 0.0738). Mesh placement was significantly correlated to pre-operative endoscopic axial length measurement (p = 0.0001), endoscopic subjective size estimate (p = 0.0301), and barium swallow read (p = 0.0211). However, mesh placement was not significantly correlated with pre-operative manometry measurements (0.2227). CONCLUSIONS: Endoscopic subjective size allocations and objective axial length measurements are associated with pre-operative objective measurements and intra-operative decisions, suggesting both can be used to guide clinical decision making. However, including axial length measurements in endoscopy reports can improve outcomes reporting.

Effective Senior Surgical Residents as Defined by Their Peers: A Qualitative Content Analysis of Nontechnical Skills Development.

OBJECTIVE: This study aims to define an effective senior resident and understand the process of leadership and nontechnical skill development in the transition from junior to senior surgery resident. SUMMARY BACKGROUND: General surgery residents are responsible for patient care, technically demanding operations, and diverse care team management. However, leadership skill development for the transition from junior to senior resident roles is often overlooked. METHODS: We conducted 15 semi-structured focus groups with surgery residents from an urban, academic institution. Focus group transcripts were inductively coded. Using content analysis and constant comparative methodology, primary codes were refined into categories and organized into higher-level themes. RESULTS: Thirty-three general surgery residents completed fifteen focus groups. Six themes were identified. Three themes describe the process of becoming an effective senior resident: how to define a senior resident's scope of practice, the transition process, and the importance of personal investment. Three themes were identified regarding effective seniors: ideal traits, teachable skills, and the team and patient impact. CONCLUSIONS: Surgery residents define an effective senior resident as the team member with the highest level of experience who manages the big picture of patient care. The transition is improved by personal engagement and acknowledgement of the transition. Ideal traits of effective seniors, including emotional intelligence and inherent personality traits, allow a resident to more naturally assume this role; however, teachable skills, such as communication, expectation setting and competence, can be taught to improve one's effectiveness. The actions of a senior resident impact the team and patient care, underscoring the importance of understanding this role.

Inpatient Satisfaction With Surgical Resident Care After Elective General and Oncologic Surgery.

OBJECTIVE: To investigate inpatient satisfaction with surgical resident care. BACKGROUND: Surgical trainees are often the primary providers of care to surgical inpatients, yet patient satisfaction with surgical resident care is not well characterized or routinely assessed. METHODS: English-speaking, general surgery inpatients recovering from elective gastrointestinal and oncologic surgery were invited to complete a survey addressing their satisfaction with surgical resident care. Patients positively identified photos of surgical senior residents and interns before completing a modified version of the Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey (S-CAHPS). Adapted S-CAHPS items were scored using the "top-box" method. RESULTS: Ninety percent of recruited patients agreed to participate (324/359, mean age=62.2, 50.3% male). Patients were able to correctly identify their seniors and interns 85% and 83% of the time, respectively ( P =0.14). On a 10-point scale, seniors had a mean rating of 9.23±1.27 and interns had a mean rating of 9.01±1.49 ( P =0.14). Ninety-nine percent of patients agreed it was important to help in the education of future surgeons. CONCLUSIONS: Surgical inpatients were able to recognize their resident physicians with high frequency and rated resident care highly overall, suggesting that they may serve as a willing source of feedback regarding residents' development of core competencies such as interpersonal skills, communication, professionalism, and patient care. Future work should investigate how to best incorporate patient evaluation of surgical resident care routinely into trainee assessment to support resident development.

Multidisciplinary perceptions on robotic surgical training: the robot is a stimulus for surgical education change.

BACKGROUND: It is unclear how to best establish successful robotic training programs or if subspecialty robotic program principles can be adapted for general surgery practice. The objective of this study is to understand the perspectives of high-volume robotic surgical educators on best practices in robotic surgery training and to provide recommendations transferable across surgical disciplines. METHODS: This multi-institutional qualitative analysis involved semi-structured interviews with high-volume robotic educators from academic general surgery (AGS), community general surgery (CGS), urology (URO), and gynecology (GYN). Purposeful sampling and snowballing ensured high-volume status and geographically balanced representation across four strata. Interviews were transcribed, deidentified, and independently, inductively coded. A codebook was developed and refined using constant comparative method until interrater reliability kappa reached 0.95. A qualitative thematic, framework analysis was completed. RESULTS: Thirty-four interviews were completed: AGS (n = 9), CGS (n = 8), URO (n = 9), and GYN (n = 8) resulting in 40 codes and four themes. Theme 1: intangibles of culture, resident engagement, and faculty and administrative buy-in are as important as tangibles of robot and simulator access, online modules, and case volumes. Theme 2: robotic OR integration stresses the trainee-autonomy versus patient-safety balance. Theme 3: trainees acquire robotic skills along individual learning curves; benchmark assessments track progress. Theme 4: AGS can learn from URO and GYN through multidisciplinary collaboration but must balance pre-existing training program use with context-specific curricular needs. CONCLUSIONS: Robotic surgical experts emphasize the importance of universal training paradigms, such as a strong educational culture that balances autonomy and patient safety, collaboration between disciplines, and routine assessments for continuous growth. Often, introduction and acceptance of the robot serves as a stimulus to discuss broader surgical education change.

Transfer of virtual reality endoscopy training to live animal colonoscopy: a randomized control trial of proficiency vs. repetition-based training.

BACKGROUND: Low first-time pass rates of the Fundamentals of Endoscopic Surgery (FES) exam stimulated development of virtual reality (VR) simulation curricula for test preparation. This study evaluates the transfer of VR endoscopy training to live porcine endoscopy performance and compares the relative effectiveness of a proficiency-based vs repetition-based VR training curriculum. METHODS: Novice endoscopists completed pretesting including the FES manual skills examination and Global Assessment of GI Endoscopic Skills (GAGES) assessment of porcine upper and lower endoscopy. Participants were randomly assigned one of two curricula: proficiency-based or repetition-based. Following curriculum completion, participants post-tested via repeat FES examination and GAGES porcine endoscopy assessments. The two cohorts pre-to-post-test differences were compared using ANCOVA. RESULTS: Twenty-two residents completed the curricula. There were no differences in demographics or clinical endoscopy experience between the groups. The repetition group spent significantly more time on the simulator (repetition: 242.2 min, SD 48.6) compared to the proficiency group (proficiency: 170.0 min, SD 66.3; p = 0.013). There was a significant improvement in porcine endoscopy (pre: 10.6, SD 2.8, post: 16.6, SD 3.4; p < 0.001) and colonoscopy (pre: 10.4, SD 2.7, post: 16.4, SD 4.2; p < 0.001) GAGES scores as well as FES manual skills performance (pre: 270.9, SD 105.5, post: 477.4, SD 68.9; p < 0.001) for the total cohort. There was no difference in post-test GAGES performance or FES manual skills exam performance between the two groups. Both the proficiency and repetition group had a 100% pass rate on the FES skills exam following VR curriculum completion. CONCLUSION: A VR endoscopy curriculum translates to improved performance in upper and lower endoscopy in a live animal model. VR curricula type did not affect FES manual skills examination or live colonoscopy outcomes; however, a proficiency curriculum is less time-consuming and can provide a structured approach to prepare for both the FES exam and clinical endoscopy.

Challenges of virtual interviewing for surgical fellowships: a qualitative analysis of applicant experiences.

BACKGROUND: The COVID-19 pandemic forced surgical fellowship programs to transition from in-person to remote applicant interviews; the virtual interviewing format presented new and unique challenges. We sought to understand applicants' perceived challenges to virtual interviewing for a surgical fellowship program. METHOD: A grounded theory-based qualitative study was performed utilizing semi-structured interviews with fellowship applicants from the 2020 fellowship match. All Fellowship Council-registered applicants were eligible. We purposefully sampled participants to balance across gender, specialty-choice, and academic versus community-program affiliation. Interviews were inductively analyzed by two researchers for prominent themes. RESULTS: Fifteen interviews were conducted. Participants were 60% male (n = 9), with 33% (n = 5) from non-academic institutions. They applied for the following fellowships: Advanced Gastrointestinal/Minimal Invasive (55%), Bariatric (30%), Hepatopancreatobiliary (10%) and Surgical Oncology (5%). Four main themes emerged to describe virtual interview process challenges: (1) perceived data deficiency, (2) superficial personal connections, (3) magnification of non-professionalism, and (4) logistical frustrations. Applicants recommend program directors provide more information about the fellowship prior to interview day and offer informal independent interactions with current and previous fellows. CONCLUSIONS: According to fellowship applicants, virtual interviews resulted in a lack of information for rank-list decision making ultimately requiring them to rely on other information avenues to base their decisions. These applicants have offered advice to fellowship program directors and future applicants to better optimize this process.

Implementation of competence by design in Canadian neurosurgery residency programs.

OBJECTIVE: The Royal College of Physicians and Surgeons of Canada (RCPSC) recently redesigned the Canadian neurosurgery residency training curriculum by implementing a competency-based model of training known as Competence by Design (CBD) centered around the assessment of Entrustable Professional Activities (EPAs). This sequential explanatory mixed-methods study evaluated potential benefits and pitfalls of CBD in Canadian neurosurgery residency education. METHODS: Two four-month interval surveys were distributed to all Canadian neurosurgery residents participating in CBD. The surveys assessed important educational components: CBD knowledge of key stakeholders, potential system barriers, and educational/psychological impacts on residents. Paired t-tests were done to assess changes over time. Based on longitudinal survey responses, semi-structured interviews were conducted to investigate in-depth residents' experience with CBD in neurosurgery. The qualitative analysis followed an explanatory approach, and a thematic analysis was performed. RESULTS: Surveys had 82% average response rate (n = 25). Over time, most residents self-reported that they retrospectively understood concepts around CBD intentions (p = 0.02). Perceived benefits included faculty evaluations with more feedback that was clearer and more objective (53% and 51%). Pitfalls included the amount of time needed to navigate through EPAs (90%) and residents forgetting to initiate EPA forms (71%). There was no significant change over time. During interviews, five key themes were found. Potential solutions identified by residents to enhance their experience included learning analytics data availability, mobile app refinement, and dedicated time to integrate EPAs in the workflow. CONCLUSION: This study was the first to assess resident-perceived benefits and pitfalls of the neurosurgery CBD training program in an educational framework context. In general, residents believed that theoretical principles behind CBD were valuable, but that technological ability and having enough time to request EPA assessments were significant barriers to success. Long-term studies are required to determine the definitive outcomes of CBD on residents' performance and ultimately, on patient care.

Psychological safety between pediatric residents and nurses and the impact of an interdisciplinary simulation curriculum.

BACKGROUND: Effective teamwork in interdisciplinary healthcare teams is necessary for patient safety. Psychological safety is a key component of effective teamwork. The baseline psychological safety on pediatric inpatient healthcare teams is unknown. The purpose of this study is to determine the baseline psychological safety between pediatric nurses and residents and examine the impact of an interdisciplinary nighttime simulation curriculum. METHODS: A convergent, multistage mixed methods approach was used. An interprofessional simulation curriculum was implemented fall 2020 to spring 2021. Qualitative focus group data and quantitative survey data on team psychological safety were collected and compared, both pre- and post-intervention and across nurses and residents. Thematic analysis of the qualitative data was conducted, and themes integrated with survey findings. RESULTS: Data were collected from 30 nurses and 37 residents pre-intervention and 32 and 38 post-intervention, respectively. Residents and nurses negatively rated psychological safety (pre-intervention mean = 3.40 [SD = 0.72]; post-intervention mean = 3.35 [SD = 0.81]). At both times psychological safety was rated significantly lower for residents (pre-intervention mean = 3.11 [SD = 0.76], post-intervention mean = 2.98 [SD = 0.84]) than nurses (pre-intervention mean = 3.76 [SD = 0.45], post-intervention mean = 3.79 [SD = 0.50]), all P < .001. Qualitative analysis identified six integrated themes: (1) influence of existing relationships on future interactions, (2) unsatisfactory manner and frequency of communication, (3) unsatisfactory resolution of disagreements (4) overwhelming resident workload impairs collaboration, (5) interpersonal disrespect disrupts teamwork, and (6) interprofessional simulation was useful but not sufficient for culture improvement. CONCLUSION: Resident-nurse team psychological safety ratings were not positive. While interprofessional simulation curriculum shows promise, additional efforts are needed to improve psychological safety among residents and nurses.

Evaluating and Comparing Emotional Intelligence and Improvement Mindset of Plastic Surgery Residents.

BACKGROUND: Emotional Intelligence (EI) has been linked to burnout, job satisfaction, and work performance among resident physicians. Individuals with a growth mindset believe intelligence and ability are traits that can be increased or improved upon through learning. EI and mindset have yet to be studied among plastic surgeons in the United States. MATERIALS AND METHODS: An online survey was distributed to plastic surgery residents and general surgery residents at a single institution containing a validated EI survey tool (TEIQue-SF). Second, a survey was sent to all plastic surgery residents in the United States containing the TEIQue-SF and a validated survey tool to assess mindset (ITIS). RESULTS: The response rate for the local study was 82% (plastic surgery) and 75% (general surgery). Only 7.8% of plastic surgery residents had any prior formal EI training or education. Mean global EI scores of local plastic surgery residents were higher than the normative population sample (P <0.0001). Plastic surgery and general surgery residents had similar EI scores. Integrated residents and junior plastic surgery residents had higher Well-Being scores compared to independent (P = 0.04) and senior residents (P = 0.04). Sixty-four plastic surgery residents completed the national survey. No correlation was found between EI and ITIS scores among the national plastic surgery resident cohort. CONCLUSION: Despite different work profiles, general surgery and plastic surgery residents may have similar EI profiles. Mindset does not appear to correlate to EI domains. Future studies will focus on how EI and mindset may develop over the course of residency training and their relationship to overall resident wellness.

A Needs Assessment of Video-based Education Resources Among General Surgery Residents.

BACKGROUND: Video-based education (VBE) is an effective tool for knowledge and skill acquisition for medical students, but its utility is less clear for resident physicians. We sought to determine how to incorporate VBE into a general surgery resident operative curriculum. METHODS: We conducted a single-institution, survey-based needs assessment of general surgery residents to determine desired content and format of an operative VBE module. RESULTS: The response rate was 84% (53/63), with 66% senior (postgraduate year ≥3) resident respondents. VBE was the most commonly cited resource that residents used to prepare for an operation (93%) compared with surgical textbooks (89%) and text-based website content (57%). Junior residents were more likely to utilize text-based website content than senior residents (P < 0.01). The three most important operative video components were accuracy, length, and cost. Senior residents significantly preferred videos that were peer-reviewed (P < 0.05) and featured attending surgeons whom they knew (P = 0.03). A majority of residents (59%) believed 5-10 min is the ideal length of an operative video. Across all postgraduate year levels, residents indicated that detailed instruction of each operative step was the most important content of a VBE module. Senior residents believed that the overall indications and details of each step of the operation were the most important contents of VBE for a junior resident. CONCLUSIONS: At this institution, general surgery residents preferentially use VBE resources for operative preparation. A centralized, standardized operative resource would likely improve resident studying efficiency, but would require personalized learning options to work for both junior and senior surgery residents.

#SurgEdVidz: Using Social Media to Create a Supplemental Video-Based Surgery Didactic Curriculum.

BACKGROUND: A general surgery resident may have competing clinical responsibilities that limit the ability to participate in educational didactics. Social media is an alternative delivery method for educational content that can be viewed at a resident's convenience. We sought to assess the feasibility of using social media to provide educational content for surgical residents. MATERIALS AND METHODS: We created a novel, supplemental surgical curriculum consisting of two-minute videos based on the This Week in SCORE modules. Videos were posted to Twitter weekly, indexed with the #SurgEdVidz hashtag and uploaded to YouTube to create a catalog of videos. Twitter and YouTube analytics were used to calculate public viewership and impact. RESULTS: A total of 43 videos were disseminated between January 30, 2019 and January 31, 2020. An average of 6.9 h (SD: 1.4; range: 5.0-10.0) was required to generate each video, including 1.5 h (SD: 0.6; range: 0-2.0) for content review and 5.4 h (SD: 1.1; range: 4.0-8.0) for video production. Between the two platforms, videos generated a total of 51,313 views with an average of 1193 views (SD: 715; range: 412-4096) per video. Account followers are geographically distributed across 28 states, 28 countries, and six continents. CONCLUSIONS: Social media can serve as an effective tool for the distribution of surgical educational content. Twitter, in conjunction with YouTube, allows for rapid dissemination of didactic content packaged into brief videos that is flexible in viewership with low time commitment. Social media offers visibility and engagement beyond the classroom without geographical or temporal borders.

Surgeons With Five or More Actual Cricothyrotomies Perform Significantly Better on a Virtual Reality Simulator.

BACKGROUND: Discriminating performance of learners with varying experience is essential to developing and validating a surgical simulator. For rare and emergent procedures such as cricothyrotomy (CCT), the criteria to establish such groups are unclear. This study is to investigate the impact of surgeons' actual CCT experience on their virtual reality simulator performance and to determine the minimum number of actual CCTs that significantly discriminates simulator scores. Our hypothesis is that surgeons who performed more actual CCT cases would perform better on a virtual reality CCT simulator. METHODS: 47 clinicians were recruited to participate in this study at the 2018 annual conference of the Society of American Gastrointestinal and Endoscopic Surgeons. We established groups based on three different experience thresholds, that is, the minimal number of CCT cases performed (1, 5, and 10), and compared simulator performance between these groups. RESULTS: Participants who had performed more clinical cases manifested higher mean scores in completing CCT simulation tasks, and those reporting at least 5 actual CCTs had significantly higher (P = 0.014) simulator scores than those who had performed fewer cases. Another interesting finding was that classifying participants based on experience level, that is, attendings, fellows, and residents, did not yield statistically significant differences in skills related to CCT. CONCLUSIONS: The simulator was sensitive to prior experience at a threshold of 5 actual CCTs performed.

Identification of Specific Educational Targets to Improve the Student Surgical Clerkship Experience.

BACKGROUND: This study describes the relationship between medical student perception of surgery, frequency of positive surgery clerkship activities, and overall surgical clerkship experience. METHODS: Medical students at four academic hospitals completed pre- and post-clerkship surveys assessing 1) surgery clerkship activities/experiences and 2) perceptions of surgery during the 2017-2018 academic year. RESULTS: Ninety-one percent of students completed both a pre- and post-clerkship survey (n = 162 of 179). Student perception of surgery significantly improved across the clerkship overall (P < 0.0001) and for 7 of 21 specific items. Eighty-six percent of students agreed that the clerkship was a meaningful experience. Sixty-six percent agreed that the operating room was a positive learning environment. Multivariable logistic regression identified one-on-one mentoring from a resident (OR [95% CI] = 2.12 [1.11-4.04], P = 0.02) and establishing a meaningful relationship with a surgical patient (OR = 2.21 [1.12-4.37], P = 0.02) as activities predictive of student agreement that the surgical clerkship was meaningful. Making an incision (OR = 2.92 [1.54-5.56], P = 0.001) and assisting in dissection (OR = 1.67 [1.03-2.69], P = 0.035) were predictive of student agreement that the operating room was a positive learning environment. Positive student perception of surgery before the clerkship was associated with increased frequency of positive clerkship activities including operative involvement (r = 0.26, P = 0.001) and relationships with surgical attendings (r = 0.20, P = 0.01), residents (r = 0.41, P < 0.0001), and patients (r = 0.24, P = 0.003). CONCLUSIONS: Interventions to improve surgery clerkship quality should target enhancing student relationships with residents and surgical patients as well as providing opportunity for student operative involvement beyond just suturing. In addition, fostering positive perceptions of surgery in the preclinical period may increase meaningfulness and experience with the later surgery clerkship.

Feasibility and Perceived Usefulness of Using Head-Mounted Cameras for Resident Video Portfolios.

BACKGROUND: There is limited guidance on how to longitudinally administer simulation materials or to incorporate video recordings into assessment portfolios of simulated surgical skills. MATERIALS AND METHODS: We launched a longitudinal weekly simulation curriculum for PGY1-PGY3 surgical residents based on the ACS/APDS Curriculum. Residents underwent monthly objective structured assessment of technical skills (OSATS) while wearing head-mounted cameras. Videos of OSATS performance accrued into individual online video portfolios. Residents were surveyed about their attitudes toward video recording. RESULTS: Twenty-seven general surgical residents participated, completing 161 OSATS encompassing 11 distinct skills and generating 258 videos of simulated skills performance. The overall survey response rate was 88%. Residents viewed the curriculum favorably overall, and 36.4% of residents accessed their videos. Of those who did not watch their videos, 78.6% cited not having enough time, whereas 28.6% did not think the videos would be useful. Over 95% of surveyed residents expressed interest in having a video library of attending-performed procedures, 59.1% were interested in having their own operations recorded, and 45.5% were interested in video-based coaching. CONCLUSIONS: Residents viewed longitudinal administration of the ACS/APDS Curriculum positively. Although video recording in simulation is feasible, resident interest may be higher for intraoperative recordings than for simulated skills.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

A proficiency-based virtual reality endoscopy curriculum improves performance on the fundamentals of endoscopic surgery examination.

INTRODUCTION: The fundamentals of endoscopic surgery (FES) examination is a national test of knowledge and skill in flexible gastrointestinal endoscopy. The skill portion of the examination involves five tasks that assesses the following skills: scope navigation, loop reduction, mucosal inspection, retroflexion, and targeting. This project aimed to assess the efficacy of a proficiency-based virtual reality (VR) curriculum in preparing residents for the FES skills exam. METHODS: Experienced (>100 career colonoscopies) and inexperienced endoscopists (<50 career colonoscopies) were recruited to participate. Six VR modules were identified as reflecting the skills tested in the exam. All participants were asked to perform each of the selected modules twice, and median performance was compared between the two groups. Inexperienced endoscopists were subsequently randomized in matched pairs into a repetition (10 repetitions of each task) or proficiency curriculum. After completion of the respective curriculum, FES scores and pass rates were compared to national data and historical institutional control data (endoscopy-rotation training alone). RESULTS: Five experienced endoscopists and twenty-three inexperienced endoscopists participated. Construct valid metrics were identified for six modules and proficiency benchmarks were set at the median performance of experienced endoscopists. FES scores of inexperienced endoscopists in the proficiency group had significantly higher FES scores (530 ± 86) versus historical control (386.7 ± 92.2, p = 0.0003) and higher pass rate (proficiency: 100%, historical control 61.5%, p = 0.01). CONCLUSION: Trainee engagement in a VR curriculum yields superior FES performance compared to an endoscopy rotation alone. Compared to the 2012-2016 national resident pass rate of 80, 100% of trainees in a proficiency-based curriculum passed the FES manual skills examination.

Development of a train-to-proficiency curriculum for the technical skills component of the fundamentals of endoscopic surgery exam.

BACKGROUND: The demonstration of competency in endoscopy is required prior to obtaining American Board of Surgery Certification. To demonstrate competency, the resident must pass a national high-stakes cognitive test and a technical skills exam on a virtual reality simulator. The purpose of this preliminary study was to design a proficiency-based endoscopy simulation curriculum to meet this competency requirement. METHODS: This is a mixed methods prospective cohort study at a single academic medical institution. Prior to taking the national exam, surgery residents were required to participate in a skills lab and demonstrate proficiency on 10 simulation tasks. Proficiency was based on time and percent of objects targeted/mucosa seen. Simulation practice time, number of task repetitions to proficiency, and prior endoscopic experience were recorded. Resident's self-reported confidence scores in endoscopic skills prior to and following simulation lab training were obtained. RESULTS: From January 1, 2016 through August 1, 2017, 20 surgical residents (8 PGY2, 8 PGY3, 4 PGY4) completed both a faculty-supervised endoscopy skills lab and independent learning with train-to-proficiency simulation tasks. Median overall simulator time per resident was 306 min (IQR: 247-405 min). Median overall time to proficiency in all tasks was 235 min (IQR: 208-283 min). The median time to proficiency decreased with increasing PGY status (r = 0.4, P = 0.05). There was no correlation between prior real-time endoscopic experience and time to proficiency. Reported confidence in endoscopic skills increased significantly from mean of 5.75 prior to 7.30 following the faculty-supervised endoscopy skills lab (P = 0.0002). All 20 residents passed the national exam. CONCLUSIONS: In this preliminary study, a train-to-proficiency curriculum in endoscopy improved surgical resident's confidence in their endoscopic skills and 100% of residents passed the FES technical skills test on their first attempt. Our findings also indicate that uniform proficiency was not achieved by real-time experience alone.

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study.

STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.