Standard-Setting Methods for Assessment in a Post-Graduate Medical College.

CONTEXT: Standard-setting procedures assess candidates' competence in an examination. Different standard-setting methods produce different pass scores, and no gold standard exists currently. The quality of the standard-setting process is critical in medical examinations where true competency needs to be determined for safe medical practice. AIMS: This study assessed the standard-setting methods the college uses to determine the pass scores in the various parts of the fellowship examinations and compared these methods with the arbitrary 50% previously used. SETTINGS AND DESIGN: A cross-sectional comparative study to assess the standard-setting methods adopted, which was applied to the September/October/November 2023 fellowship examinations. METHODS: This was a total population survey involving the 16 faculties of the College. Secondary data from a compilation of approved results was used. DATA ANALYSIS: Descriptive and analytical statistics in Microsoft Excel program. RESULTS: The methods for standard-setting adopted by the college were assessed, and their mean pass scores were analysed. The mean pass score for the primary multiple choice questions examinations was 46.7%, lower than the previously used arbitrary 50% mark, and this difference was statistically significant (P < 0.05). The mean pass scores for the other examinations were higher than the previously used arbitrary 50% mark, but these differences were not statistically significant (P > 0.05). CONCLUSION: Implementation of the approved standard-setting methods can be improved with more training and retraining of faculties and examiners so that results that are consistent with their purpose, and that align well with other measures of competency can be produced.

Performance of GPT-4 with Vision on Text- and Image-based ACR Diagnostic Radiology In-Training Examination Questions.

Background Recent advancements, including image processing capabilities, present new potential applications of large language models such as ChatGPT (OpenAI), a generative pretrained transformer, in radiology. However, baseline performance of ChatGPT in radiology-related tasks is understudied. Purpose To evaluate the performance of GPT-4 with vision (GPT-4V) on radiology in-training examination questions, including those with images, to gauge the model's baseline knowledge in radiology. Materials and Methods In this prospective study, conducted between September 2023 and March 2024, the September 2023 release of GPT-4V was assessed using 386 retired questions (189 image-based and 197 text-only questions) from the American College of Radiology Diagnostic Radiology In-Training Examinations. Nine question pairs were identified as duplicates; only the first instance of each duplicate was considered in ChatGPT's assessment. A subanalysis assessed the impact of different zero-shot prompts on performance. Statistical analysis included χ2 tests of independence to ascertain whether the performance of GPT-4V varied between question types or subspecialty. The McNemar test was used to evaluate performance differences between the prompts, with Benjamin-Hochberg adjustment of the P values conducted to control the false discovery rate (FDR). A P value threshold of less than.05 denoted statistical significance. Results GPT-4V correctly answered 246 (65.3%) of the 377 unique questions, with significantly higher accuracy on text-only questions (81.5%, 159 of 195) than on image-based questions (47.8%, 87 of 182) (χ2 test, P < .001). Subanalysis revealed differences between prompts on text-based questions, where chain-of-thought prompting outperformed long instruction by 6.1% (McNemar, P = .02; FDR = 0.063), basic prompting by 6.8% (P = .009, FDR = 0.044), and the original prompting style by 8.9% (P = .001, FDR = 0.014). No differences were observed between prompts on image-based questions with P values of .27 to >.99. Conclusion While GPT-4V demonstrated a level of competence in text-based questions, it showed deficits interpreting radiologic images. © RSNA, 2024 See also the editorial by Deng in this issue.

Educational needs assessment for health advocate role in family medicine residency training in Turkey: A Delphi study.

BACKGROUND: Health advocacy (HA) involves purposeful actions to inform, mobilise, and organise activities to address social determinants of health affecting individuals or communities. It is a fundamental component of medical practice, deemed mandatory by professional and educational bodies. Therefore, including health advocacy training in pre- and post-graduate medical education is crucial. OBJECTIVES: In this study, we aimed to determine the need for HA training for family physicians (FPs) based on expert opinions. METHODS: We conducted a modified Delphi study with 105 academic experts and active FPs to explore HA training needs. Using a three-round technique, experts first answered five open-ended questions on HA competencies, teaching and assessment methods, learning environments, and integration in residency training. In the second round, statements from the responses were rated on a 5-point Likert scale, in the third round, statements below the 85% consensus level were revised and re-evaluated. RESULTS: The panel consisted of 41 experts (33 academicians, 8 practitioners) who accepted the invitation and completed the study. At the end of the three rounds, consensus was reached on 38 statements for HA competencies, 15 for teaching methods, 8 for assessment methods, and 20 for integration for HA training. CONCLUSION: Competencies for the HA role are very broad in perspective and show commonalities with the FPs' 'professional', 'expert' and 'leader' roles. Longitudinally integration of the HA training into the national 'Family Medicine Residency Training Core Curriculum' through participatory processes and training of FM trainers in HA is strongly recommended.

The competencies expected for the health advocacy role of the family physicians show commonalities with the family physicians' 'professional', 'expert' and 'leader' roles.It is important to longitudinally integrate health advocacy training into the family medicine residency training core curriculum at the national level.

Teaching With the GlobalSurgBox: Trainer Perceptions of a Portable Surgical Simulator.

OBJECTIVES: Evaluate the utility of a low cost, portable surgical simulator (GlobalSurgBox) for surgical teaching and its ability to dismantle barriers faced by trainers when attempting to use surgical simulation. DESIGN: An anonymous survey was administered to surgical trainers who were involved in leading simulation events using the GlobalSurgBox in the past 2 years. The survey was designed to understand current barriers to using simulation as a trainer, and the utility of the GlobalSurgBox in overcoming these barriers. SETTING: Academic medical training centers or conferences in the United States, Rwanda and Kenya. PARTICIPANTS: 10 practicing surgeons, 3 practicing physicians, 11 surgical residents, 15 medical students and 1 anesthesia resident. RESULTS: The top 3 barriers for effective teaching were lack of convenient access to the simulator (50%), lack of trainer time (43%) and cost (28%). After using the GlobalSurgBox, 100% and 98% of respondents felt that it encourages more practice and offers significant advantages over current simulators in their program. About 90%, 88% and 70% of respondents believed that the GlobalSurgBox makes surgical simulation more convenient, affordable, and compatible with trainer time limitations, respectively. 83% of trainers agreed that it is a good replica of the operating room experience, and 85% practicing physicians were more likely to give autonomy to trainees after demonstrating competence on the GlobalSurgBox. CONCLUSION: The GlobalSurgBox mitigates several barriers surgical educators experience when practicing surgical skills with trainees. The convenience of the GlobalSurgBox can help facilitate the development of foundational surgical skills outside of the operating room.

Colonoscopy Training on Virtual-Reality Simulators or Physical Model Simulators: A Randomized Controlled Trial.

OBJECTIVE: This study employed a randomized controlled trial to assess the efficacy of virtual-reality (VR) simulators and physical model simulators on colonoscopy training to explore the optimal and evidence-based simulation training. DESIGN: Forty participants were divided into 2 groups and randomized as dyads: the VR simulator group and the physical model simulator group. All the participants performed a baseline test through porcine colonoscopy. After a 6 h simulation training, each participant underwent a post-test on a pig after bowel preparation, and the procedures were video-recorded. Both the baseline test and the post-test were blindly assessed by 2 experienced assistant director physicians based on the GAGES-C scoring system. SETTING: Simulation center, Renji Hospital, Shanghai Jiaotong University School of Medicine, Shanghai. PARTICIPANTS: Forty surgical residents without colonoscopy experience. RESULTS: Both the VR simulator group and the physical model simulator group improved significantly over the baseline test. The VR simulator group performed significantly better than the physical model simulator group, p=0.042. The participants in both groups expressed a high level of simulator satisfaction. CONCLUSIONS: Novice residents can benefit from both VR simulators and physical model simulators. The VR simulator was shown to be more effective for colonoscopy training. VR simulators were more recommended for novices conducting basic colonoscopy training.

Framing our Expectations: Variability in Entrustable Professional Activity Assessments.

OBJECTIVE: To determine the ability of surgical trainees and faculty to correctly interpret entrustability of a resident learner in a modeled patient care scenario. DESIGN: Prospective study utilizing a web-based survey including 4 previously-recorded short videos of resident learners targeted to specific levels of the American Board of Surgery's (ABS) Entrustment Scale. Respondents were asked to choose the entrustment level that best corresponded to their observations of the learner in the video. Responses were subcategorized by low and high entrustment. SETTING: Online, utilizing the Qualtrics survey platform. PARTICIPANTS: Survey targeting US surgical trainees and surgical faculty via email and social media. We received 31 complete responses and 2 responses which completed > 1 video assessment question without demographic information (n = 33). Respondents included 10 trainees (32%) and 21 attending surgeons (68%). RESULTS: Neither faculty nor trainees readily identified the targeted entrustment level for Question 1 (preoperative care of a patient with acute appendicitis with high entrustment, 36% correct), though evaluations of the remaining questions (2 through 4) demonstrated more accuracy (70, 84, and 75% correct, respectively). Faculty were more readily able than trainees to identify low entrustment (level Limited Participation) in intraoperative inguinal hernia repair (95% vs 60%, p = 0.03). After subcategorization to high and low entrustment, both residents and faculty were able to accurately identify entrustment 95% overall. CONCLUSIONS: Both trainees and attending surgeons were able to identify high- and low-performing residents on short video demonstrations using the ABS EPA entrustment scale. This provides additional evidence in support of the need for frequent observations of EPAs to account for the variability in raters' perceptions in addition to complexity of clinical scenarios. Frame-of-reference training via a video-based platform may also be beneficial for both residents and faculty as an ongoing EPA implementation strategy.

Filter Out the Noise: How to Narrow Your Search for the Perfect Match by the Collaboration of Surgical Education Fellows (CoSEF).

The residency match process can be overwhelming. We are the Collaboration of Surgical Education Fellows (CoSEF), a multi-institutional group of surgical residents. Our perspectives represent our current experiences as residents at academic programs, but all authors recently underwent the general surgery resident interview and match process, during which they interviewed at programs of all kinds. Based on our collective experiences, we aim to highlight program attributes that applicants should consider to find their perfect match.

How to develop entrustable professional activities in a training dermatology program; practical tips for dermatology educators.

Entrustable professional activities (EPAs) are units of professional practice that could be observed, assessed, monitored, documented, and entrusted. EPAs when entrusted, outline the physician's qualifications, and shape the scope of practice. This insight highlights the importance of development of EPAs in all medical specialties including dermatology to ensure the best standards for patient's care. Development of EPAs-based training program is considered a challenge for clinical educators. In this paper, we describe practical tips and reflections on our experience in developing EPAs in dermatology doctoral training program that could be a guide for dermatology educators to implement EPAs in dermatology training programs.

Pilot Study to Improve Resident Experience on Vascular Surgery by Standardizing Dissemination of Operative Steps.

OBJECTIVE: Many surgical residencies have passed along attendings preferences and procedural knowledge as a highly utilized but informal resource. The objective was to assess the effect of providing operative steps and attending preferences on surgical resident performance. DESIGN: This was a prospective observational study with a survey-based design. SETTING: We created and shared vascular surgery operative steps including institutional and attending preferences with junior residents at the Massachusetts General Hospital. PARTICIPANTS: There were a total of 31 residents who completed a survey to assess self-perception of performance in operative knowledge and Accreditation Council for Graduate Medical Education (ACGME) Milestone criteria. RESULTS: Advice from colleagues was the most utilized resource, followed by web-based materials. Of the web-based materials, almost all residents utilized Google searches over other web-based resources designed to specifically help surgical trainees. The vascular surgery resource was used by 90% of residents more than 3 times per week to prepare for operative cases. There was significant improvement in patient positioning, instrument selection, operative field exposure, anatomy, sequence of procedure, procedure choices, and peri-operative care knowledge. CONCLUSIONS: Development of institutional resources that specifically capture attending surgeon procedural variations can improve resident performance, encourage resident autonomy, and provide a catalog of approaches to challenging operative situations.

Needs Assessment for the Development of a Sustainability Curriculum for Surgical Residents.

INTRODUCTION: The healthcare sector accounts for 8.5% of United States (U.S.) greenhouse gas emissions, of which one-third comes from operating rooms (ORs). As a result, there is great interest in decarbonizing the OR and surgical care. However, surgical residents are not routinely educated on the negative environmental impact of surgery or how to reduce it. In this paper, we present a formal needs assessment for a sustainability curriculum geared towards surgical residents. METHODS: Using Kern's Six-Step Framework for curriculum development, we conducted focus groups with surgical residents to perform a targeted needs assessment on 3 main topics: 1) the current state of surgical sustainability curricula; 2) resident knowledge regarding the environmental impact of surgery and barriers to sustainable practice; and 3) preferred educational methods and topics within sustainability education. We audio-recorded all focus groups and performed thematic analysis using anonymized transcripts. RESULTS: Fourteen residents participated in 3 focus groups, from which a qualitative analysis revealed 4 themes. First, surgery residents receive limited formal teaching on the negative environmental impact of surgical care or how to reduce this impact. Second, surgery residents have variable levels of prior education about and interest in sustainability in surgery. Third, several barriers prevent the implementation of sustainable changes in surgical practice, including a lack of institutional initiative, cultural inertia, concerns about workflow efficiency, and limited formal education. Finally, residents prefer to learn about practical ways to reduce waste, specifically through interactive approaches such as quality improvement initiatives. CONCLUSIONS: Given the increasing importance of sustainability in surgery, there is an urgent need for formal resident education on this topic. This needs assessment provides a valuable foundation for future sustainability curriculum development.

Navigating the learning landscape: Comprehensive training in third space endoscopy - training, techniques, and practical recommendations.

Third space endoscopy (TSE), including ESD, POEM, or STER are advanced procedures requiring precise endoscopic control and tissue recognition. Despite its increasing adoption, evidence-based curricula, and standardized training protocols for TSE are lacking. This review explores training methods, cognitive skills, and technical proficiency requirements for endoscopists performing TSE, with a primary emphasis on POEM. Generally, it seems wise to recommend a step-up approach to TSE training, starting with ex-vivo models or POEM simulators; mechanical and virtual reality (VR) simulators are commonly used during early training. Preclinical training involving ex-vivo and live animal models is suggested to prepare trainees for safe and effective procedures. Studies suggest varying numbers of procedures for training, with approximately 20-40 cases needed before a first plateau is achieved in terms of complications and speed. The duration of on-patient clinical training varies depending on prior experience. Mentorship programs, workshops, and case discussions may facilitate dynamic knowledge transfer. In addition, adverse event management is a crucial aspect of any TSE training program. Existing evidence supports the use of preclinical models and emphasizes the importance of specialized training programs for TSE in alignment with our proposed step-up training approach. This review outlines practical recommendations for the theoretical knowledge and technical skills required before commencing TSE training, covering clinical understanding, diagnostic and outcome assessment, procedural requirements, and the role of mentorship programs.

Integrating Trauma-Informed Clinical Interviewing Skills in Medical Education for Screening Domestic Violence and Abuse.

A pre-post-training workshop was conducted among 22 postgraduate trainees of the Psychiatry and Emergency Department of a public sector hospital in Karachi. The workshop aimed to build the capacity of postgraduate trainees in trauma management by providing training on screening, clinical interviewing skills, and Psychological first aid to those exposed to domestic violence (DV). The effectiveness of each training module was evaluated by conducting a brief assessment before and after each module and scores were compared using the Wilcoxon signed-rank test. The post-test showed significant improvement in participants' knowledge on screening of domestic violence and trauma symptoms (p = 0.026) and psychological first aid (p <0.001). Postgraduate training programmes need to integrate trauma screening and management education, to prepare healthcare professionals to deal with sensitive clinical presentations at their work settings. Key Words: Postgraduate trainees, Trauma, Domestic violence, Medical education, Workshop.

Feasibility of initiating robotic surgery during the early stages of gastrointestinal surgery education.

PURPOSE: Minimally invasive surgery for gastrointestinal cancers is rapidly advancing; therefore, surgical education must be changed. This study aimed to examine the feasibility of early initiation of robotic surgery education for surgical residents. METHODS: The ability of staff physicians and residents to handle robotic surgical instruments was assessed using the da Vinci® skills simulator (DVSS). The short-term outcomes of 32 patients with colon cancer who underwent robot-assisted colectomy (RAC) by staff physicians and residents, supervised by a dual console system, between August 2022 and March 2024 were compared. RESULTS: The performances of four basic exercises were assessed after implementation of the DVSS. Residents required less time to complete these exercises and achieved a higher overall score than staff physicians. There were no significant differences in the short-term outcomes, operative time, blood loss, incidence of postoperative complications, and length of the postoperative hospital stay of the two surgeon groups. CONCLUSION: Based on the evaluation involving the DVSS and RAC results, it appears feasible to begin robotic surgery training at an early stage of surgical education using a dual console system.

From FUSE to a hands-on electrosurgery course using a cadaveric model.

INTRODUCTION: Surgical procedures in contemporary practice frequently employ energy-based devices, yet comprehensive education surrounding their safety and effectiveness remains deficient. We propose an innovative course for residents that aims to provide basic electrosurgery knowledge and promote the safe use of these devices. METHODS: We developed a simulated training course for first-year general surgery and orthopedic residents. First, a survey was conducted regarding their knowledge perception about energy devices. The course consisted of two online theoretical sessions, followed by three in-person practical sessions. First-year residents performed three video-recorded attempts using a cadaveric model and were assessed through a digital platform using the Objective Structured Assessment of Technical Skill (OSATS), a Specific Rating Scale (SRS), and a surgical energy-based devices scale (SEBS). Third-year residents were recruited as a control group. RESULTS: The study included 20 first-year residents and 5 third-year residents. First-year residents perceived a knowledge gap regarding energy devices. Regarding practical performance, both OSATS and checklist scores were statistically different between novices at their first attempt and the control group. When we analyzed the novice's performance, we found a significant increase in OSATS (13 vs 21), SRS (13 vs 17.5), and SEBS (5 vs 7) pre- and post-training scores. The amount of feedback referred to skin burns with the electro-scalpel reduced from 18 feedbacks in the first attempt to 2 in the third attempt (p-value = 0.0002). When comparing the final session of novices with the control group, no differences were found in the SRS (p = 0.22) or SEBS (p = 0.97), but differences remained in OSATS (p = 0.017). CONCLUSION: This study supports the implementation of structured education in electrosurgery among surgical trainees. By teaching first-year residents about electrosurgery, they can acquire a skill set equivalent to that of third-year residents. The integration of such courses can mitigate complications associated with energy device misuse, ultimately enhancing patient safety.

LapBot-Safe Chole: validation of an artificial intelligence-powered mobile game app to teach safe cholecystectomy.

BACKGROUND: Gaming can serve as an educational tool to allow trainees to practice surgical decision-making in a low-stakes environment. LapBot is a novel free interactive mobile game application that uses artificial intelligence (AI) to provide players with feedback on safe dissection during laparoscopic cholecystectomy (LC). This study aims to provide validity evidence for this mobile game. METHODS: Trainees and surgeons participated by downloading and playing LapBot on their smartphone. Players were presented with intraoperative LC scenes and required to locate their preferred location of dissection of the hepatocystic triangle. They received immediate accuracy scores and personalized feedback using an AI algorithm ("GoNoGoNet") that identifies safe/dangerous zones of dissection. Player scores were assessed globally and across training experience using non-parametric ANOVA. Three-month questionnaires were administered to assess the educational value of LapBot. RESULTS: A total of 903 participants from 64 countries played LapBot. As game difficulty increased, average scores (p < 0.0001) and confidence levels (p < 0.0001) decreased significantly. Scores were significantly positively correlated with players' case volume (p = 0.0002) and training level (p = 0.0003). Most agreed that LapBot should be incorporated as an adjunct into training programs (64.1%), as it improved their ability to reflect critically on feedback they receive during LC (47.5%) or while watching others perform LC (57.5%). CONCLUSIONS: Serious games, such as LapBot, can be effective educational tools for deliberate practice and surgical coaching by promoting learner engagement and experiential learning. Our study demonstrates that players' scores were correlated to their level of expertise, and that after playing the game, most players perceived a significant educational value.

An Ethical Framework for Disclosing the Training Status and Roles of Resident-Level Surgeons to Patients.

The concept of informed consent includes disclosure of all information that a reasonable patient would need to make a well-informed decision about whether to undergo a surgical procedure. This has traditionally been defined as including diagnosis, details about the procedure, prognosis, potential risks, and alternative treatments. The operating surgeon has final say and responsibility for the case, but the actual operation may be done (under supervision) by a surgeon in training. In this paper, we discuss the ethical dimensions of disclosing resident involvement, reviewing considerations such as established legal and professional standards, consequences for patients and for the surgical educators responsible for preparing future generations of surgeons, and patient rights. We conclude by offering a novel ethical framework intended to serve as a guide to disclosing resident involvement as part of the overall consent process.

The Plan-Do-Study-Act (PDSA): An Iterative Approach to Optimize Residents Performance in the American Board of Surgery in-Training Exam (ABSITE).

INTRODUCTION: American Board of Surgery (ABS) In-Training Examination (ITE), or ABSITE, preparation requires an effective study approach. In 2014, the ABS announced the alignment of ABSITE to the SCORE® Curriculum. We hypothesized that implementing a Plan-Do-Study-Act (PDSA) approach would help surgery residents improve their performance on the ABSITE. METHOD: Over 20 years, in a single institution, residents' ABSITE performance was evaluated over 3 timeframes: Time A (2004-2013), no specific curriculum; Time B (2014-2019), an annual comprehensive ABSITE-simulated SCORE®-based multiple-choice exam (MCQ) was administered; and Time C (2020-2023), like Time B with the addition of the PDSA approach for those with less than 60% correct on the ABSITE-simulated SCORE®-based exam. At the beginning of the academic year, in July, all residents are encouraged to (1) initiate a study plan for the upcoming ABSITE using SCORE® guided by the published ABSITE outlines content topics (Plan), (2) take an ABSITE-simulated SCORE®-based exam in October (Do), (3) assess the results/scores (Study), and (4) identify appropriate next steps (Act). Correlational analysis was performed to evaluate the association between ABSITE scores and ABSITE-simulated SCORE®-based exam scores in Time B and Time C. The primary outcome was the change in the proportions of ABSITE scores <30th percentile. RESULTS: A total of 294 ABSITE scores of 94 residents (34 females and 60 males) were analyzed. We found stronger correlation between the correct percentage on ABSITE and ABSITE-simulated SCORE®-based exam scores in Time C (r = 0.73, p < 0.0001) compared to Time B (0.62, p < 0.0001). The percentage of residents with ABSITE scores lower than 30th percentile dropped significantly from 14.0% to 3.7% (p = 0.016). CONCLUSION: Implementing the Plan-Do-Study-Act (PDSA) approach using the SCORE® curriculum significantly enhances residents' performance on the ABSITE exam. Surgery residents are encouraged to use this approach and to utilize the SCORE-contents outlined by the ABS in their study plan.