Impact of Question Bank Use for In-Training Examination Preparation by OBGYN Residents - A Multicenter Study.

OBJECTIVE: To examine the impact of access to and utilization of a commercially available question bank (TrueLearn) for in-training examination (ITE) preparation in Obstetrics and Gynecology (OBGYN). DESIGN: This was a retrospective cohort study examining the impact of TrueLearn usage on ITE examination performance outcomes. Produced by the educational arm of the American College of Obstetricians and Gynecologists, the Council on Resident Education in Obstetrics and Gynecology (CREOG) exam is a multiple-choice test given to all residents annually. Residency programs participating in this study provided residency program mean CREOG scores from the year prior (2015), and the first (2016) and second (2017) years of TrueLearn usage. Programs also contributed resident-specific CREOG scores for each resident for 2016 and 2017. This data was combined with each resident's TrueLearn usage data that was provided by TrueLearn with residency program consent. The CREOG scores consisted of the CREOG score standardized to all program years, the CREOG score standardized to the same program year (PGY) and the total percent (%) correct. TrueLearn usage data included number of practice questions completed, number of practice tests taken, average number of days between successive tests, and percent correct of answered practice questions. SETTING: OBGYN Residency Training Programs. PARTICIPANTS: OBGYN residency programs that purchased and utilized TrueLearn for the 2016 CREOG examination were eligible for participation (n = 14). Ten residency programs participated, which consisted of 212 residents in 2016 and 218 residents in 2017. RESULTS: TrueLearn was used by 78.8% (167/212) of the residents in 2016 and 84.9% (185/218) of the residents in 2017. No significant difference was seen in the average CREOG scores available on a per- program level before versus after the first year of implementation either using the CREOG score standardized to all PGYs (mean difference 1.0; p = 0.58) or standardized to the same PGY (mean difference 3.1; p = 0.25). Using resident-level data, there was no significant difference in mean CREOG score standardized to all PGYs between users and non-users of TrueLearn in 2016 (mean, 199.4 vs 196.7; p = 0.41) or 2017 (mean, 198.2 vs 203.4; p = 0.19). The percent of practice questions answered correctly on TrueLearn was positively correlated with the CREOG score standardized to all PGYs (r = 0.47 for 2016 and r = 0.60 for 2017), as well as with the CREOG total percent correct (r = 0.47 for 2016 and r = 0.61 for 2017). Based on a simple linear regression, for every 500 practice questions completed, the CREOG score significantly increased for PGY-2 residents by an average (±SE) of 7.3 ± 2.8 points (p = 0.013); the average increase was 0.7 ± 2.5 (p = 0.79) for PGY-3 residents and 5.8 ± 3.3 points (p = 0.09) for PGY-4 residents. CONCLUSIONS: Adoption of an online question bank did not result in higher mean CREOG scores at participating institutions. However, performance on the TrueLearn questions correlated with ITE performance, supporting predictive validity and the use of this question bank as a formative assessment for resident education and exam preparation.

Can ACGME Milestones predict surgical specialty board passage: an example in Obstetrics and Gynecology.

BACKGROUND: Multiple tools including Accreditation Council for Graduate Medical Education (ACGME) standardized milestones can be utilized to assess trainee and residency program performance. However, little is known regarding the objective validation of these tools in predicting written board passage. METHODS: In this retrospective study, data was gathered on n = 45 Wayne State University Obstetrics and Gynecology program graduates over the five-year period ending July 2018. United States Medical Licensing Examination (USMLE) scores, Council on Resident Education in Obstetrics and Gynecology (CREOG) in-training scores and ACGME milestones were used to predict American Board of Obstetrics and Gynecology (ABOG) board passage success on first attempt. Significance was set at p < 0.05. RESULTS: Written board passage was associated with average CREOGs (p = 0.01) and milestones (p = 0.008) while USMLE1 was not significantly associated (p = 0.055). USMLE1 <217 (Positive predictive value (PPV) = 96%). CREOGs <197 (PPV = 100%) and milestones <3.25 (PPV = 100%), particularly practice-based learning and systems-based practice milestones were most strongly correlated with board failure. Using a combination of these two milestones, it is possible to correctly predict board passage using our model (PPV = 86%). DISCUSSION: This study is the first validating the utility of milestones in a surgical specialty by demonstrating their ability to predict board passage. Residents with CREOGs or milestones below thresholds are at risk for board failure and may warrant early intervention.

Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training.

AIM: The study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination. METHODS: In this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05. RESULTS: Mean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1. CONCLUSIONS: The change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant's future performance in residency.

The Effect of a Targeted Educational Activity on Obstetrics and Gynecology Resident In-Training Examination Scores.

The objective of this study was to compare the examination scores before and after implementation of a study program based on high yield topics on the Council of Resident Education in Obstetrics and Gynecology (CREOG) In-Training Examination. This prospective cohort study compared scores from academic years 2012 to 2014 of University of Hawai'i obstetrics and gynecology residents who participated in a directed study program based on selected high yield topics from the CREOG Test Item Summary Booklet. Topics were considered high yield if more than 75% of the program residents answered the topic questions incorrectly during the immediately preceding CREOG In-Training Examination administration. Residents were assigned topics to research and present at monthly teaching sessions. The presentations were made accessible in a wiki website. The intervention was initiated in 2012 and discontinued in 2013. The primary outcome was the difference among CREOG In-Training Examination scores before the study program, during the study program, and after the study program was discontinued. Only scores of residents who sat for all three exams were included. Eleven residents were present during the duration of the study period and sat for all three CREOG examinations. During the year of the educational activity, paired individual resident CREOG exams scores increased significantly from the 2012 CREOG administration (mean = 194.7) to the 2013 CREOG administration (mean = 208.2). These findings demonstrate that the CREOG Test Item Summary Booklet and the wiki platform can be used to effectively direct educational efforts resulting in improvements in CREOG examination performance.

Interactive case-based learning improves resident knowledge and confidence in reproductive endocrinology and infertility.

Resident physicians' scores on the REI section of the CREOG exam are traditionally low, and nearly 40% of house staff nation-wide perceive their REI knowledge to be poor. We aimed to assess whether an interactive case-based group-learning curriculum would narrow the REI knowledge gap by improving understanding and retention of core REI concepts under the time constraints affecting residents. A three-hour case-based workshop was developed to address four primary CREOG objectives. A multiple-choice test was administered immediately before and after the intervention and 7 weeks post-workshop, to evaluate both knowledge and confidence. Following the intervention, residents self-reported increased confidence with counseling and treatment of PCOS, ovulation induction cycle monitoring, counseling and treatment of POI, and breaking bad news related to infertility (p < 0.05). The multiple-choice exam was re-administered 7 weeks post-intervention, and scores remained significantly improved compared to pre-workshop scores (p < 0.05). At that time, all residents either strongly agreed (91.7%) or agreed (8.3%) that the case-based interactive format was preferable to traditional lecture-based teaching. In conclusion, a nontraditional curriculum aimed at teaching core REI concepts to residents through interactive case-based learning can be successfully integrated into a residency curriculum, and significantly improves knowledge and confidence of critical concepts in REI.