Which Internal Medicine Clerkship Characteristics Are Associated With Students' Performance on the NBME Medicine Subject Exam? A Multi-Institutional Analysis.

PURPOSE: To identify which internal medicine clerkship characteristics may relate to NBME Medicine Subject Examination scores, given the growing trend toward earlier clerkship start dates. METHOD: The authors used linear mixed effects models (univariable and multivariable) to determine associations between medicine exam performance and clerkship characteristics (longitudinal status, clerkship length, academic start month, ambulatory clinical experience, presence of a study day, involvement in a combined clerkship, preclinical curriculum type, medicine exam timing). Additional covariates included number of NBME clinical subject exams used, number of didactic hours, use of a criterion score for passing the medicine exam, whether medicine exam performance was used to designate clerkship honors, and United States Medical Licensing Examination Step 1 performance. The sample included 24,542 examinees from 62 medical schools spanning 3 academic years (2011-2014). RESULTS: The multivariable analysis found no significant association between clerkship length and medicine exam performance (all pairwise P > .05). However, a small number of examinees beginning their academic term in January scored marginally lower than those starting in July (P < .001). Conversely, examinees scored higher on the medicine exam later in the academic year (all pairwise P < .001). Examinees from schools that used a criterion score for passing the medicine exam also scored higher than those at schools that did not (P < .05). Step 1 performance remained positively associated with medicine exam performance even after controlling for all other variables in the model (P < .001). CONCLUSIONS: In this sample, the authors found no association between many clerkship variables and medicine exam performance. Instead, Step 1 performance was the most powerful predictor of medicine exam performance. These findings suggest that medicine exam performance reflects the overall medical knowledge students accrue during their education rather than any specific internal medicine clerkship characteristics.

A Guide for Increasing Scholarship for Medical Educators.

Disseminating scholarly work as a clinician educator is critical to furthering new knowledge in medical education, creating an evidence base for new practices, and increasing the likelihood of promotion. Knowing how to initiate scholarship and develop habits to support it, however, may not be clear. This perspective is designed to help readers choose and narrow their focus of scholarly interest, garner mentors, find potential project funding, and identify outside support through involvement with national organizations, collaborators, and faculty development programs. By incorporating these suggestions into their daily work, educators can find ways to connect their clinical and educational interests and make their daily work count toward scholarship.

Crowdsourcing for assessment items to support adaptive learning.

PURPOSE: Adaptive learning requires frequent and valid assessments for learners to track progress against their goals. This study determined if multiple-choice questions (MCQs) "crowdsourced" from medical learners could meet the standards of many large-scale testing programs. METHODS: Users of a medical education app (Osmosis.org, Baltimore, MD) volunteered to submit case-based MCQs. Eleven volunteers were selected to submit MCQs targeted to second year medical students. Two hundred MCQs were subjected to duplicate review by a panel of internal medicine faculty who rated each item for relevance, content accuracy, and quality of response option explanations. A sample of 121 items was pretested on clinical subject exams completed by a national sample of U.S. medical students. RESULTS: Seventy-eight percent of the 200 MCQs met faculty reviewer standards based on relevance, accuracy, and quality of explanations. Of the 121 pretested MCQs, 50% met acceptable statistical criteria. The most common reasons for exclusion were that the item was too easy or had a low discrimination index. CONCLUSIONS: Crowdsourcing can efficiently yield high-quality assessment items that meet rigorous judgmental and statistical criteria. Similar models may be adopted by students and educators to augment item pools that support adaptive learning.

Performance on the Nephrology In-Training Examination and ABIM Nephrology Certification Examination Outcomes.

BACKGROUND AND OBJECTIVES: Medical specialty and subspecialty fellowship programs administer subject-specific in-training examinations to provide feedback about level of medical knowledge to fellows preparing for subsequent board certification. This study evaluated the association between the American Society of Nephrology In-Training Examination and the American Board of Internal Medicine Nephrology Certification Examination in terms of scores and passing status. DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS: The study included 1684 nephrology fellows who completed the American Society of Nephrology In-Training Examination in their second year of fellowship training between 2009 and 2014. Regression analysis examined the association between In-Training Examination and first-time Nephrology Certification Examination scores as well as passing status relative to other standardized assessments. RESULTS: This cohort included primarily men (62%) and international medical school graduates (62%), and fellows had an average age of 32 years old at the time of first completing the Nephrology Certification Examination. An overwhelming majority (89%) passed the Nephrology Certification on their first attempt. In-Training Examination scores showed the strongest association with first-time Nephrology Certification Examination scores, accounting for approximately 50% of the total explained variance in the model. Each SD increase in In-Training Examination scores was associated with a difference of 30 U (95% confidence interval, 27 to 33) in certification performance. In-Training Examination scores also were significantly associated with passing status on the Nephrology Certification Examination on the first attempt (odds ratio, 3.46 per SD difference in the In-Training Examination; 95% confidence interval, 2.68 to 4.54). An In-Training Examination threshold of 375, approximately 1 SD below the mean, yielded a positive predictive value of 0.92 and a negative predictive value of 0.50. CONCLUSIONS: American Society of Nephrology In-Training Examination performance is significantly associated with American Board of Internal Medicine Nephrology Certification Examination score and passing status.

Performance on the Cardiovascular In-Training Examination in Relation to the ABIM Cardiovascular Disease Certification Examination.

The American College of Cardiology In-Training Exam (ACC-ITE) is incorporated into most U.S. training programs, but its relationship to performance on the American Board of Internal Medicine Cardiovascular Disease (ABIM CVD) Certification Examination is unknown. ACC-ITE scores from third-year fellows from 2011 to 2014 (n = 1,918) were examined. Covariates for regression analyses included sex, age, medical school country, U.S. Medical Licensing Examination Step, and ABIM Internal Medicine Certification Examination scores. A secondary analysis examined fellows (n = 511) who took the ACC-ITE in the first and third years. ACC-ITE scores were the strongest predictor of ABIM CVD scores (p < 0.0001), and the most significant predictor of passing (p < 0.0001). The change in ACC-ITE scores from first to third year was a strong predictor of the ABIM CVD score (p < 0.001). The ACC-ITE is strongly associated with performance on the ABIM CVD Certification Examination.

The SGIM TEACH Program: A Curriculum for Teachers of Clinical Medicine.

BACKGROUND: Demand for faculty with teaching expertise is increasing as medical education is becoming well established as a career pathway. Junior faculty may be expected to take on teaching responsibilities with minimal training in teaching skills. AIM: To address the faculty development needs of junior clinician-educators with teaching responsibilities and those changing their career focus to include teaching. SETTING: Sessions at two Society of General Internal Medicine (SGIM) annual meetings combined with local coaching and online learning during the intervening year. PARTICIPANTS: Eighty-nine faculty scholars in four consecutive annual cohorts from 2013 to 2016. PROGRAM DESCRIPTION: Scholars participate in a full-day core teaching course as well as selective workshops at the annual meetings. Between meetings they receive direct observation and feedback on their teaching from a local coach and participate in an online discussion group. PROGRAM EVALUATION: Sessions were evaluated using a post-session survey. Overall content rating was 4.48 (out of 5). Eighty-nine percent of participants completed all requirements. Of these, 100% agreed that they had gained valuable knowledge and skills. DISCUSSION: The TEACH certificate program provides inexperienced faculty teachers an opportunity to develop core skills. Satisfaction is high. Future research should focus on the impact that this and similar programs have on teaching skills.

Using National Health Care Databases and Problem-Based Practice Analysis to Inform Integrated Curriculum Development.

One challenge when implementing case-based learning, and other approaches to contextualized learning, is determining which clinical problems to include. This article illustrates how health care utilization data, readily available from the National Center for Health Statistics (NCHS), can be incorporated into an educational needs assessment to identify medical problems physicians are likely to encounter in clinical practice. The NCHS survey data summarize patient demographics, diagnoses, and interventions for tens of thousands of patients seen in various settings, including emergency departments (EDs), clinics, and hospitals.Selected data from the National Hospital Ambulatory Medical Care Survey: Emergency Department illustrate how instructional materials can be derived from the results of such public-use health care data. Using fever as the reason for visit to the ED, the patient management path is depicted in the form of a case drill-down by exploring the most common diagnoses, blood tests, diagnostic studies, procedures, and medications associated with fever.Although these types of data are quite useful, they should not serve as the sole basis for determining which instructional cases to include. Additional sources of information should be considered to ensure the inclusion of cases that represent infrequent but high-impact problems and those that illustrate fundamental principles that generalize to other cases.

Testing and evaluation: the present and future of the assessment of medical professionals.

The aim of this review is to highlight recent and potential future enhancements to the United States Licensing Examination (USMLE) program. The USMLE program is co-owned by the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards. The USMLE includes four examinations: Step 1, Step 2 Clinical Knowledge, Step 2 Clinical Skills, and Step 3; every graduate of Liaison Committee on Medical Education-accredited allopathic medical schools and all international medical graduates must pass this examination series to practice medicine in the United States. From 2006 to 2009, the program underwent an indepth review resulting in five accepted recommendations. These recommendations have been the primary driver for many of the recent enhancements, such as an increased emphasis on foundational science and changes in the clinical skills examination, including more advanced communication skills assessment. These recommendations will continue to inform future changes such as access to references (e.g., a map of metabolic pathways) or decision-making tools for use during the examination. The NBME also provides assessment services globally to medical schools, students, residency programs, and residents. In 2015, >550,000 assessments were provided through the subject examination program, NBME self-assessment services, and customized assessment services.

Performance on the adult rheumatology in-training examination and relationship to outcomes on the rheumatology certification examination.

OBJECTIVE: The American College of Rheumatology (ACR) Adult Rheumatology In-Training Examination (ITE) is a feedback tool designed to identify strengths and weaknesses in the content knowledge of individual fellows-in-training and the training program curricula. We determined whether scores on the ACR ITE, as well as scores on other major standardized medical examinations and competency-based ratings, could be used to predict performance on the American Board of Internal Medicine (ABIM) Rheumatology Certification Examination. METHODS: Between 2008 and 2012, 629 second-year fellows took the ACR ITE. Bivariate correlation analyses of assessment scores and multiple linear regression analyses were used to determine whether ABIM Rheumatology Certification Examination scores could be predicted on the basis of ACR ITE scores, United States Medical Licensing Examination scores, ABIM Internal Medicine Certification Examination scores, fellowship directors' ratings of overall clinical competency, and demographic variables. Logistic regression was used to evaluate whether these assessments were predictive of a passing outcome on the Rheumatology Certification Examination. RESULTS: In the initial linear model, the strongest predictors of the Rheumatology Certification Examination score were the second-year fellows' ACR ITE scores (ß = 0.438) and ABIM Internal Medicine Certification Examination scores (ß = 0.273). Using a stepwise model, the strongest predictors of higher scores on the Rheumatology Certification Examination were second-year fellows' ACR ITE scores (ß = 0.449) and ABIM Internal Medicine Certification Examination scores (ß = 0.276). Based on the findings of logistic regression analysis, ACR ITE performance was predictive of a pass/fail outcome on the Rheumatology Certification Examination (odds ratio 1.016 [95% confidence interval 1.011-1.021]). CONCLUSION: The predictive value of the ACR ITE score with regard to predicting performance on the Rheumatology Certification Examination supports use of the Adult Rheumatology ITE as a valid feedback tool during fellowship training.

The relationship between performance on the Infectious Diseases In-Training and Certification Examinations.

BACKGROUND: The Infectious Diseases Society of America In-Training Examination (IDSA ITE) is a feedback tool used to help fellows track their knowledge acquisition during fellowship training. We determined whether the scores on the IDSA ITE and from other major medical knowledge assessments predict performance on the American Board of Internal Medicine (ABIM) Infectious Disease Certification Examination. METHODS: The sample was 1021 second-year fellows who took the IDSA ITE and ABIM Infectious Disease Certification Examination from 2008 to 2012. Multiple regression analysis was used to determine if ABIM Infectious Disease Certification Examination scores were predicted by IDSA ITE scores, prior United States Medical Licensing Examination (USMLE) scores, ABIM Internal Medicine Certification Examination scores, fellowship director ratings of medical knowledge, and demographic variables. Logistic regression was used to evaluate if these same assessments predicted a passing outcome on the certification examination. RESULTS: IDSA ITE scores were the strongest predictor of ABIM Infectious Disease Certification Examination scores (ß = .319), followed by prior ABIM Internal Medicine Certification Examination scores (ß = .258), USMLE Step 1 scores (ß = .202), USMLE Step 3 scores (ß = .130), and fellowship directors' medical knowledge ratings (ß = .063). IDSA ITE scores were also a significant predictor of passing the Infectious Disease Certification Examination (odds ratio, 1.017 [95% confidence interval, 1.013-1.021]). CONCLUSIONS: The significant relationship between the IDSA ITE score and performance on the ABIM Infectious Disease Certification Examination supports the use of the ITE as a valid feedback tool in fellowship training.

The impact of postgraduate training on USMLE® step 3® and its computer-based case simulation component.

BACKGROUND: The United States Medical Licensing Examination® (USMLE®) Step 3® examination is a computer-based examination composed of multiple choice questions (MCQ) and computer-based case simulations (CCS). The CCS portion of Step 3 is unique in that examinees are exposed to interactive patient-care simulations. OBJECTIVE: The purpose of the following study is to investigate whether the type and length of examinees' postgraduate training impacts performance on the CCS component of Step 3, consistent with previous research on overall Step 3 performance. DESIGN: Retrospective cohort study PARTICIPANTS: Medical school graduates from U.S. and Canadian institutions completing Step 3 for the first time between March 2007 and December 2009 (n = 40,588). METHODS: Post-graduate training was classified as either broadly focused for general areas of medicine (e.g. pediatrics) or narrowly focused for specific areas of medicine (e.g. radiology). A three-way between-subjects MANOVA was utilized to test for main and interaction effects on Step 3 and CCS scores between the demographic characteristics of the sample and type of residency. Additionally, to examine the impact of postgraduate training, CCS scores were regressed on Step 1 and Step 2 Clinical Knowledge (CK) scores. Residuals from the resulting regressions were plotted. RESULTS: There was a significant difference in CCS scores between broadly focused (µ = 216, σ = 17) and narrowly focused (µ=211, σ = 16) residencies (p < 0.001). Examinees in broadly focused residencies performed better overall and as length of training increased, compared to examinees in narrowly focused residencies. Predictors of Step 1 and Step 2 CK explained 55% of overall Step 3 variability and 9% of CCS score variability. CONCLUSIONS: Factors influencing performance on the CCS component may be similar to those affecting Step 3 overall. Findings are supportive of the validity of the Step 3 program and may be useful to program directors and residents in considering readiness to take this examination.

Gender differences in examinee performance on the Step 2 Clinical Skills data gathering (DG) and patient note (PN) components.

Multiple studies examining the relationship between physician gender and performance on examinations have found consistent significant gender differences, but relatively little information is available related to any gender effect on interviewing and written communication skills. The United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) examination is a multi-station examination where examinees (physicians in training) interact with, and are rated by, standardized patients (SPs) portraying cases in an ambulatory setting. Data from a recent complete year (2009) were analyzed via a series of hierarchical linear models to examine the impact of examinee gender on performance on the data gathering (DG) and patient note (PN) components of this examination. Results from both components show that not only do women have higher scores on average, but women continue to perform significantly better than men when other examinee and case variables are taken into account. Generally, the effect sizes are moderate, reflecting an approximately 2% score advantage by encounter. The advantage for female examinees increased for encounters that did not require a physical examination (for the DG component only) and for encounters that involved a Women's Health issue (for both components). The gender of the SP did not have an impact on the examinee gender effect for DG, indicating a desirable lack of interaction between examinee and SP gender. The implications of the findings, especially with respect to the validity of the use of the examination outcomes, are discussed.

F-type testlets and the effects of feedback and case-specificity.

BACKGROUND: A novel type of item sets, "f-type" testlets, was recently introduced on the United States Medical Licensing Examination. These testlets contain two or more questions associated with a common clinical scenario. In some cases, as the scenario unfolds, examinees are indirectly provided with feedback about their response to a testlet question. The effects of this format and of the provision of feedback to examinees about their performance are investigated. METHOD: Examinee behavior is predicted using an item response model, and observed examinee responses are compared with model expectations for f-type testlets. Mean model-data discrepancies among specific examinee groups are compared to study the dependencies across within-testlet items (i.e., case-specificity) and the impact of providing feedback. RESULTS: Findings showed that case-specificity effects were present (on average) for all examinee subgroups except examinees who both responded unsuccessfully to the initial item within an f-type testlet and received feedback. Case-specificity effects were negative for examinees who responded unsuccessfully to the initial testlet item but did not receive feedback. For those who responded successfully to the initial testlet items, case-specificity effects were positive. CONCLUSIONS: Results suggest that responses to test questions within an f-type testlet are not independent-even after accounting for examinee proficiency and item characteristics. Case-specificity effects (i.e., dependencies) were observed on average for all examinees except those who both responded unsuccessfully to the initial item within an f-type testlet and received feedback. Research into modeling these effects through the use of more general item response models is recommended.

What new residents do during their initial months of training.

BACKGROUND: Studies completed over the past decade suggest the presence of a gap between what students learn during medical school and their clinical responsibilities as first-year residents. The purpose of this survey was to verify on a large scale the responsibilities of residents during their initial months of training. METHOD: Practice analysis surveys were mailed in September 2009 to 1,104 residency programs for distribution to an estimated 8,793 first-year residents. Surveys were returned by 3,003 residents from 672 programs; 2,523 surveys met inclusion criteria and were analyzed. RESULTS: New residents performed a wide range of activities, from routine but important communications (obtain informed consent) to complex procedures (thoracentesis), often without the attending physician present or otherwise involved. CONCLUSIONS: Medical school curricula and the content of competence assessments prior to residency should consider more thorough coverage of the complex knowledge and skills required early in residency.

Colorectal cancer screening: patients' and physicians' perspectives on decision-making factors.

While patient barriers to colorectal cancer (CRC) screening have been identified, how well this knowledge is utilized during the patient-physician interaction is not fully understood. This study aims to assess among primary care physicians the degree of consensus between perceived and actual patients' CRC screening decision-making influential factors. During 2004-2006, 30 patients were interviewed to identify factors influencing screening decisions and 66 physicians were interviewed to understand what factors they thought were important to patients. The factors were categorized using the PRECEDE-PROCEED framework, and perspectives were compared. The researchers found little consensus on CRC screening decision-making influential factors between family practitioners, general internists, and patients. The recommendations to reach consensus are provided on the individual (e.g., updating the contents of a physician's screening recommendation to proactively address patients' decision-making needs) and population (e.g., providing cross-cultural training to medical students enabling them to better understand their patients) levels.

A complementary and alternative medicine workshop using standardized patients improves knowledge and clinical skills of medical students.

OBJECTIVES: As the use of complementary and alternative medicine (CAM) has increased in the general population, so has the interest in CAM education among medical students and medical educators. The purpose of this study is to determine the impact of a CAM workshop using standardized patients (SP) on knowledge and clinical skills of third-year medical students. DESIGN: A 4-hour CAM workshop was developed as part of a new curriculum for a required third-year 4-week primary care internal medicine clerkship. The CAM workshop and 3 other novel workshops were randomized for delivery to half of the rotational groups. The CAM workshop incorporates 4 SP cases representing different clinical challenges. All students in every rotation group are assigned CAM readings. At the end of the rotation, all students take a 100-item written exam (7 CAM items) and 9-station SP exam (1 CAM station) including a post-SP encounter open-ended written exercise. Scores on the written exam CAM items, CAM SP checklist, and CAM open-ended written exercise of workshop participants and nonparticipants were analyzed with simple means, standard deviations, and multiple regression approaches. RESULTS: The CAM workshop was delivered to 12 of the 24 rotation groups during the 2004-2005 and 2005-2006 academic years. Ninety-two students participated in the workshop, and 94 did not. Workshop participants performed significantly better than nonparticipants on the CAM-specific SP checklist items (58 vs 36.6%, P<.0001), post-SP encounter written exercise (76.9 vs 63.3%, P<.0001), and 7 CAM written exam items (84.8 vs 76.3%, P<.0001). CONCLUSIONS: Students participating in a 4-hour SP workshop exhibit superior CAM knowledge as assessed by SP checklist, open-ended exercises, and multiple choice items. It appears that practice with SPs assists in acqusition and application of CAM knowledge and deferential counseling skills.