Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance.

The Accreditation Council for Graduate Medical Education's Next Accreditation System requires training programs to demonstrate that fellows are achieving competence in medical knowledge (MK), as part of a global assessment of clinical competency. Passing American Board of Internal Medicine (ABIM) certification examinations is recognized as a metric of MK competency. This study examines several in-training MK assessment approaches and their ability to predict performance on the ABIM Hematology or Medical Oncology Certification Examinations. Results of a Hematology In-Service Examination (ISE) and an Oncology In-Training Examination (ITE), program director (PD) ratings, demographic variables, United States Medical Licensing Examination (USMLE), and ABIM Internal Medicine (IM) Certification Examination were compared. Stepwise multiple regression and logistic regression analyses evaluated these assessment approaches as predictors of performance on the Hematology or Medical Oncology Certification Examinations. Hematology ISE scores were the strongest predictor of Hematology Certification Examination scores (ß = 0.41) (passing odds ratio [OR], 1.012; 95 % confidence interval [CI], 1.008-1.015), and the Oncology ITE scores were the strongest predictor of Medical Oncology Certification Examination scores (ß = 0.45) (passing OR, 1.013; 95 % CI, 1.011-1.016). PD rating of MK was the weakest predictor of Medical Oncology Certification Examination scores (ß = 0.07) and was not significantly predictive of Hematology Certification Examination scores. Hematology and Oncology ITEs are better predictors of certification examination performance than PD ratings of MK, reinforcing the effectiveness of ITEs for competency-based assessment of MK.

Developing an in-training examination for fellows: the experience of the American Society of Clinical Oncology.

The American Society of Clinical Oncology (ASCO) developed its own test -- the Medical Oncology In-Training Examination (MedOnc ITE) -- as a tool to assess trainees' knowledge of the clinical oncology subspecialty, establish consistency in educational standards across training programs, identify areas of strength and weakness in individual programs, and stimulate intraprogrammatic reading and discussion. The Accreditation Council for Graduate Medical Education Outcome Project provided additional incentive for ASCO to develop an ITE. The examination was developed in 4 years. The concept of the examination and the budget were approved by the ASCO governing board. The National Board of Medical Examiners was selected to work with ASCO. Fellowship programs were contacted to determine if they had the information technology support to hold the examination. A blueprint for the examination was developed. The test format, including the number of questions and the selection of case-based single best answers, was determined. Physician volunteers to write the questions were solicited from among program directors, various ASCO committees, and disease experts. A workshop was held to teach volunteers how to write proper case-based questions. From this pool, a smaller group of physicians was selected to develop the test and review all test questions. The final examination was developed and administered in February 2008, with scores provided to fellows and program directors in April 2008. Feedback received after the examination will be helpful for developing future MedOnc ITEs. The process ASCO went through to develop the MedOnc ITE serves as a model for other subspecialties interested in developing their own ITEs.