Do OSCE progress test scores predict performance in a national high-stakes examination?

CONTEXT: Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a relatively new form of assessment that is used to assess the progression of clinical skills. The purpose of this study was to establish further evidence for the use of an OSCE progress test by demonstrating an association between scores from this assessment method and those from a national high-stakes examination. METHODS: The results of 8 years' of data from an Internal Medicine Residency OSCE (IM-OSCE) progress test were compared with scores on the Royal College of Physicians and Surgeons of Canada Comprehensive Objective Examination in Internal Medicine (RCPSC IM examination), which is comprised of both a written and performance-based component (n = 180). Correlations between scores in the two examinations were calculated. Logistic regression analyses were performed comparing IM-OSCE progress test scores with an 'elevated risk of failure' on either component of the RCPSC IM examination. RESULTS: Correlations between scores from the IM-OSCE (for PGY-1 residents to PGY-4 residents) and those from the RCPSC IM examination ranged from 0.316 (p = 0.001) to 0.554 (<.001) for the performance-based component and 0.305 (p = 0.002) to 0.516 (p < 0.001) for the written component. Logistic regression models demonstrated that PGY-2 and PGY-4 scores from the IM-OSCE were predictive of an 'elevated risk of failure' on both components of the RCPSC IM examination. CONCLUSIONS: This study provides further evidence for the use of OSCE progress testing by demonstrating a correlation between scores from an OSCE progress test and a national high-stakes examination. Furthermore, there is evidence that OSCE progress test scores are predictive of future performance on a national high-stakes examination.

The historic predictive value of Canadian orthopedic surgery residents' orthopedic in-training examination scores on their success on the RCPSC certification examination.

BACKGROUND: Positive correlation between the orthopedic in-training examination (OITE) and success in the American Board of Orthopaedic Surgery examination has been reported. Canadian training programs in internal medicine, anesthesiology and urology have found a positive correlation between in-training examination scores and performance on the Royal College of Physicians and Surgeons of Canada (RCPSC) certification examination. We sought to determine the potential predictive value of the OITE scores of Canadian orthopedic surgery residents on their success on their RCPSC examinations. METHODS: A total of 118 Canadian orthopedic surgery residents had their annual OITE scores during their 5 years of training matched to the RCPSC examination oral and multiple-choice questions and to overall examination pass/fail scores. We calculated Pearson correlations between the in-training examination for each postgraduate year and the certification oral and multiple-choice questions and pass/fail marks. RESULTS: There was a predictive association between the OITE and success on the RCPSC examination. The association was strongest between the OITE and the written multiple-choice examination and weakest between the OITE and the overall examination pass/fail marks. CONCLUSION: Overall, the OITE was able to provide useful feedback to Canadian orthopedic surgery residents and their training programs in preparing them for their RCPSC examinations. However, when these data were collected, truly normative data based on a Canadian sample were not available. Further study is warranted based on a more refined analysis of the OITE, which is now being produced and includes normative percentile data based on Canadian residents.

CONTEXTE: On a signalé une corrélation positive entre l'examen intermédiaire en orthopédie (EIO) et la réussite aux examens de l'American Board of Orthopaedic Surgery. Les programmes canadiens de formation en médecine interne, en anesthésiologie et en urologie ont constaté une corrélation positive entre les notes aux EIO et les résultats aux examens du Collège royal des médecins et chirurgiens du Canada (CRMCC). Nous avons cherché à déterminer la valeur prédictive potentielle des notes des résidents en chirurgie orthopédique à l'EIO pour ce qui est de leur réussite aux examens du CRMCC. MÉTHODES: Les notes de 118 résidents en chirurgie orthopédique du Canada aux EIO pendant leurs 5 années de formation ont été comparées à leurs résultats aux examens oraux et à choix multiples, ainsi qu'à leur note globale de passage ou d'échec. Nous avons calculé les corrélations de Pearson entre les résultats à l'EIO pour chaque année de formation et les notes aux examens de certification oraux et à choix multiples, et les notes de passage ou d'échec. RÉSULTATS: Il y avait un rapport prédictif entre le résultat à l'EIO et la réussite de l'examen du CRMCC. Le rapport était le plus étroit entre les résultats à l'EIO et les résultats à l'examen écrit à choix multiples, et il était le plus faible entre les résultats à l'EIO et les notes globales de passage ou d'échec. CONCLUSION: Dans l'ensemble, l'EIO a produit une rétroaction utile pour les résidents en chirurgie orthopédique et leurs programmes de formation pour les préparer aux examens du CRMCC. Toutefois, lorsque ces données ont été recueillies, de véritables données normatives fondées sur un échantillon canadien n'étaient pas disponibles. Une étude plus poussée s'impose à partir d'une analyse plus approfondie de l'EIO; cette analyse est en voie de réalisation et comprend des données normatives percentiles sur les résidents canadiens.

Simulation in healthcare: a taxonomy and a conceptual framework for instructional design and media selection.

BACKGROUND: Simulation in healthcare lacks a dedicated framework and supporting taxonomy for instructional design (ID) to assist educators in creating appropriate simulation learning experiences. AIMS: This article aims to fill the identified gap. It provides a conceptual framework for ID of healthcare simulation. METHODS: The work is based on published literature and authors' experience with simulation-based education. RESULTS: The framework for ID itself presents four progressive levels describing the educational intervention. Medium is the mode of delivery of instruction. Simulation modality is the broad description of the simulation experience and includes four modalities (computer-based simulation, simulated patient (SP), simulated clinical immersion, and procedural simulation) in addition to mixed, hybrid simulations. Instructional method describes the techniques used for learning. Presentation describes the detailed characteristics of the intervention. The choice of simulation as a learning medium is based on a matrix of simulation relating acuity (severity) to opportunity (frequency) of events, with a corresponding zone of simulation. An accompanying chart assists in the selection of appropriate media and simulation modalities based on learning outcomes. CONCLUSION: This framework should help educators incorporate simulation in their ID efforts. It also provides a taxonomy to streamline future research and ID efforts in simulation.

Validating objectives and training in Canadian paediatrics residency training programmes.

BACKGROUND: Changing health care systems and learning environments with reduction in resident work hours raises the question: "Are we adequately training our paediatricians?" AIMS: (1) Identify clinical competencies to be acquired during paediatric residency training to enable graduates to practise as consultant paediatricians; (2) Identify gaps in preparedness during training and; (3) Review and validate competencies contained in the Royal College of Physicians and Surgeons of Canada (RCPSC) objectives of training (OTR) for paediatrics. METHODS: A questionnaire with 19 classification domains containing 92 clinical competencies was administered to RCPSC certified paediatricians who completed residency training in Canada from June 2004 to June 2008. For each competency, paediatricians were asked to indicate the importance and their degree of preparedness upon entering practice. Gap scores (GSs) between importance and preparedness were calculated. RESULTS: Response rate was 43% (187/435); 91.3% (84/92) of competencies in the RCPSC OTR were identified as important. Paediatricians felt less than adequately prepared for 25% (23/92) of competencies; 40 competencies had GSs >10%. CONCLUSIONS: The unique approach used in this study is useful in validating OTR as well as the preparation of residents in relation to OTR. The results indicate a potential need for additional training in specific competencies.

Performance of the Ebel standard-setting method in spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisted of multiple-choice questions.

PURPOSE: It aimed to know the performance of the Ebel standard-setting method in in spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisted of multiple-choice questions. Specifically followings were searched: the inter-rater agreement; the correlation between Ebel scores and item facility indices; raters' knowledge of correct answers' impact on the Ebel score; and affection of rater's specialty on theinter-rater agreement and Ebel scores. METHODS: Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. Ebel's method was applied to 203 MCQs by 49 raters. Facility indices came from 194 candidates. We computed Fleiss' kappa and the Pearson correlation between Ebel scores and item facility indices. We investigated differences in the Ebel score (correct answers provided or not) and differences between internists and other specialists with t-tests. RESULTS: Kappa was below 0.15 for facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same, whether the correct answers were provided or not. Inter-rater agreement and Ebel scores was not differentbetween internists and other specialists. CONCLUSION: Inter-rater agreement and correlations between item Ebel scores and facility indices wee consistently low; furthermore, raters' knowledge of correct answer and rater specialty had no effect on Ebel scores in the present setting.

Bortezomib-induced histiocytoid Sweet syndrome.

Bortezomib, a proteasome inhibitor approved for the treatment of multiple myeloma, has been reported to be associated with Sweet syndrome. However, careful review of the histopathology of the first reported case and our case revealed similar histologic and immunohistochemical findings (a mononuclear dermal infiltrate) and not the usual neutrophilic infiltrate of Sweet syndrome. We suggest that the dermatitis induced by bortezomib is best classified as "histiocytoid Sweet syndrome."

Assessing postgraduate trainees in Canada: are we achieving diversity in methods?

BACKGROUND: Resident evaluation is a complex and challenging task, and little is known about what assessment methods, predominate within or across specialties. AIMS: To determine the methods program directors in Canada use to assess residents and their perceptions of how evaluation could be improved. METHODS: We conducted a web-based survey of program directors from The Royal College of Physicians and Surgeons of Canada (RCPSC)-accredited training programs, to examine the use of the In-Training Evaluation Report (ITER), the use of non-ITER tools and program directors' perceived needs for improvement in evaluation methods. RESULTS: One hundred forty-nine of the eligible 280 program directors participated in the survey. ITERs were used by all but one program. Of the non-ITER tools, multiple choice questions (71.8%) and oral examinations (85.9%) were most utilized, whereas essays (11.4%) and simulations (28.2%) were least used across all specialties. Surgical specialties had significantly higher multiple choice questions and logbook utilization, whereas medical specialties were significantly more likely to include Objective Stuctured Clinical Examinations (OSCEs). Program directors expressed a strong need for national collaboration between programs within a specialty to improve the resident evaluation processes. CONCLUSIONS: Program directors use a variety of methods to assess trainees. They continue to rely heavily on the ITER, but are using other tools.

CanMEDS evaluation in Canadian postgraduate training programmes: tools used and programme director satisfaction.

CONTEXT: The Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDS framework is being incorporated into specialty education worldwide. However, the literature on how to evaluate trainees in the CanMEDS competencies remains sparse. OBJECTIVES: The goals of this study were to examine the assessment tools used and programme directors' perceptions of how well they evaluate performance of the CanMEDS roles in Canadian postgraduate training programmes. METHODS: We conducted a web-based survey of programme directors of RCPSC-accredited training programmes. The survey consisted of two questions. Question 1 was designed to establish which assessment tools were used to assess each of the CanMEDS roles. Question 2 was intended to assess programme directors' perceived satisfaction with CanMEDS evaluation in their programmes. RESULTS: A total of 149 of the eligible 280 programme directors participated in the survey. Programme directors used a variety of assessment tools to evaluate trainees in CanMEDS competencies. Programmes used more tools to evaluate the Medical Expert (mean = 4.03, standard deviation [SD] = 1.59) and Communicator (mean = 2.36, SD = 1.02) roles. Programme directors used the fewest tools for the Collaborator (mean = 1.75, SD = 1.10) and Manager (mean = 1.75, SD = 1.18) roles. More than 92% of the programmes used in-training evaluation reports to evaluate all the CanMEDS roles. Programme directors were satisfied with their evaluation of the Medical Expert role, but less so with assessment of the other CanMEDS competencies. CONCLUSIONS: This study demonstrates that Canadian postgraduate training programmes use a variety of assessment tools to evaluate the CanMEDS competencies. Programme directors are neutral or concerned about how the CanMEDS roles other than that of Medical Expert are evaluated in their programmes. Further efforts are required to establish best practice in CanMEDS evaluation.

Assessing cardiac physical examination skills using simulation technology and real patients: a comparison study.

OBJECTIVE: High-stakes assessments of doctors' physical examination skills often employ standardised patients (SPs) who lack physical abnormalities. Simulation technology provides additional opportunities to assess these skills by mimicking physical abnormalities. The current study examined the relationship between internists' cardiac physical examination competence as assessed with simulation technology compared with that assessed with real patients (RPs). METHODS: The cardiac physical examination skills and bedside diagnostic accuracy of 28 internists were assessed during an objective structured clinical examination (OSCE). The OSCE included 3 modalities of cardiac patients: RPs with cardiac abnormalities; SPs combined with computer-based, audio-video simulations of auscultatory abnormalities, and a cardiac patient simulator (CPS) manikin. Four cardiac diagnoses and their associated cardiac findings were matched across modalities. At each station, 2 examiners independently rated a participant's physical examination technique and global clinical competence. Two investigators separately scored diagnostic accuracy. RESULTS: Inter-rater reliability between examiners for global ratings (GRs) ranged from 0.75-0.78 for the different modalities. Although there was no significant difference between participants' mean GRs for each modality, the correlations between participants' performances on each modality were low to modest: RP versus SP, r = 0.19; RP versus CPS, r = 0.22; SP versus CPS, r = 0.57 (P < 0.01). CONCLUSIONS: Methodological limitations included variability between modalities in the components contributing to examiners' GRs, a paucity of objective outcome measures and restricted case sampling. No modality provided a clear 'gold standard' for the assessment of cardiac physical examination competence. These limitations need to be addressed before determining the optimal patient modality for high-stakes assessment purposes.

Assessing the relationship between cardiac physical examination technique and accurate bedside diagnosis during an objective structured clinical examination (OSCE).

BACKGROUND: Many standardized patient (SP) encounters employ SPs without physical findings and, thus, assess physical examination technique. The relationship between technique, accurate bedside diagnosis, and global competence in physical examination remains unclear. METHOD: Twenty-eight internists undertook a cardiac physical examination objective structured clinical examination, using three modalities: real cardiac patients (RP), "normal" SPs combined with related cardiac audio-video simulations, and a cardiology patient simulator (CPS). Two examiners assessed physical examination technique and global bedside competence. Accuracy of cardiac diagnosis was scored separately. RESULTS: The correlation coefficients between participants' physical examination technique and diagnostic accuracy were 0.39 for RP (P < .05), 0.29 for SP, and 0.30 for CPS. Patient modality impacted the relative weighting of technique and diagnostic accuracy in the determination of global competence. CONCLUSIONS: Assessments of physical examination competence should evaluate both technique and diagnostic accuracy. Patient modality affects the relative contributions of each outcome towards a global rating.

Does physical examination competence correlate with bedside diagnostic acumen? An observational study.

AIM: To examine the relationship between a physician's ability to examine a standardized patient (SP) and their ability to correctly identify related clinical findings created with simulation technology. METHOD: The authors conducted an observational study of 347 candidates during a Canadian national specialty examination at the end of post-graduate internal medicine training. Stations were created that combined physical examination of an SP with evaluation of a related audio-video simulation of a patient abnormality, in the domains of cardiology and neurology. Examiners evaluated a candidate's competence at performing a physical examination of an SP and their accuracy in diagnosing a related audio-video simulation. RESULTS: For the cardiology stations, the correlation between the physical examination scores and recognition of simulation abnormalities was 0.31 (p < 0.01). For the neurology stations, the correlation was 0.27 (p < 0.01). Addition of the simulations identified 18% of 197 passing candidates on the cardiology stations and 17% of 240 passing candidates on the neurology stations who were competent in their physical examination technique but did not achieve the passing score for diagnostic skills. CONCLUSIONS: Assessments incorporating SPs without physical findings may need to include other methodologies to assess bedside diagnostic acumen.

Admission factors associated with international medical graduate certification success: a collaborative retrospective review of postgraduate medical education programs in Ontario.

BACKGROUND: The failure rate on certification examinations of The College of Family Physicians of Canada (CFPC) and the Royal College of Physicians and Surgeons of Canada (RCPSC) is significantly higher for international medical graduates than for Canadian medical school graduates. The purpose of the current study was to generate evidence that supports or refutes the validity of hypotheses proposed to explain the lower success rates. METHODS: We conducted retrospective analyses of admissions and certification data to determine the factors associated with success of international medical graduate residents on the certification examinations. International medical graduates who entered an Ontario residency program between 2005 and 2012 and had written a certification examination by the time of the analysis (2015) were included in the study. Data available at the time of admission for each resident, including demographic characteristics, previous experiences and previous professional experiences, were collected from each of the 6 Ontario medical schools and matched with certification examination results provided by The CFPC and the RCPSC. We developed logistic regression models to determine the association of each factor with success on the examinations. RESULTS: Data for 900 residents were analyzed. The models revealed resident age to be strongly associated with performance across all examinations. Fluency in English, female sex and the Human Development Index value associated with the country of medical school training had differential associations across the examinations. INTERPRETATION: The findings should contribute to an improved understanding of certification success by international medical graduates, help residency programs identify at-risk residents and underpin the development of specific educational and remedial interventions. In considering the results, it should be kept in mind that some variables are not amenable to changes in selection criteria.

Incorporating simulation technology in a canadian internal medicine specialty examination: a descriptive report.

High-stakes assessment of clinical performance through the use of standardized patients (SPs) is limited by the SP's lack of real physical abnormalities. The authors report on the development and implementation of physical examination stations that combine simulation technology in the form of digitized cardiac auscultation videos with an SP assessment for the 2003 Royal College of Physicians and Surgeons of Canada's Comprehensive Objective Examination in Internal Medicine. The authors assessed candidates on both the traditional stations and the stations that combined the traditional SP examination with the digitized cardiac auscultation video. For the combined stations, candidates first completed a physical examination of the SP, watched and listened to a computer simulation, and then described their auscultatory findings. The candidates' mean scores for both types of stations were similar, as were the mean discrimination indices for both types of stations, suggesting that the combined stations were of a testing standard similar to the traditional stations. Combining an SP with simulation technology may be one approach to the assessment of clinical competence in high-stakes testing situations.

Tretinoin and the prevention of keratinocyte carcinoma (Basal and squamous cell carcinoma of the skin): a veterans affairs randomized chemoprevention trial.

Keratinocyte carcinoma (KC) is the most common cancer in the United States, with no proven means for prevention other than systemic retinoids, which have significant toxicity, and sunscreen. Topical tretinoin has been used for KC chemoprevention, although this use is unproven. Hence, we conducted the randomized Veterans Affairs Topical Tretinoin Chemoprevention Trial of high-dose topical tretinoin for KC prevention. We randomized 1,131 patients to topical 0.1% tretinoin or a matching vehicle control for 1.5-5.5 years. The primary outcomes were time to development of new basal cell carcinoma (BCC) and new invasive squamous cell carcinoma (SCC) on the face or ears. The effects were not significant (P=0.3 for BCC and P=0.4 for SCC). The proportions of the tretinoin and control groups who developed a BCC at 5 years were 53 and 54% and an invasive SCC at 5 years were 28 and 31%. These differences (95% confidence intervals) were: for BCC, 1.0% (-6.5, 8.6%); for SCC, 3.6% (-3.1, 10.3%). No differences were observed in any cancer-related end points or in actinic keratosis counts. The only quality of life difference was worse symptoms in the tretinoin group at 12 months after randomization. This trial in high-risk patients demonstrates that high-dose topical tretinoin is ineffective at reducing risk of KCs.

Development and validation of a cardiac findings checklist for use with simulator-based assessments of cardiac physical examination competence.

INTRODUCTION: Objective outcome measures for use with simulator-based assessments of cardiac physical examination competence are lacking. The current study describes the development and validation of an approach to scoring performance using a cardiac findings checklist. METHODS: A cardiac findings checklist was developed and implemented for use with a simulator-based assessment of cardiac physical examination competence at a Canadian national specialty examination in internal medicine. Candidate performance as measured using the checklist was compared with global ratings of clinical performance on the cardiac patient simulator and with overall examination performance. RESULTS: Interrater reliability for scoring the checklist ranged from 0.95 for scoring correct findings to 0.72 for scoring incorrect findings. A summary checklist score had a Pearson correlation of 0.60 with overall candidate performance on the simulator-based station. CONCLUSION: Use of a cardiac findings checklist provides one objective measure of cardiac physical examination competence that may be used with simulator-based assessments.

Topical tretinoin therapy and all-cause mortality.

OBJECTIVE: To evaluate the relation of topical tretinoin, a commonly used retinoid cream, with all-cause mortality in the Veterans Affairs Topical Tretinoin Chemoprevention Trial (VATTC). The planned outcome of this trial was risk of keratinocyte carcinoma, and systemic administration of certain retinoid compounds has been shown to reduce risk of this cancer but has also been associated with increased mortality risk among smokers. DESIGN: The VATTC Trial was a blinded randomized chemoprevention trial, with 2- to 6-year follow-up. Oversight was provided by multiple independent committees. SETTING: US Department of Veterans Affairs medical centers. Patients A total of 1131 veterans were randomized. Their mean age was 71 years. Patients with a very high estimated short-term risk of death were excluded. Interventions Application of tretinoin, 0.1%, or vehicle control cream twice daily to the face and ears. MAIN OUTCOME MEASURES: Death, which was not contemplated as an end point in the original study design. RESULTS: The intervention was terminated 6 months early because of an excessive number of deaths in the tretinoin-treated group. Post hoc analysis of this difference revealed minor imbalances in age, comorbidity, and smoking status, all of which were important predictors of death. After adjusting for these imbalances, the difference in mortality between the randomized groups remained statistically significant. CONCLUSIONS: We observed an association of topical tretinoin therapy with death, but we do not infer a causal association that current evidence suggests is unlikely.