Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System.

Competency based medical education is developed utilizing a program of assessment that ideally supports learners to reflect on their knowledge and skills, allows them to exercise a growth mindset that prepares them for coaching and eventual lifelong learning, and can support important progression and certification decisions. Examinations can serve as an important anchor to that program of assessment, particularly when considering their strength as an independent, third-party assessment with evidence that they can predict future physician performance and patient outcomes. This paper describes the aims of the Royal College of Physicians and Surgeons of Canada's ("the Royal College") certification examinations, their future role, and how they relate to the Competence by Design model, particularly as the culture of workplace assessment and the evidence for validity evolves. For example, high-stakes examinations are stressful to candidates and focus learners on exam preparation rather than clinical learning opportunities, particularly when they should be developing greater autonomy. In response, the Royal College moved the written examination earlier in training and created an exam quality review, by a specialist uninvolved in development, to review the exam for clarity and relevance. While learners are likely to continue to focus on the examination as an important hurdle to overcome, they will be preparing earlier in training, allowing them the opportunity to be more present and refine their knowledge when discussing clinical cases with supervisors in the Transition to Practice phase. The quality review process better aligns the exam to clinical practice and can improve the educational impact of the examination preparation process.

Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance.

BACKGROUND: Traditional Objective Structured Clinical Examinations (OSCEs) are psychometrically sound but have the limitation of fragmenting complex clinical cases into brief stations. We describe a pilot study of a modified OSCE that attempts to balance a typical OSCE format with a semblance of a continuous, complex, patient case. METHODS: Two OSCE scenarios were developed. Each scenario involved a single standardized patient and was subdivided into three sequential 10 minute sections that assessed separate content areas and competencies. Twenty Canadian PGY-4 internal medicine trainees were assessed by trained examiner pairs during each OSCE scenario. Paired examiners rated participant performance independent of each other, on each section of each scenario using a validated global rating scale. Inter-rater reliabilities and Pearson correlations between ratings of the 3 sections of each scenario were calculated. A generalizability study was conducted. Participant and examiner satisfaction was surveyed. RESULTS: There was no main effect of section or scenario. Inter-rater reliability was acceptable. The g-coefficient was 0.68; four scenarios would achieve 0.80. Moderate correlations between sections of a scenario suggest a possible halo effect. The majority of examiners and participants felt that the modified OSCE provided a sense of patient continuity. CONCLUSIONS: The modified OSCE provides another approach to the assessment of clinical performance. It attempts to balance the advantages of a traditional OSCE with a sense of patient continuity.

Assessing cardiac physical examination skills using simulation technology and real patients: a comparison study.

OBJECTIVE: High-stakes assessments of doctors' physical examination skills often employ standardised patients (SPs) who lack physical abnormalities. Simulation technology provides additional opportunities to assess these skills by mimicking physical abnormalities. The current study examined the relationship between internists' cardiac physical examination competence as assessed with simulation technology compared with that assessed with real patients (RPs). METHODS: The cardiac physical examination skills and bedside diagnostic accuracy of 28 internists were assessed during an objective structured clinical examination (OSCE). The OSCE included 3 modalities of cardiac patients: RPs with cardiac abnormalities; SPs combined with computer-based, audio-video simulations of auscultatory abnormalities, and a cardiac patient simulator (CPS) manikin. Four cardiac diagnoses and their associated cardiac findings were matched across modalities. At each station, 2 examiners independently rated a participant's physical examination technique and global clinical competence. Two investigators separately scored diagnostic accuracy. RESULTS: Inter-rater reliability between examiners for global ratings (GRs) ranged from 0.75-0.78 for the different modalities. Although there was no significant difference between participants' mean GRs for each modality, the correlations between participants' performances on each modality were low to modest: RP versus SP, r = 0.19; RP versus CPS, r = 0.22; SP versus CPS, r = 0.57 (P < 0.01). CONCLUSIONS: Methodological limitations included variability between modalities in the components contributing to examiners' GRs, a paucity of objective outcome measures and restricted case sampling. No modality provided a clear 'gold standard' for the assessment of cardiac physical examination competence. These limitations need to be addressed before determining the optimal patient modality for high-stakes assessment purposes.

Assessing the relationship between cardiac physical examination technique and accurate bedside diagnosis during an objective structured clinical examination (OSCE).

BACKGROUND: Many standardized patient (SP) encounters employ SPs without physical findings and, thus, assess physical examination technique. The relationship between technique, accurate bedside diagnosis, and global competence in physical examination remains unclear. METHOD: Twenty-eight internists undertook a cardiac physical examination objective structured clinical examination, using three modalities: real cardiac patients (RP), "normal" SPs combined with related cardiac audio-video simulations, and a cardiology patient simulator (CPS). Two examiners assessed physical examination technique and global bedside competence. Accuracy of cardiac diagnosis was scored separately. RESULTS: The correlation coefficients between participants' physical examination technique and diagnostic accuracy were 0.39 for RP (P < .05), 0.29 for SP, and 0.30 for CPS. Patient modality impacted the relative weighting of technique and diagnostic accuracy in the determination of global competence. CONCLUSIONS: Assessments of physical examination competence should evaluate both technique and diagnostic accuracy. Patient modality affects the relative contributions of each outcome towards a global rating.

Does physical examination competence correlate with bedside diagnostic acumen? An observational study.

AIM: To examine the relationship between a physician's ability to examine a standardized patient (SP) and their ability to correctly identify related clinical findings created with simulation technology. METHOD: The authors conducted an observational study of 347 candidates during a Canadian national specialty examination at the end of post-graduate internal medicine training. Stations were created that combined physical examination of an SP with evaluation of a related audio-video simulation of a patient abnormality, in the domains of cardiology and neurology. Examiners evaluated a candidate's competence at performing a physical examination of an SP and their accuracy in diagnosing a related audio-video simulation. RESULTS: For the cardiology stations, the correlation between the physical examination scores and recognition of simulation abnormalities was 0.31 (p < 0.01). For the neurology stations, the correlation was 0.27 (p < 0.01). Addition of the simulations identified 18% of 197 passing candidates on the cardiology stations and 17% of 240 passing candidates on the neurology stations who were competent in their physical examination technique but did not achieve the passing score for diagnostic skills. CONCLUSIONS: Assessments incorporating SPs without physical findings may need to include other methodologies to assess bedside diagnostic acumen.