Assessment Practices in Continuing Professional Development Activities in Health Professions: A Scoping Review.

CONTEXT: In continuing professional development (CPD), educators face the need to develop and implement innovative assessment strategies to adhere to accreditation standards and support lifelong learning. However, little is known about the development and validation of these assessment practices. We aimed to document the breadth and depth of what is known about the development and implementation of assessment practices within CPD activities. METHODS: We conducted a scoping review using the framework proposed by Arksey and O'Malley (2005) and updated in 2020. We examined five databases and identified 1733 abstracts. Two team members screened titles and abstracts for inclusion/exclusion. After data extraction, we conducted a descriptive analysis of quantitative data and a thematic analysis of qualitative data. RESULTS: A total of 130 studies were retained for the full review. Most reported assessments are written assessments (n = 100), such as multiple-choice items (n = 79). In 99 studies, authors developed an assessment for research purpose rather than for the CPD activity itself. The assessment validation process was detailed in 105 articles. In most cases, the authors examined the content with experts (n = 57) or pilot-tested the assessment (n = 50). We identified three themes: 1-satisfaction with assessment choices; 2-difficulties experienced during the administration of the assessment; and 3-complexity of the validation process. CONCLUSION: Building on the adage "assessment drives learning," it is imperative that the CPD practices contribute to the intended learning and limit the unintended negative consequences of assessment. Our results suggest that validation processes must be considered and adapted within CPD contexts.

Development and Validation of the Transition Readiness Assessment Instrument in Type 1 Diabetes "On TRAck".

OBJECTIVES: Transition to adult care is challenging for youth with type 1 diabetes (T1D) and their caregivers. We have developed the diabetes-specific "On TRAck" transition readiness scale, and in this study we assess its reliability and validity compared with TRANSITION-Q, a generic transition readiness questionnaire. METHODS: We systematically created 3 versions of On TRAck: adolescent, parent and health-care provider (HCP) versions (for case managers and physicians). Among adolescents 13 to 18 years of age with T1D at a single academic centre, we conducted an exploratory factor analysis and assessed interrater agreement, internal consistency and relationship with age; recent glycated hemoglobin (A1C); and recent diabetic ketoacidosis (DKA) with On TRAck and TRANSITION-Q. RESULTS: One hundred fifteen adolescents (aged 15.8±1.6 years and diabetes duration 6.7±4.1 years), their caregivers and diabetes HCPs participated. The final 24-item adolescent and parent scales (with 3 subscales: "Self-efficacy," "Autonomy" and "Support & maturity") and the 3-item HCP version had a Cronbach's alpha of 0.86 to 0.93. Adolescent scores correlated with parents (r=0.64), case managers (r=0.39) and physicians (r=0.28). Mean adolescent score was 190.3±27.1 points out of 240. Adolescent scores were 3.4 points higher per year of age (p=0.03) and 4.4 points higher for every 1% lower A1C (p=0.01), but were not associated with DKA. TRANSITION-Q was associated with age. On TRAck HCP scores were associated with adolescent's age, A1C and DKA. CONCLUSIONS: On TRAck represents a new psychometrically comprehensive diabetes-specific scale that can be used in adolescent diabetes clinics for measuring transition readiness. It is a multidimensional instrument with ease of use and high reliability scores.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles.

Introduction: Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles. Methods: We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance. Results: Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each. Discussion: Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

Self-assessment differences between genders in a low-stakes objective structured clinical examination (OSCE).

OBJECTIVE: Physicians and medical students are generally poor-self assessors. Research suggests that this inaccuracy in self-assessment differs by gender among medical students whereby females underestimate their performance compared to their male counterparts. However, whether this gender difference in self-assessment is observable in low-stakes scenarios remains unclear. Our study's objective was to determine whether self-assessment differed between male and female medical students when compared to peer-assessment in a low-stakes objective structured clinical examination. RESULTS: Thirty-three (15 males, 18 females) third-year students participated in a 5-station mock objective structured clinical examination. Trained fourth-year student examiners scored their performance on a 6-point Likert-type global rating scale. Examinees also scored themselves using the same scale. To examine gender differences in medical students' self-assessment abilities, mean self-assessment global rating scores were compared with peer-assessment global rating scores using an independent samples t test. Overall, female students' self-assessment scores were significantly lower compared to peer-assessment (p < 0.001), whereas no significant difference was found between self- and peer-assessment scores for male examinees (p = 0.228). This study provides further evidence that underestimation in self-assessment among females is observable even in a low-stakes formative objective structured clinical examination facilitated by fellow medical students.

A student-initiated objective structured clinical examination as a sustainable cost-effective learning experience.

BACKGROUND: The objective structured clinical examination (OSCE) has gained widespread use as a form of performance assessment. However, opportunities for students to participate in practice OSCEs are limited by the financial, faculty and administrative investments required. OBJECTIVES: To determine the feasibility and acceptability of a student-run mock OSCE (MOSCE) as a learning experience for medical students of all 4 years. DESIGN: We conducted a five-station MOSCE for third-year students. This involved fourth-year students as examiners and first-/second-year students as standardized patients (SPs). Each examiner scored examinees using a checklist and global rating scale while providing written and verbal feedback. MOSCE stations and checklists were designed by students and reviewed by a faculty supervisor. Following the MOSCE, participants completed surveys which elucidated their perceptions on the roles they took during the MOSCE. RESULTS: Fifty examinees participated in the MOSCE. Of these, 42 (84%) consented to participate in the study and submitted completed questionnaires. Twenty-four examiners participated in the OSCE and consented to participate in the study, with 22 (92%) submitting completed questionnaires. Fifty-three of 60 SPs (88%) agreed to take part in this study, and 51 (85%) completed questionnaires. The internal consistency of the five-station OSCE was calculated as a Cronbach's alpha of 0.443. Students commented positively on having the opportunity to network and engage in mentorship activities and reinforce clinical concepts. CONCLUSIONS: Examinees, examiners, and SPs all perceived the MOSCE to be a beneficial learning experience. We found the MOSCE to be a feasible and acceptable means of providing additional OSCE practice to students prior to higher-stakes evaluations.