Assessing commitment to reflection: perceptions of medical students.

Background: While developing reflection skills is considered important by educators, the assessment of these skills is often associated with unintended negative consequences. In the context of a mandatory longitudinal course that aims to promote the development of reflection on professional identity, we assessed students' commitment to reflection. This study explores students' perception of this assessment by their mentor. Methods: We conducted a qualitative descriptive study using semi-structured interviews with twenty-one 1st and six 2nd year medical students. Thematic analysis was informed by Braun and Clarke's six-step approach. Results: We identified four main themes: 1- assessment as a motivator, 2- consequences on authenticity, 3- perception of inherent subjectivity, and 4 - relationship with the mentor. Conclusions: In the context of assessing reflection skills in future physicians, we observed that students -when assessed on the process of reflection- experienced high motivation but were ambivalent on the question of authenticity. The subjectivity of the assessment as well as the relationship with their mentor also raises questions. Nevertheless, this assessment approach for reflective skills appears to be promising in terms of limiting the negative consequences of assessment.

Contexte: Malgré l'importance que les éducateurs attribuent à l'acquisition de compétences de réflexion, l'évaluation de ces compétences entraîne souvent des conséquences négatives involontaires. Dans le cadre d'un cours longitudinal obligatoire visant à promouvoir le développement de la réflexion sur l'identité professionnelle, nous avons évalué l'engagement des étudiants à cultiver leurs compétences de réflexion. Cette étude explore leur perception de cette évaluation menée par leur mentor. Méthodes: Nous avons réalisé une étude qualitative descriptive à l'aide d'entretiens semi-structurés avec vingt-et-un étudiants en médecine de première année et six étudiants en médecine de deuxième année. Notre analyse thématique repose sur l'approche en six étapes de Braun et Clarke. Résultats: Nous avons identifié quatre thèmes principaux : 1 ­ l'évaluation comme facteur de motivation, 2 ­ les conséquences sur l'authenticité, 3 ­ la perception de la subjectivité inhérente, et 4 ­ la relation avec le mentor. Conclusions: Dans le contexte de l'évaluation des compétences de réflexion des futurs médecins, focalisée plus particulièrement sur le processus de réflexion, les étudiants se sont montrés très motivés, mais incertains quant à son authenticité. La subjectivité de l'évaluation et la relation avec leur mentor soulèvent également des interrogations. Néanmoins, cette approche d'évaluation des compétences réflexives semble prometteuse dans la mesure où elle permet de limiter les conséquences négatives de l'évaluation.

Considering the Structured Oral Examinations Beyond Its Psychometrics Properties.

Decisions to set aside Structured Oral Examinations (SOE) are, almost invariably, based on their poor psychometric properties. However, considering the perspectives of the stakeholders might help us to understand its potential contribution. To explore this, we conducted focus groups and individual interviews with stakeholders: students, assessors, and administrators. Students and assessors perceived the SOE as a window on students' clinical reasoning, as an authentic assessment, but as a subjective and stressful method. Administrators emphasized the organizational consequences such as logistical challenges. Consequences must be considered when making decisions about SOE and our results support important positive consequences.

Quality of Narratives in Assessment: Piloting a List of Evidence-Based Quality Indicators.

Background & Need for Innovation: Appraising the quality of narratives used in assessment is challenging for educators and administrators. Although some quality indicators for writing narratives exist in the literature, they remain context specific and not always sufficiently operational to be easily used. Creating a tool that gathers applicable quality indicators and ensuring its standardized use would equip assessors to appraise the quality of narratives. Steps taken for Development and Implementation of innovation: We used DeVellis' framework to develop a checklist of evidence-informed indicators for quality narratives. Two team members independently piloted the checklist using four series of narratives coming from three different sources. After each series, team members documented their agreement and achieved a consensus. We calculated frequencies of occurrence for each quality indicator as well as the interrater agreement to assess the standardized application of the checklist. Outcomes of Innovation: We identified seven quality indicators and applied them on narratives. Frequencies of quality indicators ranged from 0% to 100%. Interrater agreement ranged from 88.7% to 100% for the four series. Critical Reflection: Although we were able to achieve a standardized application of a list of quality indicators for narratives used in health sciences education, it does not exclude the fact that users would need training to be able to write good quality narratives. We also noted that some quality indicators were less frequent than others and we suggested a few reflections on this.

Narrative Assessments in Higher Education: A Scoping Review to Identify Evidence-Based Quality Indicators.

PURPOSE: Narrative comments are increasingly used in assessment to document trainees' performance and to make important decisions about academic progress. However, little is known about how to document the quality of narrative comments, since traditional psychometric analysis cannot be applied. The authors aimed to generate a list of quality indicators for narrative comments, to identify recommendations for writing high-quality narrative comments, and to document factors that influence the quality of narrative comments used in assessments in higher education. METHOD: The authors conducted a scoping review according to Arksey & O'Malley's framework. The search strategy yielded 690 articles from 6 databases. Team members screened abstracts for inclusion and exclusion, then extracted numerical and qualitative data based on predetermined categories. Numerical data were used for descriptive analysis. The authors completed the thematic analysis of qualitative data with iterative discussions until they achieved consensus for the interpretation of the results. RESULTS: After the full-text review of 213 selected articles, 47 were included. Through the thematic analysis, the authors identified 7 quality indicators, 12 recommendations for writing quality narratives, and 3 factors that influence the quality of narrative comments used in assessment. The 7 quality indicators are (1) describes performance with a focus on particular elements (attitudes, knowledge, skills); (2) provides a balanced message between positive elements and elements needing improvement; (3) provides recommendations to learners on how to improve their performance; (4) compares the observed performance with an expected standard of performance; (5) provides justification for the mark/score given; (6) uses language that is clear and easily understood; and (7) uses a nonjudgmental style. CONCLUSIONS: Assessors can use these quality indicators and recommendations to write high-quality narrative comments, thus reinforcing the appropriate documentation of trainees' performance, facilitating solid decision making about trainees' progression, and enhancing the impact of narrative feedback for both learners and programs.

Reflecting on professional identity in undergraduate medical education: implementation of a novel longitudinal course.

BACKGROUND: Today's healthcare professionals face numerous challenges. Improving reflection skills has the potential to contribute to the better management of complex patients and healthcare systems, as well as to improve professional practice. However, the question of how reflection skills can inform professional identity development at the undergraduate medical education level remains unanswered. APPROACH: The authors developed and implemented a 4-year course that aims to engage students in a reflective process to increase their awareness of their professional identity development. The course is structured around three types of pedagogical activities: workshops, reflections deposited in an electronic portfolio, and individual discussions with mentors. EVALUATION: Sixty-four 1st year students (33%) and 17 mentors (50%) from the 2017-2018 cohort completed evaluation questionnaires. For the 2018-2019 academic year, 73 1st year students (34%) and 27 2nd year students (14%), as well as 20 1st year (59%) and 19 2nd year mentors (57%) replied. Students and mentors considered that the pedagogical activities contributed to the development of students' professional identity through the acquisition of reflection skills, but some elements were perceived as challenging, notably, completing the portfolio, finding a subject to reflect about and the timing of the proposed activities. REFLECTION: An important preoccupation when wanting to foster the development of professional identity through the acquisition of reflection skills is the authenticity of students' reflection. We tried to favor authentic reflection, by having a mentee-mentor pair throughout the entire 4­year course. A rigorous evaluation process helped us identify and promptly correct issues as they surfaced.

COVID-19 as the tipping point for integrating e-assessment in higher education practices.

The COVID-19 pandemic provoked an urgency for many educators to integrate digital information and communication technologies in their educational practices. We explored how faculty members tackled the task of adapting their assessment practices during the pandemic to identify what is required to sustain and favour future quality development and implementation of e-assessment in higher education. Employing a qualitative descriptive approach, we conducted semi-structured interviews with thirty-one individuals six months into the COVID-19 pandemic. We identified four major themes in participants' discourse about the integration of e-assessment during the COVID-19 pandemic: (a) the considerations they had for the potential consequences on students and how they considered this while deciding how to move forward, (b) the preoccupations for the potential for cheating, (c) the importance of pedagogical alignment, and (d) the affordances available to them. While the COVID-19 pandemic highlighted the fact that higher education institutions were not prepared for a pivot to- or greater integration of- e-assessment, it also provided the tipping-point to do so. In other words, it offered an unprecedented opportunity to critically appraise and change assessment practices, this opportunity was also a very challenging balancing act of considering the social consequences of assessment, and the alignment within set affordances.

Strategies identified by program directors to improve adoption of the CanMEDS framework.

BACKGROUND: Challenges associated with the use of the CanMEDS physician competency framework (CanMEDS) have been the subject of several studies. Most of these have focused on the adoption of specific roles in an Anglophone context. This study aims to investigate how Francophone postgraduate medical education (PGME) program directors have integrated the CanMEDS framework into their programs. METHODS: We invited Francophone PGME program directors to participate in group interviews aimed at exploring their experiences using the CanMEDS framework. We used an open-ended interview guide and realized a thematic analysis of the transcripts. RESULTS: We held five group interviews between February and December 2014 with 17 Francophone program directors representing 13 out of a maximum of 62 different specialties/subspecialties. Although program directors endorsed the framework, its integration was seen as challenging, particularly the assessment of non-medical expert roles. To overcome these challenges, they relied on common strategies including a longitudinal approach to the framework, improving inter-program collaboration, and subcontracting the teaching of certain roles. CONCLUSION: While integrating the CanMEDS framework into their programs, Francophone program directors struggled with teaching and assessing non-medical expert roles and ensuring their longitudinal integration over time. Directors relied on various strategies, some of which (e.g., subcontracting) may ultimately limit the adoption of the framework as a whole. ___.

CONTEXTE: Les défis associés à l'utilisation du référentiel de compétences CanMEDS pour les médecins ont fait l'objet de plusieurs études. La plupart de ces études ont portées sur l'adoption de rôles spécifiques dans un contexte anglophone. Cette étude vise à explorer comment les directeurs de programmes d'études médicales postdoctorales (EMP) francophones ont intégré CanMEDS dans leurs programmes. MÉTHODES: Nous avons invité les directeurs de programmes EMP francophones à participer à des entrevues de groupe. Ces entrevues visaient à explorer leur expérience de l'utilisation du référentiel CanMEDS. Nous avons utilisé un guide d'entrevue ouvert et nous avons fait une analyse thématique des transcriptions. RÉSULTATS: Nous avons tenu cinq entrevues de groupe entre février et décembre 2014 avec 17 directeurs de programmes de 13 des 62 spécialités/sous-spécialités. Bien que les directeurs de programmes appuient le référentiel, son intégration a été perçue comme un défi, notamment en ce qui a trait à l'évaluation des rôles autres que celui d'expert médical. Pour surmonter ces défis, ils se sont appuyés sur des stratégies communes, notamment une approche longitudinale du référentiel, l'amélioration de la collaboration entre les programmes et la sous-traitance de l'enseignement de certains rôles. CONCLUSIONS: À travers le processus d'intégration du référentiel CanMEDS, les directeurs de programmes EMP francophones ont de la difficulté à enseigner et à évaluer les rôles autres que celui d'expert médical ainsi qu'à veiller à leur intégration respective et continue au fil du temps. Ils ont eu recours à diverses stratégies, dont certaines (p. ex., la sous-traitance) pourraient ultimement limiter l'adoption du référentiel dans son ensemble.

Exploring residents' spontaneous collaborative skills in a simulated setting context: an exploratory study on CanMEDS collaborator role.

BACKGROUND: Collaboration is an important competence to be acquired by residents. Although improving residents' collaboration via interprofessional education has been investigated in many studies, little is known about the residents' spontaneous collaborative behavior. The purpose of this exploratory study was to describe how residents spontaneously collaborate. METHODS: Seven first-year residents (postgraduate year 1; three from family medicine and one each from ear, nose, and throat, obstetrics/gynecology, general surgery, and orthopedic surgery) participated in two collaborative meetings with actors performing the part of other health professionals (ie, occupational therapist, physiotherapist, nurse, or social worker). Both meetings were built around an issue or conflict with the patients' families reported by one professional. The residents were required to lead the meeting to collect proper information to reach a joint decision. Two team members analyzed the video recordings of the meetings using an emerging-theme qualitative methodology. RESULTS: Although the residents spontaneously knew how to successfully communicate with other professionals, they seemed to struggle with the patient-centered approach and the shared decision-making process. DISCUSSION: Even if the residents performed communication-wise in their collaborative role, they seemed to have perceived themselves as decision makers instead of collaborators in the joint decision process. The results of this study can inform future studies on learning strategies to improve behaviors that would more likely need attention in interprofessional education.