Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education.

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

Current use of simulation for EPA assessment in emergency medicine.

OBJECTIVE: Approximately five years ago, the Royal College emergency medicine programs in Canada implemented a competency-based paradigm and introduced the use of Entrustable Professional Activities (EPAs) for assessment of units of professional activity to assess trainees. Many competency-based medical education (CBME) based curricula, involve assessing for entrustment through observations of EPAs. While EPAs are frequently assessed in clinical settings, simulation is also used. This study aimed to characterize the use of simulation for EPA assessment. METHODS: A study interview guide was jointly developed by all study authors and followed best practices for survey development. A national interview was conducted with program directors or assistant program directors across all the Royal College emergency medicine programs across Canada. Interviews were conducted over Microsoft Teams, interviews were recorded and transcribed, using Microsoft Teams transcribing service. Sample transcripts were analyzed for theme development. Themes were then reviewed by co-authors to ensure they were representative of the participants' views. RESULTS: A 64.7% response rate was achieved. Simulation has been widely adopted by EM training programs. All interviewees demonstrated support for the use of simulation for EPA assessment for many reasons, however, PDs acknowledged limitations and thematic analysis revealed certain themes and tensions for using simulation for EPA assessment. Thematic analysis revealed six major themes: widespread support for the use of simulation for EPA assessment, concerns regarding the potential for EPA assessment to become a "tick- box" exercise, logistical barriers limiting the use of simulation for EPA assessment, varied perceptions about the authenticity of using simulation for EPA assessment, the potential for simulation for EPA assessment to compromise learner psychological safety, and suggestions for the optimization of use of simulation for EPA assessment. CONCLUSIONS: Our findings offer insight for other programs and specialties on how simulation for EPA assessment can best be utilized. Programs should use these findings when considering using simulation for EPA assessment.

RéSUMé: OBJECTIF: Il y a environ cinq ans, les programmes de médecine d'urgence du Collège royal au Canada ont mis en place un paradigme basé sur les compétences et ont introduit l'utilisation d'activités professionnelles confiables (APC) pour l'évaluation des unités d'activité professionnelle afin d'évaluer les stagiaires. De nombreux programmes d'enseignement médical basés sur les compétences (CBME) prévoient l'évaluation des compétences par l'observation des APC. Bien que les APC soient fréquemment évaluées en milieu clinique, la simulation est également utilisée. Cette étude visait à caractériser l'utilisation de la simulation pour l'évaluation de l'APC. MéTHODES: Un guide d'entretien d'étude a été élaboré conjointement par tous les auteurs de l'étude et a suivi les meilleures pratiques en matière d'élaboration d'enquêtes. Un entretien national a été réalisé avec les directeurs de programmes ou les directeurs adjoints de tous les programmes de médecine d'urgence du Collège royal au Canada. Les entretiens ont été menés sur Microsoft Teams, enregistrés et transcrits à l'aide du service de transcription de Microsoft Teams. Les transcriptions des échantillons ont été analysées pour développer des thèmes. Les thèmes ont ensuite été revus par les co-auteurs pour s'assurer qu'ils étaient représentatifs des points de vue des participants. RéSULTATS: Un taux de réponse de 64,7 % a été obtenu. La simulation a été largement adoptée par les programmes de formation en médecine d'urgence. Toutes les personnes interrogées se sont montrées favorables à l'utilisation de la simulation pour l'évaluation de l'APE pour de nombreuses raisons. Cependant, les DP ont reconnu des limites et l'analyse thématique a révélé certains thèmes et tensions liés à l'utilisation de la simulation pour l'évaluation de l'APC. L'analyse thématique a révélé six thèmes majeurs : un appui généralisé à l'utilisation de la simulation pour l'évaluation de l'APC, inquiétudes concernant la possibilité que l'évaluation de l'APC devienne un exercice de type « case à cocher ¼, des obstacles logistiques limitant l'utilisation de la simulation pour l'évaluation de l'APC, les perceptions variées quant à l'authenticité de l'utilisation de la simulation pour l'évaluation de l'APC, le potentiel de la simulation pour l'évaluation de l'APC de compromettre la sécurité psychologique des apprenants, et des suggestions pour l'optimisation de l'utilisation de la simulation pour l'évaluation de l'APC. CONCLUSIONS: Nos résultats offrent un aperçu à d'autres programmes et spécialités sur la meilleure façon d'utiliser la simulation pour l'évaluation de l'APC. Les programmes devraient utiliser ces résultats lorsqu'ils envisagent d'utiliser la simulation pour l'évaluation de l'APC.

Learner Assessment and Program Evaluation: Supporting Precision Education.

ABSTRACT: Precision education (PE) systematically leverages data and advanced analytics to inform educational interventions that, in turn, promote meaningful learner outcomes. PE does this by incorporating analytic results back into the education continuum through continuous feedback cycles. These data-informed sequences of planning, learning, assessing, and adjusting foster competence and adaptive expertise. PE cycles occur at individual (micro), program (meso), or system (macro) levels. This article focuses on program- and system-level PE.Data for PE come from a multitude of sources, including learner assessment and program evaluation. The authors describe the link between these data and the vital role evaluation plays in providing evidence of educational effectiveness. By including prior program evaluation research supporting this claim, the authors illustrate the link between training programs and patient outcomes. They also describe existing national reports providing feedback to programs and institutions, as well as 2 emerging, multiorganization program- and system-level PE efforts. The challenges encountered by those implementing PE and the continuing need to advance this work illuminate the necessity for increased cross-disciplinary collaborations and a national cross-organizational data-sharing effort.Finally, the authors propose practical approaches for funding a national initiative in PE as well as potential models for advancing the field of PE. Lessons learned from successes by others illustrate the promise of these recommendations.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Needs Assessment Survey for Stroke Care Core Competency-Based Training for Neuroscience Nurses.

BACKGROUND: Assessment of stroke care core competency for neuroscience nurses is crucial for developing training programs to improve the quality of care for patients who have a stroke. The goal of this study was to determine the priorities for competency-based training using an importance-performance analysis of stroke care core competency among neuroscience nurses. METHOD: A cross-sectional descriptive survey was conducted using a convenience sample of 154 neuroscience nurses. Differences in importance and performance scores were analyzed with paired t tests. The training priority was determined using the importance-performance matrix. RESULTS: The overall performance score was significantly lower than the importance score. The highest priority areas of training were (a) principles of stroke care and (b) cardiovascular and respiratory effects. CONCLUSION: Competency-based continuing education is needed in implementing stroke best practices for neuroscience nurses to achieve optimal competency. [J Contin Educ Nurs. 2024;55(2):63-68.].

The Next Era of Assessment: Can Ensuring High-Quality, Equitable Patient Care Be the Defining Characteristic?

ABSTRACT: Previous eras of assessment in medical education have been defined by how assessment is done, from knowledge exams popularized in the 1960s to the emergence of work-based assessment in the 1990s to current efforts to integrate multiple types and sources of performance data through programmatic assessment. Each of these eras was a response to why assessment was performed (e.g., assessing medical knowledge with exams; assessing communication, professionalism, and systems competencies with work-based assessment). Despite the evolution of assessment eras, current evidence highlights the graduation of trainees with foundational gaps in the ability to provide high-quality care to patients presenting with common problems, and training program leaders report they graduate trainees they would not trust to care for themselves or their loved ones. In this article, the authors argue that the next era of assessment should be defined by why assessment is done: to ensure high-quality, equitable care. Assessment should place focus on demanding graduates possess the knowledge, skills, attitudes, and adaptive expertise to meet the needs of all patients and ensuring that graduates are able to do this in an equitable fashion. The authors explore 2 patient-focused assessment approaches that could help realize the promise of this envisioned era: entrustable professional activities (EPAs) and resident sensitive quality measures (RSQMs)/TRainee Attributable and Automatable Care Evaluations in Real-time (TRACERs). These examples illustrate how the envisioned next era of assessment can leverage existing and new data to provide precision education assessment that focuses on providing formative and summative feedback to trainees in a manner that seeks to ensure their learning outcomes prepare them to ensure high-quality, equitable patient outcomes.

A Theoretical Foundation to Inform the Implementation of Precision Education and Assessment.

ABSTRACT: Precision education (PE) uses personalized educational interventions to empower trainees and improve learning outcomes. While PE has the potential to represent a paradigm shift in medical education, a theoretical foundation to guide the effective implementation of PE strategies has not yet been described. Here, the authors introduce a theoretical foundation for the implementation of PE, integrating key learning theories with the digital tools that allow them to be operationalized. Specifically, the authors describe how the master adaptive learner (MAL) model, transformative learning theory, and self-determination theory can be harnessed in conjunction with nudge strategies and audit and feedback dashboards to drive learning and meaningful behavior change. The authors also provide practical examples of these theories and tools in action by describing precision interventions already in use at one academic medical center, concretizing PE's potential in the current clinical environment. These examples illustrate how a firm theoretical grounding allows educators to most effectively tailor PE interventions to fit individual learners' needs and goals, facilitating efficient learning and ultimately improving patient and health system outcomes.

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps.

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.

Use of learner-driven, formative, ad-hoc, prospective assessment of competence in physical therapist clinical education in the United States: a prospective cohort study

PURPOSE: The purpose of this project was to implement a process for learner-driven, formative, prospective, ad-hoc, entrustment assessment in Doctor of Physical Therapy clinical education. Our goals were to develop an innovative entrustment assessment tool, and then explore whether the tool detected (1) differences between learners at different stages of development and (2) differences within learners across the course of a clinical education experience. We also investigated whether there was a relationship between the number of assessments and change in performance. METHODS: A prospective, observational, cohort of clinical instructors (CIs) was recruited to perform learner-driven, formative, ad-hoc, prospective, entrustment assessments. Two entrustable professional activities (EPAs) were used: (1) gather a history and perform an examination and (2) implement and modify the plan of care, as needed. CIs provided a rating on the entrustment scale and provided narrative support for their rating. RESULTS: Forty-nine learners participated across 4 clinical experiences (CEs), resulting in 453 EPA learner-driven assessments. For both EPAs, statistically significant changes were detected both between learners at different stages of development and within learners across the course of a CE. Improvement within each CE was significantly related to the number of feedback opportunities. CONCLUSION: The results of this pilot study provide preliminary support for the use of learner-driven, formative, ad-hoc assessments of competence based on EPAs with a novel entrustment scale. The number of formative assessments requested correlated with change on the EPA scale, suggesting that formative feedback may augment performance improvement.

Meeting the AACN's Essentials for Competency-Based Education With Hands-Free Video-Recorded Skills Assessment.

BACKGROUND: In the American Association of Colleges of Nursing's updated Essentials, competency-based education (CBE) is the expected standard for nursing curricula. CBE approaches include observation, practice repetition, demonstration, and video supervision guided by a framework of competency, feedback, and self-reflection. METHOD: The promotion of skills competency among nursing students was facilitated by self-regulated learning using video skills-based competency testing. Video surveillance enabled students to practice and self-test their competency skills. RESULTS: Self-videotaping by students provided a comfortable learning environment for repeated practice and fostered clinical judgment and accountability. Students reported increased confidence, decreased anxiety, and improved performance. Students developed skills and attitudes regarding self-reflection and self-remediation. Additionally, faculty reported improved visibility, a more efficient workload, and greater achievement of student outcomes. CONCLUSION: Integrating CBE and incorporating video technology during skills testing enhances the opportunity for students to demonstrate skills competency and gain comfort in their own skills acquisition. [J Nurs Educ. 2023;62(12):733-736.].

Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment.

BACKGROUND: Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment. OBJECTIVES: We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME). METHODS: In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale. RESULTS: One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05). CONCLUSIONS: We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.

Usability of ENTRUST as an Assessment Tool for Entrustable Professional Activities (EPAs): A Mixed Methods Analysis.

OBJECTIVE: As the American Board of Surgery transitions to a competency-based model of surgical education centered upon entrustable professional activities (EPAs), there is a growing need for objective tools to determine readiness for entrustment. This study evaluates the usability of ENTRUST, an innovative virtual patient simulation platform to assess surgical trainees' decision-making skills in preoperative, intra-operative, and post-operative settings. DESIGN: This is a mixed-methods analysis of the usability of the ENTRUST platform. Quantitative data was collected using the system usability scale (SUS) and Likert responses. Analysis was performed with descriptive statistics, bivariate analysis, and multivariable linear regression. Qualitative analysis of open-ended responses was performed using the Nielsen-Shneiderman Heuristics framework. SETTING: This study was conducted at an academic institution in a proctored exam setting. PARTICIPANTS: The analysis includes n = 47 (PGY 1-5) surgical residents who completed an online usability survey following the ENTRUST Inguinal Hernia EPA Assessment. RESULTS: The ENTRUST platform had a median SUS score of 82.5. On bivariate and multivariate analyses, there were no significant differences between usability based on demographic characteristics (all p > 0.05), and SUS score was independent of ENTRUST performance (r = 0.198, p = 0.18). Most participants agreed that the clinical workup of the patient was engaging (91.5%) and felt realistic (85.1%). The most frequent heuristics represented in the qualitative analysis included feedback, visibility, match, and control. Additional themes of educational value, enjoyment, and ease-of-use highlighted participants' perspectives on the usability of ENTRUST. CONCLUSIONS: ENTRUST demonstrates high usability in this population. Usability was independent of ENTRUST score performance and there were no differences in usability identified in this analysis based on demographic subgroups. Qualitative analysis highlighted the acceptability of ENTRUST and will inform ongoing development of the platform. The ENTRUST platform holds potential as a tool for the assessment of EPAs in surgical residency programs.

Core Competencies Self-Assessment and Patient-Practitioner Orientation during the First Year of a Brazilian Orthopedic Residency / Autoavaliação de competências fundamentais e orientação médico durante o primeiro ano de residência ortopédica brasileira

Abstract Objective Training a competent physician requires to direct the resident profile of graduate students for practice activities. We sought to identify the doctor-patient relationship orientation and the self-assessment of the core competencies, which they pointed out needed to be developed. Methods All 56 orthopedic residents admitted between 2016 and 2019 participated in the present prospective observational study. The Patient Practitioner Orientation Scale (PPOS) and a self-assessment questionnaire were answered at the beginning and end of the first year of residency (R1) in Orthopedics and Traumatology. We calculated mean and standard deviation for PPOS items and scores and analyzed them through the paired t-test. Self-Assessment Questionnaire answer options were "yes" or "I need to improve it" and skills were classified in decreasing order of the frequency of "I need to improve it" responses with description of absolute number and percentage. We compared frequencies using Fisher Test. P-values < 0.05 were considered statistically significant. GraphPad Prism 8.4.3 (GraphPad Software, San Diego, CA, USA) and Microsoft Excel (Microsoft Corporation, Redmond, WA, USA) were used for statistical analysis. Results In the period between the beginning and the end of R1, the total PPOS mean score significantly decreased from 4.63 to 4.50 (p= 0.024), more biomedical-focused. Around one-third of the residents identified competencies of patient care, practice-based learning and improvement, and interpersonal and communication skills as needed to improve. Conclusions The PPOS and self-assessment activities could promote reflection practices and are possible tools for learner-centered competency assessment. Biomedical guidance tends to prevail as the training of physicians progresses, and periodic self-assessments can be worked on to build a growth mindset.

Resumo Objetivo A formação de um médico competente requer direcionar o perfil de pós-graduação residente para atividades práticas. Buscou-se identificar a orientação de relacionamento médico-paciente e a autoavaliação das competências fundamentais que eles apontaram que precisavam ser desenvolvidas. Métodos Todos os 56 residentes em ortopedia admitidos entre 2016 e 2019 participaram do presente estudo observacional prospectivo. A Escala de Orientação Médico-Paciente (Patient Practitioner Orientation Scale [PPOS, na sigla em inglês]) e um questionário de autoavaliação foram respondidos no início e no final do primeiro ano de residência (R1) em Ortopedia e Traumatologia. Calculamos o desvio médio e padrão para itens e pontuações de PPOS e os analisamos através do teste t emparelhado. As opções de resposta do Questionário de Autoavaliação foram "sim" ou "preciso melhorar" e as habilidades foram classificadas na ordem decrescente da frequência das respostas "preciso melhorar" com descrição de número absoluto e percentual. Comparamos frequências usando o teste de Fisher. Consideramos significativos valores-p < 0,05. Os programas GraphPad Prism 8.4.3 (GraphPad Software, San Diego, CA, EUA) e Microsoft Excel (Microsoft Corporation, Redmond, WA, EUA) foram utilizados para análise estatística. Resultados No período entre o início e o final do R1, a média total de PPOS diminuiu significativamente, de 4,63 para 4,50 (p= 0,024), mais focada em biomédica. Cerca de um terço dos residentes identificou competências do cuidado ao paciente, aprendizagem e melhoria baseadas na prática e habilidades interpessoais e de comunicação, como necessitando melhorar. Conclusões As atividades de PPOS e autoavaliação podem promover práticas de reflexão e são possíveis ferramentas para avaliação de competência centrada no aluno. A orientação biomédica tende a prevalecer à medida que a formação dos médicos progride e as autoavaliações periódicas podem ser trabalhadas para construir uma mentalidade de crescimento.

Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program.

BACKGROUND: Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design. METHODS: We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program's use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs. RESULTS: Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a "numbers game" residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning. CONCLUSION: Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training.

Call to Action: Leveraging American Nurses Credentialing Center Nursing Continuing Professional Development Credit for Prior Learning Assessment-Part 1.

American Nurses Credentialing Center (ANCC) Accreditation in Nursing Continuing Professional Development (NCPD) provides rigorous and objective evidence of the quality of education that supports nursing professional practice. Accredited, competency-based education (CBE) that is outcome driven demonstrates the value of education to the nurse, to the profession, and to the patient. The ANCC as a global standard setter and innovator in NCPD has led this effort with the inception of the OB-CE Model©. Further, the ANCC has created a vision for CBE to be a preferred method for educational design and learner assessment for change. The ANCC NCPD accreditation program has led the charge for academic institutions to rethink their position on CBE and acceptance of accredited NCPD as evidence of prior learning for academic credit. [J Contin Educ Nurs. 2023;54(8):341-343.].

Competency Assessment of General Surgery Trainees: A Perspective From the Global South, in a CBME-Naive Context.

OBJECTIVE: Before proceeding with local implementation of competency-based medical education-related assessment practices designed and evaluated in the Global North, we sought to challenge the assumption that this would be perceived as both necessary and acceptable in our context where training and assessment is based on a traditional, knowledge-focused approach. The aim of this study was to determine the perspectives of general surgery trainees and consultants towards the assessment of competence, how this has been achieved previously, and how it should be performed in the future at the University of Cape Town (UCT), South Africa. DESIGN: Semi-structured interviews were conducted with consultants and trainees. Interviews were transcribed and then analyzed using a Reflexive Thematic Analysis approach. SETTING AND PARTICIPANTS: Ten consultants (5 senior and 5 junior) and 10 trainees (5 South African and 5 international) from the Division of General Surgery at UCT in August 2022. RESULTS: Five unique themes were developed: (1) Assessment of competence is essential, (2) competence includes multiple domains of practice, (3) a surgeon must be able to operate, (4) previously used methods were inadequate to assess competence, and (5) frequent assessment with feedback is desired. The themes were considered in the context of Situated Learning Theory, particularly Communities of Practice and their role in the training for, and authentic assessment of, competence in general surgery trainees. CONCLUSIONS: Participants described a need to develop and implement a new competency assessment program for general surgery training in this context, which is aligned with described competency-based medical education principles. Thoughtful integration of the formative and summative use of direct observation in the workplace, with a clear emphasis on procedural ability and the provision of high-quality feedback, may enhance the successful implementation of a strategy for competency-based assessment in general surgery training programs.

Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison.

Background: Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment. Methods: EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members. Results: Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively. Conclusions: The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.

Contexte: Pour les activités professionnelles confiables (APC) qui sont rarement observées ou difficiles à évaluer, une évaluation en séance de simulation peut compléter celle en milieu de travail. Nous avons comparé le recours à une évaluation en séance de simulation pour les APC axées sur la réanimation dans trois programmes de formation médicale postdoctorale et décrit les perceptions de membres du corps professoral à propos de cette modalité d'évaluation. Méthodes: Nous avons extrait les scores et le cadre (simulation ou lieu de travail) d'évaluation des APC de 2017 à 2020 pour les résidents en médecine interne, en médecine d'urgence et en fondements chirurgicaux aux étapes de transition vers la discipline et de fondements de la discipline. Un questionnaire a été distribué aux membres des comités des compétences cliniques. Résultats: Onze pour cent des évaluations d'APC étaient faites lors de séances de simulation. Cette proportion était la même pour tous les programmes, mais dans le cadre des fondements chirurgicaux elle était différente selon qu'il s'agissait de l'étape de transition (38 %) ou de celle des fondements (4 %). Les scores de confiance différaient selon le cadre de l'évaluation uniquement pour les résidents en médecine d'urgence à l'étape de la transition (simulation : 4,82 ± 0,60; lieu de travail : 3,74 ± 0,93). Le questionnaire a été rempli par 70 % des membres des comités (n=20). Parmi ceux qui avaient eu recours à une évaluation en séance de simulation, 45 % avaient interprété les données de l'évaluation différemment de la façon dont ils interprètent les données d'évaluation en milieu de travail. Soixante-treize pour cent et 100 % d'entre eux font confiance à la simulation pour les évaluations à enjeux élevés et à faibles enjeux, respectivement. Conclusions: La proportion d'évaluations en séance de simulation pour les APC axées sur la réanimation était la même dans trois programmes de formation médicale postdoctorale. Les membres des comités de compétences cliniques n'ont pas interprété les données de ce type d'évaluation de manière uniforme. Tous les répondants font confiance à l'évaluation en séance de simulation pour les évaluations à faibles enjeux, et la plupart d'entre eux pour les évaluations à enjeux élevés. Ces données ont des implications pratiques pour l'intégration de la simulation dans les programmes d'évaluation.

A Program of Assessment Model for Point-of-Care Ultrasound Training for Pediatric Critical Care Providers: A Comprehensive Approach to Enhance Competency-Based Point-of-Care Ultrasound Training.

Point-of-care ultrasound (POCUS) is increasingly accepted in pediatric critical care medicine as a tool for guiding the evaluation and treatment of patients. POCUS is a complex skill that requires user competency to ensure accuracy, reliability, and patient safety. A robust competency-based medical education (CBME) program ensures user competency and mitigates patient safety concerns. A programmatic assessment model provides a longitudinal, holistic, and multimodal approach to teaching, assessing, and evaluating learners. The authors propose a fit-for-purpose and modifiable CBME model that is adaptable for different institutions' resources and needs for any intended competency level. This educational model drives and supports learning, ensures competency attainment, and creates a clear pathway for POCUS education while enhancing patient care and safety.