Toward competency based continuing professional development for practicing surgeons.

BACKGROUND: There has been a recent call for formal competency assessments of practicing physicians and surgeons to form a framework of competency based continuing professional development (CBCPD). METHODS: An email questionnaire was conducted regarding CBCPD. Responses were further used to inform development of semi-structured interviews. RESULTS: There were 58 questionnaire respondents (42%). There was moderate support for assessment of surgeons' technical skills (50.9%) or decision making (56.6%). Support was highest for a mechanism to flag surgeons in need of a focused competence assessment (83.0%). Eight surgeons participated in interviews. Interviews identified a range of benefits of CBCPD but also several challenges to implementation, including the need for fair, data-driven assessments, taking into account patient outcomes. CONCLUSIONS: Through listening to surgeon concerns and recommendations for implementation strategies, this study's findings may support development of an effective CBCPD strategy with the potential to be embraced by surgeons and foster an environment of improved safety and performance.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Setting standards of performance expected in neurosurgery residency: A study on entrustable professional activities in competency-based medical education.

BACKGROUND: Competency-based medical education requires evaluations of residents' performances of tasks of the discipline (ie. entrustable professional activities (EPAs)). Using neurosurgical Faculty perspectives, this study investigated whether a sample of neurosurgical EPAs accurately reflected the expectations of general neurosurgical practice. METHOD: A questionnaire was sent to all Canadian neurosurgery Faculty using a SurveyMonkey® platform. RESULTS: The proportion of respondents who believed the EPAs were representative of general neurosurgery competences varied significantly across all EPAs [47%-100%] (p < 0.0001). For 9/15 proposed EPAs, ≥75% agreed they were appropriate for general neurosurgery training and expected residents to attain the highest standard of performance. However, a range of 27-53% of the respondents felt the other six EPAs would be more appropriate for fellowship training and thus, require a lower standard of performance from graduating residents. CONCLUSION: The shift towards subspecialization in neurosurgery has implications for curriculum design, delivery and certification of graduating residents.

A propósito del artículo Integración de la dimensión ambiental en la formación académica / About the article Integration of the Environmental Dimension in Academic Training

La Revista Cubana de Salud Pública en su vol. 44(4) de 2018 publicó al artículo titulado Integración de la dimensión ambiental en la formación académica en hospitales del municipio de Holguín. Este trabajo tenía como objetivo proponer un sistema de indicadores para evaluar la eficacia de la integración de la dimensión ambiental en la formación de los recursos humanos en hospitales holguineros. Lo que permitió a sus autores concluir que este tipo de integración constituye una herramienta metodológica efectiva para mejorar el desempeño de la organización, asegurar...(AU)

Aligning Requirements of Training and Assessment in Radiation Treatment Planning in the Era of Competency-Based Medical Education.

PURPOSE: Radiation treatment planning (RTP) is a unique skill that requires interdisciplinary collaboration among radiation oncologists (ROs), dosimetrists, and medical physicists (MP) to train and assess residents. With the adoption of competency-based medical education (CBME) in Canada, it is essential residency program curricula focuses on developing competencies in RTP to facilitate entrustment. Our study investigates how radiation oncology team members' perspectives on RTP education align with requirements of the CBME approach, and its implications for improving residency training. METHODS AND MATERIALS: This qualitative research study took place in the Department of Oncology at a midsize academic institution. Through convenience sampling, focus groups were conducted with radiation oncologists (n = 11), dosimetrists (n = 7), medical physicists (n = 7), and residents (n = 7). Thematic design was adopted to analyze the transcripts through open coding resulting in 3 overarching themes. RESULTS: The results identified existing strengths and weaknesses of the residency program and future opportunities to redesign the curriculum and assessment process within a CBME model. Three overarching themes emerged from the analysis: (1) the strengths of RTP in the CBME environment; (2) challenges of RTP in CBME; and (3) opportunities for change. Stakeholders were optimistic CBME will help enrich resident learning with the increased frequency and quality of competency-based assessments. Participants suggested building a library of cases and developing computer-based learning resources to provide a safe environment to develop skills in contouring, dosimetry, and plan evaluation, in accordance with CBME training. CONCLUSIONS: This study identified future opportunities to redesign the RTP curriculum and assessment process within a CBME model. The need for innovative teaching and learning strategies, including case libraries, computer-based learning, and quality assessments, were highlighted in designing an innovative RTP planning curriculum.

Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate?

PURPOSE: Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method. METHOD: In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment. RESULTS: Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74). CONCLUSIONS: EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.

The Resident Report Card: A Tool for Operative Feedback and Evaluation of Technical Skills.

BACKGROUND: Competency-based medical education surgical curriculums will require frequent, recorded trainee performance evaluations. It is our hypothesis that written feedback after each operation can be used to chart surgical progress, can identify underperforming trainees, and will prove beneficial for resident learning. METHODS: The resident report card (RRC) is an online, easy-to-use evaluation tool designed to facilitate the creation and distribution of resident technical assessments. RRC data were collected from urologic trainees and analyzed using ANOVA and post hoc testing to confirm our hypothesis. A standardized survey was sent to residents, gauging their views on the RRC. RESULTS: Over a 5-y period, 958 RRCs with the resident listed as the primary operator were collected across 29 different procedures. Resident cohort and individual performance scores stratified by postgraduate year (PGY) were shown to significantly improve when all procedures (cohort, 6.5 ± 1.9 [PGY-1] to 9.1 ± 1.0 [PGY-5]; individual [resident M], 8.8 ± 1.8 [PGY-3] to 9.4 ± 0.7 [PGY-5], P < 0.01) and specific procedures (laparoscopic donor nephrectomy: cohort, 7.3 ± 1.3 [PGY-3] to 8.9 ± 1.0 [PGY-5]; individual [resident I], 7.2 ± 1.3 [PGY-3] to 9.5 ± 0.6 [PGY-5], P < 0.01) were analyzed. Individual residents were able to be compared to their own peer group and to the average scores across all evaluated residents. Surveyed residents were overwhelmingly positive about the RRC. CONCLUSIONS: The RRC adds further evidence to the fact that standardized, formative, and timely assessment can capture trainee performance over time and against comparator cohorts in an acceptable format to residents and academic training programs.

Judging residents' performance: a qualitative study using grounded theory.

BACKGROUND: Although program directors judge residents' performance for summative decisions, little is known about how they do this. This study examined what information program directors use and how they value this information in making a judgment of residents' performance and what residents think of this process. METHODS: Sixteen semi-structured interviews were held with residents and program directors from different hospitals in the Netherlands in 2015-2016. Participants were recruited from internal medicine, surgery and radiology. Transcripts were analysed using grounded theory methodology. Concepts and themes were identified by iterative constant comparison. RESULTS: When approaching semi-annual meetings with residents, program directors report primarily gathering information from the following: assessment tools, faculty members and from their own experience with residents. They put more value on faculty's comments during meetings and in the corridors than on feedback provided in the assessment tools. They are influenced by their own beliefs about learning and education in valuing feedback. Residents are aware that faculty members discuss their performance in meetings, but they believe the assessment tools provide the most important proof to demonstrate their clinical competency. CONCLUSIONS: Residents think that feedback in the assessment tools is the most important proof to demonstrate their performance, whereas program directors scarcely use this feedback to form a judgment about residents' performance. They rely heavily on remarks of faculty in meetings instead. Therefore, residents' performance may be better judged in group meetings that are organised to enhance optimal information sharing and decision making about residents' performance.

Microsurgery Training Resource Variation among US Integrated Plastic Surgery Residency Programs.

BACKGROUND: Given emerging focus on competency-based surgical training and work-hour limitations, surgical skills laboratories play an increasingly important role in resident education. This study was designed to investigate educational opportunities in microsurgery across integrated residency programs. METHODS: Senior residents (PGY 4-6) at integrated plastic surgery programs were surveyed during the 2016 to 2017 academic year to determine each program's access to: training microscopes and anastomosis models, video-based skills assessment, pre-requisite skills exams, flap courses, or a formal microsurgical training curriculum. Programs were stratified based on large size (>18 residents) and presence of microsurgery fellows. Chi-squared analysis was performed with p < 0.05 to assess statistical significance. RESULTS: Survey responses were collected from 32 of 60 eligible programs (53% response rate). Sixty-nine percent provide access to one to two training microscopes, 25% provide three or more, and 6% provide none. Sixty-nine percent of programs train anastomosis with nonliving prosthetics, 66% with living biologics, and 50% with nonliving biologics. Large program size or having microsurgical fellows was not associated with increased access to training microscopes or specific anastomosis models. Programs without microsurgery fellows reported more often that a formal microsurgery curriculum would be helpful (90 vs. 58% of programs with fellows, p = 0.0003). Respondents who indicated that creating a formal curriculum would not be helpful elaborated that their program already has a formal curriculum or a high volume of microsurgery cases. CONCLUSION: This study demonstrates the current variation in microsurgery training at integrated plastic surgery residency programs. A formal microsurgical training curriculum is commonly viewed as being helpful, particularly at programs without microsurgery fellows.

Entrustable Professional Activities: Una propuesta innovadora para la evaluación de competencias médicas / Entrustable professional activities: a new proposal for the evaluation of the medical competencies

Medical education migrated from a practice-based to a knowledge-based discipline after the publication of the Flexner Report. The emergence of competence-based medical education led to a greater standardization of teaching, allowing students to integrate knowledge, skills and attitudes for the execution of a given task. A challenge is the evaluation of learning. Complex evaluation systems and a consequent atomization that independently assesses different competence components. However, the evaluation carried out at the clinical practice sites allows assessing the overall level of learning. Supervisors observe students' performance and decide if the apprentice can execute a specific task independently. This decision is based upon the trust that the tutor places on the student. Consequently, Ten Cate (2005) proposed the term Entrustable Professional Activities (EPAs), as a framework for professional practice tasks or responsibilities that can be fully entrusted to students, when they demonstrate the competences that are necessary to execute such activity with an increasing level of autonomy.

[Development of a national competency-based learning objective catalogue for undergraduate medical education in Germany]. / Entwicklung eines kompetenzbasierten Lernzielkatalogs Nuklearmedizin für das Studium der Humanmedizin in Deutschland.

AIMS: Implementation of the guidelines on the Competency-based Learning Objective Catalogue for Undergraduate Medical Education for a Nuclear Medicine curriculum on behalf of the committee on professional training and continuing education of the German Association of Nuclear Medicine (DGN) METHODS:: In 7 domains 100 learning objectives (LOs) were subject to a prioritization in 3 categories (A, B and C) by means of a questionnaire as part of a Delphi method, in collaboration with all members of the DGN holding a "venia legendi" as experts. Category A defined the essential LOs for each medical practitioner. The prioritization was made by ranking the frequency of the A-classifications. In the 2nd step of the Delphi method, a list of LOs with the ranking positions 1-5 in each domain was presented to the first round's experts as a core curriculum, asking either for acceptance or modifications. RESULTS: The results of the 1st step of the Delphi method deliver a return rate of 29% of the questionnaires (55 out of 184). The 2nd round shows a return rate of 30.9% (57 out of 184) and full approval of the proposed LOs in all LO domains by in median 72 % of the experts consulted (61%-81%). The present final version contains 37 competency-based LOs in the LO domains "legal basis and radiation protection", "basic science", indications and contra-indications for "PET/CT", "scintigraphy and SPECT", "patient preparation", "image interpretation" as well as "therapy". CONCLUSION: The Competency-based Learning Objective Catalogue for Nuclear Medicine describes the knowledge and competencies, every physician should have at the end of his medical studies. The LO catalogue is a living document, which needs to be adapted continuously to the progress of the medical and technological development.

The current state of resident trauma training: Are we losing a generation?

SUMMARY: General surgeons provide life-saving trauma care to a large portion of Canadians. Although trauma care has evolved significantly over the last few decades and now requires fewer operations, when a life-saving operation is required the expectation of competence to perform this operation has not been reduced. A recent study from the United States found decreased resident case-log volumes of trauma operations. Such findings raise the alarm of declining exposure of residents to trauma operations and beg the question of whether graduating residents are competent to care for trauma patients. Examination of the Canadian setting reveals a dearth of published information about the actual exposure of Canadian general surgery residents to trauma care. With the forthcoming evolution of general surgery education into competency-based medical education, we sound a call to action to ensure that all graduating general surgeons are able to provide the care that both the Royal College of Physicians and Surgeons of Canada and the Canadian public demand.

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure.

BACKGROUND: The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum. METHODS: Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module 'Guided Vaginal Cuff Closure' six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach's alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups' method. RESULTS: The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach's alpha). The experienced surgeons' mean composite score for all six repetitions were significantly better than the novice surgeons' (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives). CONCLUSION: Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy.

BACKGROUND: Specific assessment tools can accelerate trainees' learning through structured feedback and ensure that trainees attain the knowledge and skills required to practice as competent, independent surgeons (competency-based surgical education). The objective was to develop an assessment tool for video-assisted thoracoscopic surgery (VATS) lobectomy by achieving consensus within an international group of VATS experts. METHOD: The Delphi method was used as a structured process for collecting and distilling knowledge from a group of internationally recognized VATS experts. Opinions were obtained in an iterative process involving answering repeated rounds of questionnaires. Responses to one round were summarized and integrated into the next round of questionnaires until consensus was reached. RESULTS: Thirty-one VATS experts were included and four Delphi rounds were conducted. The response rate for each round were 68.9% (31/45), 100% (31/31), 96.8% (30/31), and 93.3% (28/30) for the final round where consensus was reached. The first Delphi round contained 44 items and the final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with rating anchors: (1) localization of tumor and other pathological tissue, (2) dissection of the hilum and veins, (3) dissection of the arteries, (4) dissection of the bronchus, (5) dissection of lymph nodes, (6) retrieval of lobe in bag, (7) respect for tissue and structures, and (8) technical skills in general. CONCLUSION: A novel and dedicated assessment tool for VATS lobectomy was developed based on VATS experts' consensus. The VATSAT can support the learning of VATS lobectomy by providing structured feedback and help supervisors make the important decision of when trainees have acquired VATS lobectomy competencies for independent performance.

Development and validation of a written credentialing examination for overseas-educated dietitians.

AIM: Health professionals seeking employment in foreign countries are commonly required to undertake competency assessment in order to practice. The present study aims to outline the development and validation of a written examination for Dietetic Skills Recognition (DSR), to assess the knowledge, skills, capabilities and professional judgement of overseas-educated dietitians against the competency standards applied to dietetic graduates in Australia. METHODS: The present study reviews the design, rationale, validation and outcomes of a multiple choice question (MCQ) written examination for overseas-educated dietitians based on 5 years of administration. The validity of the exam is evaluated using Messick's validity framework, which focuses on five potential sources of validity evidence-content, internal structure, relationships with other variables, response process and consequences. The reference point for the exam pass mark or "cutscore" is the minimum standard required for safe practice. RESULTS: In total, 114 candidates have completed the MCQ examination at least once, with an overall pass rate of 52% on the first attempt. Pass rates are higher from countries where dietetic education more closely reflects the Australian model. While the pass rate for each exam tends to vary with each cohort, the cutscore has remained relatively stable over eight administrations. CONCLUSIONS: The findings provide important data supporting the validity of the MCQ exam. A more complete evaluation of the validity of the exam must be sought within the context of the whole DSR program of assessment. The DSR written component may serve as a model for use of the MCQ format for dietetic and other professional credentialing organisations.

How Do Thresholds of Principle and Preference Influence Surgeon Assessments of Learner Performance?

OBJECTIVE: The present study asks whether intraoperative principles are shared among faculty in a single residency program and explores how surgeons' individual thresholds between principles and preferences might influence assessment. BACKGROUND: Surgical education continues to face significant challenges in the implementation of intraoperative assessment. Competency-based medical education assumes the possibility of a shared standard of competence, but intersurgeon variation is prevalent and, at times, valued in surgical education. Such procedural variation may pose problems for assessment. METHODS: An entire surgical division (n = 11) was recruited to participate in video-guided interviews. Each surgeon assessed intraoperative performance in 8 video clips from a single laparoscopic radical left nephrectomy performed by a senior learner (>PGY5). Interviews were audio recorded, transcribed, and analyzed using the constant comparative method of grounded theory. RESULTS: Surgeons' responses revealed 5 shared generic principles: choosing the right plane, knowing what comes next, recognizing normal and abnormal, making safe progress, and handling tools and tissues appropriately. The surgeons, however, disagreed both on whether a particular performance upheld a principle and on how the performance could improve. This variation subsequently shaped their reported assessment of the learner's performance. CONCLUSIONS: The findings of the present study provide the first empirical evidence to suggest that surgeons' attitudes toward their own procedural variations may be an important influence on the subjectivity of intraoperative assessment in surgical education. Assessment based on intraoperative entrustment may harness such subjectivity for the purpose of implementing competency-based surgical education.

Desarrollo de una matriz conceptual para la innovación curricular en profesiones de la salud / Concept frame for curricular innovation in health care schools

We herein describe the conceptual dimension of the curricular innovation process carried out in the Faculty of Medicine of the University of Chile. We describe the context of innovation. The theoretical pertinence and relevance of a competence driven curriculum for health care professionals is discussed. The epistemological, ontological and didactic dimensions of the curricular innovation are examined. A main issue is the notion of competence and its significance in professional training. The curriculum is essentially considered as a moral endeavor, especially for health care professionals and their quest to improve the quality of life of the population.