Towards a program of assessment for health professionals: from training into practice.

Despite multifaceted attempts to "protect the public," including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

Development of eConsult reflective learning tools for healthcare providers: a pragmatic mixed methods approach.

BACKGROUND: Electronic consultation (eConsult) programs are crucial components of modern healthcare that facilitate communication between primary care providers (PCPs) and specialists. eConsults between PCPs and specialists. They also provide a unique opportunity to use real-world patient scenarios for reflective learning as part of professional development. However, tools that guide and document learning from eConsults are limited. The purpose of this study was to develop and pilot two eConsult reflective learning tools (RLTs), one for PCPs and one for specialists, for those participating in eConsults. METHODS: We performed a four-phase pragmatic mixed methods study recruiting PCPs and specialists from two public health systems located in two countries: eConsult BASE in Canada and San Francisco Health Network eConsult in the United States. In phase 1, subject matter experts developed preliminary RLTs for PCPs and specialists. During phase 2, a Delphi survey among 20 PCPs and 16 specialists led to consensus on items for each RLT. In phase 3, we conducted cognitive interviews with three PCPs and five specialists as they applied the RLTs on previously completed consults. In phase 4, we piloted the RLTs with eConsult users. RESULTS: The RLTs were perceived to elicit critical reflection among participants regarding their knowledge and practice habits and could be used for quality improvement and continuing professional development. CONCLUSION: PCPs and specialists alike perceived that eConsult systems provided opportunities for self-directed learning wherein they were motivated to investigate topics further through the course of eConsult exchanges. We recommend the RLTs be subject to further evaluation through implementation studies at other sites.

A virtual rural medicine self-learning module for preclerkship students.

Implication Statement All Ontario medical schools have a mandatory preclerkship rural placement. Despite these mandatory placements, there is a lack of Canadian rural medicine pre-departure training for students. We describe a virtual self-learning module (SLM) aimed to enhance medical students' abilities to provide compassionate care during their rural medicine placements. This SLM improves students' knowledge about rural health issues while also encouraging exploration of rural career options. Better preparation for a rural medicine placement may lead to an improved experience and eventually an increase in the number of students pursuing rural medicine careers.

Énoncé des implications de la rechercheLe stage de pré-externat en milieu rural est obligatoire dans toutes les facultés de médecine de l'Ontario. En dépit de ces stages obligatoires, la formation que les étudiants reçoivent pour les y préparer avant leur départ est insuffisante. Nous décrivons un module virtuel d'auto-apprentissage visant à améliorer l'aptitude des étudiants à offrir des soins avec compassion dans le cadre de leurs stages en médecine rurale. Ce module enrichit leurs connaissances sur les enjeux de la santé en milieu rural au Canada tout en les encourageant à explorer les possibilités de carrière dans ce milieu. Une meilleure préparation peut améliorer la qualité de l'expérience de stage et entraîner à terme une augmentation du nombre d'étudiants qui choisiront de poursuivre une carrière en médecine rurale.

Casting Into The Future: Effectiveness of a 3D-Printed Fishhook Removal Task Trainer.

While participation in both recreational and commercial fisheries is common, it is not risk-free. Puncture wounds caused by fishhooks are commonly incurred by people who fish recreationally and commercially. Despite literature that details the challenges of treating fishhook injuries and specific techniques for fishhook removal, only a single publication focuses on teaching fishhook removal techniques to medical trainees and staff physicians. The aim of this technical report is to investigate the efficacy of using a 3D-printed task trainer for simulating and teaching fishhook removal techniques. To facilitate this, the 3D-printed Fishhook Emergency Removal Simulator (FISH-ER 3D) was designed by the Memorial University of Newfoundland (MUN) MED 3D Network and satellite research partner, Carbonear Institute for Rural Reach and Innovation by the Sea (CIRRIS). A sample of 22 medical residents and staff physicians were asked to evaluate the task trainer by way of a practical session, which was then followed by an evaluation survey. The overall realism of the 3D-printed task trainer components was ranked as "realistic" or "very realistic" by 86% of the evaluators. The majority of evaluators rated acquiring and performing various fishhook removal techniques using the simulator as "easy" or "somewhat easy". Most evaluators found that using the task trainer increased user competence and confidence with fishhook removal techniques, and 100% of the evaluators rated the task trainer as a "very valuable" or "valuable" training tool. The results of this report demonstrate support for the FISH-ER 3D as an efficacious simulator for building competence in fishhook removal techniques.

Implementing Competency-Based Medical Education in Family Medicine: A Narrative Review of Current Trends in Assessment.

BACKGROUND AND OBJECTIVES: The implementation of effective competency-based medical education (CBME) relies on building a coherent and integrated system of assessment across the continuum of training to practice. As such, the developmental progression of competencies must be assessed at all stages of the learning process, including continuing professional development (CPD). Yet, much of the recent discussion revolves mostly around residency programs. The purpose of this review is to synthesize the findings of studies spanning the last 2 decades that examined competency-based assessment methods used in family medicine residency and CPD, and to identify gaps in their current practices. METHODS: We adopted a modified form of narrative review and searched five online databases and the gray literature for articles published between 2000 and 2020. Data analysis involved mixed methods including quantitative frequency analysis and qualitative thematic analysis. RESULTS: Thirty-seven studies met inclusion criteria. Fourteen were formal evaluation studies that focused on the outcome and impact evaluation of assessment methods. Articles that focused on formative assessment were prevalent. The most common levels of educational outcomes were performance and competence. There were few studies on CBME assessment among practicing family physicians. Thematic analysis of the literature identified several challenges the family medicine educational community faces with CBME assessment. CONCLUSIONS: We recommend that those involved in health education systematically evaluate and publish their CBME activities, including assessment-related content and evaluations. The highlighted themes may offer insights into ways in which current CBME assessment practices might be improved to align with efforts to improve health care.

Implementing Competency-Based Medical Education in Family Medicine: A Scoping Review on Residency Programs and Family Practices in Canada and the United States.

BACKGROUND AND OBJECTIVES: While family medicine has been one of the first specialties to implement competency-based medical education (CBME) in residency, the nature and level of its integration with continuing professional development (CPD) is neither well understood nor well studied. The purpose of this review was to examine the current state of CBME implementation in family medicine residency and CPD programs in the North American education literature, with the aim of identifying implementation concepts and strategies that are generalizable to other medical settings to inform the design and implementation of residency training and CPD. METHODS: Using an Arksey and O'Malley six-step framework, we searched five online databases and the gray literature over the period between January 2000 through April 2017. We included full-text articles that focused on the key words CBME, residency, CPD, and family medicine. RESULTS: Of the articles reviewed, 37 met the inclusion criteria and were selected for full review. Eighty six percent of included articles focused on foundation elements related to designing competency-based curriculum and assessment strategies rather than program evaluation or other outcome measures. Only 19% of the articles were related to CPD that focused only on the implementation at the program and/or institution/organization levels. CONCLUSIONS: Given that the implementation of CBME is in its relative infancy, the pattern of implementation activities described in this scoping review reflected a limited focus on a broad range of issues related to fidelity of implementation of this complex intervention.

Outcomes in CME/CPD - Special Collection: How to make the "pyramid" a perpetuum mobile.

Continuing medical education (CME) should not be an end in itself, but as expressed in Moore's pyramid, help to improve both individual patient and ultimately community, health. However, there are numerous barriers to translation of physician competence into improvements in community health. To enhance the effect CME may achieve in improving community health the authors suggest a kick-off/keep-on continuum of medical competence, and integration of aspects of public health at all levels from planning to delivery and outcomes measurement in CME.

Self-assessment and continuing professional development: the Canadian perspective.

INTRODUCTION: Several recent studies highlight that physicians are not very accurate at assessing their competence in clinical domains when compared to objective measures of knowledge and performance. Instead of continuing to try to train physicians to be more accurate self-assessors, the research suggests that physicians will benefit from learning programs that encourage them to reflect on their clinical practice, continuously seek answers to clinical problems they face, compare their knowledge and skills to clinical practice guidelines and benchmarks, and seek feedback from peers and their health care team. METHODS: This article describes the self-assessment learning activities of the College of Family Physicians of Canada Maintenance of Proficiency program (Mainpro) and the Royal College of Physicians and Surgeons of Canada Maintenance of Certification program. (MOC) RESULTS: The MOC and the Mainpro programs incorporate several self-evaluation learning processes and tools that encourage physicians to assess their professional knowledge and clinical performance against objective measures as well as guided self-audit learning activities that encourage physicians to gather information about their practices and reflect on it individually, with peers and their health care team. Physicians are also rewarded with extra credits when they participate in either of these kinds of learning activities. DISCUSSION: In the future, practice-based learning that incorporates self-assessment learning activities will play an increasingly important role as regulators mandate that all physicians participate in continuing professional development activities. Research in this area should be directed to understanding more about reflection in practice and how we can enable physicians to be more mindful.

Evolving Alignment in International Continuing Professional Development Accreditation.

Several of the world's accreditation systems for continuing professional development (CPD) are evolving to encourage continuous improvement in the competence and performance of health care providers and in the organizations in which they provide patient care. Clinicians learn best when they can to choose from a diverse array of activities and formats that are relevant and meet their needs. Since choice and diversity are key to meeting clinicians' needs, several CPD accreditors have been engaging in deliberate, concerted efforts to identify a core set of principles that can serve as the basis for determining substantive equivalency between CPD accreditation systems. Substantive equivalency is intended to support the mobility of learners, allowing them to access accredited learning activities that are recognized by various CPD accreditation systems in a manner that maximizes the value of those accreditation systems, while minimizing the burden of adhering to their requirements. In this article, we propose a set of core principles that all CPD accreditation systems must express as the basis for determining substantive equivalency between CPD accreditation systems. The article will illustrate how five CPD accreditation systems (two in the USA, two in Canada, and one in Qatar), differing in focus (activity-based versus provider-based), context, and culture, express these values and metrics, and concludes by identifying the value of substantive equivalency for learners, medical regulators, and CPD accreditation systems.

A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology.

OBJECTIVES: To investigate whether training in writing for scholarly publication, journal editing, or manuscript peer review effectively improves educational outcomes related to the quality of health research reporting. STUDY DESIGN AND SETTING: We searched MEDLINE, Embase, ERIC, PsycINFO, and the Cochrane Library for comparative studies of formalized, a priori-developed training programs in writing for scholarly publication, journal editing, or manuscript peer review. Comparators included the following: (1) before and after administration of a training program, (2) between two or more training programs, or (3) between a training program and any other (or no) intervention(s). Outcomes included any measure of effectiveness of training. RESULTS: Eighteen reports of 17 studies were included. Twelve studies focused on writing for publication, five on peer review, and none fit our criteria for journal editing. CONCLUSION: Included studies were generally small and inconclusive regarding the effects of training of authors, peer reviewers, and editors on educational outcomes related to improving the quality of health research. Studies were also of questionable validity and susceptible to misinterpretation because of their risk of bias. This review highlights the gaps in our knowledge of how to enhance and ensure the scientific quality of research output for authors, peer reviewers, and journal editors.

Society for Academic Continuing Medical Education Intervention Guideline Series: Guideline 2, Practice Facilitation.

The Society for Academic Continuing Medical Education commissioned a study to clarify and, if possible, standardize the terminology for a set of important educational interventions. In the form of a guideline, this article describes one such intervention, practice facilitation, which is a common strategy in primary care to help practices develop capacity and infrastructure to support their ability to improve patient care. Based on a review of recent evidence and a facilitated discussion with US and Canadian experts, we describe practice facilitation, its terminology, and other important information about the intervention. We encourage leaders and researchers to consider and build on this guideline as they plan, implement, evaluate, and report practice facilitation efforts. Clear and consistent use of terminology is imperative, along with complete and accurate descriptions of interventions, to improve the use and study of practice facilitation.

The Royal College experience and plans for the maintenance of certification program.

The Royal College of Physicians and Surgeons of Canada, in 2001, implemented a mandatory maintenance of certification (MOC) program that is required for fellows to maintain membership and fellowship. Participation in the MOC program is one of the recognized pathways approved by provincial medical regulatory authorities in Canada by which specialists can demonstrate their commitment to continued competent performance in practice. This article traces the historical beginnings of the MOC program, highlighting the educational foundation and scientific evidence that influenced its philosophy, goals, and strategic priorities. The MOC program has evolved into a complex system of continuing professional development to facilitate and enable a "cultural shift'' in how we conceptualize and support the continuing professional development (CPD) of specialists. The MOC program is an educational strategy that supports a learning culture where specialists are able to design, implement and document their accomplishments from multiple learning activities to build evidence-informed practices. In the future, the MOC Program must evolve from assisting fellows to use effective educational resources "for credit" to enable fellows, leveraging a competency-based CPD model, to demonstrate their capacity to continuously improve practice. This will require innovative methods to capture learning and practice improvements in real time, integrate learning during the delivery of health care, expand automation of reporting strategies, and facilitate new sociocultural methods of emergent learning and practice change. Collectively, these directions will require a research agenda that will generate evidence for how transformative cultural change in continuing professional education of the profession can be realized.

Just-in-time information improved decision-making in primary care: a randomized controlled trial.

BACKGROUND: The "Just-in-time Information" (JIT) librarian consultation service was designed to provide rapid information to answer primary care clinical questions during patient hours. This study evaluated whether information provided by librarians to answer clinical questions positively impacted time, decision-making, cost savings and satisfaction. METHODS AND FINDING: A randomized controlled trial (RCT) was conducted between October 2005 and April 2006. A total of 1,889 questions were sent to the service by 88 participants. The object of the randomization was a clinical question. Each participant had clinical questions randomly allocated to both intervention (librarian information) and control (no librarian information) groups. Participants were trained to send clinical questions via a hand-held device. The impact of the information provided by the service (or not provided by the service), additional resources and time required for both groups was assessed using a survey sent 24 hours after a question was submitted. The average time for JIT librarians to respond to all questions was 13.68 minutes/question (95% CI, 13.38 to 13.98). The average time for participants to respond their control questions was 20.29 minutes/question (95% CI, 18.72 to 21.86). Using an impact assessment scale rating cognitive impact, participants rated 62.9% of information provided to intervention group questions as having a highly positive cognitive impact. They rated 14.8% of their own answers to control question as having a highly positive cognitive impact, 44.9% has having a negative cognitive impact, and 24.8% with no cognitive impact at all. In an exit survey measuring satisfaction, 86% (62/72 responses) of participants scored the service as having a positive impact on care and 72% (52/72) indicated that they would use the service frequently if it were continued. CONCLUSIONS: In this study, providing timely information to clinical questions had a highly positive impact on decision-making and a high approval rating from participants. Using a librarian to respond to clinical questions may allow primary care professionals to have more time in their day, thus potentially increasing patient access to care. Such services may reduce costs through decreasing the need for referrals, further tests, and other courses of action. TRIAL REGISTRATION: Controlled-Trials.com ISRCTN96823810.