Development, Implementation, and Meta-Evaluation of a National Approach to Programmatic Assessment in Canadian Family Medicine Residency Training.

The growing international adoption of competency-based medical education has created a desire for descriptions of innovative assessment approaches that generate appropriate and sufficient information to allow for informed, defensible decisions about learner progress. In this article, the authors provide an overview of the development and implementation of the approach to programmatic assessment in postgraduate family medicine training programs in Canada, called Continuous Reflective Assessment for Training (CRAFT). CRAFT is a principles-guided, high-level approach to workplace-based assessment that was intentionally designed to be adaptable to local contexts, including size of program, resources available, and structural enablers and barriers. CRAFT has been implemented in all 17 Canadian family medicine residency programs, with each program taking advantage of the high-level nature of the CRAFT guidelines to create bespoke assessment processes and tools appropriate for their local contexts. Similarities and differences in CRAFT implementation between 5 different family medicine residency training programs, representing both English- and French-language programs from both Western and Eastern Canada, are described. Despite the intentional flexibility of the CRAFT guidelines, notable similarities in assessment processes and procedures across the 5 programs were seen. A meta-evaluation of findings from programs that have published evaluation information supports the value of CRAFT as an effective approach to programmatic assessment. While CRAFT is currently in place in family medicine residency programs in Canada, given its adaptability to different contexts as well as promising evaluation data, the CRAFT approach shows promise for application in other training environments.

Making sense of adaptive expertise for frontline clinical educators: a scoping review of definitions and strategies.

Adaptive expertise has been promoted as an emerging model of expertise in health professions education in response to the inherent complexities of patient care; however, as the concept increasingly influences the structure of professional training and practice, it creates the potential for misunderstandings of the definition and implications of adaptive expertise. To foster a common understanding of the concept, we conducted a scoping review to explore how adaptive expertise has been discussed within health professions education literature. Five databases-MedLine, PubMed, ERIC, CINAHL, and PsycINFO-were searched using the exact term "adaptive expertise", producing 212 unique articles. Fifty-eight articles met inclusion criteria. In the included articles, authors discussed the conceptual implications of adaptive expertise for health professions education, strategies for training for adaptive expertise, and research findings aimed at supporting the development of adaptive expertise or utilizing adaptive expertise as a theoretical framework. The goal of this scoping review is to establish a resource for frontline educators tasked with fostering the development of adaptive expertise in learners through education initiatives. A common understanding of adaptive expertise is essential to ensuring effective implementation in training programs.

Future practice of comprehensive care: Practice intentions of exiting family medicine residents in Canada.

OBJECTIVE: To describe exiting family medicine (FM) residents' reported practice intentions after completing a Triple C Competency-based Curriculum. DESIGN: The surveys were intended to capture residents' perceptions of FM, their perceptions of their competency-based training, and their intentions to practise FM. Entry (T1) and exit (T2) self-reported survey results were compared considering the influence of the curriculum change. Unmatched aggregate-level data were reviewed. The T1 survey was administered in the summer of 2012 and the T2 survey was administered in the spring of 2014. SETTING: Six Canadian FM residency programs across 4 provinces in Canada (Alberta, Saskatchewan, Ontario, and Quebec). PARTICIPANTS: Overall, 341 entering FM residents in 2012 responded to the T1 survey and 325 exiting FM residents completing their residency programs in spring 2014 responded to the T2 survey. MAIN OUTCOME MEASURES: Self-reported data on FM residents' future practice intentions related to comprehensive care, providing care across clinical domains and settings, and providing comprehensive care individually or in teams. RESULTS: A total of 341 (71.3%) residents responded to the T1 survey and a total of 325 (71.4%) residents responded to the T2 survey. Of these, 78.7% responded that they intended to provide comprehensive FM in multiple clinical settings in their future practices, with 70.8% indicating a comprehensive care practice with a special interest and 36.6% intending to provide care in a focused practice. Overall, 92.9% reported that they intended to work in group practice environments. Ninety percent reported they intended to work in interprofessional team practices. CONCLUSION: While an upward trend toward the practice of comprehensive care was demonstrated, findings also showed an increased trend toward providing care in focused practices. Further research is needed to better determine how FM residents understand the definition of comprehensive FM and its practice models. The survey provides an opportunity to explore questions related to practice intentions that could be helpful in work force planning. As the first study to compare entry and exit data from learners who have been exposed to a Triple C competency-based approach, this survey provides important baseline data for use by many.

Extended family medicine training: Measuring training flows at a time of substantial pedagogic change.

OBJECTIVE: To examine trends in family medicine training at a time when substantial pedagogic change is under way, focusing on factors that relate to extended family medicine training. DESIGN: Aggregate-level secondary data analysis based on the Canadian Post-MD Education Registry. SETTING: Canada. PARTICIPANTS: All Canadian citizens and permanent residents who were registered in postgraduate family medicine training programs within Canadian faculties of medicine from 1995 to 2013. MAIN OUTCOME MEASURES: Number and proportion of family medicine residents exiting 2-year and extended (third-year and above) family medicine training programs, as well as the types and numbers of extended training programs offered in 2015. RESULTS: The proportion of family medicine trainees pursuing extended training almost doubled during the study period, going from 10.9% in 1995 to 21.1% in 2013. Men and Canadian medical graduates were more likely to take extended family medicine training. Among the 5 most recent family medicine exit cohorts (from 2009 to 2013), 25.9% of men completed extended training programs compared with 18.3% of women, and 23.1% of Canadian medical graduates completed extended training compared with 13.6% of international medical graduates. Family medicine programs vary substantially with respect to the proportion of their trainees who undertake extended training, ranging from a low of 12.3% to a high of 35.1% among trainees exiting from 2011 to 2013. CONCLUSION: New initiatives, such as the Triple C Competency-based Curriculum, CanMEDS-Family Medicine, and Certificates of Added Competence, have emerged as part of family medicine education and credentialing. In acknowledgment of the potential effect of these initiatives, it is important that future research examine how pedagogic change and, in particular, extended training shapes the care family physicians offer their patients. As part of that research it will be important to measure the breadth and uptake of extended family medicine training programs.

Giving curriculum planners an edge: using entrance surveys to design family medicine education.

OBJECTIVE: To pilot a survey of family medicine residents entering residency, describing their exposure to family medicine and their perspectives related to their future intentions to practise family medicine, in order to inform curriculum planners; and to test the methodology, feasibility, and utility of delivering a longitudinal survey to multiple residency programs. DESIGN: Pilot study using surveys. SETTING: Five Canadian residency programs. PARTICIPANTS: A total of 454 first-year family medicine residents were surveyed. MAIN OUTCOME MEASURES: Residents' previous exposure to family medicine, perspectives on family medicine, and future practice intentions. RESULTS: Overall, 70% of first-year residents surveyed responded (n = 317). Although only 5 residency programs participated, respondents included graduates from each of the medical schools in Canada, as well as international medical graduates. Among respondents, 92% felt positive or strongly positive about their choice to be family physicians. Most (73%) indicated they had strong or very strong exposure to family medicine in medical school, yet more than 40% had no or minimal exposure to key clinical domains of family medicine like palliative care, home care, and care of underserved groups. Similar responses were found about residents' lack of intention to practise in these domains. CONCLUSION: Exposure to clinical domains in family medicine could influence future practice intentions. Surveys at entrance to residency can help medical school and family medicine residency planners consider important learning experiences to include in training.

Defining competency-based evaluation objectives in family medicine: professionalism.

OBJECTIVE: To develop and describe observable evaluation objectives for assessing competence in professionalism, which are grounded in the experience of practising physicians. DESIGN: Modified nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using an iterative process, the expert group defined a list of observable behaviours that are indicative of professionalism, or not, in the family medicine setting. Themes relate to professional behaviour in family medicine; specific observable behaviours are those that family physicians believe are indicative of professionalism for each theme. MAIN FINDINGS: The expert group identified 12 themes and 140 specific observable behaviours to assist in the observation and discussion of professional behaviour in family medicine workplace settings. CONCLUSION: Competency-based education literature emphasizes the importance of formative evaluation and feedback. Such feedback is particularly challenging in the domain of professionalism because of its personal nature and the potential for emotional reactions. Effective dialogue between learners and teachers begins with clear expectations and reference to descriptions of relevant, specific behaviour. This research has generated a competency-based resource to assist the assessment of professional behaviour in family medicine educational programs.

Defining competency-based evaluation objectives in family medicine: procedure skills.

OBJECTIVE: To develop evaluation objectives for assessing competence in procedure skills using a key-features approach. This was part of a multiyear project to develop competency-based evaluation objectives for Certification in Family Medicine. DESIGN: Nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using a nominal group technique, the expert group developed the general key features for procedure skills. The expert group also linked the key features to already established skill dimensions in the domain of competence, to the 4 principles of family medicine, and to the CanMEDS roles. MAIN FINDINGS: The general key features were developed after 5 iterations. Ten key features were outlined and were shown to reflect all the essential skill dimensions in the domain of competence for family medicine. The key features were linked to 2 of the 4 principles of family medicine and to 4 of the CanMEDS roles. CONCLUSION: The general key features for procedure skills were developed to assess competence in procedure skills in family medicine.

Defining competency-based evaluation objectives in family medicine: communication skills.

OBJECTIVE: To provide a pragmatic approach to the evaluation of communication skills using observable behaviours, as part of a multiyear project to develop competency-based evaluation objectives for Certification in family medicine. DESIGN: A nominal group technique was used to develop themes and subthemes and to identify positive and negative observable behaviours that demonstrate competence in communication in family medicine. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: The group used the nominal group technique to derive a list of observable behaviours that would constitute a detailed operational definition of competence in communication skills; multiple iterations were used until saturation was achieved. The group met several times a year, and membership remained unchanged during the 4 years in which the work was conducted. The iterative process was undertaken twice--once for communication with patients and once for communication with colleagues. MAIN FINDINGS: Five themes, 5 subthemes, and 106 positive and negative observable behaviours were generated. The subtheme of charting skills was defined using a key-features analysis. CONCLUSION: Communication skills were defined in terms of themes and observable behaviours. These definitions were intended to help assess family physicians' competence at the start of independent practice.

Defining competency-based evaluation objectives in family medicine: key-feature approach.

OBJECTIVE: To develop key features for priority topics previously identified by the College of Family Physicians of Canada that, together with skill dimensions and phases of the clinical encounter, broadly describe competence in family medicine. DESIGN: Modified nominal group methodology, which was used to develop key features for each priority topic through an iterative process. SETTING: The College of Family Physicians of Canada. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian family medicine context with respect to region, sex, language, community type, and experience. METHODS: The group used a modified Delphi process to derive a detailed operational definition of competence, using multiple iterations until consensus was achieved for the items under discussion. The group met 3 to 4 times a year from 2000 to 2007. MAIN FINDINGS: The group analyzed 99 topics and generated 773 key features. There were 2 to 20 (average 7.8) key features per topic; 63% of the key features focused on the diagnostic phase of the clinical encounter. CONCLUSION: This project expands previous descriptions of the process of generating key features for assessment, and removes this process from the context of written examinations. A key-features analysis of topics focuses on higher-order cognitive processes of clinical competence. The project did not define all the skill dimensions of competence to the same degree, but it clearly identified those requiring further definition. This work generates part of a discipline-specific, competency-based definition of family medicine for assessment purposes. It limits the domain for assessment purposes, which is an advantage for the teaching and assessment of learners. A validation study on the content of this work would ensure that it truly reflects competence in family medicine.