Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and practice.

INTRODUCTION: Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. METHODS: After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. RESULTS AND DISCUSSION: A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.

Defining competency-based evaluation objectives in family medicine: professionalism.

OBJECTIVE: To develop and describe observable evaluation objectives for assessing competence in professionalism, which are grounded in the experience of practising physicians. DESIGN: Modified nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using an iterative process, the expert group defined a list of observable behaviours that are indicative of professionalism, or not, in the family medicine setting. Themes relate to professional behaviour in family medicine; specific observable behaviours are those that family physicians believe are indicative of professionalism for each theme. MAIN FINDINGS: The expert group identified 12 themes and 140 specific observable behaviours to assist in the observation and discussion of professional behaviour in family medicine workplace settings. CONCLUSION: Competency-based education literature emphasizes the importance of formative evaluation and feedback. Such feedback is particularly challenging in the domain of professionalism because of its personal nature and the potential for emotional reactions. Effective dialogue between learners and teachers begins with clear expectations and reference to descriptions of relevant, specific behaviour. This research has generated a competency-based resource to assist the assessment of professional behaviour in family medicine educational programs.

Defining competency-based evaluation objectives in family medicine: procedure skills.

OBJECTIVE: To develop evaluation objectives for assessing competence in procedure skills using a key-features approach. This was part of a multiyear project to develop competency-based evaluation objectives for Certification in Family Medicine. DESIGN: Nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using a nominal group technique, the expert group developed the general key features for procedure skills. The expert group also linked the key features to already established skill dimensions in the domain of competence, to the 4 principles of family medicine, and to the CanMEDS roles. MAIN FINDINGS: The general key features were developed after 5 iterations. Ten key features were outlined and were shown to reflect all the essential skill dimensions in the domain of competence for family medicine. The key features were linked to 2 of the 4 principles of family medicine and to 4 of the CanMEDS roles. CONCLUSION: The general key features for procedure skills were developed to assess competence in procedure skills in family medicine.

Effect of field notes on confidence and perceived competence: survey of faculty and residents.

OBJECTIVE: To evaluate the effectiveness of field notes in assessing teachers' confidence and perceived competence, and the effect of field notes on residents' perceptions of their development of competence. DESIGN: A faculty and resident survey completed 5 years after field notes were introduced into the program. SETTING: Five Dalhousie University family medicine sites--Fredericton, Moncton, and Saint John in New Brunswick, and Halifax and Sydney in Nova Scotia. PARTICIPANTS: First- and second-year family medicine residents (as of May 2009) and core family medicine faculty. MAIN OUTCOME MEASURES: Residents' outcome measures included beliefs about the effects of field notes on performance, learning, reflection, clinical skills development, and feedback received. Faculty outcome measures included beliefs about the effect of field notes on guiding feedback, teaching, and reflection on clinical practice. RESULTS: Forty of 88 residents (45.5%) participated. Fifteen of 50 faculty (30.0%) participated, which only permitted a discussion of trends for faculty. Residents believed field note-directed feedback reinforced their performance (81.1%), helped them learn (67.6%), helped them reflect on practice and learning (66.7%), and focused the feedback they received, making it more useful (62.2%) (P < .001 for all); 63.3% believed field note-directed feedback helped with clinical skills development (P < .01). Faculty believed field notes helped to provide more focused (86.7%) and effective feedback (78.6%), improved teaching (75.0%), and encouraged reflection on their own clinical practice (73.3%). CONCLUSION: Most surveyed residents believed field note use improved the feedback they received and helped them to develop competence through improved performance, learning, reflection, and clinical skills development. The trends from faculty information suggested faculty believed field notes were an effective teaching, feedback, and reflection tool.

Defining competency-based evaluation objectives in family medicine: communication skills.

OBJECTIVE: To provide a pragmatic approach to the evaluation of communication skills using observable behaviours, as part of a multiyear project to develop competency-based evaluation objectives for Certification in family medicine. DESIGN: A nominal group technique was used to develop themes and subthemes and to identify positive and negative observable behaviours that demonstrate competence in communication in family medicine. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: The group used the nominal group technique to derive a list of observable behaviours that would constitute a detailed operational definition of competence in communication skills; multiple iterations were used until saturation was achieved. The group met several times a year, and membership remained unchanged during the 4 years in which the work was conducted. The iterative process was undertaken twice--once for communication with patients and once for communication with colleagues. MAIN FINDINGS: Five themes, 5 subthemes, and 106 positive and negative observable behaviours were generated. The subtheme of charting skills was defined using a key-features analysis. CONCLUSION: Communication skills were defined in terms of themes and observable behaviours. These definitions were intended to help assess family physicians' competence at the start of independent practice.

Defining competency-based evaluation objectives in family medicine: key-feature approach.

OBJECTIVE: To develop key features for priority topics previously identified by the College of Family Physicians of Canada that, together with skill dimensions and phases of the clinical encounter, broadly describe competence in family medicine. DESIGN: Modified nominal group methodology, which was used to develop key features for each priority topic through an iterative process. SETTING: The College of Family Physicians of Canada. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian family medicine context with respect to region, sex, language, community type, and experience. METHODS: The group used a modified Delphi process to derive a detailed operational definition of competence, using multiple iterations until consensus was achieved for the items under discussion. The group met 3 to 4 times a year from 2000 to 2007. MAIN FINDINGS: The group analyzed 99 topics and generated 773 key features. There were 2 to 20 (average 7.8) key features per topic; 63% of the key features focused on the diagnostic phase of the clinical encounter. CONCLUSION: This project expands previous descriptions of the process of generating key features for assessment, and removes this process from the context of written examinations. A key-features analysis of topics focuses on higher-order cognitive processes of clinical competence. The project did not define all the skill dimensions of competence to the same degree, but it clearly identified those requiring further definition. This work generates part of a discipline-specific, competency-based definition of family medicine for assessment purposes. It limits the domain for assessment purposes, which is an advantage for the teaching and assessment of learners. A validation study on the content of this work would ensure that it truly reflects competence in family medicine.