Refresh of a Clinical Skills Assessment for Physician Trainees.

Purpose: The Royal Australasian College of Physicians (RACP) oversees physician training across Australia and Aotearoa New Zealand. Success in a written examination and clinical skills assessment (known as the clinical examination) at the mid-point of training is a requirement to progress from basic to advanced training. The clinical examination had evolved over many years without a review process. This paper describes the approach taken, the changes made and the evaluation undertaken as part of a formal review. Methods: A working party that included education experts and examiners experienced in the assessment of clinical skills was established. The purpose of the clinical examination and competencies being assessed were clarified and were linked to learning objectives. Significant changes to the marking and scoring approaches resulted in a more holistic approach to the assessment of candidate performance with greater transparency of standards. Evaluation over a 2-year period was undertaken before the adoption of the new approach in 2019. Results: In 2017 testing of a new marking rubric occurred during the annual examination cycle which confirmed feasibility and acceptability. The following year an extensive trial utilising the new marking rubric and a new scoring approach took place involving 1142 examiners, 880 candidates and 5280 scoresheets which led to some minor modifications to the scoring system. The final marking and scoring approaches resulted in unchanged pass rates and improved inter-rater reliability. Feedback from examiners confirmed that the new marking and scoring approaches were easier to use and enabled better feedback on performance for candidates. Conclusion: The refresh of the RACP clinical examination has resulted in an assessment that has clarity of purpose, is linked to learning objectives, has greater transparency of expected standards, has improved inter-rater reliability, is well accepted by examiners and enables feedback on examination performance to candidates.

Twitter metrics complement traditional conference evaluations to evaluate knowledge translation at a National Emergency Medicine Conference.

OBJECTIVES: Conferences are designed for knowledge translation, but traditional conference evaluations are inadequate. We lack studies that explore alternative metrics to traditional evaluation metrics. We sought to determine how traditional evaluation metrics and Twitter metrics performed using data from a conference of the Canadian Association of Emergency Physicians (CAEP). METHODS: This study used a retrospective design to compare social media posts and tradition evaluations related to an annual specialty conference. A post ("tweet") on the social media platform Twitter was included if it associated with a session. We differentiated original and discussion tweets from retweets. We weighted the numbers of tweets and retweets to comprise a novel Twitter Discussion Index. We extracted the speaker score from the conference evaluation. We performed descriptive statistics and correlation analyses. RESULTS: Of a total of 3,804 tweets, 2,218 (58.3%) were session-specific. Forty-eight percent (48%) of all sessions received tweets (mean = 11.7 tweets; 95% CI of 0 to 57.5; range, 0-401), with a median Twitter Discussion Index score of 8 (interquartile range, 0 to 27). In the 111 standard presentations, 85 had traditional evaluation metrics and 71 received tweets (p > 0.05), while 57 received both. Twenty (20 of 71; 28%) moderated posters and 44% (40 of 92) posters or oral abstracts received tweets without traditional evaluation metrics. We found no significant correlation between Twitter Discussion Index and traditional evaluation metrics (R = 0.087). CONCLUSIONS: We found no correlation between traditional evaluation metrics and Twitter metrics. However, in many sessions with and without traditional evaluation metrics, audience created real-time tweets to disseminate knowledge. Future conference organizers could use Twitter metrics as a complement to traditional evaluation metrics to evaluate knowledge translation and dissemination.

Teaching M&M rounds skills: enhancing and assessing patient safety competencies using the Ottawa M&M model.

BACKGROUND: Postgraduate medical education bodies and national patient safety institutes recommend that trainees develop patient safety competencies such as those for Morbidity and Mortality (M&M) rounds, yet there exists no model for their educational delivery. OBJECTIVE: We studied the effect of a single educational intervention on emergency medicine residents' aptitudes in selecting and analysing M&M rounds cases. METHODS: In this before-and-after study, participants attended an 1 h educational session based on the previously described Ottawa Morbidity and Mortality Model (OM3). Residents were asked to submit a case suitable for M&M rounds both preintervention and postintervention. A novel M&M rounds case critique tool was developed based on OM3 and used to assign a numerical score to each submitted case. Our primary outcome was an increase in mean scores between phases using the case critique tool. An a priori score increase of 1 was defined as educationally significant. Data were analysed using a paired Student's t test. RESULTS: A total of 19 residents were recruited for our pre-intervention and 15 residents for the post-intervention analysis. Mean M&M rounds case critique scores increased from 5.53 to 8.67 (p<0.01) between phases. Residents reported higher comfort with structured case selection and analysis, with an increase in five-point Likert scale means of 2.32 and 3.69 (p<0.01). CONCLUSIONS: We found that residents were more effective at M&M rounds case selection and analysis after our focused 1 h educational intervention. Training programmes should consider an M&M rounds training model to ensure future physicians have these skills for 21st-century practice.

Emergency medicine point-of-care ultrasonography: a national needs assessment of competencies for general and expert practice.

INTRODUCTION: Emergency medicine point-of-care ultrasonography (EM-PoCUS) is a core competency for residents in the Royal College of Physicians and Surgeons of Canada and College of Family Physicians of Canada emergency medicine (EM) training programs. Although EM-PoCUS fellowships are currently offered in Canada, there is little consensus regarding what training should be included in a Canadian EM-PoCUS fellowship curriculum or how this contrasts with the training received in an EM residency.Objectives To conduct a systematic needs assessment of major stakeholders to define the essential elements necessary for a Canadian EM-PoCUS fellowship training curriculum. METHODS: We carried out a national survey of experts in EM-PoCUS, EM residency program directors, and EM residents. Respondents were asked to identify competencies deemed either nonessential to EM practice, essential for general EM practice, essential for advanced EM practice, or essential for EM-PoCUS fellowship trained (''expert'') practice. RESULTS: The response rate was 81% (351 of 435). PoCUS was deemed essential to general EM practice for basic cardiac, aortic, trauma, and procedural imaging. PoCUS was deemed essential to advanced EM practice in undifferentiated symptomatology, advanced chest pathologies, and advanced procedural applications. Expert-level PoCUS competencies were identified for administrative, pediatric, and advanced gynecologic applications. Eighty-seven percent of respondents indicated that there was a need for EM-PoCUS fellowships, with an ideal length of 6 months. CONCLUSION: This is the first needs assessment of major stakeholders in Canada to identify competencies for expert training in EM-PoCUS. The competencies should form the basis for EM-PoCUS fellowship programs in Canada.

Point-of-care ultrasonography adoption in Canada: using diffusion theory and the Evaluation Tool for Ultrasound skills Development and Education (ETUDE).

OBJECTIVE: Point-of-care ultrasonography (PoCUS) first appeared in the 1980s in North America, but the extent of the diffusion of its adoption is unknown. We characterized early PoCUS adoption by emergency physicians in Canada and its barriers to use using Rogers' diffusion of innovations theory. METHODS: We developed a questionnaire based on a pilot study and literature review to assess past, current, and potential use of PoCUS and potential barriers to adoption. A Dillman technique for electronic surveys was used for dissemination. Using Rogers' diffusion of innovations theory, we developed and validated the Evaluation Tool for Ultrasound skills Development and Education (ETUDE). ETUDE scores allowed categorization of respondents into innovators, early adopters, majority, and nonadopters. Descriptive statistics, correlations, and χ² statistics were used to analyze the data. RESULTS: The 296 respondents (36.4% of 814 surveyed) had a median age of 40 and were 72.5% male. Adoption scores using ETUDE revealed nonadopters (18.8%), majority (28.7%), early adopters (34.5%), and innovators (18.0%). Respondents endorsed "always" using PoCUS currently and in the future for focused assessment with sonography in trauma (FAST) (current 41.8%/future 88.4%), first trimester pregnancy (current 23.3%/future 73.7%), suspected abdominal aortic aneurysm (current 32.7%/future 92.6%), basic cardiac indications (current 30.7%/future 87.5%), and central venous catheterization (current 17.0%/future 80.3%). Several barriers to PoCUS were identified for part-time emergency physicians and those working in inner-city/urban/suburban settings. CONCLUSION: This is the first study to determine the state of adoption and barriers to the introduction of PoCUS in Canadian emergency medicine practice. The novel validated ETUDE instrument should be used to evaluate the uptake of PoCUS over time.

Clinician investigator training in Canada: a review.

PURPOSE: The Royal College of Physicians and Surgeons of Canada undertook a review of its Clinician Investigator Program (CIP), 13 years after launching the program in response to shortages in clinical investigators. The primary study goals were to determine the outcomes, impact, strengths and weaknesses of CIP. METHODS: Focus groups and telephone interviews with current and past program directors (PD) and a detailed survey of current and former trainees were conducted. Thirteen PD and 45% of current and former trainees from 10 CIP participated. RESULTS: Since 1995, 12 CIP have been accredited and 553 residents have enrolled in CIP, with 194 completing CIP and residency training by 2008. PD recognized CIP as an excellent program that produces highly qualified clinical investigators; important for faculty renewal. Both trainees and PD identified the need to improve CIP funding. Most (84%) CIP trainees did not have prior graduate degrees. Most alumni had completed Masters (58%) or Doctoral (39%) programs during CIP and published on their CIP research (97%). Among alumni who completed CIP and residency, many obtained an academic appointment with protected time for research, with 39% receiving an external career award. Many (60%) alumni reported no drawbacks to CIP and recognized the added values included Royal College recognition, structured training, pursuit of graduate studies, integration of clinical/research training and enhanced mentorship. CONCLUSION: Since the progam's inception, the number of CIP in Canada has grown. CIP are recognized as important mechanisms for integrating clinical and research training during residency to produce highly qualified clinician investigators.

Canadian residents' perceived manager training needs.

BACKGROUND: Despite widespread endorsement for administrative training during residency, teaching and learning in this area remains intermittent and limited in most programmes. AIM: To inform the development of a Manager Train-the-Trainer program for faculty, the Royal College of Physicians and Surgeons of Canada undertook a survey of perceived Manager training needs among postgraduate trainees. METHODS: A representative sample of Canadian specialty residents received a web-based questionnaire in 2009 assessing their perceived deficiencies in 13 Manager knowledge and 11 Manager skill domains, as determined by gap scores (GSs). GSs were defined as the difference between residents' perceived current and desired level of knowledge or skill in selected Manager domains. Residents' educational preferences for furthering their Manager knowledge and skills were also elicited. RESULTS: Among the 549 residents who were emailed the survey, 199 (36.2%) responded. Residents reported significant gaps in most knowledge and skills domains examined. Residents' preferred educational methods for learning Manager knowledge and skills included workshops, web-based formats and interactive small groups. CONCLUSION: The results of this national survey, highlighting significant perceived gaps in multiple Manager knowledge and skills domains, may inform the development of Manager curricula and faculty development activities to address deficiencies in training in this important area.

A new emergency medicine clerkship program: students' perceptions of what works.

OBJECTIVE: Emergency medicine is an evolving discipline in Canadian medical schools. Little has been published regarding student preferences for emergency medicine training during the clerkship phase of MD programs. We assessed medical students' perceptions of a newly developed emergency medicine clerkship rotation involving multiple learning modalities. The evaluation process included assessment of the rotation's instructional elements and overall educational value. METHODS: The first cohort of medical students to complete this new emergency medicine clerkship was invited to answer a questionnaire just before graduation. Students rated their preferences for components of the rotation using paired comparisons. Open ended questions explored students' satisfaction with the emergency medicine clerkship as well as perceptions of the rotation's impact on career development. RESULTS: Of the 94 students in the first clerkship cohort, 81 (86%) responded to the survey. Students found the emergency medicine clerkship highly valuable, citing the broad range of cases seen, close supervision, and opportunities to develop clinical assessment, decision making and procedural skills. Students' curricular preferences were for advanced cardiac life support (ACLS) (26.4%), clinical shifts (20.6%), supervised clinical shifts (17.8%), procedural skills laboratories (14.8%), tutorials (10.8%) and preceptor assisted learning sessions (9.8%). CONCLUSION: This new emergency medicine clerkship program incorporated multiple learning methods within a 4-week rotation and was highly rated by students. Although clinical shifts and ACLS were generally preferred activities, students had varying individual preferences for specific learning activities. Multiple learning methods allowed all students to benefit from the rotation. This study makes a compelling case for including an emergency medicine rotation with multiple learning modalities as a core element of clerkship at every medical school.

National survey of Canadian general surgery program directors regarding focused assessment with sonography for trauma.

INTRODUCTION: Focused assessment with sonography for trauma (FAST) is an important modality in the initial assessment of trauma patients. Information regarding the current status, availability, as well as educational and training processes in the use of FAST is limited. The purpose of this study was to survey Canadian general surgery residency program directors regarding FAST. METHODS: A Web-based survey was developed to assess the availability and use of FAST, education and training, as well as the role of FAST in general surgery residency training programs across Canada. Two experts reviewed the survey for validity and reliability. The survey was then pilot tested by 2 general surgeons. RESULTS: The survey response rate was 75.0% (12/16). FAST was available at all the institutions that responded. FAST was used 100% of the time in the initial assessment of the trauma patient. In all, 75.0% (9/12) respondents have no formal residency training in FAST, 91.7% (11/12) of respondents agreed or strongly agreed that training in FAST should be included in the residency curriculum, and 66.7% (8/12) of respondents perceive that general surgery residents are not competent in the use of FAST. CONCLUSIONS: Overall, Canadian general surgery program directors do not feel that surgical residents are competent in the use of FAST. Current training is limited, and program directors believe that training and education in FAST should be included in the postgraduate education curriculum.

Emergency medicine ultrasonography: national survey of family medicine-emergency medicine program directors.

OBJECTIVE: To survey program directors of family medicine-emergency medicine (CCFP[EM]) training programs regarding current and future emergency medicine ultrasonography (EMUS) training. DESIGN: A Web-based survey using a modified Dillman method. Two academic emergency physicians reviewed the validity and reliability of the survey. SETTING: Canada. PARTICIPANTS: Program directors of all 17 Canadian CCFP(EM) residency training programs in 2006. MAIN OUTCOME MEASURES: Characteristics of EMUS training currently offered and program directors' perceptions of needs for future EMUS training. RESULTS: The survey, performed in 2006, had a response rate of 100% (17/17), although not all respondents answered all questions. At the time of the study, 82.4% of respondents' programs used EMUS. Although all program directors recommended that residents attend introductory EMUS courses, only 71.4% (10/14) of programs offered such courses; 60.0% (9/15) of those were mandatory. In one-third of the programs, more than 75% of the attending staff used EMUS. A total of 76.5% of program directors thought that introductory courses in EMUS should be mandatory; 62.5% (10/16) believed that residents were able to acquire sufficient experience to use EMUS independently to make practice decisions before completion of their residency; and 88.2% believed that EMUS should be a part of the scope of practice for emergency medicine physicians. Only 58.8% believed that there should be questions about EMUS on the CCFP(EM) Certification examination. Open responses indicated that funding, resources, and standardization were issues that needed to be addressed. CONCLUSION: Formal EMUS training for CCFP(EM) programs is being introduced in Canada. Quality assurance needs to be strengthened. Most program directors thought that an introductory course in EMUS should be mandatory. Fewer directors, however, believed EMUS should be on the CCFP(EM) Certification examination until further funding, resources, and standardization of EMUS programs were in place.

Effectiveness of a novel training program for emergency medicine residents in ultrasound-guided insertion of central venous catheters.

OBJECTIVE: Insertion of central venous catheters (CVCs) is an essential competency for emergency physicians. Ultrasound-guided (USG) insertion of CVCs has been shown to be safer than the traditional landmark technique. There is no clear consensus on effective methods for training physicians in USG insertion of CVCs. We developed and evaluated a novel educational training program in the USG technique for insertion of CVCs. METHODS: Sixteen emergency medicine residents volunteered for a pre- and postprogram evaluation study, which was approved by our research ethics board. After their previous experience was determined, each participant was videotaped inserting a USG CVC in the right internal jugular vein on models. Participants then reviewed a Web-based instructional module and had a practical session. Participants were again videotaped inserting a USG CVC. The primary outcome was the change in score before and after the training program, using an expert-validated performance evaluation tool used to review the videotaped performances in a blinded fashion. Participants also completed a questionnaire to measure their satisfaction with the training program and any change in their perceived competence. RESULTS: Participants ranged from residency year 1 to 5. Thirteen of 16 (81%) had never attempted USG insertion of a CVC. Participants reported that the models were realistic. Performance scores (12/19 to 13.2/19) and global ratings assessments (3.5/7 to 5.5/7) improved significantly (p < 0.01; the effect size, Cohen d = 1.12 before and 1.28 after) after the instruction. There was good interrater reliability between evaluators of the videotaped performances regarding performance scores (r = 0.68) and global rating scores (r = 0.75). All participants felt their confidence and technical skills were improved (p < 0.01) and all felt satisfied with the training program. CONCLUSION: This brief innovative multimethod training program was effective in enhancing emergency medicine resident competence in USG insertion of CVCs.

Assessing the quality of supervisors' completed clinical evaluation reports.

CONTEXT: Although concern has been raised about the value of clinical evaluation reports for discriminating among trainees, there have been few efforts to formalise the dimensions and qualities that distinguish effective versus less useful styles of form completion. METHODS: Using brainstorming and a modified Delphi technique, a focus group determined the key features of high-quality completed evaluation reports. These features were used to create a rating scale to evaluate the quality of completed reports. The scale was pilot-tested locally; the results were psychometrically analysed and used to modify the scale. The scale was then tested on a national level. Psychometric analysis and final modification of the scale were completed. RESULTS: Sixteen features of high-quality reports were identified and used to develop a rating scale: the Completed Clinical Evaluation Report Rating (CCERR). The reliability of the scale after a national field test with 55 raters assessing 18 in-training evaluation reports (ITERs) was 0.82. Further revisions were made; the final version of the CCERR contains nine items rated on a 5-point scale. With this version, the mean ratings of three groups of 'gold-standard' ITERs (previously judged to be of high, average and poor quality) differed significantly (P < 0.05). DISCUSSION: The CCERR is a validated scale that can be used to help train supervisors to complete and assess the quality of evaluation reports.