A systems approach for institutional CBME adoption at Queen's University.

The Royal College of Physicians and Surgeons of Canada (RCPSC) has begun the transition to Competency by Design (CBD), a new curricular model for residency education that 'ensure[s] competence, but teaches for excellence'. By 2022, all Canadian specialty programs are anticipated to have completed the CBD cohort process which includes workshops facilitated by a Royal College Clinician Educator. Queen's University in Ontario, Canada, was granted approval by the RCPSC to embark upon an accelerated path to competency-based medical education (CBME) for all our postgraduate specialties. This accelerated path allowed us to take an institutional approach for CBME implementation and ensure that all specialities were part of a system-wide change. Our unique institution-wide approach to CBD is the first of its kind across Canada. From both a theoretical and practical perspective we undertook CBME using a systems approach that allowed us to build the foundations for CBME, implement the change, and plan for sustainability. This has created opportunities to bridge and connect the various programs involved in the implementation of CBME on Queen's campus. The systems approach was an essential part of our strategy to develop a community dedicated to ensuring a successful CBME implementation.

Competency-based resident education-The Canadian perspective.

Canada has been a leader in competency-based medical education for some years. Postgraduate training programs are typically 5 years in duration with opportunities to pursue 2-year subspecialty training after certification in a primary specialty. The introduction of competency-based models in Canada has progressed from a single orthopedic surgery training program at the University of Toronto through the adoption of competency-based medical education in 29 training programs at a single medical school, and the implementation across all 68 disciplines overseen by the Royal College of Physicians and Surgeons of Canada. This article outlines the introduction of competency-based medical education in postgraduate medical education in Canada.

Comparison of Personal Video Technology for Teaching and Assessment of Surgical Skills.

BACKGROUND: Improvements in personal technology have made video recording for teaching and assessment of surgical skills possible. OBJECTIVE: This study compared 5 personal video-recording devices based on their utility (image quality, hardware, mounting options, and accessibility) in recording open surgical procedures. METHODS: Open procedures in a simulated setting were recorded using smartphones and tablets (MOB), laptops (LAP), sports cameras such as GoPro (SC), single-lens reflex cameras (DSLR), and spy camera glasses (SPY). Utility was rated by consensus between 2 investigators trained in observation of technology using a 5-point Likert scale (1, poor, to 5, excellent). RESULTS: A total of 150 hours of muted video were reviewed with a minimum 1 hour for each device. Image quality was good (3.8) across all devices, although this was influenced by the device-mounting requirements (4.2) and its proximity to the area of interest. Device hardware (battery life and storage capacity) was problematic for long procedures (3.8). Availability of devices was high (4.2). CONCLUSIONS: Personal video-recording technology can be used for assessment and teaching of open surgical skills. DSLR and SC provide the best images. DSLR provides the best zoom capability from an offset position, while SC can be placed closer to the operative field without impairing sterility. Laptops provide best overall utility for long procedures due to video file size. All devices require stable recording platforms (eg, bench space, dedicated mounting accessories). Head harnesses (SC, SPY) provide opportunities for "point-of-view" recordings. MOB and LAP can be used for multiple concurrent recordings.

Delivering on the promise of competency based medical education - an institutional approach.

The Royal College of Physicians and Surgeons of Canada (RCPSC) adopted a plan to transform, over a seven-year horizon (2014-2021), residency education across all specialties to competency-based medical education (CBME) curriculum models. The RCPSC plan recommended implementing a more responsive and accountable training model with four discrete stages of training, explicit, specialty specific entrustable professional activities, with associated milestones, and a programmatic approach to assessment across residency education. Embracing this vision, the leadership at Queen's University (in Kingston, Ontario, Canada) applied for and was granted special permission by the RCPSC to embark on an accelerated institutional path. Over a three-year period, Queen's took CBME from concept to reality through the development and implementation of a comprehensive strategic plan. This perspective paper describes Queen's University's approach of creating a shared institutional vision, outlines the process of developing a centralized CBME executive team and twenty-nine CBME program teams, and summarizes proactive measures to ensure program readiness for launch. In so doing, Queen's created a community of support and CBME expertise that reinforces shared values including fostering co-production, cultivating responsive leadership, emphasizing diffusion of innovation, and adopting a systems-based approach to transformative change.

Student Reflections on the Queen's Accelerated Route to Medical School Programme.

CONTEXT: Since its inception more than 150 years ago, the School of Medicine at Queen's University has aspired 'to advance the tradition of preparing excellent physicians and leaders in health care by embracing a spirit of inquiry and innovation in education and research'. As part of this continuing commitment, Queen's School of Medicine developed the Queen's University Accelerated Route to Medical School (QuARMS). As Canada's only 2-year accelerated-entry premedical programme, QuARMS was designed to reduce training time, the associated expense of medical training, and to encourage a collaborative premedical experience. Students enter QuARMS directly from high school and then spend 2 years enrolled in an undergraduate degree programme. They then are eligible to enter the first-year MD curriculum. The 2-year QuARMS academic curriculum includes traditional undergraduate coursework, small group sessions, and independent activities. The QuARMS curriculum is built on 4 pillars: communication skills, critical thinking, the role of physician (including community service learning [CSL]), and scientific foundations. Self-regulated learning (SRL) is explicitly developed throughout all aspects of the curriculum. Medical educators have defined SRL as the cyclical control of academic and clinical performance through several key processes that include goal-directed behaviour, use of specific strategies to attain goals, and the adaptation and modification to behaviours or strategies that optimize learning and performance. Based on Zimmerman's social cognitive framework, this definition includes relationships among the individual, his or her behaviour, and the environment, with the expectation that individuals will monitor and adjust their behaviours to influence future outcomes. OBJECTIVES: This study evaluated the students' learning as perceived by them at the conclusion of their first 2 academic years. METHODS: At the end of the QuARMS learning stream, the first and second cohorts of students completed a 26-item, 4-point Likert-type instrument with space for optional narrative details for each question. A focus group with each group explored emergent issues. Consent was obtained from 9 out of 10 and 7 out of 8 participants to report the 2015 survey and focus group data, respectively, and from 10 out of 10 and 9 out of 10 participants to report the 2016 survey and focus group data, respectively. Thematic analysis and a constructivist interpretive paradigm were used. A distanced facilitator, standard protocols, and a dual approach assured consistency and trustworthiness of data. RESULTS: Both analyses were congruent. Students described experiences consistent with curricular goals including critical thinking, communication, role of a physician, CSL, and SRL. Needs included additional mentorship, more structure for CSL, more feedback, explicit continuity between in-class sessions, and more clinical experience. Expectations of students towards engaging in independent learning led to some feelings of disconnectedness. CONCLUSIONS: Participants described benefit from the sessions and an experience consistent with the curricular goals, which were intentionally focused on foundational skills. In contrast to the goal of SRL, students described a need for an explicit educational structure. Thus, scaffolding of the curriculum from more structured in year 1 to less structured in year 2 using additional mentorship and feedback is planned for subsequent years. Added clinical exposure may increase relevance but poses challenges for integration with the first-year medical class.

Comparison of faculty versus structured peer-feedback for acquisitions of basic and intermediate-level surgical skills.

PURPOSE: Video feedback and faculty feedback has been shown to improve surgical performance; however, consistent access to faculty is challenging. We studied the utility of structured peer-feedback (PF) compared to faculty-feedback (FF) during acquisition of basic and intermediate surgical skills. METHODOLOGY: Two randomized non-inferiority trials were conducted with 1st (n = 30) and 2nd year (n = 29) medical students learning skin-lesion excision and closure (S), and single-layer hand-sewn bowel anastomosis (B), respectively. Five attempts were performed. PF participants used an Objective Structured Assessment of Technical Skills tool to guide feedback. Blinded raters assessed video-recorded performance, time and Integrity of the completed task were also assessed. RESULTS: For both tasks performance by PF was comparable to FF (P = 0.111). Both groups improved significantly: performance (B:P < 0.0001, S:P = 0.035), time (B:P = 0.043, S:P < 0.0001) and integrity (B:P < 0.0001, S:P < 0.032). CONCLUSION: Structured peer-feedback is equivalent to faculty-feedback in the acquisition of basic and intermediate surgical skills, giving students freedom to practice independently.

Eight-year outcomes of a competency-based residency training program in orthopedic surgery.

Background: The Division of Orthopaedic Surgery at the University of Toronto implemented a pilot residency training program that used a competency-based framework in July of 2009. The competency-based curriculum (CBC) deployed an innovative, modularized approach that dramatically intensified both the structured learning elements and the assessment processes. Methods: This paper discusses the initial curriculum design of the CBC pilot program; the refinement of the curriculum using curriculum mapping that allowed for efficiencies in educational delivery; details of evaluating resident competence; feedback from external reviews by accrediting bodies; and trainee and program outcomes for the first eight years of the program's implementation. Results: Feedback from the residents, the faculty, and the postgraduate residency training accreditation bodies on the CBC has been positive and suggests that the essential framework of the program may provide a valuable tool to other programs that are contemplating embarking on transition to competency-based education. Conclusions: While the goal of the program was not to shorten training per se, efficiencies gained through a modular, competency-based program have resulted in shortened time to completion of residency training for some learners.

The Surgical Skills and Technology Elective Program (SSTEP): A comprehensive simulation-based surgical skills initiative for preclerkship medical students.

BACKGROUND: The Surgical Skills and Technology Elective Program (SSTEP) is a voluntary preclerkship surgical bootcamp that uses simulation learning to build procedural knowledge and technical skills before clerkship. METHODS: Eighteen second year students (n = 18) participated in simulation workshops over the course of 7 days to learn clerkship-level procedural skills. A manual was supplied with the program outline. Assessment of the participants involved: 1) a written exam 2) a single videotaped Objective Structured Assessment of Technical Skill (OSATS) station 3) an exit survey to document changes in career choices. RESULTS: Compared to the mean written pre-test score students scored significantly higher on the written post-test (35.83 ± 6.56 vs. 52.11 ± 5.95 out of 73) (p = 0.01). Technical skill on the OSATS station demonstrated improved performance and confidence following the program (10.10 vs. 17.94 out of 25) (p = 0.05). Most participants (72%) re-considered their choices of surgical electives. CONCLUSIONS: A preclerkship surgical skills program not only stimulates interest in surgery but can also improve surgical knowledge and technical skills prior to clerkship.

Comparison of Canadian and Swiss Surgical Training Curricula: Moving on Toward Competency-Based Surgical Education.

OBJECTIVE: Quality of surgical training in the era of resident duty-hour restrictions (RDHR) is part of an ongoing debate. Most training elements are provided during surgical service. As exposure to surgical procedures is important but time-consuming, RDHR may affect quality of surgical training. Providing structured training elements may help to compensate for this shortcoming. DESIGN: This binational anonymous questionnaire-based study evaluates frequency, time, and structure of surgical training programs at 2 typical academic teaching hospitals with different RDHR. SETTING: Departments of Surgery of University of Basel (Basel, Switzerland) and the Queen's University (Kingston, Ontario, Canada). PARTICIPANTS: Surgical consultants and residents of the Queen's University Hospital (Kingston, Ontario, Canada) and the University Hospital Basel (Basel, Switzerland) were eligible for this study. RESULTS: Questionnaire response rate was 37% (105/284). Queen's residents work 80 hours per week, receiving 7 hours of formal training (8.8% of workweek). Basel residents work 60 hours per week, including 1 hour of formal training (1.7% of working time). Queen's faculty and residents rated their program as "structured" or "rather structured" in contrast to Basel faculty and residents who rated their programs as "neutral" in structure or "unstructured." Respondents identified specific structured training elements more frequently at Queen's than in Basel. Two-thirds of residents responded that they seek out additional surgical experiences through voluntary extra work. Basel participants articulated a stronger need for improvement of current surgical training. Although Basel residents and consultants in both institutions fear negative influence of RDHR on the training program, this was not the case in Queen's residents. CONCLUSIONS: Providing more structured surgical training elements may be advantageous in providing optimal-quality surgical education in an era of work-hour restrictions.

Simulation for Teaching Orthopaedic Residents in a Competency-based Curriculum: Do the Benefits Justify the Increased Costs?

BACKGROUND: Although simulation-based training is becoming widespread in surgical education and research supports its use, one major limitation is cost. Until now, little has been published on the costs of simulation in residency training. At the University of Toronto, a novel competency-based curriculum in orthopaedic surgery has been implemented for training selected residents, which makes extensive use of simulation. Despite the benefits of this intensive approach to simulation, there is a need to consider its financial implications and demands on faculty time. QUESTIONS/PURPOSES: This study presents a cost and faculty work-hours analysis of implementing simulation as a teaching and evaluation tool in the University of Toronto's novel competency-based curriculum program compared with the historic costs of using simulation in the residency training program. METHODS: All invoices for simulation training were reviewed to determine the financial costs before and after implementation of the competency-based curriculum. Invoice items included costs for cadavers, artificial models, skills laboratory labor, associated materials, and standardized patients. Costs related to the surgical skills laboratory rental fees and orthopaedic implants were waived as a result of special arrangements with the skills laboratory and implant vendors. Although faculty time was not reimbursed, faculty hours dedicated to simulation were also evaluated. The academic year of 2008 to 2009 was chosen to represent an academic year that preceded the introduction of the competency-based curriculum. During this year, 12 residents used simulation for teaching. The academic year of 2010 to 2011 was chosen to represent an academic year when the competency-based curriculum training program was functioning parallel but separate from the regular stream of training. In this year, six residents used simulation for teaching and assessment. The academic year of 2012 to 2013 was chosen to represent an academic year when simulation was used equally among the competency-based curriculum and regular stream residents for teaching (60 residents) and among 14 competency-based curriculum residents and 21 regular stream residents for assessment. RESULTS: The total costs of using simulation to teach and assess all residents in the competency-based curriculum and regular stream programs (academic year 2012-2013) (CDN 155,750, USD 158,050) were approximately 15 times higher than the cost of using simulation to teach residents before the implementation of the competency-based curriculum (academic year 2008-2009) (CDN 10,090, USD 11,140). The number of hours spent teaching and assessing trainees increased from 96 to 317 hours during this period, representing a threefold increase. CONCLUSIONS: Although the financial costs and time demands on faculty in running the simulation program in the new competency-based curriculum at the University of Toronto have been substantial, augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more efficient learning outcomes. CLINICAL RELEVANCE: The higher costs and demands on faculty time associated with implementing simulation for teaching and assessment must be considered when it is used to enhance surgical training.

Gastrointestinal Endoscopy Competency Assessment Tool: reliability and validity evidence.

BACKGROUND: Rigorously developed and validated direct observational assessment tools are required to support competency-based colonoscopy training to facilitate skill acquisition, optimize learning, and ensure readiness for unsupervised practice. OBJECTIVE: To examine reliability and validity evidence of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) for colonoscopy for use within the clinical setting. DESIGN: Prospective, observational, multicenter validation study. Sixty-one endoscopists performing 116 colonoscopies were assessed using the GiECAT, which consists of a 7-item global rating scale (GRS) and 19-item checklist (CL). A second rater assessed procedures to determine interrater reliability by using intraclass correlation coefficients (ICCs). Endoscopists' first and second procedure scores were compared to determine test-retest reliability by using ICCs. Discriminative validity was examined by comparing novice, intermediate, and experienced endoscopists' scores. Concurrent validity was measured by correlating scores with colonoscopy experience, cecal and terminal ileal intubation rates, and physician global assessment. SETTING: A total of 116 colonoscopies performed by 33 novice (<50 previous procedures), 18 intermediate (50-500 previous procedures), and 10 experienced (>1000 previous procedures) endoscopists from 6 Canadian hospitals. MAIN OUTCOME MEASUREMENTS: Interrater and test-retest reliability, discriminative, and concurrent validity. RESULTS: Interrater reliability was high (total: ICC=0.85; GRS: ICC=0.85; CL: ICC=0.81). Test-retest reliability was excellent (total: ICC=0.91; GRS: ICC=0.93; CL: ICC=0.80). Significant differences in GiECAT scores among novice, intermediate, and experienced endoscopists were noted (P<.001). There was a significant positive correlation (P<.001) between scores and number of previous colonoscopies (total: ρ=0.78, GRS: ρ=0.80, CL: Spearman's ρ=0.71); cecal intubation rate (total: ρ=0.81, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.75); ileal intubation rate (total: Spearman's ρ=0.82, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.77); and physician global assessment (total: Spearman's ρ=0.90, GRS: Spearman's ρ=0.94, CL: Spearman's ρ=0.77). LIMITATIONS: Nonblinded assessments. CONCLUSION: This study provides evidence supporting the reliability and validity of the GiECAT for use in assessing the performance of live colonoscopies in the clinical setting.

Student-led learning: a new teaching paradigm for surgical skills.

BACKGROUND: Competency-based education and simulation are being used more frequently in surgical skills curricula. We explored a novel student-led learning paradigm, which allows trainees to become more active participants in the learning process while maintaining expert guidance and supervision. METHODS: Twelve first-year orthopedic residents were randomized to either a student-led (SL) or a traditional instructor-led group during an intensive, month-long, laboratory-based technical skills training course. A rigorous qualitative-description approach was used for analysis. RESULTS: Four prominent themes emerged: instructional style, feedback, peer and instructor collaboration, and self-efficacy. Compared with the instructor-led group, there was more peer assistance, feedback, collaboration, and hands-on and active learning observed in the SL group. CONCLUSIONS: The flexible and socially rich nature of the SL learning environment may aid in development of both technical and nontechnical skills early in residency and ultimately privilege later clinical learning.

Lessons learned in the pursuit of a dream.

CONTEXT: The author describes a career in which he combined clinical surgery with the formal study of medical education. In the 1980s, when the author embarked on this career track, it was an uncommon pathway. Over the last 30 years there has been an exponential increase in the number of individuals who have made medical education their principal academic focus. This paper provides examples from the author's personal story and lessons derived from that experience. PROCESS: The author outlines his experience of attaining formal training in education and concludes that this training was a foundational element in his pursuit of a career in health education research. The author describes his involvement in the transition from paper and pencil-based tests to performance-based testing in high-stakes examinations. He describes the development of a research centre in health professions education and the establishment of a simulation centre. The author's experiences in the development of an examination intended to measure technical skills, in the adoption of surgical safety checklists and in the elaboration of a programme in competency-based education are discussed. DISCUSSION: The author describes several of the lessons learned in the course of his career in medical education. He argues that successful enterprises in scholarship in medicine are almost invariably the product of interdisciplinarity. He describes the power of a joint venture between a university and an academic hospital. He argues that the geographical footprint of an emerging centre is critical. He discusses the importance of graduate studentship in an emerging discipline and enterprise.

Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy.

BACKGROUND: Ensuring competence remains a seminal objective of endoscopy training programs, professional organizations, and accreditation bodies; however, no widely accepted measure of endoscopic competence currently exists. OBJECTIVE: By using Delphi methodology, we aimed to develop and establish the content validity of the Gastrointestinal Endoscopy Competency Assessment Tool for colonoscopy. DESIGN: An international panel of endoscopy experts rated potential checklist and global rating items for their importance as indicators of the competence of trainees learning to perform colonoscopy. After each round, responses were analyzed and sent back to the experts for further ratings until consensus was reached. MAIN OUTCOME MEASUREMENTS: Consensus was defined a priori as ≥80% of experts, in a given round, scoring ≥4 of 5 on all remaining items. RESULTS: Fifty-five experts agreed to be part of the Delphi panel: 43 gastroenterologists, 10 surgeons, and 2 endoscopy managers. Seventy-three checklist and 34 global rating items were generated through a systematic literature review and survey of committee members. An additional 2 checklist and 4 global rating items were added by Delphi panelists. Five rounds of surveys were completed before consensus was achieved, with response rates ranging from 67% to 100%. Seven global ratings and 19 checklist items reached consensus as good indicators of the competence of clinicians performing colonoscopy. LIMITATIONS: Further validation required. CONCLUSION: Delphi methodology allowed for the rigorous development and content validation of a new measure of endoscopic competence, reflective of practice across institutions. Although further evaluation is required, it is a promising step toward the objective assessment of competency for use in colonoscopy training, practice, and research.

A novel approach to assessing technical competence of colorectal surgery residents: the development and evaluation of the Colorectal Objective Structured Assessment of Technical Skill (COSATS).

OBJECTIVE: To develop and evaluate an objective method of technical skills assessment for graduating subspecialists in colorectal (CR) surgery-the Colorectal Objective Structured Assessment of Technical Skill (COSATS). BACKGROUND: It may be reasonable for the public to assume that surgeons certified as competent have had their technical skills assessed. However, technical skill, despite being the hallmark of a surgeon, is not directly assessed at the time of certification by surgical boards. METHODS: A procedure-based, multistation technical skills examination was developed to reflect a sample of the range of skills necessary for CR surgical practice. These consisted of bench, virtual reality, and cadaveric models. Reliability and construct validity were evaluated by comparing 10 graduating CR residents with 10 graduating general surgery (GS) residents from across North America. Expert CR surgeons, blinded to level of training, evaluated performance using a task-specific checklist and a global rating scale. The mean global rating score was used as the overall examination score and a passing score was set at "borderline competent for CR practice." RESULTS: The global rating scale demonstrated acceptable interstation reliability (0.69) for a homogeneous group of examinees. Both the overall checklist and global rating scores effectively discriminated between CR and GS residents (P < 0.01), with 27% of the variance attributed to level of training. Nine CR residents but only 3 GS residents were deemed competent. CONCLUSIONS: The Colorectal Objective Structured Assessment of Technical Skill effectively discriminated between CR and GS residents. With further validation, the Colorectal Objective Structured Assessment of Technical Skill could be incorporated into the colorectal board examination where it would be the first attempt of a surgical specialty to formally assess technical skill at the time of certification.

Toronto orthopaedic boot camp III: examining the efficacy of student-regulated learning during an intensive, laboratory-based surgical skills course.

BACKGROUND: Previous studies have presented compelling data that a 1-month "boot-camp"-style course can be a highly effective mechanism for teaching and developing targeted technical skills. In the current study, we examine whether performance of these targeted skills is improved when residents are trained using directed, student-led (SL) learning methods compared with traditional instructor-led (IL) learning methods. METHODS: Twelve first-year orthopedic residents began their training with a 1-month, intensive skills course. Six residents were taught basic surgical skills using a format that focused on deliberate, SL exploration and practice of the skills under instructor supervision (SL group). The remaining residents were taught the same surgical skills using more traditional IL methods that included complete demonstration of the surgical task by an orthopedic surgeon, followed by an extended period of instruction (IL group). Performance on 4 targeted technical skills (sawing, bone drilling, suturing, and plaster splint application) was tested using an objective, structured assessment of technical skills examination for the 2 groups at the beginning and the end of the skills course. RESULTS: Before the start of the skills course, there were no differences in performance scores between the 2 groups. On completion of the skills course, mean global rating scores for the 4 surgical skills tasks were greater for the SL group compared with the IL group: SL, 3.95 ± 0.1; IL, 3.42 ± 0.1; F(1,10) = 7.66 P < .02. A similar pattern of results was revealed by the checklists scores, with the SL group outperforming the IL group: SL, 94.9 ± 2.1; IL, 86.4 ± 2.1; F(1,10) = 8.512; P < .02. CONCLUSION: Previous work has demonstrated the effectiveness of teaching basic surgical skills through an intensive course at the onset of residency. The present study shows that allowing surgical trainees to take a directed, student-regulated approach to learning basic surgical skills can further improve performance of these skills.