Setting Performance Standards for Technical and Nontechnical Competence in General Surgery.

OBJECTIVES: The objectives of this study were to (1) create a technical and nontechnical performance standard for the laparoscopic cholecystectomy, (2) assess the classification accuracy and (3) credibility of these standards, (4) determine a trainees' ability to meet both standards concurrently, and (5) delineate factors that predict standard acquisition. BACKGROUND: Scores on performance assessments are difficult to interpret in the absence of established standards. METHODS: Trained raters observed General Surgery residents performing laparoscopic cholecystectomies using the Objective Structured Assessment of Technical Skill (OSATS) and the Objective Structured Assessment of Non-Technical Skills (OSANTS) instruments, while as also providing a global competent/noncompetent decision for each performance. The global decision was used to divide the trainees into 2 contrasting groups and the OSATS or OSANTS scores were graphed per group to determine the performance standard. Parametric statistics were used to determine classification accuracy and concurrent standard acquisition, receiver operator characteristic (ROC) curves were used to delineate predictive factors. RESULTS: Thirty-six trainees were observed 101 times. The technical standard was an OSATS of 21.04/35.00 and the nontechnical standard an OSANTS of 22.49/35.00. Applying these standards, competent/noncompetent trainees could be discriminated in 94% of technical and 95% of nontechnical performances (P < 0.001). A 21% discordance between technically and nontechnically competent trainees was identified (P < 0.001). ROC analysis demonstrated case experience and trainee level were both able to predict achieving the standards with an area under the curve (AUC) between 0.83 and 0.96 (P < 0.001). CONCLUSIONS: The present study presents defensible standards for technical and nontechnical performance. Such standards are imperative to implementing summative assessments into surgical training.

Practice does not always make perfect: need for selection curricula in modern surgical training.

BACKGROUND: It is hypothesized that not all surgical trainees are able to reach technical competence despite ongoing practice. The objectives of the study were to assess a trainees' ability to reach technical competence by assessing learning patterns of the acquisition of surgical skills. Furthermore, it aims to determine whether individuals' learning patterns were consistent across a range of open and laparoscopic tasks of variable difficulty. METHODS: Sixty-five preclinical medical students participated in a training curriculum with standardized feedback over forty repetitions of the following laparoscopic and open technical tasks: peg transfer (PT), circle cutting (CC), intracorporeal knot tie (IKT), one-handed tie, and simulated laparotomy closure. Data mining techniques were used to analyze the prospectively collected data and stratify the students into four learning clusters. Performance was compared between groups, and learning curve characteristics unique to trainees who have difficulty reaching technical competence were quantified. RESULTS: Top performers (22-35%) and high performers (32-42%) reached proficiency in all tasks. Moderate performers (25-37%) reached proficiency for all open tasks but not all laparoscopic tasks. Low performers (8-15%) failed to reach proficiency in four of five tasks including all laparoscopic tasks (PT 7.8%; CC 9.4%; IKT 15.6%). Participants in lower performance clusters demonstrated sustained performance disadvantage across tasks, with widely variable learning curves and no evidence of progression towards a plateau phase. CONCLUSIONS: Most students reached proficiency across a range of surgical tasks, but low-performing trainees failed to reach competence in laparoscopic tasks. With increasing use of laparoscopy in surgical practice, screening potential candidates to identify the lowest performers may be beneficial.

Overarching challenges to the implementation of competency-based medical education.

Medical education is under increasing pressure to more effectively prepare physicians to meet the needs of patients and populations. With its emphasis on individual, programmatic, and institutional outcomes, competency-based medical education (CBME) has the potential to realign medical education with this societal expectation. Implementing CBME, however, comes with significant challenges. This manuscript describes four overarching challenges that must be confronted by medical educators worldwide in the implementation of CBME: (1) the need to align all regulatory stakeholders in order to facilitate the optimization of training programs and learning environments so that they support competency-based progression; (2) the purposeful integration of efforts to redesign both medical education and the delivery of clinical care; (3) the need to establish expected outcomes for individuals, programs, training institutions, and health care systems so that performance can be measured; and (4) the need to establish a culture of mutual accountability for the achievement of these defined outcomes. In overcoming these challenges, medical educators, leaders, and policy-makers will need to seek collaborative approaches to common problems and to learn from innovators who have already successfully made the transition to CBME.

Can We Predict Technical Aptitude?: A Systematic Review.

OBJECTIVE: To identify background characteristics and cognitive tests that may predict surgical trainees' future technical performance, and therefore be used to supplement existing surgical residency selection criteria. BACKGROUND: Assessment of technical skills is not commonly incorporated as part of the selection process for surgical trainees in North America. Emerging evidence, however, suggests that not all trainees are capable of reaching technical competence. Therefore, incorporating technical aptitude into selection processes may prove useful. METHODS: A systematic search was carried out of the MEDLINE, PsycINFO, and Embase online databases to identify all studies that assessed associations between surrogate markers of innate technical abilities in surgical trainees, and whether these abilities correlate with technical performance. The quality of each study was evaluated using the Grading of Recommendations, Assessment, Development, and Evaluation system. RESULTS: A total of 8035 records were identified. After screening by title, abstract, and full text, 52 studies were included. Very few surrogate markers were found to predict technical performance. Significant associations with technical performance were seen for 1 of 23 participant-reported surrogate markers, 2 of 25 visual spatial tests, and 2 of 19 dexterity tests. The assessment of trainee Basic Performance Resources predicted technical performance in 62% and 75% of participants. CONCLUSIONS: To date, no single test has been shown to reliably predict the technical performance of surgical trainees. Strategies that rely on assessing multiple innate abilities, their interaction, and their relationship with technical skill may ultimately be more likely to serve as reliable predictors of future surgical performance.

Assessing Technical Competence in Surgical Trainees: A Systematic Review.

OBJECTIVE: To systematically examine the literature describing the methods by which technical competence is assessed in surgical trainees. BACKGROUND: The last decade has witnessed an evolution away from time-based surgical education. In response, governing bodies worldwide have implemented competency-based education paradigms. The definition of competence, however, remains elusive, and the impact of these education initiatives in terms of assessment methods remains unclear. METHODS: A systematic review examining the methods by which technical competence is assessed was conducted by searching MEDLINE, EMBASE, PsychINFO, and the Cochrane database of systematic reviews. Abstracts of retrieved studies were reviewed and those meeting inclusion criteria were selected for full review. Data were retrieved in a systematic manner, the validity and reliability of the assessment methods was evaluated, and quality was assessed using the Grading of Recommendations Assessment, Development and Evaluation classification. RESULTS: Of the 6814 studies identified, 85 studies involving 2369 surgical residents were included in this review. The methods used to assess technical competence were categorized into 5 groups; Likert scales (37), benchmarks (31), binary outcomes (11), novel tools (4), and surrogate outcomes (2). Their validity and reliability were mostly previously established. The overall Grading of Recommendations Assessment, Development and Evaluation for randomized controlled trials was high and low for the observational studies. CONCLUSIONS: The definition of technical competence continues to be debated within the medical literature. The methods used to evaluate technical competence predominantly include instruments that were originally created to assess technical skill. Very few studies identify standard setting approaches that differentiate competent versus noncompetent performers; subsequently, this has been identified as an area with great research potential.

Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education.

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Surgical fellowship training in Canada: what is its current status and is improvement required?

This paper examines current issues concerning surgical fellowship training in Canada. Other than information from a few studies of fellowship training in North America, there are scant data on this subject in the literature. Little is known about the demographic characteristics of those who pursue fellowship training in Canada, what the experiences and expectations are of fellows and their supervisors with respect to the strengths and weaknesses of this level of training, or how this level of education fits in with Canadian undergraduate and postgraduate medical training. We summarize current knowledge about fellowship training in Canada as it pertains to demographic characteristics, finances, work hours, residency training, preparation for clinical and research work and satisfaction with training. Most information on surgical fellowship training comes from the United States. As such, we used information from American studies to supplement the Canadian data. Because a surgical fellowship experience in Canada may be different from that in the United States, we propose that Canadian surgical fellows and their supervisors should be surveyed to gain an understanding of such information. This knowledge could be used to improve surgical fellowship training in Canada.

Surgeons' and residents' double-gloving practices at 2 teaching hospitals in Ontario.

BACKGROUND: Surgeons and residents are at increased risk of exposure to blood-borne pathogens owing to percutaneous injury (PI) and contamination. One method known to reduce risk is double-gloving (DG) during surgery. METHODS: All surgeons and residents affiliated with the University of Western Ontario (UWO) and McMaster University in 2005 were asked to participate in a Web-based survey. The survey asked respondents their specialty, the number of operations they participated in per week, their age and sex, the proportion of surgeries in which they double-gloved (DG in ≥75% surgeries was considered to be routine), and the average number of PIs they sustained per year and whether or not they reported them to an employee health service. RESULTS: In total, 155 of 331 (47%) eligible surgeons and residents responded; response rates for UWO and McMaster surgeons were 50% and 39%, respectively, and for UWO and McMaster residents, they were 52% and 47%, respectively. A total of 43% of surgeons and residents reported routine DG; 50% from McMaster and 36% from UWO. Using logistic regression to simultaneously adjust for participant characteristics, we confirmed that DG was more frequent at McMaster than at UWO, with an odds ratio of 3.32 (95% confidence interval 1.35-8.17). Surgeons and residents reported an average of 3.3 surgical PIs per year (2.2 among McMaster participants and 4.5 among UWO participants). Of the 77% who reported at least 1 injury/year, 67% stated that they had not reported it to an employee health service. CONCLUSION: Percutaneous injuries occur frequently during surgery, yet routine DG, an effective means of reducing risk, was carried out by less than half of the surgeons and residents participating in this study. This highlights the need for a more concerted and broad-based approach to increase the use of a measure that is effective, inexpensive and easily carried out.

Identifying the policy implications of competency-based education.

At their 2009 consensus conference, the International CBME Collaborators proposed a number of central tenets of CBME in order to advance the field of medical education. Although the proposed conceptualization of CBME offers several advantages and opportunities, including a greater emphasis on outcomes, a mechanism for the promotion of learner-centred curricula, and the potential to move away from time-based training and credentialing in medicine, it is also associated with several significant barriers to adoption. This paper examines the concepts of CBME through a broad educational policy lens, identifying considerations for medical education leaders, health care institutions, and policy-makers at both the meso (program, institutional) and macro (health care system, inter-jurisdictional, and international) levels. Through this analysis, it is clear that CBME is associated with a number of complex challenges and questions, and cannot be considered in isolation from the complex systems in which it functions. Much more work is needed to engage stakeholders in dialogue, to debate the issues, and to identify possible solutions.

Competency-based medical education: theory to practice.

Although competency-based medical education (CBME) has attracted renewed interest in recent years among educators and policy-makers in the health care professions, there is little agreement on many aspects of this paradigm. We convened a unique partnership - the International CBME Collaborators - to examine conceptual issues and current debates in CBME. We engaged in a multi-stage group process and held a consensus conference with the aim of reviewing the scholarly literature of competency-based medical education, identifying controversies in need of clarification, proposing definitions and concepts that could be useful to educators across many jurisdictions, and exploring future directions for this approach to preparing health professionals. In this paper, we describe the evolution of CBME from the outcomes movement in the 20th century to a renewed approach that, focused on accountability and curricular outcomes and organized around competencies, promotes greater learner-centredness and de-emphasizes time-based curricular design. In this paradigm, competence and related terms are redefined to emphasize their multi-dimensional, dynamic, developmental, and contextual nature. CBME therefore has significant implications for the planning of medical curricula and will have an important impact in reshaping the enterprise of medical education. We elaborate on this emerging CBME approach and its related concepts, and invite medical educators everywhere to enter into further dialogue about the promise and the potential perils of competency-based medical curricula for the 21st century.

Competency-based resident education-The Canadian perspective.

Canada has been a leader in competency-based medical education for some years. Postgraduate training programs are typically 5 years in duration with opportunities to pursue 2-year subspecialty training after certification in a primary specialty. The introduction of competency-based models in Canada has progressed from a single orthopedic surgery training program at the University of Toronto through the adoption of competency-based medical education in 29 training programs at a single medical school, and the implementation across all 68 disciplines overseen by the Royal College of Physicians and Surgeons of Canada. This article outlines the introduction of competency-based medical education in postgraduate medical education in Canada.

Developing the Blueprint for a General Surgery Technical Skills Certification Examination: A Validation Study.

INTRODUCTION: There is a recognized need to develop high-stakes technical skills assessments for decisions of certification and resident promotion. High-stakes examinations requires a rigorous approach in accruing validity evidence throughout the developmental process. One of the first steps in development is the creation of a blueprint which outlines the potential content of examination. The purpose of this validation study was to develop an examination blueprint for a Canadian General Surgery assessment of technical skill certifying examination. METHODS: A Delphi methodology was used to gain consensus amongst Canadian General Surgery program directors as to the content (tasks or procedures) that could be included in a certifying Canadian General Surgery examination. Consensus was defined a priori as a Cronbach's α ≥ 0.70. All procedures or tasks reaching a positive consensus (defined as ≥80% of program directors rated items as ≥4 on the 5-point Likert scale) were then included in the final examination blueprint. RESULTS: Two Delphi rounds were needed to reach consensus. Of the 17 General Surgery Program directors across the country, 14 (82.4%) and 10 (58.8%) program directors responded to the first and second round, respectively. A total of 59 items and procedures reached positive consensus and were included in the final examination blueprint. CONCLUSIONS: The present study has outlined the development of an examination blueprint for a General Surgery certifying examination using a consensus-based methodology. This validation study will serve as the foundational work from which simulated model will be developed, pilot tested and evaluated.

The association between a surgeon's learning curve with endovascular aortic aneurysm repair and previous institutional experience.

The purpose of the present study was to determine whether an institution's prior endovascular experience influenced the learning curve of subsequent surgeons. A prospective analysis of the initial 70 endovascular abdominal aortic aneurysm repair (EVAR) cases attempted by an individual surgeon was performed with the primary outcome variable being achievement and 30-day maintenance of initial clinical success. Along with standard statistical analyses, the cumulative sum failure method (CUSUM) was used to analyze the learning curve, with a predetermined acceptable failure rate of 10%. Seventy elective EVAR cases were performed by this surgeon during a 4-year period (2000-2004) (mean age, 73.7 -/+ 5.4 years; mean aneurysm diameter 63.3 -/+ 7.2 mm). Initial clinical success was achieved in 68 of 70 cases (97%), which differed significantly with that of our initial surgeon (88.5%, P = .01). Causes of failure in the present series included 1 early mortality (1.4%) and 1 case of conversion to open repair with no instances of type I endoleak or endograft limb thrombosis. Both surgeons' cases were plotted sequentially with CUSUM curves revealing a significantly shorter learning curve for the second surgeon. Optimal results were achieved following 10 to 20 EVAR cases, as opposed to 60 cases in the initial series. Such an analysis confirms that as an institution's experience with EVAR increases, an individual surgeon's learning curve shortens considerably.

Endovascular repair of thoracic aortic disease: early and midterm experience.

Thoracic aorta disease remains a challenging problem, and despite improvements, open repair techniques are still associated with significant morbidity and mortality. This is a retrospective review of 53 consecutive patients with thoracic aortic pathology who were treated with endovascular repair between September 1998 and December 2004 at a tertiary-care hospital. Endovascular stent graft placement was performed on 23 elective and 30 emergent patients (34 male patients, mean age 66 years, 21 to 85 years). Completion angiography revealed no endoleak in 47 (89%) patients, a type I endoleak in 4 patients, and a type II endoleak in 2 patients. Operative 30-day mortality for elective aneurysms (n = 22), emergent aneurysms (n = 10), dissection (n = 3), penetrating aortic ulcers (n = 7), and trauma (n = 11) was 0%, 40%, 0%, 29%, and 9%, respectively. In total, 46 (87%) patients survived 30 days, and 36 (78.3%) of the survivors were discharged home free of complications. Two patients (4%) experienced paraplegia. Median follow-up was 22 months (1 to 72 months). Intermediate-term results revealed 41 (89%) patients free of endoleak, stent migration, or aneurysmal expansion. Two (4%) patients required reintervention with an additional stent graft. There were 2 (4%) patients with late aortic-related deaths and four (9%) patients with non-aorticrelated late deaths. Endovascular stent graft placement for thoracic aorta disease can be performed successfully and safely with good perioperative and intermediate-term outcomes. Stent graft complication and reintervention rates are low, whereas intermediate survival rates are good. Long-term efficacy still needs to be evaluated.

Midterm results of endovascular infrarenal abdominal aortic aneurysm repair in high-risk patients.

Short-term and midterm clinical outcomes after endovascular repair of abdominal aortic aneurysms (AAAs) have been well documented. Evaluation of longer term outcomes is now possible. Here we describe our initial 100 high-risk patients treated with endovascular aneurysm repair (EVAR), all with a minimum of 5 years of follow-up. A retrospective review of prospectively recorded data in a departmental database was undertaken for the first 100 consecutive EVAR patients with a minimum of 5 years (range, 60-105 months) of follow-up performed between December 1997 and June 2001. Information was obtained from surgical follow-up visits and family doctors' offices. Endovascular repair of AAA in high-risk patients can be achieved with acceptably low postoperative mortality and morbidity. Longer term results in this high-risk cohort suggest that EVAR is effective in preventing aneurysm-related deaths at 5 years and beyond. All late mortalities were due to patients' comorbid diseases.

Strategies for increasing the feasibility of performance assessments during competency-based education: Subjective and objective evaluations correlate in the operating room.

BACKGROUND: Competency-based education necessitates assessments that determine whether trainees have acquired specific competencies. The evidence on the ability of internal raters (staff surgeons) to provide accurate assessments is mixed; however, this has not yet been directly explored in the operating room. This study's objective is to compare the ratings given by internal raters vs an expert external rater (independent to the training process) in the operating room. METHODS: Raters assessed general surgery residents during a laparoscopic cholecystectomy for their technical and nontechnical performance. RESULTS: Fifteen cases were observed. There was a moderately positive correlation (rs = .618, P = .014) for technical performance and a strong positive correlation (rs = .731, P = .002) for nontechnical performance. The internal raters were less stringent for technical (mean rank 3.33 vs 8.64, P = .007) and nontechnical (mean rank 3.83 vs 8.50, P = .01) performances. CONCLUSIONS: This study provides evidence to help operationalize competency-based assessments.

Optimizing the Selection of General Surgery Residents: A National Consensus.

BACKGROUND: Surgical programs strive to recruit trainees who will graduate as competent surgeons; however, selection processes vary between institutions. The purpose of the present study was to (1) solicit program directors' (PDs) opinions on the proportion of trainees who have difficulty achieving competence and (2) establish consensus on the desired attributes of general surgery (GS) candidates and the technical skills that would be most indicative of future performance. METHODS: Delphi consensus methodology was used. An open-ended questionnaire, followed by a closed-ended questionnaire, formulated as a 5-point Likert scale, was administered. A Cronbach α ≥ 0.8 with 80% of responses in agreement (4-agree and 5-strongly agree) determined the threshold for consensus. RESULTS: The first and second rounds were completed by 14 and 11, of a potential 17, GS PDs, respectively. PDs felt that 5% or less of trainees have difficulty reaching competence in clinical knowledge, 5% to 10% in decision-making, and 5% to 15% in technical skill by the time of completion of training. Consensus was excellent (α = 0.92). The top attributes for success in GS included work ethic and passion for surgery. Technical skills that felt to be most appropriate were open tasks (one-handed tie and subcuticular suture) and laparoscopic tasks (coordination, grasping, and cutting). CONCLUSION: PDs indicate that of the 3 domains, the largest proportion of trainees had difficulty reaching competence in technical skill. Consensus among PDs suggests that top personal attributes include work ethic and passion for surgery. Consensus of technical tasks for inclusion into selection was basic open and laparoscopic skills.

Inhalation of carbon monoxide prevents liver injury and inflammation following hind limb ischemia/reperfusion.

The induction of heme oxygenase (HO), the rate limiting enzyme in the conversion of heme into carbon monoxide (CO) and biliverdin, limits liver injury following remote trauma such as hind limb ischemia/reperfusion (I/R). Using intravital video microscopy, we tested the hypothesis that inhaled CO (250 ppm) would mimic HO-derived liver protection. Hind limb I/R significantly decreased sinusoidal diameter and volumetric flow, increased leukocyte accumulation within sinusoids, increased leukocyte rolling and adhesion within postsinusoidal venules, and significantly increased hepatocyte injury compared with naive animals. Inhalation of CO alone did not alter any microcirculatory or inflammatory parameters. Inhalation of CO following I/R restored volumetric flow, decreased stationary leukocytes within sinusoids, decreased leukocyte rolling and adhesion within postsinusoidal venules, and significantly reduced hepatocellular injury following hind limb I/R. HO inhibition did not alter microcirculatory parameters in naive mice, but did increase inflammation, as well as increase hepatocyte injury following hind limb I/R. Inhalation of CO during HO inhibition significantly reduced such microcirculatory deficits, hepatic inflammation, and injury in response to hind limb I/R. In conclusion, these results suggest that HO-derived hepatic protection is mediated by CO, and inhalation of low concentrations of CO may represent a novel therapeutic approach to prevent remote organ injury during systemic inflammatory response syndrome, or SIRS.