Usability of ENTRUST as an Assessment Tool for Entrustable Professional Activities (EPAs): A Mixed Methods Analysis.

OBJECTIVE: As the American Board of Surgery transitions to a competency-based model of surgical education centered upon entrustable professional activities (EPAs), there is a growing need for objective tools to determine readiness for entrustment. This study evaluates the usability of ENTRUST, an innovative virtual patient simulation platform to assess surgical trainees' decision-making skills in preoperative, intra-operative, and post-operative settings. DESIGN: This is a mixed-methods analysis of the usability of the ENTRUST platform. Quantitative data was collected using the system usability scale (SUS) and Likert responses. Analysis was performed with descriptive statistics, bivariate analysis, and multivariable linear regression. Qualitative analysis of open-ended responses was performed using the Nielsen-Shneiderman Heuristics framework. SETTING: This study was conducted at an academic institution in a proctored exam setting. PARTICIPANTS: The analysis includes n = 47 (PGY 1-5) surgical residents who completed an online usability survey following the ENTRUST Inguinal Hernia EPA Assessment. RESULTS: The ENTRUST platform had a median SUS score of 82.5. On bivariate and multivariate analyses, there were no significant differences between usability based on demographic characteristics (all p > 0.05), and SUS score was independent of ENTRUST performance (r = 0.198, p = 0.18). Most participants agreed that the clinical workup of the patient was engaging (91.5%) and felt realistic (85.1%). The most frequent heuristics represented in the qualitative analysis included feedback, visibility, match, and control. Additional themes of educational value, enjoyment, and ease-of-use highlighted participants' perspectives on the usability of ENTRUST. CONCLUSIONS: ENTRUST demonstrates high usability in this population. Usability was independent of ENTRUST score performance and there were no differences in usability identified in this analysis based on demographic subgroups. Qualitative analysis highlighted the acceptability of ENTRUST and will inform ongoing development of the platform. The ENTRUST platform holds potential as a tool for the assessment of EPAs in surgical residency programs.

Correlation of Performance on ENTRUST and Traditional Oral Objective Structured Clinical Examination for High-Stakes Assessment in the College of Surgeons of East, Central, and Southern Africa.

BACKGROUND: To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence. STUDY DESIGN: In partnership with the College of Surgeons of East, Central, and Southern Africa, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n = 110) completed the traditional 11-station oral objective structured clinical examinations (OSCEs), followed by 3 ENTRUST cases, authored to query similar clinical content of 3 corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Examination outcome using independent sample t tests. Correlation of ENTRUST scores to MCS Examination Percentage and OSCE station scores was calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance. RESULTS: ENTRUST performance was significantly higher in examinees who passed the MCS examination compared with those who failed (p < 0.001). The ENTRUST score was positively correlated with MCS Examination Percentage (p < 0.001) and combined OSCE station scores (p < 0.001). On multivariate analysis, there was a strong association between MCS Examination Percentage and ENTRUST Grand Total Score (p < 0.001), Simulation Total Score (p = 0.018), and Question Total Score (p < 0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST. CONCLUSIONS: This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.

Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA).

OBJECTIVE: As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence. DESIGN: This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses. SETTING: This study was conducted at an academic institution in a proctored exam setting. PARTICIPANTS: Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment. RESULTS: Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56). CONCLUSIONS: Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

Diversity, equity, and inclusion in presidential leadership of academic medical and surgical societies.

BACKGROUND: Our aim was to identify gender and racial disparities in presidential leadership for national medical and surgical organizations. METHODS: We located publicly sourced information on national medical organizations. Years between or since the first diverse presidents were analyzed using descriptive statistics and Mann Whitney U tests. RESULTS: Sixty-seven national medical and surgical organizations were surveyed. 70.8% (n = 34) diversified via gender first (White-female), whereas 26.1% (n = 14) had racial diversity first. Organizations with gender diversity first followed with an African American male president sooner than organizations who first diversified by race (14.7 ± 11.8 v. 27.6 ± 11.3 years, p = 0.018). No significant difference was observed for the third tier of diversification. CONCLUSIONS: Significant gender and racial leadership disparities in national medical organizations are still present. It is notable that organizations with female leaders had a shorter timeline to racial diversity. These findings help to inform strategies to promote and increase diversity, equity, and inclusion in national leadership.

Peer Assessment in Medical Student Education: A Study of Feasibility, Benefit, and Worth.

BACKGROUND: Direct experience with medical procedures is an important component of medical school training, yet opportunities for medical students have dwindled for various reasons. To offset this, simulated procedures are being integrated into training. However, this comes with additional time commitments required of teaching surgeons regarding assessment of simulation. A solution to this could be peer assessment. We hypothesize that there will be no significant difference between peer assessment when compared to that of a teaching surgeon. METHODS: Third-year medical students were shown 3 simulated procedures by teaching surgeon and provided a grading rubric. Student performances were independently graded by peer assessment and by teaching surgeons. All peer assessment grades and surgeon grades were compared. RESULTS: Four hundred fifty-nine medical students completed the simulation procedures. Comparisons between the teaching surgeons and peer assessment evaluations demonstrated a 99% interobserver agreement for pass-fail designation and 98% agreement for individual data points (kappa = .78). Survey results demonstrated a significant increase in confidence in performing the tested items and comfort with peer assessment. DISCUSSION: This analysis demonstrates that the inclusion of peer assessment within medical school is highly comparable to teaching surgeon assessments.

Diversity, Equity, and Inclusion: A Current Analysis of General Surgery Residency Programs.

BACKGROUND: Local, regional, and national diversity, equity, and inclusion (DEI) initiatives have been established to combat barriers to entry and promote retention in surgery residency programs. Our study evaluates changes in diversity in general surgery residency programs. We hypothesize that diversity trends have remained stable nationally and regionally. MATERIALS AND METHODS: General surgery residents in all postgraduate years were queried regarding their self-reported sex, race, and ethnicity following the 2020 ABSITE. Residents were then grouped into geographic regions. Data were analyzed utilizing descriptive statistics, Kruskal-Wallis test, and chi-square analyses. RESULTS: A total of 9276 residents responded. Nationally, increases in female residents were noted from 38.0 to 46.0% (P < .001) and in Hispanic or Latinx residents from 7.3 to 8.3% (P = .031). Across geographic regions, a significant increase in female residents was noted in the Northwest (51.9 to 58.3%, P = .039), Midwest (36.9 to 43.3%, P = .006), and Southwest (35.8 to 47.5%, P = .027). A significant increase in black residents was only noted in the Northwest (0 to 15.8%, P = .031). The proportion of white residents decreased nationally by 8.9% and in the Mid-Atlantic, Southeast, and Southwest between 5.5 and 15.9% (P < .05). DISCUSSION: In an increasingly diverse society, expanding the numbers of underrepresented surgeons in training, and ultimately in practice, is a necessity. This study shows that there are region-specific increases in diversity, despite minimal change on a national level. This finding may suggest the need for region-specific DEI strategies and initiatives. Future studies will seek to evaluate individual programs with DEI plans and determine if there is a correlation to changing demographics.

Validity Evidence for Vascular Skills Assessment: The Feasibility of Fundamentals of Vascular Surgery in General Surgery Residency.

OBJECTIVE: As the Fundamentals of Laparoscopic Surgery (FLS) and Fundamentals of Endoscopic Surgery (FES) have been used for general surgery assessment, the Fundamentals of Vascular Surgery (FVS) has recently been developed to evaluate core operative skills for vascular trainees. This study examines the 3 year implementation of FVS for general surgery residents and it gathers validity evidence using Messick's framework. We hypothesized that the curriculum and assessment tool enhance general surgery resident training and assessment. DESIGN: This is a retrospective review of FVS assessments of residents using descriptive and multivariate analyses. SETTING: This study was conducted at an academic institution, where simulation-based teaching sessions occur in coordination between the general surgery and the integrated vascular surgery residency programs. PARTICIPANTS: Seventeen general surgery residents were assessed in FVS skills by an expert rater from 2018 to 2020. RESULTS: Overall, 86 assessments were completed. CONTENT: Assessment focuses on 3 open vascular skills (End-to-Side Anastomosis, Patch Angioplasty and Clockface Suturing). Response Process: 7 items comprise a graded rating for a skills score. Additionally, a global summary score is designated. Internal Structure: The assessment tool has a Cronbach's alpha of 0.87, demonstrating good internal consistency. Addition of the second rater correlated with Cohen's kappa -0.69 (p < 0.001), indicating poor interrater reliability. Relationships to other variables: The most significant improvement occurred in total scores between PGY2s (17.4 ± 2.37) and PGY4s (23.2 ± 3.00), p < 0.001, indicating adequate level discernment. CONCLUSIONS: The validity evidence of FVS assessment in this study supports its use in general surgery residency at a time when opportunities for open vascular skills assessment may be decreasing due to case availability and shifting paradigms. Further study into quality rater training is needed to optimize national implementation of FVS and ensure consistency in grading.

Isolating steps instead of learners: Use of deliberate practice and validity evidence in coronavirus disease (COVID)-era procedural assessment.

BACKGROUND: In surgical training, assessment tools based on strong validity evidence allow for standardized evaluation despite changing external circumstances. At a large academic institution, surgical interns undergo a multimodal curriculum for central line placement that uses a 31-item binary assessment at the start of each academic year. This study evaluated this practice within increased in-person learning restrictions. We hypothesized that external constraints would not affect resident performance nor assessment due to a robust curriculum and assessment checklist. METHODS: From 2018 to 2020, 81 residents completed central line training and assessment. In 2020, this curriculum was modified to conform to in-person restrictions and social distancing guidelines. Resident score reports were analyzed using multivariate analyses to compare performance, objective scoring parameters, and subjective assessments among "precoronavirus disease" years (2018 and 2019) and 2020. RESULTS: There were no significant differences in average scores or objective pass rates over 3 years. Significant differences between 2020 and precoronavirus disease years occurred in subjective pass rates and in first-time success for 4 checklist items: patient positioning, draping, sterile ultrasound probe cover placement, and needle positioning before venipuncture. CONCLUSION: Modifications to procedural training within current restrictions did not adversely affect residents' overall performance. However, our data suggest that in 2020, expert trainers may not have ensured learner acquisition of automated procedural steps. Additionally, although 2020 raters could have been influenced by logistical barriers leading to more lenient grading, the assessment tool ensured training and assessment integrity.

From Listening to Action: Academic Surgical Departmental Response to Social Injustice Through Curricular Development.

OBJECTIVE: The aim of this study was to describe the development and evaluation of a structured department wide cultural competency curriculum. SUMMARY BACKGROUND DATA: Despite numerous organizational policies and statements, social injustice and bias still exist. Our department committed to assist individuals of the entire department to develop foundational knowledge and skills to combat implicit bias and systemic racism through the creation of a cultural competency curriculum. The purpose of this manuscript is to detail our curriculum and the evaluation of its effectiveness. METHODS: Using a well-established curriculum development framework, a cultural competency curriculum was developed focusing on knowledge, skills and attitudes at the individual level, for all members of the department. The curriculum was implemented through 6-hour-long sessions over a 9-week period. Effectiveness was assessed through a post curriculum survey. RESULTS: Twenty percent of the respondents had experienced bias based on race, ethnicity, or sexual orientation in the past 12 months, whereas 30% had experienced bias based on sex. Seventy-one percent independently explored related topics. The curriculum was overall well received and generally achieved the goals and objectives. CONCLUSION: Using a standard curriculum development framework, an effective department-wide cultural competency curriculum can be developed and implemented.

The what? How? And Who? Of video based assessment.

BACKGROUND: Currently, there is significant variability in the development, implementation and overarching goals of video review for assessment of surgical performance. METHODS: This paper evaluates the current methods in which video review is used for evaluation of surgical performance and identifies which processes are critical for successful, widespread implementation of video-based assessment. RESULTS: Despite the advances in video capture technology and growing interest in video-based assessment, there is a notable gap in the implementation and longitudinal use of formative and summative assessment using video. CONCLUSION: Validity, scalability and discoverability are current but removable barriers to video-based assessment.

The Change in Surgical Case Diversity Over the Past 15 Years and the Influence on the Pursuit of Surgical Fellowship.

The surgical community has expressed concern that residents do not receive the same caliber training as their predecessors and the increase in fellowships have been described as secondary to perceived lack of preparation. Yet, data show no change in total cases even after implementation of the 80-hour workweek. It is hypothesized that the increasing subspecialization of general surgery may decrease in certain resident case numbers. Data were collected from the Accreditation Council for Graduate Medical Education (ACGME) General Surgery Case Logs National Data Report (1999-2014) of mean number of procedures per resident for 19 surgical categories. Statistical analysis was performed with analysis of variance over three time periods between 1999 and 2014. The number of total cases performed by residents has not changed significantly. There was a statistically significant difference observed in the variety of cases: vascular, esophageal, breast, and trauma cases decreased (P < 0.01), whereas major intestinal, hernia, liver, pancreatic, and biliary cases increased (P < 0.01). There are many reasons to pursue additional training after residency. The demonstrated change in case variability, presumably secondary to increasing fellowships, may play a significant role on training and preparation. Close monitoring of curriculums is essential to ensure a comprehensive general surgical education.

The Economics of Private Practice versus Academia in Surgery.

OBJECTIVE: Residents often make career decisions regarding future practice without adequate knowledge to the realities of professional life. Currently there is a paucity of data regarding economic differences between practice models. This study seeks to illuminate the financial differences of surgical subspecialties between academic and private practice. DESIGN: Data were collected from the Association of American Medical College (AAMC) and the Medical Group Management Association's (MGMA) 2015 reports of average annual salaries. Salaries were analyzed for general surgery and 7 subspecialties. Fixed time of practice was set at 30 years. Assumptions included 5 years as assistant professor, 10 years as associate professor, and 15 years as full professor. Formula used: (average yearly salary) × [years of practice (30 yrs - fellowship/research yrs)] + ($50,000 × yrs of fellowship/research) = total adjusted lifetime revenue. RESULTS: As a full professor, academic surgeons in all subspecialties make significantly less than their private practice counterparts. The largest discrepancy is in vascular and cardiothoracic surgery, with full professors earning 16% and 14% less than private practitioners. Plastic surgery and general surgery are the only 2 disciplines that have similar lifetime revenues to private practitioners, earning 2% and 6% less than their counterparts' lifetime revenue. CONCLUSIONS: Academic surgeons in all surgical subspecialties examined earn less lifetime revenue compared to those in private practice. This difference in earnings decreases but remains substantial as an academic surgeon advances. With limited exposure to the diversity of professional arenas, residents must be aware of this discrepancy.

Time crunch: increasing the efficiency of assessment of technical surgical skill via brief video clips.

BACKGROUND: Video review for assessment of surgical performance is gaining popularity but is time consuming for busy expert reviewers, making review delays inevitable. Decreasing review time and including nonexpert reviewers may facilitate more timely reviews. We hypothesized that a shorter duration video clip would not affect the quality of expert ratings compared with full-length review. A secondary aim was to examine the reliability between expert and novice raters and how it was affected by video clip duration. METHODS: Videos of laparoscopic suturing performed on a live porcine model by premedical students, surgery residents, and fully trained surgeons were edited into 3 different durations: full, part, and 30-second versions. Video clips (n = 36) were reviewed by experienced surgeons (n = 3) and novice volunteers (n = 4) using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) assessment. Videos were assigned randomly during 3 review cycles held 1 week apart. Each cycle included 1 iteration of the 12 performance videos. We assessed the impact of the duration of the video on reviewer scores and inter-rater reliability. RESULTS: Variance in scores for 2 of 4 GOALS domains was dependent on the duration of the video clip (P < .05). Total GOALS scores were greater for part and 30-second clips compared with full clips (P < .05). Inter-rater reliability was greatest for full clips (intraclass correlation = 0.68) and decreased significantly for shorter clips. CONCLUSION: Our hypothesis was rejected as shorter video durations for surgical performance assessment led to inflated reviewer ratings both for expert and novice reviewers. Shortening duration of the video cannot be recommended for accurate performance assessment.

Developing a robust suturing assessment: validity evidence for the intracorporeal suturing assessment tool.

BACKGROUND: Assessment tools specific to intracorporeal suturing are lacking. The purpose of this study was to validate a novel Intracorporeal Suturing Assessment Tool (ISAT) by comparing it with existing measures that have been reported to have validity evidence. METHODS: Videos of laparoscopic suturing were assessed by 3 blinded laparoscopic experts using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) scale and the ISAT. Correlations between these instruments were calculated, and sensitivity analyses compared both tools with objective suturing scores. A factor analysis was also performed. RESULTS: The ISAT and GOALS ratings were significantly correlated with the objective suturing score (r = 0.58 and 0.61, respectively; P < .0001), and with each other (r = 0.92, P < .0001). A weighted κ test indicated significantly higher agreement than expected between these instruments (P < .0001). All ISAT items had a factor loading approaching or greater than 0.50. CONCLUSION: The ISAT accurately assessed laparoscopic suturing skill related to other instruments. ISAT was highly correlated with GOALS, which is often used for laparoscopic performance assessment. Unlike the generic GOALS, ISAT includes specific information that can provide feedback on trainee suturing ability and targeted performance improvements. ISAT may provide a better alternative for intracorporeal suturing assessment.

The Economics of Academic Advancement Within Surgery.

BACKGROUND: The success of an academic surgeon's career is often viewed as directly related to academic appointment; therefore, the sequence of promotion is a demanding, rigorous process. This paper seeks to define the financial implication of academic advancement across different surgical subspecialties. STUDY DESIGN: Data was collected from the Association of American Medical College's 2015 report of average annual salaries. Assumptions included 30 years of practice, 5 years as assistant professor, and 10 years as associate professor before advancement. The base formula used was: (average annual salary) × (years of practice [30 years - fellowship/research years]) + ($50,000 × years of fellowship/research) = total adjusted lifetime salary income. RESULTS: There was a significant increase in lifetime salary income with advancement from assistant to associate professor in all subspecialties when compared to an increase from associate to full professor. The greatest increase in income from assistant to associate professor was seen in transplant and cardiothoracic surgery (35% and 27%, respectively). Trauma surgery and surgical oncology had the smallest increases of 8% and 9%, respectively. With advancement to full professor, the increase in lifetime salary income was significantly less across all subspecialties, ranging from 1% in plastic surgery to 8% in pediatric surgery. CONCLUSION: When analyzing the economics of career advancement in academic surgery, there is a substantial financial benefit in lifetime income to becoming an associate professor in all fields; whereas, advancement to full professor is associated with a drastically reduced economic benefit.

Using Simulation to Improve Systems-Based Practices.

BACKGROUND: Ensuring the safe, effective management of patients requires efficient processes of care within a smoothly operating system in which highly reliable teams of talented, skilled health care providers are able to use the vast array of high-technology resources and intensive care techniques available. Simulation can play a unique role in exploring and improving the complex perioperative system by proactively identifying latent safety threats and mitigating their damage to ensure that all those who work in this critical health care environment can provide optimal levels of patient care. METHODS: A panel of five experts from a wide range of institutions was brought together to discuss the added value of simulation-based training for improving systems-based aspects of the perioperative service line. Panelists shared the way in which simulation was demonstrated at their institutions. The themes discussed by each panel member were delineated into four avenues through which simulation-based techniques have been used. RESULTS: Simulation-based techniques are being used in (1) testing new clinical workspaces and facilities before they open to identify potential latent conditions; (2) practicing how to identify the deteriorating patient and escalate care in an effective manner; (3) performing prospective root cause analyses to address system weaknesses leading to sentinel events; and (4) evaluating the efficiency and effectiveness of the electronic health record in the perioperative setting. CONCLUSION: This focused review of simulation-based interventions to test and improve components of the perioperative microsystem, which includes literature that has emerged since the panel's presentation, highlights the broad-based utility of simulation-based technologies in health care.

Is it All About the Money? Not All Surgical Subspecialization Leads to Higher Lifetime Revenue when Compared to General Surgery.

OBJECTIVE: It is believed that spending additional years gaining expertise in surgical subspecialization leads to higher lifetime revenue. Literature shows that more surgeons are pursuing fellowship training and dedicated research years; however, there are no data looking at the aggregate economic impact when training time is accounted for. It is hypothesized that there will be a discrepancy in lifetime income when delay to practice is considered. DESIGN: Data were collected from the Medical Group Management Association's 2015 report of average annual salaries. Fixed time of practice was set at 30 years, and total adjusted revenue was calculated based on variable years spent in research and fellowship. All total revenue outcomes were compared to general surgery and calculated in US dollars. PARTICIPANTS: The financial data on general surgeons and 9 surgical specialties (vascular, pediatric, plastic, breast, surgical oncology, cardiothoracic, thoracic primary, transplant, and trauma) were examined. RESULTS: With fellowship and no research, breast and surgical oncology made significantly less than general surgery (-$1,561,441, -$1,704,958), with a difference in opportunity cost equivalent to approximately 4 years of work. Pediatric and cardiothoracic surgeons made significantly more than general surgeons, with an increase of opportunity cost equivalent to $5,301,985 and $3,718,632, respectively. With 1 research year, trauma surgeons ended up netting less than a general surgeon by $325,665. With 2 research years, plastic and transplant surgeons had total lifetime revenues approximately equivalent to that of a general surgeon. CONCLUSIONS: Significant disparities exist in lifetime total revenue between surgical subspecialties and in comparison, to general surgery. Although most specialists do gross more than general surgeons, breast and surgical oncologists end up netting significantly less over their lifetime as well as trauma surgeons if they do 1 year of research. Thus, the economic advantage of completing additional training is dependent on surgical field and duration of research.

Multicenter longitudinal assessment of resident technical skills.

BACKGROUND: Our aim was to report the longitudinal assessment of technical performance of general surgery residents on select tasks from multiple programs over a 2-year period. METHODS: An institutional review board-approved, multi-institutional collaborative study was undertaken with yearly resident performance assessments over a 3-year period. General surgery residents (postgraduate year [PGY] 1 to 5) were tested on 3 laparoscopic and 5 open simulated surgical tasks. Resident performance was compared individually over time and among interns and more senior residents. RESULTS: Forty-one residents from 4 residency programs were evaluated. Scores increased in all tasks with each assessment, plateauing at a lower PGY level for open tasks compared with laparoscopic tasks. Change in performance scores between assessments were higher for interns compared with more senior residents (P < .003). CONCLUSIONS: Resident performance on basic open and laparoscopic tasks assessed over time improved the most between the PGY 1 and 2 levels and was dependent on task difficulty. This documented skill evolution may allow tailoring of skills curricula to both meet existing needs and minimize performance variability.

Using simulation to improve root cause analysis of adverse surgical outcomes.

OBJECTIVE: The purpose of this study was to develop and test a simulation method of conducting investigation of the causality of adverse surgical outcomes. DESIGN: Six hundred and thirty-one closed claims of a major medical malpractice insurance company were reviewed. Each case had undergone conventional root cause analysis (RCA). Claims were categorized by comparing the predominant underlying cause documented in the case files. Three cases were selected for simulation. SETTING: All records (medical and legal) were analyzed. Simulation scenarios were developed by abstracting data from the records and then developing paper and electronic medical records, choosing appropriate STUDY PARTICIPANTS: including test subjects and confederates, scripting the simulation and choosing the appropriate simulated environment. INTERVENTION: In a simulation center, each case simulation was run 6-7 times and recorded, with participants debriefed at the conclusion. MAIN OUTCOME MEASURES: Sources of error identified during simulation were compared with those noted in the closed claims. Test subject decision-making was assessed qualitatively. RESULTS: Simulation of adverse outcomes (SAOs) identified more system errors and revealed the way complex decisions were made by test subjects. Compared with conventional RCA, SAO identified root causes less focused on errors by individuals and more on systems-based error. CONCLUSIONS: The use of simulation for investigation of adverse surgical outcomes is feasible and identifies causes that may be more amenable to effective systems changes than conventional RCA. The information that SAO provides may facilitate the implementation of corrective measures, decreasing the risk of recurrence and improving patient safety.

Development and verification of a taxonomy of assessment metrics for surgical technical skills.

PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.