Opportunities and Applications of Educational Technologies in Surgical Education and Assessment.

BACKGROUND: Surgical education is challenged by continuously increasing clinical content, greater subspecialization, and public scrutiny of access to high quality surgical care. Since the last Blue Ribbon Committee on surgical education, novel technologies have been developed including artificial intelligence and telecommunication. OBJECTIVES AND METHODS: The goals of this Blue Ribbon Sub-Committee were to describe the latest technological advances and construct a framework for applying these technologies to improve the effectiveness and efficiency of surgical education and assessment. An additional goal was to identify implementation frameworks and strategies for centers with different resources and access. All sub-committee recommendations were included in a Delphi consensus process with the entire Blue Ribbon Committee (N=67). RESULTS: Our sub-committee found several new technologies and opportunities that are well poised to improve the effectiveness and efficiency of surgical education and assessment (see Tables 1-3). Our top recommendation was that a Multidisciplinary Surgical Educational Council be established to serve as an oversight body to develop consensus, facilitate implementation, and establish best practices for technology implementation and assessment. This recommendation achieved 93% consensus during the first round of the Delphi process. CONCLUSION: Advances in technology-based assessment, data analytics, and behavioral analysis now allow us to create personalized educational programs based on individual preferences and learning styles. If implemented properly, education technology has the promise of improving the quality and efficiency of surgical education and decreasing the demands on clinical faculty.

Comprehensive Literature Search to Identify Assessment Tools for Operating Room Nontechnical Skills to Determine Common Critical Components.

BACKGROUND: Effective use of nontechnical skills (NTS) contributes to the provision of safe, quality care in the fast-paced, dynamic setting of the operating room (OR). Inter-professional education of NTS to OR team members can improve performance. Such training requires the accurate measurement of NTS in order to identify gaps in their utilization by OR teams. Although several instruments for measuring OR NTS exist in the literature, each tool tends to define specific NTS differently. AIM: We aimed to determine commonalities in defined measurements among existing OR NTS tools. METHODS: We undertook a comprehensive literature review of assessment tools for OR NTS to determine the critical components common to these instruments. A PubMed search of the literature from May 2009 to May 2019 combined various combinations of keywords and Medical Subject Headings (MeSH) related to the following subjects: teamwork, teams, assessment, debriefing, surgery, operating room, nontechnical, communication. From this start, articles were selected describing specific instruments. Three reviewers then identified the common components measured among these assessment tools. Reviewers collated kin constructs within each instrument using frequency counts of similarly termed and conceptualized components. RESULTS: The initial PubMed search produced 119 articles of which 24 articles satisfied the inclusion criteria. Within these articles, 10 assessment tools evaluated OR NTS. Kin constructs were grouped into six NTS categories in the following decreasing frequency order: communication, situation awareness, teamwork, leadership, decision making, and task management/decision making (equal). CONCLUSION: NTS OR assessment tools in the literature have a variety of kin constructs related to the specific measured components within the instruments. Such kin constructs contain thematic cohesion across six primary NTS groupings with some variation in scale and scope. Future plans include using this information to develop an easy-to-use assessment tool to assist with debriefing in the clinical environment.

Improvement in student-led debriefing analysis after simulation-based team training using a revised teamwork assessment tool.

BACKGROUND: Team debriefing is an important teamwork development intervention for improving team outputs in healthcare. Debriefing is a key component of experiential team training teamwork development interventions such as simulation-based training. Improving the quality of debriefing of healthcare teams, therefore, has multiple benefits. We investigated whether the quality of student-led debriefing improved using a shortened guide. METHODS: Senior medical students, nurse anesthesia students, and senior undergraduate nursing students participated in student operating room team training at a health sciences center in the southeastern United States. Student teams participated in a dual-scenario simulation-based training session with immediate after-action debriefings after each scenario. In 2018, student teams conducted the second debriefing using as a guide the teamwork assessment scale, an 11-item, 3-subscale, 6-point Likert-type instrument. In 2019, they used a shortened, revised, 5-item version of the teamwork assessment scale, the quick teamwork assessment scale. Trained observers rated the quality of the student-led debriefings using the Objective Structured Assessment of Debriefing, an 8-item, 5-point instrument. The Wilcoxon-Mann-Whitney test was used to compare the teamwork assessment scale-guided and the quick teamwork assessment scale-guided mean item debriefing scores. RESULTS: Two observers rated 3 student-led team debriefings using the teamwork assessment scales as a guide in 2018, and 6 such debriefings happened using the quick teamwork assessment scale as a guide in 2019. For each debriefing, observer scores were averaged for each Objective Structured Assessment of Debriefing item; these mean scores were then averaged with other mean scores for each year. The use of the quick teamwork assessment scale resulted in a statistically significant higher mean score for the Analysis Objective Structured Assessment of Debriefing item compared with the use of the teamwork assessment scale (4.92 [standard deviation 0.20] versus 3.83 [standard deviation 0.76], P = .023). CONCLUSION: The use of a shortened teamwork assessment instrument as a debriefing guide for student teams in student operating room team training was more effective in analysis of actions than the original, longer tool. Next steps include determining the efficacy of the quick teamwork assessment scale in an actual clinical setting.

eAssessment: development of an electronic version of the Objective Structured Assessment of Debriefing tool to streamline evaluation of video recorded debriefings.

Objective: The Objective Structured Assessment of Debriefing (OSAD) is an evidence-based, 8-item tool that uses a behaviorally anchored rating scale in paper-based form to evaluate the quality of debriefing in medical education. The objective of this project was twofold: 1) to create an easy-to-use electronic format of the OSAD (eOSAD) in order to streamline data entry; and 2) to pilot its use on videoed debriefings. Materials and Methods: The eOSAD was developed in collaboration with the LSU Health New Orleans Epidemiology Data Center using SurveyGizmo (Widgix Software, LLC, Boulder, CO, USA) software. The eOSAD was then piloted by 2 trained evaluators who rated 37 videos of faculty teams conducting pre-briefing and debriefing after a high-fidelity trauma simulation. Inter-rater reliability was assessed, and evaluators' qualitative feedback was obtained. Results: Inter-rater reliability was good [prebrief, intraclass correlation coefficient, ICC = 0.955 (95% CI, 0.912-0.977), P < .001; debrief, ICC = 0.853 (95% CI, 0.713-0.924), P < .001]. Qualitative feedback from evaluators found that the eOSAD was easy to complete, simple to read and add comments, and reliably stored data that were readily retrievable, enabling the smooth dissemination of information collected. Discussion: The eOSAD features a secure login, sharable internet access link for distant evaluators, and the immediate exporting of data into a secure database for future analysis. It provided convenience for end-users, produced reliable assessments among independent evaluators, and eliminated multiple sources of possible data corruption. Conclusion: The eOSAD tool format advances the post debriefing evaluation of videoed inter-professional team training in high-fidelity simulation.

Using Simulation to Improve Systems-Based Practices.

BACKGROUND: Ensuring the safe, effective management of patients requires efficient processes of care within a smoothly operating system in which highly reliable teams of talented, skilled health care providers are able to use the vast array of high-technology resources and intensive care techniques available. Simulation can play a unique role in exploring and improving the complex perioperative system by proactively identifying latent safety threats and mitigating their damage to ensure that all those who work in this critical health care environment can provide optimal levels of patient care. METHODS: A panel of five experts from a wide range of institutions was brought together to discuss the added value of simulation-based training for improving systems-based aspects of the perioperative service line. Panelists shared the way in which simulation was demonstrated at their institutions. The themes discussed by each panel member were delineated into four avenues through which simulation-based techniques have been used. RESULTS: Simulation-based techniques are being used in (1) testing new clinical workspaces and facilities before they open to identify potential latent conditions; (2) practicing how to identify the deteriorating patient and escalate care in an effective manner; (3) performing prospective root cause analyses to address system weaknesses leading to sentinel events; and (4) evaluating the efficiency and effectiveness of the electronic health record in the perioperative setting. CONCLUSION: This focused review of simulation-based interventions to test and improve components of the perioperative microsystem, which includes literature that has emerged since the panel's presentation, highlights the broad-based utility of simulation-based technologies in health care.

Learning how to "teach one": A needs assessment of the state of faculty development within the Consortium of the American College of Surgeons Accredited Education Institutes.

BACKGROUND: Developing faculty competencies in curriculum development, teaching, and assessment using simulation is critical for the success of the Consortium of the American College of Surgeons Accredited Education Institutes program. The state of and needs for faculty development in the Accredited Education Institute community are unknown currently. The Faculty Development Committee of the Consortium of the Accredited Education Institutes conducted a survey of Accredited Education Institutes to ascertain what types of practices are used currently, with what frequency, and what needs are perceived for further programs and courses to guide the plan of action for the Faculty Development Committee. METHODS: The Faculty Development Committee created a 20-question survey with quantitative and qualitative items aimed at gathering data about practices of faculty development and needs within the Consortium of Accredited Education Institutes. The survey was sent to all 83 Accredited Education Institutes program leaders via Survey Monkey in January 2015 with 2 follow-up reminders. Quantitative data were compiled and analyzed using descriptive statistics, and qualitative data were interpreted for common themes. RESULTS: Fifty-four out of the 83 programs (65%) responded to the survey. Two-thirds of the programs had from 1 to 30 faculty teaching at their Accredited Education Institutes. More than three-quarters of the programs taught general surgery, emergency medicine, or obstetrics/gynecology. More than 60% of programs had some form of faculty development, but 91% reported a need to expand their offerings for faculty development with "extreme value" for debriefing skills (70%), assessment (47%), feedback (40%), and curriculum development (40%). Accredited Education Institutes felt that the Consortium could assist with faculty development through such activities as the provision of online resources, sharing of best practices, provision of a blueprint for development of a faculty curriculum and information related to available, credible master programs of faculty development and health professions education. CONCLUSION: Many Accredited Education Institutes programs are engaged in faculty development activities, but almost all see great needs in faculty development related to debriefing, assessment, and curricular development. These results should help to guide the action and decision-making of the Consortium Faculty Development Committee to improve teaching within the American College of Surgeons Accredited Education Institutes.

Multicenter longitudinal assessment of resident technical skills.

BACKGROUND: Our aim was to report the longitudinal assessment of technical performance of general surgery residents on select tasks from multiple programs over a 2-year period. METHODS: An institutional review board-approved, multi-institutional collaborative study was undertaken with yearly resident performance assessments over a 3-year period. General surgery residents (postgraduate year [PGY] 1 to 5) were tested on 3 laparoscopic and 5 open simulated surgical tasks. Resident performance was compared individually over time and among interns and more senior residents. RESULTS: Forty-one residents from 4 residency programs were evaluated. Scores increased in all tasks with each assessment, plateauing at a lower PGY level for open tasks compared with laparoscopic tasks. Change in performance scores between assessments were higher for interns compared with more senior residents (P < .003). CONCLUSIONS: Resident performance on basic open and laparoscopic tasks assessed over time improved the most between the PGY 1 and 2 levels and was dependent on task difficulty. This documented skill evolution may allow tailoring of skills curricula to both meet existing needs and minimize performance variability.

What is the future of training in surgery? Needs assessment of national stakeholders.

BACKGROUND: The Curriculum Committee of the American College of Surgeons-Accredited Educational Institutes conducted a need assessment to (1) identify gaps between ideal and actual practices in areas of surgical care, (2) explore educational solutions for addressing these gaps, and (3) shape a vision to advance the future of training in surgery. METHODS: National stakeholders were recruited from the committee members' professional network and interviewed via telephone. Interview questions targeted areas for improving surgical patient care, optimal educational solutions for training in surgery including simulation roles, and entities that should primarily bear training costs. We performed an iterative, qualitative analysis including member checking to identify key themes. RESULTS: Twenty-two interviewees included state/national board representatives, risk managers, multispecialty faculty/program directors, nurses, trainees, an industry representative, and a patient. Surgeons' communication with patients, families, and team members was raised consistently by stakeholders as a way to establish clear expectations regarding pre-, peri-, and postoperative care. Other comments highlighted the surgeon's development and demonstration and maintenance of cognitive and technical skills, including surgical judgment. Stakeholders also reiterated the critical need for surgeons to engage in on-going self-assessment and professional development to identify and remediate recognized limitations. Recommended learning modalities for meeting surgeons' needs included active learning (deliberate practice, diverse patient experiences), experiential learning (simulation), and peer and mentored learning (preceptorship). CONCLUSION: This first formal needs assessment of education for surgeons points to opportunities for educational programs in patient-centered communication, learning models that match preferences of new generations of trainees, and training in interprofessional/interdisciplinary team communication and teamwork.

Implementation of a preoperative briefing protocol improves accuracy of teamwork assessment in the operating room.

This study examined the effect of implementing a new preoperative briefing protocol on self- and peer-assessments of individual operating room (OR) teamwork behaviors. From July 2006 to February 2007, OR teamwork performance at a rural community hospital was evaluated before and after training and implementation of the protocol. After each case, every member on the team completed a 360-degree type teamwork behavior evaluation containing both self- and peer-assessments using a six-point Likert type scale (1 = definitely no to 6 = definitely yes). Individual behavior change was measured using the mean scale score of pre and postprotocol assessments. Statistical analysis included t test for both pre/post and self/peer differences. Data were available for one general surgeon and nine OR staff (pre = 20 cases, post = 16 cases). The preprotocol self-assessment mean score was significantly higher than peer-assessment (5.63 vs 5.29, P < 0.0267). Pre and postprotocol peer assessment mean scores revealed a statistically significant gain in teamwork behaviors. No difference was observed in postassessment mean scores for self- and peer-assessments. Individuals overestimated their teamwork behaviors before protocol implementation. Using a preoperative protocol seems to improve OR staff teamwork behaviors and self-assessment accuracy. The use of a 360-degree assessment method targeting specific, observable behaviors may be useful in evaluating team-based interventions and enhancing teamwork effectiveness.