Simulation Based Training and Assessment in Open Vascular Surgery: A Systematic Review.

OBJECTIVE: The aim of this study was to systematically review the literature and give evidence based recommendations for future initiatives for simulation based training (SBT) and assessment in open vascular surgery. DATA SOURCES: PubMed, Embase, and the Cochrane Library. REVIEW METHODS: A systematic review of PubMed, Embase, and the Cochrane Library was performed, with the last search on 31 March 2020, to identify studies describing SBT and assessment in open vascular surgery. Kirkpatrick's levels for efficacy of training were evaluated. Validity evidence for assessment tools was evaluated according to the recommended contemporary framework by Messick. RESULTS: Of 2 844 studies, 51 were included for data extraction. A high degree of heterogeneity in reporting standards and varying types of simulation was found. Vascular anastomosis was the most frequently simulated technical skill (43%). Assessment was mostly carried out using the Objective Structured Assessment of Technical Skills (55%). Validity evidence for assessment tools was found using outdated frameworks, and only one study used Messick's framework. Self directed training is valuable, the low trainer to trainee ratio is important to maximise efficiency, and experienced vascular surgeons are the most effective trainers. CONCLUSION: Carefully designed and structured SBT is effective and can improve technical skills, especially in less experienced trainees. However, the supporting evidence lacks homogeneity in the reporting standards and types of simulations. Pass/fail standards that support proficiency based learning and studies investigating skills transfer should be the focus in future studies. Validity evidence of assessment tools needs to be addressed using contemporary frameworks.

European Society for Vascular Surgery (ESVS) Certification of Theoretical and Practical Competences in Basic Vascular Ultrasound: Validity Investigation of the Assessment Tools.

OBJECTIVE: The aim of this study was to gather validity evidence for the Assessment of basic Vascular Ultrasound Expertise (AVAUSE) tool, and to establish a pass/fail score for each component, to support decisions for certification. METHODS: A cross sectional validation study performed during the European Society for Vascular Surgery's annual meeting. Validity evidence was sought for the theoretical test and two practical tests based on Messick's framework. The participants were vascular surgeons, vascular surgical trainees, sonographers, and nurses with varying experience levels. Five vascular ultrasound experts developed the theoretical and two practical test components of the AVAUSE tool for each test component. Two stations were set up for carotid examinations and two for superficial venous incompetence (SVI) examinations. Eight raters were assigned in pairs to each station. Three methods were used to set pass/fail scores: contrasting groups' method; rater consensus; and extended Angoff. RESULTS: Nineteen participants were enrolled. Acceptable internal consistency reliability (Cronbach's alpha) for the AVAUSE theoretical (0.93), carotid (0.84), and SVI (0.65) practical test were shown. In the carotid examination, inter-rater reliability (IRR) for the two rater pairs was good: 0.68 and 0.78, respectively. The carotid scores correlated significantly with years of experience (Pearson's r = 0.56, p = .013) but not with number of examinations in the last five years. For SVI, IRR was excellent at 0.81 and 0.87. SVI performance scores did not correlate with years of experience and number of examinations. The pass/fail score set by the contrasting groups' method was 29 points out of 50. The rater set pass/fail scores were 3.0 points for both carotid and SVI examinations and were used to determine successful participants. Ten of 19 participants passed the tests and were certified. CONCLUSION: Validity evidence was sought and established for the AVAUSE comprehensive tool, including pass/fail standards. AVAUSE can be used to assess competences in basic vascular ultrasound, allowing operators to progress towards independent practice.

Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool.

OBJECTIVE: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard. METHODS: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score. RESULTS: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed. CONCLUSION: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

Achieving Consensus to Define Curricular Content for Simulation Based Education in Vascular Surgery: A Europe Wide Needs Assessment Initiative.

OBJECTIVE: To gather consensus among European educators about technical procedures that should be included in a future simulation based curriculum in vascular surgery. METHODS: A three round modified Delphi survey was initiated among 189 key opinion leaders (KOL) from 34 countries across Europe who were identified according to their positions in the European Society for Vascular Surgery, the European Journal of Vascular and Endovascular Surgery, and Union Européenne des Médecins Spécialistes Section and Board of Vascular Surgery. The first round was a brainstorming phase to identify technical procedures that a newly qualified vascular surgeon should be able to perform. The answers were analysed qualitatively. The second round investigated how often the identified procedures are performed, the number of vascular surgeons that should be able to perform these procedures, whether the procedures pose a risk to the patients, and whether simulation based education (SBE) is feasible. In the third round, elimination and re-ranking of procedures were performed. Only procedures that gained more than 70% support were included. An international steering group consisting of open and endovascular surgeons and medical educators governed the process. RESULTS: Response rates in the three rounds were 75% (142/189), 89% (126/142), and 85% (107/126), respectively. In the final prioritised list of 30 technical procedures for SBE, the top five procedures focus on basic open vascular skills, basic endovascular skills, vascular imaging interpretation, femoral endarterectomy, and open peripheral bypass. Twenty-six procedures were eliminated, including peripheral pressure measurement, wound management, open management of complications, major amputations, and highly advanced endovascular skills. CONCLUSION: The prioritised list of technical procedures from this ESVS supported project could be used to guide planning and development of future SBE programs to meet the needs of vascular surgeons across Europe.