Development and Testing of Step, Error, and Event Frameworks to Evaluate Technical Performance in Peripheral Endovascular Interventions.

OBJECTIVE: Tools for endovascular performance assessment are necessary in competency based education. This study aimed to develop and test a detailed analysis tool to assess steps, errors, and events in peripheral endovascular interventions (PVI). METHODS: A modified Delphi consensus was used to identify steps, errors, and events in iliac-femoral-popliteal endovascular interventions. International experts in vascular surgery, interventional radiology, cardiology, and angiology were identified, based on their scientific track record. In an initial open ended survey round, experts volunteered a comprehensive list of steps, errors, and events. The items were then rated on a five point Likert scale until consensus was reached with a pre-defined threshold (Cronbach's alpha > 0.7) and > 70% expert agreement. An experienced endovascular surgeon applied the finalised frameworks on 10 previously videorecorded elective PVI cases. RESULTS: The expert consensus panel was formed by 28 of 98 invited proceduralists, consisting of three angiologists, seven interventional radiologists, five cardiologists, and 13 vascular surgeons, with 29% from North America and 71% from Europe. The Delphi process was completed after three rounds (Cronbach's alpha; αsteps = 0.79; αerrors = 0.90; αevents = 0.90), with 15, 26, and 18 items included in the final step (73 - 100% agreement), error (73 - 100% agreement), and event (73 - 100% agreement) frameworks, respectively. The median rating time per case was 4.3 hours (interquartile range [IQR] 3.2, 5 hours). A median of 55 steps (IQR 40, 67), 27 errors (IQR 21, 49), and two events (IQR 1, 6) were identified per case. CONCLUSION: An evaluation tool for the procedural steps, errors, and events in iliac-femoral-popliteal endovascular procedures was developed through a modified Delphi consensus and applied to recorded intra-operative data to identify hazardous steps, common errors, and events. Procedural mastery may be promoted by using the frameworks to provide endovascular proceduralists with detailed technical performance feedback.

Surgeons' Leadership Style and Team Behavior in the Hybrid Operating Room: Prospective Cohort Study.

OBJECTIVE: This study aimed to assess the relationship between surgeons' leadership style and team behavior in the hybrid operating room through video coding. Secondly, possible fluctuations possible fluctuations in leadership styles and team behavior during operative phases were studied. BACKGROUND: Leadership is recognized as a key component to successful team functioning in high-risk industries. The 'full range of leadership' theory is commonly used to evaluate leadership, marking transformational, transactional, and passive. Few studies have examined the effects of these leadership styles on team behavior in surgery and/or their fluctuations during surgery. METHODS: A single-center study included patients planned for routine endovascular procedures. A medical data capture system was used to allow post hoc video coding through Behavior Anchored Rating Scales. Multilevel statistical analysis was performed to assess possible correlations between leadership style and 3 team behavior indicators (speaking up, knowledge sharing, and collaboration) on an operative phase level. RESULTS: Twenty-two cases were analyzed (47 hours recording). Transformational leadership is positively related to the extent to which team members work together (γ=0.20, P <0.001), share knowledge (γ=0.45, P <0.001), and speak up (γ=0.64, P <0.001). Passive leadership is significantly positively correlated with speaking up (γ=0.29, P =0.004). Leadership style and team behavior clearly fluctuate during a procedure, with similar patterns across different types of endovascular procedures. CONCLUSIONS: Consistent with other professional fields, surgeons' transformational leadership enhances team behavior, especially during the most complex operative phases. This suggests that encouraging surgeons to learn and actively implement a transformational leadership style is meaningful to enhance patient safety and team performance.

Massed Training is Logistically Superior to Distributed Training in Acquiring Basic Endovascular Skills.

OBJECTIVE: A "PROficiency based StePwise Endovascular Curricular Training" (PROSPECT) has proven its superiority over traditional training in a randomised controlled trial to acquire basic endovascular skills outside theatre, but real life adherence is low. This study aimed to compare the original distributed training format, where trainees learn at their own pace, with a massed training format offering the same content within a limited time span while exempt from clinical duties. Secondly, long term skills retention was evaluated. METHODS: A multicentre, prospective study allocated participants to the distributed D-PROSPECT or to a massed, compact version (C-PROSPECT) based on logistics such as travel time, participant and instructor availability. A multiple choice question (MCQ) test (max. score 20) tested cognitive skills. Technical skills were assessed using a global rating scale (GRS) (max. score 55), examiner's checklist (max. score 85), and validated simulator metrics. Data were collected pre- and post-programme and at three, six, and 12 months after programme completion. RESULTS: Over four years and in two countries, D-PROSPECT was implemented in two centres and C-PROSPECT in three. A total of 22 participants completed D-PROSPECT with a 41% dropout rate, and 21 completed C-PROSPECT with 0% dropout rate. All participants showed significant improvement for all performance parameters after programme completion: MCQ test (median 14.5 vs. 18; p < .001), GRS (median 20 vs. 41; p < .001), examiner's checklist (median 49 vs. 78.5; p < .001), and simulation metrics (p < .001). Scores of C- or D-PROSPECT participants were not significantly different. No significant differences were seen between groups during the retention period. CONCLUSION: PROSPECT significantly improves the quality of simulated endovascular performances using a massed or distributed training format. A massed training format of PROSPECT may be preferred to decrease dropout during standardised training to obtain basic endovascular skills in existing surgical curricula.

Development, Feasibility, and Knowledge Impact of a Massive Open Online Course on Radiation Safety: A Multicentre Prospective Cohort Study.

OBJECTIVE: This study reports the development of an innovative, interactive Massive Open Online Course (MOOC) teaching radiation safety principles in the vascular workplace, using stepwise e-learning with multiple choice question tests (MCQs), educational videos, and a serious game. The aim was to study the MOOC impact on radiation safety knowledge and assess its feasibility and acceptability. METHODS: An international multicentre prospective study included team members active in the hybrid operating room. The MOOC was offered voluntarily via a secure online learning platform. A standardised MCQ test (15 questions) assessed radiation safety knowledge pre- and post-course (range 0 - 100%). Acceptability and feasibility were tested via the previously validated, Evaluating e-learning system success (EESS) model, using five point Likert scales. RESULTS: In eight centres across four countries, 150 of 203 invited endovascular team members consented. Over a seven week study period, surgeons (28%, including vascular surgery trainees and consultants), nurses (27%, including scrub, circulating and anaesthetic nurses), anaesthetists (43%, including trainees and consultants), and radiographers (3%) participated. Of those, 67% completed the course. The average radiation knowledge improved by 22.8% (95% CI 19.5 - 26.0%; p < .001) after MOOC completion, from 48% to 71% (standard deviation [SD] 14 and 15% respectively), requiring a mean time investment of 169 minutes (SD 89 minutes). In centres with a radiographer, mean knowledge gain after MOOC completion was significantly smaller (14%, SD 19% vs. 24%, SD 16%, p = .036). The course was deemed feasible and acceptable according to the EESS model with a total mean score of 3.68/5. CONCLUSION: This newly developed, multimodal MOOC was deemed feasible and effective across multiple international centres. The MOOC significantly contributes to radiation safety education of the entire endovascular team, improving radiation safety knowledge. The course may optimise workplace radiation safety behaviour and therefore enhance team and patient safety.

Step, Error, and Event Frameworks in Endovascular Aortic Repair.

OBJECTIVE: Competency-based surgical education requires detailed and actionable feedback to ensure adequate and efficient skill development. Comprehensive operative capture systems such as the Operating Room Black Box (ORBB; Surgical Safety Technologies, Inc), which continuously records and synchronizes multiple sources of intraoperative data, have recently been integrated into hybrid rooms to provide targeted feedback to endovascular teams. The objective of this study is to develop step, error, and event frameworks to evaluate technical performance in elective endovascular aortic repair (EVAR) comprehensively captured by the ORBB (Surgical Safety Technologies, Inc; Toronto, Canada). METHODS: This study is based upon a modified Delphi consensus process to create evaluation frameworks for steps, errors, and events in EVAR. International experts from Vascular Surgery and Interventional Radiology were identified, based on their records of publications and invited presentations, or serving on relevant journal editorial boards. In an initial open-ended survey round, experts were asked to volunteer a comprehensive list of steps, errors, and events for a standard EVAR of an infrarenal aorto-iliac aneurysm (AAA). In subsequent survey rounds, the identified items were presented to the expert panel to rate on a 5-point Likert scale. Delphi survey rounds were repeated until the process reached consensus with a predefined agreement threshold (Cronbach α>0.7). The final frameworks were constructed with items achieving an agreement (responses of 4 or 5) from greater than 70% of experts. RESULTS: Of 98 invited proceduralists, 38 formed the expert consensus panel (39%), consisting of 29 vascular surgeons and 9 interventional radiologists, with 34% from North America and 66% from Europe. Consensus criteria were met following the third round of the Delphi consensus process (Cronbach α=0.82-0.93). There were 15, 32, and 25 items in the error, step, and event frameworks, respectively (within-item agreement=74%-100%). CONCLUSION: A detailed evaluation tool for the procedural steps, errors, and events in infrarenal EVAR was developed. This tool will be validated on recorded procedures in future work: It may focus skill development on common errors and hazardous steps. This tool might be used to provide high-quality feedback on technical performance of trainees and experienced surgeons alike, thus promoting surgical mastery.

Endovascular Training Using a Simulation Based Curriculum is Less Expensive than Training in the Hybrid Angiosuite.

OBJECTIVE/BACKGROUND: This study aimed to determine the cost-effectiveness of a PROficiency based StePwise Endovascular Curricular Training (PROSPECT) program, including e-learning and hands on virtual reality simulation. METHODS: A prospective, single blinded, randomised controlled trial (RCT) was carried out to evaluate the impact of a PROSPECT training program on real life operative performance. Under supervision, all subjects performed two endovascular interventions on patients with symptomatic iliac and/or superficial femoral artery stenosis. Primary outcomes were technical performance (Global Rating Scale, Examiner Checklist), operative metrics, and patient outcomes, adjusted for case difficulty and the trainee's experience. Additionally, an analysis of costs and savings related to implementation of this endovascular training program was performed. Thirty-two general surgery trainees were randomised into three groups: group 1 (n = 11) received e-learning and simulation training (PROSPECT program); group 2 (n = 10) only had access to e-learning; group 3 (n = 11) did not receive supplementary education besides clinical training. Developmental cost, implementation cost, training time cost, and the operational cost of PROSPECT were determined. Time spent studying and practicing was converted to indirect saving of operating time. The costs of logistics, faculty time supervising simulation sessions, and 30 day complication rates were registered. Sensitivity analysis was performed to assess the robustness of the results. RESULTS: Fifty-eight peripheral endovascular interventions, performed by 29 surgical trainees (three dropouts) were included in this RCT from October 2014 to February 2016. Annual costs from the perspective of the hospital were €6589 for curriculum design, €31,484 for implementation, and €1143 in operational costs. Per trainee, simulation based training until proficiency cost €3806. In comparison, if endovascular proficiency levels were obtained with conventional training only, this may have cost €5001 per trainee. CONCLUSION: Simulation based training in endovascular procedures may be cost saving, because training occurs outside the angiosuite. It is possible that cost savings are underestimated as, in contrast to the literature, prevented costs related to complications could not be defined.

Defining the Key Competencies in Radiation Protection for Endovascular Procedures: A Multispecialty Delphi Consensus Study.

OBJECTIVES: Radiation protection training courses currently focus on broad knowledge topics which may not always be relevant in daily practice. The goal of this study was to determine the key competencies in radiation protection that every endovascular team member should possess and apply routinely, through multispecialty clinical content expert consensus. METHODS: Consensus was obtained through a two round modified Delphi methodology. The expert panel consisted of European vascular surgeons, interventional radiologists, and interventional cardiologists/angiologists experienced in endovascular procedures. An initial list of statements, covering knowledge skills, technical skills and attitudes was created, based on a literature search. Additional statements could be suggested by the experts in the first Delphi round. Each of the statements had to be rated on a 5- point Likert scale. A statement was considered to be a key competency when the internal consistency was greater than alpha = 0.80 and at least 80% of the experts agreed (rating 4/5) or strongly agreed (rating 5/5) with the statement. Questionnaires were emailed to panel members using the Surveymonkey service. RESULTS: Forty-one of 65 (63.1%) invited experts agreed to participate in the study. The response rates were 36 out of 41 (87.8%): overall 38 out of 41(92.6%) in the first round and 36 out of 38 (94.7%) in the second round. The 71 primary statements were supplemented with nine items suggested by the panel. The results showed excellent consensus among responders (Cronbach's alpha = 0.937 first round; 0.958 s round). Experts achieved a consensus that 30 of 33 knowledge skills (90.9%), 23 of 27 technical skills (82.1%), and 15 of 20 attitudes (75.0%) should be considered as key competencies. CONCLUSIONS: A multispecialty European endovascular expert panel reached consensus about the key competencies in radiation protection. These results may serve to create practical and relevant radiation protection training courses in the future, enhancing radiation safety for both patients and the entire endovascular team.

Assessing endovascular team performances in a hybrid room using the Black Box system: a prospective cohort study.

BACKGROUND: The hybrid room (HR) is a complex, high-risk environment, requiring teams (surgeons, anesthesiologists, nurses, technologists) to master various skills, including the 'As Low As Reasonably Achievable' principle of radiation safety. This prospective single center cohort reports the first use of the Operating Room Black Box (ORBB) in a HR. This medical data recording system captures procedural and audio-visual data to facilitate structured team performance analysis. METHODS: Patients planned for endovascular repair of an infrarenal abdominal aortic aneurysm (EVAR) or treatment of symptomatic iliac-femoral-popliteal atherosclerotic disease (Peripheral Vascular Interventions or PVI) were included. Validated measures and established assessment tools were used to assess (non-)technical skills, radiation safety performance and environmental distractions. RESULTS: Six EVAR and sixteen PVI procedures were captured. Technical performance for one EVAR was rated 19/35 on the procedure-specific scale, below the 'acceptable' score of 21. Technical skills were rated above acceptable in all PVI procedures. Shared decision making and leadership were rated highly in 12/22 cases, whereas surgical communication and nurses' task management were rated low in 14/22 cases. Team members rarely stepped back from the C-arm during digital subtraction angiography. Radiation safety behavior was scored below 'acceptable' in 14/22 cases. A median (interquartile range) number of 12 (6-23) auditory distractions was observed per procedure. CONCLUSIONS: The ORBB facilitates holistic workplace-based assessment of endovascular performance in a HR by combining objective assessment parameters and rating scale-based evaluations. Strengths and weaknesses were identified in team members' (non-)technical and radiation safety practices. This technology has the potential to improve vascular surgical practice, though human input remains crucial. (NCT04854278).

Radiation Safety Performance is More than Simply Measuring Doses! Development of a Radiation Safety Rating Scale.

PURPOSE: Radiation safety performance is often evaluated using dose parameters measured by personal dosimeters and/or the C-arm, which provide limited information about teams' actual radiation safety behaviors. This study aimed to develop a rating scale to evaluate team radiation safety behaviors more accurately and investigate its reliability. MATERIALS AND METHODS: A modified Delphi consensus was organized involving European vascular surgeons (VS), interventional radiologists, and interventional cardiologists. Initial items and anchors were drafted a priori and rated using five-point Likert scales. Participants could suggest additional items or adjustments. Consensus was defined as ≥ 80% agreement (rating ≥ 4) with Cronbach's alpha ≥ .80. Two VS with expertise in radiation safety evaluated 15 video-recorded endovascular repairs of infrarenal aortic aneurysms (EVAR) to assess usability, inter and intrarater reliability. RESULTS: Thirty-one of 46 invited specialists completed three rating rounds to generate the final rating scale. Five items underwent major adjustments. In the final round, consensus was achieved for all items (alpha = .804; agreement > 87%): 'Pre-procedural planning', 'Preparation in angiosuite/operating room', 'Shielding equipment', 'Personal protective equipment', 'Position of operator/team', 'Radiation usage awareness', 'C-arm handling', 'Adjusting image quality', 'Additional dose reducing functions', 'Communication/leadership', and 'Overall radiation performance and ALARA principle'. All EVARs were rated, yielding excellent Cronbach's alpha (.877) with acceptable interrater and excellent intrarater reliability (ICC = .782; ICC = .963, respectively). CONCLUSION: A reliable framework was developed to assess radiation safety behaviors in endovascular practice and provide teams with formative feedback. The final scale is provided in this publication.

Team perception of the radiation safety climate in the hybrid angiography suite: A cross-sectional study.

BACKGROUND: Good radiation safety practice in the angiosuite is essential to protect patients and healthcare workers. Most strategies aim to advance radiation safety through technological upgrades and educational initiatives. However, safety literature suggests that additional ways to improve radiation safety in the angiosuite do exist. The safety climate reflects the way team members perceive various key characteristics of their work environment and is closely related to relevant safety outcomes. A specific 'radiation safety climate' has not been described nor studied in the hybrid angiosuite. This study explores the radiation safety climate in the hybrid angiosuite and its relation to team members' radiation safety behavior, knowledge and motivation. MATERIALS AND METHODS: Vascular surgeons, fellows/trainees and operating room nurses active in the angiosuite at five hospitals were invited to complete an online self-report questionnaire assessing the radiation safety climate (28 items); radiation safety behavior; radiation safety knowledge and radiation safety motivation. Relations between climate scores and behavior were investigated using Pearson correlations. Mediation was analyzed using the Baron and Kenny analysis. P-Values < 0.05 were considered statistically significant. RESULTS: No major differences were identified in total radiation safety climate scores between centers or team member functions. Scale reliability for radiation safety climate was good to excellent (α > 0.663). Total radiation safety climate scores were positively related to the radiation safety behavior score (r = 0.403; p = 0.015). This relation was partially mediated by radiation safety knowledge (ß = 0.1730; 95% CI: [0.0475; 0.3512]), while radiation safety motivation did not act as a mediator: (ß = 0.010; 95% CI: [-0.0561; 0.0998]). CONCLUSION: A well-developed radiation safety climate in the hybrid angiosuite fosters positive radiation safety behaviors, which may partially be explained through improved radiation safety knowledge transfer. Further research on (radiation) safety climate and its impact on radiation safety-related outcome measures for patients is recommended.

Introduction of a surgical Black Box system in a hybrid angiosuite: Challenges and opportunities.

PURPOSE: In laparoscopy, the Operating Room Black Box® (ORBB) provides insights into operative performance to improve patient safety. This technology may also enhance endovascular surgical practice; however, the use of a C-arm and X-rays pose important challenges, hindering transferability to an endovascular context. We describe the first implementation of ORBB technology in a hybrid angiosuite and illustrate its value in evaluating surgeons' radiation safety, technical and non-technical performance. METHODS: Team members (surgeons, nurses, anaesthesiologists) and stakeholders were informed during several information sessions. Together with teams from Surgical Safety Technologies (Toronto, Canada), an implementation plan was developed, and video-evaluation frameworks were chosen. Radiation safety was assessed using dose measurements and video-evaluation of safety-related behaviours. Technical performance was assessed using 'global' (GRS) and 'procedure-specific' (PRS) rating scales and the 'EndoVascular Aortic Repair Assessment of Technical Expertise' (EVARATE) framework. Surgeons' non-technical skills were assessed with the NOTSS framework. RESULTS: The system captures Audio-visual data from four ceiling-mounted cameras, three ceiling-array microphones, the fluoroscopy screen and anaesthesia monitor. After patient and team consent, an elective endovascular aneurysm repair was successfully analysed. Dose-Area-Product and Air Kerma were 71094 mGy.cm2 and 270 mGy, respectively. Behavioural analysis revealed deficiencies in stepping back and radiation safety communication. Technical skill assessment was feasible: GRS: 29/40; 'PRS': 27/35; EVARATE: 29/35. Non-technical analysis highlighted surgeons' leadership qualities. CONCLUSION: An innovative data capture platform has been successfully installed to evaluate overall performance during endovascular procedures. This technology may facilitate identification of (radiation) safety-related errors and instigate educational interventions based on real-world issues.

Non-technical attributes and surgical experience: A cross-sectional study comparing communication styles and attitudes in surgical staff, trainees and applicants.

BACKGROUND: This monocentric study aimed to explore whether key non-technical attributes can be reliably measured in a mixed population of candidates applying for surgical training, surgical trainees and staff and to identify any differences between these groups. MATERIALS & METHODS: Candidates applying for surgical training, surgical trainees and staff from four surgical specialties (general surgery, orthopedics, plastic surgery or urology) at a tertiary academic teaching hospital were all sent an online self-report questionnaire. The Communication Styles Inventory (CSI, 96 items) was used to assess a six-dimensional behavioral model of participant communication styles (expressiveness, preciseness, verbal aggressiveness, questioningness, emotionality and impression manipulativeness). Attitudes toward uncertainty and risks were assessed with the Physicians' Reaction toward Uncertainty (PRU, 15 items) and Physician Risk Attitudes (PRA, 6 items) scales respectively. Data was encoded and analyzed using parametric testing. RESULTS: The questionnaire was completed by 177 participants (110 candidates; 42 trainees; 25 staff). All scales had very good internal consistency (Cronbach's alpha >0.80). After controlling for gender-based differences, surgical candidates scored significantly higher on 'expressiveness' (P = 0.012) and were significantly less risk-averse (P = 0.006) than trainees and staff. Surgical trainees scored lowest on the CSI 'questioningness' subscale (P = 0.019) and had significantly more difficulties dealing with uncertainty, characterized by their highest scores on the 'concern about bad outcome' (P = 0.021) and reluctance to disclose uncertainty to patients' (P = 0.05) subscales. Multiple subscales revealed gender-based differences in candidate and trainee groups, which were not noted for surgical staff. CONCLUSIONS: Meaningful differences in non-technical attributes of surgical staff, trainees and candidates have been identified, which may be explained by differences in clinical experience and learning and may suggest that these develop over time. Further research on assessment of non-technical attributes during surgical selections and the role of both technical and non-technical attributes in surgery at large is needed.