Standardised assessment of competence in Focused Assessment with Sonography for Trauma.

PURPOSE: The aim of this study was to develop and gather validity evidence for a standardised test of competence in Focused Assessment with Sonography for Trauma (FAST) and to define the appropriate cut-off point in simulation-based learning of the FAST protocol. METHODS: A 20-item simulation-based test for assessing competence in FAST was created. The test was administered to thirteen novices and twelve radiologists experienced in abdominal ultrasound diagnostics. The Contrasting Groups' method was used to establish a credible passing score. RESULTS: The internal consistency was high (Cronbach's α = 0.90) and the test had good discriminatory ability (P < .001). The mean score was 16.9 (95% CI: 15.5-18.3) in the experienced group and 8.0 (95% CI: 5.8-10.2) in the novice group, corresponding to 85% and 40% of the total score, respectively. A pass/fail standard of 14 points was established using the Contrasting Groups' method. CONCLUSIONS: The FAST simulation-based test provided valid assessment of competence in FAST. The FAST test could be used to guide training and ensure basic competence of physicians using FAST.

Assessment of Competence in EVAR Stent Graft Sizing and Selection.

OBJECTIVES AND BACKGROUND: The aims of this study were to develop a test of competence in endovascular aortic repair (EVAR) stent graft sizing and selection; to examine the test for evidence of validity; and to explore the experience required for the task. METHODS: The test was developed based on a literature review resulting in 22 anatomical assessment points and a graft selection. Validity evidence was explored in an international cross sectional study. Twenty-two consultants with varying levels of experience in the field (novices, intermediates, and experts) were presented with computed tomography angiography of the aortic vessels from three patients. Test scores were based on summed z-scores using the anatomical measurements and graft choices of the experts as a reference. A proficiency score was established using the contrasting groups standard setting method. RESULTS: The assessment was shown to be reliable with an intraclass correlation coefficient of 0.83 (p<.001) and high internal consistency with a Cronbach's α of .91 (p<.001). Mann-Whitney U test showed that experts performed significantly better than novices and intermediates (p<.002 and p<.005, respectively). Regarding anatomical measurements, Mann-Whitney U test could discriminate between experts and novices (p=.002), between experts and intermediates (p=.010), and between novices and intermediates (p=.036). In stent selection the experts performed significantly better than both the novices and the intermediates (p=.002 and p=.007, respectively), while there was no significant difference between the two non-expert groups (p=1). A credible passing standard with appropriate consequences was established using the contrasting groups methods. CONCLUSION: This study presents a standardised and objective assessment tool of competence in vessel analysis and stent graft selection for endovascular aortic repair. This was supported by strong validity evidence with good internal consistency and discriminatory ability. The tool may be used to facilitate training and certification of future endovascular specialists.

Assessment of Competence in EVAR Procedures: A Novel Rating Scale Developed by the Delphi Technique.

OBJECTIVE/BACKGROUND: To develop a procedure specific global rating scale for assessment of operator competence in endovascular aortic repair (EVAR). METHODS: A Delphi approach was used to achieve expert consensus. A panel of 32 international experts (median 300 EVAR procedures, range 200-3000) from vascular surgery (n = 21) and radiology (n = 11) was established. The first Delphi round was based on a review of endovascular skills assessment papers, stent graft instructions for use, and structured interviews. It led to a primary pool of 83 items that were formulated as global rating scale items with tentative anchors. Iterative Delphi rounds were executed. The panellists rated the importance of each item on a 5 point Likert scale. Consensus was defined as 80% of the panel rating an item 4 or 5 in the primary round and 90% in subsequent rounds. Consensus on the final assessment tool was defined as Cronbach's alpha > .8 after a minimum of three rounds. RESULTS: Thirty-two of 35 invited experts participated. Three rounds of surveys were completed with a completion rate of 100% in the first two rounds and 91% in round three. The 83 primary assessment items were supplemented with five items suggested by the panel and reduced to seven pivotal assessment items that reached consensus, Cronbach's alpha = 0.82. The seven item rating scale covers key elements of competence in EVAR stent placement and deployment. Each item has well defined grades with explicit anchors at unacceptable, acceptable, and superior performance on a 5 point Likert scale. CONCLUSION: The Delphi methodology allowed for international consensus on a new procedure specific global rating scale for assessment of competence in EVAR. The resulting scale, EndoVascular Aortic Repair Assessment of Technical Expertise (EVARATE), represents key elements in the procedure. EVARATE constitutes an assessment tool for providing structured feedback to endovascular operators in training.

A National Needs Assessment to Identify Technical Procedures in Vascular Surgery for Simulation Based Training.

OBJECTIVES AND BACKGROUND: Practical skills training in vascular surgery is facing challenges because of an increased number of endovascular procedures and fewer open procedures, as well as a move away from the traditional principle of "learning by doing." This change has established simulation as a cornerstone in providing trainees with the necessary skills and competences. However, the development of simulation based programs often evolves based on available resources and equipment, reflecting convenience rather than a systematic educational plan. The objective of the present study was to perform a national needs assessment to identify the technical procedures that should be integrated in a simulation based curriculum. DESIGN AND METHODS: A national needs assessment using a Delphi process was initiated by engaging 33 predefined key persons in vascular surgery. Round 1 was a brainstorming phase to identify technical procedures that vascular surgeons should learn. Round 2 was a survey that used a needs assessment formula to explore the frequency of procedures, the number of surgeons performing each procedure, risk and/or discomfort, and feasibility for simulation based training. Round 3 involved elimination and ranking of procedures. RESULTS: The response rate for round 1 was 70%, with 36 procedures identified. Round 2 had a 76% response rate and resulted in a preliminary prioritised list after exploring the need for simulation based training. Round 3 had an 85% response rate; 17 procedures were eliminated, resulting in a final prioritised list of 19 technical procedures. CONCLUSION: A national needs assessment using a standardised Delphi method identified a list of procedures that are highly suitable and may provide the basis for future simulation based training programs for vascular surgeons in training.

Randomized trial to examine procedure-to-procedure transfer in laparoscopic simulator training.

BACKGROUND: Laparoscopic simulation has become a standard component of surgical training, but there is limited knowledge regarding skills transfer between procedural tasks. The objective was to investigate the specificity of procedural simulator training. METHODS: This was randomized single-centre educational superiority trial. Surgical novices practised basic skills on a laparoscopic virtual reality simulator. On reaching proficiency, participants were randomized to proficiency-based training. The intervention group practised two procedures on the simulator (appendicectomy followed by salpingectomy), whereas the control group trained on only one procedure (salpingectomy). The main outcomes were number of repetitions and time to proficiency for the second procedure. RESULTS: Ninety-six participants were randomized, of whom 74 per cent were women, with a median age of 26 years. The intervention group needed significantly fewer attempts than the control group to reach proficiency in the second procedure: median (i.q.r.) 22 (17-34) versus 32 (26-41) attempts, which corresponded to 24·1 per cent fewer attempts as assessed by multivariable analysis (P = 0·004). The intervention group required significantly less time than the control group to reach proficiency: median (i.q.r.) 88 (63-127) versus 131 (101-153) min respectively, corresponding to a difference of 31·1 min as assessed by multivariable analysis (P = 0·001). CONCLUSION: Practising two procedures, compared with only one, reduced the number of attempts and time to reach proficiency in the second procedure. Skills transfer is seen between two tasks in laparoscopic simulator training; however, task specificity is still present when practising procedures. REGISTRATION NUMBER: NCT02069951 (http://www.clinicaltrials.gov).

Evaluating structured assessment of anaesthesiologists' non-technical skills.

BACKGROUND: Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties. METHODS: An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session. RESULTS: Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training. CONCLUSION: The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.

Simulation-Based Abdominal Ultrasound Training - A Systematic Review.

PURPOSE: The aim is to provide a complete overview of the different simulation-based training options for abdominal ultrasound and to explore the evidence of their effect. MATERIALS AND METHODS: This systematic review was performed according to the PRISMA guidelines and Medline, Embase, Web of Science, and the Cochrane Library was searched. Articles were divided into three categories based on study design (randomized controlled trials, before-and-after studies and descriptive studies) and assessed for level of evidence using the Oxford Centre for Evidence Based Medicine (OCEBM) system and for bias using the Cochrane Collaboration risk of bias assessment tool. RESULTS: Seventeen studies were included in the analysis: four randomized controlled trials, eight before-and-after studies with pre- and post-test evaluations, and five descriptive studies. No studies scored the highest level of evidence, and 14 had the lowest level. Bias was high for 11 studies, low for four, and unclear for two. No studies used a test with established evidence of validity or examined the correlation between obtained skills on the simulators and real-life clinical skills. Only one study used blinded assessors. CONCLUSION: The included studies were heterogeneous in the choice of simulator, study design, participants, and outcome measures, and the level of evidence for effect was inadequate. In all studies simulation training was equally or more beneficial than other instructions or no instructions. Study designs had significant built-in bias and confounding issues; therefore, further research should be based on randomized controlled trials using tests with validity evidence and blinded assessors.

Validity of a cross-specialty test in basic laparoscopic techniques (TABLT).

BACKGROUND: Box trainer systems have been developed that include advanced skills such as suturing. There is still a need for a portable, cheap training and testing system for basic laparoscopic techniques that can be used across different specialties before performing supervised surgery on patients. The aim of this study was to establish validity evidence for the Training and Assessment of Basic Laparoscopic Techniques (TABLT) test, a tablet-based training system. METHODS: Laparoscopic surgeons and trainees were recruited from departments of general surgery, gynaecology and urology. Participants included novice, intermediate and experienced surgeons. All participants performed the TABLT test. Performance scores were calculated based on time taken and errors made. Evidence of validity was explored using a contemporary framework of validity. RESULTS: Some 60 individuals participated. The TABLT was shown to be reliable, with an intraclass correlation coefficient of 0.99 (P < 0.001). ANOVA showed a difference between the groups with different level of experience (P < 0.001). The Bonferroni correction was used to confirm this finding. A Pearson's r value of 0.73 (P < 0.001) signified a good positive correlation between the level of laparoscopic experience and performance score. A reasonable pass-fail standard was established using contrasting groups methods. CONCLUSION: TABLT can be used for the assessment of basic laparoscopic skills and can help novice surgical trainees in different specialties gain basic laparoscopic competencies.

Correlation between human papillomavirus and p16 overexpression in oropharyngeal tumours: a systematic review.

BACKGROUND: A significant proportion of squamous cell carcinomas of the oropharynx (OP-SCC) are related to human papillomavirus (HPV) infection and p16 overexpression. This subgroup proves better prognosis and survival but no evidence exists on the correlation between HPV and p16 overexpression based on diagnostic measures and definition of p16 overexpression. We evaluated means of p16 and HPV diagnostics, and quantified overexpression of p16 in HPV-positive and -negative OP-SCCs by mode of immunohistochemical staining of carcinoma cells. METHODS: PubMed, Embase, and the Cochrane Library were searched from 1980 until October 2012. We applied the following inclusion criteria: a minimum of 20 cases of site-specific OP-SCCs, and HPV and p16 results present. Studies were categorised into three groups based on their definition of p16 overexpression: verbal definition, nuclear and cytoplasmatic staining between 5 and 69%, and ≥70% staining. RESULTS: We identified 39 studies with available outcome data (n=3926): 22 studies (n=1980) used PCR, 6 studies (n=688) used ISH, and 11 studies (n=1258) used both PCR and ISH for HPV diagnostics. The methods showed similar HPV-positive results. Overall, 52.5% of the cases (n=2062) were HPV positive. As to p16 overexpression, 17 studies (n=1684) used a minimum of 5-69% staining, and 7 studies (n=764) used ≥70% staining. Fifteen studies (n=1478) referred to a verbal definition. Studies showed high heterogeneity in diagnostics of HPV and definition of p16. The correlation between HPV positivity and p16 overexpression proved best numerically in the group applying ≥70% staining for p16 overexpression. The group with verbal definitions had a significantly lower false-positive rate, but along with the group applying 5-69% staining showed a worse sensitivity compared with ≥70% staining. CONCLUSIONS: There are substantial differences in how studies diagnose HPV and define p16 overexpression. Numerically, p16 staining is better to predict the presence of HPV (i.e. larger sensitivity), when the cutoff is set at ≥70% of cytoplasmatic and nuclear staining.

Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination.

OBJECTIVE: To assess the validity and reliability of performance measures, develop credible performance standards and explore learning curves for a virtual-reality simulator designed for transvaginal gynecological ultrasound examination. METHODS: A group of 16 ultrasound novices, along with a group of 12 obstetrics/gynecology (Ob/Gyn) consultants, were included in this experimental study. The first two performances of the two groups on seven selected modules on a high-fidelity ultrasound simulator were used to identify valid and reliable metrics. Performance standards were determined and novices were instructed to continue practicing until they attained the performance level of an expert subgroup (n = 4). RESULTS: All 28 participants completed the selected modules twice and all novices reached the expert performance level. Of 153 metrics, 48 were able to be used to discriminate between the two groups' performance. The ultrasound novices scored a median of 43.8% (range, 17.9-68.9%) and the Ob/Gyn consultants scored a median of 82.8% (range, 60.4-91.7%) of the maximum sum score (P < 0.001). The ultrasound novices reached the expert level (88.4%) within a median of five iterations (range, 5-6), corresponding to an average of 219 min (range, 150-251 min) of training. The test/retest reliability was high, with an intraclass correlation coefficient of 0.93. CONCLUSIONS: Competence in the performance of gynecological ultrasound examination can be assessed in a valid and reliable way using virtual-reality simulation. The novices' performance improved with practice and their learning curves plateaued at the level of expert performance, following between 3 and 4 h of simulator training.

Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer.

BACKGROUND AND STUDY AIMS: Fine-needle aspiration (FNA) guided by endoscopic ultrasonography (EUS) is important in mediastinal staging of non-small cell lung cancer (NSCLC). Training standards and implementation strategies of this technique are currently under discussion. The aim of this study was to explore the reliability and validity of a newly developed EUS Assessment Tool (EUSAT) designed to measure competence in EUS - FNA for mediastinal staging of NSCLC. PATIENTS AND METHODS: A total of 30 patients with proven or suspected NSCLC underwent EUS - FNA for mediastinal staging by three trainees and three experienced physicians. Their performances were assessed prospectively by three experts in EUS under direct observation and again 2 months later in a blinded fashion using digital video-recordings. Based on the assessments, intra-rater reliability, inter-rater reliability, and construct validity were explored. RESULTS: The intra-rater reliability was good (Cronbach's α = 0.80), but comparison of results based on direct observations and blinded video-recordings indicated a significant bias favoring consultants (P = 0.022). Inter-rater reliability was very good (Cronbach's α = 0.93). However, one rater assessing five procedures or two raters each assessing four procedures were necessary to secure a generalizability coefficient of 0.80. The assessment tool demonstrated construct validity by discriminating between trainees and experienced physicians (P = 0.034). CONCLUSIONS: Competency in mediastinal staging of NSCLC using EUS and EUS - FNA can be assessed in a reliable and valid way using the EUSAT assessment tool. Measuring and defining competency and training requirements could improve EUS quality and benefit patient care.

Reliable test of clinicians' mastery in skin cancer diagnostics.

Differentiating between benign and malignant skin lesions can be very difficult and should only be done by sufficiently trained and skilled clinicians. To our knowledge there are no validated tests for reliable assessments of clinicians' ability to perform skin cancer diagnostics. To develop and gather validity evidence for a test in skin cancer diagnostics, a multiple-choice questionnaire (MCQ) was developed based on informal interviews with seven content experts from five skin cancer centers in Denmark. Validity evidence for the test was gathered from May until July 2019 using Messick's validity framework (content, response process, internal structure, relationship to other variables and consequences). Item content was revised through a Delphi-like review process and then piloted on 36 medical students and 136 doctors using a standardized response process. Results enabled an analysis of the internal structure and relationship to other variables of the test. Finally, the contrasting groups method was used to investigate the test's consequences (pass-fail standard). The initial 90-item MCQ was reduced to 40 items during the Delphi-like review process. Item analysis revealed that 25 of the 40 selected items were level I-III quality items with a high internal consistency (Cronbach's α = 0.83) and highly significant (P ≤ 0.0001) differences in test scores between participants with different occupations or levels of experience. A pass-fail standard of 12 (48%) correct answers was established using the contrasting groups' method. The skin cancer diagnostics MCQ developed in this study can be used for reliable assessments of clinicians' competencies.

The Implementation Gap in Laparoscopic Simulation Training.

BACKGROUND AND AIMS: Simulation-based training in laparoscopy can improve patient safety and efficiency of care, but it depends on how it is used. Research in medical education has moved from demonstrating transferability of simulation training to the operating room to how to best implement it. This study aims to investigate how simulation-based training in laparoscopy has been implemented Scandinavia. MATERIAL AND METHODS: An online survey was sent out to medical doctors at surgical, gynecological, and urological departments at 138 hospitals in Denmark, Norway, and Sweden. The questionnaire included questions on respondents' baseline characteristics, opinions, access, and actual use of simulation-based training in laparoscopy. RESULTS: In total, 738 respondents completed the survey. Of these, 636 (86.2%) of respondents agreed or strongly agreed that simulation-based training in laparoscopy should be mandatory. A total of 602 (81.6%) had access to simulation-based training in laparoscopy. Of the total 738 respondents, 141 (19.1%) were offered structured training courses, 129 (17.5%) were required to reach a predefined level of competency, and 66 (8.9%) had mandatory courses in laparoscopy. In all, 72 (9.8%) had never used simulation-based training in laparoscopy. CONCLUSION: An implementation gap in laparoscopic simulation-based training still exists in Scandinavia. Simulation equipment is generally available, but there is a lack of structured simulation-based training.

Amputation-Free Survival after Crural Percutaneous Transluminal Angioplasty for Critical Limb Ischemia.

BACKGROUND AND AIM: To evaluate the amputation-free survival after below the knee percutaneous transluminal angioplasty in a consecutive group of patients with critical ischemia of the lower extremity. MATERIALS AND METHODS: A total of 70 consecutive patients with critical ischemia were treated with below the knee percutaneous transluminal angioplasty at the vascular center at Rigshospitalet with the purpose of limb salvage. All patients were deemed unfit for major surgery due to anatomical limitations or severe co-morbidity, and no prior attempts of revascularization were performed. Follow-up clinical examinations were performed within 6 weeks and after 1 year. All medical records were crosschecked with the national vascular registry ensuring a valid 1-year status in 97% of the patients. RESULTS: A total of 15 major amputations were performed during follow-up, with 11 amputations performed within the first year. Complications after percutaneous transluminal angioplasty were rare. Cumulative mortality after 1 and 2 years was 22% and 34%, respectively. Amputation-free survival at 1 and 2 years of follow-up was 68% and 58%, respectively. There were no association between known risk factors such as diabetes, ischemic ulcers, cardiac disease, history of smoking, major amputation, or overall amputation. CONCLUSION: Below the knee percutaneous transluminal angioplasty in patients with end-stage peripheral arterial disease and critical limb ischemia is a safe procedure in relieving critical ischemia, reducing the short-term rate of a major amputation as opposed to best medical treatment alone.

Peak and ceiling effects in final-product analysis of mastoidectomy performance.

BACKGROUND: Virtual reality surgical simulation of mastoidectomy is a promising training tool for novices. Final-product analysis for assessing novice mastoidectomy performance could be limited by a peak or ceiling effect. These may be countered by simulator-integrated tutoring. METHODS: Twenty-two participants completed a single session of self-directed practice of the mastoidectomy procedure in a virtual reality simulator. Participants were randomised for additional simulator-integrated tutoring. Performances were assessed at 10-minute intervals using final-product analysis. RESULTS: In all, 45.5 per cent of participants peaked before the 60-minute time limit. None of the participants achieved the maximum score, suggesting a ceiling effect. The tutored group performed better than the non-tutored group but tutoring did not eliminate the peak or ceiling effects. CONCLUSION: Timing and adequate instruction is important when using final-product analysis to assess novice mastoidectomy performance. Improved real-time feedback and tutoring could address the limitations of final product based assessment.

Module based training improves and sustains surgical skills: a randomised controlled trial.

PURPOSE: Traditional surgical training is challenged by factors such as patient safety issues, economic considerations and lack of exposure to surgical procedures due to short working hours. A module-based clinical training model promotes rapidly acquired and persistent surgical skills. METHODS: A randomised controlled trial concerning supervised hernia repair in eight training hospitals in Denmark was performed. The participants were 18 registrars [Post graduate year (PGY) 3 or more] in their first year of surgical specialist training. The intervention consisted of different modules with a skills-lab course followed by 20 supervised Lichtenstein hernia repairs. Operative performance was video recorded and blindly rated by two consultants using a previously validated skills rating scale (8-40 points). Outcome measures were change in the ratings of operative skills and operative time. RESULTS: In the intervention group (n = 10) the average rating of operative skills before intervention was 22.5 (20.6-24.3) and after 26.2 (23.5-28.8), p = 0.044. At follow-up after 1 year, rating was 26.9 (23.4-30.4), p = 0.019. In the conventionally trained group average rating was 23.4 (19.4-27.3) at start and 21.7 (17.3-26.1) at end, p = 0.51. At start no difference was detected between the two groups, p = 0.59; by 1 year the difference was statistically significant favouring intervention, p = 0.044. Operative time showed similar results in favour of the intervention. CONCLUSIONS: A module-based training model in Lichtenstein hernia repair was preferable in both short and long-term compared with standard clinical training. The model will probably be applicable to other surgical training procedures.

Reliable and valid assessment of Lichtenstein hernia repair skills.

PURPOSE: Lichtenstein hernia repair is a common surgical procedure and one of the first procedures performed by a surgical trainee. However, formal assessment tools developed for this procedure are few and sparsely validated. The aim of this study was to determine the reliability and validity of an assessment tool designed to measure surgical skills in Lichtenstein hernia repair. METHODS: Key issues were identified through a focus group interview. On this basis, an assessment tool with eight items was designed. Ten surgeons and surgical trainees were video recorded while performing Lichtenstein hernia repair, (four experts, three intermediates, and three novices). The videos were blindly and individually assessed by three raters (surgical consultants) using the assessment tool. Based on these assessments, validity and reliability were explored. RESULTS: The internal consistency of the items was high (Cronbach's alpha = 0.97). The inter-rater reliability was very good with an intra-class correlation coefficient (ICC) = 0.93. Generalizability analysis showed a coefficient above 0.8 even with one rater. The coefficient improved to 0.92 if three raters were used. One-way analysis of variance found a significant difference between the three groups which indicates construct validity, p < 0.001. CONCLUSIONS: Lichtenstein hernia repair skills can be assessed blindly by a single rater in a reliable and valid fashion with the new procedure-specific assessment tool. We recommend this tool for future assessment of trainees performing Lichtenstein hernia repair to ensure that the objectives of competency-based surgical training are met.