Development and validation of the TOCO-TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour.

BACKGROUND: The current shift towards competency-based residency training has increased the need for objective assessment of skills. In this study, we developed and validated an assessment tool that measures technical and non-technical competency in transurethral resection of bladder tumour (TURBT). METHODS: The 'Test Objective Competency' (TOCO)-TURBT tool was designed by means of cognitive task analysis (CTA), which included expert consensus. The tool consists of 51 items, divided into 3 phases: preparatory (n = 15), procedural (n = 21), and completion (n = 15). For validation of the TOCO-TURBT tool, 2 TURBT procedures were performed and videotaped by 25 urologists and 51 residents in a simulated setting. The participants' degree of competence was assessed by a panel of eight independent expert urologists using the TOCO-TURBT tool. Each procedure was assessed by two raters. Feasibility, acceptability and content validity were evaluated by means of a quantitative cross-sectional survey. Regression analyses were performed to assess the strength of the relation between experience and test scores (construct validity). Reliability was analysed by generalizability theory. RESULTS: The majority of assessors and urologists indicated the TOCO-TURBT tool to be a valid assessment of competency and would support the implementation of the TOCO-TURBT assessment as a certification method for residents. Construct validity was clearly established for all outcome measures of the procedural phase (all r > 0.5, p < 0.01). Generalizability-theory analysis showed high reliability (coefficient Phi ≥ 0.8) when using the format of two assessors and two cases. CONCLUSIONS: This study provides first evidence that the TOCO-TURBT tool is a feasible, valid and reliable assessment tool for measuring competency in TURBT. The tool has the potential to be used for future certification of competencies for residents and urologists. The methodology of CTA might be valuable in the development of assessment tools in other areas of clinical practice.

The Simbla TURBT Simulator in Urological Residency Training: From Needs Analysis to Validation.

OBJECTIVE: To investigate the value of the physical Simbla Transurethral Resection of a Bladder Tumor (TURBT) simulator as an educational tool within urological residency training, by means of a training needs analysis (TNA) and assessment of its feasibility, acceptability, and face, content, and construct validity. METHODS: To analyze the training needs for TURBT, procedural steps and pitfalls were identified and the TNA was completed during an expert consensus meeting. Participants (n = 76) were divided into three groups based on their experience in TURBT: novices, intermediates, and experts. Participants performed two standardized TURBT procedures on the simulator. Face validity and content validity, as well as feasibility and acceptability, were assessed with a quantitative survey. Construct validity was assessed by comparing the performance of novices, intermediates, and experts on resection time, quality of tumor resection, and overall performance. RESULTS: Of the 21 procedural steps and 17 pitfalls defined in TNA, 13 steps and 8 pitfalls were covered by the Simbla. Participants rated the Simbla's overall realism (face validity) with a score of 8 of 10 (range 6-9). The simulator was judged to be most useful (content validity) for learning eye-hand coordination: score 8 (6-10). All aspects regarding realism and usefulness were rated above the acceptability threshold of 6/10. Intermediates (100%) and experts (96%) considered the Simbla to be a useful educational tool within the urological curriculum. Resection time was longer for novices than for experts (p < 0.05; construct validity). In addition, the overall performance of novices was rated lower compared with intermediates and experts, and novices showed more irradical resections and bladder perforations (all p < 0.05). CONCLUSIONS: The Simbla TURBT simulator is a valid, feasible, and acceptable educational tool for training procedural skills and may be implemented in the urological curriculum to complement learning in clinical practice. TNA is valuable in defining training objectives and evaluating the educational value of a simulator.

Effect of comprehensive oncogenetics training interventions for general practitioners, evaluated at multiple performance levels.

General practitioners (GPs) are increasingly called upon to identify patients at risk for hereditary cancers, and their genetic competencies need to be enhanced. This article gives an overview of a research project on how to build effective educational modules on genetics, assessed by randomized controlled trials (RCTs), reflecting the prioritized educational needs of primary care physicians. It also reports on an ongoing study to investigate long-term increase in genetic consultation skills (1-year follow-up) and interest in and satisfaction with a supportive website on genetics among GPs. Three oncogenetics modules were developed: an online Continuing Professional Development (G-eCPD) module, a live genetic CPD module, and a "GP and genetics" website (huisartsengenetica.nl) providing further genetics information applicable in daily practice. Three assessments to evaluate the effectiveness (1-year follow-up) of the oncogenetic modules were designed: 1.An online questionnaire on self-reported genetic competencies and changes in referral behaviour, 2.Referral rates from GPs to clinical genetics centres and 3.Satisfaction questionnaire and visitor count analytics of supportive genetics website. The setting was Primary care in the Netherlands and three groups of study participants were included in the reported studies:. Assessment 1. 168 GPs responded to an email invitation and were randomly assigned to an intervention or control group, evaluating the G-eCPD module (n = 80) or the live module (n = 88). Assessment 2. Referral rates by GPs were requested from the clinical genetics centres, in the northern and southern parts of the Netherlands (Amsterdam and Maastricht), for the two years before (2010 [n = 2510] and 2011 [n = 2940]) and the year after (2012 [n = 2875]) launch of the oncogenetics CPD modules and the website. Assessment 3. Participants of the website evaluation were all recruited online. When they visited the website during the month of February 2013, a pop-up invitation came up. Of the 1350 unique visitors that month, only 38 completed the online questionnaire. Main outcomes measure showed long-term (self-reported) genetic consultation skills (i.e. increased genetics awareness and referrals to clinical genetics centres) among GPs who participated in the oncogenetic training course, and interest in and satisfaction with the supportive website. 42 GPs (52%) who previously participated in the G-eCPD evaluation study and 50 GPs (57%) who participated in the live training programme responded to the online questionnaire on long-term effects of educational outcome. Previous RCTs showed that the genetics CPD modules achieved sustained improvement of oncogenetic knowledge and consultation skills (3-months follow-up). Participants of these RCTs reported being more aware of genetic problems long term; this was reported by 29 GPs (69%) and 46 GPs (92%) participating in the G-eCPD and live module evaluation studies, respectively (Chisquare test, p<0.005). One year later, 68% of the respondents attending the live training reported that they more frequently referred patients to the clinical genetics centres, compared to 29% of those who attended the online oncogenetics training (Chisquare test, p<0.0005). However, the clinical genetics centres reported no significant change in referral numbers one year after the training. Website visitor numbers increased, as did satisfaction, reflected in a 7.7 and 8.1 (out of 10) global rating of the website (by G-eCPD and live module participants, respectively). The page most often consulted was "family tree drawing". Self-perceived genetic consultation skills increased long-term and GPs were interested in and satisfied with the supportive website. Further studies are necessary to see whether the oncogenetics CPD modules result in more efficient referral. The results presented suggest we have provided a flexible and effective framework to meet the need for effective educational programmes for non-geneticist healthcare providers, enabling improvement of genetic medical care.

Effectiveness of oncogenetics training on general practitioners' consultation skills: a randomized controlled trial.

PURPOSE: General practitioners are increasingly called upon to deliver genetic services and could play a key role in translating potentially life-saving advancements in oncogenetic technologies to patient care. If general practitioners are to make an effective contribution in this area, their genetics competencies need to be upgraded. The aim of this study was to investigate whether oncogenetics training for general practitioners improves their genetic consultation skills. METHODS: In this pragmatic, blinded, randomized controlled trial, the intervention consisted of a 4-h training (December 2011 and April 2012), covering oncogenetic consultation skills (family history, familial risk assessment, and efficient referral), attitude (medical ethical issues), and clinical knowledge required in primary-care consultations. Outcomes were measured using observation checklists by unannounced standardized patients and self-reported questionnaires. RESULTS: Of 88 randomized general practitioners who initially agreed to participate, 56 completed all measurements. Key consultation skills significantly and substantially improved; regression coefficients after intervention were equivalent to 0.34 and 0.28 at 3-month follow-up, indicating a moderate effect size. Satisfaction and perceived applicability of newly learned skills were highly scored. CONCLUSION: The general practitioner-specific training proved to be a feasible, satisfactory, and clinically applicable method to improve oncogenetics consultation skills and could be used as an educational framework to inform future training activities with the ultimate aim of improving medical care.

Sustained effects of online genetics education: a randomized controlled trial on oncogenetics.

Medical professionals are increasingly expected to deliver genetic services in daily patient care. However, genetics education is considered to be suboptimal and in urgent need of revision and innovation. We designed a Genetics e-learning Continuing Professional Development (CPD) module aimed at improving general practitioners' (GPs') knowledge about oncogenetics, and we conducted a randomized controlled trial to evaluate the outcomes at the first two levels of the Kirkpatrick framework (satisfaction, learning and behavior). Between September 2011 and March 2012, a parallel-group, pre- and post-retention (6-month follow-up) controlled group intervention trial was conducted, with repeated measurements using validated questionnaires. Eighty Dutch GP volunteers were randomly assigned to the intervention or the control group. Satisfaction with the module was high, with the three item's scores in the range 4.1-4.3 (5-point scale) and a global score of 7.9 (10-point scale). Knowledge gains post test and at retention test were 0.055 (P<0.05) and 0.079 (P<0.01), respectively, with moderate effect sizes (0.27 and 0.31, respectively). The participants appreciated applicability in daily practice of knowledge aspects (item scores 3.3-3.8, five-point scale), but scores on self-reported identification of disease, referral to a specialist and knowledge about the possibilities/limitations of genetic testing were near neutral (2.7-2.8, five-point scale). The Genetics e-learning CPD module proved to be a feasible, satisfactory and clinically applicable method to improve oncogenetics knowledge. The educational effects can inform further development of online genetics modules aimed at improving physicians' genetics knowledge and could potentially be relevant internationally and across a wider range of potential audiences.

Results of the European Basic Laparoscopic Urological Skills examination.

BACKGROUND: In 2011, the European Basic Laparoscopic Urological Skills (E-BLUS) examination was introduced as a pilot for the examination of final-year urologic residents. OBJECTIVE: In this study, we aimed to answer the following research questions: What level of laparoscopic skills do final-year residents in urology have in Europe, and do the participants of the E-BLUS pass the examination according to the validated criteria? DESIGN, SETTING, AND PARTICIPANTS: Participants of the examination were final-year urology residents from different European countries taking part in the European Urology Residents Education Program in 2011 and 2012. SURGICAL PROCEDURE: The E-BLUS exam consists of five tasks validated for the training of basic urologic laparoscopic skills. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: Performances of the tasks were recorded on DVD and analysed by an objective rater. Time and number of errors made in tasks 1-4 were noted. Furthermore, all expert laparoscopic urologists were asked to score participants on a global rating scale (1-5) based on three items: depth perception, bimanual dexterity, and efficiency. Participants were asked to complete a questionnaire on prior training and laparoscopic experience. RESULTS AND LIMITATIONS: Seventy DVD recordings were analysed. Most participants did not pass the time criteria on task 4 (90%), task 2 (85.7%), task 1 (74.3%), and task 5 (71.4%). Task 3 was passed by 84.3%. The overall quality score was passed by 64%. When combining time and quality, only three participants (4.2%) passed the examination according to the validated criteria. According to the questionnaire, 61% did not have the opportunity to train in laparoscopic skills. CONCLUSIONS: The results of the E-BLUS examination show that the level of basic laparoscopic skills among European residents is low. Although quality of performance is good, most residents do not pass the validated time criteria. Regular laparoscopic training or a dedicated fellowship should improve the laparoscopic level of residents in urology.

Program for laparoscopic urological skills assessment: setting certification standards for residents.

AIM: There is growing pressure from the government and the public to define proficiency standards for surgical skills. Aim of this study was to estimate the reliability of the Program for Laparoscopic Urological Skills (PLUS) assessment and to set a certification standard for second-year urological residents. METHODS: Fifty participants were assessed on performance time and performance quality to investigate the reliability of the PLUS assessment. Generalisability coefficient of 0.8, on a scale of 0 to 1.0, was considered to indicate good reliability for assessment purposes. Pass/fail standards were based on laparoscopic experience: Novices, intermediates, and experts (>100 procedures). The pass/fail standards were investigated for the PLUS performances of 33 second-year urological residents. RESULTS: Fifteen novices, twenty-three intermediates and twelve experts were included. An inter-trial reliability of >0.80 was reached with two trials for each task. Inter-rater reliability of the quality measurements was 0.79 for two judges. Pass/fail scores were determined for the novice/intermediate boundary and the intermediate/expert boundary. Pass rates for second-year residents were 63.64% and 9.09%, respectively. CONCLUSION: The PLUS assessment is reliable for setting a certification standard for second-year urological residents that serves as a starting point for residents to proceed to the next level of laparoscopic competency.

Program for laparoscopic urologic skills: a newly developed and validated educational program.

OBJECTIVE: To develop and evaluate a program for laparoscopic urologic skills (PLUS) to determine the face, content, and construct validation to achieve uniformity and standardization in training residents in urology. METHODS: The PLUS consists of 5 basic laparoscopic tasks. Three tasks were abstracted from the Fundamentals of Laparoscopic Surgery program, and 2 additional tasks were developed under continuous evaluation by expert urologists. Fifty participants were recruited from different hospitals and performed the final PLUS training. They all completed a questionnaire after performance. Three outcome parameters were measured: performance quality, time, and dropped objects. The relationship between laparoscopic experience and the outcome parameters was investigated. RESULTS: Of the 50 participants, 13 were students, 20 were residents, and 17 were urologists. Double-log linear regression analysis for all 5 tasks showed a significant effect (effect size range 0.53-0.82; P < .0005) for laparoscopic experience on performance time. Substantial correlations were found between experience and quality ratings (log-linear regression effect size 0.37; P = .012) and the number of dropped objects (Spearman correlation effect size 0.49; P < .01). The usefulness of the PLUS model as a training tool for basic laparoscopic skills was rated 4.55 on a scale from 1 (not useful) to 5 (useful) (standard deviation 0.58; range 3-5). CONCLUSION: The results of the present study indicated the face, content, and construct validity for the PLUS. The training is considered appropriate for use as a primary training tool for an entry test or as part of a step-wise training program in which basic and procedural laparoscopic skills are integrated.

Effect of distraction on the performance of endourological tasks: a randomized controlled trial.

OBJECTIVE: • To establish the effect of distraction on the performance of cystoscopy and basic endourological tasks by using a virtual reality (VR) simulator. SUBJECTS AND METHODS: • A total of 86 third-year medical students from Maastricht University, who had no previous experience in performing the tasks on a VR simulator, were randomly assigned to an intervention or control group. • All participants performed three endourological tasks on the VR simulator. Participants in the intervention group were distracted 1 min into the third task. The distraction consisted of being asked to answer questions about a medical case that had been presented to all the participants before the hands-on session. After two adequate verbal responses the conversation was terminated. • Number of traumata, number of missed lesions in the bladder and time to completion were measured by the VR simulator. RESULTS: • Number of traumata and missed lesions, as well as time to completion were significantly higher in the intervention than in the control group with effect sizes (using Cohen's categorization) of 0.48, 0.41 and 0.50 respectively. • Nevertheless, only 9.5% of the participants in the intervention group reported feeling burdened by the distraction. CONCLUSIONS: • Distraction during the performance of endourological skills results in significantly poorer performance by medical students on all the variables measured in a controlled learning environment. • Most students do not realize they are affected by distraction. • Further research is needed to determine the impact of distraction on more experienced participants and on patient safety.

The effect of a low-fidelity model on cystoscopic skill training: a single-blinded randomized controlled trial.

INTRODUCTION: Models for training urological procedures without burdening patients are available at varying costs. We examined the value of training on a low-fidelity model in addition to training on a high-fidelity simulator in a cystoscopy training program. METHODS: Thirty-two medical students were randomized to an intervention and a control group. The former started by performing cystoscopy on a low-cost, low-fidelity, glass globe model before moving on to training on the URO Mentor (UM), a computerized simulator. The control group took part in the same UM training program but not in the low-fidelity training. Performance on UM was assessed by a global rating score, percentage of correctly inspected areas of the bladder (% inspected areas), time, and number of traumas caused. RESULTS: The intervention group had generally higher scores. Its global rating score on task 1 was significantly higher than that of the control group (Mann-Whitney U test, P = 0.046, effect size 0.6) and the group also scored higher, albeit not significantly, on time and % inspected areas. All students said they valued training with UM, but the appreciation of the intervention group was stronger (mean 8.9 vs. 8.1 on a scale from 1 to 10, P = 0.017, effect size 1.8). CONCLUSION: A low-fidelity glass globe model seemed to be an inexpensive educational tool to practice the first steps of cystoscopy. It may reduce training time on the UM simulator. The combined use of a low- and high-fidelity training model may provide an optimal learning effect.

Transfer of cysto-urethroscopy skills from a virtual-reality simulator to the operating room: a randomized controlled trial.

OBJECTIVE: To assess whether real-time cysto-urethroscopy (CUS) performance improves by simulator-based training (criterion or predictive validity), addressing the research question 'Does practical skills training on the URO Mentor (UM, Simbionix USA Corp., Cleveland, OH, USA) virtual-reality simulator improve the performance of flexible CUS in patients'. SUBJECTS AND METHODS: Participants (71 interns from Catharina Hospital Eindhoven, CHE, and 29 from University Medical Centre Groningen, UMCG) were randomized to carry out CUS in a patient after training on the UM (UM-trained, 50) or without training on UM (control, 50). The assessment of real-time performance consisted of scoring on a Global Rating Scale (GRS) by supervisors unaware of training status. Data were analysed using stepwise multiple linear regression. The effect size (ES) indication for correlations was used to interpret the magnitude of a standard regression coefficient (beta); an ES of 0.10, 0.30 and 0.50 were considered small, moderate and large, respectively. The study was approved by the Medical Review Ethics Committees of the participating hospitals. RESULTS: Overall, the group that received training performed significantly better than the controls (P < or = 0.003, beta range 0.30-0.47). There was no effect of training for participants with a specific preference for a surgical speciality in two of five GRS scores. Participants from CHE obtained higher GRS 3 scores than those from UMCG. Significantly more UMCG trainees indicated having had stress than those from CHE (P < 0.001). CONCLUSIONS: The results showed that interns who had trained on UM outperformed controls for a CUS procedure in a patient. Training for CUS on the UM is to be recommended for learning to respect tissue, procedural knowledge, flow of procedure and forward planning. Use of the UM to train interns with a specific interest in a surgical speciality in handling instruments, and time and motion, seems to be of limited value.

Acquisition of flexible cystoscopy skills on a virtual reality simulator by experts and novices.

OBJECTIVE: To assess the construct validity of the URO Mentor (Simbionix Corp., Cleveland, OH, USA) virtual reality training model for several variables of skills training in cysto-urethroscopy, addressing two research questions: (i) Does training on the URO Mentor significantly improve novices' performance in terms of time, trauma, areas inspected and Global Rating Scale (GRS) score?; (ii) is discrimination between different levels of expertise possible using the URO Mentor? METHODS: Thirty experts and 50 novices performed seven tasks on the URO Mentor during one training session. The first, fourth and seventh tasks were 'test tasks' to evaluate participants' performance. The simulator recorded procedure time and trauma; a supervisor scored which areas were inspected and gave scores on the GRS. A two-way analysis of variance with repeated-measures test was used to analyse experts' and novices' performances, with P < 0.05 considered to indicate statistical significance. Effect sizes (ES) were calculated to quantify the practical significance of the results; ES of 0.10, 0.30, and 0.50 were considered small, medium and large, respectively. RESULTS: Novices' performances showed a significant improvement with large ES in time (linear trend of learning curve P < 0.001, ES 0.66) and mean GRS score (linear trend P < 0.001, ES 0.84, quadratic trend P = 0.018, ES 0.24). There was a medium improvement for trauma (linear trend P < 0.001, ES 0.40) and a small improvement in areas inspected (linear trend P = 0.032, ES 0.21). That the 95% confidence intervals of the measures on the first task of experts and novices did not coincide indicates that differentiation between experts and novices on the four variables measured can be achieved using the URO Mentor. CONCLUSIONS: Training on the URO Mentor appears to result in a medium to large improvement of novices' performances for time, trauma, areas inspected and GRS scores. Moreover, discrimination between different levels of expertise is possible using this simulator.