Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment.

BACKGROUND: Direct Observation of Procedural Skills (DOPS) is an established competence assessment tool in endoscopy. In July 2016, the DOPS scoring format changed from a performance-based scale to a supervision-based scale. We aimed to evaluate the impact of changes to the DOPS scale format on the distribution of scores in novice trainees and on competence assessment. METHODS: We performed a prospective, multicenter (n = 276), observational study of formative DOPS assessments in endoscopy trainees with ≤ 100 lifetime procedures. DOPS were submitted in the 6-months before July 2016 (old scale) and after (new scale) for gastroscopy (n = 2998), sigmoidoscopy (n = 1310), colonoscopy (n = 3280), and polypectomy (n = 631). Scores for old and new DOPS were aligned to a 4-point scale and compared. RESULTS: 8219 DOPS (43 % new and 57 % old) submitted for 1300 trainees were analyzed. Compared with old DOPS, the use of the new DOPS was associated with greater utilization of the lowest score (2.4 % vs. 0.9 %; P < 0.001), broader range of scores, and a reduction in competent scores (60.8 % vs. 86.9 %; P < 0.001). The reduction in competent scores was evident on subgroup analysis across all procedure types (P < 0.001) and for each quartile of endoscopy experience. The new DOPS was superior in characterizing the endoscopy learning curve by demonstrating progression of competent scores across quartiles of procedural experience. CONCLUSIONS: Endoscopy assessors applied a greater range of scores using the new DOPS scale based on degree of supervision in two cohorts of trainees matched for experience. Our study provides construct validity evidence in support of the new scale format.

Development and implementation of the Structured Training Trainer Assessment Report (STTAR) in the English National Training Programme for laparoscopic colorectal surgery.

BACKGROUND: There is a lack of educational tools available for surgical teaching critique, particularly for advanced laparoscopic surgery. The aim was to develop and implement a tool that assesses training quality and structures feedback for trainers in the English National Training Programme for laparoscopic colorectal surgery. METHODS: Semi-structured interviews were performed and analysed, and items were extracted. Through the Delphi process, essential items pertaining to desirable trainer characteristics, training structure and feedback were determined. An assessment tool (Structured Training Trainer Assessment Report-STTAR) was developed and tested for feasibility, acceptability and educational impact. RESULTS: Interview transcripts (29 surgical trainers, 10 trainees, four educationalists) were analysed, and item lists created and distributed for consensus opinion (11 trainers and seven trainees). The STTAR consisted of 64 factors, and its web-based version, the mini-STTAR, included 21 factors that were categorised into four groups (training structure, training behaviour, trainer attributes and role modelling) and structured around a training session timeline (beginning, middle and end). The STTAR (six trainers, 48 different assessments) demonstrated good internal consistency (α = 0.88) and inter-rater reliability (ICC = 0.75). The mini-STTAR demonstrated good inter-item reliability (α = 0.79) and intra-observer reliability on comparison of 85 different trainer/trainee combinations (r = 0.701, p = <0.001). Both were found to be feasible and acceptable. The educational report for trainers was found to be useful (4.4 out of 5). CONCLUSIONS: An assessment tool that evaluates training quality was developed and shown to be reliable, acceptable and of educational value. It has been successfully implemented into the English National Training Programme for laparoscopic colorectal surgery.

The national colonoscopy audit: a nationwide assessment of the quality and safety of colonoscopy in the UK.

OBJECTIVE: To perform a comprehensive audit of all colonoscopy undertaken in the UK over a 2-week period. DESIGN: Multi-centre survey. All adult (≥16 years of age) colonoscopies that took place in participating National Health Service hospitals between 28 February 2011 and 11 March 2011 were included. RESULTS: Data on 20,085 colonoscopies and 2681 colonoscopists were collected from 302 units. A validation exercise indicated that data were collected on over 94% of all procedures performed nationally. The unadjusted caecal intubation rate (CIR) was 92.3%. When adjusted for impassable strictures and poor bowel preparation the CIR was 95.8%. The polyp detection rate was 32.1%. The polyp detection rate for larger polyps (≥10 mm diameter) was 11.7%. 92.3% of resected polyps were retrieved. 90.2% of procedures achieved acceptable levels of patient comfort. A total of eight perforations and 52 significant haemorrhages were reported. Eight patients underwent surgery as a consequence of a complication. CONCLUSIONS: This is the first national audit of colonoscopy that has successfully captured the majority of adult colonoscopies performed across an entire nation during a defined time period. The data confirm that there has been a significant improvement in the performance of colonoscopy in the UK since the last study reported seven years ago (CIR 76.9%) and that performance is above the required national standards.

The Global Rating Scale in clinical practice: a comprehensive quality assurance programme for endoscopy departments.

BACKGROUND: The Global Rating Scale is an endoscopy quality assurance programme, successfully implemented in England. It remains uncertain whether it is applicable in another health care setting. AIM: To assess the applicability of the Global Rating Scale as benchmark tool in an international context. METHODS: Eleven Dutch endoscopy departments were included for a Global Rating Scale-census, performed as a cross-sectional evaluation, July 2010. Two Global Rating Scale-dimensions - 'clinical quality' and 'patient experience' - were assessed across six items using a range of levels: from level-D (basic) to level-A (excellent). Construct validity was assessed by comparing department-specific colonoscopy audit data to GRS-levels. RESULTS: For 'clinical quality', variable scores were achieved in items 'safety' (9%=B, 27%=C, 64%=D) and 'communication' (46%=A, 18%=C, 36%=D). All departments achieved a basic score in 'quality' (100%=D). For 'patient experience', variable scores were achieved in 'timeliness' (18%=A, 9%=B, 73%=D) and 'booking-choice' (36%=B, 46%=C, 18%=D). All departments achieved basic scores in 'equality' (100%=D). Departments obtaining level-C or above in 'information', 'comfort', 'communication', 'timeliness' and 'aftercare', achieved significantly better audit outcomes compared to those obtaining level-D (p<0.05). CONCLUSION: The Global Rating Scale is appropriate to use outside England. There was significant variance across departments in dimensions. Most Global Rating Scale-levels were in line with departments' audit outcomes, indicating construct validity.