Correction to: Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

The citation for Reference 22 should be replaced with: Kumar NL, Kugener G, Perencevich ML, et al (2018) The SAFE-T assessment tool: derivation and validation of a web-based application for point-of-care evaluation of gastroenterology fellow performance in colonoscopy. Gastrointest Endosc 87(1):262-269.

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

BACKGROUND: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. METHODS: This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. RESULTS: In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. CONCLUSION: This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training.

INTRODUCTION: Formative colonoscopy direct observation of procedural skills (DOPS) assessments were updated in 2016 and incorporated into UK training but lack validity evidence. We aimed to appraise the validity of DOPS assessments, benchmark performance, and evaluate competency development during training in diagnostic colonoscopy. METHODS: This prospective national study identified colonoscopy DOPS submitted over an 18-month period to the UK training e-portfolio. Generalizability analyses were conducted to evaluate internal structure validity and reliability. Benchmarking was performed using receiver operator characteristic analyses. Learning curves for DOPS items and domains were studied, and multivariable analyses were performed to identify predictors of DOPS competency. RESULTS: Across 279 training units, 10,749 DOPS submitted for 1,199 trainees were analyzed. The acceptable reliability threshold (G > 0.70) was achieved with 3 assessors performing 2 DOPS each. DOPS competency rates correlated with the unassisted caecal intubation rate (rho 0.404, P < 0.001). Demonstrating competency in 90% of assessed items provided optimal sensitivity (90.2%) and specificity (87.2%) for benchmarking overall DOPS competence. This threshold was attained in the following order: "preprocedure" (50-99 procedures), "endoscopic nontechnical skills" and "postprocedure" (150-199), "management" (200-249), and "procedure" (250-299) domain. At item level, competency in "proactive problem solving" (rho 0.787) and "loop management" (rho 0.780) correlated strongest with the overall DOPS rating (P < 0.001) and was the last to develop. Lifetime procedure count, DOPS count, trainer specialty, easier case difficulty, and higher cecal intubation rate were significant multivariable predictors of DOPS competence. DISCUSSION: This study establishes milestones for competency acquisition during colonoscopy training and provides novel validity and reliability evidence to support colonoscopy DOPS as a competency assessment tool.

Training and assessment in flexible sigmoidoscopy: using a novel direct observation of procedural skills (DOPS) assessment tool.

BACKGROUND AND AIMS: Data supporting milestone development during flexible sigmoidoscopy (FS) training are lacking. We aimed to present validity evidence for our formative direct observation of procedural skills (DOPS) assessment in FS, and use DOPS to establish competency benchmarks and define learning curves for a national training cohort. METHODS: This prospective UK-wide (211 centres) study included all FS formative DOPS assessments submitted to the national e-portfolio. Reliability was estimated from generalisability theory analysis. Item and global DOPS scores were correlated with lifetime procedure count to study learning curves, with competency benchmarks defined using contrasting groups analysis. Multivariable binary logistic regression was performed to identify independent predictors of DOPS competence. RESULTS: This analysis included 3,616 DOPS submitted for 468 trainees. From generalisability analysis, sources of overall competency score variance included: trainee ability (27%), assessor stringency (15%), assessor subjectivity attributable to the trainee (18%) and case-to-case variation (40%), which enabled the modelling of reliability estimates. The competency benchmark (mean DOPS score: 3.84) was achieved after 150-174 procedures. Across the cohort, competency development occurred in the order of: pre-procedural (50-74), non-technical (75-149), technical (125-174) and post-procedural (175-199) skills. Lifetime procedural count (p<0.001), case difficulty (p<0.001), and lifetime formative DOPS count (p=0.001) were independently associated with DOPS competence, but not trainee or assessor specialty. CONCLUSION: Sigmoidoscopy DOPS can provide valid and reliable assessments of competency during training and can be used to chart competency development. Contrary to earlier studies, based on destination-orientated endpoints, overall competency in sigmoidoscopy was attained after 150 lifetime procedures.

Implementing workplace-based assessment across the medical specialties in the United Kingdom.

OBJECTIVES: To evaluate the reliability and feasibility of assessing the performance of medical specialist registrars (SpRs) using three methods: the mini-clinical evaluation exercise (mini-CEX), directly observed procedural skills (DOPS) and multi-source feedback (MSF) to help inform annual decisions about the outcome of SpR training. METHODS: We conducted a feasibility study and generalisability analysis based on the application of these assessment methods and the resulting data. A total of 230 SpRs (from 17 specialties) in 58 UK hospitals took part from 2003 to 2004. Main outcome measures included: time taken for each assessment, and variance component analysis of mean scores and derivation of 95% confidence intervals for individual doctors' scores based on the standard error of measurement. Responses to direct questions on questionnaires were analysed, as were the themes emerging from open-comment responses. RESULTS: The methods can provide reliable scores with appropriate sampling. In our sample, all trainees who completed the number of assessments recommended by the Royal Colleges of Physicians had scores that were 95% certain to be better than unsatisfactory. The mean time taken to complete the mini-CEX (including feedback) was 25 minutes. The DOPS required the duration of the procedure being assessed plus an additional third of this time for feedback. The mean time required for each rater to complete his or her MSF form was 6 minutes. CONCLUSIONS: This is the first attempt to evaluate the use of comprehensive workplace assessment across the medical specialties in the UK. The methods are feasible to conduct and can make reliable distinctions between doctors' performances. With adaptation, they may be appropriate for assessing the workplace performance of other grades and specialties of doctor. This may be helpful in informing foundation assessment.

Improving the quality of outpatient clinic letters using the Sheffield Assessment Instrument for Letters (SAIL).

AIM: To improve the quality of outpatient letters used as communication between hospital and primary care doctors. METHODS: On 2 separate occasions, 15 unselected outpatient letters written by each of 7 hospital practitioners were rated by another hospital doctor and a general practitioner (GP) using the Sheffield Assessment Instrument for Letters (SAIL). Individualised feedback was provided to participants following the rating of the first set of letters. The audit cycle was completed 3 months later without forewarning by repeat assessment by the same hospital and GP assessors using the SAIL tool to see if there was any improvement in correspondence. SETTING: Single centre: general paediatric outpatient department in a large district general hospital. RESULTS: All 7 doctors available for reassessment completed the audit loop, each providing 15 outpatient letters per assessment. The mean of the quality scores, derived for each letter from the summation of a 20-point checklist and a global score, improved from 23.3 (95% CI 22.1-24.4) to 26.6 (95% CI 25.8-27.4) (P = 0.001). CONCLUSIONS: The SAIL provides a feasible and reliable method of assessing the quality and content of outpatient clinic letters. This study demonstrates that it can also provide feedback with a powerful educational impact. This approach holds real potential for appraisal and revalidation, providing an effective means for the quality improvement required by clinical governance.