Validity evidence for observational ERCP competency assessment tools: a systematic review.

BACKGROUND : Assessment of competence in endoscopic retrograde cholangiopancreatography (ERCP) is critical for supporting learning and documenting attainment of skill. Validity evidence supporting ERCP observational assessment tools has not been systematically evaluated. METHODS : We conducted a systematic search using electronic databases and hand-searching from inception until August 2021 for studies evaluating observational assessment tools of ERCP performance. We used a unified validity framework to characterize validity evidence from five sources: content, response process, internal structure, relations to other variables, and consequences. Each domain was assigned a score of 0-3 (maximum score 15). We assessed educational utility and methodological quality using the Accreditation Council for Graduate Medical Education framework and the Medical Education Research Quality Instrument, respectively. RESULTS : From 2769 records, we included 17 studies evaluating 7 assessment tools. Five tools were studied for clinical ERCP, one for simulated ERCP, and one for simulated and clinical ERCP. Validity evidence scores ranged from 2 to 12. The Bethesda ERCP Skills Assessment Tool (BESAT), ERCP Direct Observation of Procedural Skills Tool (ERCP DOPS), and The Endoscopic Ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) had the strongest validity evidence, with scores of 10, 12, and 11, respectively. Regarding educational utility, most tools were easy to use and interpret, and required minimal additional resources. Overall methodological quality (maximum score 13.5) was strong, with scores ranging from 10 to 12.5. CONCLUSIONS : The BESAT, ERCP DOPS, and TEESAT had strong validity evidence compared with other assessments. Integrating tools into training may help drive learners' development and support competency decision making.

Advanced technology for assessment of endoscopic and histological activity in ulcerative colitis: a systematic review and meta-analysis.

Background: Advanced endoscopic technologies led to significant progress in the definition of endoscopic remission of ulcerative colitis (UC) and correlate better with histological changes, compared with standard endoscopy. However, while studies have assessed the diagnostic accuracy of endoscope technologies individually, there are currently limited data comparing between technologies. As such, the aim of this systematic review was to pool data from the existing literature and compare the correlations between endoscopy and histologic disease activity scores across endoscope technologies. Methods: We searched PubMed and Embase until February 2021 for eligible studies reporting the correlation between endoscopy and histology activity scores in UC. Studies were grouped by endoscope technology as standard-definition white light (SD-WLE), high-definition white light (HD-WLE) or electronic virtual chromoendoscopy (VCE) and comparisons made between these groups. Results: A total of N = 27 studies were identified, of which N = 12 were included in a meta-analysis of correlations between endoscopic and histological activity scores. Combining these studies identified considerable heterogeneity (I 2: 89-93%) and returned a pooled correlation coefficient (ρ) for the SD-WLE group of 0.74, which did not differ significantly from HD-WLE (ρ: 0.65, p = 0.521) or VCE (ρ: 0.70, p = 0.801). In addition, N = 4 studies reported the accuracy of endoscopic activity scores on WLE and VCE to diagnose histological remission. Pooling these found significantly higher accuracy for VCE, compared with WLE [risk ratio: 1.13, 95% confidence interval (CI): 1.07-1.19, p < 0.001]. Conclusion: Activity scores assessed using endoscopy are strongly correlated with activity on histology regardless of endoscopic technology. VCE seems to be more accurate in predicting histological remission than WLE. However, given the heterogeneity between the included studies, head-to-head trials are warranted to confirm these findings.

Roadmap to resuming care for liver diseases after coronavirus disease-2019.

The global pandemic of coronavirus disease-2019 (COVID-19) has led to significant disruptions in healthcare delivery. Patients with chronic liver diseases require a high level of care and are therefore particularly vulnerable to disruptions in medical services during COVID-19. Recent data have also identified chronic liver disease as an independent risk factor for COVID-19 related hospital mortality. In response to the pandemic, national and international societies have recommended interim changes to the management of patients with liver diseases. These modifications included the implementation of telehealth, postponement or cancelation of elective procedures, and other non-urgent patient care-related activities. There is concern that reduced access to diagnosis and treatment can also lead to increased morbidity in patients with liver diseases and we may witness a delayed surge of hospitalizations related to decompensated liver disease after the COVID-19 pandemic has receded. Therefore, it is paramount that liver practices craft a comprehensive plan for safe resumption of clinical operations while minimizing the risk of exposure to patients and health-care professionals. Here, we provide a broad roadmap for how to safely resume care for patients with chronic liver disease according to various phases of the pandemic with particular emphasis on outpatient care, liver transplantation, liver cancer care, and endoscopy.

Correction to: Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

The citation for Reference 22 should be replaced with: Kumar NL, Kugener G, Perencevich ML, et al (2018) The SAFE-T assessment tool: derivation and validation of a web-based application for point-of-care evaluation of gastroenterology fellow performance in colonoscopy. Gastrointest Endosc 87(1):262-269.

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

BACKGROUND: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. METHODS: This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. RESULTS: In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. CONCLUSION: This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training.

INTRODUCTION: Formative colonoscopy direct observation of procedural skills (DOPS) assessments were updated in 2016 and incorporated into UK training but lack validity evidence. We aimed to appraise the validity of DOPS assessments, benchmark performance, and evaluate competency development during training in diagnostic colonoscopy. METHODS: This prospective national study identified colonoscopy DOPS submitted over an 18-month period to the UK training e-portfolio. Generalizability analyses were conducted to evaluate internal structure validity and reliability. Benchmarking was performed using receiver operator characteristic analyses. Learning curves for DOPS items and domains were studied, and multivariable analyses were performed to identify predictors of DOPS competency. RESULTS: Across 279 training units, 10,749 DOPS submitted for 1,199 trainees were analyzed. The acceptable reliability threshold (G > 0.70) was achieved with 3 assessors performing 2 DOPS each. DOPS competency rates correlated with the unassisted caecal intubation rate (rho 0.404, P < 0.001). Demonstrating competency in 90% of assessed items provided optimal sensitivity (90.2%) and specificity (87.2%) for benchmarking overall DOPS competence. This threshold was attained in the following order: "preprocedure" (50-99 procedures), "endoscopic nontechnical skills" and "postprocedure" (150-199), "management" (200-249), and "procedure" (250-299) domain. At item level, competency in "proactive problem solving" (rho 0.787) and "loop management" (rho 0.780) correlated strongest with the overall DOPS rating (P < 0.001) and was the last to develop. Lifetime procedure count, DOPS count, trainer specialty, easier case difficulty, and higher cecal intubation rate were significant multivariable predictors of DOPS competence. DISCUSSION: This study establishes milestones for competency acquisition during colonoscopy training and provides novel validity and reliability evidence to support colonoscopy DOPS as a competency assessment tool.

ERCP assessment tool: evidence of validity and competency development during training.

BACKGROUND: The endoscopic retrograde cholangiopancreatography (ERCP) direct observation of procedural skills (DOPS) is a 27-item competency assessment tool that was developed to support UK ERCP training. We evaluated validity of ERCP DOPS and competency development during training. METHODS: This prospective study analyzed ERCP DOPS performed in the UK between July 2016 and October 2018. Reliability was measured using Cronbach's alpha, and DOPS scores were benchmarked using the contrasting groups method. The percentage of competent scores was averaged for each item, domain, and overall rating, and stratified by lifetime procedure count to evaluate learning curves. Multivariable analyses were performed to identify predictors of DOPS competence. RESULTS: 818 DOPS (109 trainees, 80 UK centers) were analyzed. Overall Cronbach's alpha was 0.961. Attaining competency in 87 % of assessed DOPS items provided the optimal competency benchmark. This was achieved in the domain sequence of: pre-procedure, post-procedure management, endoscopic non-technical skills, cannulation & imaging, and execution of selected therapy, and across all items after 200 - 249 procedures (89 %). After 300 procedures, the benchmark was reached for selective cannulation (89 %), but not for stenting (plastic 73 %; metal 70 %), sphincterotomy (80 %), and sphincteroplasty (56 %). On multivariable analysis, lifetime procedure count (P = 0.002), easier case difficulty (P < 0.001), trainee grade (P = 0.03), and higher lifetime DOPS count (P = 0.01) were predictors of DOPS competence. CONCLUSION: This study provides novel validity, reliability, and learning curve data for ERCP DOPS. Trainees should have a minimum of 300 hands-on ERCP procedures before undertaking summative assessment for independent practice.

Training and assessment in flexible sigmoidoscopy: using a novel direct observation of procedural skills (DOPS) assessment tool.

BACKGROUND AND AIMS: Data supporting milestone development during flexible sigmoidoscopy (FS) training are lacking. We aimed to present validity evidence for our formative direct observation of procedural skills (DOPS) assessment in FS, and use DOPS to establish competency benchmarks and define learning curves for a national training cohort. METHODS: This prospective UK-wide (211 centres) study included all FS formative DOPS assessments submitted to the national e-portfolio. Reliability was estimated from generalisability theory analysis. Item and global DOPS scores were correlated with lifetime procedure count to study learning curves, with competency benchmarks defined using contrasting groups analysis. Multivariable binary logistic regression was performed to identify independent predictors of DOPS competence. RESULTS: This analysis included 3,616 DOPS submitted for 468 trainees. From generalisability analysis, sources of overall competency score variance included: trainee ability (27%), assessor stringency (15%), assessor subjectivity attributable to the trainee (18%) and case-to-case variation (40%), which enabled the modelling of reliability estimates. The competency benchmark (mean DOPS score: 3.84) was achieved after 150-174 procedures. Across the cohort, competency development occurred in the order of: pre-procedural (50-74), non-technical (75-149), technical (125-174) and post-procedural (175-199) skills. Lifetime procedural count (p<0.001), case difficulty (p<0.001), and lifetime formative DOPS count (p=0.001) were independently associated with DOPS competence, but not trainee or assessor specialty. CONCLUSION: Sigmoidoscopy DOPS can provide valid and reliable assessments of competency during training and can be used to chart competency development. Contrary to earlier studies, based on destination-orientated endpoints, overall competency in sigmoidoscopy was attained after 150 lifetime procedures.

Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment.

BACKGROUND: Direct Observation of Procedural Skills (DOPS) is an established competence assessment tool in endoscopy. In July 2016, the DOPS scoring format changed from a performance-based scale to a supervision-based scale. We aimed to evaluate the impact of changes to the DOPS scale format on the distribution of scores in novice trainees and on competence assessment. METHODS: We performed a prospective, multicenter (n = 276), observational study of formative DOPS assessments in endoscopy trainees with ≤ 100 lifetime procedures. DOPS were submitted in the 6-months before July 2016 (old scale) and after (new scale) for gastroscopy (n = 2998), sigmoidoscopy (n = 1310), colonoscopy (n = 3280), and polypectomy (n = 631). Scores for old and new DOPS were aligned to a 4-point scale and compared. RESULTS: 8219 DOPS (43 % new and 57 % old) submitted for 1300 trainees were analyzed. Compared with old DOPS, the use of the new DOPS was associated with greater utilization of the lowest score (2.4 % vs. 0.9 %; P < 0.001), broader range of scores, and a reduction in competent scores (60.8 % vs. 86.9 %; P < 0.001). The reduction in competent scores was evident on subgroup analysis across all procedure types (P < 0.001) and for each quartile of endoscopy experience. The new DOPS was superior in characterizing the endoscopy learning curve by demonstrating progression of competent scores across quartiles of procedural experience. CONCLUSIONS: Endoscopy assessors applied a greater range of scores using the new DOPS scale based on degree of supervision in two cohorts of trainees matched for experience. Our study provides construct validity evidence in support of the new scale format.