Validity evidence for observational ERCP competency assessment tools: a systematic review.

BACKGROUND : Assessment of competence in endoscopic retrograde cholangiopancreatography (ERCP) is critical for supporting learning and documenting attainment of skill. Validity evidence supporting ERCP observational assessment tools has not been systematically evaluated. METHODS : We conducted a systematic search using electronic databases and hand-searching from inception until August 2021 for studies evaluating observational assessment tools of ERCP performance. We used a unified validity framework to characterize validity evidence from five sources: content, response process, internal structure, relations to other variables, and consequences. Each domain was assigned a score of 0-3 (maximum score 15). We assessed educational utility and methodological quality using the Accreditation Council for Graduate Medical Education framework and the Medical Education Research Quality Instrument, respectively. RESULTS : From 2769 records, we included 17 studies evaluating 7 assessment tools. Five tools were studied for clinical ERCP, one for simulated ERCP, and one for simulated and clinical ERCP. Validity evidence scores ranged from 2 to 12. The Bethesda ERCP Skills Assessment Tool (BESAT), ERCP Direct Observation of Procedural Skills Tool (ERCP DOPS), and The Endoscopic Ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) had the strongest validity evidence, with scores of 10, 12, and 11, respectively. Regarding educational utility, most tools were easy to use and interpret, and required minimal additional resources. Overall methodological quality (maximum score 13.5) was strong, with scores ranging from 10 to 12.5. CONCLUSIONS : The BESAT, ERCP DOPS, and TEESAT had strong validity evidence compared with other assessments. Integrating tools into training may help drive learners' development and support competency decision making.

Accuracy of self-assessment in gastrointestinal endoscopy: a systematic review and meta-analysis.

BACKGROUND: Assessment is necessary to ensure both attainment and maintenance of competency in gastrointestinal (GI) endoscopy, and this can be accomplished through self-assessment. We conducted a systematic review with meta-analysis to evaluate the accuracy of self-assessment among GI endoscopists. METHODS: This was an individual participant data meta-analysis of studies that investigated self-assessment of endoscopic competency. We performed a systematic search of the following databases: Ovid MEDLINE, Ovid EMBASE, Wiley Cochrane CENTRAL, and ProQuest Education Resources Information Center. We included studies if they were primary investigations of self-assessment accuracy in GI endoscopy that used statistical analyses to determine accuracy. We conducted a meta-analysis of studies using a limits of agreement (LoA) approach to meta-analysis of Bland-Altman studies. RESULTS: After removing duplicate entries, we screened 7138 records. After full-text review, we included 16 studies for qualitative analysis and three for meta-analysis. In the meta-analysis, we found that the LoA were wide (-41.0 % to 34.0 %) and beyond the clinically acceptable difference. Subgroup analyses found that both novice and intermediate endoscopists had wide LoA (-45.0 % to 35.1 % and -54.7 % to 46.5 %, respectively) and expert endoscopists had narrow LoA (-14.2 % to 21.4 %). CONCLUSIONS: GI endoscopists are inaccurate in self-assessment of their endoscopic competency. Subgroup analyses demonstrated that novice and intermediate endoscopists were inaccurate, while expert endoscopists have accurate self-assessment. While we advise against the sole use of self-assessment among novice and intermediate endoscopists, expert endoscopists may wish to integrate it into their practice.

Interventions to improve the quality of screening-related colonoscopy: protocol for a systematic review and network meta-analysis of randomised controlled trials.

INTRODUCTION: Colonoscopy quality can vary depending on endoscopist-related factors. Quality indicators, such as adenoma detection rate (ADR), have been adopted to reduce variations in care. Several interventions aim to improve ADR, but these fall into several domains that have traditionally been difficult to compare. We will conduct a systematic review and network meta-analysis of randomised controlled trials evaluating the efficacies of interventions to improve colonoscopy quality and report our findings according to clinically relevant interventional domains. METHODS AND ANALYSIS: We will search MEDLINE (Ovid), PubMed, EMBASE, CINAHL, Web of Science, Scopus and Evidence-Based Medicine from inception to September 2022. Four reviewers will screen for eligibility and abstract data in parallel, with two accordant entries establishing agreement and with any discrepancies resolved by consensus. The primary outcome will be ADR. Two authors will independently conduct risk of bias assessments. The analyses of the network will be conducted under a Bayesian random-effects model using Markov-chain Monte-Carlo simulation, with 10 000 burn-ins and 100 000 iterations. We will calculate the ORs and corresponding 95% credible intervals of network estimates with a consistency model. We will report the impact of specific interventions within each domain against standard colonoscopy. We will perform a Bayesian random-effects pairwise meta-analysis to assess heterogeneity based on the I2 statistic. We will assess the certainty of evidence using the Grading of Recommendations Assessment, Development and Evaluation framework for network meta-analyses. ETHICS AND DISSEMINATION: Our study does not require research ethics approval given the lack of patient-specific data being collected. The results will be disseminated at national and international gastroenterology conferences and peer-reviewed journals. PROSPERO REGISTRATION NUMBER: CRD42021291814.

Financial Conflicts of Interest in Propensity Score-Matched Studies Evaluating Biologics and Biosimilars for Inflammatory Bowel Disease.

Background: Propensity score matching (PSM), a statistical technique that estimates a treatment effect by accounting for predictor covariates, has been used to evaluate biologics for inflammatory bowel disease (IBD). Financial conflicts of interest are prevalent in the marketing of biologic medications. It is unclear whether this burden of conflicts is present among authors of PSM studies comparing IBD biologics and biosimilars. Objective: This study was aimed to determine the prevalence of financial conflicts of interest among authors of PSM studies evaluating IBD biologics and biosimilars. Methods: We conducted a systematic search for PSM studies comparing biologics and biosimilars in IBD treatment. We identified 21 eligible studies. Two independent authors extracted self-declared conflicts from the disclosures section. Each participating author was searched on the Centers for Medicare & Medicaid Services Open Payments to identify payment amounts and undisclosed conflicts. Primary outcome was the prevalence of author conflicts. Secondary analyses assessed for an association between conflict prevalence and reporting of positive outcomes. Results: Among 283 authors, conflicts were present among 41.0% (116 of 283). Twenty-three per cent (27 of 116) of author conflicts involved undisclosed payments. Studies with positive outcomes were significantly more likely to include conflicted authors than neutral studies (relative risk = 2.34, 95% confidence interval: 1.71 to 3.21, P < 0.001). Conclusions: Overall, we found a high burden of undisclosed conflicts among authors of PSM studies comparing IBD biologics and biosimilars. Given the importance of PSM studies as a means for biologic comparison and the potential for undue industry influence from these payments, authors should ensure greater transparency with reporting of industry relationships.

Self-assessment of Competence in Endoscopy: Challenges and Insights.

BACKGROUND: Endoscopists use self-assessment to monitor the development and maintenance of their skills. The accuracy of these self-assessments, which reflects how closely one's own rating corresponds to an external rating, is unclear. METHODS: In this narrative review, we critically examine the current literature on self-assessment in gastrointestinal endoscopy with the aim of informing training and practice and identifying opportunities to improve the methodological rigor of future studies. RESULTS: In the seven included studies, the evidence regarding self-assessment accuracy was mixed. When stratified by experience level, however, novice endoscopists were least accurate in their self-assessments and tended to overestimate their performance. Studies examining the utility of video-based interventions using observation of expert benchmark performances show promise as a mechanism to improve self-assessment accuracy among novices. CONCLUSIONS: Based on the results of this review, we highlight problematic areas, identify opportunities to improve the methodological rigor of future studies on endoscopic self-assessment and outline potential avenues for further exploration.

Do authors of research funded by the Canadian Institutes of Health Research comply with its open access mandate?: A meta-epidemiologic study.

BACKGROUND: Since 2008, the Canadian Institutes of Health Research (CIHR) has mandated that studies it funds either in whole or in part are required to publish their results as open access (OA) within 12 months of publication using either online repositories and/or OA journals. Yet, there is evidence that authors are poorly compliant with this mandate. Specifically, there has been an apparent decrease in OA publication after 2015, which coincides with a change in the OA policy during the same year. One particular policy change that may have contributed to this decline was lifting the requirement that authors deposit their article in an OA repository immediately upon publication. We investigated the proportion of OA compliance of CIHR-funded studies in the period before and after the policy change of 2015 with manual confirmation of both CIHR funding and OA status. METHODS AND FINDINGS: We identified CIHR-funded studies published between the years 2014 to 2017 using a comprehensive search in the Web of Science (WoS). We took a stratified random sample from all four years (i.e. 2014 to 2017), with 250 studies from each year. Two authors independently reviewed the final full-text publications retrieved from the journal web page to determine to confirm CIHR funding, as indicated in the acknowledgements or elsewhere in the paper. For each study, we also collected bibliometric data that included citation count and Altmetric attention score Statistical analyses were conducted using two-tailed Fisher's exact test with relative risk (RR). Among the 851 receiving CIHR funding published from 2014 to 2017, the percentage of CIHR-funded studies published as OA significantly decreased from 79.6% in 2014 to 70.3% in 2017 (RR = 0.88, 95% CI: 0.79-0.99, P = 0.028). When considering all four years, there was no significant difference in the percentage of CIHR-funded studies published as OA in both 2014 and 2015 compared to both 2016 and 2017 (RR = 0.97, 95% CI: 0.90-1.05, P = 0.493). Additionally, OA publications had significantly higher citation count (both in year of publication and in total) and higher attention scores (P<0.05). CONCLUSIONS: Overall, we found that there was a significant decrease in the proportion of CIHR funded studies published as OA from 2014 compared to 2017, though this difference did not persist when comparing both 2014-2015 to 2016-2017. The primary limitation was the reliance of self-reported data from authors on CIHR funding status. We posit that this decrease may be attributable to CIHR's OA policy change in 2015. Further exploration is warranted to both validate these studies using a larger dataset and, if valid, investigate the effects of potential interventions to improve the OA compliance, such as use of a CIHR publication database, and reinstatement of a policy for authors to immediately submit their findings to OA repositories upon publication.

Financial Conflicts of Interest in Clinical Practice Guidelines: A Systematic Review.

OBJECTIVE: To systematically evaluate the prevalence of disclosed and undisclosed financial conflicts of interest (FCOI) among clinical practice guidelines (CPGs). METHODS: In this systematic review, we ascertained the prevalence and types of FCOI for CPGs from January 1, 1980, to March 3, 2019. The primary outcome was the prevalence of FCOI among authors of CPGs. FCOI disclosures were compared between medical subspecialties and societies producing CPGs. RESULTS: Among the 37 studies including 14,764 total guideline authors, 45% had at least one FCOI. The prevalence of FCOI per study ranged from 6% to 100%. More authors had FCOI involving general payments (39%) compared with research payments (29%). Oncology, neurology, and gastroenterology had the highest prevalence of FCOI compared with other medical specialties. Among the 8 studies that included the monetary values in US dollars of FCOI, average payments per author ranged from $578 to $242,300. Among the 10 studies that included data on undisclosed FCOI, 32% of authors had undisclosed industry payments. CONCLUSION: There are numerous FCOI among authors of CPGs, many of which are undisclosed. Our study found a significant difference in FCOI prevalence based on types of FCOI and CPG sponsor society. Additional research is required to quantify the implications of FCOI on clinical judgment and patient care.

Colonoscopy competence assessment tools: a systematic review of validity evidence.

BACKGROUND: Assessment tools are essential for endoscopy training, being required to support feedback provision, optimize learner capabilities, and document competence. We aimed to evaluate the strength of validity evidence that supports the available colonoscopy direct observation assessment tools using the unified framework of validity. METHODS: We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until 8 April 2020. We extracted data outlining validity evidence (content, response process, internal structure, relations to other variables, and consequences) from the five sources and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI). RESULTS: From 10 841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 each assessed cognitive and integrative skills. Validity evidence scores ranged from 1-15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret, and required minimal resources. MERSQI scores ranged from 9.5-11.5 (maximum score 14.5). CONCLUSIONS: The ACE, DOPS, and GiECAT have strong validity evidence compared with other assessments. Future studies should identify barriers to widespread implementation and report on the use of these tools in credentialing examinations.

Undisclosed payments by pharmaceutical and medical device manufacturers to authors of endoscopy guidelines in the United States.

BACKGROUND AND AIMS: Payments from pharmaceutical and medical device manufacturers to authors of clinical practice guidelines (CPGs) may influence practice recommendations. It is therefore important to evaluate the completeness of financial conflict of interest (FCOI) declarations among CPG authors. METHODS: We performed a cross-sectional analysis of industry payments to authors of endoscopy guidelines published by 5 GI societies between 2014 and 2017. For each author we identified payments using the disclosure section of CPGs and the Centers for Medicare & Medicaid Services Open Payments (CMS-OP) database. We calculated the prevalence, monetary value, and type of declared and undeclared payments among authors. Payments were assessed for the calendar year of and before publication. RESULTS: Thirty-seven CPGs were included in the analysis comprising 569 author entries (91 unique individuals; 66.43% men, 92.6% physicians, 66.4% academically affiliated). Four hundred fifty-one episodes (79%) involved FCOIs, 451 (79%) had undisclosed FCOIs in the CMS-OP, and 445 (77%) had FCOIs relevant to a CPG recommendation. The median undisclosed payment value was $4807.26 (interquartile range, $334.84-$20,579.75). Male authors (odds ratio, 2.23; 95% confidence interval, 1.47-3.39) and academically affiliated authors (odds ratio, 8.87; 95% confidence interval, 5.57-14.13) were significantly more likely to have undeclared payments (P < .001). No CPGs met all National Academy of Medicine criteria. CONCLUSIONS: Recognizing concerns about the accuracy of the CMS-OP, there are substantial discrepancies between industry-reported payments and author self-disclosure. Additionally, there is a high prevalence of undisclosed payments by pharmaceutical and medical device manufacturers to these authors. Given the potential impact of these discrepancies and undisclosed payments on CPGs, more accurate reporting and alternative strategies for managing FCOI are needed.

Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial.

Background and study aims Novice endoscopists are inaccurate in self-assessment of procedures. One means of improving self-assessment accuracy is through video-based feedback. We aimed to determine the comparative effectiveness of three video-based interventions on novice endoscopists' self-assessment accuracy of endoscopic competence. Materials and methods Novice endoscopists (performed < 20 previous procedures) were recruited. Participants completed a simulated esophagogastroduodenoscopy (EGD) on a virtual reality simulator. They were then randomized to one of three groups: self-video review (SVR), which involved watching a recorded video of their own performance; benchmark review (BVR), which involved watching a video of a simulated EGD completed by an expert; and self- and benchmark video (SBVR), which involved both videos. Participants then completed two additional simulated EGD cases. Self-assessments were conducted immediately after the first procedure, after the video intervention and after the additional two procedures. External assessments were conducted by two experienced endoscopists, who were blinded to participant identity and group assignment through video recordings. External and self-assessments were completed using the global rating scale component of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT GRS). Results Fifty-one participants completed the study. The BVR group had significantly improved self-assessment accuracy in the short-term, compared to the SBVR group ( P  = .005). The SBVR group demonstrated significantly improved self-assessment accuracy over time ( P  = .016). There were no significant effects of group or of time for the SVR group. Conclusions Video-based interventions, particularly combined use of self- and benchmark video review, can improve accuracy of self-assessment of endoscopic competence among novices.

Financial Conflicts of Interest in Inflammatory Bowel Disease Guidelines.

BACKGROUND: Industry payments can lead to financial conflicts of interest (FCOI) among authors of clinical practice guidelines (CPGs). Guidelines for inflammatory bowel disease (IBD) may be at particularly high risk. We determined the prevalence of FCOI in IBD CPGs produced by various gastroenterology societies. METHODS: We conducted a cross-sectional analysis of FCOI disclosure among CPGs related to the management of IBD. We ascertained the prevalence and types of FCOI for each guideline and determined adherence to National Academy of Medicine (NAM) standards. FCOI disclosures were compared between societies producing CPGs. RESULTS: We identified 11 relevant CPGs with 173 total authors. There were 117 (68%) authors who declared a payment. A total of 107 (62%) authors declared FCOI related to a medication recommended in the guideline. There was a significant difference (P < 0.001) between the proportion of authors with FCOI between countries or regions. Authors of US CPGs had a significantly lower FCOI prevalence (19%) compared with other societies. Authors of UK CPGs had a significantly lower FCOI prevalence (56%) compared with Canadian (84%) and European (94%) CPGs. Three (27%) guidelines adhered to both NAM standards. CONCLUSIONS: A substantial portion of authors of IBD CPGs had FCOI. Our study found a significant difference in FCOI prevalence based on CPG sponsor nationality. Most CPGs for IBD did not adhere to NAM standards for FCOI disclosure.

Can Pediatric Endoscopists Accurately Assess Their Clinical Competency? A Comparison Across Skill Levels.

BACKGROUND: Assessment is critical to support pediatric endoscopy training. Although trainee engagement in assessment is encouraged, the use of self-assessment and its accuracy among pediatric endoscopists is not well described. We aimed to determine the self-assessment accuracy of novice, intermediate, and experienced pediatric endoscopists. METHODS: Novice (performed <50 previous colonoscopies), intermediate (50-500), and experienced (>1000) pediatric endoscopists from 3 North American academic teaching hospitals each performed a clinical colonoscopy. Endoscopists were assessed in real-time by 2 experienced endoscopists using the Gastrointestinal Endoscopy Competency Assessment Tool for Pediatric Colonoscopy (GiECATKIDS). In addition, participants self-assessed their performance using the same instrument. Self-assessment accuracy between the externally assessed and self-assessed scores was evaluated using absolute difference scores, intraclass correlation coefficients, and Bland-Altman analyses. RESULTS: Forty-seven endoscopists participated (21 novices, 16 intermediates, and 10 experienced). Overall, there was moderate agreement of externally assessed and self-assessed GiECATKIDS total scores with an intraclass correlation coefficient of 0.72 (95% confidence interval, 0.55-0.83). The absolute difference scores among the 3 groups were significantly different (P = 0.005), with experienced endoscopists demonstrating a more accurate self-assessment compared to novices (P = 0.003). Bland-Altman plots revealed that novice endoscopists' self-assessed scores tended to be higher than their externally assessed scores, indicating they overestimated their performance. CONCLUSIONS: We found that endoscopic experience was positively associated with self-assessment accuracy among pediatric endoscopists. Novices were inaccurate in assessing their endoscopic competence and were prone to overestimation of their performances. Our findings suggest novices may benefit from targeted interventions aimed at improving their insight and self-awareness.

Impact of experience on self-assessment accuracy of clinical colonoscopy competence.

BACKGROUND AND AIMS: Self-assessment is important for life-long learning and a recommended assessment method for endoscopy skills. Prior literature has not investigated self-assessment accuracy of colonoscopic competence in the clinical setting. This study aimed to determine the self-assessment accuracy of novice, intermediate, and experienced endoscopists. METHODS: Novice (performed <50 previous colonoscopies), intermediate (50-500), and experienced (>1000) endoscopists from 5 hospitals each performed a clinical colonoscopy. Video recordings of procedures were independently assessed by 2 blinded expert endoscopists by using the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT). Externally assessed and self-assessed GiECAT scores were defined as the mean of the 2 video-based ratings and as participants' own assigned ratings, respectively. Self-assessment accuracy between the externally assessed and self-assessed scores was evaluated by using absolute difference scores, intraclass correlation coefficients, and the Bland-Altman analysis. RESULTS: Twenty novice, 10 intermediate, and 10 experienced endoscopists participated. There was moderate agreement of externally assessed and self-assessed GiECAT scores, with an intraclass correlation coefficient of 0.65 (95% confidence interval, 0.44-0.80). The absolute difference scores among the 3 groups were significantly different (P = .002), with experienced endoscopists demonstrating a more accurate self-assessment ability compared with novices (P = .002). Bland-Altman plots suggest that novice and experienced endoscopists tend to overrate and underrate their clinical competence, respectively; no specific trends were associated with intermediates. CONCLUSION: Participants demonstrated moderate self-assessment accuracy of clinical competence. Endoscopist experience was positively associated with self-assessment accuracy; novices demonstrated lower self-assessment accuracy compared with experienced endoscopists. Moreover, novices tended to overestimate their performances. Novice endoscopists may benefit from targeted interventions to improve self-assessment accuracy.