Assessing the robustness of positive vascular surgery randomized controlled trials using their fragility index.

OBJECTIVE: The fragility index (FI) measures the robustness of statistically significant findings in randomized controlled trials (RCTs) by quantifying the minimum number of event conversions required to reverse a dichotomous outcome's statistical significance. In vascular surgery, many clinical guidelines and critical decision-making points are informed by a handful of key RCTs, especially regarding open surgical versus endovascular treatment. The objective of this study is to evaluate the FI of RCTs with statistically significant primary outcomes that compared open vs endovascular surgery in vascular surgery. METHODS: In this meta-epidemiological study and systematic review, MEDLINE, Embase, and CENTRAL were searched for RCTs comparing open versus endovascular treatments for abdominal aortic aneurysms, carotid artery stenosis, and peripheral arterial disease to December 2022. RCTs with statistically significant primary outcomes were included. Data screening and extraction were conducted in duplicate. The FI was calculated by adding an event to the group with the smaller number of events while subtracting a nonevent to the same group until Fisher's exact test produced a nonstatistically significant result. The primary outcome was the FI and proportion of outcomes where the loss to follow-up was greater than the FI. The secondary outcomes assessed the relationship of the FI to disease state, presence of commercial funding, and study design. RESULTS: Overall, 5133 articles were captured in the initial search with 21 RCTs reporting 23 different primary outcomes being included in the final analysis. The median FI (first quartile, third quartile) was 3 (3, 20) with 16 outcomes (70%) reporting a loss to follow-up greater than its FI. Mann-Whitney U test revealed that commercially funded RCTs and composite outcomes had greater FIs (median, 20.0 [5.5, 24.5] vs median, 3.0 [2.0, 5.5], P = .035; median, 21 [8, 38] vs median, 3.0 [2.0, 8.5], P = .01, respectively). The FI did not vary between disease states (P = .285) or between index and follow-up trials (P = .147). There were significant correlations between the FI and P values (Pearson r = 0.90; 95% confidence interval, 0.77-0.96), and the number of events (r = 0.82; 95% confidence interval, 0.48-0.97). CONCLUSIONS: A small number of event conversions (median, 3) are needed to alter the statistical significance of primary outcomes in vascular surgery RCTs evaluating open surgical and endovascular treatments. Most studies had loss to follow-up greater than its FI, which can call into question trial results, and commercially funded studies had a greater FI. The FI and these findings should be considered in future trial design in vascular surgery.

Completeness of reporting in systematic reviews and meta-analyses in vascular surgery.

OBJECTIVE: Systematic reviews (SRs) and meta-analyses are essential in informing clinical guidelines and decision-making. Complete reporting of SRs through compliance to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines promotes transparency, reproducibility, and consistency across the literature. The purpose of this meta-epidemiological study is to assess the completeness of reporting of SRs in the vascular surgery literature. METHODS: MEDLINE and Embase were used to search through four major vascular surgery journals and four high impact general medical journals for SRs published between 2018 and October 2022 evaluating clinical treatments for any pathology treated by a vascular surgeon. Data screening and extraction were conducted in duplicate. The reporting completeness of each included SR was measured with reference to the 27-item PRISMA checklist, and methodological quality was evaluated using the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR 2) tool. Parametric tests were used to evaluate for associations between PRISMA score and study funding, protocol registration, publication in a higher impact factor journal, and AMSTAR 2 score. The protocol is available online: 10.17605/OSF.IO/VBC5N. RESULTS: Of 1653 articles captured in the initial search, 162 SRs were included in the final analysis. All SRs had more than one incomplete PRISMA item. The mean PRISMA score was 21.2/27 (standard deviation: 2.9, 78.5% compliance), and the mean AMSTAR 2 score was 11.7/16 (standard deviation: 1.9, 73%). SRs that had a prospectively registered protocol had a higher PRISMA score (22.9 vs 20.6, P < .001) as did those that were published in higher impact factor journals (23.3 vs 21.0, P = .017). There was a large positive correlation between an SR's PRISMA and AMSTAR 2 scores (Pearson r = 0.655, 95% confidence interval: 0.55-0.74). There were no associations between the PRISMA score and publication year (P = .067) or funding status (P = .076). CONCLUSIONS: Overall, the reporting of SRs and meta-analyses in vascular surgery is less than ideal, with several key items being consistently under-reported. Prospective registration and methodological quality as measured by AMSTAR 2 scores are positively associated with improved reporting. Authors, reviewers, and journal editors should consider these findings moving forward to encourage completeness of SR reporting. Raising awareness surrounding the value of complete reporting of SRs can aid in enhancing the quality of evidence, and journals should consider these findings in methods used to promote SR reporting.