Statistical techniques to assess publication integrity in groups of randomized trials: a narrative review.

OBJECTIVES: To describe statistical tools available for assessing publication integrity of groups of randomized controlled trials (RCTs). STUDY DESIGN AND SETTING: Narrative review. RESULTS: Freely available statistical tools have been developed that compare the observed distributions of baseline variables with the expected distributions that would occur if successful randomization occurred. For continuous variables, the tools assess baseline means, baseline P values, and the occurrence of identical means and/or standard deviation. For categorical variables, they assess baseline P values, frequency counts for individual or all variables, numbers of trial participants randomized or withdrawing, and compare reported with independently calculated P values. The tools have been used to identify publication integrity concerns in RCTs from individual groups, and performed at an acceptable level in discriminating intentionally fabricated baseline summary data from genuine RCTs. The tools can be used when concerns have been raised about RCT(s) from an individual/group and when the whole body of their work is being examined, when conducting systematic reviews, and could be adapted to aid screening of RCTs at journal submission. CONCLUSION: Statistical tools are useful for the assessment of publication integrity of groups of RCTs.

Ten Years later: Assessments of the integrity of publications from one research group with multiple retractions.

When a research group has multiple retracted publications and/or research misconduct by a member is evident, there is a risk that its other publications are unreliable, so a comprehensive assessment of the group's publications is advisable. We analyzed the comprehensiveness of assessment of the integrity of 300 publications by a research group with numerous retractions and known research misconduct, for 292 of which we raised concerns to publishers and academic institutions between 3/2013 and 2/2020. By 4/2023, 91 (30%) publications had not been assessed by either publisher or academic institution. Publishers had assessed 185 (63%) publications. The 4 academic institutions had assessed 5/36 (14%), 56/216 (26%), 30/50 (60%) and 40/66 (61%) publications. Unprompted assessments, those undertaken without our notification of concerns, occurred for 24 (8%) publications, 3 (1%) by publishers and 21 (7%) by academic institutions. Among 32 journals with ≥2 affected publications, no unprompted assessments of the remaining publication(s) occurred after notification of concerns about the index publication(s). Publishers retracted 58/84 (69%) publications which institutions also assessed and decided needed no editorial action. These analyses demonstrate the failure of publishers and institutions to comprehensively and spontaneously determine the integrity of publications in a setting of known misconduct and multiple retractions.

Responses to concerns raised about publications don't address the concerns raised.

Recently in the Journal, Amanda Williams described her experience of raising concerns about a group of trials with "untrustworthy data". We were inspired by the work of Williams and colleagues to examine these and other trials by the same research group. Similar to Williams, we found that the patterns of reported data differed from the patterns expected to arise from valid randomisation. We also identified a high proportion of reported baseline p-values for categorial variables that differed from independently calculated p-values. We reported these findings to the affected journals but none of the concerns were addressed and no action will be taken about the majority. Despite the large number of unresolved concerns about these trials, readers will be unaware of the issues, which seems entirely unsatisfactory.

Increased workload without clinical benefit: Results following implementation of the ACR-TIRADS system for thyroid nodules.

OBJECTIVE: The ACR-TIRADS system for stratifying thyroid nodule malignancy risk has been widely promoted and implemented. We audited its introduction at a large public hospital in Auckland, New Zealand. DESIGN: Audit of outcomes following thyroid nodule fine needle aspiration (FNA) before/after ACR-TIRADS. PATIENTS: Individuals undergoing thyroid FNA 2017-2019. MEASUREMENTS: From medical records, we obtained details from the pre-FNA ultrasound (nodule size, TIRADS points/levels, radiologist recommendation for FNA), Bethesda (B) cytology classification, histology and post-FNA follow-up. RESULTS: Four hundred and twenty-two individuals had 564 FNAs, 163 had surgery and 54 (13%) had cancer in the primary nodule. 37/54 (69%) cancers were papillary thyroid carcinoma (median size 25 mm, 87% ≥10 mm, 61% ≥20 mm). Following ACR-TIRADS introduction, FNA recommendations increased greater than twofold, FNAs performed by 71%-83%, and the monthly rate of FNAs and operations by 60% and 40%, respectively. However, the proportion of cancers/FNA remained similar (9.9% post-TIRADS vs. 8.7% pre-TIRADS). The proportions of FNA results remained stable for B2-B4 categories, but doubled (11% vs. 5%) for B5-B6: 15 FNAs were needed to identify an additional B5/B6 lesion. TIRADS-5 nodules had a higher proportion of B5/B6 (20%) and a lower proportion of B2 (30%) than TIRADS-3 (2%, 57%, respectively) and TIRADS-4 (9%, 56%) nodules. About 5 additional cancers/year were diagnosed, but they were more often small (49% vs. 8% <2 cm, 17% vs. 0% <1 cm). CONCLUSION: ACR-TIRADS introduction increased workload (FNAs and operations), without increasing the proportion of cancers/FNA. It led to a few more cancers being diagnosed, but many were small and of uncertain clinical significance.

Distributions of baseline categorical variables were different from the expected distributions in randomized trials with integrity concerns.

BACKGROUND AND OBJECTIVES: Comparing observed and expected distributions of baseline continuous variables in randomized controlled trials (RCTs) can be used to assess publication integrity. We explored whether baseline categorical variables could also be used. METHODS: The observed and expected (binomial) distribution of all baseline categorical variables were compared in four sets of RCTs: two controls, and two with publication integrity concerns. We also compared baseline calculated and reported P-values. RESULTS: The observed and expected distributions of baseline categorical variables were similar in the control datasets, both for frequency counts (and percentages) and for between-group differences in frequency counts. However, in both sets of RCTs with publication integrity concerns, about twice as many variables as expected had between-group differences in frequency counts of one or 2, and far fewer variables than expected had between-group differences of >4 (P < 0.001 for both datasets). Furthermore, about one in six reported P-values for baseline categorial variables differed by > 0.1 from the calculated P-value in trials with publication integrity concerns. CONCLUSION: Comparing the observed and expected distributions and reported and calculated P-values of baseline categorical variables may help in the assessment of publication integrity of a body of RCTs.