Statistical techniques to assess publication integrity in groups of randomized trials: a narrative review.

OBJECTIVES: To describe statistical tools available for assessing publication integrity of groups of randomized controlled trials (RCTs). STUDY DESIGN AND SETTING: Narrative review. RESULTS: Freely available statistical tools have been developed that compare the observed distributions of baseline variables with the expected distributions that would occur if successful randomization occurred. For continuous variables, the tools assess baseline means, baseline P values, and the occurrence of identical means and/or standard deviation. For categorical variables, they assess baseline P values, frequency counts for individual or all variables, numbers of trial participants randomized or withdrawing, and compare reported with independently calculated P values. The tools have been used to identify publication integrity concerns in RCTs from individual groups, and performed at an acceptable level in discriminating intentionally fabricated baseline summary data from genuine RCTs. The tools can be used when concerns have been raised about RCT(s) from an individual/group and when the whole body of their work is being examined, when conducting systematic reviews, and could be adapted to aid screening of RCTs at journal submission. CONCLUSION: Statistical tools are useful for the assessment of publication integrity of groups of RCTs.

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions.

INTRODUCTION: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare-related interventions. METHODS AND ANALYSIS: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: (1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, (2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, (3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in, (4) a consensus meeting to select checks to be included in a draft tool and to determine its format and (5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies and will help patients by protecting them from the influence of false data on their healthcare. ETHICS AND DISSEMINATION: The University of Manchester ethics decision tool was used, and this returned the result that ethical approval was not required for this project (30 September 2022), which incorporates secondary research and surveys of professionals about subjects relating to their expertise. Informed consent will be obtained from all survey participants. All results will be published as open-access articles. The final tool will be made freely available.

Responses to concerns raised about publications don't address the concerns raised.

Recently in the Journal, Amanda Williams described her experience of raising concerns about a group of trials with "untrustworthy data". We were inspired by the work of Williams and colleagues to examine these and other trials by the same research group. Similar to Williams, we found that the patterns of reported data differed from the patterns expected to arise from valid randomisation. We also identified a high proportion of reported baseline p-values for categorial variables that differed from independently calculated p-values. We reported these findings to the affected journals but none of the concerns were addressed and no action will be taken about the majority. Despite the large number of unresolved concerns about these trials, readers will be unaware of the issues, which seems entirely unsatisfactory.

Ten Years later: Assessments of the integrity of publications from one research group with multiple retractions.

When a research group has multiple retracted publications and/or research misconduct by a member is evident, there is a risk that its other publications are unreliable, so a comprehensive assessment of the group's publications is advisable. We analyzed the comprehensiveness of assessment of the integrity of 300 publications by a research group with numerous retractions and known research misconduct, for 292 of which we raised concerns to publishers and academic institutions between 3/2013 and 2/2020. By 4/2023, 91 (30%) publications had not been assessed by either publisher or academic institution. Publishers had assessed 185 (63%) publications. The 4 academic institutions had assessed 5/36 (14%), 56/216 (26%), 30/50 (60%) and 40/66 (61%) publications. Unprompted assessments, those undertaken without our notification of concerns, occurred for 24 (8%) publications, 3 (1%) by publishers and 21 (7%) by academic institutions. Among 32 journals with ≥2 affected publications, no unprompted assessments of the remaining publication(s) occurred after notification of concerns about the index publication(s). Publishers retracted 58/84 (69%) publications which institutions also assessed and decided needed no editorial action. These analyses demonstrate the failure of publishers and institutions to comprehensively and spontaneously determine the integrity of publications in a setting of known misconduct and multiple retractions.

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions.

Introduction: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INSPECT-SR project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare related interventions. Methods and analysis: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: 1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, 2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, 3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in 4) a consensus meeting to select checks to be included in a draft tool and to determine its format, 5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies, and will help patients by protecting them from the influence of false data on their healthcare.

Increased workload without clinical benefit: Results following implementation of the ACR-TIRADS system for thyroid nodules.

OBJECTIVE: The ACR-TIRADS system for stratifying thyroid nodule malignancy risk has been widely promoted and implemented. We audited its introduction at a large public hospital in Auckland, New Zealand. DESIGN: Audit of outcomes following thyroid nodule fine needle aspiration (FNA) before/after ACR-TIRADS. PATIENTS: Individuals undergoing thyroid FNA 2017-2019. MEASUREMENTS: From medical records, we obtained details from the pre-FNA ultrasound (nodule size, TIRADS points/levels, radiologist recommendation for FNA), Bethesda (B) cytology classification, histology and post-FNA follow-up. RESULTS: Four hundred and twenty-two individuals had 564 FNAs, 163 had surgery and 54 (13%) had cancer in the primary nodule. 37/54 (69%) cancers were papillary thyroid carcinoma (median size 25 mm, 87% ≥10 mm, 61% ≥20 mm). Following ACR-TIRADS introduction, FNA recommendations increased greater than twofold, FNAs performed by 71%-83%, and the monthly rate of FNAs and operations by 60% and 40%, respectively. However, the proportion of cancers/FNA remained similar (9.9% post-TIRADS vs. 8.7% pre-TIRADS). The proportions of FNA results remained stable for B2-B4 categories, but doubled (11% vs. 5%) for B5-B6: 15 FNAs were needed to identify an additional B5/B6 lesion. TIRADS-5 nodules had a higher proportion of B5/B6 (20%) and a lower proportion of B2 (30%) than TIRADS-3 (2%, 57%, respectively) and TIRADS-4 (9%, 56%) nodules. About 5 additional cancers/year were diagnosed, but they were more often small (49% vs. 8% <2 cm, 17% vs. 0% <1 cm). CONCLUSION: ACR-TIRADS introduction increased workload (FNAs and operations), without increasing the proportion of cancers/FNA. It led to a few more cancers being diagnosed, but many were small and of uncertain clinical significance.

Distributions of baseline categorical variables were different from the expected distributions in randomized trials with integrity concerns.

BACKGROUND AND OBJECTIVES: Comparing observed and expected distributions of baseline continuous variables in randomized controlled trials (RCTs) can be used to assess publication integrity. We explored whether baseline categorical variables could also be used. METHODS: The observed and expected (binomial) distribution of all baseline categorical variables were compared in four sets of RCTs: two controls, and two with publication integrity concerns. We also compared baseline calculated and reported P-values. RESULTS: The observed and expected distributions of baseline categorical variables were similar in the control datasets, both for frequency counts (and percentages) and for between-group differences in frequency counts. However, in both sets of RCTs with publication integrity concerns, about twice as many variables as expected had between-group differences in frequency counts of one or 2, and far fewer variables than expected had between-group differences of >4 (P < 0.001 for both datasets). Furthermore, about one in six reported P-values for baseline categorial variables differed by > 0.1 from the calculated P-value in trials with publication integrity concerns. CONCLUSION: Comparing the observed and expected distributions and reported and calculated P-values of baseline categorical variables may help in the assessment of publication integrity of a body of RCTs.

Is PHARMAC's decision-making fair, cost-effective and clinically effective? Observations from the real world.

Nil.

Procrastination and inconsistency: Expressions of concern for publications with compromised integrity.

Expressions of concern (EoC) can reduce the adverse effects of unreliable publications by alerting readers to concerns about publication integrity while assessment is undertaken. We investigated the use of EoC for 463 publications by two research groups for which we notified concerns about publication integrity to 142 journals and 44 publishers between March 2013 and February 2020. By December 2021, 95 papers had had an EoC, and 83 were retracted without an EoC. Median times from notification of concerns to EoC (10.4mo) or retraction without EoC (13.1mo) were similar. Among the 95 EoCs, 29 (30.5%) were followed by retraction after a median of 5.4mo, none was lifted, and 66 (69.5%) remained in place after a median of 18.1mo. Publishers with >10 notified publications issued EoCs for 0-81.8% of papers: for several publishers the proportions of notified papers for which EoCs were issued varied considerably between the 2 research groups. EoCs were issued for >30% of notified publications of randomized clinical trials and letters to the editor, and <20% of other types of research. These results demonstrate inconsistent application of EoCs between and within publishers, and prolonged times to issue and resolve EoCs.

Outpatient synacthen testing in a large metropolitan region: a clinical audit.

AIM: To audit short synacthen tests (SSTs) performed at a single laboratory within the greater Auckland area. METHODS: Two hundred and eighty-seven SSTs conducted in 286 individuals between September 2016 and September 2019 were assessed. Test requests were not triaged. We assessed source of referrals, indications for testing, adequacy of pre-test information and test outcomes. RESULTS: Seventy-one percent of referrals were for women. Fifty-six percent were from primary care and 18% from rheumatology. One-hundred and fifteen (40%) of those referred had been taking corticosteroids within the previous three months: this information was only provided in 49 referrals. In 32% of referrals, no serum cortisol measurement had been undertaken within the previous six months. In 20% of referrals, no indication was provided. Thirty-three (11%) SSTs were abnormal. Of these, 29 were in patients taking corticosteroids. No SSTs were abnormal among 64 patients with pre-test serum cortisol >300nmol/L or >400nmol/L according to the cortisol assay in use. CONCLUSIONS: Referrals for SSTs often lack important information, such as the indication for testing and recent corticosteroid exposure. Up to one quarter of SSTs could be avoided if a serum cortisol was routinely measured prior to referral. Adopting a structured referral form that mandates provision of important clinical and biochemical data might reduce unnecessary testing.

A randomized trial alerting authors, with or without coauthors or editors, that research they cited in systematic reviews and guidelines has been retracted.

Retracted clinical trials may be influential in citing systematic reviews and clinical guidelines. We assessed the influence of 27 retracted trials on systematic reviews and clinical guidelines (citing publications), then alerted authors to these retractions. Citing publications were randomized to up to three e-mails to contact author with/without up to two coauthors, with/without the editor. After one year we assessed corrective action. We included 88 citing publications; 51% (45/88) had findings likely to change if retracted trials were removed, 87% (39/45) likely substantially. 51% (44/86) of contacted citing publications replied. Including three authors rather than the contact author alone was more likely to elicit a reply (P = 0.03). Including the editor did not increase replies (P = 0.66). Whether findings were judged likely to change, and size of the likely change, had no effect on response rate or action taken. One year after e-mails were sent only nine publications had published notifications. E-Mail alerts to authors and editors are inadequate to correct the impact of retracted publications in citing systematic reviews and guidelines. Changes to bibliographic and referencing systems, and submission processes are needed. Citing publications with retracted citations should be marked until authors resolve concerns.

Bone Mineral Density and Bone Turnover 10 Years After a Single 5 mg Dose or Two 5-Yearly Lower Doses of Zoledronate in Osteopenic Older Women: An Open-Label Extension of a Randomized Controlled Trial.

Intravenous zoledronate reduces fracture risk (5 mg at 18-month intervals) and prevents bone loss (doses of 1 to 5 mg for 3 to >5 years), but the duration of action of a single 5 mg dose and the effects of lower doses beyond 5 years are unknown. We report the second open-label extension (years 5 to 10) of a 2-year randomized, multidose, placebo-controlled, double-blinded trial. A total of 116 older women who completed 5 years of participation either continued observation without further treatment (zoledronate 5 mg and placebo at baseline) or received repeat doses of 1 or 2.5 mg zoledronate (zoledronate 1 mg and zoledronate 2.5 mg at baseline, respectively). Outcomes were spine, hip, and total body bone mineral density (BMD) and serum markers of bone turnover. After a single 5 mg dose of zoledronate, mean BMD at the lumbar spine and total hip was maintained at or above baseline levels for 9 and 10 years, respectively. The mean level of the bone resorption marker ß-C-terminal telopeptide of type I collagen (ß-CTX) was at least 25% lower than that in the placebo group for 9 years. In women administered 5-yearly doses of 2.5 mg zoledronate, mean BMD at the total hip and lumbar spine was maintained at or above baseline levels for 9 and 10 years, respectively. Redosing with 1 or 2.5 mg zoledronate at 5 years reduced bone turnover markers for 3 to 4 years. BMD increased for 3 to 4 years after redosing with 1 mg zoledronate. In the group given 5-yearly 2.5 mg zoledronate, ß-CTX was at least 20% lower than that in the placebo group for 10 years. Both a single baseline 5 mg dose of zoledronate and 5-yearly doses of 1 and 2.5 mg zoledronate prevented bone loss at hip and spine for 8 to 10 years in older postmenopausal women. Clinical trials to evaluate the effects on fracture risk of these very infrequent and lower doses of zoledronate are justified. © 2021 American Society for Bone and Mineral Research (ASBMR).

Timeliness and content of retraction notices for publications by a single research group.

Publications of expressions of concern and retractions should be timely, accurate and comprehensive. We as sessed these characteristics for 292 publications by a research group about which we submitted concerns about publication integrity to 77 journals and 29 publishers between March 2013 and February 2020. By October 2020, 115 publications were corrected (3), had expressions of concern (18), or were retracted (94). The median (95% CI) time from submission of concerns to the first journal correction was 22.1 (18.2-26.9) months: this did not diminish by year of submission of concerns, varied between publishers, and was shorter for journals with higher impact factors. Eighty-four publications of original research were retracted. The median (range) proportion of concerns raised with the journal that were mentioned in the ensuing retraction notices was 9.5% (2-49). At least 75% of retraction notices included the suggested content for 7/9, 3/9 and 3/16 items in the Committee for Publication Ethics and Retraction Watch minimum and optimal recommended formats, respectively. Thus, assessment of concerns about publication integrity was delayed and incomplete. Adherence to recommended content of retraction notices was moderate at best. Strategies are needed to improve the efficiency, accuracy and transparency of processes for resolving concerns about publication integrity.

Citation of retracted publications: A challenging problem.

Scientific publications with compromised integrity should be retracted. Papers citing retracted publications might need correction if findings depend on the retracted publication. While many studies have reported on post-retraction citations, few have focused on citations made before the retraction. We investigated the citation profile for a research group with 113 published concerns regarding publication integrity (CRPI). We identified 376 of their source publications that were cited by 5577 articles, and whether the source publication had a published CRPI. Of 6926 references to a source publication in these citing articles, for 3925 (57%) the source article had a published CRPI, while for 3001 (43%) it did not. Of these 3925 references, 3688 were in citing articles published before the source article CRPI was published. 166 citing articles containing 198 references to source publications were published after the corresponding source article CRPI was published (range 1-5 such references/article; 19/166 (11%) articles had >1 reference). In summary, many articles cite retracted publications, with the majority of these references occurring before the retraction. However, very few publications assess the impact of the retracted citations, even though the findings of many might be altered, at least in part, by removal of the retracted citation.