Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses.

BACKGROUND: Systematic reviews are performed manually despite the exponential growth of scientific literature. OBJECTIVE: To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews. DESIGN: Diagnostic test accuracy study. SETTING: Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations. PARTICIPANTS: None. MEASUREMENTS: A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened. RESULTS: Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level. LIMITATIONS: Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time. CONCLUSION: The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level. PRIMARY FUNDING SOURCE: None.

Psychometric properties and domains covered by patient-reported outcome measures used in trials assessing interventions for chronic pain.

OBJECTIVES: To identify the patient-reported outcome measures (PROMs) used in clinical trials assessing interventions for chronic pain, describe their psychometric properties, and the clinical domains they cover. STUDY DESIGN AND SETTING: We identified phase 3 or 4 interventional trials: on adult participants (aged >18 years), registered in clinicaltrials.gov between January 1, 2021 and December 31, 2022, and which provided "chronic pain" as a keyword condition. We excluded diagnostic studies and phase 1 or 2 trials. In each trial, one reviewer extracted all outcomes registered and identified those captured using PROMs. For each PROM used in more than 1% of identified trials, two reviewers assessed whether it covered the six important clinical domains from the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT): pain, emotional functioning, physical functioning, participant ratings of global improvement and satisfaction with treatment, symptoms and adverse events, and participant disposition (eg, adherence to medication). Second, reviewers searched PubMed for both the initial publication and latest review reporting the psychometric properties of each PROM and assessed their content validity, structural validity, internal consistency, reliability, measurement error, hypotheses testing, criterion validity, and responsiveness using published criteria from the literature. RESULTS: In total, 596 trials assessing 4843 outcomes were included in the study (median sample size 60, interquartile range 40-100). Trials evaluated behavioral (22%), device-based (21%), and drug-based (10%) interventions. Of 495 unique PROMs, 55 were used in more than 1% trials (16 were generic pain measures; 8 were pain measures for specific diseases; and 30 were measures of other symptoms or consequences of pain). About 50% PROMs had more than 50% of psychometric properties rated as sufficient. Scales often focused on a single clinical domain. Only 25% trials measured at least three clinical domains from IMMPACT. CONCLUSION: Half of PROMs used in trials assessing interventions for chronic pain had sufficient psychometric properties for more than 50% of criteria assessed. Few PROMs assessed more than one important clinical domain. Only 25% of trials measured more than 3/6 clinical domains considered important by IMMPACT.

Methods used to develop the SPIRIT 2024 and CONSORT 2024 Statements.

OBJECTIVES: To describe, and explain the rationale for, the methods used and decisions made during development of the updated SPIRIT 2024 and CONSORT 2024 reporting guidelines. METHODS: We developed SPIRIT 2024 and CONSORT 2024 together to facilitate harmonization of the two guidelines, and incorporated content from key extensions. We conducted a scoping review of comments suggesting changes to SPIRIT 2013 and CONSORT 2010, and compiled a list of other possible revisions based on existing SPIRIT and CONSORT extensions, other reporting guidelines, and personal communications. From this, we generated a list of potential modifications or additions to SPIRIT and CONSORT, which we presented to stakeholders for feedback in an international online Delphi survey. The Delphi survey results were discussed at an online expert consensus meeting attended by 30 invited international participants. We then drafted the updated SPIRIT and CONSORT checklists and revised them based on further feedback from meeting attendees. RESULTS: We compiled 83 suggestions for revisions or additions to SPIRIT and/or CONSORT from the scoping review and 85 from other sources, from which we generated 33 potential changes to SPIRIT (n = 5) or CONSORT (n = 28). Of 463 participants invited to take part in the Delphi survey, 317 (68%) responded to Round 1, 303 (65%) to Round 2 and 290 (63%) to Round 3. Two additional potential checklist changes were added to the Delphi survey based on Round 1 comments. Overall, 14/35 (SPIRIT n = 0; CONSORT n = 14) proposed changes reached the predefined consensus threshold (≥80% agreement), and participants provided 3580 free-text comments. The consensus meeting participants agreed with implementing 11/14 of the proposed changes that reached consensus in the Delphi and supported implementing a further 4/21 changes (SPIRIT n = 2; CONSORT n = 2) that had not reached the Delphi threshold. They also recommended further changes to refine key concepts and for clarity. CONCLUSION: The forthcoming SPIRIT 2024 and CONSORT 2024 Statements will provide updated, harmonized guidance for reporting randomized controlled trial protocols and results, respectively. The simultaneous development of the SPIRIT and CONSORT checklists has been informed by current empirical evidence and extensive input from stakeholders. We hope that this report of the methods used will be helpful for developers of future reporting guidelines.

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses.

BACKGROUND: This study examined the synthesis methods used in meta-analyses pooling data from observational studies (OSs) and randomised controlled trials (RCTs) from various medical disciplines. METHODS: We searched Medline via PubMed to identify reports of systematic reviews of interventions, including and pooling data from RCTs and OSs published in 110 high-impact factor general and specialised journals between 2015 and 2019. Screening and data extraction were performed in duplicate. To describe the synthesis methods used in the meta-analyses, we considered the first meta-analysis presented in each article. RESULTS: Overall, 132 reports were identified with a median number of included studies of 14 [9-26]. The median number of OSs was 6.5 [3-12] and that of RCTs was 3 [1-6]. The effect estimates recorded from OSs (i.e., adjusted or unadjusted) were not specified in 82% (n = 108) of the meta-analyses. An inverse-variance common-effect model was used in 2% (n = 3) of the meta-analyses, a random-effects model was used in 55% (n = 73), and both models were used in 40% (n = 53). A Poisson regression model was used in 1 meta-analysis, and 2 meta-analyses did not report the model they used. The mean total weight of OSs in the studied meta-analyses was 57.3% (standard deviation, ± 30.3%). Only 44 (33%) meta-analyses reported results stratified by study design. Of them, the results between OSs and RCTs had a consistent direction of effect in 70% (n = 31). Study design was explored as a potential source of heterogeneity in 79% of the meta-analyses, and confounding factors were investigated in only 10% (n = 13). Publication bias was assessed in 70% (n = 92) of the meta-analyses. Tau-square was reported in 32 meta-analyses with a median of 0.07 [0-0.30]. CONCLUSION: The inclusion of OSs in a meta-analysis on interventions could provide useful information. However, considerations of several methodological and conceptual aspects of OSs, that are required to avoid misleading findings, were often absent or insufficiently reported in our sample.

Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials.

BACKGROUND: Preprints are increasingly used to disseminate research results, providing multiple sources of information for the same study. We assessed the consistency in effect estimates between preprint and subsequent journal article of COVID-19 randomized controlled trials. METHODS: The study utilized data from the COVID-NMA living systematic review of pharmacological treatments for COVID-19 (covid-nma.com) up to July 20, 2022. We identified randomized controlled trials (RCTs) evaluating pharmacological treatments vs. standard of care/placebo for patients with COVID-19 that were originally posted as preprints and subsequently published as journal articles. Trials that did not report the same analysis in both documents were excluded. Data were extracted independently by pairs of researchers with consensus to resolve disagreements. Effect estimates extracted from the first preprint were compared to effect estimates from the journal article. RESULTS: The search identified 135 RCTs originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs that did not meet the eligibility criteria, of which 13 RCTs reported an interim analysis in the preprint and a final analysis in the journal article. Overall, 109 preprint-article RCTs were included in the analysis. The median (interquartile range) delay between preprint and journal article was 121 (73-187) days, the median sample size was 150 (71-464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias was rated as 'some concern' for 80% of RCTs. We found that 81 preprint-article pairs of RCTs were consistent for all outcomes reported. There were nine RCTs with at least one outcome with a discrepancy in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect. Furthermore, six RCTs had at least one outcome missing in the journal article and 14 RCTs had at least one outcome added in the journal article compared to the preprint. There was a change in the direction of effect in one RCT. No changes in statistical significance or conclusions were found. CONCLUSIONS: Effect estimates were generally consistent between COVID-19 preprints and subsequent journal articles. The main results and interpretation did not change in any trial. Nevertheless, some outcomes were added and deleted in some journal articles.

Protocol for a living evidence synthesis on variants of concern and COVID-19 vaccine effectiveness.

BACKGROUND: It is evident that COVID-19 will remain a public health concern in the coming years, largely driven by variants of concern (VOC). It is critical to continuously monitor vaccine effectiveness as new variants emerge and new vaccines and/or boosters are developed. Systematic surveillance of the scientific evidence base is necessary to inform public health action and identify key uncertainties. Evidence syntheses may also be used to populate models to fill in research gaps and help to prepare for future public health crises. This protocol outlines the rationale and methods for a living evidence synthesis of the effectiveness of COVID-19 vaccines in reducing the morbidity and mortality associated with, and transmission of, VOC of SARS-CoV-2. METHODS: Living evidence syntheses of vaccine effectiveness will be carried out over one year for (1) a range of potential outcomes in the index individual associated with VOC (pathogenesis); and (2) transmission of VOC. The literature search will be conducted up to May 2023. Observational and database-linkage primary studies will be included, as well as RCTs. Information sources include electronic databases (MEDLINE; Embase; Cochrane, L*OVE; the CNKI and Wangfang platforms), pre-print servers (medRxiv, BiorXiv), and online repositories of grey literature. Title and abstract and full-text screening will be performed by two reviewers using a liberal accelerated method. Data extraction and risk of bias assessment will be completed by one reviewer with verification of the assessment by a second reviewer. Results from included studies will be pooled via random effects meta-analysis when appropriate, or otherwise summarized narratively. DISCUSSION: Evidence generated from our living evidence synthesis will be used to inform policy making, modelling, and prioritization of future research on the effectiveness of COVID-19 vaccines against VOC.

No evidence of important difference in summary treatment effects between COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study.

OBJECTIVES: Preprints became a major source of research communication during the COVID-19 pandemic. We aimed to evaluate whether summary treatment effect estimates differ between preprint and peer-reviewed journal trials. STUDY DESIGN AND SETTING: A meta-epidemiological study. Data were derived from the COVID-NMA living systematic review (covid-nma.com) up to July 20, 2022. We identified all meta-analyses evaluating pharmacological treatments vs. standard of care or placebo for patients with COVID-19 that included at least one preprint and one peer-reviewed journal article. Difference in effect estimates between preprint and peer-reviewed journal trials were estimated by the ratio of odds ratio (ROR); ROR <1 indicated larger effects in preprint trials. RESULTS: Thirty-seven meta-analyses including 114 trials (44 preprints and 70 peer-reviewed publications) were selected. The median number of randomized controlled trials (RCTs) per meta-analysis was 2 (interquartile range [IQR], 2-4; maximum, 11), median sample size of RCTs was 199 (IQR, 99-478). Overall, there was no statistically significant difference in summary effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71-1.09; I2 = 17.8%; τ2 = 0.06). CONCLUSION: We did not find an important difference between summary treatment effects of preprints and summary treatment effects of peer-reviewed publications. Systematic reviewers and guideline developers should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.

A structured approach to information retrieval improved identification of funding and researchers' conflicts of interest in trials included in Cochrane reviews.

OBJECTIVES: To compare the contemporary Cochrane review approach for retrieving information on trial funding and researchers' conflicts of interest with a structured approach for information retrieval. STUDY DESIGN AND SETTING: Methodological study of 100 Cochrane reviews from August to December 2020 and one randomly selected trial from each review. Reporting of trial funding and researchers' conflicts of interest in reviews was compared with information identified using a structured retrieval process, and time to retrieve information was noted. We also formulated a guide to systematic reviewers for efficient information retrieval. RESULTS: Sixty-eight of 100 Cochrane reviews reported trial funding and 24 reported trial researchers' conflicts of interest. A simple structured approach, searching only trial publications (including conflicts of interest disclosure forms), identified funding for 16 additional trials and conflicts of interest information for 39 additional trials. A comprehensive structured approach, searching multiple information sources, identified funding for two additional trials and conflicts of interest for 14 additional trials. The median time to retrieve information was 10 minutes per trial (interquartile range: 7-15) for the simple approach and 20 minutes (11-43) for the comprehensive approach. CONCLUSION: A structured information retrieval approach improves identification of funding and researchers' conflicts of interest in trials included in Cochrane reviews.

Interleukin-6 blocking agents for treating COVID-19: a living systematic review.

BACKGROUND: It has been reported that people with COVID-19 and pre-existing autoantibodies against type I interferons are likely to develop an inflammatory cytokine storm responsible for severe respiratory symptoms. Since interleukin 6 (IL-6) is one of the cytokines released during this inflammatory process, IL-6 blocking agents have been used for treating people with severe COVID-19. OBJECTIVES: To update the evidence on the effectiveness and safety of IL-6 blocking agents compared to standard care alone or to a placebo for people with COVID-19. SEARCH METHODS: We searched the World Health Organization (WHO) International Clinical Trials Registry Platform, the Living OVerview of Evidence (L·OVE) platform, and the Cochrane COVID-19 Study Register to identify studies on 7 June 2022. SELECTION CRITERIA: We included randomized controlled trials (RCTs) evaluating IL-6 blocking agents compared to standard care alone or to placebo for people with COVID-19, regardless of disease severity. DATA COLLECTION AND ANALYSIS: Pairs of researchers independently conducted study selection, extracted data and assessed risk of bias. We assessed the certainty of evidence using the GRADE approach for all critical and important outcomes. In this update we amended our protocol to update the methods used for grading evidence by establishing minimal important differences for the critical outcomes. MAIN RESULTS: This update includes 22 additional trials, for a total of 32 trials including 12,160 randomized participants all hospitalized for COVID-19 disease. We identified a further 17 registered RCTs evaluating IL-6 blocking agents without results available as of 7 June 2022.  The mean age range varied from 56 to 75 years; 66.2% (8051/12,160) of enrolled participants were men. One-third (11/32) of included trials were placebo-controlled. Twenty-two were published in peer-reviewed journals, three were reported as preprints, two trials had results posted only on registries, and results from five trials were retrieved from another meta-analysis. Eight were funded by pharmaceutical companies.  Twenty-six included studies were multicenter trials; four were multinational and 22 took place in single countries. Recruitment of participants occurred between February 2020 and June 2021, with a mean enrollment duration of 21 weeks (range 1 to 54 weeks). Nineteen trials (60%) had a follow-up of 60 days or more. Disease severity ranged from mild to critical disease. The proportion of participants who were intubated at study inclusion also varied from 5% to 95%. Only six trials reported vaccination status; there were no vaccinated participants included in these trials, and 17 trials were conducted before vaccination was rolled out. We assessed a total of six treatments, each compared to placebo or standard care. Twenty trials assessed tocilizumab, nine assessed sarilumab, and two assessed clazakizumab. Only one trial was included for each of the other IL-6 blocking agents (siltuximab, olokizumab, and levilimab). Two trials assessed more than one treatment. Efficacy and safety of tocilizumab and sarilumab compared to standard care or placebo for treating COVID-19 At day (D) 28, tocilizumab and sarilumab probably result in little or no increase in clinical improvement (tocilizumab: risk ratio (RR) 1.05, 95% confidence interval (CI) 1.00 to 1.11; 15 RCTs, 6116 participants; moderate-certainty evidence; sarilumab: RR 0.99, 95% CI 0.94 to 1.05; 7 RCTs, 2425 participants; moderate-certainty evidence). For clinical improvement at ≥ D60, the certainty of evidence is very low for both tocilizumab (RR 1.10, 95% CI 0.81 to 1.48; 1 RCT, 97 participants; very low-certainty evidence) and sarilumab (RR 1.22, 95% CI 0.91 to 1.63; 2 RCTs, 239 participants; very low-certainty evidence). The effect of tocilizumab on the proportion of participants with a WHO Clinical Progression Score (WHO-CPS) of level 7 or above remains uncertain at D28 (RR 0.90, 95% CI 0.72 to 1.12; 13 RCTs, 2117 participants; low-certainty evidence) and that for sarilumab very uncertain (RR 1.10, 95% CI 0.90 to 1.33; 5 RCTs, 886 participants; very low-certainty evidence). Tocilizumab reduces all cause-mortality at D28 compared to standard care/placebo (RR 0.88, 95% CI 0.81 to 0.94; 18 RCTs, 7428 participants; high-certainty evidence). The evidence about the effect of sarilumab on this outcome is very uncertain (RR 1.06, 95% CI 0.86 to 1.30; 9 RCTs, 3305 participants; very low-certainty evidence). The evidence is uncertain for all cause-mortality at ≥ D60 for tocilizumab (RR 0.91, 95% CI 0.80 to 1.04; 9 RCTs, 2775 participants; low-certainty evidence) and very uncertain for sarilumab (RR 0.95, 95% CI 0.84 to 1.07; 6 RCTs, 3379 participants; very low-certainty evidence). Tocilizumab probably results in little to no difference in the risk of adverse events (RR 1.03, 95% CI 0.95 to 1.12; 9 RCTs, 1811 participants; moderate-certainty evidence). The evidence about adverse events for sarilumab is uncertain (RR 1.12, 95% CI 0.97 to 1.28; 4 RCT, 860 participants; low-certainty evidence).  The evidence about serious adverse events is very uncertain for tocilizumab (RR 0.93, 95% CI 0.81 to 1.07; 16 RCTs; 2974 participants; very low-certainty evidence) and uncertain for sarilumab (RR 1.09, 95% CI 0.97 to 1.21; 6 RCTs; 2936 participants; low-certainty evidence). Efficacy and safety of clazakizumab, olokizumab, siltuximab and levilimab compared to standard care or placebo for treating COVID-19 The evidence about the effects of clazakizumab, olokizumab, siltuximab, and levilimab comes from only one or two studies for each blocking agent, and is uncertain or very uncertain. AUTHORS' CONCLUSIONS: In hospitalized people with COVID-19, results show a beneficial effect of tocilizumab on all-cause mortality in the short term and probably little or no difference in the risk of adverse events compared to standard care alone or placebo. Nevertheless, both tocilizumab and sarilumab probably result in little or no increase in clinical improvement at D28. Evidence for an effect of sarilumab and the other IL-6 blocking agents on critical outcomes is uncertain or very uncertain. Most of the trials included in our review were done before the waves of different variants of concern and before vaccination was rolled out on a large scale. An additional 17 RCTs of IL-6 blocking agents are currently registered with no results yet reported. The number of pending studies and the number of participants planned is low. Consequently, we will not publish further updates of this review.

Reminding Peer Reviewers of Reporting Guideline Items to Improve Completeness in Published Articles: Primary Results of 2 Randomized Trials.

Importance: Numerous studies have shown that adherence to reporting guidelines is suboptimal. Objective: To evaluate whether asking peer reviewers to check if specific reporting guideline items were adequately reported would improve adherence to reporting guidelines in published articles. Design, Setting, and Participants: Two parallel-group, superiority randomized trials were performed using manuscripts submitted to 7 biomedical journals (5 from the BMJ Publishing Group and 2 from the Public Library of Science) as the unit of randomization, with peer reviewers allocated to the intervention or control group. Interventions: The first trial (CONSORT-PR) focused on manuscripts that presented randomized clinical trial (RCT) results and reported following the Consolidated Standards of Reporting Trials (CONSORT) guideline, and the second trial (SPIRIT-PR) focused on manuscripts that presented RCT protocols and reported following the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guideline. The CONSORT-PR trial included manuscripts that described RCT primary results (submitted July 2019 to July 2021). The SPIRIT-PR trial included manuscripts that contained RCT protocols (submitted June 2020 to May 2021). Manuscripts in both trials were randomized (1:1) to the intervention or control group; the control group received usual journal practice. In the intervention group of both trials, peer reviewers received an email from the journal that asked them to check whether the 10 most important and poorly reported CONSORT (for CONSORT-PR) or SPIRIT (for SPIRIT-PR) items were adequately reported in the manuscript. Peer reviewers and authors were not informed of the purpose of the study, and outcome assessors were blinded. Main Outcomes and Measures: The difference in the mean proportion of adequately reported 10 CONSORT or SPIRIT items between the intervention and control groups in published articles. Results: In the CONSORT-PR trial, 510 manuscripts were randomized. Of those, 243 were published (122 in the intervention group and 121 in the control group). A mean proportion of 69.3% (95% CI, 66.0%-72.7%) of the 10 CONSORT items were adequately reported in the intervention group and 66.6% (95% CI, 62.5%-70.7%) in the control group (mean difference, 2.7%; 95% CI, -2.6% to 8.0%). In the SPIRIT-PR trial, of the 244 randomized manuscripts, 178 were published (90 in the intervention group and 88 in the control group). A mean proportion of 46.1% (95% CI, 41.8%-50.4%) of the 10 SPIRIT items were adequately reported in the intervention group and 45.6% (95% CI, 41.7% to 49.4%) in the control group (mean difference, 0.5%; 95% CI, -5.2% to 6.3%). Conclusions and Relevance: These 2 randomized trials found that it was not useful to implement the tested intervention to increase reporting completeness in published articles. Other interventions should be assessed and considered in the future. Trial Registration: ClinicalTrials.gov Identifiers: NCT05820971 (CONSORT-PR) and NCT05820984 (SPIRIT-PR).

Completeness of reporting in abstracts of randomized controlled trials assessing interventional radiology for liver disease.

PURPOSE: To assess the completeness of reporting in abstracts of published randomized controlled trials (RCTs) assessing interventional radiology (IR) for liver disease; to assess whether publication of the 2017 CONSORT update for nonpharmacological treatments (NPT) resulted in changes in abstract reporting; and to identify factors associated with better reporting. MATERIAL AND METHODS: MEDLINE and Embase were searched to identify RCTs of IR for liver disease (January 2015-September 2020). Two reviewers assessed the completeness of abstract reporting according to the CONSORT-NPT-2017-update. The primary outcome was the mean number of CONSORT items completely reported among 10 items reported in <50% of the abstracts published in 2015. A time series analysis assessed the evolution trend over time. A multivariate regression model was used to identify factors associated with better reporting. RESULTS: A total of 107 abstracts of RCTs published in 61 journals were included. Overall, 74% (45/61) of journals endorsed the main CONSORT guidelines, of which 60% (27/45) had a policy to implement them. The mean number of primary outcome items completely reported increased by 0.19 over the study period. The publication of the CONSORT-NPT update did not lead to an increase in the trend of items reported (increase of 0.04 items/month before vs. 0.02 after, P=0.41). Factors associated with more complete reporting were impact factor (OR=1.13; 95%CI:1.07-1.18) and endorsement of CONSORT with an implementation policy (OR=8.29; 95%CI:2.04-33.65). CONCLUSION: Completeness of reporting is incomplete in abstracts of trials of IR liver disease and did not improve after publication of the CONSORT-NPT-2017 update with abstract guidance.

Sharing of Individual Patient-Level Data by Trialists of Randomized Clinical Trials of Pharmacological Treatments for COVID-19.

This study describes access to individual patient-level data from randomized clinical trials during the COVID-19 pandemic to determine whether the intent to share what was reported in the registry, publication, or preprint was consistent with actual data access.

metaCOVID: A web-application for living meta-analyses of COVID-19 trials.

Outputs from living evidence syntheses projects have been used widely during the pandemic by guideline developers to form evidence-based recommendations. However, the needs of different stakeholders cannot be accommodated by solely providing pre-defined non amendable numerical summaries. Stakeholders also need to understand the data and perform their own exploratory analyses. This requires resources, time, statistical expertise, software knowledge as well as relevant clinical expertise to avoid spurious conclusions. To assist them, we created the metaCOVID application which, based on automation processes, facilitates the fast exploration of the data and the conduct of sub-analyses tailored to end-users needs. metaCOVID has been created in R and is freely available as an R-Shiny application. Based on the COVID-NMA platform (https://covid-nma.com/) the application conducts living meta-analyses of randomized controlled trials related to COVID-19 treatments and vaccines for several outcomes. Several options are available for subgroup and sensitivity analyses. The results are presented in downloadable forest plots. We illustrate metaCOVID through three examples involving well-known treatments and vaccines for COVID-19. The application is freely available from https://covid-nma.com/metacovid/.

Conceptualizing the reporting of living systematic reviews.

OBJECTIVES: As part of an effort to develop an extension of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 statement for living systematic reviews (LSRs), we discuss conceptual issues relevant to the reporting of LSRs and highlight a few challenges. METHODS: Discussion of conceptual issues based on a scoping review of the literature and discussions among authors. RESULTS: We first briefly describe aspects of the LSR production process relevant to reporting. The production cycles differ by whether the literature surveillance identifies new evidence and whether newly identified evidence is judged to be consequential. This impacts the timing, content, and format of LSR versions. Second, we discuss four types of information that are specific to the reporting of LSRs: justification for adopting the living mode, LSR specific methods, changes between LSR versions, and LSR updating status. We also discuss the challenge of conveying changes between versions to the reader. Third, we describe two commonly used reporting formats of LSRs: full and partial reports. Although partial reports are easier to produce and publish, they lead to the scattering of information across different versions. Full reports ensure the completeness of reporting. We discuss the implications for the extension of the PRISMA 2020 statement for LSRs. CONCLUSION: We argue that a dynamic publication platform would facilitate complete and timely reporting of LSRs.

Empirical evaluation of the methods used in systematic reviews including observational studies and randomized trials.

OBJECTIVES: To examine the methodological characteristics of systematic reviews and meta-analyses including observational studies (OSs) and randomized controlled trials (RCTs), in various medical disciplines. STUDY DESIGN AND SETTING: We searched Medline via PubMed to identify systematic reviews of interventions including RCTs and OSs published in 110 journals from 2015 to 2019. We extracted in duplicate general and methodological characteristics of the systematic review. RESULTS: We identified 402 systematic reviews. Only 39% (n = 160) of them reported the availability of a pre-established protocol. A rationale for including observational data in the systematic review was clearly reported in 25% (n = 102) of the systematic reviews. Thirty two percent (n = 130) of the reviews reported a search strategy intending to identify published and unpublished data for RCTs and OSs. The risk of bias of the individual studies was assessed in 89% (n = 359) of the systematic reviews. In 74% (n = 266) it was assessed for both RCTs and OSs; 180 (50%) used different tools. Information about confounding factors was reported in only 11% of systematic reviews and the type of effect estimates (crude or adjusted) used was specified in only 22% of the systematic reviews. Among the 385 systematic reviews that performed data synthesis, only 132 (33%) pooled OSs and RCTs in the same meta-analysis. CONCLUSION: Including OSs in systematic reviews of interventions could provide useful information but such an approach could also be misleading; thus, several methodological details are needed to ensure appropriate handling of OS and valid results. Our study revealed, although, that substantial methodological information is missing in reports published in high-impact factor general and specialty journals.

A scoping review identifies multiple comments suggesting modifications to SPIRIT 2013 and CONSORT 2010.

OBJECTIVES: To identify, summarize, and analyse comments on the core reporting guidelines for protocols of randomized trials (Standard Protocol Items: Recommendations for Interventional Trials [SPIRIT] 2013) and for completed trials (Consolidated Standards of Reporting Trials [CONSORT] 2010), with special emphasis on suggestions for guideline modifications. METHODS: We included documents written in English and published after 2010 that explicitly commented on SPIRIT 2013 or CONSORT 2010. We searched four bibliographic databases (Embase and MEDLINE to June 2022; Web of Science and Google Scholar to April 2022) and other sources (e.g., the EQUATOR Network website, the BMC Blog Network, and the BMJ rapid response section). Two authors independently assessed documents for eligibility and extracted data on basic characteristics and the wording of the main comments. We categorized comments as 'suggestion for modification to the wording of an existing guideline item,' 'suggestion for a new item,' or 'reflections on challenges or strengths.' We provided a summary and examples of the proposed suggestions and categorized comments into those that were directly linked to empirical investigations, were continuations of previous methodological discussions, or reflected new methodological developments. RESULTS: We assessed full text of 2,320 potentially eligible documents and included 93 documents with 114 comments. In total, 37 comments suggested modifications to existing guideline items. The participant flow section of CONSORT 2010 received the most comments (eight comments made different suggestions, e.g., one comment suggested to add numbers on nonrandomized screened participants). There were 46 comments suggesting new items. Multiple suggestions were related to trial interventions (eight comments made different suggestions, e.g., one comment suggested to add content on cointerventions), blinding (six comments suggested to add content on risk of unblinding), statistical methods (five comments made different suggestions, e.g., one comment suggested to add content on blinding of statisticians), and participant flow (seven comments made different suggestions, e.g., three comments suggested to add content on missing data). Half (53%) of the suggestions were directly linked to empirical investigations. Six (7%) suggestions were continuations of previous methodological discussions and five (6%) suggestions reflected new methodological developments related to conflicts of interest and funding, data sharing, and patient and public involvement. CONCLUSION: The issues raised provide context to authors, peer reviewers, editors, and readers of trials using SPIRIT 2013 and CONSORT 2010 and inform the planned updates of the core guidelines.

Using Risk of Bias 2 to assess results from randomised controlled trials: guidance from Cochrane.

A systematic review identifies, appraises and synthesises all the empirical evidence from studies that meet prespecified eligibility criteria to answer a specific research question. As part of the appraisal, researchers use explicit methods to assess risk of bias in the results' from included studies that contribute to the review's findings, to improve our confidence in the review's conclusions. Randomised controlled trials included in Cochrane Reviews have used a specific risk of bias tool to assess these included studies since 2008. In 2019, a new version of this tool, Risk of Bias 2 (RoB 2), was launched to improve its usability and to reflect current understanding of how the causes of bias can influence study results. Cochrane implemented RoB 2 in a phased approach, with users of the tool informing guidance development. This paper highlights learning for all systematic reviewers (Cochrane and non-Cochrane) from the phased implementation, highlighting differences between the original version of the tool and RoB 2, consideration of reporting systematic review protocols or full review reports that have used RoB 2, and some tips shared by authors during the pilot phase of the implementation.

Considering multiple outcomes with different weights informed the hierarchy of interventions in network meta-analysis.

OBJECTIVES: Ranking metrics in network meta-analysis (NMA) are computed separately for each outcome. Our aim is to 1) present graphical ways to group competing interventions considering multiple outcomes and 2) use conjoint analysis for placing weights on the various outcomes based on the stakeholders' preferences. STUDY DESIGN AND SETTING: We used multidimensional scaling (MDS) and hierarchical tree clustering to visualize the extent of similarity of interventions in terms of the relative effects they produce through a random effect NMA. We reanalyzed a published network of 212 psychosis trials taking three outcomes into account as follows: reduction in symptoms of schizophrenia, all-cause treatment discontinuation, and weight gain. RESULTS: Conjoint analysis provides a mathematical method to transform judgements into weights that can be subsequently used to visually represent interventions on a two-dimensional plane or through a dendrogram. These plots provide insightful information about the clustering of interventions. CONCLUSION: Grouping interventions can help decision makers not only to identify the optimal ones in terms of benefit-risk balance but also choose one from the best cluster based on other grounds, such as cost, implementation etc. Placing weights on outcomes allows considering patient profile or preferences.