Evaluating a shrinkage estimator for the treatment effect in clinical trials.

The main objective of most clinical trials is to estimate the effect of some treatment compared to a control condition. We define the signal-to-noise ratio (SNR) as the ratio of the true treatment effect to the SE of its estimate. In a previous publication in this journal, we estimated the distribution of the SNR among the clinical trials in the Cochrane Database of Systematic Reviews (CDSR). We found that the SNR is often low, which implies that the power against the true effect is also low in many trials. Here we use the fact that the CDSR is a collection of meta-analyses to quantitatively assess the consequences. Among trials that have reached statistical significance we find considerable overoptimism of the usual unbiased estimator and under-coverage of the associated confidence interval. Previously, we have proposed a novel shrinkage estimator to address this "winner's curse." We compare the performance of our shrinkage estimator to the usual unbiased estimator in terms of the root mean squared error, the coverage and the bias of the magnitude. We find superior performance of the shrinkage estimator both conditionally and unconditionally on statistical significance.

Standardization and other approaches to meta-analyze differences in means.

Meta-analysts often use standardized mean differences (SMD) to combine mean effects from studies in which the dependent variable has been measured with different instruments or scales. In this tutorial we show how the SMD is properly calculated as the difference in means divided by a between-subject reference-group, control-group, or pooled pre-intervention SD, usually free of measurement error. When combining mean effects from controlled trials and crossovers, most meta-analysts have divided by either the pooled SD of change scores, the pooled SD of post-intervention scores, or the pooled SD of pre- and post-intervention scores, resulting in SMDs that are biased and difficult to interpret. The frequent use of such inappropriate standardizing SDs by meta-analysts in three medical journals we surveyed is due to misleading advice in peer-reviewed publications and meta-analysis packages. Even with an appropriate standardizing SD, meta-analysis of SMDs increases heterogeneity artifactually via differences in the standardizing SD between settings. Furthermore, the usual magnitude thresholds for standardized mean effects are not thresholds for clinically important differences. We therefore explain how to use other approaches to combining mean effects of disparate measures: log transformation of factor effects (response ratios) and of percent effects converted to factors; rescaling of psychometrics to percent of maximum range; and rescaling with minimum clinically important differences. In the absence of clinically important differences, we explain how standardization after meta-analysis with appropriately transformed or rescaled pre-intervention SDs can be used to assess magnitudes of a meta-analyzed mean effect in different settings.

Calcium supplementation for the prevention of pre-eclampsia: Challenging the evidence from meta-analyses.

OBJECTIVE: To investigate the validity of the conclusion from Cochrane reviews and meta-analyses that treatment with calcium supplementation during pregnancy reduces the risk for pre-eclampsia by 55%, which has been influential in international guidelines and future research. DESIGN: Sensitivity analysis of data from Cochrane reviews of trials evaluating high-dose calcium supplementation (of at least 1 g/day) for reduction of pre-eclampsia risk. SETTING: Systematic review and meta-analysis. POPULATION: The Cochrane reviews and meta-analyses included 13 trials enrolling a total of 15 730 women. Random-effects meta-analysis of these studies resulted in a mean risk ratio (RR, calcium/placebo) of 0.45 (95% confidence interval [CI] 0.31-0.65; p < 0.0001). METHODS: We carried out a sensitivity analysis of evidence from the relevant Cochrane review, to examine the impact of study size. MAIN OUTCOME MEASURES: pre-eclampsia. RESULTS: In the three largest studies, accounting for 13 815 (88%) of total recruitment, mean RR was 0.92 (95% CI 0.80-1.06) and there was no evidence of heterogeneity between studies (I2 = 0). With inclusion of the smaller studies, mean RR decreased to 0.45 and I2 increased to 70%. CONCLUSIONS: In assessment of the effect of calcium supplementation on pre-eclampsia risk, the naive focus on the mean of the random-effects meta-analysis in the presence of substantial heterogeneity is highly misleading.

Reliability of the risk of bias assessment in randomized controlled trials for nursing: A cross-sectional study.

AIM: To evaluate the percentage and reasons for disagreements in the risk of bias (RoB) assessments for randomized controlled trials (RCTs) included in more than one Cochrane review in the field of nursing. BACKGROUND: Disagreement in RoB assessments reduces the credibility of the evidence summarized by systematic reviews (SRs). There is no study that evaluates the reliability of RoB assessments in nursing studies. DESIGN: Secondary data analysis based on research reports. METHODS: RCTs included in more than one review in the nursing have been included. The disagreement of the assessment was analysed, and the possible reasons for disagreements were investigated. RESULTS: Twenty-three RCTs were included in more than one review. The agreement of assessment ranged from 36.84% for "selective reporting" to 91.30% for "random sequence generation". "Allocation concealment" showed the optimal agreement (84.21%). The items "blinding of participants and personnel", "blinding of outcome assessment" and "incomplete outcome data" showed poor agreement, with 50.00%, 58.82% and 66.67%, respectively. Most disagreements came from extracting incomplete or different RCTs' information. CONCLUSIONS: The level of agreement of the assessment between reviews has varied greatly in the field of nursing. More complete and accurate information of RCTs needs to be collected when conducting a SR.

Cochrane systematic reviews in orthodontics: trends across updates.

BACKGROUND: Systematic reviews (SR) are regularly updated to reflect new evidence. However, updates are time-consuming and costly, and therefore should ideally be informed by new high-quality research. The purpose of this study is to assess trends in the quantity, quality, and recency of evidence intervening updates of orthodontic SR. METHODS: SR relevant to orthodontics with at least two versions were identified from the Cochrane Database. The number, risk of bias, and year of publication of included trials were recorded for each update. Multivariate regression was conducted to assess factors affecting the risk of bias in trials, and the proportions within SR. RESULTS: Forty-five SR inclusive of updates were included. The median number of trials was three per review and this increased across subsequent versions. Seven reviews (15.6%) included no evidence, and 40.74% of updates included no new evidence. Most of the primary research was considered high risk of bias (57.3%), although this was reduced marginally across updates. The proportion of studies considered low risk did not improve significantly between updates. There was no impact of publication year of clinical trials on the risk of bias (P = 0.349). However, average age of trials included in a systematic review significantly affected the proportion of low risk-of-bias studies (P = 0.039). CONCLUSIONS: SR are frequently updated without including new evidence. New evidence that is included is commonly deemed to be at high risk of bias. Targeted strategies to improve the efficient use of resources and improve research quality should be considered.

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses.

BACKGROUND: Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses. METHODS: In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro-Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile-quantile (Q-Q) plot of study-specific standardized residuals for visually assessing between-study normality. RESULTS: Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q-Q plots were fair or moderate. CONCLUSIONS: The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.

Prioritising Cochrane reviews to be updated with health equity focus.

BACKGROUND: The prioritisation of updating published systematic reviews of interventions is vital to prevent research waste and ensure relevance to stakeholders. The consideration of health equity in reviews is also important to ensure interventions will not exacerbate the existing inequities of the disadvantaged if universally implemented. This study aimed to pilot a priority setting exercise based on systematic reviews of interventions published in the Cochrane Library, to identify and prioritise reviews to be updated with a focus on health equity. METHODS: We conducted a priority setting exercise with a group of 13 international stakeholders. We identified Cochrane reviews of interventions that showed a reduction in mortality, had at least one Summary of Findings table and that focused on one of 42 conditions with a high global burden of disease from the 2019 WHO Global Burden of Disease report. This included 21 conditions used as indicators of success of the United Nations Universal Health Coverage in attaining the Sustainable Development Goals. Stakeholders prioritised reviews that were relevant to disadvantaged populations, or to characteristics of potential disadvantage within the general population. RESULTS: After searching for Cochrane reviews of interventions within 42 conditions, we identified 359 reviews that assessed mortality and included at least one Summary of Findings table. These pertained to 29 of the 42 conditions; 13 priority conditions had no reviews with the outcome mortality. Reducing the list to only reviews showing a clinically important reduction in mortality left 33 reviews. Stakeholders ranked these reviews in order of priority to be updated with a focus on health equity. CONCLUSIONS: This project developed and implemented a methodology to set priorities for updating systematic reviews spanning multiple health topics with a health equity focus. It prioritised reviews that reduce overall mortality, are relevant to disadvantaged populations, and focus on conditions with a high global burden of disease. This approach to the prioritisation of systematic reviews of interventions that reduce mortality provides a template that can be extended to reducing morbidity, and the combination of mortality and morbidity as represented in Disability-Adjusted Life Years and Quality-Adjusted Life Years.

Health Chatbots in Africa: Scoping Review.

BACKGROUND: This scoping review explores and summarizes the existing literature on the use of chatbots to support and promote health in Africa. OBJECTIVE: The primary aim was to learn where, and under what circumstances, chatbots have been used effectively for health in Africa; how chatbots have been developed to the best effect; and how they have been evaluated by looking at literature published between 2017 and 2022. A secondary aim was to identify potential lessons and best practices for others chatbots. The review also aimed to highlight directions for future research on the use of chatbots for health in Africa. METHODS: Using the 2005 Arksey and O'Malley framework, we used a Boolean search to broadly search literature published between January 2017 and July 2022. Literature between June 2021 and July 2022 was identified using Google Scholar, EBSCO information services-which includes the African HealthLine, PubMed, MEDLINE, PsycInfo, Cochrane, Embase, Scopus, and Web of Science databases-and other internet sources (including gray literature). The inclusion criteria were literature about health chatbots in Africa published in journals, conference papers, opinion, or white papers. RESULTS: In all, 212 records were screened, and 12 articles met the inclusion criteria. Results were analyzed according to the themes they covered. The themes identified included the purpose of the chatbot as either providing an educational or information-sharing service or providing a counselling service. Accessibility as a result of either technical restrictions or language restrictions was also noted. Other themes that were identified included the need for the consideration of trust, privacy and ethics, and evaluation. CONCLUSIONS: The findings demonstrate that current data are insufficient to show whether chatbots are effectively supporting health in the region. However, the review does reveal insights into popular chatbots and the need to make them accessible through language considerations, platform choice, and user trust, as well as the importance of robust evaluation frameworks to assess their impact. The review also provides recommendations on the direction of future research.

Repositioning for pressure injury prevention in adults: a commentary on a Cochrane review.

Repositioning is regarded as an important intervention to prevent the development of pressure injuries in patients who are immobile. However, there is uncertainty as to the optimal regimen in terms of frequency and method of repositioning. This commentary summarises and critically appraises a Cochrane systematic review that assessed the clinical and costeffectiveness of different repositioning regimens on the prevention of pressure injuries in adults in any setting.

Effectiveness of home-based end-of-life care.

Commentary on: Shepperd S, Gonçalves-Bradley DC, Straus SE, Wee B. Hospital at home: home-based end-of-life care. Cochrane Database of Systematic Reviews. 2021;Issue 3. 10.1002/14651858.CD009231.pub3 When a person has been diagnosed with a terminal illness and has less than 6 months to live, and curative treatments are no longer effective, end-of-life care or hospice care may be initiated. Studies suggest that approximately 7 million people a year are given this type of care, which strives to reduce distress and improve quality of life for patients and their families by providing comprehensive physical, psychosocial, and spiritual support. Surveys show that most individuals would prefer to receive this care at home when given the option. However, there are still some uncertainties regarding the effects of end-of-life care at home on a range of important patient outcomes. As a result, a Cochrane review was conducted/updated to study the effects of receiving end-of-life care at home, looking at these outcomes. The aim of this commentary is to critically appraise this Cochrane review and expand upon the findings of the review in context to practice.

Advancing pressure ulcer prevention: evaluating the impact of patient and lay carer education.

Decubitus ulcers, also known as bed ulcers or pressure ulcers, result from prolonged pressure on the skin, with contributing factors such as shear forces, friction and excessive moisture. Pressure ulcers have significant physical, social and psychological consequences for patients and impose a substantial financial burden on healthcare providers. Patient and caregiver education has been suggested as a potential approach for preventing pressure ulcers. In order to investigate the potential preventive impact, O'Connor et al (2021) conducted a Cochrane systematic review. Their study aimed to assess the effectiveness of educational interventions for patients and caregivers in preventing pressure ulcers. This commentary aims to critically appraise the methods used within the Cochrane systematic review by O'Connor et al (2022) and expand on the findings in the context of clinical practice.

Interventions for Basal Cell Carcinoma: Summary of Main Comparisons From a Cochrane Review and a Practical Interpretation of Findings. / Intervenciones para el carcinoma basocelular cutáneo (revisión Cochrane): Resumen de las principales comparaciones e interpretación práctica de los resultados.

Numerous surgical and nonsurgical modalities are available to treat basal cell carcinoma (BCC), but their true effectiveness and safety is unknown. This article summarizes the evidence presented in a recent Cochrane review and aims to facilitate the interpretation of the review's findings for the Spanish and Latin American scientific communities. Much of the evidence the reviewers found came from single studies, preventing meta-analysis. Conventional surgical excision continues to be the most effective treatment for low-risk BCC. Most studies had small sample sizes, and some had problems with blinding, limitations which will have affected the assessment of subjective outcomes, such as pain and cosmetic results. The authors identified a lack of standardization in relation to recurrences and cosmetic outcomes that threatens not only the internal validity of the studies but also their external validity and reproducibility.

Effectiveness of letters to patients with or without Cochrane blogshots on 10-year cardiovascular risk change among women in menopausal transition: 6-month three-arm randomized controlled trial.

BACKGROUND: Health information and patient education on lifestyle changes may have a positive effect on the prevention of many chronic conditions, especially cardiovascular diseases (CVDs). We performed a parallel, three-arm randomized controlled trial (RCT) of 6-month educational intervention in a form of letters containing a reminder of the participant's CVD risk with or without Cochrane blogshots to reduce CVD risk among women aged 45-65 with one or more known CVD risk factors. METHODS: The control group received a letter about their CVD risk at the beginning of the trial. The intervention groups received the initial letter about their CVD risk and remainder letters about their CVD risk every 2 months, with or without Cochrane blogshots: (1) effect of calcium in the prevention of high blood pressure, (2) effect of reducing saturated fat acids in eating habits, and (3) effects of green and black tea in CVD prevention. The primary outcome was CVD risk reduction calculated as the difference between the baseline and 6-month score for a 10-year risk of fatal CVD according to the ACC/AHA guidelines. RESULTS: After both interventions, CVD risk reduction was significantly higher compared to the control group (P < 0.001, Kruskal-Wallis H test). The number of participants who decreased their CV risk was 29% (20/70) in the control group, 69% (48/70) in the group receiving the reminder letters, and 70% (49/70) in the group receiving the reminder letters and blogshots. The number needed to treat to achieve risk reduction was 2.41 (95% CI = 1.77 to 3.78) for letters with a CVD risk reminder and 2.50 (1.81 to 4.03) for letters with a reminder and a blogshot. The group receiving reminder letters with Cochrane blogshots had a significant change in the category of CVD risk, mainly from high to moderate and from moderate to low CVD risk category. CONCLUSIONS: A simple and inexpensive intervention method in a form of letters reminding women about their CVD risk with or without providing additional health information in the form of Cochrane blogshots about interventions for important CVD risk factors may be effective in CVD management and could be considered by primary care providers. TRIAL REGISTRATION: ClinicalTrials.gov, NCT04601558. Retrospectively registered on October 19, 2020.

Empirical Comparisons of 12 Meta-analysis Methods for Synthesizing Proportions of Binary Outcomes.

BACKGROUND: Meta-analysis is increasingly used to synthesize proportions (e.g., disease prevalence). It can be implemented with widely used two-step methods or one-step methods, such as generalized linear mixed models (GLMMs). Existing simulation studies have shown that GLMMs outperform the two-step methods in some settings. It is, however, unclear whether these simulation settings are common in the real world. We aim to compare the real-world performance of various meta-analysis methods for synthesizing proportions. METHODS: We extracted datasets of proportions from the Cochrane Library and applied 12 two-step and one-step methods to each dataset. We used Spearman's ρ and the Bland-Altman plot to assess their results' correlation and agreement. The GLMM with the logit link was chosen as the reference method. We calculated the absolute difference and fold change (ratio of estimates) of the overall proportion estimates produced by each method vs. the reference method. RESULTS: We obtained a total of 43,644 datasets. The various methods generally had high correlations (ρ > 0.9) and agreements. GLMMs had computational issues more frequently than the two-step methods. However, the two-step methods generally produced large absolute differences from the GLMM with the logit link for small total sample sizes (< 50) and crude event rates within 10-20% and 90-95%, and large fold changes for small total event counts (< 10) and low crude event rates (< 20%). CONCLUSIONS: Although different methods produced similar overall proportion estimates in most datasets, one-step methods should be considered in the presence of small total event counts or sample sizes and very low or high event rates.

How large should the next study be? Predictive power and sample size requirements for replication studies.

We use information derived from over 40K trials in the Cochrane Collaboration database of systematic reviews (CDSR) to compute the replication probability, or predictive power of an experiment given its observed (two-sided) P$$ P $$ -value. We find that an exact replication of a marginally significant result with P=.05$$ P=.05 $$ has less than 30% chance of again reaching significance. Moreover, the replication of a result with P=.005$$ P=.005 $$ still has only 50% chance of significance. We also compute the probability that the direction (sign) of the estimated effect is correct, which is closely related to the type S error of Gelman and Tuerlinckx. We find that if an estimated effect has P=.05$$ P=.05 $$ , there is a 93% probability that its sign is correct. If P=.005$$ P=.005 $$ , then that probability is 99%. Finally, we compute the required sample size for a replication study to achieve some specified power conditional on the p$$ p $$ -value of the original study. We find that the replication of a result with P=.05$$ P=.05 $$ requires a sample size more than 16 times larger than the original study to achieve 80% power, while P=.005$$ P=.005 $$ requires at least 3.5 times larger sample size. These findings confirm that failure to replicate the statistical significance of a trial does not necessarily indicate that the original result was a fluke.

Intervention effect estimates in randomised controlled trials conducted in primary care versus secondary or tertiary care settings: a meta-epidemiological study.

BACKGROUND: Many clinical practice guidelines are based on randomised controlled trials conducted in secondary or tertiary care setting and general practitioners frequently question their relevance for primary care patients. Our aim was to compare the intervention effect estimates between primary care setting randomised controlled trials (PC-RCTs) and secondary or tertiary care setting randomised controlled trials (ST-RCTs). METHODS: Meta-epidemiological study of meta-analyses (MAs) of a binary outcome including at least one PC-RCT and one ST-RCT. PC-RCTs were defined as trials recruiting patients in general practices, primary care practices, family practices, community centers or community pharmacies. ST-RCTs were defined as trials recruiting in hospitals, including hospitalized patients, hospital outpatients and patients from emergency departments. For each MA, we estimated a ratio of odds ratio (ROR) by using random-effects meta-regression, with an ROR less than 1 indicating lower estimates of the intervention effect in PC-RCTs than ST-RCTs. Finally, we estimated a combined ROR across MAs by using a random-effects meta-analysis. We performed subgroup analyses considering the type of outcomes (objective vs subjective), type of experimental intervention (pharmacological vs non-pharmacological), and control group (active vs inactive) as well as analyses adjusted on items of the Cochrane Risk of Bias tool. RESULTS: Among 1765 screened reviews, 76 MAs with 230 PC-RCTs and 384 ST-RCTs were selected. The main medical fields were pneumology (13.2%) and psychiatry or addictology (38.2%). Intervention effect estimates did not significantly differ between PC-RCTs and ST-RCTs (ROR = 0.97, 95% confidence interval 0.88 to 1.08), with moderate heterogeneity across MAs (I2 = 45%). Subgroup and adjusted analyses led to consistent results. CONCLUSION: We did not observe any significant difference in intervention effect estimates between PC-RCTs and ST-RCTs. Nevertheless, most of the medical fields in this meta-epidemiological study were not representative of the pathologies encountered in primary care. Further studies with pathologies more frequently encountered in primary care are needed.

An investigation of the constancy of effect in Cochrane systematic reviews in context with the assumptions for noninferiority trials.

When designing a noninferiority (NI) study one of the most important steps is to set the noninferiority (NI) limit. The NI limit is an acceptable loss of efficacy for a new investigative treatment compared to an active control treatment - often standard care. The limit should be a value so small that the loss efficacy is clinically zero. An approach to the setting of a noninferiority limit such that an effect over placebo can be shown through an indirect comparison to placebo-controlled trials where the active control treatment was compared to placebo. In this context, the setting of the NI limit depends on three assumptions: assay sensitivity, bias minimisation, and the constancy assumption. The last assumption of constancy assumes the effect of the active control over placebo is constant. This paper aims to assess the constancy assumption in placebo-controlled trials. METHODS: 236 Cochrane reviews of placebo-controlled trials published in 2015-2016 were collected and used to assess the relation between the placebo, active treatment, and the standardised treatment different (SMD) with the time (year of publication). RESULTS: The analysis showed that both the size of the study and the treatment effect were associated with year of publication. The three main variables that affect the estimate of any future trial are the estimate from the meta-analysis of previous trials prior to the trial, the year difference in the meta-analysis, and the year of the trial conduction. The regression analysis showed that an increase of one unit in the point estimate of the historical meta-analysis would lead to an increase in the predicted estimate of future trial on the SMD scale by 0.88. This result suggests the final trial results are 12% smaller than that from the meta-analysis of trials until that point. CONCLUSION: The result of this study indicates that assuming constancy of the treatment difference between the active control and placebo can be questioned. It is therefore important to consider the effect of time in estimating the treatment response if indirect comparisons are being used as the basis of a NI limit.

Conclusiveness, linguistic characteristics and readability of Cochrane plain language summaries of intervention reviews: a cross-sectional study.

BACKGROUND: One of the most important formats to disseminate the evidence in health to different populations are Cochrane Plain Language Summaries (PLSs). PLSs should be written in a simplified language, easily understandable and providing clear message for the consumer. The aim of this study was to examine the extent to which PLSs are customized for lay persons, specifically by providing conclusive, comprehensible, and readable messages. METHODS: The study analyzed Cochrane PLSs of interventional studies (N = 4360) in the English language published from 1995 to 2019. We categorized the conclusiveness into one of the following categories: "positive", "positive inconclusive", "no evidence", "no opinion", "negative", "negative inconclusive", "unclear", "equal", "equal inconclusive". Language characteristics were analyzed using Linguistic Inquiry and Word Count (LIWC) software. The level of readability was measured by SMOG (Simple Measure of Gobbledygook) index, indicating the number of years of education required to read the text. For each PLS, we also collected the following data: Cochrane Review Network, year of publication and number of authors. RESULTS: Most of the PLSs (80%) did not have a conclusive message. In 53% PLSs there was no concluding opinion about the studied intervention or the conclusion was unclear. The most frequent conclusiveness category was "no opinion" (30%), and its frequency increased over time. The conclusiveness categories were similarly dispersed across Cochrane Networks. PLSs were written in an objective style, with high levels of analytical tone and clout above neutral, but a lower relation to authenticity and tone. The median number of years of non-specific education needed to read the PLSs was 14.9 (IQR 13.8 to 16.1), indicating that the person needs almost 15 years of general education to read the content with ease. CONCLUSION: Most of the Cochrane PLSs provided no concluding opinion or unclear conclusion regarding the effects of analyzed intervention. Analysis of readability indicated that they may be difficult to read for the lay population without medical education. Our results indicate that PLSs may not be so plain, and that the writing of Cochrane PLSs requires more effort. Tools used in this study could improve PLSs and make them better suited for lay audiences.

Restrictions and their reporting in systematic reviews of effectiveness: an observational study.

BACKGROUND: Restrictions in systematic reviews (SRs) can lead to bias and may affect conclusions. Therefore, it is important to report whether and which restrictions were used. This study aims to examine the use of restrictions regarding language, publication period, and study type, as well as the transparency of reporting in SRs of effectiveness. METHODS: A retrospective observational study was conducted with a random sample of 535 SRs of effectiveness indexed in PubMed between 2000 and 2019. The use of restrictions and their reporting were analysed using descriptive statistics. RESULTS: Of the total 535 SRs included, four out of every ten (41.3%) lacked information on at least one of the three restrictions considered (language, publication period, or study type). Overall, 14.6% of SRs did not provide information on restrictions regarding publication period, 19.1% regarding study type, and 18.3% regarding language. Of all included SRs, language was restricted in 46.4%, and in more than half of the SRs with restricted language (130/248), it was unclear whether the restriction was applied during either the search or the screening process, or both. The restrictions were justified for publication period in 22.2% of the respective SRs (33/149), study type in 6.5% (28/433), and language in 3.2% (8/248). Differences in reporting were found between countries as well as between Cochrane and non-Cochrane reviews. CONCLUSIONS: This study suggests that there is a lack of transparency in reporting on restrictions in SRs. Authors as well as editors and reviewers should be encouraged to improve the reporting and justification of restrictions to increase the transparency of SRs.

Implications of analysing time-to-event outcomes as binary in meta-analysis: empirical evidence from the Cochrane Database of Systematic Reviews.

BACKGROUND: Systematic reviews and meta-analysis of time-to-event outcomes are frequently published within the Cochrane Database of Systematic Reviews (CDSR). However, these outcomes are handled differently across meta-analyses. They can be analysed on the hazard ratio (HR) scale or can be dichotomized and analysed as binary outcomes using effect measures such as odds ratios (OR) or risk ratios (RR). We investigated the impact of reanalysing meta-analyses from the CDSR that used these different effect measures. METHODS: We extracted two types of meta-analysis data from the CDSR: either recorded in a binary form only ("binary"), or in binary form together with observed minus expected and variance statistics ("OEV"). We explored how results for time-to-event outcomes originally analysed as "binary" change when analysed using the complementary log-log (clog-log) link on a HR scale. For the data originally analysed as HRs ("OEV"), we compared these results to analysing them as binary on a HR scale using the clog-log link or using a logit link on an OR scale. RESULTS: The pooled HR estimates were closer to 1 than the OR estimates in the majority of meta-analyses. Important differences in between-study heterogeneity between the HR and OR analyses were also observed. These changes led to discrepant conclusions between the OR and HR scales in some meta-analyses. Situations under which the clog-log link performed better than logit link and vice versa were apparent, indicating that the correct choice of the method does matter. Differences between scales arise mainly when event probability is high and may occur via differences in between-study heterogeneity or via increased within-study standard error in the OR relative to the HR analyses. CONCLUSIONS: We identified that dichotomising time-to-event outcomes may be adequate for low event probabilities but not for high event probabilities. In meta-analyses where only binary data are available, the complementary log-log link may be a useful alternative when analysing time-to-event outcomes as binary, however the exact conditions need further exploration. These findings provide guidance on the appropriate methodology that should be used when conducting such meta-analyses.