Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power.

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

"Expression of concern": publication bias for positive preclinical cardioprotection studies.

The present analysis reports on the robustness of preclinical cardioprotection studies with infarct size as endpoint which were published in Basic Research in Cardiology, Cardiovascular Research, and Circulation Research between January 2013 and December 2023. Only 26 out of 269 papers with technically robust analysis of infarct size by triphenyltetrazolium chloride staining, magnetic resonance imaging or single photon emission tomography applied a prospective power analysis. A retrospective power calculation revealed that only 75% of the reported data sets with statistically significant positive results from all these studies had a statistical power of ≥ 0.9, and an additional 9% had a statistical power ≥ 0.8. The remaining 16% of all significant positive data sets did not even reach the 0.8 threshold. Only 13% of all analyzed data sets were neutral. We conclude that neutral studies are underreported and there is indeed a significant lack of robustness in many of the published preclinical cardioprotection studies which may contribute to the difficulties of translating cardioprotection to patient benefit.

Efficacy of mesenchymal stem cell transplantation on major adverse cardiovascular events and cardiac function indices in patients with chronic heart failure: a meta-analysis of randomized controlled trials.

BACKGROUND: The effects of mesenchymal stem cells (MSCs) on heart failure (HF) have been controversial. This study was conducted to investigate whether the transplantation of MSCs after HF could help improve clinical outcomes and myocardial performance indices. METHODS: Using a systematic approach, electronic databases were searched for randomized controlled trials (RCTs), which evaluated the transplantation of MSCs after HF. The outcomes owf interest included clinical outcomes and myocardial function indices. We also assessed the role of age, cause of heart failure, cell origin, cell number, type of donor (autologous/allogeneic), and route of cell delivery on these outcomes. Using the random-effects method, a relative risk (RR) or mean difference (MD) and their corresponding 95% confidence intervals (CI) were pooled. RESULTS: Seventeen RCTs including 1684 patients (927 and 757 patients in the intervention and control arms, respectively) were enrolled. The RR (95% CI) of mortality was 0.78 (0.62; 0.99, p = 0.04) in the MSC group compared to the controls. HF rehospitalization decreased in the MSC group (RR = 0.85 (0.71-1.01), p = 0.06), but this was only significant in those who received autologous MSCs (RR = 0.67 (0.49; 0.90), p = 0.008). LVEF was significantly increased among those who received MSC (MD = 3.38 (1.89; 4.87), p < 0.001). LVESV (MD = -9.14 (-13.25; -5.03), p < 0.001), LVEDV (MD = -8.34 -13.41; -3.27), p < 0.001), and scar size (standardized MD = -0.32 (-0.60; -0.05), p = 0.02) were significantly decreased. NYHA class (MD = -0.19 (-0.34; -0.06), p = 0.006), BNP level (standardized MD = -0.28 (-0.50; -0.06), p = 0.01), and MLHFQ (MD = -11.55 (-16.77; -6.33), p = 0.005) significantly decreased and 6-min walk test significantly improved (MD = 36.86 (11.22; 62.50), p = 0.001) in the MSC group. Trials were not affected by the participants' etiology of heart failure, while trials with the autologous source of cells, MSC doses lower than 100 million cells, and intracoronary injection performed significantly better in some of the outcomes. CONCLUSION: Transplantation of MSCs for ischemic or dilated heart failure patients may reduce all-cause mortality and improve clinical condition. Moreover, this treatment would improve left ventricular function indices and reduce scar size.

Head-to-head comparison of influenza vaccines in children: a systematic review and meta-analysis.

Although vaccination is considered the most effective weapon against influenza, coverage rates, national vaccination policies, and funding vary largely around the globe. Despite their huge potential for achieving herd immunity, child-focused national vaccination strategies that favor pain-free nasal vaccines are uncommon. CENTRAL, Embase, and MEDLINE were last searched on November 13, 2023. Active-controlled randomized controlled trials comparing the live-attenuated intranasal vaccine with the inactivated intramuscular influenza vaccine in children were included. Event rates of laboratory-confirmed influenza virus infection, all-cause mortality, hospitalization, serious adverse events, adverse events, and financial outcomes were extracted based on the PRISMA 2020 Guideline. PROSPERO: CRD42021285412. Pooled odds ratios (ORs) with 95% confidence intervals (CI) were calculated using the random-effects model when at least three comparable outcomes were available. We found no significant difference between quadrivalent live-attenuated intranasal and trivalent inactivated intramuscular (OR = 1.48; 95% CI 0.49-4.45) or between trivalent live-attenuated intranasal and inactivated intramuscular vaccines (OR = 0.77, CI = 0.44-1.34) regarding their efficacy. However, the subgroup analysis of large, multi-center trials indicated that the trivalent live attenuated intranasal influenza vaccine was superior to the trivalent inactivated intramuscular influenza vaccine (12,154 people, OR = 0.50, CI = 0.28-0.88). Only 23 "vaccine-related serious adverse events" were recorded among 17 833 individuals, with no significant difference between methods. The widespread initiation of pediatric national flu vaccination programs prioritizing the live-attenuated intranasal influenza vaccine would be beneficial. Multi-continent, high-quality studies that include children younger than two years old and those living in subtropical and tropical regions are needed to further enhance our understanding.

Chronic atrophic gastritis and risk of incident upper gastrointestinal cancers: a systematic review and meta-analysis.

BACKGROUND: Previous literature has explored the relationship between chronic atrophic gastritis (CAG) and isolated cancers within the upper gastrointestinal cancers; However, an integrative synthesis across the totality of upper gastrointestinal cancers was conspicuously absent. The research objective was to assess the relationship between CAG and the risk of incident upper gastrointestinal cancers, specifically including gastric cancer, oesophageal cancer, and oesophagogastric junction cancer. METHODS: Rigorous systematic searches were conducted across three major databases, namely PubMed, Embase and Web of Science, encompassing the timeline from database inception until August 10, 2023. We extracted the necessary odds ratio (OR) and their corresponding 95% confidence interval (CI) for subsequent meta-analysis. Statistical analyses were conducted using Stata 17.0 software. RESULTS: This meta-analysis included a total of 23 articles encompassing 5858 patients diagnosed with upper gastrointestinal cancers. CAG resulted in a statistically significant 4.12-fold elevated risk of incident gastric cancer (OR = 4.12, 95% CI 3.20-5.30). Likewise, CAG was linked to a 2.08-fold increased risk of incident oesophageal cancer (OR = 2.08, 95%CI 1.60-2.72). Intriguingly, a specific correlation was found between CAG and the risk of incident oesophageal squamous cell carcinoma (OR = 2.29, 95%CI 1.77-2.95), while no significant association was detected for oesophageal adenocarcinoma (OR = 0.62, 95%CI 0.17-2.26). Moreover, CAG was correlated with a 2.77-fold heightened risk of oesophagogastric junction cancer (OR = 2.77, 95%CI 2.21-3.46). Notably, for the same type of upper gastrointestinal cancer, it was observed that diagnosing CAG through histological methods was linked to a 33-77% higher risk of developing cancer compared to diagnosing CAG through serological methods. CONCLUSION: This meta-analysis indicated a two- to fourfold increased risk of gastric cancer, oesophageal cancer, and oesophagogastric junction cancer in patients with CAG. Importantly, for the same upper gastrointestinal cancer, the risk of incident cancer was higher when CAG was diagnosed histologically compared to serological diagnosis. Further rigorous study designs are required to explore the impact of CAG diagnosed through both diagnostic methods on the risk of upper gastrointestinal cancers.

Posttraumatic stress and delay discounting: a meta-analytic review.

Delay discounting-the extent to which individuals show a preference for smaller immediate rewards over larger delayed rewards-has been proposed as a transdiagnostic neurocognitive process across mental health conditions, but its examination in relation to posttraumatic stress disorder (PTSD) is comparatively recent. To assess the aggregated evidence for elevated delay discounting in relation to posttraumatic stress, we conducted a meta-analysis on existing empirical literature. Bibliographic searches identified 209 candidate articles, of which 13 articles with 14 independent effect sizes were eligible for meta-analysis, reflecting a combined sample size of N = 6897. Individual study designs included case-control (e.g. examination of differences in delay discounting between individuals with and without PTSD) and continuous association studies (e.g. relationship between posttraumatic stress symptom severity and delay discounting). In a combined analysis of all studies, the overall relationship was a small but statistically significant positive association between posttraumatic stress and delay discounting (r = .135, p < .0001). The same relationship was statistically significant for continuous association studies (r = .092, p = .027) and case-control designs (r = .179, p < .001). Evidence of publication bias was minimal. The included studies were limited in that many did not concurrently incorporate other psychiatric conditions in the analyses, leaving the specificity of the relationship to posttraumatic stress less clear. Nonetheless, these findings are broadly consistent with previous meta-analyses of delayed reward discounting in relation to other mental health conditions and provide further evidence for the transdiagnostic utility of this construct.

Unpublished trials of alprazolam XR and their influence on its apparent efficacy for panic disorder.

OBJECTIVE: To test for publication bias with alprazolam, the most widely prescribed benzodiazepine, by comparing its efficacy for panic disorder using trial results from (1) the published literature and (2) the US Food and Drug Administration (FDA). METHODS: From FDA reviews, we included data from all phase 2/3 efficacy trials of alprazolam extended-release (Xanax XR) for the treatment of panic disorder. A search for matching publications was performed using PubMed and Google Scholar. Publication bias was examined by comparing: (1) overall trial results (positive or not) according to the FDA v. corresponding publications; (2) effect size (Hedges's g) based on FDA data v. published data. RESULTS: The FDA review showed that five trials were conducted, only one of which (20%) was positive. Of the four not-positive trials, two were published conveying a positive outcome; the other two were not published. Thus, according to the published literature, three trials were conducted and all (100%) were positive. Alprazolam's effect size calculated using FDA data was 0.33 (CI95% 0.07-0.60) v. 0.47 (CI95% 0.30-0.65) using published data, an increase of 0.14, or 42%. CONCLUSIONS: Publication bias substantially inflates the apparent efficacy of alprazolam XR.

Editorial Perspective: Misaligned incentives in mental health research - the case for Registered Reports.

Current incentive structures reward mental health researchers for producing positive, novel, and clean results. This can promote questionable research practices which contribute to a distorted evidence base, in turn limiting progress in mental health research. Registered Reports (RRs) offer a solution to realign the incentives towards conducting high-quality, rigorous, and accurate studies, by preventing publication and reporting biases. However, the uptake of RRs in mental health research has so far been limited. This editorial perspective highlights the advantages of RRs for mental health research, before discussing potential challenges and how they can be addressed. Greater uptake of RRs in mental health research could help to promote a fairer research culture, limit publication bias and questionable research practices, and ultimately, improve understanding of mental health.

Sensitivity analysis for publication bias in meta-analysis of sparse data based on exact likelihood.

Meta-analysis is a powerful tool to synthesize findings from multiple studies. The normal-normal random-effects model is widely used to account for between-study heterogeneity. However, meta-analyses of sparse data, which may arise when the event rate is low for binary or count outcomes, pose a challenge to the normal-normal random-effects model in the accuracy and stability in inference since the normal approximation in the within-study model may not be good. To reduce bias arising from data sparsity, the generalized linear mixed model can be used by replacing the approximate normal within-study model with an exact model. Publication bias is one of the most serious threats in meta-analysis. Several quantitative sensitivity analysis methods for evaluating the potential impacts of selective publication are available for the normal-normal random-effects model. We propose a sensitivity analysis method by extending the likelihood-based sensitivity analysis with the $t$-statistic selection function of Copas to several generalized linear mixed-effects models. Through applications of our proposed method to several real-world meta-analyses and simulation studies, the proposed method was proven to outperform the likelihood-based sensitivity analysis based on the normal-normal model. The proposed method would give useful guidance to address publication bias in the meta-analysis of sparse data.

Nonparametric worst-case bounds for publication bias on the summary receiver operating characteristic curve.

The summary receiver operating characteristic (SROC) curve has been recommended as one important meta-analytical summary to represent the accuracy of a diagnostic test in the presence of heterogeneous cutoff values. However, selective publication of diagnostic studies for meta-analysis can induce publication bias (PB) on the estimate of the SROC curve. Several sensitivity analysis methods have been developed to quantify PB on the SROC curve, and all these methods utilize parametric selection functions to model the selective publication mechanism. The main contribution of this article is to propose a new sensitivity analysis approach that derives the worst-case bounds for the SROC curve by adopting nonparametric selection functions under minimal assumptions. The estimation procedures of the worst-case bounds use the Monte Carlo method to approximate the bias on the SROC curves along with the corresponding area under the curves, and then the maximum and minimum values of PB under a range of marginal selection probabilities are optimized by nonlinear programming. We apply the proposed method to real-world meta-analyses to show that the worst-case bounds of the SROC curves can provide useful insights for discussing the robustness of meta-analytical findings on diagnostic test accuracy.

Dissemination and outcome reporting bias in clinical malaria intervention trials: a cross-sectional analysis.

BACKGROUND: Dissemination and outcome reporting biases are a significant problem in clinical research, with far-reaching implications for both scientific understanding and clinical decision-making. This study investigates the prevalence of dissemination- and outcome reporting biases in registered interventional malaria research. METHODS: All malaria interventional trials registered on ClinicalTrials.gov from 2010 to 2020 were identified. Subsequently, publications that matched the registration were searched. The primary outcome measures were the percentage of registered studies that resulted in subsequent publication of study results, the concordance between registered outcomes, and reported outcomes. Secondary outcomes were compliance with WHO standards for timely publication (issued in 2017) of summary study results in the respective trial registry (within 12 months of study completion) or peer-reviewed publication (within 24 months of study completion) was evaluated. RESULTS: A total of 579 trials were identified on ClinicalTrials.gov, of which 544 met the inclusion criteria. Notably, almost 36.6% of these trials (199/544) were registered retrospectively, with 129 (23.7%) registered after the first patient enrolment and 70 (12.9%) following study completion. Publications were identified for 351 out of 544 registered trials (64.5%), involving 1,526,081 study participants. Conversely, publications were not found for 193 of the 544 registrations (35.5%), which aimed to enrol 417,922 study participants. Among these 544 registrations, 444 (81.6%) did not meet the WHO standard to post summary results within 12 months of primary study completion (the last visit of the last subject for collection of data on the primary outcome), while 386 out of 544 registrations (71.0%) failed to publish their results in a peer-reviewed journal within 24 months of primary study completion. Discrepancies were noted in the reported primary outcomes compared to the registered primary outcomes in 47.6% (222/466) of the published trials, and an even higher discordance rate of 73.2% (341/466 publications) for secondary outcomes. CONCLUSIONS: Non-dissemination remains a significant issue in interventional malaria research, with most trials failing to meet WHO standards for timely dissemination of summary results and peer-reviewed journal publications. Additionally, outcome reporting bias is highly prevalent across malaria publications. To address these challenges, it is crucial to implement strategies that enhance the timely reporting of research findings and reduce both non-dissemination and outcome reporting bias.

The diagnostic performance of salivary gland ultrasound elastography in Sjögren's syndrome and sicca symptoms: a systematic review and meta-analysis.

OBJECTIVE: To systematically evaluate the diagnostic performance of ultrasound elastography (USE) in distinguishing primary Sjögren's syndrome (pSS) from healthy/disease controls. METHODS: We searched the PubMed, Embase, Web of Science, and Cochrane Library databases for published literature on USE for diagnosing pSS. Bivariate random effects models were used to calculate the pooled sensitivity and specificity of USE. To determine the factors influencing heterogeneity, meta-regression and subgroup analyses were performed to assess country, diagnostic criteria, imaging mechanisms, shear wave elastography techniques, measurement location, control group category, and patient age. Publication bias was assessed using the asymmetry of the Deeks funnel plot. RESULTS: Fifteen articles covering 816 patients and 735 control participants were included. USE showed a pooled sensitivity of 0.80 (95% CI: 0.71-0.87) and specificity of 0.87 (95% CI: 0.78-0.92). Meta-regression and subgroup analyses revealed that shear wave elastography techniques, measurement location, and patient age were significant factors that affected study heterogeneity (p < 0.05). Elastography performs better in diagnosing patients aged ≤ 51 years compared to patients aged > 51 years. There was no significant publication bias. CONCLUSION: USE demonstrates high accuracy in differentiating between pSS and healthy/disease control groups. CLINICAL RELEVANCE STATEMENT: Ultrasound elastography, as a non-invasive and cost-effective technique, can be used to distinguish primary Sjögren's syndrome from disease/healthy control groups by measuring the stiffness of salivary glands. KEY POINTS: • Ultrasound elastography is an acceptable technique for the diagnosis of primary Sjögren's syndrome. • The pooled sensitivity and specificity of ultrasound elastography for diagnosing primary Sjögren's syndrome were 0.80 and 0.87, respectively. • In patients aged ≤ 51 years with primary Sjögren's syndrome, ultrasound elastography showed good diagnostic performance.

A Review of Meta-Analyses in Plastic Surgery: Need for Adequate Assessment of Publication Bias.

INTRODUCTION: Publication bias describes a phenomenon in which significant positive results have a higher likelihood of being published compared to negative or nonsignificant results. Publication bias can confound the estimated therapeutic effect in meta-analyses and needs to be adequately assessed in the surgical literature. METHODS: A review of meta-analyses published in five plastic surgery journals from 2002 to 2022 was conducted. The inclusion criteria for meta-analyses were factors that demonstrated an obligation to assess publication bias, such as interventions with comparable treatment groups and enough power for statistical analysis. Acknowledgment of publication bias risk, quality of bias assessment, methods used in assessment, and individual article factors were analyzed. RESULTS: 318 unique meta-analyses were identified in literature search, and after full-text reviews, 143 met the inclusion criteria for obligation to assess publication bias. 64% of eligible meta-analyses acknowledged the confounding potential of publication bias, and only 46% conducted a formal assessment. Of those who conducted an assessment, 49% used subjective inspection of funnel plots alone, while 47% used any statistical testing in analysis. Overall, only 9/143 (6.3%) assessed publication bias and attempted to correct for its effect. Journals with a higher average impact factor were associated with mention and assessment of publication bias, but more recent publication year and higher number of primary articles analyzed were not. CONCLUSIONS: This review identified low rates of proper publication bias assessment in meta-analyses published in five major plastic surgery journals. Assessment of publication bias using objective statistical testing is necessary to ensure quality literature within surgical disciplines.

Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials.

BACKGROUND: Preprints are increasingly used to disseminate research results, providing multiple sources of information for the same study. We assessed the consistency in effect estimates between preprint and subsequent journal article of COVID-19 randomized controlled trials. METHODS: The study utilized data from the COVID-NMA living systematic review of pharmacological treatments for COVID-19 (covid-nma.com) up to July 20, 2022. We identified randomized controlled trials (RCTs) evaluating pharmacological treatments vs. standard of care/placebo for patients with COVID-19 that were originally posted as preprints and subsequently published as journal articles. Trials that did not report the same analysis in both documents were excluded. Data were extracted independently by pairs of researchers with consensus to resolve disagreements. Effect estimates extracted from the first preprint were compared to effect estimates from the journal article. RESULTS: The search identified 135 RCTs originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs that did not meet the eligibility criteria, of which 13 RCTs reported an interim analysis in the preprint and a final analysis in the journal article. Overall, 109 preprint-article RCTs were included in the analysis. The median (interquartile range) delay between preprint and journal article was 121 (73-187) days, the median sample size was 150 (71-464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias was rated as 'some concern' for 80% of RCTs. We found that 81 preprint-article pairs of RCTs were consistent for all outcomes reported. There were nine RCTs with at least one outcome with a discrepancy in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect. Furthermore, six RCTs had at least one outcome missing in the journal article and 14 RCTs had at least one outcome added in the journal article compared to the preprint. There was a change in the direction of effect in one RCT. No changes in statistical significance or conclusions were found. CONCLUSIONS: Effect estimates were generally consistent between COVID-19 preprints and subsequent journal articles. The main results and interpretation did not change in any trial. Nevertheless, some outcomes were added and deleted in some journal articles.

Effect of powered circular stapler in colorectal anastomosis after left-sided colic resection: systematic review and meta-analysis.

PURPOSE: Anastomotic leak (AL) remains the most important complication after left-sided colic anastomoses and technical complications during anastomotic construction are responsible of higher leakage incidence. Powered circular stapler (PCS) in colorectal surgery has been introduced in order to reduce technical errors and post-operative complications due to the manual circular stapler (MCS). METHODS: A systematic review and meta-analysis were performed. An electronic systematic search was performed using Web of Science, PubMed, and Embase of studies comparing PCS and MCS. The incidence of AL, anastomotic bleeding (AB), conversion, and reoperation were assessed. PROSPERO Registration Number: CRD42024512644. RESULTS: Five observational studies were eligible for inclusion reporting on 2379 patients. The estimated pooled Risk Ratios for AL and AB rates following PCS were significantly lower than those observed with MCS (0.44 and 0.23, respectively; both with p < 0.01). Conversion and reoperation rate did not show any significant difference: 0.41 (95% CI 0.09-1.88; p = 0.25) and 0.78 (95% CI 0.33-1.84; p = 0.57); respectively. CONCLUSION: The use of PCS demonstrates a lower incidence of AL and AB compared to MCS but does not exhibit a discernible influence on reintervention or conversion rates. The call for future randomized clinical trials aims to definitively clarify these issues and contribute to further advancements in refining surgical strategies for left-sided colonic resection.

Comparison of low versus high (standard) intraabdominal pressure during laparoscopic colorectal surgery: systematic review and meta-analysis.

BACKGROUND: To evaluate outcomes of low with high intraabdominal pressure during laparoscopic colorectal resection surgery. METHODS: A systematic search of multiple electronic data sources was conducted, and all studies comparing low with high (standard) intraabdominal pressures were included. Our primary outcomes were post-operative ileus occurrence and return of bowel movement/flatus. The evaluated secondary outcomes included: total operative time, post-operative haemorrhage, anastomotic leak, pneumonia, surgical site infection, overall post-operative complications (categorised by Clavien-Dindo grading), and length of hospital stay. Revman 5.4 was used for data analysis. RESULTS: Six randomised controlled trials (RCTs) and one observational study with a total of 771 patients (370 surgery at low intraabdominal pressure and 401 at high pressures) were included. There was no statistically significant difference in all the measured outcomes; post-operative ileus [OR 0.80; CI (0.42, 1.52), P = 0.50], time-to-pass flatus [OR -4.31; CI (-12.12, 3.50), P = 0.28], total operative time [OR 0.40; CI (-10.19, 11.00), P = 0.94], post-operative haemorrhage [OR 1.51; CI (0.41, 5.58, P = 0.53], anastomotic leak [OR 1.14; CI (0.26, 4.91), P = 0.86], pneumonia [OR 1.15; CI (0.22, 6.09), P = 0.87], SSI [OR 0.69; CI (0.19, 2.47), P = 0.57], overall post-operative complications [OR 0.82; CI (0.52, 1.30), P = 0.40], Clavien-Dindo grade ≥ 3 [OR 1.27; CI (0.59, 2.77), P = 0.54], and length of hospital stay [OR -0.68; CI (-1.61, 0.24), P = 0.15]. CONCLUSION: Low intraabdominal pressure is safe and feasible approach to laparoscopic colorectal resection surgery with non-inferior outcomes to standard or high pressures. More robust and well-powered RCTs are needed to consolidate the potential benefits of low over high pressure intra-abdominal surgery.

A case study evaluating the effect of clustering, publication bias, and heterogeneity on the meta-analysis estimates in implant dentistry.

Meta-analyses may provide imprecise estimates when important meta-analysis parameters are not considered during the synthesis. The aim of this case study was to highlight the influence of meta-analysis parameters that can affect reported estimates using as an example pre-existing meta-analyses on the association between implant survival and sinus membrane perforation. PubMed was searched on 7 July 2021 for meta-analyses comparing implant failure in perforated and non-perforated sinus membranes. Primary studies identified in these meta-analyses were combined in a new random-effects model with odds ratios (ORs), confidence intervals (CIs), and prediction intervals reported. Using this new meta-analysis, further meta-analyses were then undertaken considering the clinical, methodological, and statistical heterogeneity of the primary studies, publication bias, and clustering effects. The meta-analyses with the greatest number and more homogeneous studies provided lower odds of implant failure in non-perforated sites (OR 0.49, 95 % CI = [0.26, 0.92]). However, when considering heterogeneity, publication bias, and clustering (number of implants), the confidence in these results was reduced. Interpretation of estimates reported in systematic reviews can vary depending on the assumptions made in the meta-analysis. Users of these analyses need to carefully consider the impact of heterogeneity, publication bias, and clustering, which can affect the size, direction, and interpretation of the reported estimates.

Publication Bias and Selective Outcome Reporting in Randomized Controlled Trials Related to Rehabilitation: A Literature Review.

OBJECTIVE: To investigate the rate of registered protocols published as research papers as a measure of publication bias, and the concordance rates of the primary outcomes between research protocol and published papers as a measure of selective outcome reporting bias in randomized controlled trials (RCTs) related to rehabilitation. DATA SOURCES: Protocols related to RCTs were extracted from electronic databases, the University Hospital Medical Information Network (UMIN), International Standard Research Clinical Trial Number (ISRCTN), ClinicalTrials.gov, and MEDLINE. Published papers were retrieved from MEDLINE. STUDY SELECTION: The inclusion criteria were as follows: (1) initial registration (UMIN, ISRCTN, ClinicalTrials.gov) within the designated period; (2) published as a paper from a research protocol in MEDLINE (PubMed); and (3) written in English or Japanese. The search period was from January 1, 2013, to December 31, 2020. DATA EXTRACTION: The outcome of this study was set as the rate of published papers that were consistent with the extracted research protocol and the concordance rate between the primary outcomes in published papers and in protocols. The concordance rate of the primary outcomes was evaluated by checking whether the description in the research protocol matched that in the paper's abstract and main text. DATA SYNTHESIS: Out of the 5597 research protocols registered, only 727 were published (13.0%). The concordance rates of the primary outcomes were 48.7% and 72.6% in the abstract and main text, respectively. CONCLUSIONS: This study revealed major discrepancies between the number of research protocols and published papers, and difference of description regarding the primary outcomes in published papers which were already defined in the research protocols.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology.

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies.

When comparing the performance of two or more competing tests, simulation studies commonly focus on statistical power. However, if the size of the tests being compared are either different from one another or from the nominal size, comparing tests based on power alone may be misleading. By analogy with diagnostic accuracy studies, we introduce relative positive and negative likelihood ratios to factor in both power and size in the comparison of multiple tests. We derive sample size formulas for a comparative simulation study. As an example, we compared the performance of six statistical tests for small-study effects in meta-analyses of randomized controlled trials: Begg's rank correlation, Egger's regression, Schwarzer's method for sparse data, the trim-and-fill method, the arcsine-Thompson test, and Lin and Chu's combined test. We illustrate that comparing power alone, or power adjusted or penalized for size, can be misleading, and how the proposed likelihood ratio approach enables accurate comparison of the trade-off between power and size between competing tests.