Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability.

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement, and implement the computations in the R package IRR2FPR.

Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics.

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Exploring open science practices in behavioural public policy research.

In their book 'Nudge: Improving Decisions About Health, Wealth and Happiness', Thaler & Sunstein (2009) argue that choice architectures are promising public policy interventions. This research programme motivated the creation of 'nudge units', government agencies which aim to apply insights from behavioural science to improve public policy. We closely examine a meta-analysis of the evidence gathered by two of the largest and most influential nudge units (DellaVigna & Linos (2022 Econometrica 90, 81-116 (doi:10.3982/ECTA18709))) and use statistical techniques to detect reporting biases. Our analysis shows evidence suggestive of selective reporting. We additionally evaluate the public pre-analysis plans from one of the two nudge units (Office of Evaluation Sciences). We identify several instances of excellent practice; however, we also find that the analysis plans and reporting often lack sufficient detail to evaluate (unintentional) reporting biases. We highlight several improvements that would enhance the effectiveness of the pre-analysis plans and reports as a means to combat reporting biases. Our findings and suggestions can further improve the evidence base for policy decisions.

Association between Antenatal Antibiotic Exposure and Bronchopulmonary Dysplasia: A Systematic Review and Bayesian Model-Averaged Meta-Analysis.

INTRODUCTION: Antenatal antibiotic exposure has been suggested as a risk factor for bronchopulmonary dysplasia (BPD). We aimed to summarize the evidence from randomized controlled trials (RCTs) and observational studies on this potential association. METHODS: PubMed/Medline and Embase databases were searched. BPD was classified as BPD28 (supplemental oxygen during 28 days or at postnatal day 28), BPD36 (supplemental oxygen at 36 weeks postmenstrual age), BPD36 or death, and BPD-associated pulmonary hypertension (BPD-PH). Bayesian model-averaged (BMA) meta-analysis was used to calculate Bayes factors (BFs). The BF10 is the ratio of the probability of the data under the alternative hypothesis (H1) over the probability of the data under the null hypothesis (H0). RESULTS: We included 6 RCTs and 27 observational studies (126,614 infants). Regarding BPD28, BMA showed that the evidence in favor of H0 (lack of association with antenatal antibiotics) was weak for the RCTS (BF10 = 0.506, 6 studies) and moderate for the observational studies (BF10 = 0.286, 10 studies). Regarding BPD36, the evidence in favor of H0 was moderate for the RCTs (BF10 = 0.127, 2 studies) and weak for the observational studies (BF10 = 0.895, 14 studies). Evidence in favor of H0 was also weak for the associations with BPD36 or death (BF10 = 0.429, 2 studies) and BPD-PH (BF10 = 0.384, 2 studies). None of the meta-analyses showed evidence in favor of H1. CONCLUSIONS: The currently available evidence suggests a lack of association between antenatal antibiotics and BPD. However, our results should not be interpreted as an argument for widespread use of antibiotics in the setting of preterm delivery.

"This behavior strikes us as ideal": assessment and anticipations of Huisman (2022).

Huisman (Psychonomic Bulletin & Review, 1-10. 2022) argued that a valid measure of evidence should indicate more support in favor of a true alternative hypothesis when sample size is large than when it is small. Bayes factors may violate this pattern and hence Huisman concluded that Bayes factors are invalid as a measure of evidence. In this brief comment we call attention to the following: (1) Huisman's purported anomaly is in fact dictated by probability theory; (2) Huisman's anomaly has been discussed and explained in the statistical literature since 1939; the anomaly was also highlighted in the Psychonomic Bulletin & Review article by Rouder et al. (2009), who interpreted the anomaly as "ideal": an interpretation diametrically opposed to that of Huisman. We conclude that when intuition clashes with probability theory, chances are that it is intuition that needs schooling.

Platelet Counts and Risk of Severe Retinopathy of Prematurity: A Bayesian Model-Averaged Meta-Analysis.

BACKGROUND: We aimed to conduct a systematic review and Bayesian model-averaged meta-analysis (BMA) on the association between platelet counts and severe retinopathy of prematurity (ROP). METHODS: We searched for studies reporting on platelet counts (continuous variable) or thrombocytopenia (categorical variable) and severe ROP or aggressive posterior ROP (APROP). The timing of platelet counts was divided into Phase 1 (<2 weeks) and Phase 2 (around ROP treatment). BMA was used to calculate Bayes factors (BFs). The BF10 is the ratio of the probability of the data under the alternative hypothesis (H1) over the probability of the data under the null hypothesis (H0). RESULTS: We included 21 studies. BMA showed an association between low platelet counts and severe ROP. The evidence was strong (BF10 = 13.5, 7 studies) for phase 1 and very strong (BF10 = 51.0, 9 studies) for phase 2. Thrombocytopenia (<100 × 109/L) in phase 2 was associated with severe ROP (BF10 = 28.2, 4 studies). Following adjustment for publication bias, only the association of severe ROP with thrombocytopenia remained with moderate evidence in favor of H1 (BF10 = 4.30). CONCLUSIONS: Thrombocytopenia is associated with severe ROP. However, the evidence for this association was tempered when results were adjusted for publication bias.

Patent Ductus Arteriosus and Bronchopulmonary Dysplasia-Associated Pulmonary Hypertension: A Bayesian Meta-Analysis.

Importance: Bronchopulmonary dysplasia (BPD) is often associated with pulmonary vascular disease and secondary pulmonary hypertension (PH). The pathogenesis of BPD-associated PH (BPD-PH) is complex and involves prenatal and postnatal factors that disrupt pulmonary vascular development, and patent ductus arteriosus (PDA) is a factor potentially associated with risk of BPD-PH that has been identified in very recent studies. Objective: To explore the association of PDA with BPD-PH using a bayesian model-averaged (BMA) meta-analysis of studies. Data Sources: PubMed and Embase were searched up to April 2023. Key search terms included BPD and PH. Study Selection: Studies examining infants with gestational age 32 weeks or less and reporting data on PDA and risk of BPD-PH. Data Extraction and Synthesis: This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses and the Meta-Analysis of Observational Studies in Epidemiology reporting guidelines. Two independent reviewers extracted data, with a third reviewer checking for accuracy and completeness. Data pooling and effect size calculations were performed by BMA. Main Outcomes and Measures: The primary outcome was BPD-PH. BMA was used to calculate Bayes factors (BFs). The BF10 is the ratio of the probability of the data under the alternative hypothesis (H1, association of PDA with BPD-HP) over the probability of the data under the null hypothesis (H0). Results: A total of 32 studies (8513 infants) were included. BMA showed that the evidence in favor of H1 was weak for any PDA (BF10 = 2.90; 10 studies), moderate for hemodynamically significant PDA (BF10 = 3.77; 3 studies), and extreme for surgically ligated or catheter-occluded PDA (BF10 = 294.9; 16 studies). In contrast, the evidence in favor of H0 was weak for medically treated PDA (BF10 = 0.55; 6 studies). In addition, BMA found strong evidence in favor of H1 when prolonged exposure to PDA was analyzed as a dichotomous variable (BF10 = 11.80; 6 studies) and extreme evidence (BF10 = 113.60; 3 studies) when PDA exposure time was analyzed as a continuous variable. Conclusions and Relevance: In this bayesian meta-analysis, the data suggest that prolonged exposure to PDA might be associated with increased risk of pulmonary vascular disease in extremely preterm infants. This highlights the need to monitor for PH in high-risk preterm infants with prolonged exposure to PDA and to incorporate PH risk into clinical decisions regarding PDA management.

Sex differences in the risk of retinopathy of prematurity: a systematic review, frequentist and Bayesian meta-analysis, and meta-regression.

BACKGROUND: Retinopathy of prematurity (ROP) is generally considered to be more frequent in males than in females. However, it is not known whether sex differences in ROP affect all degrees of the condition, are global and have changed as neonatology has developed. We aimed to conduct a systematic review and meta-analysis of studies addressing sex differences in the risk of developing ROP. METHODS: PubMed/MEDLINE and Embase databases were searched. The frequentist, random-effects risk ratio (RR) and 95% confidence interval (CI) were calculated. Bayesian model averaged (BMA) meta-analysis was used to calculate the Bayes factors (BFs). The BF10 is the ratio of the probability of the data under the alternative hypothesis (H1) over the probability of the data under the null hypothesis (H0). RESULTS: We included 205 studies (867,252 infants). Frequentist meta-analysis showed a positive association between male sex and severe ROP (113 studies, RR = 1.14, 95% CI = 1.07-1.22) but no association with any ROP (144 studies, RR = 1.00, 95% CI = 0.96-1.03). BMA showed extreme evidence in favor of H1 for severe ROP (BF10 = 71,174) and strong evidence in favor of H0 for any ROP (BF10 = 0.05). The association between male sex and severe ROP remained stable over time and was present only in cohorts from countries with a high or high-middle sociodemographic index. CONCLUSIONS: Our study confirms the presence of a male disadvantage in severe ROP but not in less severe forms of the disease. There are variations in the sex differences in ROP, depending on geographical location and sociodemographic level of the countries.

Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values.

The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to settle this question empirically. Here we propose a new way to estimate the false positive risk and apply the method to the results of (randomized) clinical trials in top medical journals. Contrary to claims that most published results are false, we find that the traditional significance criterion of α = .05 produces a false positive risk of 13%. Adjusting α to.01 lowers the false positive risk to less than 5%. However, our method does provide clear evidence of publication bias that leads to inflated effect size estimates. These results provide a solid empirical foundation for evaluations of the trustworthiness of medical research.

Meta-analyses in psychology often overestimate evidence for and size of effects.

Adjusting for publication bias is essential when drawing meta-analytic inferences. However, most methods that adjust for publication bias do not perform well across a range of research conditions, such as the degree of heterogeneity in effect sizes across studies. Sladekova et al. 2022 (Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods. Psychol. Methods) tried to circumvent this complication by selecting the methods that are most appropriate for a given set of conditions, and concluded that publication bias on average causes only minimal over-estimation of effect sizes in psychology. However, this approach suffers from a 'Catch-22' problem-to know the underlying research conditions, one needs to have adjusted for publication bias correctly, but to correctly adjust for publication bias, one needs to know the underlying research conditions. To alleviate this problem, we conduct an alternative analysis, robust Bayesian meta-analysis (RoBMA), which is not based on model-selection but on model-averaging. In RoBMA, models that predict the observed results better are given correspondingly larger weights. A RoBMA reanalysis of Sladekova et al.'s dataset reveals that more than 60% of meta-analyses in psychology notably overestimate the evidence for the presence of the meta-analytic effect and more than 50% overestimate its magnitude.

Multibridge: an R package to evaluate informed hypotheses in binomial and multinomial models.

The multibridge R package allows a Bayesian evaluation of informed hypotheses [Formula: see text] applied to frequency data from an independent binomial or multinomial distribution. multibridge uses bridge sampling to efficiently compute Bayes factors for the following hypotheses concerning the latent category proportions 𝜃: (a) hypotheses that postulate equality constraints (e.g., 𝜃1 = 𝜃2 = 𝜃3); (b) hypotheses that postulate inequality constraints (e.g., 𝜃1 < 𝜃2 < 𝜃3 or 𝜃1 > 𝜃2 > 𝜃3); (c) hypotheses that postulate combinations of inequality constraints and equality constraints (e.g., 𝜃1 < 𝜃2 = 𝜃3); and (d) hypotheses that postulate combinations of (a)-(c) (e.g., 𝜃1 < (𝜃2 = 𝜃3),𝜃4). Any informed hypothesis [Formula: see text] may be compared against the encompassing hypothesis [Formula: see text] that all category proportions vary freely, or against the null hypothesis [Formula: see text] that all category proportions are equal. multibridge facilitates the fast and accurate comparison of large models with many constraints and models for which relatively little posterior mass falls in the restricted parameter space. This paper describes the underlying methodology and illustrates the use of multibridge through fully reproducible examples.

Unrestricted weighted least squares represent medical research better than random effects in 67,308 Cochrane meta-analyses.

OBJECTIVES: To evaluate how well meta-analysis mean estimators represent reported medical research and establish which meta-analysis method is better using widely accepted model selection measures: Akaike information criterion (AIC) and Bayesian information criterion (BIC). STUDY DESIGN AND SETTING: We compiled 67,308 meta-analyses from the Cochrane Database of Systematic Reviews (CDSR) published between 1997 and 2020, collectively encompassing nearly 600,000 medical findings. We compared unrestricted weighted least squares (UWLS) vs. random effects (RE); fixed effect was also secondarily considered. RESULTS: The probability that a randomly selected systematic review from the CDSR would favor UWLS over RE is 79.4% (95% confidence interval [CI95%]: 79.1; 79.7). The odds ratio that a Cochrane systematic review would substantially favor UWLS over RE is 9.33 (CI95%: 8.94; 9.73) using the conventional criterion that a difference in AIC (or BIC) of two or larger represents a 'substantial' improvement. UWLS's advantage over RE is most prominent in the presence of low heterogeneity. However, UWLS also has a notable advantage in high heterogeneity research, across different sizes of meta-analyses and types of outcomes. CONCLUSION: UWLS frequently dominates RE in medical research, often substantially. Thus, the UWLS should be reported routinely in the meta-analysis of clinical trials.

Association of Funisitis with Short-Term Outcomes of Prematurity: A Frequentist and Bayesian Meta-Analysis.

The fetal systemic inflammatory response associated with intra-amniotic inflammation may play a key role in the pathogenesis of complications of preterm birth. Funisitis is the histologic equivalent of the fetal inflammatory response, whereas chorioamnionitis represents a maternal inflammatory response. We conducted a frequentist and Bayesian model average (BMA) meta-analysis of studies investigating the effects of funisitis on short-term outcomes of prematurity. Thirty-three studies (12,237 infants with gestational age ≤ 34 weeks) were included. Frequentist meta-analysis showed that funisitis was associated with an increased risk of any bronchopulmonary dysplasia (BPD), moderate/severe BPD, retinopathy of prematurity (ROP), intraventricular hemorrhage (IVH), periventricular leukomalacia (PVL), any sepsis, early-onset sepsis (EOS), and mortality. However, Bayesian meta-analysis showed that the evidence in favor of the alternative hypothesis (i.e., funisitis is associated with an increased risk of developing the outcome) was strong for any IVH, moderate for severe IVH and EOS, and weak for the other outcomes. When the control group was restricted to infants having chorioamnionitis without funisitis, the only outcome associated with funisitis was any IVH. In conclusion, our data suggest that the presence of funisitis does not add an additional risk to preterm birth when compared to chorioamnionitis in the absence of fetal inflammatory response.

Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods.

Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition-dependent, all-or-none choice between competing methods and conflicting results, we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models adjusting for small-study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of complementary publication bias adjustment methods.

Robust Bayesian meta-analysis: Addressing publication bias with model-averaging.

Meta-analysis is an important quantitative tool for cumulative science, but its application is frustrated by publication bias. In order to test and adjust for publication bias, we extend model-averaged Bayesian meta-analysis with selection models. The resulting robust Bayesian meta-analysis (RoBMA) methodology does not require all-or-none decisions about the presence of publication bias, can quantify evidence in favor of the absence of publication bias, and performs well under high heterogeneity. By model-averaging over a set of 12 models, RoBMA is relatively robust to model misspecification and simulations show that it outperforms existing methods. We demonstrate that RoBMA finds evidence for the absence of publication bias in Registered Replication Reports and reliably avoids false positives. We provide an implementation in R so that researchers can easily use the new methodology in practice. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Informed Bayesian survival analysis.

BACKGROUND: We provide an overview of Bayesian estimation, hypothesis testing, and model-averaging and illustrate how they benefit parametric survival analysis. We contrast the Bayesian framework to the currently dominant frequentist approach and highlight advantages, such as seamless incorporation of historical data, continuous monitoring of evidence, and incorporating uncertainty about the true data generating process. METHODS: We illustrate the application of the outlined Bayesian approaches on an example data set, retrospective re-analyzing a colon cancer trial. We assess the performance of Bayesian parametric survival analysis and maximum likelihood survival models with AIC/BIC model selection in fixed-n and sequential designs with a simulation study. RESULTS: In the retrospective re-analysis of the example data set, the Bayesian framework provided evidence for the absence of a positive treatment effect of adding Cetuximab to FOLFOX6 regimen on disease-free survival in patients with resected stage III colon cancer. Furthermore, the Bayesian sequential analysis would have terminated the trial 10.3 months earlier than the standard frequentist analysis. In a simulation study with sequential designs, the Bayesian framework on average reached a decision in almost half the time required by the frequentist counterparts, while maintaining the same power, and an appropriate false-positive rate. Under model misspecification, the Bayesian framework resulted in higher false-negative rate compared to the frequentist counterparts, which resulted in a higher proportion of undecided trials. In fixed-n designs, the Bayesian framework showed slightly higher power, slightly elevated error rates, and lower bias and RMSE when estimating treatment effects in small samples. We found no noticeable differences for survival predictions. We have made the analytic approach readily available to other researchers in the RoBSA R package. CONCLUSIONS: The outlined Bayesian framework provides several benefits when applied to parametric survival analyses. It uses data more efficiently, is capable of considerably shortening the length of clinical trials, and provides a richer set of inferences.

DE-PASS Best Evidence Statement (BESt): modifiable determinants of physical activity and sedentary behaviour in children and adolescents aged 5-19 years-a protocol for systematic review and meta-analysis.

INTRODUCTION: Physical activity among children and adolescents remains insufficient, despite the substantial efforts made by researchers and policymakers. Identifying and furthering our understanding of potential modifiable determinants of physical activity behaviour (PAB) and sedentary behaviour (SB) is crucial for the development of interventions that promote a shift from SB to PAB. The current protocol details the process through which a series of systematic literature reviews and meta-analyses (MAs) will be conducted to produce a best-evidence statement (BESt) and inform policymakers. The overall aim is to identify modifiable determinants that are associated with changes in PAB and SB in children and adolescents (aged 5-19 years) and to quantify their effect on, or association with, PAB/SB. METHODS AND ANALYSIS: A search will be performed in MEDLINE, SportDiscus, Web of Science, PsychINFO and Cochrane Central Register of Controlled Trials. Randomised controlled trials (RCTs) and controlled trials (CTs) that investigate the effect of interventions on PAB/SB and longitudinal studies that investigate the associations between modifiable determinants and PAB/SB at multiple time points will be sought. Risk of bias assessments will be performed using adapted versions of Cochrane's RoB V.2.0 and ROBINS-I tools for RCTs and CTs, respectively, and an adapted version of the National Institute of Health's tool for longitudinal studies. Data will be synthesised narratively and, where possible, MAs will be performed using frequentist and Bayesian statistics. Modifiable determinants will be discussed considering the settings in which they were investigated and the PAB/SB measurement methods used. ETHICS AND DISSEMINATION: No ethical approval is needed as no primary data will be collected. The findings will be disseminated in peer-reviewed publications and academic conferences where possible. The BESt will also be shared with policy makers within the DE-PASS consortium in the first instance. SYSTEMATIC REVIEW REGISTRATION: CRD42021282874.

Bayesian model-averaged meta-analysis in medicine.

We outline a Bayesian model-averaged (BMA) meta-analysis for standardized mean differences in order to quantify evidence for both treatment effectiveness δ and across-study heterogeneity τ . We construct four competing models by orthogonally combining two present-absent assumptions, one for the treatment effect and one for across-study heterogeneity. To inform the choice of prior distributions for the model parameters, we used 50% of the Cochrane Database of Systematic Reviews to specify rival prior distributions for δ and τ . The relative predictive performance of the competing models and rival prior distributions was assessed using the remaining 50% of the Cochrane Database. On average, ℋ1r -the model that assumes the presence of a treatment effect as well as across-study heterogeneity-outpredicted the other models, but not by a large margin. Within ℋ1r , predictive adequacy was relatively constant across the rival prior distributions. We propose specific empirical prior distributions, both for the field in general and for each of 46 specific medical subdisciplines. An example from oral health demonstrates how the proposed prior distributions can be used to conduct a BMA meta-analysis in the open-source software R and JASP. The preregistered analysis plan is available at https://osf.io/zs3df/.

Parity moderates the effect of delivery mode on maternal ratings of infant temperament.

OBJECTIVE: Cesarean section (CS) rates are rising rapidly around the world but no conclusive evidence has been obtained about the possible short- and long-term effects of CS on child behavior. We evaluated prospectively the association between CS and infant temperament across the first 9 postpartum months, controlling for indications for CS and investigating parity and infant sex as moderators. METHODS: The sample consisted of mothers and their healthy infants. Infant temperament was measured using the Infant Characteristics Questionnaire completed by the mothers at 6 weeks (n = 452) and 9 months (n = 258) postpartum. Mode of birth was classified into spontaneous vaginal birth (n = 347 for 6 weeks sample; 197 for 9 months sample), CS planned for medical reasons (n = 55; 28) and emergency CS (n = 50; 33). RESULTS: Multiple regression analysis revealed no main effects of birth mode, but showed a significant interaction between birth mode and parity indicating that emergency CS in firstborn infants was associated with more difficult temperament at 6 weeks. There were no significant associations between indications for CS and infant temperament, although breech presentation predicted difficult temperament at 9 months. CONCLUSION: We largely failed to support the association between CS and infant temperament. Although our results suggest that emergency CS may be associated with temperament in firstborns, further research is needed to replicate this finding, preferably using observational measures to assess child temperament.