Application of causal inference methods in individual-participant data meta-analyses in medicine: addressing data handling and reporting gaps with new proposed reporting guidelines.

Observational data provide invaluable real-world information in medicine, but certain methodological considerations are required to derive causal estimates. In this systematic review, we evaluated the methodology and reporting quality of individual-level patient data meta-analyses (IPD-MAs) conducted with non-randomized exposures, published in 2009, 2014, and 2019 that sought to estimate a causal relationship in medicine. We screened over 16,000 titles and abstracts, reviewed 45 full-text articles out of the 167 deemed potentially eligible, and included 29 into the analysis. Unfortunately, we found that causal methodologies were rarely implemented, and reporting was generally poor across studies. Specifically, only three of the 29 articles used quasi-experimental methods, and no study used G-methods to adjust for time-varying confounding. To address these issues, we propose stronger collaborations between physicians and methodologists to ensure that causal methodologies are properly implemented in IPD-MAs. In addition, we put forward a suggested checklist of reporting guidelines for IPD-MAs that utilize causal methods. This checklist could improve reporting thereby potentially enhancing the quality and trustworthiness of IPD-MAs, which can be considered one of the most valuable sources of evidence for health policy.

Multiple imputation of incomplete multilevel data using Heckman selection models.

Missing data is a common problem in medical research, and is commonly addressed using multiple imputation. Although traditional imputation methods allow for valid statistical inference when data are missing at random (MAR), their implementation is problematic when the presence of missingness depends on unobserved variables, that is, the data are missing not at random (MNAR). Unfortunately, this MNAR situation is rather common, in observational studies, registries and other sources of real-world data. While several imputation methods have been proposed for addressing individual studies when data are MNAR, their application and validity in large datasets with multilevel structure remains unclear. We therefore explored the consequence of MNAR data in hierarchical data in-depth, and proposed a novel multilevel imputation method for common missing patterns in clustered datasets. This method is based on the principles of Heckman selection models and adopts a two-stage meta-analysis approach to impute binary and continuous variables that may be outcomes or predictors and that are systematically or sporadically missing. After evaluating the proposed imputation model in simulated scenarios, we illustrate it use in a cross-sectional community survey to estimate the prevalence of malaria parasitemia in children aged 2-10 years in five regions in Uganda.

Confounder Adjustment Using the Disease Risk Score: A Proposal for Weighting Methods.

Propensity score analysis is a common approach to addressing confounding in nonrandomized studies. Its implementation, however, requires important assumptions (e.g., positivity). The disease risk score (DRS) is an alternative confounding score that can relax some of these assumptions. Like the propensity score, the DRS summarizes multiple confounders into a single score, on which conditioning by matching allows the estimation of causal effects. However, matching relies on arbitrary choices for pruning out data (e.g., matching ratio, algorithm, and caliper width) and may be computationally demanding. Alternatively, weighting methods, common in propensity score analysis, are easy to implement and may entail fewer choices, yet none have been developed for the DRS. Here we present 2 weighting approaches: One derives directly from inverse probability weighting; the other, named target distribution weighting, relates to importance sampling. We empirically show that inverse probability weighting and target distribution weighting display performance comparable to matching techniques in terms of bias but outperform them in terms of efficiency (mean squared error) and computational speed (up to >870 times faster in an illustrative study). We illustrate implementation of the methods in 2 case studies where we investigate placebo treatments for multiple sclerosis and administration of aspirin in stroke patients.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed.

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement. STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs. RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD. CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

Application of Causal Inference Methods to Pooled Longitudinal Non- Randomized Studies: A Methodological Systematic Review.

Observational data provide invaluable real-world information in medicine, but certain methodological considerations are required to derive causal estimates. In this systematic review, we evaluated the methodology and reporting quality of individual-level patient data meta-analyses (IPD-MAs) published in 2009, 2014, and 2019 that sought to estimate a causal relationship in medicine. We screened over 16,000 titles and abstracts, reviewed 45 full-text articles out of the 167 deemed potentially eligible, and included 29 into the analysis. Unfortunately, we found that causal methodologies were rarely implemented, and reporting was generally poor across studies. Specifically, only three of the 29 articles used quasi-experimental methods, and no study used G-methods to adjust for time-varying confounding. To address these issues, we propose stronger collaborations between physicians and methodologists to ensure that causal methodologies are properly implemented in IPD-MAs. In addition, we put forward a suggested checklist of reporting guidelines for IPD-MAs that utilize causal methods. This checklist could improve reporting thereby potentially enhancing the quality and trustworthiness of IPD-MAs, which can be considered one of the most valuable sources of evidence for health policy.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population.

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Development of treatment-decision algorithms for children evaluated for pulmonary tuberculosis: an individual participant data meta-analysis.

BACKGROUND: Many children with pulmonary tuberculosis remain undiagnosed and untreated with related high morbidity and mortality. Recent advances in childhood tuberculosis algorithm development have incorporated prediction modelling, but studies so far have been small and localised, with limited generalisability. We aimed to evaluate the performance of currently used diagnostic algorithms and to use prediction modelling to develop evidence-based algorithms to assist in tuberculosis treatment decision making for children presenting to primary health-care centres. METHODS: For this meta-analysis, we identified individual participant data from a WHO public call for data on the management of tuberculosis in children and adolescents and referral from childhood tuberculosis experts. We included studies that prospectively recruited consecutive participants younger than 10 years attending health-care centres in countries with a high tuberculosis incidence for clinical evaluation of pulmonary tuberculosis. We collated individual participant data including clinical, bacteriological, and radiological information and a standardised reference classification of pulmonary tuberculosis. Using this dataset, we first retrospectively evaluated the performance of several existing treatment-decision algorithms. We then used the data to develop two multivariable prediction models that included features used in clinical evaluation of pulmonary tuberculosis-one with chest x-ray features and one without-and we investigated each model's generalisability using internal-external cross-validation. The parameter coefficient estimates of the two models were scaled into two scoring systems to classify tuberculosis with a prespecified sensitivity target. The two scoring systems were used to develop two pragmatic, treatment-decision algorithms for use in primary health-care settings. FINDINGS: Of 4718 children from 13 studies from 12 countries, 1811 (38·4%) were classified as having pulmonary tuberculosis: 541 (29·9%) bacteriologically confirmed and 1270 (70·1%) unconfirmed. Existing treatment-decision algorithms had highly variable diagnostic performance. The scoring system derived from the prediction model that included clinical features and features from chest x-ray had a combined sensitivity of 0·86 [95% CI 0·68-0·94] and specificity of 0·37 [0·15-0·66] against a composite reference standard. The scoring system derived from the model that included only clinical features had a combined sensitivity of 0·84 [95% CI 0·66-0·93] and specificity of 0·30 [0·13-0·56] against a composite reference standard. The scoring system from each model was placed after triage steps, including assessment of illness acuity and risk of poor tuberculosis-related outcomes, to develop treatment-decision algorithms. INTERPRETATION: We adopted an evidence-based approach to develop pragmatic algorithms to guide tuberculosis treatment decisions in children, irrespective of the resources locally available. This approach will empower health workers in primary health-care settings with high tuberculosis incidence and limited resources to initiate tuberculosis treatment in children to improve access to care and reduce tuberculosis-related mortality. These algorithms have been included in the operational handbook accompanying the latest WHO guidelines on the management of tuberculosis in children and adolescents. Future prospective evaluation of algorithms, including those developed in this work, is necessary to investigate clinical performance. FUNDING: WHO, US National Institutes of Health.

Measuring the performance of prediction models to personalize treatment choice.

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Adjusting for misclassification of an exposure in an individual participant data meta-analysis.

A common problem in the analysis of multiple data sources, including individual participant data meta-analysis (IPD-MA), is the misclassification of binary variables. Misclassification may lead to biased estimators of model parameters, even when the misclassification is entirely random. We aimed to develop statistical methods that facilitate unbiased estimation of adjusted and unadjusted exposure-outcome associations and between-study heterogeneity in IPD-MA, where the extent and nature of exposure misclassification may vary across studies. We present Bayesian methods that allow misclassification of binary exposure variables to depend on study- and participant-level characteristics. In an example of the differential diagnosis of dengue using two variables, where the gold standard measurement for the exposure variable was unavailable for some studies which only measured a surrogate prone to misclassification, our methods yielded more accurate estimates than analyses naive with regard to misclassification or based on gold standard measurements alone. In a simulation study, the evaluated misclassification model yielded valid estimates of the exposure-outcome association, and was more accurate than analyses restricted to gold standard measurements. Our proposed framework can appropriately account for the presence of binary exposure misclassification in IPD-MA. It requires that some studies supply IPD for the surrogate and gold standard exposure, and allows misclassification to follow a random effects distribution across studies conditional on observed covariates (and outcome). The proposed methods are most beneficial when few large studies that measured the gold standard are available, and when misclassification is frequent.

Personalizing treatment in end-stage kidney disease: deciding between haemodiafiltration and haemodialysis based on individualized treatment effect prediction.

Background: Previous studies suggest that haemodiafiltration reduces mortality compared with haemodialysis in patients with end-stage kidney disease (ESKD), but the controversy surrounding its benefits remains and it is unclear to what extent individual patients benefit from haemodiafiltration. This study is aimed to develop and validate a treatment effect prediction model to determine which patients would benefit most from haemodiafiltration compared with haemodialysis in terms of all-cause mortality. Methods: Individual participant data from four randomized controlled trials comparing haemodiafiltration with haemodialysis on mortality were used to derive a Royston-Parmar model for the prediction of absolute treatment effect of haemodiafiltration based on pre-specified patient and disease characteristics. Validation of the model was performed using internal-external cross validation. Results: The median predicted survival benefit was 44 (Q1-Q3: 44-46) days for every year of treatment with haemodiafiltration compared with haemodialysis. The median survival benefit with haemodiafiltration ranged from 2 to 48 months. Patients who benefitted most from haemodiafiltration were younger, less likely to have diabetes or a cardiovascular history and had higher serum creatinine and albumin levels. Internal-external cross validation showed adequate discrimination and calibration. Conclusion: Although overall mortality is reduced by haemodiafiltration compared with haemodialysis in ESKD patients, the absolute survival benefit can vary greatly between individuals. Our results indicate that the effects of haemodiafiltration on survival can be predicted using a combination of readily available patient and disease characteristics, which could guide shared decision-making.

Reducing antibiotic use in uncomplicated urinary tract infections in adult women: a systematic review and individual participant data meta-analysis.

BACKGROUND: Randomised controlled trials (RCTs) investigated analgesics, herbal formulations, delayed prescription of antibiotics, and placebo to prevent overprescription of antibiotics in women with uncomplicated urinary tract infections (uUTI). OBJECTIVES: To estimate the effect of these strategies and to identify symptoms, signs, or other factors that indicate a benefit from these strategies. DATA SOURCES: MEDLINE, EMBASE, Web of Science, LILACS, Cochrane Database of Systematic Reviews and of Controlled Trials, and ClinicalTrials. STUDY ELIGIBILITY CRITERIA, PARTICIPANTS AND INTERVENTIONS: RCTs investigating any strategies to reduce antibiotics vs. immediate antibiotics in adult women with uUTI in primary care. METHODS: We extracted individual participant data (IPD) if available, otherwise aggregate data (AD). Bayesian random-effects meta-analysis of the AD was used for pairwise comparisons. Candidate moderators and prognostic indicators of treatment effects were investigated using generalised linear mixed models based on IPD. RESULTS: We analysed IPD of 3524 patients from eight RCTs and AD of 78 patients. Non-antibiotic strategies increased the rates of incomplete recovery (OR 3.0; 95% credible interval (CrI), 1.7-5.5; Bayesian p-value (pB) = 0.0017; τ = 0.6), subsequent antibiotic treatment (OR 3.5; 95% CrI, 2.1-5.8; pB = 0.0003) and pyelonephritis (OR 5.6; 95% CrI, 2.3-13.9; pB = 0.0003). Conversely, they decreased overall antibiotic use by 63%. Patients positive for urinary erythrocytes and urine culture were at increased risk for incomplete recovery (OR 4.7; 95% CrI, 2.1-10.8; pB = 0.0010), but no difference was apparent where both were negative (OR 0.8; 95% CrI, 0.3-2.0; pB = 0.667). In patients treated using non-antibiotic strategies, urinary erythrocytes and positive urine culture were independent prognostic indicators for subsequent antibiotic treatment and pyelonephritis. CONCLUSIONS: Compared to immediate antibiotics, non-antibiotic strategies reduce overall antibiotic use but result in poorer clinical outcomes. The presence of erythrocytes and tests to confirm bacteria in urine could be used to target antibiotic prescribing.

Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis.

OBJECTIVE: To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19. DESIGN: Two stage individual participant data meta-analysis. SETTING: Secondary and tertiary care. PARTICIPANTS: 46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021. DATA SOURCES: Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge. MODEL SELECTION AND ELIGIBILITY CRITERIA: Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor. METHODS: Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters. MAIN OUTCOME MEASURES: 30 day mortality or in-hospital mortality. RESULTS: Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28). CONCLUSION: The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Systematic Review Reveals Lack of Causal Methodology Applied to Pooled Longitudinal Observational Infectious Disease Studies.

OBJECTIVES: Among ID studies seeking to make causal inferences and pooling individual-level longitudinal data from multiple infectious disease cohorts, we sought to assess what methods are being used, how those methods are being reported, and whether these factors have changed over time. STUDY DESIGN AND SETTING: Systematic review of longitudinal observational infectious disease studies pooling individual-level patient data from 2+ studies published in English in 2009, 2014, or 2019. This systematic review protocol is registered with PROSPERO (CRD42020204104). RESULTS: Our search yielded 1,462 unique articles. Of these, 16 were included in the final review. Our analysis showed a lack of causal inference methods and of clear reporting on methods and the required assumptions. CONCLUSION: There are many approaches to causal inference which may help facilitate accurate inference in the presence of unmeasured and time-varying confounding. In observational ID studies leveraging pooled, longitudinal IPD, the absence of these causal inference methods and gaps in the reporting of key methodological considerations suggests there is ample opportunity to enhance the rigor and reporting of research in this field. Interdisciplinary collaborations between substantive and methodological experts would strengthen future work.