Global disease burden of and risk factors for acute lower respiratory infections caused by respiratory syncytial virus in preterm infants and young children in 2019: a systematic review and meta-analysis of aggregated and individual participant data.

BACKGROUND: Infants and young children born prematurely are at high risk of severe acute lower respiratory infection (ALRI) caused by respiratory syncytial virus (RSV). In this study, we aimed to assess the global disease burden of and risk factors for RSV-associated ALRI in infants and young children born before 37 weeks of gestation. METHODS: We conducted a systematic review and meta-analysis of aggregated data from studies published between Jan 1, 1995, and Dec 31, 2021, identified from MEDLINE, Embase, and Global Health, and individual participant data shared by the Respiratory Virus Global Epidemiology Network on respiratory infectious diseases. We estimated RSV-associated ALRI incidence in community, hospital admission, in-hospital mortality, and overall mortality among children younger than 2 years born prematurely. We conducted two-stage random-effects meta-regression analyses accounting for chronological age groups, gestational age bands (early preterm, <32 weeks gestational age [wGA], and late preterm, 32 to <37 wGA), and changes over 5-year intervals from 2000 to 2019. Using individual participant data, we assessed perinatal, sociodemographic, and household factors, and underlying medical conditions for RSV-associated ALRI incidence, hospital admission, and three severity outcome groups (longer hospital stay [>4 days], use of supplemental oxygen and mechanical ventilation, or intensive care unit admission) by estimating pooled odds ratios (ORs) through a two-stage meta-analysis (multivariate logistic regression and random-effects meta-analysis). This study is registered with PROSPERO, CRD42021269742. FINDINGS: We included 47 studies from the literature and 17 studies with individual participant-level data contributed by the participating investigators. We estimated that, in 2019, 1 650 000 (95% uncertainty range [UR] 1 350 000-1 990 000) RSV-associated ALRI episodes, 533 000 (385 000-730 000) RSV-associated hospital admissions, 3050 (1080-8620) RSV-associated in-hospital deaths, and 26 760 (11 190-46 240) RSV-attributable deaths occurred in preterm infants worldwide. Among early preterm infants, the RSV-associated ALRI incidence rate and hospitalisation rate were significantly higher (rate ratio [RR] ranging from 1·69 to 3·87 across different age groups and outcomes) than for all infants born at any gestational age. In the second year of life, early preterm infants and young children had a similar incidence rate but still a significantly higher hospitalisation rate (RR 2·26 [95% UR 1·27-3·98]) compared with all infants and young children. Although late preterm infants had RSV-associated ALRI incidence rates similar to that of all infants younger than 1 year, they had higher RSV-associated ALRI hospitalisation rate in the first 6 months (RR 1·93 [1·11-3·26]). Overall, preterm infants accounted for 25% (95% UR 16-37) of RSV-associated ALRI hospitalisations in all infants of any gestational age. RSV-associated ALRI in-hospital case fatality ratio in preterm infants was similar to all infants. The factors identified to be associated with RSV-associated ALRI incidence were mainly perinatal and sociodemographic characteristics, and factors associated with severe outcomes from infection were mainly underlying medical conditions including congenital heart disease, tracheostomy, bronchopulmonary dysplasia, chronic lung disease, or Down syndrome (with ORs ranging from 1·40 to 4·23). INTERPRETATION: Preterm infants face a disproportionately high burden of RSV-associated disease, accounting for 25% of RSV hospitalisation burden. Early preterm infants have a substantial RSV hospitalisation burden persisting into the second year of life. Preventive products for RSV can have a substantial public health impact by preventing RSV-associated ALRI and severe outcomes from infection in preterm infants. FUNDING: EU Innovative Medicines Initiative Respiratory Syncytial Virus Consortium in Europe.

Association between pregnancy-related complications and development of type 2 diabetes and hypertension in women: an umbrella review.

BACKGROUND: Despite many systematic reviews and meta-analyses examining the associations of pregnancy complications with risk of type 2 diabetes mellitus (T2DM) and hypertension, previous umbrella reviews have only examined a single pregnancy complication. Here we have synthesised evidence from systematic reviews and meta-analyses on the associations of a wide range of pregnancy-related complications with risk of developing T2DM and hypertension. METHODS: Medline, Embase and Cochrane Database of Systematic Reviews were searched from inception until 26 September 2022 for systematic reviews and meta-analysis examining the association between pregnancy complications and risk of T2DM and hypertension. Screening of articles, data extraction and quality appraisal (AMSTAR2) were conducted independently by two reviewers using Covidence software. Data were extracted for studies that examined the risk of T2DM and hypertension in pregnant women with the pregnancy complication compared to pregnant women without the pregnancy complication. Summary estimates of each review were presented using tables, forest plots and narrative synthesis and reported following Preferred Reporting Items for Overviews of Reviews (PRIOR) guidelines. RESULTS: Ten systematic reviews were included. Two pregnancy complications were identified. Gestational diabetes mellitus (GDM): One review showed GDM was associated with a 10-fold higher risk of T2DM at least 1 year after pregnancy (relative risk (RR) 9.51 (95% confidence interval (CI) 7.14 to 12.67) and although the association differed by ethnicity (white: RR 16.28 (95% CI 15.01 to 17.66), non-white: RR 10.38 (95% CI 4.61 to 23.39), mixed: RR 8.31 (95% CI 5.44 to 12.69)), the between subgroups difference were not statistically significant at 5% significance level. Another review showed GDM was associated with higher mean blood pressure at least 3 months postpartum (mean difference in systolic blood pressure: 2.57 (95% CI 1.74 to 3.40) mmHg and mean difference in diastolic blood pressure: 1.89 (95% CI 1.32 to 2.46) mmHg). Hypertensive disorders of pregnancy (HDP): Three reviews showed women with a history of HDP were 3 to 6 times more likely to develop hypertension at least 6 weeks after pregnancy compared to women without HDP (meta-analysis with largest number of studies: odds ratio (OR) 4.33 (3.51 to 5.33)) and one review reported a higher rate of T2DM after HDP (hazard ratio (HR) 2.24 (1.95 to 2.58)) at least a year after pregnancy. One of the three reviews and five other reviews reported women with a history of preeclampsia were 3 to 7 times more likely to develop hypertension at least 6 weeks postpartum (meta-analysis with the largest number of studies: OR 3.90 (3.16 to 4.82) with one of these reviews reporting the association was greatest in women from Asia (Asia: OR 7.54 (95% CI 2.49 to 22.81), Europe: OR 2.19 (95% CI 0.30 to 16.02), North and South America: OR 3.32 (95% CI 1.26 to 8.74)). CONCLUSIONS: GDM and HDP are associated with a greater risk of developing T2DM and hypertension. Common confounders adjusted for across the included studies in the reviews were maternal age, body mass index (BMI), socioeconomic status, smoking status, pre-pregnancy and current BMI, parity, family history of T2DM or cardiovascular disease, ethnicity, and time of delivery. Further research is needed to evaluate the value of embedding these pregnancy complications as part of assessment for future risk of T2DM and chronic hypertension.

Discontinuation of anti-tumour necrosis factor alpha treatment owing to blood test abnormalities, and cost-effectiveness of alternate blood monitoring strategies.

BACKGROUND: There is no evidence base to support the use of 6-monthly monitoring blood tests for the early detection of liver, blood and renal toxicity during established anti-tumour necrosis factor alpha (TNFα) treatment. OBJECTIVES: To evaluate the incidence and risk factors of anti-TNFα treatment cessation owing to liver, blood and renal side-effects, and to estimate the cost-effectiveness of alternate intervals between monitoring blood tests. METHODS: A secondary care-based retrospective cohort study was performed. Data from the British Association of Dermatologists Biologic and Immunomodulators Register (BADBIR) were used. Patients with at least moderate psoriasis prescribed their first anti-TNFα treatment were included. Treatment discontinuation due to a monitoring blood test abnormality was the primary outcome. Patients were followed-up from start of treatment to the outcome of interest, drug discontinuation, death, 31 July 2021 or up to 5 years, whichever came first. The incidence rate (IR) and 95% confidence intervals (CIs) of anti-TNFα discontinuation with monitoring blood test abnormality was calculated. Multivariate Cox regression was used to examine the association between risk factors and outcome. A mathematical model evaluated costs and quality-adjusted life years (QALYs) associated with increasing the length of time between monitoring blood tests during anti-TNFα treatment. RESULTS: The cohort included 8819 participants [3710 (42.1%) female, mean (SD) age 44.76 (13.20) years] that contributed 25 058 person-years (PY) of follow-up and experienced 125 treatment discontinuations owing to a monitoring blood test abnormality at an IR of 5.85 (95% CI 4.91-6.97)/1000 PY. Of these, 64 and 61 discontinuations occurred within the first year and after the first year of treatment start, at IRs of 8.62 (95% CI 6.74-11.01) and 3.44 (95% CI 2.67-4.42)/1000 PY, respectively. Increasing age (in years), diabetes and liver disease were associated with anti-TNFα discontinuation after a monitoring blood test abnormality [adjusted hazard ratios of 1.02 (95% CI 1.01-1.04), 1.68 (95% CI 1.00-2.81) and 2.27 (95% CI 1.26-4.07), respectively]. Assuming a threshold of £20 000 per QALY gained, no monitoring was most cost-effective, but all extended periods were cost-effective vs. 3- or 6-monthly monitoring. CONCLUSIONS: Anti-TNFα drugs were uncommonly discontinued owing to abnormal monitoring blood tests after the first year of treatment. Extending the duration between monitoring blood tests was cost-effective. Our results produce evidence for specialist society guidance to reduce patient monitoring burden and healthcare costs.

Calibration plots for multistate risk predictions models.

INTRODUCTION: There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation. METHODS: We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. RESULTS: The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability. CONCLUSIONS: We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.

Development and external validation of the eFalls tool: a multivariable prediction model for the risk of ED attendance or hospitalisation with a fall or fracture in older adults.

BACKGROUND: Falls are common in older adults and can devastate personal independence through injury such as fracture and fear of future falls. Methods to identify people for falls prevention interventions are currently limited, with high risks of bias in published prediction models. We have developed and externally validated the eFalls prediction model using routinely collected primary care electronic health records (EHR) to predict risk of emergency department attendance/hospitalisation with fall or fracture within 1 year. METHODS: Data comprised two independent, retrospective cohorts of adults aged ≥65 years: the population of Wales, from the Secure Anonymised Information Linkage Databank (model development); the population of Bradford and Airedale, England, from Connected Bradford (external validation). Predictors included electronic frailty index components, supplemented with variables informed by literature reviews and clinical expertise. Fall/fracture risk was modelled using multivariable logistic regression with a Least Absolute Shrinkage and Selection Operator penalty. Predictive performance was assessed through calibration, discrimination and clinical utility. Apparent, internal-external cross-validation and external validation performance were assessed across general practices and in clinically relevant subgroups. RESULTS: The model's discrimination performance (c-statistic) was 0.72 (95% confidence interval, CI: 0.68 to 0.76) on internal-external cross-validation and 0.82 (95% CI: 0.80 to 0.83) on external validation. Calibration was variable across practices, with some over-prediction in the validation population (calibration-in-the-large, -0.87; 95% CI: -0.96 to -0.78). Clinical utility on external validation was improved after recalibration. CONCLUSION: The eFalls prediction model shows good performance and could support proactive stratification for falls prevention services if appropriately embedded into primary care EHR systems.

Clinical prediction models and the multiverse of madness.

BACKGROUND: Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice. MAIN BODY: We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice. CONCLUSIONS: Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time.

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population.

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Developing prediction models to estimate the risk of two survival outcomes both occurring: A comparison of techniques.

INTRODUCTION: This study considers the prediction of the time until two survival outcomes have both occurred. We compared a variety of analytical methods motivated by a typical clinical problem of multimorbidity prognosis. METHODS: We considered five methods: product (multiply marginal risks), dual-outcome (directly model the time until both events occur), multistate models (msm), and a range of copula and frailty models. We assessed calibration and discrimination under a variety of simulated data scenarios, varying outcome prevalence, and the amount of residual correlation. The simulation focused on model misspecification and statistical power. Using data from the Clinical Practice Research Datalink, we compared model performance when predicting the risk of cardiovascular disease and type 2 diabetes both occurring. RESULTS: Discrimination was similar for all methods. The product method was poorly calibrated in the presence of residual correlation. The msm and dual-outcome models were the most robust to model misspecification but suffered a drop in performance at small sample sizes due to overfitting, which the copula and frailty model were less susceptible to. The copula and frailty model's performance were highly dependent on the underlying data structure. In the clinical example, the product method was poorly calibrated when adjusting for 8 major cardiovascular risk factors. DISCUSSION: We recommend the dual-outcome method for predicting the risk of two survival outcomes both occurring. It was the most robust to model misspecification, although was also the most prone to overfitting. The clinical example motivates the use of the methods considered in this study.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review.

BACKGROUND: Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome. METHODS: We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size. RESULTS: A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84). CONCLUSIONS: Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Stability of clinical prediction models developed using statistical or machine learning methods.

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

Global Disease Burden of Respiratory Syncytial Virus in Preterm Children in 2019: A Systematic Review and Individual Participant Data Meta-Analysis Protocol.

Existing guidelines on respiratory syncytial virus (RSV) prophylaxis differ greatly by gestational age (GA) and other underlying risk factors, highlighting the data gaps in RSV disease burden among preterm infants. We will conduct a systematic review and individual participant data (IPD) meta-analysis of RSV global disease burden among preterm-born children. Three databases, Medline, Embase, and Global Health, will be searched for relevant studies on RSV disease burden for 2019 or before in preterm-born children published between 1 January 1995 and 31 December 2021. IPD will be sought by contacting the investigators identified from published literature and from existing collaboration networks. One-stage and 2-stage random-effects meta-analyses will be used to combine information from IPD and non-IPD studies to produce summary RSV burden estimates of incidence rate, hospital admission rate, and in-hospital case fatality ratio. The framework will be extended to examine subgroup(s) with the most substantial RSV disease burden.

A Guide to Estimating the Reference Range From a Meta-Analysis Using Aggregate or Individual Participant Data.

Clinicians frequently must decide whether a patient's measurement reflects that of a healthy "normal" individual. Thus, the reference range is defined as the interval in which some proportion (frequently 95%) of measurements from a healthy population is expected to fall. One can estimate it from a single study or preferably from a meta-analysis of multiple studies to increase generalizability. This range differs from the confidence interval for the pooled mean and the prediction interval for a new study mean in a meta-analysis, which do not capture natural variation across healthy individuals. Methods for estimating the reference range from a meta-analysis of aggregate data that incorporates both within- and between-study variations were recently proposed. In this guide, we present 3 approaches for estimating the reference range: one frequentist, one Bayesian, and one empirical. Each method can be applied to either aggregate or individual-participant data meta-analysis, with the latter being the gold standard when available. We illustrate the application of these approaches to data from a previously published individual-participant data meta-analysis of studies measuring liver stiffness by transient elastography in healthy individuals between 2006 and 2016.

Predicting relapse or recurrence of depression: systematic review of prognostic models.

BACKGROUND: Relapse and recurrence of depression are common, contributing to the overall burden of depression globally. Accurate prediction of relapse or recurrence while patients are well would allow the identification of high-risk individuals and may effectively guide the allocation of interventions to prevent relapse and recurrence. AIMS: To review prognostic models developed to predict the risk of relapse, recurrence, sustained remission, or recovery in adults with remitted major depressive disorder. METHOD: We searched the Cochrane Library (current issue); Ovid MEDLINE (1946 onwards); Ovid Embase (1980 onwards); Ovid PsycINFO (1806 onwards); and Web of Science (1900 onwards) up to May 2021. We included development and external validation studies of multivariable prognostic models. We assessed risk of bias of included studies using the Prediction model risk of bias assessment tool (PROBAST). RESULTS: We identified 12 eligible prognostic model studies (11 unique prognostic models): 8 model development-only studies, 3 model development and external validation studies and 1 external validation-only study. Multiple estimates of performance measures were not available and meta-analysis was therefore not necessary. Eleven out of the 12 included studies were assessed as being at high overall risk of bias and none examined clinical utility. CONCLUSIONS: Due to high risk of bias of the included studies, poor predictive performance and limited external validation of the models identified, presently available clinical prediction models for relapse and recurrence of depression are not yet sufficiently developed for deploying in clinical settings. There is a need for improved prognosis research in this clinical area and future studies should conform to best practice methodological and reporting guidelines.

Calculating the power to examine treatment-covariate interactions when planning an individual participant data meta-analysis of randomized trials with a binary outcome.

Before embarking on an individual participant data meta-analysis (IPDMA) project, researchers and funders need assurance it is worth their time and cost. This should include consideration of how many studies are promising their IPD and, given the characteristics of these studies, the power of an IPDMA including them. Here, we show how to estimate the power of a planned IPDMA of randomized trials to examine treatment-covariate interactions at the participant level (ie, treatment effect modifiers). We focus on a binary outcome with binary or continuous covariates, and propose a three-step approach, which assumes the true interaction size is common to all trials. In step one, the user must specify a minimally important interaction size and, for each trial separately (eg, as obtained from trial publications), the following aggregate data: the number of participants and events in control and treatment groups, the mean and SD for each continuous covariate, and the proportion of participants in each category for each binary covariate. This allows the variance of the interaction estimate to be calculated for each trial, using an analytic solution for Fisher's information matrix from a logistic regression model. Step 2 calculates the variance of the summary interaction estimate from the planned IPDMA (equal to the inverse of the sum of the inverse trial variances from step 1), and step 3 calculates the corresponding power based on a two-sided Wald test. Stata and R code are provided, and two examples given for illustration. Extension to allow for between-study heterogeneity is also considered.

Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome.

Previous articles in Statistics in Medicine describe how to calculate the sample size required for external validation of prediction models with continuous and binary outcomes. The minimum sample size criteria aim to ensure precise estimation of key measures of a model's predictive performance, including measures of calibration, discrimination, and net benefit. Here, we extend the sample size guidance to prediction models with a time-to-event (survival) outcome, to cover external validation in datasets containing censoring. A simulation-based framework is proposed, which calculates the sample size required to target a particular confidence interval width for the calibration slope measuring the agreement between predicted risks (from the model) and observed risks (derived using pseudo-observations to account for censoring) on the log cumulative hazard scale. Precise estimation of calibration curves, discrimination, and net-benefit can also be checked in this framework. The process requires assumptions about the validation population in terms of the (i) distribution of the model's linear predictor and (ii) event and censoring distributions. Existing information can inform this; in particular, the linear predictor distribution can be approximated using the C-index or Royston's D statistic from the model development article, together with the overall event risk. We demonstrate how the approach can be used to calculate the sample size required to validate a prediction model for recurrent venous thromboembolism. Ideally the sample size should ensure precise calibration across the entire range of predicted risks, but must at least ensure adequate precision in regions important for clinical decision-making. Stata and R code are provided.

Bayesian network meta-analysis methods for combining individual participant data and aggregate data from single arm trials and randomised controlled trials.

BACKGROUND: Increasingly in network meta-analysis (NMA), there is a need to incorporate non-randomised evidence to estimate relative treatment effects, and in particular in cases with limited randomised evidence, sometimes resulting in disconnected networks of treatments. When combining different sources of data, complex NMA methods are required to address issues associated with participant selection bias, incorporating single-arm trials (SATs), and synthesising a mixture of individual participant data (IPD) and aggregate data (AD). We develop NMA methods which synthesise data from SATs and randomised controlled trials (RCTs), using a mixture of IPD and AD, for a dichotomous outcome. METHODS: We propose methods under both contrast-based (CB) and arm-based (AB) parametrisations, and extend the methods to allow for both within- and across-trial adjustments for covariate effects. To illustrate the methods, we use an applied example investigating the effectiveness of biologic disease-modifying anti-rheumatic drugs for rheumatoid arthritis (RA). We applied the methods to a dataset obtained from a literature review consisting of 14 RCTs and an artificial dataset consisting of IPD from two SATs and AD from 12 RCTs, where the artificial dataset was created by removing the control arms from the only two trials assessing tocilizumab in the original dataset. RESULTS: Without adjustment for covariates, the CB method with independent baseline response parameters (CBunadjInd) underestimated the effectiveness of tocilizumab when applied to the artificial dataset compared to the original dataset, albeit with significant overlap in posterior distributions for treatment effect parameters. The CB method with exchangeable baseline response parameters produced effectiveness estimates in agreement with CBunadjInd, when the predicted baseline response estimates were similar to the observed baseline response. After adjustment for RA duration, there was a reduction in across-trial heterogeneity in baseline response but little change in treatment effect estimates. CONCLUSIONS: Our findings suggest incorporating SATs in NMA may be useful in some situations where a treatment is disconnected from a network of comparator treatments, due to a lack of comparative evidence, to estimate relative treatment effects. The reliability of effect estimates based on data from SATs may depend on adjustment for covariate effects, although further research is required to understand this in more detail.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review.

BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review.

BACKGROUND: While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. METHODS: We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item. RESULTS: Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). CONCLUSION: Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste. SYSTEMATIC REVIEW REGISTRATION: PROSPERO, CRD42019161764.

Development and validation of a risk prediction model of preterm birth for women with preterm labour symptoms (the QUIDS study): A prospective cohort study and individual participant data meta-analysis.

BACKGROUND: Timely interventions in women presenting with preterm labour can substantially improve health outcomes for preterm babies. However, establishing such a diagnosis is very challenging, as signs and symptoms of preterm labour are common and can be nonspecific. We aimed to develop and externally validate a risk prediction model using concentration of vaginal fluid fetal fibronectin (quantitative fFN), in combination with clinical risk factors, for the prediction of spontaneous preterm birth and assessed its cost-effectiveness. METHODS AND FINDINGS: Pregnant women included in the analyses were 22+0 to 34+6 weeks gestation with signs and symptoms of preterm labour. The primary outcome was spontaneous preterm birth within 7 days of quantitative fFN test. The risk prediction model was developed and internally validated in an individual participant data (IPD) meta-analysis of 5 European prospective cohort studies (2009 to 2016; 1,783 women; mean age 29.7 years; median BMI 24.8 kg/m2; 67.6% White; 11.7% smokers; 51.8% nulliparous; 10.4% with multiple pregnancy; 139 [7.8%] with spontaneous preterm birth within 7 days). The model was then externally validated in a prospective cohort study in 26 United Kingdom centres (2016 to 2018; 2,924 women; mean age 28.2 years; median BMI 25.4 kg/m2; 88.2% White; 21% smokers; 35.2% nulliparous; 3.5% with multiple pregnancy; 85 [2.9%] with spontaneous preterm birth within 7 days). The developed risk prediction model for spontaneous preterm birth within 7 days included quantitative fFN, current smoking, not White ethnicity, nulliparity, and multiple pregnancy. After internal validation, the optimism adjusted area under the curve was 0.89 (95% CI 0.86 to 0.92), and the optimism adjusted Nagelkerke R2 was 35% (95% CI 33% to 37%). On external validation in the prospective UK cohort population, the area under the curve was 0.89 (95% CI 0.84 to 0.94), and Nagelkerke R2 of 36% (95% CI: 34% to 38%). Recalibration of the model's intercept was required to ensure overall calibration-in-the-large. A calibration curve suggested close agreement between predicted and observed risks in the range of predictions 0% to 10%, but some miscalibration (underprediction) at higher risks (slope 1.24 (95% CI 1.23 to 1.26)). Despite any miscalibration, the net benefit of the model was higher than "treat all" or "treat none" strategies for thresholds up to about 15% risk. The economic analysis found the prognostic model was cost effective, compared to using qualitative fFN, at a threshold for hospital admission and treatment of ≥2% risk of preterm birth within 7 days. Study limitations include the limited number of participants who are not White and levels of missing data for certain variables in the development dataset. CONCLUSIONS: In this study, we found that a risk prediction model including vaginal fFN concentration and clinical risk factors showed promising performance in the prediction of spontaneous preterm birth within 7 days of test and has potential to inform management decisions for women with threatened preterm labour. Further evaluation of the risk prediction model in clinical practice is required to determine whether the risk prediction model improves clinical outcomes if used in practice. TRIAL REGISTRATION: The study was approved by the West of Scotland Research Ethics Committee (16/WS/0068). The study was registered with ISRCTN Registry (ISRCTN 41598423) and NIHR Portfolio (CPMS: 31277).