Large-scale external validation and comparison of prognostic models: an application to chronic obstructive pulmonary disease.

BACKGROUND: External validations and comparisons of prognostic models or scores are a prerequisite for their use in routine clinical care but are lacking in most medical fields including chronic obstructive pulmonary disease (COPD). Our aim was to externally validate and concurrently compare prognostic scores for 3-year all-cause mortality in mostly multimorbid patients with COPD. METHODS: We relied on 24 cohort studies of the COPD Cohorts Collaborative International Assessment consortium, corresponding to primary, secondary, and tertiary care in Europe, the Americas, and Japan. These studies include globally 15,762 patients with COPD (1871 deaths and 42,203 person years of follow-up). We used network meta-analysis adapted to multiple score comparison (MSC), following a frequentist two-stage approach; thus, we were able to compare all scores in a single analytical framework accounting for correlations among scores within cohorts. We assessed transitivity, heterogeneity, and inconsistency and provided a performance ranking of the prognostic scores. RESULTS: Depending on data availability, between two and nine prognostic scores could be calculated for each cohort. The BODE score (body mass index, airflow obstruction, dyspnea, and exercise capacity) had a median area under the curve (AUC) of 0.679 [1st quartile-3rd quartile = 0.655-0.733] across cohorts. The ADO score (age, dyspnea, and airflow obstruction) showed the best performance for predicting mortality (difference AUCADO - AUCBODE = 0.015 [95% confidence interval (CI) = -0.002 to 0.032]; p = 0.08) followed by the updated BODE (AUCBODE updated - AUCBODE = 0.008 [95% CI = -0.005 to +0.022]; p = 0.23). The assumption of transitivity was not violated. Heterogeneity across direct comparisons was small, and we did not identify any local or global inconsistency. CONCLUSIONS: Our analyses showed best discriminatory performance for the ADO and updated BODE scores in patients with COPD. A limitation to be addressed in future studies is the extension of MSC network meta-analysis to measures of calibration. MSC network meta-analysis can be applied to prognostic scores in any medical field to identify the best scores, possibly paving the way for stratified medicine, public health, and research.

Correction to: Multiple Score Comparison: a network meta-analysis approach to comparison and external validation of prognostic scores.

CORRECTION: Following publication of the original article [1], a member of the writing group reported that his name is misspelt. The paper should appear in Pubmed under "Ter Riet G", bot as "Riet GT".

Multiple Score Comparison: a network meta-analysis approach to comparison and external validation of prognostic scores.

BACKGROUND: Prediction models and prognostic scores have been increasingly popular in both clinical practice and clinical research settings, for example to aid in risk-based decision making or control for confounding. In many medical fields, a large number of prognostic scores are available, but practitioners may find it difficult to choose between them due to lack of external validation as well as lack of comparisons between them. METHODS: Borrowing methodology from network meta-analysis, we describe an approach to Multiple Score Comparison meta-analysis (MSC) which permits concurrent external validation and comparisons of prognostic scores using individual patient data (IPD) arising from a large-scale international collaboration. We describe the challenges in adapting network meta-analysis to the MSC setting, for instance the need to explicitly include correlations between the scores on a cohort level, and how to deal with many multi-score studies. We propose first using IPD to make cohort-level aggregate discrimination or calibration scores, comparing all to a common comparator. Then, standard network meta-analysis techniques can be applied, taking care to consider correlation structures in cohorts with multiple scores. Transitivity, consistency and heterogeneity are also examined. RESULTS: We provide a clinical application, comparing prognostic scores for 3-year mortality in patients with chronic obstructive pulmonary disease using data from a large-scale collaborative initiative. We focus on the discriminative properties of the prognostic scores. Our results show clear differences in performance, with ADO and eBODE showing higher discrimination with respect to mortality than other considered scores. The assumptions of transitivity and local and global consistency were not violated. Heterogeneity was small. CONCLUSIONS: We applied a network meta-analytic methodology to externally validate and concurrently compare the prognostic properties of clinical scores. Our large-scale external validation indicates that the scores with the best discriminative properties to predict 3 year mortality in patients with COPD are ADO and eBODE.

Prediction models for exacerbations in patients with COPD.

Personalised medicine aims to tailor medical decisions to the individual patient. A possible approach is to stratify patients according to the risk of adverse outcomes such as exacerbations in chronic obstructive pulmonary disease (COPD). Risk-stratified approaches are particularly attractive for drugs like inhaled corticosteroids or phosphodiesterase-4 inhibitors that reduce exacerbations but are associated with harms. However, it is currently not clear which models are best to predict exacerbations in patients with COPD. Therefore, our aim was to identify and critically appraise studies on models that predict exacerbations in COPD patients. Out of 1382 studies, 25 studies with 27 prediction models were included. The prediction models showed great heterogeneity in terms of number and type of predictors, time horizon, statistical methods and measures of prediction model performance. Only two out of 25 studies validated the developed model, and only one out of 27 models provided estimates of individual exacerbation risk, only three out of 27 prediction models used high-quality statistical approaches for model development and evaluation. Overall, none of the existing models fulfilled the requirements for risk-stratified treatment to personalise COPD care. A more harmonised approach to develop and validate high- quality prediction models is needed to move personalised COPD medicine forward.

Annealed and mean-field formulations of disease dynamics on static and adaptive networks.

We use the annealed formulation of complex networks to study the dynamical behavior of disease spreading on both static and adaptive networked systems. This unifying approach relies on the annealed adjacency matrix, representing one network ensemble, and allows to solve the dynamical evolution of the whole network ensemble all at once. Our results accurately reproduce those obtained by extensive numerical simulations showing a large improvement with respect to the usual heterogeneous mean-field formulation. Moreover, by means of the annealed formulation we derive a new heterogeneous mean-field formulation that correctly reproduces the epidemic dynamics.

Dynamical organization towards consensus in the Axelrod model on complex networks.

We analyze the dynamics toward cultural consensus in the Axelrod model on scale-free networks. By looking at the microscopic dynamics of the model, we are able to show how culture traits spread across different cultural features. We compare the diffusion at the level of cultural features to the growth of cultural consensus at the global level, finding important differences between these two processes. In particular, we show that even when most of the cultural features have reached macroscopic consensus, there are still no signals of globalization. Finally, we analyze the topology of consensus clusters both for global culture and at the feature level of representation.