Measuring the performance of prediction models to personalize treatment choice.

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Bayesian models for aggregate and individual patient data component network meta-analysis.

Network meta-analysis can synthesize evidence from studies comparing multiple treatments for the same disease. Sometimes the treatments of a network are complex interventions, comprising several independent components in different combinations. A component network meta-analysis (CNMA) can be used to analyze such data and can in principle disentangle the individual effect of each component. However, components may interact with each other, either synergistically or antagonistically. Deciding which interactions, if any, to include in a CNMA model may be difficult, especially for large networks with many components. In this article, we present two Bayesian CNMA models that can be used to identify prominent interactions between components. Our models utilize Bayesian variable selection methods, namely the stochastic search variable selection and the Bayesian LASSO, and can benefit from the inclusion of prior information about important interactions. Moreover, we extend these models to combine data from studies providing aggregate information and studies providing individual patient data (IPD). We illustrate our models in practice using three real datasets, from studies in panic disorder, depression, and multiple myeloma. Finally, we describe methods for developing web-applications that can utilize results from an IPD-CNMA, to allow for personalized estimates of relative treatment effects given a patient's characteristics.

Comparing methods for estimating patient-specific treatment effects in individual patient data meta-analysis.

Meta-analysis of individual patient data (IPD) is increasingly used to synthesize data from multiple trials. IPD meta-analysis offers several advantages over meta-analyzing aggregate data, including the capacity to individualize treatment recommendations. Trials usually collect information on many patient characteristics. Some of these covariates may strongly interact with treatment (and thus be associated with treatment effect modification) while others may have little effect. It is currently unclear whether a systematic approach to the selection of treatment-covariate interactions in an IPD meta-analysis can lead to better estimates of patient-specific treatment effects. We aimed to answer this question by comparing in simulations the standard approach to IPD meta-analysis (no variable selection, all treatment-covariate interactions included in the model) with six alternative methods: stepwise regression, and five regression methods that perform shrinkage on treatment-covariate interactions, that is, least absolute shrinkage and selection operator (LASSO), ridge, adaptive LASSO, Bayesian LASSO, and stochastic search variable selection. Exploring a range of scenarios, we found that shrinkage methods performed well for both continuous and dichotomous outcomes, for a variety of settings. In most scenarios, these methods gave lower mean squared error of the patient-specific treatment effect as compared with the standard approach and stepwise regression. We illustrate the application of these methods in two datasets from cardiology and psychiatry. We recommend that future IPD meta-analysis that aim to estimate patient-specific treatment effects using multiple effect modifiers should use shrinkage methods, whereas stepwise regression should be avoided.

Comparison of revascularization strategies in patients with acute coronary syndrome and multivessel coronary disease: A systematic review and network meta-analysis.

BACKGROUND: Whether revascularization should be performed as multivessel intervention at the time of index procedure (MV-index), staged procedure (MV-staged), or culprit only intervention (COI) in patients with multivessel disease (MVD) presenting with acute coronary syndrome (ACS) is unclear. We performed a systematic review and network meta-analysis of randomized controlled trials to assess the optimal revascularization strategy in this patient population. METHODS: PubMed, Embase, and Cochrane Central databases were systematically searched to identify all relevant studies. The outcomes assessed were major cardiac adverse events (MACE), all-cause mortality, cardiovascular mortality, myocardial infarction (MI), and revascularization. A Bayesian random-effects network meta-analysis was used to calculate odds ratio (OR) with credible interval (CrI). RESULTS: Thirteen studies with 8,066 patients were included in the analysis. There was a decreased risk of MACE (MV-index vs. COI: OR, 0.35; 95% CrI, 0.23-0.55; MV-staged vs COI: OR, 0.52; 95% CrI, 0.31-0.81) and revascularization (MV-index vs. COI: OR, 0.27; 95% CrI, 0.15-0.49; MV-staged vs. COI: OR, 0.38; 95% CrI, 0.19-0.70) with MV-index intervention and MV-staged intervention compared with COI. However, MV-index intervention and not MV-staged intervention was associated with a decreased risk of MI (MV-index vs. COI: OR, 0.35; 95% CrI, 0.12-0.93; MV-staged vs. COI: OR, 0.65; 95% CrI, 0.24-1.59) compared with COI. CONCLUSIONS: Our analysis suggests that multivessel intervention either at index procedure or as staged intervention may be more efficacious compared to COI in patients with MVD presenting with ACS.

TB and HIV induced immunosenescence: where do vaccines play a role?

This paper tackles the complex interplay between Human Immunodeficiency virus (HIV-1) and Mycobacterium tuberculosis (M. tuberculosis) infections, particularly their contribution to immunosenescence, the age-related decline in immune function. Using the current literature, we discuss the immunological mechanisms behind TB and HIV-induced immunosenescence and critically evaluate the BCG (Bacillus Calmette-Guérin) vaccine's role. Both HIV-1 and M. tuberculosis demonstrably accelerate immunosenescence: M. tuberculosis through DNA modification and heightened inflammation, and HIV-1 through chronic immune activation and T cell production compromise. HIV-1 and M. tuberculosis co-infection further hastens immunosenescence by affecting T cell differentiation, underscoring the need for prevention and treatment. Furthermore, the use of the BCG tuberculosis vaccine is contraindicated in patients who are HIV positive and there is a lack of investigation regarding the use of this vaccine in patients who develop HIV co-infection with possible immunosenescence. As HIV does not currently have a vaccine, we focus our review more so on the BCG vaccine response as a result of immunosenescence. We found that there are overall limitations with the BCG vaccine, one of which is that it cannot necessarily prevent re-occurrence of infection due to effects of immunosenescence or protect the elderly due to this reason. Overall, there is conflicting evidence to show the vaccine's usage due to factors involving its production and administration. Further research into developing a vaccine for HIV and improving the BCG vaccine is warranted to expand scientific understanding for public health and beyond.

Developing prediction models when there are systematically missing predictors in individual patient data meta-analysis.

Clinical prediction models are widely used in modern clinical practice. Such models are often developed using individual patient data (IPD) from a single study, but often there are IPD available from multiple studies. This allows using meta-analytical methods for developing prediction models, increasing power and precision. Different studies, however, often measure different sets of predictors, which may result to systematically missing predictors, that is, when not all studies collect all predictors of interest. This situation poses challenges in model development. We hereby describe various approaches that can be used to develop prediction models for continuous outcomes in such situations. We compare four approaches: a "restrict predictors" approach, where the model is developed using only predictors measured in all studies; a multiple imputation approach that ignores study-level clustering; a multiple imputation approach that accounts for study-level clustering; and a new approach that develops a prediction model in each study separately using all predictors reported, and then synthesizes all predictions in a multi-study ensemble. We explore in simulations the performance of all approaches under various scenarios. We find that imputation methods and our new method outperform the restrict predictors approach. In several scenarios, our method outperformed imputation methods, especially for few studies, when predictor effects were small, and in case of large heterogeneity. We use a real dataset of 12 trials in psychotherapies for depression to illustrate all methods in practice, and we provide code in R.

Combining individual patient data from randomized and non-randomized studies to predict real-world effectiveness of interventions.

Meta-analysis of randomized controlled trials is generally considered the most reliable source of estimates of relative treatment effects. However, in the last few years, there has been interest in using non-randomized studies to complement evidence from randomized controlled trials. Several meta-analytical models have been proposed to this end. Such models mainly focussed on estimating the average relative effects of interventions. In real-life clinical practice, when deciding on how to treat a patient, it might be of great interest to have personalized predictions of absolute outcomes under several available treatment options. This paper describes a general framework for developing models that combine individual patient data from randomized controlled trials and non-randomized study when aiming to predict outcomes for a set of competing medical interventions applied in real-world clinical settings. We also discuss methods for measuring the models' performance to identify the optimal model to use in each setting. We focus on the case of continuous outcomes and illustrate our methods using a data set from rheumatoid arthritis, comprising patient-level data from three randomized controlled trials and two registries from Switzerland and Britain.

Personalized Prediction of Alzheimer's Disease and Its Treatment Effects by Donepezil: An Individual Participant Data Meta-Analysis of Eight Randomized Controlled Trials.

BACKGROUND: Patient characteristics may predict the progression of Alzheimer's disease (AD) and may moderate the effects of donepezil. OBJECTIVE: To build a personalized prediction model for patients with AD and to estimate patient-specific treatment effects of donepezil, using individual patient characteristics. METHODS: We systematically searched for all double-masked randomized controlled trials comparing oral donepezil and pill placebo in the treatment of AD and requested individual participant data through its developer, Eisai. The primary outcome was cognitive function at 24 weeks, measured with the Alzheimer's Disease Assessment Scale-cognitive component (ADAS-cog). We built a Bayesian meta-analytical prediction model for patients receiving placebo and we performed an individual patient data meta-analysis to estimate patient-level treatment effects. RESULTS: Eight studies with 3,156 participants were included. The Bayesian prediction model suggested that more severe cognitive and global function at baseline and younger age were associated with worse cognitive function at 24 weeks. The individual participant data meta-analysis showed that, on average, donepezil was superior to placebo in cognitive function (ADAS-cog scores, -3.2; 95% Credible Interval (CrI) -4.2 to -2.1). In addition, our results suggested that antipsychotic drug use at baseline might be associated with a lower effect of donepezil in ADAS-cog (2.0; 95% CrI, -0.02 to 4.3). CONCLUSION: Although our results suggested that donepezil is somewhat efficacious for cognitive function for most patients with AD, use of antipsychotic drugs may be associated with lower efficacy of the drug. Future research with larger sample sizes, more patient covariates, and longer treatment duration is needed.

The Kilim plot: A tool for visualizing network meta-analysis results for multiple outcomes.

Network meta-analysis (NMA) can be used to compare multiple competing treatments for the same disease. In practice, usually a range of outcomes is of interest. As the number of outcomes increases, summarizing results from multiple NMAs becomes a nontrivial task, especially for larger networks. Moreover, NMAs provide results in terms of relative effect measures that can be difficult to interpret and apply in every-day clinical practice, such as the odds ratios. In this article, we aim to facilitate the clinical decision-making process by proposing a new graphical tool, the Kilim plot, for presenting results from NMA on multiple outcomes. Our plot compactly summarizes results on all treatments and all outcomes; it provides information regarding the strength of the statistical evidence of treatment effects, while it illustrates absolute, rather than relative, effects of interventions. Moreover, it can be easily modified to include considerations regarding clinically important effects. To showcase our method, we use data from a network of studies in antidepressants. All analyses are performed in R and we provide the source code needed to produce the Kilim plot, as well as an interactive web application.

Can personalized treatment prediction improve the outcomes, compared with the group average approach, in a randomized trial? Developing and validating a multivariable prediction model in a pragmatic megatrial of acute treatment for major depression.

BACKGROUND: Clinical trials have traditionally been analysed at the aggregate level, assuming that the group average would be applicable to all eligible and similar patients. We re-analyzed a mega-trial of antidepressant therapy for major depression to explore whether a multivariable prediction model may lead to different treatment recommendations for individual participants. METHODS: The trial compared the second-line treatment strategies of continuing sertraline, combining it with mirtazapine or switching to mirtazapine after initial failure to remit on sertraline among 1,544 patients with major depression. The outcome was the Personal Health Questionnaire-9 (PHQ-9) at week 9: the original analyses showed that both combining and switching resulted in greater reduction in PHQ-9 by 1.0 point than continuing. We considered several models of penalized regression or machine learning. RESULTS: Models using support vector machines (SVMs) provided the best performance. Using SVMs, continuing sertraline was predicted to be the best treatment for 123 patients, combining for 696 patients, and switching for 725 patients. In the last two subgroups, both combining and switching were equally superior to continuing by 1.2 to 1.4 points, resulting in the same treatment recommendations as with the original aggregate data level analyses; in the first subgroup, however, switching was substantively inferior to combining (-3.1, 95%CI: -5.4 to -0.5). LIMITATIONS: Stronger predictors are needed to make more precise predictions. CONCLUSIONS: The multivariable prediction models led to improved recommendations for a minority of participants than the group average approach in a megatrial.

Distinguishing modes of cell death using the ImageStream multispectral imaging flow cytometer.

BACKGROUND: Here we demonstrate the ability of the ImageStream 100 Multispectral Imaging Cytometer to discriminate between live, necrotic, and early and late apoptotic cells, using unique combinations of photometric and morphometric features. METHODS: Live, necrotic, and early and late apoptotic cells were prepared and analyzed by immunofluorescence microscopy, conventional flow cytometry, and imaging flow cytometry, both as single populations and as a heterogeneous mixture of cells. RESULTS: Live (annexin V(-), 7-AAD(-)) and early apoptotic (annexin V(+), 7-AAD(-)) cells were readily identifiable using either conventional or ImageStream based flow cytometric methods. However, inspection of multispectral images of cells demonstrated that the annexin V(+), 7-AAD(+) population contained both necrotic and late-stage apoptotic cells. Although these cells could not be distinguished using conventional flow cytometric techniques, they were separable using unique combinations of photometric and morphometric measures available using ImageStream technologies. CONCLUSIONS: Using multispectral imagery, morphologically distinct cell populations can be distinguished using features not available with conventional flow cytometers. In particular, the ability to couple morphometric with photometric measures makes it possible to distinguish live cells from cells in the early phases of apoptosis, as well as late apoptotic cells from necrotic cells.