Prognostic models for predicting clinical disease progression, worsening and activity in people with multiple sclerosis.

BACKGROUND: Multiple sclerosis (MS) is a chronic inflammatory disease of the central nervous system that affects millions of people worldwide. The disease course varies greatly across individuals and many disease-modifying treatments with different safety and efficacy profiles have been developed recently. Prognostic models evaluated and shown to be valid in different settings have the potential to support people with MS and their physicians during the decision-making process for treatment or disease/life management, allow stratified and more precise interpretation of interventional trials, and provide insights into disease mechanisms. Many researchers have turned to prognostic models to help predict clinical outcomes in people with MS; however, to our knowledge, no widely accepted prognostic model for MS is being used in clinical practice yet. OBJECTIVES: To identify and summarise multivariable prognostic models, and their validation studies for quantifying the risk of clinical disease progression, worsening, and activity in adults with MS. SEARCH METHODS: We searched MEDLINE, Embase, and the Cochrane Database of Systematic Reviews from January 1996 until July 2021. We also screened the reference lists of included studies and relevant reviews, and references citing the included studies. SELECTION CRITERIA: We included all statistically developed multivariable prognostic models aiming to predict clinical disease progression, worsening, and activity, as measured by disability, relapse, conversion to definite MS, conversion to progressive MS, or a composite of these in adult individuals with MS. We also included any studies evaluating the performance of (i.e. validating) these models. There were no restrictions based on language, data source, timing of prognostication, or timing of outcome. DATA COLLECTION AND ANALYSIS: Pairs of review authors independently screened titles/abstracts and full texts, extracted data using a piloted form based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS), assessed risk of bias using the Prediction Model Risk Of Bias Assessment Tool (PROBAST), and assessed reporting deficiencies based on the checklist items in Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD). The characteristics of the included models and their validations are described narratively. We planned to meta-analyse the discrimination and calibration of models with at least three external validations outside the model development study but no model met this criterion. We summarised between-study heterogeneity narratively but again could not perform the planned meta-regression. MAIN RESULTS: We included 57 studies, from which we identified 75 model developments, 15 external validations corresponding to only 12 (16%) of the models, and six author-reported validations. Only two models were externally validated multiple times. None of the identified external validations were performed by researchers independent of those that developed the model. The outcome was related to disease progression in 39 (41%), relapses in 8 (8%), conversion to definite MS in 17 (18%), and conversion to progressive MS in 27 (28%) of the 96 models or validations. The disease and treatment-related characteristics of included participants, and definitions of considered predictors and outcome, were highly heterogeneous amongst the studies. Based on the publication year, we observed an increase in the percent of participants on treatment, diversification of the diagnostic criteria used, an increase in consideration of biomarkers or treatment as predictors, and increased use of machine learning methods over time. Usability and reproducibility All identified models contained at least one predictor requiring the skills of a medical specialist for measurement or assessment. Most of the models (44; 59%) contained predictors that require specialist equipment likely to be absent from primary care or standard hospital settings. Over half (52%) of the developed models were not accompanied by model coefficients, tools, or instructions, which hinders their application, independent validation or reproduction. The data used in model developments were made publicly available or reported to be available on request only in a few studies (two and six, respectively). Risk of bias We rated all but one of the model developments or validations as having high overall risk of bias. The main reason for this was the statistical methods used for the development or evaluation of prognostic models; we rated all but two of the included model developments or validations as having high risk of bias in the analysis domain. None of the model developments that were externally validated or these models' external validations had low risk of bias. There were concerns related to applicability of the models to our research question in over one-third (38%) of the models or their validations. Reporting deficiencies Reporting was poor overall and there was no observable increase in the quality of reporting over time. The items that were unclearly reported or not reported at all for most of the included models or validations were related to sample size justification, blinding of outcome assessors, details of the full model or how to obtain predictions from it, amount of missing data, and treatments received by the participants. Reporting of preferred model performance measures of discrimination and calibration was suboptimal. AUTHORS' CONCLUSIONS: The current evidence is not sufficient for recommending the use of any of the published prognostic prediction models for people with MS in clinical routine today due to lack of independent external validations. The MS prognostic research community should adhere to the current reporting and methodological guidelines and conduct many more state-of-the-art external validation studies for the existing or newly developed models.

Association between post-operative delirium and use of volatile anesthetics in the elderly: A real-world big data approach.

STUDY OBJECTIVE: Early post-operative delirium is a common perioperative complication in the post anesthesia care unit. To date it is unknown if a specific anesthetic regime can affect the incidence of delirium after surgery. Our objective was to examine the effect of volatile anesthetics on post-operative delirium. DESIGN: Single Center Observational Study. SETTING: Post Anesthesia Care Units at a German tertiary medical center. PATIENTS: 30,075 patients receiving general anesthesia for surgery. MEASUREMENTS: Delirium was assessed with the Nursing Delirium Screening Scale at the end of the recovery period. Subgroup-specific effects of volatile anesthetics on post-operative delirium were estimated using generalized-linear-model trees with inverse probability of treatment weighting. We further assessed the age-specific effect of volatiles using logistic regression models. MAIN RESULTS: Out of 30,075 records, 956 patients (3.2%) developed delirium in the post anesthesia care unit. On average, patients who developed delirium were older than patients without delirium. We found volatile anesthetics to increase the risk (Odds exp. (B) for delirium in the elderly 1.8-fold compared to total intravenous anesthesia. Odds increases with unplanned surgery 3.0-fold. In the very old (87 years or older), the increase in delirium is 6.2-fold. This result was confirmed with internal validation and in a logistic regression model. CONCLUSIONS: Our exploratory study indicates that early postoperative delirium is associated with the use of volatile anesthetics especially in the sub-cohort of patients aged 75 years and above. Further studies should include both volatile and intravenous anesthetics to find the ideal anesthetic in elderly patients.

Correction: A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses.

[This corrects the article DOI: 10.1371/journal.pone.0251194.].

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses.

Computational reproducibility is a corner stone for sound and credible research. Especially in complex statistical analyses-such as the analysis of longitudinal data-reproducing results is far from simple, especially if no source code is available. In this work we aimed to reproduce analyses of longitudinal data of 11 articles published in PLOS ONE. Inclusion criteria were the availability of data and author consent. We investigated the types of methods and software used and whether we were able to reproduce the data analysis using open source software. Most articles provided overview tables and simple visualisations. Generalised Estimating Equations (GEEs) were the most popular statistical models among the selected articles. Only one article used open source software and only one published part of the analysis code. Replication was difficult in most cases and required reverse engineering of results or contacting the authors. For three articles we were not able to reproduce the results, for another two only parts of them. For all but two articles we had to contact the authors to be able to reproduce the results. Our main learning is that reproducing papers is difficult if no code is supplied and leads to a high burden for those conducting the reproductions. Open data policies in journals are good, but to truly boost reproducibility we suggest adding open code policies.

An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action.

Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability. Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin.

Survival forests under test: Impact of the proportional hazards assumption on prognostic and predictive forests for amyotrophic lateral sclerosis survival.

We investigate the effect of the proportional hazards assumption on prognostic and predictive models of the survival time of patients suffering from amyotrophic lateral sclerosis. We theoretically compare the underlying model formulations of several variants of survival forests and implementations thereof, including random forests for survival, conditional inference forests, Ranger, and survival forests with L1 splitting, with two novel variants, namely distributional and transformation survival forests. Theoretical considerations explain the low power of log-rank-based splitting in detecting patterns in non-proportional hazards situations in survival trees and corresponding forests. This limitation can potentially be overcome by the alternative split procedures suggested herein. We empirically investigated this effect using simulation experiments and a re-analysis of the Pooled Resource Open-Access ALS Clinical Trials database of amyotrophic lateral sclerosis survival, giving special emphasis to both prognostic and predictive models.

Estimating patient-specific treatment advantages in the 'Treatment for Adolescents with Depression Study'.

The 'Treatment for Adolescents with Depression Study' (TADS, ClinicalTrials.gov, identifier: NCT00006286) was a cornerstone, randomized controlled trial evaluating the effectiveness of standard treatment options for major depression in adolescents. Whereas previous TADS analyses examined primarily effect modifications of treatment-placebo differences by various patient characteristics, less is known about the modification of inter-treatment differences, and hence, patient characteristics that might guide treatment selection. We sought to fill this gap by estimating patient-specific inter-treatment differences as a function of patients' baseline characteristics. We did so by applying the 'model-based random forest', a recently-introduced machine learning-based method for evaluating effect heterogeneity that allows for the estimation of patient-specific treatment effects as a function of arbitrary baseline characteristics. Treatment conditions were cognitive-behavioural therapy (CBT) alone, fluoxetine (FLX) alone, and the combination of CBT and fluoxetine (COMB). All inter-treatment differences (CBT vs. FLX; CBT vs. COMB; FLX vs. COMB) were evaluated across 23 potential effect modifiers extracted from previous studies. Overall, FLX was superior to CBT, while COMB was superior to both CBT and FLX. Evidence for effect heterogeneity was found for the CBT-FLX difference and the FLX-COMB difference, but not for the CBT-COMB difference. Baseline depression severity modified the CBT-FLX difference; whereas baseline depression severity, patients' treatment expectations, and childhood trauma modified the FLX-COMB difference. All modifications were quantitative rather than qualitative, however, meaning that the differences varied only in magnitude, but not direction. These findings imply that combining CBT with fluoxetine may be superior to either therapy used alone across a broad range of patients.

Subgroup identification in clinical trials: an overview of available methods and their implementations with R.

Randomized controlled trials (RCTs) usually enroll heterogeneous study population, and thus it is interesting to identify subgroups of patients for whom the treatment may be beneficial or harmful. A variety of methods have been developed to do such kind of post hoc analyses. Conventional generalized linear model is able to include prognostic variables as a main effect and predictive variables in an interaction with treatment variable. A statistically significant and large interaction effect usually indicates potential subgroups that may have different responses to the treatment. However, the conventional regression method requires to specify the interaction term, which requires knowledge of predictive variables or becomes infeasible when there is a large number of feature variables. The Least Absolute Shrinkage and Selection Operator (LASSO) method does variable selection by shrinking less clear effects (including interaction effects) to zero and in this way selects only certain variables and interactions for the model. There are many tree-based methods for subgroup identification. For example, model-based recursive partitioning incorporates parametric models such as generalized linear models into trees. The model incorporated is usually a simple model with only the treatment as covariate. Predictive and prognostic variables are found and incorporated automatically via the tree. The present article gives an overview of these methods and explains how to perform them using the free software environment for statistical computing R (version 3.3.2). A simulated dataset is employed for illustrating the performance of these methods.

Subgroup identification in dose-finding trials via model-based recursive partitioning.

An important task in early-phase drug development is to identify patients, which respond better or worse to an experimental treatment. While a variety of different subgroup identification methods have been developed for the situation of randomized clinical trials that study an experimental treatment and control, much less work has been done in the situation when patients are randomized to different dose groups. In this article, we propose new strategies to perform subgroup analyses in dose-finding trials and discuss the challenges, which arise in this new setting. We consider model-based recursive partitioning, which has recently been applied to subgroup identification in 2-arm trials, as a promising method to tackle these challenges and assess its viability using a real trial example and simulations. Our results show that model-based recursive partitioning can be used to identify subgroups of patients with different dose-response curves and improves estimation of treatment effects and minimum effective doses compared to models ignoring possible subgroups, when heterogeneity among patients is present.

Individual treatment effect prediction for amyotrophic lateral sclerosis patients.

A treatment for a complicated disease might be helpful for some but not all patients, which makes predicting the treatment effect for new patients important yet challenging. Here we develop a method for predicting the treatment effect based on patient characteristics and use it for predicting the effect of the only drug (Riluzole) approved for treating amyotrophic lateral sclerosis. Our proposed method of model-based random forests detects similarities in the treatment effect among patients and on this basis computes personalised models for new patients. The entire procedure focuses on a base model, which usually contains the treatment indicator as a single covariate and takes the survival time or a health or treatment success measurement as primary outcome. This base model is used both to grow the model-based trees within the forest, in which the patient characteristics that interact with the treatment are split variables, and to compute the personalised models, in which the similarity measurements enter as weights. We applied the personalised models using data from several clinical trials for amyotrophic lateral sclerosis from the Pooled Resource Open-Access Clinical Trials database. Our results indicate that some amyotrophic lateral sclerosis patients benefit more from the drug Riluzole than others. Our method allows gradually shifting from stratified medicine to personalised medicine and can also be used in assessing the treatment effect for other diseases studied in a clinical trial.

Model-Based Recursive Partitioning for Subgroup Analyses.

The identification of patient subgroups with differential treatment effects is the first step towards individualised treatments. A current draft guideline by the EMA discusses potentials and problems in subgroup analyses and formulated challenges to the development of appropriate statistical procedures for the data-driven identification of patient subgroups. We introduce model-based recursive partitioning as a procedure for the automated detection of patient subgroups that are identifiable by predictive factors. The method starts with a model for the overall treatment effect as defined for the primary analysis in the study protocol and uses measures for detecting parameter instabilities in this treatment effect. The procedure produces a segmented model with differential treatment parameters corresponding to each patient subgroup. The subgroups are linked to predictive factors by means of a decision tree. The method is applied to the search for subgroups of patients suffering from amyotrophic lateral sclerosis that differ with respect to their Riluzole treatment effect, the only currently approved drug for this disease.

Patterns of Lynx Predation at the Interface between Protected Areas and Multi-Use Landscapes in Central Europe.

In Central Europe, protected areas are too small to ensure survival of populations of large carnivores. In the surrounding areas, these species are often persecuted due to competition with game hunters. Therefore, understanding how predation intensity varies spatio-temporally across areas with different levels of protection is fundamental. We investigated the predation patterns of Eurasian lynx (Lynx lynx) on roe deer (Capreolus capreolus) and red deer (Cervus elaphus) in both protected areas and multi-use landscapes of the Bohemian Forest Ecosystem. Based on 359 roe and red deer killed by 10 GPS-collared lynx, we calculated the species-specific annual kill rates and tested for effects of season and lynx age, sex and reproductive status. Because roe and red deer in the study area concentrate in unprotected lowlands during winter, we modeled spatial distribution of kills separately for summer and winter and calculated-the probability of a deer killed by lynx and-the expected number of kills for areas with different levels of protection. Significantly more roe deer (46.05-74.71/year/individual lynx) were killed than red deer (1.57-9.63/year/individual lynx), more deer were killed in winter than in summer, and lynx family groups had higher annual kill rates than adult male, single adult female and subadult female lynx. In winter the probability of a deer killed and the expected number of kills were higher outside the most protected part of the study area than inside; in summer, this probability did not differ between areas, and the expected number of kills was slightly larger inside than outside the most protected part of the study area. This indicates that the intensity of lynx predation in the unprotected part of the Bohemian Forest Ecosystem increases in winter, thus mitigation of conflicts in these areas should be included as a priority in the lynx conservation strategy.