Non-parametric inference on calibration of predicted risks.

Moderate calibration, the expected event probability among observations with predicted probability z being equal to z, is a desired property of risk prediction models. Current graphical and numerical techniques for evaluating moderate calibration of risk prediction models are mostly based on smoothing or grouping the data. As well, there is no widely accepted inferential method for the null hypothesis that a model is moderately calibrated. In this work, we discuss recently-developed, and propose novel, methods for the assessment of moderate calibration for binary responses. The methods are based on the limiting distributions of functions of standardized partial sums of prediction errors converging to the corresponding laws of Brownian motion. The novel method relies on well-known properties of the Brownian bridge which enables joint inference on mean and moderate calibration, leading to a unified "bridge" test for detecting miscalibration. Simulation studies indicate that the bridge test is more powerful, often substantially, than the alternative test. As a case study we consider a prediction model for short-term mortality after a heart attack, where we provide suggestions on graphical presentation and the interpretation of results. Moderate calibration can be assessed without requiring arbitrary grouping of data or using methods that require tuning of parameters.

A parametric model to jointly characterize rate, duration, and severity of exacerbations in episodic diseases.

BACKGROUND: The natural history of many chronic diseases is characterized by periods of increased disease activity, commonly referred to as flare-ups or exacerbations. Accurate characterization of the burden of these exacerbations is an important research objective. METHODS: The purpose of this work was to develop a statistical framework for nuanced characterization of the three main features of exacerbations: their rate, duration, and severity, with interrelationships among these features being a particular focus. We jointly specified a zero-inflated accelerated failure time regression model for the rate, an accelerated failure time regression model for the duration, and a logistic regression model for the severity of exacerbations. Random effects were incorporated into each component to capture heterogeneity beyond the variability attributable to observed characteristics, and to describe the interrelationships among these components. RESULTS: We used pooled data from two clinical trials in asthma as an exemplary application to illustrate the utility of the joint modeling approach. The model fit clearly indicated the presence of heterogeneity in all three components. A novel finding was that the new therapy reduced not just the rate but also the duration of exacerbations, but did not have a significant impact on their severity. After controlling for covariates, exacerbations among more frequent exacerbators tended to be shorter and less likely to be severe. CONCLUSIONS: We conclude that a joint modeling framework, programmable in available software, can provide novel insights about how the rate, duration, and severity of episodic events interrelate, and enables consistent inference on the effect of treatments on different disease outcomes. Trial registration Ethics approval was obtained from the University of British Columbia Human Ethics Board (H17-00938).

Recommendations for the use of propensity score methods in multiple sclerosis research.

BACKGROUND: With many disease-modifying therapies currently approved for the management of multiple sclerosis, there is a growing need to evaluate the comparative effectiveness and safety of those therapies from real-world data sources. Propensity score methods have recently gained popularity in multiple sclerosis research to generate real-world evidence. Recent evidence suggests, however, that the conduct and reporting of propensity score analyses are often suboptimal in multiple sclerosis studies. OBJECTIVES: To provide practical guidance to clinicians and researchers on the use of propensity score methods within the context of multiple sclerosis research. METHODS: We summarize recommendations on the use of propensity score matching and weighting based on the current methodological literature, and provide examples of good practice. RESULTS: Step-by-step recommendations are presented, starting with covariate selection and propensity score estimation, followed by guidance on the assessment of covariate balance and implementation of propensity score matching and weighting. Finally, we focus on treatment effect estimation and sensitivity analyses. CONCLUSION: This comprehensive set of recommendations highlights key elements that require careful attention when using propensity score methods.

16-year trends in asthma hospital admissions in Canada.

BACKGROUND: Asthma hospitalizations declined rapidly in many parts of the world, including Canada, in the 1990s and early 2000s. OBJECTIVE: To examine whether the declining trend of asthma hospitalizations persisted in recent years in Canada. METHODS: Using the Canadian comprehensive nationwide hospitalization data (2002-2017), we identified hospital admissions with the main International Classification of Diseases codes for asthma. We analyzed sex-specific age-standardized trends in annual hospitalization rates among pediatric (< 19 years) and adult (19+ years) patients. We used change-point analysis to evaluate any substantial changes in the trends in the sex-age groups. RESULTS: There were 254,672 asthma-related hospital admissions (59% pediatric, 50% female) during the study period. Among children, age-adjusted annual rates per 100,000 decreased by 55% in females (152-69) and by 60% in males (270-108) from 2002 to 2017. Among adults, the rates decreased by 59% in both sexes (females: 61-25; males: 27-11). Change-point analysis indicated a substantial plateauing of the annual rate in both pediatric (from -15.3 [females] and -25.8 [males] before 2010 to -0.6 [females] and -0.8 [males] after 2010) and adult (from -5.4 [females] and -2.6 [males] before 2008 to -0.6 [females] and -0.2 [males] after 2008) groups. CONCLUSION: After a substantial decline in hospital admissions for acute asthma, there has been minimal further decline since 2010 for children and 2008 for adults. In addition to adhering to the contemporary standards of asthma care, novel, disruptive strategies are likely needed to further reduce the burden of asthma.

Dealing With Treatment-Confounder Feedback and Sparse Follow-up in Longitudinal Studies: Application of a Marginal Structural Model in a Multiple Sclerosis Cohort.

The beta-interferons are widely prescribed platform therapies for patients with multiple sclerosis (MS). We accessed a cohort of patients with relapsing-onset MS from British Columbia, Canada (1995-2013), to examine the potential survival advantage associated with beta-interferon exposure using a marginal structural model. Accounting for potential treatment-confounder feedback between comorbidity, MS disease progression, and beta-interferon exposure, we found an association between beta-interferon exposure of at least 6 contiguous months and improved survival (hazard ratio (HR) = 0.63, 95% confidence interval 0.47, 0.86). We also assessed potential effect modifications by sex, baseline age, or baseline disease duration, and found these factors to be important effect modifiers. Sparse follow-up due to variability in patient contact with the health system is one of the biggest challenges in longitudinal analyses. We considered several single-level and multilevel multiple imputation approaches to deal with sparse follow-up and disease progression information; both types of approach produced similar estimates. Compared to ad hoc imputation approaches, such as linear interpolation (HR = 0.63), and last observation carried forward (HR = 0.65), all multiple imputation approaches produced a smaller hazard ratio (HR = 0.53), although the direction of effect and conclusions drawn concerning the survival advantage remained the same.

Should the number of acute exacerbations in the previous year be used to guide treatments in COPD?

BACKGROUND: In contemporary management of chronic obstructive pulmonary disease (COPD), the frequent exacerbator phenotype, based on a 12-month history of acute exacerbation of COPD (AECOPD), is a major determinant of therapeutic recommendations. However, there is considerable debate as to the stability of this phenotype over time. METHODS: We used fundamental principles in time-to-event analysis to demonstrate that variation in the frequent exacerbator phenotype has two major sources: variability in the underlying AECOPD rate and randomness in the occurrence of individual AECOPDs. We re-analysed data from two large cohorts, the Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints (ECLIPSE) study and the SubPopulations and InteRmediate OutcoMes In COPD Study (SPIROMICS), using a Bayesian model that separated these sources of variability. We then evaluated the stability of the frequent exacerbator phenotype based on these results. RESULTS: In both cohorts, the pattern of AECOPDs strongly supported the presence of an individual-specific underlying AECOPD rate which is stable over time (Bayes Factor less than 0.001). Despite this, the observed AECOPD rate can vary markedly year-to-year within individual patients. For those with an underlying rate of 0.8-3.1 events·year-1, the frequent exacerbator classification, based on the observed rate, changes more than 30% of the time over two consecutive years due to chance alone. This value increases to more than 45% for those with an underlying rate of 1.2-2.2 events·year-1. CONCLUSIONS: While the underlying AECOPD rate is a stable trait, the frequent exacerbator phenotype based on observed AECOPD patterns is unstable, so much so that its suitability for informing treatment decisions should be questioned. Whether evaluating AECOPD history over longer durations or using multivariate prediction models can result in more stable phenotyping needs to be evaluated.

Multiple sclerosis: effect of beta interferon treatment on survival.

Worldwide, the beta interferons remain the most commonly prescribed disease-modifying drugs for multiple sclerosis. However, it is unclear if they alter survival. We investigated the association between beta interferon and mortality in the 'real-world' setting. This was a multi-centre population-based observational study of patients with relapsing-onset multiple sclerosis who were initially registered at a clinic in British Columbia, Canada (1980-2004) or Rennes, France (1976-2013). Data on this cohort were accessed from the clinical multiple sclerosis databases and from individually linked health administrative data; all data were collected prospectively. Participants were followed from the latter of their first multiple sclerosis clinic visit, 18th birthday or 1 January 1996; until death, emigration or 31 December 2013. Only those who were naïve to disease-modifying therapy and immunosuppressant treatment of multiple sclerosis at the start of their follow-up were included in the analysis. A nested case-control approach was used. Up to 20 controls, matched to cases (deaths) by country, sex, age ± 5 years, year and disability level at study entry, were randomly selected from the cohort by incidence density sampling. The associations between all-cause mortality and at least 6 months beta interferon exposure, and also cumulative exposure ('low', 6 months to 3 years; and 'high', >3 years), were estimated by conditional logistic regression adjusting for treatment with other disease-modifying therapies and age in years. Further analyses included separate analyses by sex and country, additional adjustment for comorbidity burden in the Canadian cohort, and estimation of the association between beta interferon and multiple sclerosis-related death in both countries. Among 5989 participants (75% female) with a mean age of 42 (standard deviation, SD 11) years at study entry, there were 742 deaths (70% female) and the mean age at death was 61 (SD 13) years. Of these cases, 649 were matched to between one and 20 controls. Results of the conditional logistic regression analyses are expressed as adjusted odds ratios with 95% confidence intervals. The odds of beta interferon exposure were 32% lower among cases than controls (0.68; 0.53-0.89). Increased survival was associated with >3 years beta interferon exposure (0.44; 0.30-0.66), but not between 6 months and 3 years exposure (1.00; 0.73-1.38). Findings were similar within sex and country, and for multiple sclerosis-related death. Beta interferon treatment was associated with a lower mortality risk among people with relapsing-onset multiple sclerosis. Findings were consistent between two geographically distinct regions in North America and Europe.

Adjusting for differential misclassification in matched case-control studies utilizing health administrative data.

In epidemiological studies of secondary data sources, lack of accurate disease classifications often requires investigators to rely on diagnostic codes generated by physicians or hospital systems to identify case and control groups, resulting in a less-than-perfect assessment of the disease under investigation. Moreover, because of differences in coding practices by physicians, it is hard to determine the factors that affect the chance of an incorrectly assigned disease status. What results is a dilemma where assumptions of non-differential misclassification are questionable but, at the same time, necessary to proceed with statistical analyses. This paper develops an approach to adjust exposure-disease association estimates for disease misclassification, without the need of simplifying non-differentiality assumptions, or prior information about a complicated classification mechanism. We propose to leverage rich temporal information on disease-specific healthcare utilization to estimate each participant's probability of being a true case and to use these estimates as weights in a Bayesian analysis of matched case-control data. The approach is applied to data from a recent observational study into the early symptoms of multiple sclerosis (MS), where MS cases were identified from Canadian health administrative databases and matched to population controls that are assumed to be correctly classified. A comparison of our results with those from non-differentially adjusted analyses reveals conflicting inferences and highlights that ill-suited assumptions of non-differential misclassification can exacerbate biases in association estimates.

Identification of treatment responders based on multiple longitudinal outcomes with applications to multiple sclerosis patients.

Identification of treatment responders is a challenge in comparative studies where treatment efficacy is measured by multiple longitudinally collected continuous and count outcomes. Existing procedures often identify responders on the basis of only a single outcome. We propose a novel multiple longitudinal outcome mixture model that assumes that, conditionally on a cluster label, each longitudinal outcome is from a generalized linear mixed effect model. We utilize a Monte Carlo expectation-maximization algorithm to obtain the maximum likelihood estimates of our high-dimensional model and classify patients according to their estimated posterior probability of being a responder. We demonstrate the flexibility of our novel procedure on two multiple sclerosis clinical trial datasets with distinct data structures. Our simulation study shows that incorporating multiple outcomes improves the responder identification performance; this can occur even if some of the outcomes are ineffective. Our general procedure facilitates the identification of responders who are comprehensively defined by multiple outcomes from various distributions. Copyright © 2017 John Wiley & Sons, Ltd.

Bayesian analysis of pair-matched case-control studies subject to outcome misclassification.

We examine the impact of nondifferential outcome misclassification on odds ratios estimated from pair-matched case-control studies and propose a Bayesian model to adjust these estimates for misclassification bias. The model relies on access to a validation subgroup with confirmed outcome status for all case-control pairs as well as prior knowledge about the positive and negative predictive value of the classification mechanism. We illustrate the model's performance on simulated data and apply it to a database study examining the presence of ten morbidities in the prodromal phase of multiple sclerosis.

Comparison of Statistical Approaches for Dealing With Immortal Time Bias in Drug Effectiveness Studies.

In time-to-event analyses of observational studies of drug effectiveness, incorrect handling of the period between cohort entry and first treatment exposure during follow-up may result in immortal time bias. This bias can be eliminated by acknowledging a change in treatment exposure status with time-dependent analyses, such as fitting a time-dependent Cox model. The prescription time-distribution matching (PTDM) method has been proposed as a simpler approach for controlling immortal time bias. Using simulation studies and theoretical quantification of bias, we compared the performance of the PTDM approach with that of the time-dependent Cox model in the presence of immortal time. Both assessments revealed that the PTDM approach did not adequately address immortal time bias. Based on our simulation results, another recently proposed observational data analysis technique, the sequential Cox approach, was found to be more useful than the PTDM approach (Cox: bias = -0.002, mean squared error = 0.025; PTDM: bias = -1.411, mean squared error = 2.011). We applied these approaches to investigate the association of ß-interferon treatment with delaying disability progression in a multiple sclerosis cohort in British Columbia, Canada (Long-Term Benefits and Adverse Effects of Beta-Interferon for Multiple Sclerosis (BeAMS) Study, 1995-2008).

Association between the use of selective serotonin reuptake inhibitors and multiple sclerosis disability progression.

BACKGROUND: Benefits of selective serotonin reuptake inhibitors (SSRIs) in modifying the multiple sclerosis (MS) disease course have been suggested, but their ability to delay disability progression remains unknown. We examined the association between SSRI exposure and MS disability progression. METHODS: A nested case-control study was conducted using the British Columbia (Canada) Multiple Sclerosis clinical data linked to health administrative data. The primary outcome was a sustained score of 6 (requires a cane to walk) on the Expanded Disability Status Scale (EDSS), and the secondary outcome was the onset of secondary progressive MS (SPMS, an advanced stage of MS). The cases were those who reached a study outcome and were matched with up to four randomly selected controls by sex, age, EDSS and calendar year at study entry using incidence density sampling. The associations between disability worsening and SSRI exposure were assessed with conditional logistic regression models, adjusted for confounders. RESULTS: A total of 3920 patients were included in the main analyses, of which 272 reached sustained EDSS 6 and 187 reached SPMS. SSRI exposure was significantly different between patients who reached sustained EDSS 6 and controls [adjusted odds ratio (adjOR):1.44; 95% confidence interval (CI):1.03-2.01]. However, SSRI exposure was not significantly different between those who reached SPMS and their controls (adjOR:1.35; 95%CI:0.89-2.04). CONCLUSION: We found no evidence to suggest that SSRI exposure was associated with a delay in MS disability accumulation or progression. Copyright © 2016 John Wiley & Sons, Ltd.

Sample-size calculations for short-term proof-of-concept studies of tissue protection and repair in multiple sclerosis lesions via conventional clinical imaging.

BACKGROUND: New multiple sclerosis (MS) lesion activity on magnetic resonance imaging (MRI) can test immunomodulatory therapies in proof-of-concept trials. Comparably powerful endpoints to assess tissue protection or repair are lacking. OBJECTIVE: The objective of this paper is to report sample-size calculations for assessment of new lesion recovery. METHODS: In two sets of six active MS cases, new lesions were observed by monthly MRI for approximately 12 months. Averages and quartiles of normalized (proton density/T1/T2 weighted) and quantitative (T1/T2 and mean diffusivity maps for dataset 1, T2 and magnetization transfer ratio maps for dataset 2) measures were used to compare the lesion area before lesion appearance to afterward. A linear mixed-effects model incorporating lesion- and participant-specific random effects estimated average levels and variance components for sample-size calculations. RESULTS: In both datasets, greatest statistical sensitivity was observed for the 25th percentile of normalized proton density-weighted signal. At 3T, using new lesions ⩾15 mm(3), as few as nine participants/arm may be required for a six-month placebo-controlled add-on trial postulating a therapeutic effect size of 20% and statistical power of 90%. CONCLUSION: Lesion recovery is a powerful outcome measure for proof-of-concept clinical trials of tissue protection and repair in MS. The trial design requires active cases and is therefore best implemented near disease onset.

A flexible mixed-effect negative binomial regression model for detecting unusual increases in MRI lesion counts in individual multiple sclerosis patients.

We develop a new modeling approach to enhance a recently proposed method to detect increases of contrast-enhancing lesions (CELs) on repeated magnetic resonance imaging, which have been used as an indicator for potential adverse events in multiple sclerosis clinical trials. The method signals patients with unusual increases in CEL activity by estimating the probability of observing CEL counts as large as those observed on a patient's recent scans conditional on the patient's CEL counts on previous scans. This conditional probability index (CPI), computed based on a mixed-effect negative binomial regression model, can vary substantially depending on the choice of distribution for the patient-specific random effects. Therefore, we relax this parametric assumption to model the random effects with an infinite mixture of beta distributions, using the Dirichlet process, which effectively allows any form of distribution. To our knowledge, no previous literature considers a mixed-effect regression for longitudinal count variables where the random effect is modeled with a Dirichlet process mixture. As our inference is in the Bayesian framework, we adopt a meta-analytic approach to develop an informative prior based on previous clinical trials. This is particularly helpful at the early stages of trials when less data are available. Our enhanced method is illustrated with CEL data from 10 previous multiple sclerosis clinical trials. Our simulation study shows that our procedure estimates the CPI more accurately than parametric alternatives when the patient-specific random effect distribution is misspecified and that an informative prior improves the accuracy of the CPI estimates.

Marginal structural Cox models for estimating the association between ß-interferon exposure and disease progression in a multiple sclerosis cohort.

Longitudinal observational data are required to assess the association between exposure to ß-interferon medications and disease progression among relapsing-remitting multiple sclerosis (MS) patients in the "real-world" clinical practice setting. Marginal structural Cox models (MSCMs) can provide distinct advantages over traditional approaches by allowing adjustment for time-varying confounders such as MS relapses, as well as baseline characteristics, through the use of inverse probability weighting. We assessed the suitability of MSCMs to analyze data from a large cohort of 1,697 relapsing-remitting MS patients in British Columbia, Canada (1995-2008). In the context of this observational study, which spanned more than a decade and involved patients with a chronic yet fluctuating disease, the recently proposed "normalized stabilized" weights were found to be the most appropriate choice of weights. Using this model, no association between ß-interferon exposure and the hazard of disability progression was found (hazard ratio = 1.36, 95% confidence interval: 0.95, 1.94). For sensitivity analyses, truncated normalized unstabilized weights were used in additional MSCMs and to construct inverse probability weight-adjusted survival curves; the findings did not change. Additionally, qualitatively similar conclusions from approximation approaches to the weighted Cox model (i.e., MSCM) extend confidence in the findings.

Association between beta-interferon exposure and hospital events in multiple sclerosis.

PURPOSE: A systematic evaluation of hospital events can be an important surrogate measure for drug effectiveness or adverse effects. The purpose of this study was to examine the association between beta-interferon use and hospital events in a large cohort of patients with multiple sclerosis (MS). METHODS: Retrospective cohort study comparing beta-interferon exposed and unexposed patients using clinical data from the British Columbia MS (BCMS) database linked with health administrative databases, 1996-2008. For each patient, the primary outcome was the number of hospital events in each month, analyzed by quasi Poisson regression. Beta-interferon exposure was examined two ways: current and cumulative exposure. Secondary outcomes included whether a hospital event occurred in each month for each specific primary diagnoses, grouped by International Classification of Diseases categories. RESULTS: Current exposure to beta-interferon was not associated with an altered rate of hospital events (adjusted incident rate ratio 1.018; 95% CI 0.803-1.290). Similarly, there was no association with cumulative exposure. Cumulative beta-interferon exposure was associated with a lower odds of respiratory disease-related hospital events compared to those never exposed to beta-interferon. CONCLUSIONS: Exposure to beta-interferon for MS was not associated with a change in overall hospital event rates. Preliminary evidence suggests that the beta-interferons may have a protective effect against respiratory diseases requiring hospitalization in MS patients.

Natural, innate improvements in multiple sclerosis disability.

BACKGROUND: Improvements in multiple sclerosis (MS) disability have recently been reported in immunomodulatory drug (IMD) clinical trials and observational studies. However, improvements have rarely been examined in natural history or IMD naive patients. We investigated annual and biennial improvements in Expanded Disability Status Scale (EDSS) scores in British Columbia, Canada. METHODS: The British Columbian MS database was accessed for definite MS patients (1980-2009). Consecutive IMD-free EDSS scores one and two years apart (± 3 months) were examined; improvements (≥0.5,≥1,≥2 EDSS points) and sustained improvements (confirmed at one year) were described. The influence of patient characteristics on improvements was examined using logistic regression. RESULTS: From 16,132 EDSS scores, 7653 yearly and 5845 biennial EDSS intervals were available for 2961 and 2382 patients respectively. Of the yearly intervals, 14.9% showed an improvement (≥0.5 points), 8.3% ≥1 point and 2.2% ≥2 point improvement, with nearly half being sustained. Corresponding worsenings were observed in 32.9%, 20.5% and 7.9% respectively, with stability in just over half (53%). Biennial findings were similar. Characteristics generally associated with improvements included: female sex, younger age, shorter disease duration, relapsing-onset and presence of moderate disability (compared with mild or advanced) and a previous episode of worsening (disassociated from a relapse). However, improvements were also observed after periods of stability and in primary-progressive MS. CONCLUSION: Improvements in MS disability over one or two years are not unusual. We suggest the term 'innate improvements'. Our findings have implication for the design of clinical trials and observational studies in MS targeting improvements on the EDSS.

Neutralizing antibodies to interferon beta-1b multiple sclerosis: a clinico-radiographic paradox in the BEYOND trial.

BACKGROUND: The frequency and impact of neutralizing antibodies (NAbs) to interferon beta-1b (IFNß-1b) on clinical and radiographic outcomes is controversial. OBJECTIVE: To assess NAb impact in the BEYOND study. METHODS: 2244 patients were randomized (2:2:1) to receive IFNß-1b, either 250 or 500 µg, or glatiramer acetate, 20 mg, and observed for 2-3.5 years. NAb titers were determined every 6 months. A titer ≥20 NU/ml was considered NAb positive. Efficacy was compared between NAb-positive and NAb-negative patients, using comprehensive statistical analyses, taking into account the delayed appearance of NAbs, the time-dependent changes in the relapse rate, spontaneous reversions to NAb-negative status, NAb-titer level, and also adjusting for baseline factors. RESULTS: In the IFNß-1b 250 µg group, NAb-positive titers were detected (≥ once) in 319 patients (37.0%); of these, 112 (35.1%) reverted to NAb-negative status. In the IFNß-1b 500 µg group, 340 patients (40.7%) became NAb-positive and 119 (35.0%) reverted to NAb-negative status. In both IFNß groups, especially the 250 µg arm, NAb-positive status was not associated with a convincing impact on any clinical outcome measure by any statistical analysis. By contrast, in both IFNß groups, NAbs were associated with a very consistent deleterious impact on most MRI outcomes. CONCLUSION: There was a notable dissociation between the impact of NAbs on MRI and clinical outcomes. On MRI measures, the impact was consistent and convincing, whereas on clinical measures a negative impact of NAbs was not found. The basis for this clinico-radiographic paradox is unknown but it suggests that the relationship between NAbs and the therapeutic effects of IFNß-1b is complex.

Association between use of interferon beta and progression of disability in patients with relapsing-remitting multiple sclerosis.

CONTEXT: Interferon beta is widely prescribed to treat multiple sclerosis (MS); however, its relationship with disability progression has yet to be established. OBJECTIVE: To investigate the association between interferon beta exposure and disability progression in patients with relapsing-remitting MS. DESIGN, SETTING, AND PATIENTS: Retrospective cohort study based on prospectively collected data (1985-2008) from British Columbia, Canada. Patients with relapsing-remitting MS treated with interferon beta (n = 868) were compared with untreated contemporary (n = 829) and historical (n = 959) cohorts. MAIN OUTCOME MEASURES: The main outcome measure was time from interferon beta treatment eligibility (baseline) to a confirmed and sustained score of 6 (requiring a cane to walk 100 m; confirmed at >150 days with no measurable improvement) on the Expanded Disability Status Scale (EDSS) (range, 0-10, with higher scores indicating higher disability). A multivariable Cox regression model with interferon beta treatment included as a time-varying covariate was used to assess the hazard of disease progression associated with interferon beta treatment. Analyses also included propensity score adjustment to address confounding by indication. RESULTS: The median active follow-up times (first to last EDSS measurement) were as follows: for the interferon beta-treated cohort, 5.1 years (interquartile range [IQR], 3.0-7.0 years); for the contemporary control cohort, 4.0 years (IQR, 2.1-6.4 years); and for the historical control cohort, 10.8 years (IQR, 6.3-14.7 years). The observed outcome rates for reaching a sustained EDSS score of 6 were 10.8%, 5.3%, and 23.1% in the 3 cohorts, respectively. After adjustment for potential baseline confounders (sex, age, disease duration, and EDSS score), exposure to interferon beta was not associated with a statistically significant difference in the hazard of reaching an EDSS score of 6 when either the contemporary control cohort (hazard ratio, 1.30; 95% CI, 0.92-1.83; P = .14) or the historical control cohort (hazard ratio, 0.77; 95% CI, 0.58-1.02; P = .07) were considered. Further adjustment for comorbidities and socioeconomic status, where possible, did not change interpretations, and propensity score adjustment did not substantially change the results. CONCLUSION: Among patients with relapsing-remitting MS, administration of interferon beta was not associated with a reduction in progression of disability.

Model-Based ROC Curve: Examining the Effect of Case Mix and Model Calibration on the ROC Plot.

BACKGROUND: The performance of risk prediction models is often characterized in terms of discrimination and calibration. The receiver-operating characteristic (ROC) curve is widely used for evaluating model discrimination. However, when comparing ROC curves across different samples, the effect of case mix makes the interpretation of discrepancies difficult. Further, compared with model discrimination, evaluating model calibration has not received the same level of attention. Current methods for examining model calibration require specification of smoothing or grouping factors. METHODS: We introduce the "model-based" ROC curve (mROC) to assess model calibration and the effect of case mix during external validation. The mROC curve is the ROC curve that should be observed if the prediction model is calibrated in the external population. We show that calibration-in-the-large and the equivalence of mROC and ROC curves are together sufficient conditions for the model to be calibrated. Based on this, we propose a novel statistical test for calibration that, unlike current methods, does not require any subjective specification of smoothing or grouping factors. RESULTS: Through a stylized example, we demonstrate how mROC separates the effect of case mix and model miscalibration when externally validating a risk prediction model. We present the results of simulation studies that confirm the properties of the new calibration test. A case study on predicting the risk of acute exacerbations of chronic obstructive pulmonary disease puts the developments in a practical context. R code for the implementation of this method is provided. CONCLUSION: mROC can easily be constructed and used to interpret the effect of case mix and calibration on the ROC plot. Given the popularity of ROC curves among applied investigators, this framework can further promote assessment of model calibration. HIGHLIGHTS: Compared with examining model discrimination, examining model calibration has not received the same level of attention among investigators who develop or examine risk prediction models.This article introduces the model-based ROC (mROC) curve as the basis for graphical and statistical examination of model calibration on the ROC plot.This article introduces a formal statistical test based on mROC for examining model calibration that does not require arbitrary smoothing or grouping factors.Investigators who develop or validate risk prediction models can now also use the popular ROC plot for examining model calibration, as a critical but often neglected component in predictive analytics.