Developing a predictive signature for two trial endpoints using the cross-validated risk scores method.

The existing cross-validated risk scores (CVRS) design has been proposed for developing and testing the efficacy of a treatment in a high-efficacy patient group (the sensitive group) using high-dimensional data (such as genetic data). The design is based on computing a risk score for each patient and dividing them into clusters using a nonparametric clustering procedure. In some settings, it is desirable to consider the tradeoff between two outcomes, such as efficacy and toxicity, or cost and effectiveness. With this motivation, we extend the CVRS design (CVRS2) to consider two outcomes. The design employs bivariate risk scores that are divided into clusters. We assess the properties of the CVRS2 using simulated data and illustrate its application on a randomized psychiatry trial. We show that CVRS2 is able to reliably identify the sensitive group (the group for which the new treatment provides benefit on both outcomes) in the simulated data. We apply the CVRS2 design to a psychology clinical trial that had offender status and substance use status as two outcomes and collected a large number of baseline covariates. The CVRS2 design yields a significant treatment effect for both outcomes, while the CVRS approach identified a significant effect for the offender status only after prefiltering the covariates.

Finding the best subgroup with differential treatment effect with multiple outcomes.

Precision medicine aims to identify specific patient subgroups that may benefit the most from a particular treatment than the whole population. Existing definitions for the best subgroup in subgroup analysis are based on a single outcome and do not consider multiple outcomes; specifically, outcomes of different types. In this article, we introduce a definition for the best subgroup under a multiple-outcome setting with continuous, binary, and censored time-to-event outcomes. Our definition provides a trade-off between the subgroup size and the conditional average treatment effects (CATE) in the subgroup with respect to each of the outcomes while taking the relative contribution of the outcomes into account. We conduct simulations to illustrate the proposed definition. By examining the outcomes of urinary tract infection and renal scarring in the RIVUR clinical trial, we identify a subgroup of children that would benefit the most from long-term antimicrobial prophylaxis.

Variance-components tests for genetic association with multiple interval-censored outcomes.

Massive genetic compendiums such as the UK Biobank have become an invaluable resource for identifying genetic variants that are associated with complex diseases. Due to the difficulties of massive data collection, a common practice of these compendiums is to collect interval-censored data. One challenge in analyzing such data is the lack of methodology available for genetic association studies with interval-censored data. Genetic effects are difficult to detect because of their rare and weak nature, and often the time-to-event outcomes are transformed to binary phenotypes for access to more powerful signal detection approaches. However transforming the data to binary outcomes can result in loss of valuable information. To alleviate such challenges, this work develops methodology to associate genetic variant sets with multiple interval-censored outcomes. Testing sets of variants such as genes or pathways is a common approach in genetic association settings to lower the multiple testing burden, aggregate small effects, and improve interpretations of results. Instead of performing inference with only a single outcome, utilizing multiple outcomes can increase statistical power by aggregating information across multiple correlated phenotypes. Simulations show that the proposed strategy can offer significant power gains over a single outcome approach. We apply the proposed test to the investigation that motivated this study, a search for the genes that perturb risks of bone fractures and falls in the UK Biobank.

Dealing with imperfect randomization: Inference for the highscope perry preschool program.

This paper considers the problem of making inferences about the effects of a program on multiple outcomes when the assignment of treatment status is imperfectly randomized. By imperfect randomization we mean that treatment status is reassigned after an initial randomization on the basis of characteristics that may be observed or unobserved by the analyst. We develop a partial identification approach to this problem that makes use of information limiting the extent to which randomization is imperfect to show that it is still possible to make nontrivial inferences about the effects of the program in such settings. We consider a family of null hypotheses in which each null hypothesis specifies that the program has no effect on one of several outcomes of interest. Under weak assumptions, we construct a procedure for testing this family of null hypotheses in a way that controls the familywise error rate - the probability of even one false rejection - in finite samples. We develop our methodology in the context of a reanalysis of the HighScope Perry Preschool program. We find statistically significant effects of the program on a number of different outcomes of interest, including outcomes related to criminal activity for males and females, even after accounting for the imperfectness of the randomization and the multiplicity of null hypotheses.

Permutation Tests for Assessing Potential Non-Linear Associations between Treatment Use and Multivariate Clinical Outcomes.

In many psychometric applications, the relationship between the mean of an outcome and a quantitative covariate is too complex to be described by simple parametric functions; instead, flexible nonlinear relationships can be incorporated using penalized splines. Penalized splines can be conveniently represented as a linear mixed effects model (LMM), where the coefficients of the spline basis functions are random effects. The LMM representation of penalized splines makes the extension to multivariate outcomes relatively straightforward. In the LMM, no effect of the quantitative covariate on the outcome corresponds to the null hypothesis that a fixed effect and a variance component are both zero. Under the null, the usual asymptotic chi-square distribution of the likelihood ratio test for the variance component does not hold. Therefore, we propose three permutation tests for the likelihood ratio test statistic: one based on permuting the quantitative covariate, the other two based on permuting residuals. We compare via simulation the Type I error rate and power of the three permutation tests obtained from joint models for multiple outcomes, as well as a commonly used parametric test. The tests are illustrated using data from a stimulant use disorder psychosocial clinical trial.

A generalized calibrated Bayesian hierarchical modeling approach to basket trials with multiple endpoints.

A basket trial simultaneously evaluates a treatment in multiple cancer subtypes, offering an effective way to accelerate drug development in multiple indications. Many basket trials are designed and monitored based on a single efficacy endpoint, primarily the tumor response. For molecular targeted or immunotherapy agents, however, a single efficacy endpoint cannot adequately characterize the treatment effect. It is increasingly important to use more complex endpoints to comprehensively assess the risk-benefit profile of such targeted therapies. We extend the calibrated Bayesian hierarchical modeling approach to monitor phase II basket trials with multiple endpoints. We propose two generalizations, one based on the latent variable approach and the other based on the multinomial-normal hierarchical model, to accommodate different types of endpoints and dependence assumptions regarding information sharing. We introduce shrinkage parameters as functions of statistics measuring homogeneity among subgroups and propose a general calibration approach to determine the functional forms. Theoretical properties of the generalized hierarchical models are investigated. Simulation studies demonstrate that the monitoring procedure based on the generalized approach yields desirable operating characteristics.

Differences in set-based tests for sparse alternatives when testing sets of outcomes compared to sets of explanatory factors in genetic association studies.

Set-based association tests are widely popular in genetic association settings for their ability to aggregate weak signals and reduce multiple testing burdens. In particular, a class of set-based tests including the Higher Criticism, Berk-Jones, and other statistics have recently been popularized for reaching a so-called detection boundary when signals are rare and weak. Such tests have been applied in two subtly different settings: (a) associating a genetic variant set with a single phenotype and (b) associating a single genetic variant with a phenotype set. A significant issue in practice is the choice of test, especially when deciding between innovated and generalized type methods for detection boundary tests. Conflicting guidance is present in the literature. This work describes how correlation structures generate marked differences in relative operating characteristics for settings (a) and (b). The implications for study design are significant. We also develop novel power bounds that facilitate the aforementioned calculations and allow for analysis of individual testing settings. In more concrete terms, our investigation is motivated by translational expression quantitative trait loci (eQTL) studies in lung cancer. These studies involve both testing for groups of variants associated with a single gene expression (multiple explanatory factors) and testing whether a single variant is associated with a group of gene expressions (multiple outcomes). Results are supported by a collection of simulation studies and illustrated through lung cancer eQTL examples.

Estimation and visualization of heterogeneous treatment effects for multiple outcomes.

We consider two-arm comparison in clinical trials. The objective is to identify a population with characteristics that make the treatment effective. Such a population is called a subgroup. This identification can be made by estimating the treatment effect and identifying the interactions between treatments and covariates. For a single outcome, there are several ways available to identify the subgroups. There are also multiple outcomes, but they are difficult to interpret and cannot be applied to outcomes other than continuous values. In this paper, we thus propose a new method that allows for a straightforward interpretation of subgroups and deals with both continuous and binary outcomes. The proposed method introduces latent variables and adds Lasso sparsity constraints to the estimated loadings to facilitate the interpretation of the relationship between outcomes and covariates. The interpretation of the subgroups is made by visualizing treatment effects and latent variables. Since we are performing sparse estimation, we can interpret the covariates related to the treatment effects and subgroups. Finally, simulation and real data examples demonstrate the effectiveness of the proposed method.

Mapping the risk of infections in patients with multiple sclerosis: A multi-database study in the United Kingdom Clinical Practice Research Datalink GOLD and Aurum.

BACKGROUND: People with multiple sclerosis (pwMS) have an increased risk of infections; risk factors include underlying disease, physical impairment and use of some disease-modifying treatments. OBJECTIVE: To quantify changes in population-level infection rates among pwMS and compare these to the general population and people with rheumatoid arthritis (pwRA), and identify patient characteristics predictive of infections after MS diagnosis. METHODS: We conducted a multi-database study using data on 23,226 people with MS diagnosis from the UK Clinical Practice Research Datalink Aurum and GOLD (January 2000-December 2020). PwMS were matched to MS-free controls and pwRA. We calculated infection rates, and estimated incidence rate ratios (IRR) and 95% confidence intervals (CI) of predictors for infections ⩽ 5 years after MS diagnosis using Poisson regression. RESULTS: Among pwMS, overall infection rates remained stable - 1.51-fold (1.49-1.52) that in MS-free controls and 0.87-fold (0.86-0.88) that in pwRA - although urinary tract infection rate per 1000 person-years increased from 98.7 (96.1-101) (2000-2010) to 136 (134-138) (2011-2020). Recent infection before MS diagnosis was most predictive of infections (1 infection: IRR 1.92 (1.86-1.97); ⩾2 infections: IRR 3.00 (2.89-3.10)). CONCLUSION: The population-level elevated risk of infection among pwMS has remained stable despite the introduction of disease-modifying treatments.

Effects of prenatal alcohol exposure on cognitive and behavioral development: Findings from a hierarchical meta-analysis of data from six prospective longitudinal U.S. cohorts.

BACKGROUND: Cognitive and behavioral sequelae of prenatal alcohol exposure (PAE) continue to be prevalent in the United States and worldwide. Because these sequelae are also common in other neurodevelopmental disorders, researchers have attempted to identify a distinct neurobehavioral profile to facilitate the differential diagnosis of fetal alcohol spectrum disorders (FASD). We used an innovative, individual participant meta-analytic technique to combine data from six large U.S. longitudinal cohorts to provide a more comprehensive and reliable characterization of the neurobehavioral deficits seen in FASD than can be obtained from smaller samples. METHODS: Meta-analyses were performed on data from 2236 participants to examine effects of PAE (measured as oz absolute alcohol/day (AA/day)) on IQ, four domains of cognition function (learning and memory, executive function, reading achievement, and math achievement), sustained attention, and behavior problems, after adjusting for potential confounders using propensity scores. RESULTS: The effect sizes for IQ and the four domains of cognitive function were strikingly similar to one another and did not differ at school age, adolescence, or young adulthood. Effect sizes were smaller in the more middle-class Seattle cohort and larger in the three cohorts that obtained more detailed and comprehensive assessments of AA/day. PAE effect sizes were somewhat weaker for parent- and teacher-reported behavior problems and not significant for sustained attention. In a meta-analysis of five aspects of executive function, the strongest effect was on set-shifting. CONCLUSIONS: The similarity in the effect sizes for the four domains of cognitive function suggests that PAE affects an underlying component or components of cognition involving learning and memory and executive function that are reflected in IQ and academic achievement scores. The weaker effects in the more middle-class cohort may reflect a more cognitively stimulating environment, a different maternal drinking pattern (lower alcohol dose/occasion), and/or better maternal prenatal nutrition. These findings identify two domains of cognition-learning/memory and set-shifting-that are particularly affected by PAE, and one, sustained attention, which is apparently spared.

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches.

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.

Joint modeling of multivariate nonparametric longitudinal data and survival data: A local smoothing approach.

In many clinical studies, evaluating the association between longitudinal and survival outcomes is of primary concern. For analyzing data from such studies, joint modeling of longitudinal and survival data becomes an appealing approach. In some applications, there are multiple longitudinal outcomes whose longitudinal pattern is difficult to describe by a parametric form. For such applications, existing research on joint modeling is limited. In this article, we develop a novel joint modeling method to fill the gap. In the new method, a local polynomial mixed-effects model is used for describing the nonparametric longitudinal pattern of the multiple longitudinal outcomes. Two model estimation procedures, that is, the local EM algorithm and the local penalized quasi-likelihood estimation, are explored. Practical guidelines for choosing tuning parameters and for variable selection are provided. The new method is justified by some theoretical arguments and numerical studies.

A comparison of methods for analysing multiple outcome measures in randomised controlled trials using a simulation study.

Multiple primary outcomes are sometimes collected and analysed in randomised controlled trials (RCTs), and are used in favour of a single outcome. By collecting multiple primary outcomes, it is possible to fully evaluate the effect that an intervention has for a given disease process. A simple approach to analysing multiple outcomes is to consider each outcome separately, however, this approach does not account for any pairwise correlations between the outcomes. Any cases with missing values must be ignored, unless an additional imputation step is performed. Alternatively, multivariate methods that explicitly model the pairwise correlations between the outcomes may be more efficient when some of the outcomes have missing values. In this paper, we present an overview of relevant methods that can be used to analyse multiple outcome measures in RCTs, including methods based on multivariate multilevel (MM) models. We perform simulation studies to evaluate the bias in the estimates of the intervention effects and the power of detecting true intervention effects observed when using selected methods. Different simulation scenarios were constructed by varying the number of outcomes, the type of outcomes, the degree of correlations between the outcomes and the proportions and mechanisms of missing data. We compare multivariate methods to univariate methods with and without multiple imputation. When there are strong correlations between the outcome measures (ρ > .4), our simulation studies suggest that there are small power gains when using the MM model when compared to analysing the outcome measures separately. In contrast, when there are weak correlations (ρ < .4), the power is reduced when using univariate methods with multiple imputation when compared to analysing the outcome measures separately.

A Bayesian hierarchical approach for multiple outcomes in routinely collected healthcare data.

Clinical trials are the standard approach for evaluating new treatments, but may lack the power to assess rare outcomes. Trial results are also necessarily restricted to the population considered in the study. The availability of routinely collected healthcare data provides a source of information on the performance of treatments beyond that offered by clinical trials, but the analysis of this type of data presents a number of challenges. Hierarchical methods, which take advantage of known relationships between clinical outcomes, while accounting for bias, may be a suitable statistical approach for the analysis of this data. A study of direct oral anticoagulants in Scotland is discussed and used to motivate a modeling approach. A Bayesian hierarchical model, which allows a stratification of the population into clusters with similar characteristics, is proposed and applied to the direct oral anticoagulant study data. A simulation study is used to assess its performance in terms of outcome detection and error rates.

Spie charts for quantifying treatment effectiveness and safety in multiple outcome network meta-analysis: a proof-of-concept study.

BACKGROUND: Network meta-analysis (NMA) simultaneously synthesises direct and indirect evidence on the relative efficacy and safety of at least three treatments. A decision maker may use the coherent results of an NMA to determine which treatment is best for a given outcome. However, this evidence must be balanced across multiple outcomes. This study aims to provide a framework that permits the objective integration of the comparative effectiveness and safety of treatments across multiple outcomes. METHODS: In the proposed framework, measures of each treatment's performance are plotted on its own pie chart, superimposed on another pie chart representing the performance of a hypothetical treatment that is the best across all outcomes. This creates a spie chart for each treatment, where the coverage area represents the probability a treatment ranks best overall. The angles of each sector may be adjusted to reflect the importance of each outcome to a decision maker. The framework is illustrated using two published NMA datasets comparing dietary oils and fats and psoriasis treatments. Outcome measures are plotted in terms of the surface under the cumulative ranking curve. The use of the spie chart was contrasted with that of the radar plot. RESULTS: In the NMA comparing the effects of dietary oils and fats on four lipid biomarkers, the ease of incorporating the lipids' relative importance on spie charts was demonstrated using coefficients from a published risk prediction model on coronary heart disease. Radar plots produced two sets of areas based on the ordering of the lipids on the axes, while the spie chart only produced one set. In the NMA comparing psoriasis treatments, the areas inside spie charts containing both efficacy and safety outcomes masked critical information on the treatments' comparative safety. Plotting the areas inside spie charts of the efficacy outcomes against measures of the safety outcome facilitated simultaneous comparisons of the treatments' benefits and harms. CONCLUSIONS: The spie chart is more optimal than a radar plot for integrating the comparative effectiveness or safety of a treatment across multiple outcomes. Formal validation in the decision-making context, along with statistical comparisons with other recent approaches are required.

Extensions of the probabilistic ranking metrics of competing treatments in network meta-analysis to reflect clinically important relative differences on many outcomes.

One of the key features of network meta-analysis is ranking of interventions according to outcomes of interest. Ranking metrics are prone to misinterpretation because of two limitations associated with the current ranking methods. First, differences in relative treatment effects might not be clinically important and this is not reflected in the ranking metrics. Second, there are no established methods to include several health outcomes in the ranking assessments. To address these two issues, we extended the P-score method to allow for multiple outcomes and modified it to measure the mean extent of certainty that a treatment is better than the competing treatments by a certain amount, for example, the minimum clinical important difference. We suggest to present the tradeoff between beneficial and harmful outcomes allowing stakeholders to consider how much adverse effect they are willing to tolerate for specific gains in efficacy. We used a published network of 212 trials comparing 15 antipsychotics and placebo using a random effects network meta-analysis model, focusing on three outcomes; reduction in symptoms of schizophrenia in a standardized scale, all-cause discontinuation, and weight gain.

Bayesian Network Meta-analysis of Multiple Outcomes in Dental Research.

OBJECTIVES: Dental research typically targets multiple outcomes. Interdental cleaning devices such as interdental brushes (IB) and water jet devices (WJ) share a sizable portion of the medical device market. However, recommendations for device selection are limited by the conflicting evidence from multiple outcomes in available studies and the lack of an appropriate synthesis approach to summarize evidences taken from multiple outcomes. In particular, both pairwise meta-analyses and single-outcome network meta-analyses can give discordant results. The purpose of this multioutcome, Bayesian network meta-analysis is to introduce this innovative method to the dental research community using data from interdental cleaning device studies for illustrative purposes. METHODS: We reanalyzed a network meta-analysis of interproximal oral hygiene methods in the reduction of clinical indices of inflammation, which included 22 trials assessing 10 interproximal oral hygiene aids. We focused on the primary outcome of gingival inflammation, which was measured by 2 correlated outcome variables, the Gingival Index (GI) and bleeding on probing (BOP). RESULTS: In our previous single-outcome analysis, we concluded that IB and WJ rank high for reducing gingival inflammation while toothpick and flossing rank last. In this multioutcome Bayesian network meta-analysis with equal weight on GI and BOP, the surface under the cumulative ranking curve was 0.87 for WJ and 0.85 for IB. WJ and IB remained ranked as the 2 best devices across different sets of weightings for the GI and BOP. CONCLUSION: In conclusion, multioutcome Bayesian network meta-analysis naturally takes the correlations among multiple outcomes into account, which in turn can provide more comprehensive evidence.

Bayesian optimal interval design with multiple toxicity constraints.

Most phase I dose-finding trials are conducted based on a single binary toxicity outcome to investigate the safety of new drugs. In many situations, however, it is important to distinguish between various toxicity types and different toxicity grades. By minimizing the maximum joint probability of incorrect decisions, we extend the Bayesian optimal interval (BOIN) design to control multiple toxicity outcomes at prespecified levels. The developed multiple-toxicity BOIN design can handle equally important, unequally important as well as nested toxicity outcomes. Interestingly, we find that the optimal interval boundaries with non-nested toxicity outcomes under the proposed method coincide with those under the standard single-toxicity BOIN design by treating the multiple toxicity outcomes marginally. We establish several desirable properties for the proposed interval design. We additionally extend our design to address trials with combined drugs. The finite-sample performance of the proposed methods is assessed according to extensive simulation studies and is compared with those of existing methods. Simulation results reveal that, our methods are as accurate and efficient as the more complicated model-based methods, but are more robust and much easier to implement.

Quantifying the totality of treatment effect with multiple event-time observations in the presence of a terminal event from a comparative clinical study.

To evaluate the totality of one treatment's benefit/risk profile relative to an alternative treatment via a longitudinal comparative clinical study, the timing and occurrence of multiple clinical events are typically collected during the patient's follow-up. These multiple observations reflect the patient's disease progression/burden over time. The standard practice is to create a composite endpoint from the multiple outcomes, the timing of the occurrence of the first clinical event, to evaluate the treatment via the standard survival analysis techniques. By ignoring all events after the composite outcome, this type of assessment may not be ideal. Various parametric or semiparametric procedures have been extensively discussed in the literature for the purposes of analyzing multiple event-time data. Many existing methods were developed based on extensive model assumptions. When the model assumptions are not plausible, the resulting inferences for the treatment effect may be misleading. In this article, we propose a simple, nonparametric inference procedure to quantify the treatment effect, which has an intuitive clinically meaningful interpretation. We use the data from a cardiovascular clinical trial for heart failure to illustrate the procedure. A simulation study is also conducted to evaluate the performance of the new proposal.

Risk factors associated with multiple correlated health outcomes in the 500 Cities Project.

Reducing chronic disease is a major health challenge. Risk factors for chronic diseases are often studied at the individual level, even though interventions and policies may be implemented at the city level. We use an ecologic study design with city-level data, to simultaneously assess the relative impact of unhealthy behaviors and preventive care measures on multiple chronic disease health outcomes. We analyze a newly available, large national dataset called the 500 Cities Project. We examine the associations between city-level prevalence of unhealthy behaviors, clinical preventive service use, and all chronic disease health outcomes in 500 of the largest U.S. cities for year 2014. After adjusting for age and demographic characteristics, using MANOVA we found that the top three risk factors for all health outcomes are smoking (Pillai's trace = 0.95, approx. F = 688.7, p-value < 0.0001), lack of physical activity (Pillai's trace = 0.91, approx. F = 380.0, p-value < 0.0001) and binge drinking (Pillai's trace = 0.91, approx. F = 348.8, p-value < 0.0001), which are statistically significant after adjusting for multiple comparisons. Higher prevalence of an annual dental checkup, a preventive service use measure, is correlated with lower prevalence of several chronic diseases such as diabetes (correlation coefficient r = -0.88), poor physical health (r = -0.91), stroke (r = -0.85), cardiovascular disease (r = -0.83) and poor mental health (r = -0.82). Identifying important chronic disease risk factors at the city-level may provide more actionable information for policymakers to improve urban health.