Using leave-one-out cross validation (LOO) in a multilevel regression and poststratification (MRP) workflow: A cautionary tale.

In recent decades, multilevel regression and poststratification (MRP) has surged in popularity for population inference. However, the validity of the estimates can depend on details of the model, and there is currently little research on validation. We explore how leave-one-out cross validation (LOO) can be used to compare Bayesian models for MRP. We investigate two approximate calculations of LOO: Pareto smoothed importance sampling (PSIS-LOO) and a survey-weighted alternative (WTD-PSIS-LOO). Using two simulation designs, we examine how accurately these two criteria recover the correct ordering of model goodness at predicting population and small-area estimands. Focusing first on variable selection, we find that neither PSIS-LOO nor WTD-PSIS-LOO correctly recovers the models' order for an MRP population estimand, although both criteria correctly identify the best and worst model. When considering small-area estimation, the best model differs for different small areas, highlighting the complexity of MRP validation. When considering different priors, the models' order seems slightly better at smaller-area levels. These findings suggest that, while not terrible, PSIS-LOO-based ranking techniques may not be suitable to evaluate MRP as a method. We suggest this is due to the aggregation stage of MRP, where individual-level prediction errors average out. We validate these results by applying to the real world National Health and Nutrition Examination Survey (NHANES) data in the United States. Altogether, these results show that PSIS-LOO-based model validation tools need to be used with caution and might not convey the full story when validating MRP as a method.

A nonparametric relative treatment effect for direct comparisons of censored paired survival outcomes.

A frequently addressed issue in clinical trials is the comparison of censored paired survival outcomes, for example, when individuals were matched based on their characteristics prior to the analysis. In this regard, a proper incorporation of the dependence structure of the paired censored outcomes is required and, up to now, appropriate methods are only rarely available in the literature. Moreover, existing methods are not motivated by the strive for insights by means of an easy-to-interpret parameter. Hence, we seek to develop a new estimand-driven method to compare the effectiveness of two treatments in the context of right-censored survival data with matched pairs. With the help of competing risks techniques, the so-called relative treatment effect is estimated. This estimand describes the probability that individuals under Treatment 1 have a longer lifetime than comparable individuals under Treatment 2. We derive hypothesis tests and confidence intervals based on a studentized version of the estimator, where resampling-based inference is established by means of a randomization method. In a simulation study, we demonstrate for numerous sample sizes and different amounts of censoring that the developed test exhibits a good power. Finally, we apply the methodology to a well-known benchmark data set from a trial with patients suffering from diabetic retinopathy.

Principal quantile treatment effect estimation using principal scores.

Intercurrent events and estimands play a key role in defining the treatment effects of interest precisely. Sometimes the median or other quantiles of outcomes in a principal stratum according to potential occurrence of intercurrent events are of interest in randomized clinical trials. Naïve analyses such as those based on the observed occurrence of the intercurrent events lead to biased results. Therefore, we propose principal quantile treatment effect estimators that can nonparametrically estimate the distribution of potential outcomes by principal score weighting without relying on the exclusion restriction assumption. Our simulation studies show that the proposed method works in situations where the median or quantiles may be regarded as the preferred population-level summary over the mean. We illustrate our proposed method by using data from a randomized controlled trial conducted on patients with nonerosive reflux disease.

End points in clinical trials in diffuse large B-cell lymphoma: time for more dialogue?

We observed lack of clarity and consistency in end point definitions of large randomized clinical trials in diffuse large B-cell lymphoma. These inconsistencies are such that trials might, in fact, address different clinical questions. They complicate interpretation of results, including comparisons across studies. Problems arise from different ways to account for events occurring after randomization including absence of improvement in disease status, treatment discontinuation or the initiation of new therapy. We call for more dialogue between stakeholders to define with clarity the questions of interest and corresponding end points. We illustrate that assessing different end point rules across a range of plausible patient journeys can be a powerful tool to facilitate such a discussion and contribute to better understanding of patient-relevant end points.

What is this article about? This article talks about the lack of clarity and consistency in the definitions of outcomes used in clinical trials that investigate new treatments for diffuse large B-cell lymphoma. This is mainly due to how these different outcome definitions handle events such as absence of improvement in disease status, treatment discontinuation or initiation of new treatment. The authors discuss how these inconsistencies make it hard to interpret the results of individual clinical trials and to compare results across clinical trials.Why is it important? Defining the above events and consequently defining outcomes affects what we can learn from the trials and can lead to different results. Some approaches may not reflect good and bad outcomes for patients appropriately. This makes it challenging for patients, physicians, health authorities and payors to understand the true benefit of treatments under investigation and which one is better.What are the key take-aways? This article serves as a call-to-action for more dialogue among all stakeholders involved in drug development and the decision-making process related to drug evaluations. There is an urgent need for clinical trials to be designed with more clarity and consistency on what is being measured so that relevant questions for patients and prescribing physicians are addressed. Understanding patient journeys will be key to successfully understand what truly matters to patients and how to measure the benefit of new treatments. Such discussions will contribute toward more clarity and consistency in the evaluation of new treatments.

Scaling and interpreting treatment effects in clinical trials using restricted mean survival time.

BACKGROUND: Restricted mean survival time is the expected duration of survival up to a chosen time of restriction τ. For comparison studies, the difference in restricted mean survival times between two groups provides a summary measure of the treatment effect that is free of assumptions regarding the relative shape of the two survival curves, such as proportional hazards. However, it can be difficult to judge the magnitude of the effect from a comparison of restricted means due to the truncation of observation at time τ. METHODS: In this article, we describe additional ways of expressing the treatment effect based on restricted means that can be helpful in this regard. These include the ratio of restricted means, the ratio of life-years (or time) lost, and the average integrated difference between the survival curves, equal to the difference in restricted means divided by τ. These alternative metrics are straightforward to calculate and provide a means for scaling the effect size as an aid to interpretation. Examples from two randomized, multicenter clinical trials in prostate cancer, NRG/RTOG 0521 and NRG/RTOG 0534, with primary endpoints of overall survival and biochemical/radiological progression-free survival, respectively, are presented to illustrate the ideas. RESULTS: The four effect measures (restricted mean survival time difference, restricted mean survival time ratio, time lost ratio, and average survival rate difference) were 0.45 years, 1.05, 0.81, and 0.038 for RTOG 0521 and 1.36 years, 1.17, 0.56, and 0.12 for RTOG 0534 with τ = 12 and 11 years, respectively. Thus, for example, the 0.45-year difference in the first trial translates into a 19% reduction in time lost and a 3.8% average absolute difference between the survival curves over the 12-year horizon, a modest effect size, whereas the 1.36-year difference in the second trial corresponds to a 44% reduction in time lost and a 12% absolute survival difference, a rather large effect. CONCLUSIONS: In addition to the difference in restricted mean survival times, these alternative measures can be helpful in determining whether the magnitude of the treatment effect is clinically meaningful.

Causal interpretation of the hazard ratio in randomized clinical trials.

BACKGROUND: Although the hazard ratio has no straightforward causal interpretation, clinical trialists commonly use it as a measure of treatment effect. METHODS: We review the definition and examples of causal estimands. We discuss the causal interpretation of the hazard ratio from a two-arm randomized clinical trial, and the implications of proportional hazards assumptions in the context of potential outcomes. We illustrate the application of these concepts in a synthetic model and in a model of the time-varying effects of COVID-19 vaccination. RESULTS: We define causal estimands as having either an individual-level or population-level interpretation. Difference-in-expectation estimands are both individual-level and population-level estimands, whereas without strong untestable assumptions the causal rate ratio and hazard ratio have only population-level interpretations. We caution users against making an incorrect individual-level interpretation, emphasizing that in general a hazard ratio does not on average change each individual's hazard by a factor. We discuss a potentially valid interpretation of the constant hazard ratio as a population-level causal effect under the proportional hazards assumption. CONCLUSION: We conclude that the population-level hazard ratio remains a useful estimand, but one must interpret it with appropriate attention to the underlying causal model. This is especially important for interpreting hazard ratios over time.

A safety estimand for late phase clinical trials where the analysis period varies over the subjects.

BACKGROUND/AIMS: Evaluating safety is as important as evaluating efficacy in a clinical trial, yet the tradition for safety analysis is rudimentary. This article explores more complex methodologies for safety evaluation, with the aim of improving the interpretability, as well as generalizability, of the results. METHODS: For studies where the analysis periods vary over the subjects, using the International Council for Harmonisation estimand framework, we construct a formal estimand that could be used in the setting of safety surveillance that answers the clinical question of 'What is the magnitude of the increase in risk of experiencing an adverse event if the treatment is taken, as prescribed, for a specific period of time?'. Estimation methodologies for this estimand are also discussed. RESULTS: The proposed estimand is similar to that found in the efficacy analyses of time to event data (e.g. in outcome studies), with the key difference of utilization of hypothetical intercurrent event strategy for the intercurrent event of treatment discontinuation. This is motivated by what we perceive to be a key difference for the safety objective compared to efficacy objectives, namely a desire for sensitivity (i.e. greater possibility of detecting a negative impact of the drug, if such exists) as opposed to the need to prove a positive effect of the drug in a conservative manner. CONCLUSION: It is valuable, and possible, to use the International Council for Harmonisation estimand framework not only for efficacy but also for safety evaluation, with the estimand driven by an interpretable, and relevant, clinical question.

[Special issue PRO] Considering endpoints for comparative tolerability of cancer treatments using patient report given the estimand framework.

Regulatory agencies are advancing the use of systematic approaches to collect patient experience data, including patient-reported outcomes (PROs), in cancer clinical trials to inform regulatory decision-making. Due in part to clinician under-reporting of symptomatic adverse events, there is a growing recognition that evaluation of cancer treatment tolerability should include the patient experience, both in terms of the overall side effect impact and symptomatic adverse events. Methodologies around implementation, analysis, and interpretation of "patient" reported tolerability are under development, and current approaches are largely descriptive. There is robust guidance for use of PROs as efficacy endpoints to compare cancer treatments, but it is unclear to what extent this can be relied-upon to develop tolerability endpoints. An important consideration when developing endpoints to compare tolerability between treatments is the linkage of trial design, objectives, and statistical analysis. Despite interest in and frequent collection of PRO data in oncology trials, heterogeneity in analyses and unclear PRO objectives mean that design, objectives, and analysis may not be aligned, posing substantial challenges for the interpretation of results. The recent ICH E9 (R1) estimand framework represents an opportunity to help address these challenges. Efforts to apply the estimand framework in the context of PROs have primarily focused on efficacy outcomes. In this paper, we discuss considerations for comparing the patient-reported tolerability of different treatments in an oncology trial context.

Application of estimand framework to the design and analysis of multi-regional clinical trials.

With the increasing globalization of drug development and the publication of the International Council for Harmonisation (ICH) E17 guideline (ICH International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use 2017), multi-regional clinical trials (MRCTs) have become a preferred option to accelerate the availability of new medical products by design, execution and simultaneous submission under one protocol. MRCTs, with the participation of all major regions including countries from both developed and emerging markets, surely make new drug development more efficient. Even though the proposed estimand framework (ICH E9 (R1) (2019), came later in 2019 and was not mentioned in ICH E17, the application of the estimand framework has the potential to enhance the design, execution, and analysis in MRCTs. Defining an estimand within the regional context in MRCTs is an important issue that requires careful consideration. Given that consistency evaluation of treatment effects across regions is critical in MRCTs, the utilization of the estimand framework for regional consistency evaluation is also worth discussion. This paper aims to address these two questions. The five attributes of the estimand definition are discussed within a multi-regional context. It is imperative to thoroughly consider regional intrinsic/extrinsic factors when planning the estimand and estimation of MRCTs. A holistic approach is summarized to conduct consistency evaluation. When a regional inconsistency is observed, the possible reasons need to be further explored under five attributes of the estimand framework. Two real case studies are discussed to illustrate the application of the estimand framework in the consistency evaluation.

Considerations for master protocols using external controls.

There has been an increasing use of master protocols in oncology clinical trials because of its efficiency to accelerate cancer drug development and flexibility to accommodate multiple substudies. Depending on the study objective and design, a master protocol trial can be a basket trial, an umbrella trial, a platform trial, or any other form of trials in which multiple investigational products and/or subpopulations are studied under a single protocol. Master protocols can use external data and evidence (e.g. external controls) for treatment effect estimation, which can further improve efficiency of master protocol trials. This paper provides an overview of different types of external controls and their unique features when used in master protocols. Some key considerations in master protocols with external controls are discussed including construction of estimands, assessment of fit-for-use real-world data, and considerations for different types of master protocols. Similarities and differences between regular randomized controlled trials and master protocols when using external controls are discussed. A targeted learning-based causal roadmap is presented which constitutes three key steps: (1) define a target statistical estimand that aligns with the causal estimand for the study objective, (2) use an efficient estimator to estimate the target statistical estimand and its uncertainty, and (3) evaluate the impact of causal assumptions on the study conclusion by performing sensitivity analyses. Two illustrative examples for master protocols using external controls are discussed for their merits and possible improvement in causal effect estimation.

Current developments of the estimand concept.

Since the introduction of the estimand in therapeutical studies, several adaptions have been developed. This short article highlights the important aspects of the estimand concept. A literature research was conducted to identify different extensions to this framework. Different modified strategies for intercurrent events are presented, as well as examples of methods to implement the estimand in clinical studies. The article reflects that the estimand is an ongoing research field with further exploration.

Application of hypothetical strategies in acute pain.

Since the publication of ICH E9 (R1), "Addendum to statistical principles for clinical trials: on choosing appropriate estimands and defining sensitivity analyses in clinical trials," there has been a lot of debate about the hypothetical strategy for handling intercurrent events. Arguments against the hypothetical strategy are twofold: (1) the clinical question has limited clinical/regulatory interest; (2) the estimation may need strong statistical assumptions. In this article, we provide an example of a hypothetical strategy handling use of rescue medications in the acute pain setting. We argue that the treatment effect of a drug that is attributable to the treatment alone is the clinical question of interest and is important to regulators. The hypothetical strategy is important when developing non-opioid treatment as it estimates the treatment effect due to treatment during the pre-specified evaluation period whereas the treatment policy strategy does not. Two widely acceptable and non-controversial clinical inputs are required to construct a reasonable estimator. More importantly, this estimator does not rely on additional strong statistical assumptions and is considered reasonable for regulatory decision making. In this article, we point out examples where estimators for a hypothetical strategy can be constructed without any strong additional statistical assumptions besides acceptable clinical inputs. We also showcase a new way to obtain estimation based on disease specific clinical knowledge instead of strong statistical assumptions. In the example presented, we clearly demonstrate the advantages of the hypothetical strategy compared to alternative strategies including the treatment policy strategy and a composite variable strategy.

Applying the Estimand Framework to Non­Inferiority Trials.

Most published applications of the estimand framework have focused on superiority trials. However, non-inferiority trials present specific challenges compared to superiority trials. The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use notes in their addendum on estimands and sensitivity analysis in clinical trials that there may be special considerations to the implementation of estimands in clinical trials with a non-inferiority objective yet provides little guidance. This paper discusses considerations that trial teams should make when defining estimands for a clinical trial with a non-inferiority objective. We discuss how the pre-addendum way of establishing non-inferiority can be embraced by the estimand framework including a discussion of the role of the Per Protocol analysis set. We examine what clinical questions of interest can be formulated in the context of non-inferiority trials and outline why we do not think it is sensible to describe an estimand as 'conservative'. The impact of the estimand framework on key considerations in non-inferiority trials such as whether trials should have more than one primary estimand, the choice of non-inferiority margin, assay sensitivity, switching from non-inferiority to superiority and estimation are discussed. We conclude by providing a list of recommendations, and important considerations for defining estimands for trials with a non-inferiority objective.

Decentralized Clinical Trials: Scientific Considerations Through the Lens of the Estimand Framework.

While industry and regulators' interest in decentralized clinical trials (DCTs) is long-standing, the Covid-19 pandemic accelerated and broadened the adoption and experience with these trials. The key idea in decentralization is bringing the clinical trial design, typically on-site, closer to the patient's experience (on-site or off-site). Thus, potential benefits of DCTs include reducing the burden of participation in trials, broadening access to a more diverse population, or using innovative endpoints collected off-site. This paper helps researchers to carefully evaluate the added value and the implications of DCTs beyond the operational aspects of their implementation. The proposed approach is to use the ICH E9(R1) estimand framework to guide the strategic decisions around each decentralization component. Furthermore, the framework can guide the process for clinical trialists to systematically consider the implications of decentralization, in turn, for each attribute of the estimand. We illustrate the use of this approach with a fully DCT case study and show that the proposed systematic process can uncover the scientific opportunities, assumptions, and potential risks associated with a possible use of decentralization components in the design of a trial. This process can also highlight the benefits of specifying estimand attributes in a granular way. Thus, we demonstrate that bringing a decentralization component into the design will not only impact estimators and estimation but can also correspond to addressing more granular questions, thereby uncovering new target estimands.

Estimands in CNS trials - A review of strategies for addressing intercurrent events.

Background: The estimands framework represents a significant innovation for the design, conduct, analysis, and interpretation of clinical trials. An aim of the framework is to increase precision and transparency on the handling of intercurrent events (IEs), defined as events occurring after treatment initiation and affecting the endpoint. While the experience in constructing and reporting estimands in the published literature is limited, developers performing confirmatory studies are already making use of the new paradigm, allowing to survey the strategies proposed by applicants and endorsed by regulators. Methods: To identify strategies for handling IEs in confirmatory central nervous system (CNS) trials, we searched scientific advice letters issued by the European Medicines Agency (EMA) between 2017 and 2022. We developed a categorisation of the IEs and classified, according to the strategies defined in the framework, the strategies proposed by the Applicants and recommended by the agency. Strategies proposed and recommended were summarised by category of IEs, and the rationale for the choices was analysed qualitatively. Results: In total, 170 IEs were identified in 52 confirmatory trials. A clear preference for the treatment policy strategy for treatment discontinuation and for the hypothetical strategy for pandemic-related disruptions was identified. For other categories of IEs, there are more mixed patterns. Discussion: This study highlights the multidimensional nature of choosing a strategy for an IE. For different occurring IEs in confirmatory CNS trials different strategies are of regulatory interest, depending on the trial objective, underlying disease properties, rarity of disease, as well as frequency and timing of IEs and their relatedness to the disease.

Estimand for non-inferiority influenza vaccine immunogenicity trials.

According to recent regulatory guidance, clinical trial objectives should be translated into estimands, i.e., precise descriptions of that what is to be estimated. Hence, estimands are to be formulated for influenza vaccine immunogenicity (IVI) trials, notably for one of the most popular immunogenicity trials, the non-inferiority trial. In this paper an estimand for this trial design is proposed. An estimand should state how intercurrent events are handled. Intercurrent events are events that occur after the start of the trial and that affect the endpoint's measurement or interpretation or prevent its observation. In IVI trials the intercurrent events of interest are immunological intercurrent events (IIEs). Major IIEs are identified, i.e., protocol deviations occurring during the trial that affect immunogenicity endpoints, and the consequences for the trial data collection are discussed. In the statistical analysis endpoint values that are missing or excluded from the analysis due to IIEs are to be substituted by plausible values, by means of multiple imputing. Replacing values are based on predictors of the endpoint. A distinction is made between mandatory and non-mandatory predictors. Mandatory predictors are predictors necessary to prevent biased prediction. Non-mandatory predictors are predictors that limit the additional variance due to the imputing. The four steps of the multiple imputing are explained, and available software is listed.

Response to discussant comments on "NMA, the first 20 years".

We respond to discussant comments on our paper "Twenty years of network meta-analysis: Continuing controversies and recent developments" (https://doi.org/10.1002/jrsm.1700) and raise some additional points for consideration, including: the way in which methodological guidance is generated; integration of the estimand framework with evidence synthesis; and implications of the European Joint Clinical Assessment. We ask: what properties are required of population adjustment methods to enable transparent and consistent decision-making? We also ask why individual patient data is not routinely made available to re-imbursement authorities and clinical guideline developers.

Marginal versus conditional rate estimation for count and recurrent event data with an estimand framework.

Count and recurrent event endpoints are common key efficacy endpoints in clinical research. For example, in clinical research of pulmonary diseases such as chronic obstructive pulmonary disease (COPD) or asthma, the reduction of the occurrence of a recurrent event, pulmonary exacerbation (PEx) caused by acute respiratory symptoms, is often used to measure the treatment effect. The occurrence of PEx is often analyzed with nonlinear models, such as Poisson regression or Negative Binomial regression. It is observed that model-estimated within-group PEx rates are often lower than the descriptive statistics of within-group PEx rates. Motivated by this observation, we explore their relationship mathematically and demonstrate that it is due to the difference between conditional PEx rates and population-level PEx rates (marginal rates). Our findings corroborate the recent FDA guidance (2023) [1], which discusses considerations for covariate adjustment in nonlinear models, and that conditional or subgroup treatment effects with covariate adjustment may differ from marginal treatment effects. In this article, we demonstrate how covariate adjustment impacts the estimation of event rates and rate ratios with both closed form and simulation studies. Additionally, following the ICH E9 addendum on the estimand framework [2], we discuss the estimand framework for count and recurrent event data.

Challenges and Lessons Learned in Autologous Chimeric Antigen Receptor T-Cell Therapy Development from a Statistical Perspective.

Chimeric antigen receptor (CAR) T-cell therapy is a human gene therapy product where T cells from a patient are genetically modified to enable them to recognize desired target antigen(s) more effectively. In recent years, promising antitumor activity has been seen with autologous CAR T cells. Since 2017, six CAR T-cell therapies for the treatment of hematological malignancies have been approved by the Food and Drug Administration (FDA). Despite the rapid progress of CAR T-cell therapies, considerable statistical challenges still exist for this category of products across all phases of clinical development that need to be addressed. These include (but not limited to) dose finding strategy, implementation of the estimand framework, use of real-world data in contextualizing single-arm CAR T trials, analysis of safety data and long-term follow-up studies. This paper is the first step in summarizing and addressing these statistical hurdles based on the development of the six approved CAR T-cell products.

Applying the estimand and target trial frameworks to external control analyses using observational data: a case study in the solid tumor setting.

Introduction: In causal inference, the correct formulation of the scientific question of interest is a crucial step. The purpose of this study was to apply causal inference principles to external control analysis using observational data and illustrate the process to define the estimand attributes. Methods: This study compared long-term survival outcomes of a pooled set of three previously reported randomized phase 3 trials studying patients with metastatic non-small cell lung cancer receiving front-line chemotherapy and similar patients treated with front-line chemotherapy as part of routine clinical care. Causal inference frameworks were applied to define the estimand aligned with the research question and select the estimator to estimate the estimand of interest. Results: The estimand attributes of the ideal trial were defined using the estimand framework. The target trial framework was used to address specific issues in defining the estimand attributes using observational data from a nationwide electronic health record-derived de-identified database. The two frameworks combined allow to clearly define the estimand and the aligned estimator while accounting for key baseline confounders, index date, and receipt of subsequent therapies. The hazard ratio estimate (point estimate with 95% confidence interval) comparing the randomized clinical trial pooled control arm with the external control was close to 1, which is indicative of similar survival between the two arms. Discussion: The proposed combined framework provides clarity on the causal contrast of interest and the estimator to adopt, and thus facilitates design and interpretation of the analyses.