Comments on 'standard and reference-based conditional mean imputation': Regulators and trial statisticians be aware!

Accurate frequentist performance of a method is desirable in confirmatory clinical trials, but is not sufficient on its own to justify the use of a missing data method. Reference-based conditional mean imputation, with variance estimation justified solely by its frequentist performance, has the surprising and undesirable property that the estimated variance becomes smaller the greater the number of missing observations; as explained under jump-to-reference it effectively forces the true treatment effect to be exactly zero for patients with missing data.

Comment on Oberman & Vink: Should we fix or simulate the complete data in simulation studies evaluating missing data methods?

For simulation studies that evaluate methods of handling missing data, we argue that generating partially observed data by fixing the complete data and repeatedly simulating the missingness indicators is a superficially attractive idea but only rarely appropriate to use.

Multiple imputation of missing data under missing at random: including a collider as an auxiliary variable in the imputation model can induce bias.

Epidemiological studies often have missing data, which are commonly handled by multiple imputation (MI). In MI, in addition to those required for the substantive analysis, imputation models often include other variables ("auxiliary variables"). Auxiliary variables that predict the partially observed variables can reduce the standard error (SE) of the MI estimator and, if they also predict the probability that data are missing, reduce bias due to data being missing not at random. However, guidance for choosing auxiliary variables is lacking. We examine the consequences of a poorly chosen auxiliary variable: if it shares a common cause with the partially observed variable and the probability that it is missing (i.e., it is a "collider"), its inclusion can induce bias in the MI estimator and may increase the SE. We quantify, both algebraically and by simulation, the magnitude of bias and SE when either the exposure or outcome is incomplete. When the substantive analysis outcome is partially observed, the bias can be substantial, relative to the magnitude of the exposure coefficient. In settings in which a complete records analysis is valid, the bias is smaller when the exposure is partially observed. However, bias can be larger if the outcome also causes missingness in the exposure. When using MI, it is important to examine, through a combination of data exploration and considering plausible casual diagrams and missingness mechanisms, whether potential auxiliary variables are colliders.

A new approach to evaluating loop inconsistency in network meta-analysis.

In network meta-analysis, studies evaluating multiple treatment comparisons are modeled simultaneously, and estimation is informed by a combination of direct and indirect evidence. Network meta-analysis relies on an assumption of consistency, meaning that direct and indirect evidence should agree for each treatment comparison. Here we propose new local and global tests for inconsistency and demonstrate their application to three example networks. Because inconsistency is a property of a loop of treatments in the network meta-analysis, we locate the local test in a loop. We define a model with one inconsistency parameter that can be interpreted as loop inconsistency. The model builds on the existing ideas of node-splitting and side-splitting in network meta-analysis. To provide a global test for inconsistency, we extend the model across multiple independent loops with one degree of freedom per loop. We develop a new algorithm for identifying independent loops within a network meta-analysis. Our proposed models handle treatments symmetrically, locate inconsistency in loops rather than in nodes or treatment comparisons, and are invariant to choice of reference treatment, making the results less dependent on model parameterization. For testing global inconsistency in network meta-analysis, our global model uses fewer degrees of freedom than the existing design-by-treatment interaction approach and has the potential to increase power. To illustrate our methods, we fit the models to three network meta-analyses varying in size and complexity. Local and global tests for inconsistency are performed and we demonstrate that the global model is invariant to choice of independent loops.

Handling intercurrent events and missing data in non-inferiority trials using the estimand framework: A tuberculosis case study.

INTRODUCTION: The ICH E9 addendum outlining the estimand framework for clinical trials was published in 2019 but provides limited guidance around how to handle intercurrent events for non-inferiority studies. Once an estimand is defined, it is also unclear how to deal with missing values using principled analyses for non-inferiority studies. METHODS: Using a tuberculosis clinical trial as a case study, we propose a primary estimand, and an additional estimand suitable for non-inferiority studies. For estimation, multiple imputation methods that align with the estimands for both primary and sensitivity analysis are proposed. We demonstrate estimation methods using the twofold fully conditional specification multiple imputation algorithm and then extend and use reference-based multiple imputation for a binary outcome to target the relevant estimands, proposing sensitivity analyses under each. We compare the results from using these multiple imputation methods with those from the original study. RESULTS: Consistent with the ICH E9 addendum, estimands can be constructed for a non-inferiority trial which improves on the per-protocol/intention-to-treat-type analysis population previously advocated, involving respectively a hypothetical or treatment policy strategy to handle relevant intercurrent events. Results from using the 'twofold' multiple imputation approach to estimate the primary hypothetical estimand, and using reference-based methods for an additional treatment policy estimand, including sensitivity analyses to handle the missing data, were consistent with the original study's reported per-protocol and intention-to-treat analysis in failing to demonstrate non-inferiority. CONCLUSIONS: Using carefully constructed estimands and appropriate primary and sensitivity estimators, using all the information available, results in a more principled and statistically rigorous approach to analysis. Doing so provides an accurate interpretation of the estimand.

Multiple imputation of missing data under missing at random: compatible imputation models are not sufficient to avoid bias if they are mis-specified.

OBJECTIVES: Epidemiological studies often have missing data, which are commonly handled by multiple imputation (MI). Standard (default) MI procedures use simple linear covariate functions in the imputation model. We examine the bias that may be caused by acceptance of this default option and evaluate methods to identify problematic imputation models, providing practical guidance for researchers. STUDY DESIGN AND SETTING: Using simulation and real data analysis, we investigated how imputation model mis-specification affected MI performance, comparing results with complete records analysis (CRA). We considered scenarios in which imputation model mis-specification occurred because (i) the analysis model was mis-specified or (ii) the relationship between exposure and confounder was mis-specified. RESULTS: Mis-specification of the relationship between outcome and exposure, or between exposure and confounder, can cause biased CRA and MI estimates (in addition to any bias in the full-data estimate due to analysis model mis-specification). MI by predictive mean matching can mitigate model mis-specification. Methods for examining model mis-specification were effective in identifying mis-specified relationships. CONCLUSION: When using MI methods that assume data are MAR, compatibility between the analysis and imputation models is necessary, but not sufficient to avoid bias. We propose a step-by-step procedure for identifying and correcting mis-specification of imputation models.

Propensity score matching after multiple imputation when a confounder has missing data.

One of the main challenges when using observational data for causal inference is the presence of confounding. A classic approach to account for confounding is the use of propensity score techniques that provide consistent estimators of the causal treatment effect under four common identifiability assumptions for causal effects, including that of no unmeasured confounding. Propensity score matching is a very popular approach which, in its simplest form, involves matching each treated patient to an untreated patient with a similar estimated propensity score, that is, probability of receiving the treatment. The treatment effect can then be estimated by comparing treated and untreated patients within the matched dataset. When missing data arises, a popular approach is to apply multiple imputation to handle the missingness. The combination of propensity score matching and multiple imputation is increasingly applied in practice. However, in this article we demonstrate that combining multiple imputation and propensity score matching can lead to over-coverage of the confidence interval for the treatment effect estimate. We explore the cause of this over-coverage and we evaluate, in this context, the performance of a correction to Rubin's rules for multiple imputation proposed by finding that this correction removes the over-coverage.

Effectiveness of a behavioural intervention delivered by text messages (safetxt) on sexually transmitted reinfections in people aged 16-24 years: randomised controlled trial.

OBJECTIVE: To quantify the effects of a series of text messages (safetxt) delivered in the community on incidence of chlamydia and gonorrhoea reinfection at one year in people aged 16-24 years. DESIGN: Parallel group randomised controlled trial. SETTING: 92 sexual health clinics in the United Kingdom. PARTICIPANTS: People aged 16-24 years with a diagnosis of, or treatment for, chlamydia, gonorrhoea, or non-specific urethritis in the past two weeks who owned a mobile phone. INTERVENTIONS: 3123 participants assigned to the safetxt intervention received a series of text messages to improve sex behaviours: four texts daily for days 1-3, one or two daily for days 4-28, two or three weekly for month 2, and 2-5 monthly for months 3-12. 3125 control participants received a monthly text message for one year asking for any change to postal or email address. It was hypothesised that safetxt would reduce the risk of chlamydia and gonorrhoea reinfection at one year by improving three key safer sex behaviours: partner notification at one month, condom use, and sexually transmitted infection testing before unprotected sex with a new partner. Care providers and outcome assessors were blind to allocation. MAIN OUTCOME MEASURES: The primary outcome was the cumulative incidence of chlamydia or gonorrhoea reinfection at one year, assessed by nucleic acid amplification tests. Safety outcomes were self-reported road traffic incidents and partner violence. All analyses were by intention to treat. RESULTS: 6248 of 20 476 people assessed for eligibility between 1 April 2016 and 23 November 2018 were randomised. Primary outcome data were available for 4675/6248 (74.8%). At one year, the cumulative incidence of chlamydia or gonorrhoea reinfection was 22.2% (693/3123) in the safetxt arm versus 20.3% (633/3125) in the control arm (odds ratio 1.13, 95% confidence interval 0.98 to 1.31). The number needed to harm was 64 (95% confidence interval number needed to benefit 334 to ∞ to number needed to harm 24) The risk of road traffic incidents and partner violence was similar between the groups. CONCLUSIONS: The safetxt intervention did not reduce chlamydia and gonorrhoea reinfections at one year in people aged 16-24 years. More reinfections occurred in the safetxt group. The results highlight the need for rigorous evaluation of health communication interventions. TRIAL REGISTRATION: ISRCTN registry ISRCTN64390461.

Substantive model compatible multilevel multiple imputation: A joint modeling approach.

BACKGROUND: Substantive model compatible multiple imputation (SMC-MI) is a relatively novel imputation method that is particularly useful when the analyst's model includes interactions, non-linearities, and/or partially observed random slope variables. METHODS: Here we thoroughly investigate a SMC-MI strategy based on joint modeling of the covariates of the analysis model. We provide code to apply the proposed strategy and we perform an extensive simulation work to test it in various circumstances. We explore the impact on the results of various factors, including whether the missing data are at the individual or cluster level, whether there are non-linearities and whether the imputation model is correctly specified. Finally, we apply the imputation methods to the motivating example data. RESULTS: SMC-JM appears to be superior to standard JM imputation, particularly in presence of large variation in random slopes, non-linearities, and interactions. Results seem to be robust to slight mis-specification of the imputation model for the covariates. When imputing level 2 data, enough clusters have to be observed in order to obtain unbiased estimates of the level 2 parameters. CONCLUSIONS: SMC-JM is preferable to standard JM imputation in presence of complexities in the analysis model of interest, such as non-linearities or random slopes.

Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands.

OBJECTIVES: To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials. DESIGN: Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals. DATA SOURCE: PubMed search in February 2021. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded. MAIN OUTCOME MEASURES: Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated. RESULTS: 255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die). CONCLUSIONS: The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions. SYSTEMATIC REVIEW REGISTRATION: PROSPERO CRD42021238053.

Rethinking intercurrent events in defining estimands for tuberculosis trials.

BACKGROUND/AIMS: Tuberculosis remains one of the leading causes of death from an infectious disease globally. Both choices of outcome definitions and approaches to handling events happening post-randomisation can change the treatment effect being estimated, but these are often inconsistently described, thus inhibiting clear interpretation and comparison across trials. METHODS: Starting from the ICH E9(R1) addendum's definition of an estimand, we use our experience of conducting large Phase III tuberculosis treatment trials and our understanding of the estimand framework to identify the key decisions regarding how different event types are handled in the primary outcome definition, and the important points that should be considered in making such decisions. A key issue is the handling of intercurrent (i.e. post-randomisation) events (ICEs) which affect interpretation of or preclude measurement of the intended final outcome. We consider common ICEs including treatment changes and treatment extension, poor adherence to randomised treatment, re-infection with a new strain of tuberculosis which is different from the original infection, and death. We use two completed tuberculosis trials (REMoxTB and STREAM Stage 1) as illustrative examples. These trials tested non-inferiority of new tuberculosis treatment regimens versus a control regimen. The primary outcome was a binary composite endpoint, 'favourable' or 'unfavourable', which was constructed from several components. RESULTS: We propose the following improvements in handling the above-mentioned ICEs and loss to follow-up (a post-randomisation event that is not in itself an ICE). First, changes to allocated regimens should not necessarily be viewed as an unfavourable outcome; from the patient perspective, the potential harms associated with a change in the regimen should instead be directly quantified. Second, handling poor adherence to randomised treatment using a per-protocol analysis does not necessarily target a clear estimand; instead, it would be desirable to develop ways to estimate the treatment effects more relevant to programmatic settings. Third, re-infection with a new strain of tuberculosis could be handled with different strategies, depending on whether the outcome of interest is the ability to attain culture negativity from infection with any strain of tuberculosis, or specifically the presenting strain of tuberculosis. Fourth, where possible, death could be separated into tuberculosis-related and non-tuberculosis-related and handled using appropriate strategies. Finally, although some losses to follow-up would result in early treatment discontinuation, patients lost to follow-up before the end of the trial should not always be classified as having an unfavourable outcome. Instead, loss to follow-up should be separated from not completing the treatment, which is an ICE and may be considered as an unfavourable outcome. CONCLUSION: The estimand framework clarifies many issues in tuberculosis trials but also challenges trialists to justify and improve their outcome definitions. Future trialists should consider all the above points in defining their outcomes.

Neighbourhood prevalence-to-notification ratios for adult bacteriologically-confirmed tuberculosis reveals hotspots of underdiagnosis in Blantyre, Malawi.

Local information is needed to guide targeted interventions for respiratory infections such as tuberculosis (TB). Case notification rates (CNRs) are readily available, but systematically underestimate true disease burden in neighbourhoods with high diagnostic access barriers. We explored a novel approach, adjusting CNRs for under-notification (P:N ratio) using neighbourhood-level predictors of TB prevalence-to-notification ratios. We analysed data from 1) a citywide routine TB surveillance system including geolocation, confirmatory mycobacteriology, and clinical and demographic characteristics of all registering TB patients in Blantyre, Malawi during 2015-19, and 2) an adult TB prevalence survey done in 2019. In the prevalence survey, consenting adults from randomly selected households in 72 neighbourhoods had symptom-plus-chest X-ray screening, confirmed with sputum smear microscopy, Xpert MTB/Rif and culture. Bayesian multilevel models were used to estimate adjusted neighbourhood prevalence-to-notification ratios, based on summarised posterior draws from fitted adult bacteriologically-confirmed TB CNRs and prevalence. From 2015-19, adult bacteriologically-confirmed CNRs were 131 (479/371,834), 134 (539/415,226), 114 (519/463,707), 56 (283/517,860) and 46 (258/578,377) per 100,000 adults per annum, and 2019 bacteriologically-confirmed prevalence was 215 (29/13,490) per 100,000 adults. Lower educational achievement by household head and neighbourhood distance to TB clinic was negatively associated with CNRs. The mean neighbourhood P:N ratio was 4.49 (95% credible interval [CrI]: 0.98-11.91), consistent with underdiagnosis of TB, and was most pronounced in informal peri-urban neighbourhoods. Here we have demonstrated a method for the identification of neighbourhoods with high levels of under-diagnosis of TB without the requirement for a prevalence survey; this is important since prevalence surveys are expensive and logistically challenging. If confirmed, this approach may support more efficient and effective targeting of intensified TB and HIV case-finding interventions aiming to accelerate elimination of urban TB.

Training and capacity building in medical statistics in Sub-Saharan Africa: Impact of the London School of Hygiene & Tropical Medicine MSc in Medical Statistics, 1969 to 2021.

Since its inception in 1969, the MSc in medical statistics program has placed a high priority on training students from Africa. In this article, we review how the program has shaped, and in turn been shaped by, two substantial capacity building initiatives: (a) a fellowship program, funded by the UK Medical Research Council, and run through the International Statistical Epidemiology Group at the LSHTM, and (b) the Sub-Saharan capacity building in Biostatistics (SSACAB) initiative, administered through the Developing Excellence in Leadership, Training and Science in Africa (DELTAS) program of the African Academy of Sciences. We reflect on the impact of both initiatives, and the implications for future work in this area.

Observational retrospective study calculating health service costs of patients receiving surgery for chronic rhinosinusitis in England, using linked patient-level primary and secondary care electronic data.

OBJECTIVES: Chronic rhinosinusitis (CRS) symptoms are experienced by an estimated 11% of UK adults, and symptoms have major impacts on quality of life. Data from UK and elsewhere suggest high economic burden of CRS, but detailed cost information and economic analyses regarding surgical pathway are lacking. This paper estimates healthcare costs for patients receiving surgery for CRS in England. DESIGN: Observational retrospective study examining cost of healthcare of patients receiving CRS surgery. SETTING: Linked electronic health records from the Clinical Practice Research Datalink, Hospital Episode Statistics and Office for National Statistics databases in England. PARTICIPANTS: A phenotyping algorithm using medical ontology terms identified 'definite' CRS cases who received CRS surgery. Patients were registered with a general practice in England. Data covered the period 1997-2016. A cohort of 13 462 patients had received surgery for CRS, with 9056 (67%) having confirmed nasal polyps. OUTCOME MEASURES: Information was extracted on numbers and types of primary care prescriptions and consultations, and inpatient and outpatient hospital investigations and procedures. Resource use was costed using published sources. RESULTS: Total National Health Service costs in CRS surgery patients were £2173 over 1 year including surgery. Total costs per person-quarter were £1983 in the quarter containing surgery, mostly comprising surgical inpatient care costs (£1902), and around £60 per person-quarter in the 2 years before and after surgery, of which half were outpatient costs. Outpatient and primary care costs were low compared with the peak in inpatient costs at surgery. The highest outpatient expenditure was on CT scans, peaking in the quarter preceding surgery. CONCLUSIONS: We present the first study of costs to the English healthcare system for patients receiving surgery for CRS. The total aggregate costs provide a further impetus for trials to evaluate the relative benefit of surgical intervention.

Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: Application to a melanoma network.

BACKGROUND: Synthesis of clinical effectiveness from multiple trials is a well-established component of decision-making. Time-to-event outcomes are often synthesised using the Cox proportional hazards model assuming a constant hazard ratio over time. However, with an increasing proportion of trials reporting treatment effects where hazard ratios vary over time and with differing lengths of follow-up across trials, alternative synthesis methods are needed. OBJECTIVES: To compare and contrast five modelling approaches for synthesis of time-to-event outcomes and provide guidance on key considerations for choosing between the modelling approaches. METHODS: The Cox proportional hazards model and five other methods of estimating treatment effects from time-to-event outcomes, which relax the proportional hazards assumption, were applied to a network of melanoma trials reporting overall survival: restricted mean survival time, generalised gamma, piecewise exponential, fractional polynomial and Royston-Parmar models. RESULTS: All models fitted the melanoma network acceptably well. However, there were important differences in extrapolations of the survival curve and interpretability of the modelling constraints demonstrating the potential for different conclusions from different modelling approaches. CONCLUSION: The restricted mean survival time, generalised gamma, piecewise exponential, fractional polynomial and Royston-Parmar models can accommodate non-proportional hazards and differing lengths of trial follow-up within a network meta-analysis of time-to-event outcomes. We recommend that model choice is informed using available and relevant prior knowledge, model transparency, graphically comparing survival curves alongside observed data to aid consideration of the reliability of the survival estimates, and consideration of how the treatment effect estimates can be incorporated within a decision model.

Statistical methods for non-adherence in non-inferiority trials: useful and used? A systematic review.

BACKGROUND: In non-inferiority trials with non-adherence to interventions (or non-compliance), intention-to-treat and per-protocol analyses are often performed; however, non-random non-adherence generally biases these estimates of efficacy. OBJECTIVE: To identify statistical methods that adjust for the impact of non-adherence and thus estimate the causal effects of experimental interventions in non-inferiority trials. DESIGN: A systematic review was conducted by searching the Ovid MEDLINE database (31 December 2020) to identify (1) randomised trials with a primary analysis for non-inferiority that applied (or planned to apply) statistical methods to account for the impact of non-adherence to interventions, and (2) methodology papers that described such statistical methods and included a non-inferiority trial application. OUTCOMES: The statistical methods identified, their impacts on non-inferiority conclusions, and their advantages/disadvantages. RESULTS: A total of 24 papers were included (4 protocols, 13 results papers and 7 methodology papers) reporting relevant methods on 26 occasions. The most common were instrumental variable approaches (n=9), including observed adherence as a covariate within a regression model (n=3), and modelling adherence as a time-varying covariate in a time-to-event analysis (n=3). Other methods included rank preserving structural failure time models and inverse-probability-of-treatment weighting. The methods identified in protocols and results papers were more commonly specified as sensitivity analyses (n=13) than primary analyses (n=3). Twelve results papers included an alternative analysis of the same outcome; conclusions regarding non-inferiority were in agreement on six occasions and could not be compared on six occasions (different measures of effect or results not provided in full). CONCLUSIONS: Available statistical methods which attempt to account for the impact of non-adherence to interventions were used infrequently. Therefore, firm inferences about their influence on non-inferiority conclusions could not be drawn. Since intention-to-treat and per-protocol analyses do not guarantee unbiased conclusions regarding non-inferiority, the methods identified should be considered for use in sensitivity analyses. PROSPERO REGISTRATION NUMBER: CRD42020177458.

Estimands: bringing clarity and focus to research questions in clinical trials.

Precise specification of the research question and associated treatment effect of interest is essential in clinical research, yet recent work shows that they are often incompletely specified. The ICH E9 (R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials introduces a framework that supports researchers in precisely and transparently specifying the treatment effect they aim to estimate in their clinical trial. In this paper, we present practical examples to demonstrate to all researchers involved in clinical trials how estimands can help them to specify the research question, lead to a better understanding of the treatment effect to be estimated and hence increase the probability of success of the trial.