Comment on Oberman & Vink: Should we fix or simulate the complete data in simulation studies evaluating missing data methods?

For simulation studies that evaluate methods of handling missing data, we argue that generating partially observed data by fixing the complete data and repeatedly simulating the missingness indicators is a superficially attractive idea but only rarely appropriate to use.

The marginality principle revisited: Should "higher-order" terms always be accompanied by "lower-order" terms in regression analyses?

The marginality principle guides analysts to avoid omitting lower-order terms from models in which higher-order terms are included as covariates. Lower-order terms are viewed as "marginal" to higher-order terms. We consider how this principle applies to three cases: regression models that may include the ratio of two measured variables; polynomial transformations of a measured variable; and factorial arrangements of defined interventions. For each case, we show that which terms or transformations are considered to be lower-order, and therefore marginal, depends on the scale of measurement, which is frequently arbitrary. Understanding the implications of this point leads to an intuitive understanding of the curse of dimensionality. We conclude that the marginality principle may be useful to analysts in some specific cases but caution against invoking it as a context-free recipe.

Rethinking intercurrent events in defining estimands for tuberculosis trials.

BACKGROUND/AIMS: Tuberculosis remains one of the leading causes of death from an infectious disease globally. Both choices of outcome definitions and approaches to handling events happening post-randomisation can change the treatment effect being estimated, but these are often inconsistently described, thus inhibiting clear interpretation and comparison across trials. METHODS: Starting from the ICH E9(R1) addendum's definition of an estimand, we use our experience of conducting large Phase III tuberculosis treatment trials and our understanding of the estimand framework to identify the key decisions regarding how different event types are handled in the primary outcome definition, and the important points that should be considered in making such decisions. A key issue is the handling of intercurrent (i.e. post-randomisation) events (ICEs) which affect interpretation of or preclude measurement of the intended final outcome. We consider common ICEs including treatment changes and treatment extension, poor adherence to randomised treatment, re-infection with a new strain of tuberculosis which is different from the original infection, and death. We use two completed tuberculosis trials (REMoxTB and STREAM Stage 1) as illustrative examples. These trials tested non-inferiority of new tuberculosis treatment regimens versus a control regimen. The primary outcome was a binary composite endpoint, 'favourable' or 'unfavourable', which was constructed from several components. RESULTS: We propose the following improvements in handling the above-mentioned ICEs and loss to follow-up (a post-randomisation event that is not in itself an ICE). First, changes to allocated regimens should not necessarily be viewed as an unfavourable outcome; from the patient perspective, the potential harms associated with a change in the regimen should instead be directly quantified. Second, handling poor adherence to randomised treatment using a per-protocol analysis does not necessarily target a clear estimand; instead, it would be desirable to develop ways to estimate the treatment effects more relevant to programmatic settings. Third, re-infection with a new strain of tuberculosis could be handled with different strategies, depending on whether the outcome of interest is the ability to attain culture negativity from infection with any strain of tuberculosis, or specifically the presenting strain of tuberculosis. Fourth, where possible, death could be separated into tuberculosis-related and non-tuberculosis-related and handled using appropriate strategies. Finally, although some losses to follow-up would result in early treatment discontinuation, patients lost to follow-up before the end of the trial should not always be classified as having an unfavourable outcome. Instead, loss to follow-up should be separated from not completing the treatment, which is an ICE and may be considered as an unfavourable outcome. CONCLUSION: The estimand framework clarifies many issues in tuberculosis trials but also challenges trialists to justify and improve their outcome definitions. Future trialists should consider all the above points in defining their outcomes.

Predictive values for different cancers and inflammatory bowel disease of 6 common abdominal symptoms among more than 1.9 million primary care patients in the UK: A cohort study.

BACKGROUND: The diagnostic assessment of abdominal symptoms in primary care presents a challenge. Evidence is needed about the positive predictive values (PPVs) of abdominal symptoms for different cancers and inflammatory bowel disease (IBD). METHODS AND FINDINGS: Using data from The Health Improvement Network (THIN) in the United Kingdom (2000-2017), we estimated the PPVs for diagnosis of (i) cancer (overall and for different cancer sites); (ii) IBD; and (iii) either cancer or IBD in the year post-consultation with each of 6 abdominal symptoms: dysphagia (n = 86,193 patients), abdominal bloating/distension (n = 100,856), change in bowel habit (n = 106,715), rectal bleeding (n = 235,094), dyspepsia (n = 517,326), and abdominal pain (n = 890,490). The median age ranged from 54 (abdominal pain) to 63 years (dysphagia and change in bowel habit); the ratio of women/men ranged from 50%:50% (rectal bleeding) to 73%:27% (abdominal bloating/distension). Across all studied symptoms, the risk of diagnosis of cancer and the risk of diagnosis of IBD were of similar magnitude, particularly in women, and younger men. Estimated PPVs were greatest for change in bowel habit in men (4.64% cancer and 2.82% IBD) and for rectal bleeding in women (2.39% cancer and 2.57% IBD) and lowest for dyspepsia (for cancer: 1.41% men and 1.03% women; for IBD: 0.89% men and 1.00% women). Considering PPVs for specific cancers, change in bowel habit and rectal bleeding had the highest PPVs for colon and rectal cancer; dysphagia for esophageal cancer; and abdominal bloating/distension (in women) for ovarian cancer. The highest PPVs of abdominal pain (either sex) and abdominal bloating/distension (men only) were for non-abdominal cancer sites. For the composite outcome of diagnosis of either cancer or IBD, PPVs of rectal bleeding exceeded the National Institute of Health and Care Excellence (NICE)-recommended specialist referral threshold of 3% in all age-sex strata, as did PPVs of abdominal pain, change in bowel habit, and dyspepsia, in those aged 60 years and over. Study limitations include reliance on accuracy and completeness of coding of symptoms and disease outcomes. CONCLUSIONS: Based on evidence from more than 1.9 million patients presenting in primary care, the findings provide estimated PPVs that could be used to guide specialist referral decisions, considering the PPVs of common abdominal symptoms for cancer alongside that for IBD and their composite outcome (cancer or IBD), taking into account the variable PPVs of different abdominal symptoms for different cancers sites. Jointly assessing the risk of cancer or IBD can better support decision-making and prompt diagnosis of both conditions, optimising specialist referrals or investigations, particularly in women.

A comparison of methods for analyzing a binary composite endpoint with partially observed components in randomized controlled trials.

Composite endpoints are commonly used to define primary outcomes in randomized controlled trials. A participant may be classified as meeting the endpoint if they experience an event in one or several components (eg, a favorable outcome based on a composite of being alive and attaining negative culture results in trials assessing tuberculosis treatments). Partially observed components that are not missing simultaneously complicate the analysis of the composite endpoint. An intuitive strategy frequently used in practice for handling missing values in the components is to derive the values of the composite endpoint from observed components when possible, and exclude from analysis participants whose composite endpoint cannot be derived. Alternatively, complete record analysis (CRA) (excluding participants with any missing components) or multiple imputation (MI) can be used. We compare a set of methods for analyzing a composite endpoint with partially observed components mathematically and by simulation, and apply these methods in a reanalysis of a published trial (TOPPS). We show that the derived composite endpoint can be missing not at random even when the components are missing completely at random. Consequently, the treatment effect estimated from the derived endpoint is biased while CRA results without the derived endpoint are valid. Missing at random mechanisms require MI of the components. We conclude that, although superficially attractive, deriving the composite endpoint from observed components should generally be avoided. Despite the potential risk of imputation model mis-specification, MI of missing components is the preferred approach in this study setting.

Treatment estimands in clinical trials of patients hospitalised for COVID-19: ensuring trials ask the right questions.

When designing a clinical trial, explicitly defining the treatment estimands of interest (that which is to be estimated) can help to clarify trial objectives and ensure the questions being addressed by the trial are clinically meaningful. There are several challenges when defining estimands. Here, we discuss a number of these in the context of trials of treatments for patients hospitalised with COVID-19 and make suggestions for how estimands should be defined for key outcomes. We suggest that treatment effects should usually be measured as differences in proportions (or risk or odds ratios) for outcomes such as death and requirement for ventilation, and differences in means for outcomes such as the number of days ventilated. We further recommend that truncation due to death should be handled differently depending on whether a patient- or resource-focused perspective is taken; for the former, a composite approach should be used, while for the latter, a while-alive approach is preferred. Finally, we suggest that discontinuation of randomised treatment should be handled from a treatment policy perspective, where non-adherence is ignored in the analysis (i.e. intention to treat).

Population-calibrated multiple imputation for a binary/categorical covariate in categorical regression models.

Multiple imputation (MI) has become popular for analyses with missing data in medical research. The standard implementation of MI is based on the assumption of data being missing at random (MAR). However, for missing data generated by missing not at random mechanisms, MI performed assuming MAR might not be satisfactory. For an incomplete variable in a given data set, its corresponding population marginal distribution might also be available in an external data source. We show how this information can be readily utilised in the imputation model to calibrate inference to the population by incorporating an appropriately calculated offset termed the "calibrated-δ adjustment." We describe the derivation of this offset from the population distribution of the incomplete variable and show how, in applications, it can be used to closely (and often exactly) match the post-imputation distribution to the population level. Through analytic and simulation studies, we show that our proposed calibrated-δ adjustment MI method can give the same inference as standard MI when data are MAR, and can produce more accurate inference under two general missing not at random missingness mechanisms. The method is used to impute missing ethnicity data in a type 2 diabetes prevalence case study using UK primary care electronic health records, where it results in scientifically relevant changes in inference for non-White ethnic groups compared with standard MI. Calibrated-δ adjustment MI represents a pragmatic approach for utilising available population-level information in a sensitivity analysis to explore potential departures from the MAR assumption.

Predictors of Postal or Online Response Mode and Associations With Patient Experience and Satisfaction in the English Cancer Patient Experience Survey.

BACKGROUND: Patient experience surveys are important tools for improving the quality of cancer services, but the representativeness of responders is a concern. Increasingly, patient surveys that traditionally used postal questionnaires are incorporating an online response option. However, the characteristics and experience ratings of online responders are poorly understood. OBJECTIVE: We sought to examine predictors of postal or online response mode, and associations with patient experience in the (English) Cancer Patient Experience Survey. METHODS: We analyzed data from 71,186 patients with cancer recently treated in National Health Service hospitals who responded to the Cancer Patient Experience Survey 2015. Using logistic regression, we explored patient characteristics associated with greater probability of online response and whether, after adjustment for patient characteristics, the online response was associated with a more or less critical evaluation of cancer care compared to the postal response. RESULTS: Of the 63,134 patients included in the analysis, 4635 (7.34%) responded online. In an adjusted analysis, male (women vs men: odds ratio [OR] 0.50, 95% confidence interval [CI] 0.46-0.54), younger (<55 vs 65-74 years: OR 3.49, 95% CI 3.21-3.80), least deprived (most vs least deprived quintile: OR 0.57, 95% CI 0.51-0.64), and nonwhite (nonwhite vs white ethnic group: OR 1.37, 95% CI 1.24-1.51) patients were more likely to respond online. Compared to postal responders, after adjustment for patient characteristics, online responders had a higher likelihood of reporting an overall satisfied experience of care (OR 1.24, 95% CI 1.16-1.32). For 34 of 49 other items, online responders more frequently reported a less than positive experience of care (8 reached statistical significance), and the associations were positive for the remaining 15 of 49 items (2 reached statistical significance). CONCLUSIONS: In the context of a national survey of patients with cancer, online and postal responders tend to differ in their characteristics and rating of satisfaction. Associations between online response and reported experience were generally small and mostly nonsignificant, but with a tendency toward less than positive ratings, although not consistently. Whether the observed associations between response mode and reported experience were causal needs to be examined using experimental survey designs.

How to check a simulation study.

Simulation studies are powerful tools in epidemiology and biostatistics, but they can be hard to conduct successfully. Sometimes unexpected results are obtained. We offer advice on how to check a simulation study when this occurs, and how to design and conduct the study to give results that are easier to check. Simulation studies should be designed to include some settings in which answers are already known. They should be coded in stages, with data-generating mechanisms checked before simulated data are analysed. Results should be explored carefully, with scatterplots of standard error estimates against point estimates surprisingly powerful tools. Failed estimation and outlying estimates should be identified and dealt with by changing data-generating mechanisms or coding realistic hybrid analysis procedures. Finally, we give a series of ideas that have been useful to us in the past for checking unexpected results. Following our advice may help to prevent errors and to improve the quality of published simulation studies.

Statistical analysis plan for the motor neuron disease systematic multi-arm adaptive randomised trial (MND-SMART).

BACKGROUND: MND-SMART is a platform, multi-arm, multi-stage, multi-centre, randomised controlled trial recruiting people with motor neuron disease. Initially, the treatments memantine and trazodone will each be compared against placebo, but other investigational treatments will be introduced into the trial later. The co-primary outcomes are the Amyotrophic Lateral Sclerosis Functional Rating Scale Revised (ALS-FRS-R) functional outcome, which is assessed longitudinally, and overall survival. METHODS: Initially in MND-SMART, participants are randomised 1:1:1 via a minimisation algorithm to receive placebo or one of the two investigational treatments with up to 531 to be randomised in total. The comparisons between each research arm and placebo will be conducted in four stages, with the opportunity to cease further randomisations to poorly performing research arms at the end of stages 1 or 2. The final ALS-FRS-R analysis will be at the end of stage 3 and final survival analysis at the end of stage 4. The estimands for the co-primary outcomes are described in detail. The primary analysis of ALS-FRS-R at the end of stages 1 to 3 will involve fitting a normal linear mixed model to the data to calculate a mean difference in rate of ALS-FRS-R change between each research treatment and placebo. The pairwise type 1 error rate will be controlled, because each treatment comparison will generate its own distinct and separate interpretation. This publication is based on a formal statistical analysis plan document that was finalised and signed on 18 May 2022. DISCUSSION: In developing the statistical analysis plan, we had to carefully consider several issues such as multiple testing, estimand specification, interim analyses, and statistical analysis of the repeated measurements of ALS-FRS-R. This analysis plan attempts to balance multiple factors, including minimisation of bias, maximising power and precision, and deriving clinically interpretable summaries of treatment effects. TRIAL REGISTRATION: EudraCT Number, 2019-000099-41. Registered 2 October 2019, https://www.clinicaltrialsregister.eu/ctr-search/search?query=mnd-smart ClinicalTrials.gov, NCT04302870 . Registered 10 March 2020.

Cardiovascular outcomes of type 2 diabetic patients treated with DPP­4 inhibitors versus sulphonylureas as add-on to metformin in clinical practice.

DPP-4 inhibitors (DPP-4i) and sulphonylureas remain the most widely prescribed add-on treatments after metformin. However, there is limited evidence from clinical practice comparing major adverse cardiovascular events (MACE) in patients prescribed these treatments, particularly among those without prior history of MACE and from vulnerable population groups. Using electronic health records from UK primary care, we undertook a retrospective cohort study with people diagnosed type-2 diabetes mellitus, comparing incidence of MACE (myocardial infarction, stroke, major cardiovascular surgery, unstable angina) and all-cause mortality among those prescribed DPP-4i versus sulphonylureas as add-on to metformin. We stratified analysis by history of MACE, age, social deprivation and comorbidities and adjusted for HbA1c, weight, smoking-status, comorbidities and medications. We identified 17,570 patients prescribed sulphonylureas and 6,267 prescribed DPP-4i between 2008-2017. Of these, 16.3% had pre-existing MACE. Primary incidence of MACE was similar in patients prescribed DPP-4i and sulphonylureas (10.3 vs 8.5 events per 1000 person-years; adjusted Hazard Ratio (adjHR): 0.94; 95%CI 0.80-1.14). For those with pre-existing MACE, rates for recurrence were higher overall, but similar between the two groups (21.8 vs 17.2 events per 1000 person-years; adjHR: 0.93; 95%CI 0.69-1.24). For those aged over 75 and with BMI less than 25 kg/m2 there was a protective effect for DPP-I, warranting further investigation. Patients initiating a DPP-4i had similar risk of cardiovascular outcomes to those initiating a sulphonylurea. This indicates the choice should be based on safety and cost, not cardiovascular prognosis, when deciding between a DPP-4i or sulphonylurea as add-on to metformin.

Handling Missing Values in Interrupted Time Series Analysis of Longitudinal Individual-Level Data.

BACKGROUND: In the interrupted time series (ITS) approach, it is common to average the outcome of interest at each time point and then perform a segmented regression (SR) analysis. In this study, we illustrate that such 'aggregate-level' analysis is biased when data are missing at random (MAR) and provide alternative analysis methods. METHODS: Using electronic health records from the UK, we evaluated weight change over time induced by the initiation of antipsychotic treatment. We contrasted estimates from aggregate-level SR analysis against estimates from mixed models with and without multiple imputation of missing covariates, using individual-level data. Then, we conducted a simulation study for insight about the different results in a controlled environment. RESULTS: Aggregate-level SR analysis suggested a substantial weight gain after initiation of treatment (average short-term weight change: 0.799kg/week) compared to mixed models (0.412kg/week). Simulation studies confirmed that aggregate-level SR analysis was biased when data were MAR. In simulations, mixed models gave less biased estimates than SR analysis and, in combination with multilevel multiple imputation, provided unbiased estimates. Mixed models with multiple imputation can be used with other types of ITS outcomes (eg, proportions). Other standard methods applied in ITS do not help to correct this bias problem. CONCLUSION: Aggregate-level SR analysis can bias the ITS estimates when individual-level data are MAR, because taking averages of individual-level data before SR means that data at the cluster level are missing not at random. Avoiding the averaging-step and using mixed models with or without multilevel multiple imputation of covariates is recommended.