Comment on Oberman & Vink: Should we fix or simulate the complete data in simulation studies evaluating missing data methods?

For simulation studies that evaluate methods of handling missing data, we argue that generating partially observed data by fixing the complete data and repeatedly simulating the missingness indicators is a superficially attractive idea but only rarely appropriate to use.

How to check a simulation study.

Simulation studies are powerful tools in epidemiology and biostatistics, but they can be hard to conduct successfully. Sometimes unexpected results are obtained. We offer advice on how to check a simulation study when this occurs, and how to design and conduct the study to give results that are easier to check. Simulation studies should be designed to include some settings in which answers are already known. They should be coded in stages, with data-generating mechanisms checked before simulated data are analysed. Results should be explored carefully, with scatterplots of standard error estimates against point estimates surprisingly powerful tools. Failed estimation and outlying estimates should be identified and dealt with by changing data-generating mechanisms or coding realistic hybrid analysis procedures. Finally, we give a series of ideas that have been useful to us in the past for checking unexpected results. Following our advice may help to prevent errors and to improve the quality of published simulation studies.

The marginality principle revisited: Should "higher-order" terms always be accompanied by "lower-order" terms in regression analyses?

The marginality principle guides analysts to avoid omitting lower-order terms from models in which higher-order terms are included as covariates. Lower-order terms are viewed as "marginal" to higher-order terms. We consider how this principle applies to three cases: regression models that may include the ratio of two measured variables; polynomial transformations of a measured variable; and factorial arrangements of defined interventions. For each case, we show that which terms or transformations are considered to be lower-order, and therefore marginal, depends on the scale of measurement, which is frequently arbitrary. Understanding the implications of this point leads to an intuitive understanding of the curse of dimensionality. We conclude that the marginality principle may be useful to analysts in some specific cases but caution against invoking it as a context-free recipe.

Statistical analysis plan for the motor neuron disease systematic multi-arm adaptive randomised trial (MND-SMART).

BACKGROUND: MND-SMART is a platform, multi-arm, multi-stage, multi-centre, randomised controlled trial recruiting people with motor neuron disease. Initially, the treatments memantine and trazodone will each be compared against placebo, but other investigational treatments will be introduced into the trial later. The co-primary outcomes are the Amyotrophic Lateral Sclerosis Functional Rating Scale Revised (ALS-FRS-R) functional outcome, which is assessed longitudinally, and overall survival. METHODS: Initially in MND-SMART, participants are randomised 1:1:1 via a minimisation algorithm to receive placebo or one of the two investigational treatments with up to 531 to be randomised in total. The comparisons between each research arm and placebo will be conducted in four stages, with the opportunity to cease further randomisations to poorly performing research arms at the end of stages 1 or 2. The final ALS-FRS-R analysis will be at the end of stage 3 and final survival analysis at the end of stage 4. The estimands for the co-primary outcomes are described in detail. The primary analysis of ALS-FRS-R at the end of stages 1 to 3 will involve fitting a normal linear mixed model to the data to calculate a mean difference in rate of ALS-FRS-R change between each research treatment and placebo. The pairwise type 1 error rate will be controlled, because each treatment comparison will generate its own distinct and separate interpretation. This publication is based on a formal statistical analysis plan document that was finalised and signed on 18 May 2022. DISCUSSION: In developing the statistical analysis plan, we had to carefully consider several issues such as multiple testing, estimand specification, interim analyses, and statistical analysis of the repeated measurements of ALS-FRS-R. This analysis plan attempts to balance multiple factors, including minimisation of bias, maximising power and precision, and deriving clinically interpretable summaries of treatment effects. TRIAL REGISTRATION: EudraCT Number, 2019-000099-41. Registered 2 October 2019, https://www.clinicaltrialsregister.eu/ctr-search/search?query=mnd-smart ClinicalTrials.gov, NCT04302870 . Registered 10 March 2020.

Rethinking intercurrent events in defining estimands for tuberculosis trials.

BACKGROUND/AIMS: Tuberculosis remains one of the leading causes of death from an infectious disease globally. Both choices of outcome definitions and approaches to handling events happening post-randomisation can change the treatment effect being estimated, but these are often inconsistently described, thus inhibiting clear interpretation and comparison across trials. METHODS: Starting from the ICH E9(R1) addendum's definition of an estimand, we use our experience of conducting large Phase III tuberculosis treatment trials and our understanding of the estimand framework to identify the key decisions regarding how different event types are handled in the primary outcome definition, and the important points that should be considered in making such decisions. A key issue is the handling of intercurrent (i.e. post-randomisation) events (ICEs) which affect interpretation of or preclude measurement of the intended final outcome. We consider common ICEs including treatment changes and treatment extension, poor adherence to randomised treatment, re-infection with a new strain of tuberculosis which is different from the original infection, and death. We use two completed tuberculosis trials (REMoxTB and STREAM Stage 1) as illustrative examples. These trials tested non-inferiority of new tuberculosis treatment regimens versus a control regimen. The primary outcome was a binary composite endpoint, 'favourable' or 'unfavourable', which was constructed from several components. RESULTS: We propose the following improvements in handling the above-mentioned ICEs and loss to follow-up (a post-randomisation event that is not in itself an ICE). First, changes to allocated regimens should not necessarily be viewed as an unfavourable outcome; from the patient perspective, the potential harms associated with a change in the regimen should instead be directly quantified. Second, handling poor adherence to randomised treatment using a per-protocol analysis does not necessarily target a clear estimand; instead, it would be desirable to develop ways to estimate the treatment effects more relevant to programmatic settings. Third, re-infection with a new strain of tuberculosis could be handled with different strategies, depending on whether the outcome of interest is the ability to attain culture negativity from infection with any strain of tuberculosis, or specifically the presenting strain of tuberculosis. Fourth, where possible, death could be separated into tuberculosis-related and non-tuberculosis-related and handled using appropriate strategies. Finally, although some losses to follow-up would result in early treatment discontinuation, patients lost to follow-up before the end of the trial should not always be classified as having an unfavourable outcome. Instead, loss to follow-up should be separated from not completing the treatment, which is an ICE and may be considered as an unfavourable outcome. CONCLUSION: The estimand framework clarifies many issues in tuberculosis trials but also challenges trialists to justify and improve their outcome definitions. Future trialists should consider all the above points in defining their outcomes.

Cardiovascular outcomes of type 2 diabetic patients treated with DPP­4 inhibitors versus sulphonylureas as add-on to metformin in clinical practice.

DPP-4 inhibitors (DPP-4i) and sulphonylureas remain the most widely prescribed add-on treatments after metformin. However, there is limited evidence from clinical practice comparing major adverse cardiovascular events (MACE) in patients prescribed these treatments, particularly among those without prior history of MACE and from vulnerable population groups. Using electronic health records from UK primary care, we undertook a retrospective cohort study with people diagnosed type-2 diabetes mellitus, comparing incidence of MACE (myocardial infarction, stroke, major cardiovascular surgery, unstable angina) and all-cause mortality among those prescribed DPP-4i versus sulphonylureas as add-on to metformin. We stratified analysis by history of MACE, age, social deprivation and comorbidities and adjusted for HbA1c, weight, smoking-status, comorbidities and medications. We identified 17,570 patients prescribed sulphonylureas and 6,267 prescribed DPP-4i between 2008-2017. Of these, 16.3% had pre-existing MACE. Primary incidence of MACE was similar in patients prescribed DPP-4i and sulphonylureas (10.3 vs 8.5 events per 1000 person-years; adjusted Hazard Ratio (adjHR): 0.94; 95%CI 0.80-1.14). For those with pre-existing MACE, rates for recurrence were higher overall, but similar between the two groups (21.8 vs 17.2 events per 1000 person-years; adjHR: 0.93; 95%CI 0.69-1.24). For those aged over 75 and with BMI less than 25 kg/m2 there was a protective effect for DPP-I, warranting further investigation. Patients initiating a DPP-4i had similar risk of cardiovascular outcomes to those initiating a sulphonylurea. This indicates the choice should be based on safety and cost, not cardiovascular prognosis, when deciding between a DPP-4i or sulphonylurea as add-on to metformin.

A comparison of methods for analyzing a binary composite endpoint with partially observed components in randomized controlled trials.

Composite endpoints are commonly used to define primary outcomes in randomized controlled trials. A participant may be classified as meeting the endpoint if they experience an event in one or several components (eg, a favorable outcome based on a composite of being alive and attaining negative culture results in trials assessing tuberculosis treatments). Partially observed components that are not missing simultaneously complicate the analysis of the composite endpoint. An intuitive strategy frequently used in practice for handling missing values in the components is to derive the values of the composite endpoint from observed components when possible, and exclude from analysis participants whose composite endpoint cannot be derived. Alternatively, complete record analysis (CRA) (excluding participants with any missing components) or multiple imputation (MI) can be used. We compare a set of methods for analyzing a composite endpoint with partially observed components mathematically and by simulation, and apply these methods in a reanalysis of a published trial (TOPPS). We show that the derived composite endpoint can be missing not at random even when the components are missing completely at random. Consequently, the treatment effect estimated from the derived endpoint is biased while CRA results without the derived endpoint are valid. Missing at random mechanisms require MI of the components. We conclude that, although superficially attractive, deriving the composite endpoint from observed components should generally be avoided. Despite the potential risk of imputation model mis-specification, MI of missing components is the preferred approach in this study setting.

Predictive values for different cancers and inflammatory bowel disease of 6 common abdominal symptoms among more than 1.9 million primary care patients in the UK: A cohort study.

BACKGROUND: The diagnostic assessment of abdominal symptoms in primary care presents a challenge. Evidence is needed about the positive predictive values (PPVs) of abdominal symptoms for different cancers and inflammatory bowel disease (IBD). METHODS AND FINDINGS: Using data from The Health Improvement Network (THIN) in the United Kingdom (2000-2017), we estimated the PPVs for diagnosis of (i) cancer (overall and for different cancer sites); (ii) IBD; and (iii) either cancer or IBD in the year post-consultation with each of 6 abdominal symptoms: dysphagia (n = 86,193 patients), abdominal bloating/distension (n = 100,856), change in bowel habit (n = 106,715), rectal bleeding (n = 235,094), dyspepsia (n = 517,326), and abdominal pain (n = 890,490). The median age ranged from 54 (abdominal pain) to 63 years (dysphagia and change in bowel habit); the ratio of women/men ranged from 50%:50% (rectal bleeding) to 73%:27% (abdominal bloating/distension). Across all studied symptoms, the risk of diagnosis of cancer and the risk of diagnosis of IBD were of similar magnitude, particularly in women, and younger men. Estimated PPVs were greatest for change in bowel habit in men (4.64% cancer and 2.82% IBD) and for rectal bleeding in women (2.39% cancer and 2.57% IBD) and lowest for dyspepsia (for cancer: 1.41% men and 1.03% women; for IBD: 0.89% men and 1.00% women). Considering PPVs for specific cancers, change in bowel habit and rectal bleeding had the highest PPVs for colon and rectal cancer; dysphagia for esophageal cancer; and abdominal bloating/distension (in women) for ovarian cancer. The highest PPVs of abdominal pain (either sex) and abdominal bloating/distension (men only) were for non-abdominal cancer sites. For the composite outcome of diagnosis of either cancer or IBD, PPVs of rectal bleeding exceeded the National Institute of Health and Care Excellence (NICE)-recommended specialist referral threshold of 3% in all age-sex strata, as did PPVs of abdominal pain, change in bowel habit, and dyspepsia, in those aged 60 years and over. Study limitations include reliance on accuracy and completeness of coding of symptoms and disease outcomes. CONCLUSIONS: Based on evidence from more than 1.9 million patients presenting in primary care, the findings provide estimated PPVs that could be used to guide specialist referral decisions, considering the PPVs of common abdominal symptoms for cancer alongside that for IBD and their composite outcome (cancer or IBD), taking into account the variable PPVs of different abdominal symptoms for different cancers sites. Jointly assessing the risk of cancer or IBD can better support decision-making and prompt diagnosis of both conditions, optimising specialist referrals or investigations, particularly in women.

Handling Missing Values in Interrupted Time Series Analysis of Longitudinal Individual-Level Data.

BACKGROUND: In the interrupted time series (ITS) approach, it is common to average the outcome of interest at each time point and then perform a segmented regression (SR) analysis. In this study, we illustrate that such 'aggregate-level' analysis is biased when data are missing at random (MAR) and provide alternative analysis methods. METHODS: Using electronic health records from the UK, we evaluated weight change over time induced by the initiation of antipsychotic treatment. We contrasted estimates from aggregate-level SR analysis against estimates from mixed models with and without multiple imputation of missing covariates, using individual-level data. Then, we conducted a simulation study for insight about the different results in a controlled environment. RESULTS: Aggregate-level SR analysis suggested a substantial weight gain after initiation of treatment (average short-term weight change: 0.799kg/week) compared to mixed models (0.412kg/week). Simulation studies confirmed that aggregate-level SR analysis was biased when data were MAR. In simulations, mixed models gave less biased estimates than SR analysis and, in combination with multilevel multiple imputation, provided unbiased estimates. Mixed models with multiple imputation can be used with other types of ITS outcomes (eg, proportions). Other standard methods applied in ITS do not help to correct this bias problem. CONCLUSION: Aggregate-level SR analysis can bias the ITS estimates when individual-level data are MAR, because taking averages of individual-level data before SR means that data at the cluster level are missing not at random. Avoiding the averaging-step and using mixed models with or without multilevel multiple imputation of covariates is recommended.

Treatment estimands in clinical trials of patients hospitalised for COVID-19: ensuring trials ask the right questions.

When designing a clinical trial, explicitly defining the treatment estimands of interest (that which is to be estimated) can help to clarify trial objectives and ensure the questions being addressed by the trial are clinically meaningful. There are several challenges when defining estimands. Here, we discuss a number of these in the context of trials of treatments for patients hospitalised with COVID-19 and make suggestions for how estimands should be defined for key outcomes. We suggest that treatment effects should usually be measured as differences in proportions (or risk or odds ratios) for outcomes such as death and requirement for ventilation, and differences in means for outcomes such as the number of days ventilated. We further recommend that truncation due to death should be handled differently depending on whether a patient- or resource-focused perspective is taken; for the former, a composite approach should be used, while for the latter, a while-alive approach is preferred. Finally, we suggest that discontinuation of randomised treatment should be handled from a treatment policy perspective, where non-adherence is ignored in the analysis (i.e. intention to treat).

Reliability of hospital scores for the Cancer Patient Experience Survey: analysis of publicly reported patient survey data.

OBJECTIVES: To assess the degree to which variations in publicly reported hospital scores arising from the English Cancer Patient Experience Survey (CPES) are subject to chance. DESIGN: Secondary analysis of publically reported data. SETTING: English National Health Service hospitals. PARTICIPANTS: 72 756 patients who were recently treated for cancer in one of 146 hospitals and responded to the 2016 English CPES. MAIN OUTCOME MEASURES: Spearman-Brown reliability of hospital scores on 51 evaluative questions regarding cancer care. RESULTS: Hospitals varied in respondent sample size with a median hospital sample size of 419 responses (range 31-1972). There were some hospitals with generally highly reliable scores across most questions, whereas other hospitals had generally unreliable scores (the median reliability of question scores within individual hospitals varied between 0.11 and 0.86). Similarly, there were some questions with generally high reliability across most hospitals, whereas other questions had generally low reliability. Of the 7377 individual hospital scores publically reported (146 hospitals by 51 questions, minus 69 suppressed scores), only 34% reached a reliability of 0.7, the minimum generally considered to be useful. In order for 80% of the individual hospital scores to reach a reliability of 0.7, some hospitals would require a fourfold increase in number of respondents; although in a few other hospitals sample sizes could be reduced. CONCLUSIONS: The English Patient Experience Survey represents a globally unique source for understanding experience of a patient with cancer; but in its present form, it is not reliable for high stakes comparisons of the performance of different hospitals. Revised sampling strategies and survey questions could help increase the reliability of hospital scores, and thus make the survey fit for use in performance comparisons.

Diagnostic route is associated with care satisfaction independently of tumour stage: Evidence from linked English Cancer Patient Experience Survey and cancer registration data.

BACKGROUND: Whether diagnostic route (e.g. emergency presentation) is associated with cancer care experience independently of tumour stage is unknown. METHODS: We analysed data on 18 590 patients with breast, prostate, colon, lung, and rectal cancers who responded to the 2014 English Cancer Patient Experience Survey, linked to cancer registration data on diagnostic route and tumour stage at diagnosis. We estimated odds ratios (OR) of reporting a negative experience of overall cancer care by tumour stage and diagnostic route (crude and adjusted for patient characteristic and cancer site variables) and examined their interactions with cancer site. RESULTS: After adjustment, the likelihood of reporting a negative experience was highest for emergency presenters and lowest for screening-detected patients with breast, colon, and rectal cancers (OR versus two-week-wait 1.51, 95% confidence interval [CI] 1.24-1.83; 0.88, 95% CI 0.75-1.03, respectively). Patients with the most advanced stage were more likely to report a negative experience (OR stage IV versus I 1.37, 95% CI 1.15-1.62) with little confounding between stage and route, and no evidence for cancer-stage or cancer-route interactions. CONCLUSIONS: Though the extent of disease is strongly associated with ratings of overall cancer care, diagnostic route (particularly emergency presentation or screening detection) exerts important independent effects.