Comparison of statistical methods used to meta-analyse results from interrupted time series studies: an empirical study.

BACKGROUND: The Interrupted Time Series (ITS) is a robust design for evaluating public health and policy interventions or exposures when randomisation may be infeasible. Several statistical methods are available for the analysis and meta-analysis of ITS studies. We sought to empirically compare available methods when applied to real-world ITS data. METHODS: We sourced ITS data from published meta-analyses to create an online data repository. Each dataset was re-analysed using two ITS estimation methods. The level- and slope-change effect estimates (and standard errors) were calculated and combined using fixed-effect and four random-effects meta-analysis methods. We examined differences in meta-analytic level- and slope-change estimates, their 95% confidence intervals, p-values, and estimates of heterogeneity across the statistical methods. RESULTS: Of 40 eligible meta-analyses, data from 17 meta-analyses including 282 ITS studies were obtained (predominantly investigating the effects of public health interruptions (88%)) and analysed. We found that on average, the meta-analytic effect estimates, their standard errors and between-study variances were not sensitive to meta-analysis method choice, irrespective of the ITS analysis method. However, across ITS analysis methods, for any given meta-analysis, there could be small to moderate differences in meta-analytic effect estimates, and important differences in the meta-analytic standard errors. Furthermore, the confidence interval widths and p-values for the meta-analytic effect estimates varied depending on the choice of confidence interval method and ITS analysis method. CONCLUSIONS: Our empirical study showed that meta-analysis effect estimates, their standard errors, confidence interval widths and p-values can be affected by statistical method choice. These differences may importantly impact interpretations and conclusions of a meta-analysis and suggest that the statistical methods are not interchangeable in practice.

The Effects of Statins on Cardiovascular and Inflammatory Biomarkers in Primary Prevention: A Systematic Review and Meta-Analysis.

BACKGROUND: Statins are well-established for their treatment of cardiovascular disease (CVD) due to their cholesterol-lowering effects and potential anti-inflammatory properties. Although previous systematic reviews demonstrate that statins reduce inflammatory biomarkers in the secondary prevention of CVD, none examine their effects on cardiac and inflammatory biomarkers in a primary prevention setting. METHODS: We conducted a systematic review and meta-analysis to examine the effects of statins on cardiovascular and inflammatory biomarkers among individuals without established CVD. The biomarkers included are: cardiac troponin, N-terminal pro B-type natriuretic peptide (NT-proBNP), C-reactive protein (CRP), tumour necrosis factor-alpha (TNF-α), interleukin-6 (IL-6), soluble vascular cell adhesion molecule (sVCAM), soluble intercellular adhesion molecule (sICAM), soluble E-selectin (sE-selectin) and endothelin-1 (ET-1). A literature search was performed through Ovid MEDLINE, Embase and CINAHL Plus for randomised controlled trials (RCTs) published up to June 2021. RESULTS: Overall, 35 RCTs with 26,521 participants were included in our meta-analysis. Data was pooled using random effects models presented as standardised mean differences (SMD) with 95% confidence intervals (CI). Combining 36 effect sizes from 29 RCTs, statin use resulted in a significant reduction in CRP levels (SMD -0.61; 95% CI -0.91, -0.32; P<0.001). This reduction was observed for both hydrophilic (SMD -0.39; 95% CI -0.62, -0.16; P<0.001) and lipophilic statins (SMD -0.65; 95% CI -1.01, -0.29; P<0.001). There were no significant changes in serum concentrations of cardiac troponin, NT-proBNP, TNF-α, IL-6, sVCAM, sICAM, sE-selectin and ET-1. CONCLUSION: This meta-analysis demonstrates that statin use reduces serum CRP levels in a primary prevention setting for CVD, with no clear effect on the other eight biomarkers studied.

Comparison of six statistical methods for interrupted time series studies: empirical evaluation of 190 published series.

BACKGROUND: The Interrupted Time Series (ITS) is a quasi-experimental design commonly used in public health to evaluate the impact of interventions or exposures. Multiple statistical methods are available to analyse data from ITS studies, but no empirical investigation has examined how the different methods compare when applied to real-world datasets. METHODS: A random sample of 200 ITS studies identified in a previous methods review were included. Time series data from each of these studies was sought. Each dataset was re-analysed using six statistical methods. Point and confidence interval estimates for level and slope changes, standard errors, p-values and estimates of autocorrelation were compared between methods. RESULTS: From the 200 ITS studies, including 230 time series, 190 datasets were obtained. We found that the choice of statistical method can importantly affect the level and slope change point estimates, their standard errors, width of confidence intervals and p-values. Statistical significance (categorised at the 5% level) often differed across the pairwise comparisons of methods, ranging from 4 to 25% disagreement. Estimates of autocorrelation differed depending on the method used and the length of the series. CONCLUSIONS: The choice of statistical method in ITS studies can lead to substantially different conclusions about the impact of the interruption. Pre-specification of the statistical method is encouraged, and naive conclusions based on statistical significance should be avoided.

Evaluation of statistical methods used in the analysis of interrupted time series studies: a simulation study.

BACKGROUND: Interrupted time series (ITS) studies are frequently used to evaluate the effects of population-level interventions or exposures. However, examination of the performance of statistical methods for this design has received relatively little attention. METHODS: We simulated continuous data to compare the performance of a set of statistical methods under a range of scenarios which included different level and slope changes, varying lengths of series and magnitudes of lag-1 autocorrelation. We also examined the performance of the Durbin-Watson (DW) test for detecting autocorrelation. RESULTS: All methods yielded unbiased estimates of the level and slope changes over all scenarios. The magnitude of autocorrelation was underestimated by all methods, however, restricted maximum likelihood (REML) yielded the least biased estimates. Underestimation of autocorrelation led to standard errors that were too small and coverage less than the nominal 95%. All methods performed better with longer time series, except for ordinary least squares (OLS) in the presence of autocorrelation and Newey-West for high values of autocorrelation. The DW test for the presence of autocorrelation performed poorly except for long series and large autocorrelation. CONCLUSIONS: From the methods evaluated, OLS was the preferred method in series with fewer than 12 points, while in longer series, REML was preferred. The DW test should not be relied upon to detect autocorrelation, except when the series is long. Care is needed when interpreting results from all methods, given confidence intervals will generally be too narrow. Further research is required to develop better performing methods for ITS, especially for short series.

Identification of application and interpretation errors that can occur in pairwise meta-analyses in systematic reviews of interventions: a systematic review.

OBJECTIVES: To generate a bank of items describing application and interpretation errors that can arise in pairwise meta-analyses in systematic reviews of interventions. STUDY DESIGN AND SETTING: MEDLINE, Embase, and Scopus were searched to identify studies describing types of errors in meta-analyses. Descriptions of errors and supporting quotes were extracted by multiple authors. Errors were reviewed at team meetings to determine if they should be excluded, reworded, or combined with other errors, and were categorized into broad categories of errors and subcategories within. RESULTS: Fifty articles met our inclusion criteria, leading to the identification of 139 errors. We identified 25 errors covering data extraction/manipulation, 74 covering statistical analyses, and 40 covering interpretation. Many of the statistical analysis errors related to the meta-analysis model (eg, using a two-stage strategy to determine whether to select a fixed or random-effects model) and statistical heterogeneity (eg, not undertaking an assessment for statistical heterogeneity). CONCLUSION: We generated a comprehensive bank of possible errors that can arise in the application and interpretation of meta-analyses in systematic reviews of interventions. This item bank of errors provides the foundation for developing a checklist to help peer reviewers detect statistical errors.

Development of a search filter to retrieve reports of interrupted time series studies from MEDLINE and PubMed.

BACKGROUND: Interrupted time series (ITS) studies contribute importantly to systematic reviews of population-level interventions. We aimed to develop and validate search filters to retrieve ITS studies in MEDLINE and PubMed. METHODS: A total of 1017 known ITS studies (published 2013-2017) were analysed using text mining to generate candidate terms. A control set of 1398 time-series studies were used to select differentiating terms. Various combinations of candidate terms were iteratively tested to generate three search filters. An independent set of 700 ITS studies was used to validate the filters' sensitivities. The filters were test-run in Ovid MEDLINE and the records randomly screened for ITS studies to determine their precision. Finally, all MEDLINE filters were translated to PubMed format and their sensitivities in PubMed were estimated. RESULTS: Three search filters were created in MEDLINE: a precision-maximising filter with high precision (78%; 95% CI 74%-82%) but moderate sensitivity (63%; 59%-66%), most appropriate when there are limited resources to screen studies; a sensitivity-and-precision-maximising filter with higher sensitivity (81%; 77%-83%) but lower precision (32%; 28%-36%), providing a balance between expediency and comprehensiveness; and a sensitivity-maximising filter with high sensitivity (88%; 85%-90%) but likely very low precision, useful when combined with specific content terms. Similar sensitivity estimates were found for PubMed versions. CONCLUSION: Our filters strike different balances between comprehensiveness and screening workload and suit different research needs. Retrieval of ITS studies would be improved if authors identified the ITS design in the titles.

Evaluation of statistical methods used to meta-analyse results from interrupted time series studies: A simulation study.

Interrupted time series (ITS) are often meta-analysed to inform public health and policy decisions but examination of the statistical methods for ITS analysis and meta-analysis in this context is limited. We simulated meta-analyses of ITS studies with continuous outcome data, analysed the studies using segmented linear regression with two estimation methods [ordinary least squares (OLS) and restricted maximum likelihood (REML)], and meta-analysed the immediate level- and slope-change effect estimates using fixed-effect and (multiple) random-effects meta-analysis methods. Simulation design parameters included varying series length; magnitude of lag-1 autocorrelation; magnitude of level- and slope-changes; number of included studies; and, effect size heterogeneity. All meta-analysis methods yielded unbiased estimates of the interruption effects. All random effects meta-analysis methods yielded coverage close to the nominal level, irrespective of the ITS analysis method used and other design parameters. However, heterogeneity was frequently overestimated in scenarios where the ITS study standard errors were underestimated, which occurred for short series or when the ITS analysis method did not appropriately account for autocorrelation. The performance of meta-analysis methods depends on the design and analysis of the included ITS studies. Although all random effects methods performed well in terms of coverage, irrespective of the ITS analysis method, we recommend the use of effect estimates calculated from ITS methods that adjust for autocorrelation when possible. Doing so will likely to lead to more accurate estimates of the heterogeneity variance.

Are COVID-19 systematic reviews up to date and can we tell? A cross-sectional study.

BACKGROUND: COVID-19 led to a rapid acceleration in the number of systematic reviews. Readers need to know how up to date evidence is when selecting reviews to inform decisions. This cross-sectional study aimed to evaluate how easily the currency of COVID-19 systematic reviews published early in the pandemic could be determined and how up to date these reviews were at the time of publication. METHODS: We searched for systematic reviews and meta-analyses relevant to COVID-19 added to PubMed in July 2020 and January 2021, including any that were first published as preprints. We extracted data on the date of search, number of included studies, and date first published online. For the search date, we noted the format of the date and where in the review this was reported. A sample of non-COVID-19 systematic reviews from November 2020 served as a comparator. RESULTS: We identified 246 systematic reviews on COVID-19. In the abstract of these reviews, just over half (57%) reported the search date (day/month/year or month/year) while 43% failed to report any date. When the full text was considered, the search date was missing from 6% of reviews. The median time from last search to publication online was 91 days (IQR 63-130). Time from search to publication was similar for the subset of 15 rapid or living reviews (92 days) but shorter for the 29 reviews published as preprints (37 days). The median number of studies or publications included per review was 23 (IQR 12-40). In the sample of 290 non-COVID SRs, around two-thirds (65%) reported the search date while a third (34%) did not include any date in the abstract. The median time from search to publication online was 253 days (IQR 153-381) and each review included a median of 12 studies (IQR 8-21). CONCLUSIONS: Despite the context of the pandemic and the need to easily ascertain the currency of systematic reviews, reporting of the search date information for COVID-19 reviews was inadequate. Adherence to reporting guidelines would improve the transparency and usefulness of systematic reviews to users.

Methodological systematic review recommends improvements to conduct and reporting when meta-analyzing interrupted time series studies.

OBJECTIVES: Interrupted Time Series (ITS) are a type of nonrandomized design commonly used to evaluate public health policy interventions, and the impact of exposures, at the population level. Meta-analysis may be used to combine results from ITS across studies (in the context of systematic reviews) or across sites within the same study. We aimed to examine the statistical approaches, methods, and completeness of reporting in reviews that meta-analyze results from ITS. STUDY DESIGN AND SETTINGS: Eight electronic databases were searched to identify reviews (published 2000-2019) that meta-analyzed at least two ITS. Characteristics of the included reviews, the statistical methods used to analyze the ITS and meta-analyze their results, effect measures, and risk of bias assessment tools were extracted. RESULTS: Of the 4213 identified records, 54 reviews were included. Nearly all reviews (94%) used two-stage meta-analysis, most commonly fitting a random effects model (69%). Among the 41 reviews that re-analyzed the ITS, linear regression (39%) and ARIMA (20%) were most commonly used; 38% adjusted for autocorrelation. The most common effect measure meta-analyzed was an immediate level-change (46/54). Reporting of the statistical methods and ITS characteristics was often incomplete. CONCLUSION: Improvement is needed in the conduct and reporting of reviews that meta-analyze results from ITS.

Improving adherence to acute low back pain guideline recommendations with chiropractors and physiotherapists: the ALIGN cluster randomised controlled trial.

BACKGROUND: Acute low back pain is a common condition, has high burden, and there are evidence-to-practice gaps in the chiropractic and physiotherapy setting for imaging and giving advice to stay active. The aim of this cluster randomised trial was to estimate the effects of a theory- and evidence-based implementation intervention to increase chiropractors' and physiotherapists' adherence to a guideline for acute low back pain compared with the comparator (passive dissemination of the guideline). In particular, the primary aim of the intervention was to reduce inappropriate imaging referral and improve patient low back pain outcomes, and to determine whether this intervention was cost-effective. METHODS: Physiotherapy and chiropractic practices in the state of Victoria, Australia, comprising at least one practising clinician who provided care to patients with acute low back pain, were invited to participate. Patients attending these practices were included if they had acute non-specific low back pain (duration less than 3 months), were 18 years of age or older, and were able to understand and read English. Practices were randomly assigned either to a tailored, multi-faceted intervention based on the guideline (interactive educational symposium plus academic detailing) or passive dissemination of the guideline (comparator). A statistician independent of the study team undertook stratified randomisation using computer-generated random numbers; four strata were defined by professional group and the rural or metropolitan location of the practice. Investigators not involved in intervention delivery were blinded to allocation. Primary outcomes were X-ray referral self-reported by clinicians using a checklist and patient low back pain-specific disability (at 3 months). RESULTS: A total of 104 practices (43 chiropractors, 85 physiotherapists; 755 patients) were assigned to the intervention and 106 practices (45 chiropractors, 97 physiotherapists; 603 patients) to the comparator; 449 patients were available for the patient-level primary outcome. There was no important difference in the odds of patients being referred for X-ray (adjusted (Adj) OR: 1.40; 95% CI 0.51, 3.87; Adj risk difference (RD): 0.01; 95% CI - 0.02, 0.04) or patient low back pain-specific disability (Adj mean difference: 0.37; 95% CI - 0.48, 1.21, scale 0-24). The intervention did lead to improvement for some key secondary outcomes, including giving advice to stay active (Adj OR: 1.96; 95% CI 1.20, 3.22; Adj RD: 0.10; 95% CI 0.01, 0.19) and intending to adhere to the guideline recommendations (e.g. intention to refer for X-ray: Adj OR: 0.27; 95% CI 0.17, 0.44; intention to give advice to stay active: Adj OR: 2.37; 95% CI 1.51, 3.74). CONCLUSIONS: Intervention group clinicians were more likely to give advice to stay active and to intend to adhere to the guideline recommendations about X-ray referral. The intervention did not change the primary study outcomes, with no important differences in X-ray referral and patient disability between groups, implying that hypothesised reductions in health service utilisation and/or productivity gains are unlikely to offset the direct costs of the intervention. We report these results with the caveat that we enrolled less patients into the trial than our determined sample size. We cannot recommend this intervention as a cost-effective use of resources. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry ACTRN12609001022257 . Retrospectively registered on 25 November 2009.

Creating effective interrupted time series graphs: Review and recommendations.

INTRODUCTION: Interrupted Time Series (ITS) studies may be used to assess the impact of an interruption, such as an intervention or exposure. The data from such studies are particularly amenable to visual display and, when clearly depicted, can readily show the short- and long-term impact of an interruption. Further, well-constructed graphs allow data to be extracted using digitizing software, which can facilitate their inclusion in systematic reviews and meta-analyses. AIM: We provide recommendations for graphing ITS data, examine the properties of plots presented in ITS studies, and provide examples employing our recommendations. METHODS AND RESULTS: Graphing recommendations from seminal data visualization resources were adapted for use with ITS studies. The adapted recommendations cover plotting of data points, trend lines, interruptions, additional lines and general graph components. We assessed whether 217 graphs from recently published (2013-2017) ITS studies met our recommendations and found that 130 graphs (60%) had clearly distinct data points, 100 (46%) had trend lines, and 161 (74%) had a clearly defined interruption. Accurate data extraction (requiring distinct points that align with axis tick marks and labels that allow the points to be interpreted) was possible in only 72 (33%) graphs. CONCLUSION: We found that many ITS graphs did not meet our recommendations and could be improved with simple changes. Our proposed recommendations aim to achieve greater standardization and improvement in the display of ITS data, and facilitate re-use of the data in systematic reviews and meta-analyses.

Design characteristics and statistical methods used in interrupted time series studies evaluating public health interventions: a review.

OBJECTIVES: Interrupted time series (ITS) designs are frequently used in public health to examine whether an intervention or exposure has influenced health outcomes. Few reviews have been undertaken to examine the design characteristics, statistical methods, and completeness of reporting of published ITS studies. STUDY DESIGN AND SETTING: We used stratified random sampling to identify 200 ITS studies that evaluated public health interventions or exposures from PubMed (2013-2017). Study characteristics, details of statistical models and estimation methods used, effect metrics, and parameter estimates were extracted. From the 200 studies, 230 time series were examined. RESULTS: Common statistical methods used were linear regression (31%, 72/230) and autoregressive integrated moving average (19%, 43/230). In 17% (40/230) of the series, we could not determine the statistical method used. Autocorrelation was acknowledged in 63% (145/230) of the series. An estimate of the autocorrelation coefficient was given for only 1% of the series (3/230). Measures of precision were reported for 63% of effect measures (541/852). CONCLUSION: Many aspects of the design, methods, analysis, and reporting of ITS studies can be improved, particularly description of the statistical methods and approaches to adjust for and estimate autocorrelation. More guidance on the conduct and reporting of ITS studies is needed to improve this study design.

Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol.

Background: Systematic reviews are used to inform healthcare decision making. In reviews that aim to examine the effects of organisational, policy change or public health interventions, or exposures, evidence from interrupted time series (ITS) studies may be included. A core component of many systematic reviews is meta-analysis, which is the statistical synthesis of results across studies. There is currently a lack of guidance informing the choice of meta-analysis methods for combining results from ITS studies, and there have been no studies examining the meta-analysis methods used in practice. This study therefore aims to describe current meta-analysis methods used in a cohort of reviews of ITS studies. Methods: We will identify the 100 most recent reviews (published between 1 January 2000 and 11 October 2019) that include meta-analyses of ITS studies from a search of eight electronic databases covering several disciplines (public health, psychology, education, economics). Study selection will be undertaken independently by two authors. Data extraction will be undertaken by one author, and for a random sample of the reviews, two authors. From eligible reviews we will extract details at the review level including discipline, type of interruption and any tools used to assess the risk of bias / methodological quality of included ITS studies; at the meta-analytic level we will extract type of outcome, effect measure(s), meta-analytic methods, and any methods used to re-analyse the individual ITS studies. Descriptive statistics will be used to summarise the data. Conclusions: This review will describe the methods used to meta-analyse results from ITS studies. Results from this review will inform future methods research examining how different meta-analysis methods perform, and ultimately, the development of guidance.

Design characteristics and statistical methods used in interrupted time series studies evaluating public health interventions: protocol for a review.

INTRODUCTION: An interrupted time series (ITS) design is an important observational design used to examine the effects of an intervention or exposure. This design has particular utility in public health where it may be impracticable or infeasible to use a randomised trial to evaluate health system-wide policies, or examine the impact of exposures (such as earthquakes). There have been relatively few studies examining the design characteristics and statistical methods used to analyse ITS designs. Further, there is a lack of guidance to inform the design and analysis of ITS studies.This is the first study in a larger project that aims to provide tools and guidance for researchers in the design and analysis of ITS studies. The objectives of this study are to (1) examine and report the design characteristics and statistical methods used in a random sample of contemporary ITS studies examining public health interventions or exposures that impact on health-related outcomes, and (2) create a repository of time series data extracted from ITS studies. Results from this study will inform the remainder of the project which will investigate the performance of a range of commonly used statistical methods, and create a repository of input parameters required for sample size calculation. METHODS AND ANALYSIS: We will collate 200 ITS studies evaluating public health interventions or the impact of exposures. ITS studies will be identified from a search of the bibliometric database PubMed between the years 2013 and 2017, combined with stratified random sampling. From eligible studies, we will extract study characteristics, details of the statistical models and estimation methods, effect metrics and parameter estimates. Further, we will extract the time series data when available. We will use systematic review methods in the screening, application of inclusion and exclusion criteria, and extraction of data. Descriptive statistics will be used to summarise the data. ETHICS AND DISSEMINATION: Ethics approval is not required since information will only be extracted from published studies. Dissemination of the results will be through peer-reviewed publications and presentations at conferences. A repository of data extracted from the published ITS studies will be made publicly available.

An investigation of the impact of using different methods for network meta-analysis: a protocol for an empirical evaluation.

BACKGROUND: Network meta-analysis, a method to synthesise evidence from multiple treatments, has increased in popularity in the past decade. Two broad approaches are available to synthesise data across networks, namely, arm- and contrast-synthesis models, with a range of models that can be fitted within each. There has been recent debate about the validity of the arm-synthesis models, but to date, there has been limited empirical evaluation comparing results using the methods applied to a large number of networks. We aim to address this gap through the re-analysis of a large cohort of published networks of interventions using a range of network meta-analysis methods. METHODS: We will include a subset of networks from a database of network meta-analyses of randomised trials that have been identified and curated from the published literature. The subset of networks will include those where the primary outcome is binary, the number of events and participants are reported for each direct comparison, and there is no evidence of inconsistency in the network. We will re-analyse the networks using three contrast-synthesis methods and two arm-synthesis methods. We will compare the estimated treatment effects, their standard errors, treatment hierarchy based on the surface under the cumulative ranking (SUCRA) curve, the SUCRA value, and the between-trial heterogeneity variance across the network meta-analysis methods. We will investigate whether differences in the results are affected by network characteristics and baseline risk. DISCUSSION: The results of this study will inform whether, in practice, the choice of network meta-analysis method matters, and if it does, in what situations differences in the results between methods might arise. The results from this research might also inform future simulation studies.