The Impact of Excluding Nonrandomized Studies From Systematic Reviews in Rare Diseases: "The Example of Meta-Analyses Evaluating the Efficacy and Safety of Enzyme Replacement Therapy in Patients With Mucopolysaccharidosis"

Nonrandomized studies are usually excluded from systematic reviews. This could lead to loss of a considerable amount of information on rare diseases. In this article, we explore the impact of excluding nonrandomized studies on the generalizability of meta-analyses results on mucopolysaccharidosis (MPS) disease. A comprehensive search of systematic reviews on MPS patients up to May 2020 was carried out (CRD42020191217). The primary endpoint was the rate of patients excluded from systematic reviews if only randomized studies were considered. Secondary outcomes included the differences in patient and study characteristics between randomized and nonrandomized studies, the methods used to combine data from studies with different designs, and the number of patients excluded from systematic reviews if case reports were not considered. More than 50% of the patients analyzed have been recruited in nonrandomized studies. Patient characteristics, duration of follow-up, and the clinical outcomes evaluated differ between the randomized and nonrandomized studies. There are feasible strategies to combine the data from different randomized and nonrandomized designs. The analyses suggest the relevance of including case reports in the systematic reviews, since the smaller the number of patients in the reference population, the larger the selection bias associated to excluding case reports. Our results recommend including nonrandomized studies in the systematic reviews of MPS to increase the representativeness of the results and to avoid a selection bias. The recommendations obtained from this study should be considered when conducting systematic reviews on rare diseases

No inexplicable disagreements between real-world data-based nonrandomized controlled studies and randomized controlled trials were found.

OBJECTIVES: We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs). STUDY DESIGN AND SETTING: We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements. RESULTS: We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement. CONCLUSION: Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates.

Pride and Prejudice during the COVID-19 Pandemic: The Misfortune of Inappropriate Clinical Trial Design.

Coronavirus Disease 2019 (COVID-19) is a rapidly evolving global pandemic for which more than a thousand clinical trials have been registered to secure therapeutic effectiveness, expeditiously. Most of these are single-center non-randomized studies rather than multi-center, randomized controlled trials. Single-arm trials have several limitations and may be conducted when spontaneous improvement is not anticipated, small placebo effect exists, and randomization to a placebo is not ethical. In an emergency where saving lives takes precedence, it is ethical to conduct trials with any scientifically proven design, however, safety must not be compromised. A phase II or III trial can be conducted directly in a pandemic with appropriate checkpoints and stopping rules. COVID-19 has two management paradigms- antivirals, or treatment of its complications. Simultaneous assessment of two different treatments can be done using 2 × 2 factorial schema. World Health Organization's SOLIDARITY trial is a classic example of the global research protocol which can evaluate the preferred treatment to combat COVID-19 pandemic. Short of that, a trial design must incorporate the practicality of the intervention used, and an appropriate primary endpoint which should ideally be a clinical outcome. Collaboration between institutions is needed more than ever to successfully execute and accrue in randomized trials.

Propensity weighting: how to minimise comparative bias in non-randomised studies?

Non-randomised study designs are frequently used by researchers in cardiovascular nursing and allied professions. Baseline differences between the groups to be compared may introduce bias in the results. Methods for causal inference address this issue. One such method is propensity weighting, in which two or more treatments/exposure groups are weighted to make the groups as comparable as possible. As such, it mimics a randomised controlled trial design. In this article, the Twang package is presented for propensity weighting, and its use is exemplified in a study on smoking and cannabis consumption in adults with congenital heart disease.

Risk of bias in nonrandomized studies of interventions showed low inter-rater reliability and challenges in its application.

OBJECTIVE: To assess the inter-rater reliability (IRR) and usability of the risk of bias in nonrandomized studies of interventions tool (ROBINS-I). STUDY DESIGN AND SETTING: We designed a cross-sectional study. Five raters independently applied ROBINS-I to the nonrandomized cohort studies in three systematic reviews on vaccines, opiate abuse, and rehabilitation. We calculated Fleiss' Kappa for multiple raters as a measure of IRR and discussed the application of ROBINS-I to identify difficulties and possible reasons for disagreement. RESULTS: Thirty one studies were included (195 evaluations). IRRs were slight for overall judgment (IRR 0.06, 95% CI 0.001 to 0.12) and individual domains (from 0.04, 95% CI -0.04 to 0.12 for the domain "selection of reported results" to 0.18, 95% CI 0.10 to 0.26 for the domain "deviation from intended interventions"). Mean time to apply the tool was 27.8 minutes (SD 12.6) per study. The main difficulties were due to poor reporting of primary studies, misunderstanding of the question, translation of questions into a final judgment, and incomplete guidance. CONCLUSION: We found ROBINS-I difficult and demanding, even for raters with substantial expertise in systematic reviews. Calibration exercises and intensive training before its application are needed to improve reliability.

Critical appraisal of nonrandomized studies-A review of recommended and commonly used tools.

RATIONALE, AIMS, AND OBJECTIVES: When randomized controlled trial data are limited or unavailable, or to supplement randomized controlled trial evidence, health technology assessment (HTA) agencies may rely on systematic reviews of nonrandomized studies (NRSs) for evidence of the effectiveness of health care interventions. NRS designs may introduce considerable bias into systematic reviews, and several methodologies by which to evaluate this risk of bias are available. This study aimed to identify tools commonly used to assess bias in NRS and determine those recommended by HTA bodies. METHODS: Appraisal tools used in NRS were identified through a targeted search of systematic reviews (January 2013-March 2017; MEDLINE and EMBASE [OVID SP]). Recommendations for the critical appraisal of NRS by expert review groups and HTA bodies were reviewed. RESULTS: From the 686 studies included in the narrative synthesis, 48 critical appraisal tools were identified. Commonly used tools included the Newcastle-Ottawa Scale, the methodological index for NRS, and bespoke appraisal tools. Neither the Cochrane Handbook nor the Centre for Reviews and Dissemination recommends a particular instrument for the assessment of risk of bias in NRS, although Cochrane has recently developed their own NRS critical appraisal tool. Among HTA bodies, only the Canadian Agency for Drugs and Technologies in Health recommends use of a specific critical appraisal tool-SIGN 50 (for cohort or case-control studies). Several criteria including reporting, external validity, confounding, and power were examined. CONCLUSION: There is no consensus between HTA groups on the preferred appraisal tool. Reviewers should select from a suite of tools on the basis of the design of studies included in their review.

Quality Improvement Feature Series Article 4: Advanced Designs for Quality Improvement Studies.

Each quality improvement (QI) project has an implicit study design, although these designs are not discussed as commonly as they are in clinical research. Most QI projects fall under the quasi-experimental study category, in which observations are made before and after the implementation of an intervention(s). The simplest and most commonly used for QI studies is the pre-post design, in which observations are made before and after each intervention that was implemented over a specified period. More sophisticated designs for QI studies enable a study team to draw stronger inferences about the system that is being changed and the individual effects of the interventions that are implemented. In the final commentary in this QI series, we discuss these study designs and focus on the strengths and weaknesses of more sophisticated designs, including cluster randomized, stepped-wedge, and factorial designs.

Quasi-experimental study designs series-paper 13: realizing the full potential of quasi-experiments for health research.

Although the number of quasi-experiments conducted by health researchers has increased in recent years, there clearly remains unrealized potential for using these methods for causal evaluation of health policies and programs globally. This article proposes five prescriptions for capturing the full value of quasi-experiments for health research. First, new funding opportunities targeting proposals that use quasi-experimental methods should be made available to a broad pool of health researchers. Second, administrative data from health programs, often amenable to quasi-experimental analysis, should be made more accessible to researchers. Third, training in quasi-experimental methods should be integrated into existing health science graduate programs to increase global capacity to use these methods. Fourth, clear guidelines for primary research and synthesis of evidence from quasi-experiments should be developed. Fifth, strategic investments should be made to continue to develop new innovations in quasi-experimental methodologies. Tremendous opportunities exist to expand the use of quasi-experimental methods to increase our understanding of which health programs and policies work and which do not. Health researchers should continue to expand their commitment to rigorous causal evaluation with quasi-experimental methods, and international institutions should increase their support for these efforts.

False Dichotomies and Health Policy Research Designs: Randomized Trials Are Not Always the Answer.

Some medical scientists argue that only data from randomized controlled trials (RCTs) are trustworthy. They claim data from natural experiments and administrative data sets are always spurious and cannot be used to evaluate health policies and other population-wide phenomena in the real world. While many acknowledge biases caused by poor study designs, in this article we argue that several valid designs using administrative data can produce strong findings, particularly the interrupted time series (ITS) design. Many policy studies neither permit nor require an RCT for cause-and-effect inference. Framing our arguments using Campbell and Stanley's classic research design monograph, we show that several "quasi-experimental" designs, especially interrupted time series (ITS), can estimate valid effects (or non-effects) of health interventions and policies as diverse as public insurance coverage, speed limits, hospital safety programs, drug abuse regulation and withdrawal of drugs from the market. We further note the recent rapid uptake of ITS and argue for expanded training in quasi-experimental designs in medical and graduate schools and in post-doctoral curricula.

The quality of control groups in nonrandomized studies published in the Journal of Hand Surgery.

PURPOSE: To evaluate control group selection in nonrandomized studies published in the Journal of Hand Surgery American (JHS). METHODS: We reviewed all papers published in JHS in 2013 to identify studies that used nonrandomized control groups. Data collected included type of study design and control group characteristics. We then appraised studies to determine whether authors discussed confounding and selection bias and how they controlled for confounding. RESULTS: Thirty-seven nonrandomized studies were published in JHS in 2013. The source of control was either the same institution as the study group, a different institution, a database, or not provided in the manuscript. Twenty-nine (78%) studies statistically compared key characteristics between control and study group. Confounding was controlled with matching, exclusion criteria, or regression analysis. Twenty-two (59%) papers explicitly discussed the threat of confounding and 18 (49%) identified sources of selection bias. CONCLUSIONS: In our review of nonrandomized studies published in JHS, papers had well-defined controls that were similar to the study group, allowing for reasonable comparisons. However, we identified substantial confounding and bias that were not addressed as explicit limitations, which might lead the reader to overestimate the scientific validity of the data. CLINICAL RELEVANCE: Incorporating a brief discussion of control group selection in scientific manuscripts should help readers interpret the study more appropriately. Authors, reviewers, and editors should strive to address this component of clinical importance.

Development of an interventional pain management specific instrument for methodologic quality assessment of nonrandomized studies of interventional techniques.

BACKGROUND: The major component of a systematic review is assessment of the methodologic quality and bias of randomized and nonrandomized trials. While there are multiple instruments available to assess the methodologic quality and bias for randomized controlled trials (RCTs), there is a lack of extensively utilized instruments for observational studies, specifically for interventional pain management (IPM) techniques. Even Cochrane review criteria for randomized trials is considered not to be a "gold standard," but merely an indication of the current state of the art review methodology. Recently a specific instrument to assess the methodologic quality of randomized trials has been developed for interventional techniques. OBJECTIVES: Our objective was to develop an IPM specific instrument to assess the methodological quality of nonrandomized trials or observational studies of interventional techniques. METHODS: The item generation for the instrument was based on a definition of quality, to the extent to which the design and conduct of the trial were congruent with the objectives of the study. Applicability was defined as the extent to which procedures produced by the study could be applied using contemporary IPM techniques. Multiple items based on Cochrane review criteria and Interventional Pain Management Techniques - Quality Appraisal of Reliability and Risk of Bias Assessment for Nonrandomized Studies (IPM-QRBNR) were utilized. RESULTS: A total of 16 items were developed which formed the IPM-QRBNR tool. The assessment was performed in multiple stages. The final assessment was 4 nonrandomized studies. The inter-rater agreement was moderate to good for IPM-QRBNR criteria. LIMITATIONS: Limited validity or accuracy assessment of the instrument and the large number of items to be scored were limitations. CONCLUSION: We have developed a new comprehensive instrument to assess the methodological quality of nonrandomized studies of interventional techniques. This instrument provides extensive information specific to interventional techniques is useful in assessing the methodological quality and bias of observational studies of interventional techniques.