Bias Assessment in Outcomes Research: The Role of Relative Versus Absolute Approaches.

OBJECTIVES: Bias assessment tools vary in content and detail, and the method used for assessment may produce different assessment results in a study if not carefully considered. Therefore, taking an approach to the assessment of studies that produces a similar result regardless of the tool used for assessment (tool independence) is important. METHODS: A preexisting study that used 25 different quality scales was assessed to examine tool dependence of 2 common approaches to bias assessments-absolute value judgments (defined as the qualitative risk of bias judgment based on a threshold across studies) and relative ranks (defined as the relative probability toward bias of a study relative to the best assessed study). Agreement between each of the 25 scales and a composite scale (that includes all unique safeguards across all scales) was computed (using the intraclass correlation coefficient [ICC]; consistency). Tool dependence was considered present when the ICCs were inconsistent across the 25 scales for the same study. RESULTS: We found that using relative ranks for tools with different numbers and types of items produced consistent results, with only small differences in the agreement for the various tools with the composite tool, whereas consistency (measured by the ICC) varied considerably when using absolute judgments. Inconsistency is problematic because it means that the assessment result is linked to the scale and not to the study. CONCLUSIONS: Tool independence is an important attribute of a bias assessment tool. On the basis of this study, the use of relative ranks retains tool independence and therefore produces consistent ranks for the same study across tools.

Was there really any evidence that rosiglitazone increased the risk of myocardial infarction or death from cardiovascular causes?

Rosiglitazone has previously been widely used to treat patients with type 2 diabetes mellitus, but its safety in terms of cardiovascular morbidity and mortality had been called into question. Recently, there have been doubts raised about the meta-analytic evidence with the regulatory authorities relaxing its restrictions. We hypothesized that the original analyses may have produced exaggerated results because of the small baseline risks involved. To demonstrate this, we replicated the meta-analysis of four randomized trials of greater than 12-month follow-up that made use of a randomized control group not receiving rosiglitazone and reported outcome data for all occurrences of the complementary outcomes (no myocardial infarction, no death from cardiovascular causes, and no heart failure). Data were combined by means of a fixed-effects model. In the rosiglitazone group, as compared with the control group, the relative risk for no myocardial infarction was 0.997 (95% confidence interval [CI], 0.994 to 1.000), and the relative risk for no death from cardiovascular causes was 1.001 (95%CI, 0.999 to 1.003). Finally, no heart failure had a relative risk of 0.995 (95%CI, 0.993 to 0.998). Rosiglitazone does not seem to have any significant increase in the risk of myocardial infarction or of death from cardiovascular causes associated with its use. Regulatory authorities should revisit this issue of the appropriate measure for reporting of adverse events with low baseline risks as this has implications well beyond rosiglitazone.

Estrogen and selective estrogen receptor modulators (SERMs) for the treatment of acromegaly: a meta-analysis of published observational studies.

Estrogen and selective estrogen receptor modulator (SERM) treatments for acromegaly have received limited attention since the development of newer pharmacologic therapies. There has been ongoing research evidence suggesting their utility in the biochemical control of acromegaly. Therefore, the aim of this meta-analysis was to synthesise current evidence with a view to determining to what extent and in which acromegalic patient subsets do estrogen and SERMs reduce IGF-1 levels. A literature search was conducted (finished December 2012), which included all studies pertaining to estrogen or SERM treatment and IGF-1. Seven patient subsets were identified from six published observational studies, and were pooled using meta-analytic methods. Overall, the pooled mean loss in IGF-1 was -29.09 nmol/L (95 % CI -37.23 to -20.95). A sensitivity analysis indicated that women receiving estrogen had a substantially greater reduction in IGF-1 levels compared with women receiving SERMs, with a weighted mean loss in IGF-1 of -38.12 nmol/L (95 % CI -46.78 to -29.45) compared with -22.91 nmol/L (95 % CI -32.73 to -13.09). There was a trend that did not reach statistical significance for men receiving SERM treatment at -11.41 nmol/L (95 % CI -30.14 to 7.31). It was concluded that estrogen and SERMs are a low cost and effective treatment to achieve control of IGF-1 levels in acromegalic women either as concomitant treatment for refractory disease, or where access to conventional therapy is restricted. Their use in men requires further study.

Radiographic, histologic, and arthroscopic findings in amorphous calcifications of the hip labrum.

PURPOSE: The purpose of this study was to evaluate the clinical, radiographic, histologic, and intraoperative findings of an amorphous calcification involving the acetabular labrum. METHODS: From October 2008 to April 2013, all patients who underwent arthroscopic hip surgery for symptomatic intra-articular hip disorders and were found to have the characteristic calcific deposit involving the acetabular labrum were included. These patients were reviewed retrospectively on prospectively collected data. Radiographs were retrospectively evaluated for morphologic features of impingement and characteristics of labral calcification. RESULTS: Sixteen patients were identified as having amorphous calcification at the time of arthroscopy. There were 15 women and 1 man. Mean age was 37.3 years (range, 30 to 50 years). Symptoms were present for a mean of 9.3 months (range, 3 to 48 months). All patients reported anterior groin pain. Fifteen (94%) patients had positive anterior impingement and 9 (56%) had positive results for lateral impingement. Calcifications measured on average 3.2 mm (range, 1.9 mm to 5.6 mm), and 14 had a clear separation from the rim with increased opacity compared with neighboring trabecular bone. Intraoperatively, the characteristic amorphous calcium deposit was located in the anterosuperior labrum, with the deposit found to be accessible from the capsule-labral recess in all cases. All patients had labral tears and all patients had at least one component of femoroacetabular impingement (FAI). CONCLUSIONS: Calcification in the anterosuperior acetabular labrum presents with a consistent patient demographic and distinct radiographic and arthroscopic presentation that is different from os acetabuli. As with os acetabuli, one should have a high suspicion for FAI when this lesion is encountered. LEVEL OF EVIDENCE: Level IV, therapeutic case series.

Comparison of bias adjustment in meta-analysis using data-based and opinion-based methods.

INTRODUCTION: Several methods exist for bias adjustment of meta-analysis results, but there has been no comprehensive comparison with unadjusted methods. We compare 6 bias-adjustment methods with 2 unadjusted methods to examine how these different methods perform. METHODS: We re-analyzed a meta-analysis that included 10 randomized controlled trials. Two data-based methods (Welton's data-based approach and Doi's quality effects model) and 4 opinion-informed methods (opinion-based approach, opinion-based distributions combined statistically with data-based distributions, numerical opinions informed by data-based distributions, and opinions obtained by selecting areas from data-based distributions) were used to incorporate methodological quality information into the meta-analytical estimates. The results of these 6 methods were compared with 2 unadjusted models: the DerSimonian-Laird random effects model and Doi's inverse variance heterogeneity model. RESULTS: The 4 opinion-based methods returned the random effects model estimates with wider uncertainty. The data-based and quality effects methods returned different results and aligned with the inverse variance heterogeneity method with some minor downward bias adjustment. CONCLUSION: Opinion-based methods seem to only add uncertainty rather than bias adjust.

Common tool structures and approaches to risk of bias assessment: implications for systematic reviewers.

There are numerous tools available to assess the risk of bias in individual studies in a systematic review. These tools have different structures, including scales and checklists, which may or may not separate their items by domains. There are also various approaches and guides for the process, scoring, and interpretation of risk of bias assessments, such as value judgments, quality scores, and relative ranks. The objective of this commentary, which is part of the JBI Series on Risk of Bias, is to discuss some of the distinctions among different tool structures and approaches to risk of bias assessment and the implications of these approaches for systematic reviewers.

The revised JBI critical appraisal tool for the assessment of risk of bias for quasi-experimental studies.

Systematic reviews of effectiveness offer a rigorous synthesis of the best evidence available regarding the effects of interventions or treatments. Randomized controlled trials are considered the optimal study design for evaluating the effectiveness of interventions and are the ideal study design for inclusion in a systematic review of effectiveness. In the absence of randomized controlled trials, quasi-experimental studies may be relied on to provide information on treatment or intervention effectiveness. However, such studies are subject to unique considerations regarding their internal validity and, consequently, the assessment of the risk of bias of these studies needs to consider these features of design and conduct. The JBI Effectiveness Methodology Group has recently commenced updating the suite of JBI critical appraisal tools for quantitative study designs to align with the latest advancements in risk of bias assessment. This paper presents the revised critical appraisal tool for risk of bias assessment of quasi-experimental studies; offers practical guidance for its use; provides examples for interpreting the results of risk of bias assessment; and discusses major changes from the previous version, along with the justifications for those changes.

Implementing GRADE in systematic reviews that adhere to JBI methodological conduct.

GRADE is a methodological approach used to establish certainty in a body of evidence and is now widely adopted among the evidence synthesis and guideline development community. JBI is an international evidence-based health care organization that provides guidance for a range of evidence synthesis approaches. The GRADE approach is currently endorsed for use in a subset of JBI systematic reviews; however, there is some uncertainty regarding when (and how) GRADE may be implemented in reviews that follow JBI methodology.

Predatory journals and their practices present a conundrum for systematic reviewers and evidence synthesisers of health research: A qualitative descriptive study.

Predatory journals are a blemish on scholarly publishing and academia and the studies published within them are more likely to contain data that is false. The inclusion of studies from predatory journals in evidence syntheses is potentially problematic due to this propensity for false data to be included. To date, there has been little exploration of the opinions and experiences of evidence synthesisers when dealing with predatory journals in the conduct of their evidence synthesis. In this paper, the thoughts, opinions, and attitudes of evidence synthesisers towards predatory journals and the inclusion of studies published within these journals in evidence syntheses were sought. Focus groups were held with participants who were experienced evidence synthesisers from JBI (previously the Joanna Briggs Institute) collaboration. Utilising qualitative content analysis, two generic categories were identified: predatory journals within evidence synthesis, and predatory journals within academia. Our findings suggest that evidence synthesisers believe predatory journals are hard to identify and that there is no current consensus on the management of these studies if they have been included in an evidence synthesis. There is a critical need for further research, education, guidance, and development of clear processes to assist evidence synthesisers in the management of studies from predatory journals.

Methodological quality assessment should move beyond design specificity.

OBJECTIVE: This study aimed to assess the utility of a unified tool (MASTER) for bias assessment against design-specific tools in terms of content and coverage. METHODS: Each of the safeguards in the design-specific tools was compared and matched to safeguards in the unified MASTER scale. The design-specific tools were the JBI, Scottish Intercollegiate Guidelines Network (SIGN), and the Newcastle-Ottawa Scale (NOS) tools for analytic study designs. Duplicates, safeguards that could not be mapped to the MASTER scale, and items not applicable as safeguards against bias were flagged and described. RESULTS: Many safeguards across the JBI, SIGN, and NOS tools were common, with a minimum of 10 to a maximum of 23 unique safeguards across various tools. These 3 design-specific toolsets were missing 14 to 26 safeguards from the MASTER scale. The MASTER scale had complete coverage of safeguards within the 3 toolsets for analytic designs. CONCLUSIONS: The MASTER scale provides a unified framework for bias assessment of analytic study designs, has good coverage, avoids duplication, has less redundancy, and is more convenient when used for methodological quality assessment in evidence synthesis. It also allows assessment across designs that cannot be done using a design-specific tool.

Effectiveness of dual active ingredient insecticide-treated nets in preventing malaria: A systematic review and meta-analysis.

Malaria vectors have demonstrated resistance to pyrethroid-based insecticides used in insecticide-treated nets, diminishing their effectiveness. This systematic review and meta-analysis investigated two forms of dual active-ingredient (DAI) insecticide-treated nets (ITN(s)) for malaria prevention. A comprehensive search was conducted on July 6th 2022. The databases searched included PubMed, Embase, CINAHL, amongst others. Trials were eligible if they were conducted in a region with ongoing malaria transmission. The first DAI ITN investigated were those that combined a pyrethroid with a non-pyrethroid insecticides. The second DAI ITN investigated were that combined a pyrethroid with an insect growth regulator. These interventions were compared against either a pyrethroid-only ITN, or ITNs treated with pyrethroid and piperonyl-butoxide. Assessment of risk of bias was conducted in duplicate using the Cochrane risk of bias 2 tool for cluster-randomised trials. Summary data was extracted using a custom data-extraction instrument. This was conducted by authors THB, JCS and SH. Malaria case incidence was the primary outcome and has been meta-analysed, adverse events were narratively synthesised. The review protocol is registered on PROSPERO (CRD42022333044). From 9494 records, 48 reports were screened and 13 reports for three studies were included. These studies contained data from 186 clusters and all reported a low risk of bias. Compared to pyrethroid-only ITNs, clusters that received pyrethroid-non-pyrethroid DAI ITNs were associated with 305 fewer cases per 1000-person years (from 380 fewer cases to 216 fewer cases) (IRR = 0.55, 95%CI: 0.44-0.68). However, this trend was not observed in clusters that received pyrethroid-insect growth regulator ITNs compared to pyrethroid-only ITNs (from 280 fewer cases to 135 more) (IRR = 0.90, 95%CI: 0.73-1.13). Pyrethroid-non-pyrethroid DAI ITNs demonstrated consistent reductions in malaria case incidence and other outcomes across multiple comparisons. Pyrethroid-non-pyrethroid DAI ITNs may present a novel intervention for the control of malaria.

Reporting quality and risk of bias in JBI systematic reviews evaluating the effectiveness of interventions: a methodological review protocol.

OBJECTIVE: The objective of this methodological review is to evaluate the adherence of systematic reviews of effectiveness published in JBI Evidence Synthesis to reporting guidelines and methodological quality. INTRODUCTION: Systematic reviews of effectiveness are essential tools for health practitioners and policy-makers. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines and the Risk of Bias in Systematic Reviews (ROBIS) tool are used to ensure maintenance of high reporting standards and methodological quality, respectively. This review will utilize these tools to identify strengths and shortfalls in the reporting quality of JBI systematic reviews of effectiveness. INCLUSION CRITERIA: This review will include the 20 most recent systematic reviews of effectiveness published in JBI Evidence Synthesis . METHODS: This review will search MEDLINE (PubMed) for effectiveness reviews published in JBI Evidence Synthesis . Abstract and full-text screening will be performed by 2 independent reviewers, and the most recent 20 studies will be selected for inclusion. Data regarding adherence to PRISMA 2020 and ROBIS will be extracted by 2 independent reviewers. Data will be presented descriptively with tables and synthesized narratively.

The revised JBI critical appraisal tool for the assessment of risk of bias for randomized controlled trials.

JBI recently began the process of updating and revising its suite of critical appraisal tools to ensure that these tools remain compatible with recent developments within risk of bias science. Following a rigorous development process led by the JBI Effectiveness Methodology Group, this paper presents the revised critical appraisal tool for the assessment of risk of bias for randomized controlled trials. This paper also presents practical guidance on how the questions of this tool are to be interpreted and applied by systematic reviewers, while providing topical examples. We also discuss the major changes made to this tool compared to the previous version and justification for why these changes facilitate best-practice methodologies in this field.

Assessing the risk of bias of quantitative analytical studies: introducing the vision for critical appraisal within JBI systematic reviews.

A key step in the systematic review process is the assessment of the methodological quality (or risk of bias) of the included studies. At JBI, we have developed several tools to assist with this evaluation. As evidence synthesis methods continue to evolve, it has been necessary to revise and reflect on JBI's current approach to critical appraisal and to plan a strategy for the future. In this first paper of a series focusing on risk of bias assessment, we introduce our vision for risk of bias assessment for JBI. In future papers in this series, the methodological approach taken for this revision process will be discussed, along with the revised tools and guidance for using these tools.

Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process.

JBI offers a suite of critical appraisal instruments that are freely available to systematic reviewers and researchers investigating the methodological limitations of primary research studies. The JBI instruments are designed to be study-specific and are presented as questions in a checklist. The JBI instruments have existed in a checklist-style format for approximately 20 years; however, as the field of research synthesis expands, many of the tools offered by JBI have become outdated. The JBI critical appraisal tools for quantitative studies (eg, randomized controlled trials, quasi-experimental studies) must be updated to reflect the current methodologies in this field. Cognizant of this and the recent developments in risk-of-bias science, the JBI Effectiveness Methodology Group was tasked with updating the current quantitative critical appraisal instruments. This paper details the methods and rationale that the JBI Effectiveness Methodology Group followed when updating the JBI critical appraisal instruments for quantitative study designs. We detail the key changes made to the tools and highlight how these changes reflect current methodological developments in this field.

Reporting of 3Rs Approaches in Preclinical Animal Experimental Studies-A Nationwide Study.

The 3Rs aim to refine animal welfare, reduce animal numbers, and replace animal experiments. Investigations disclose that researchers are positive towards 3Rs recommendations from peers. Communication of 3Rs approaches via primary preclinical animal experimental literature may become a fast-forward extension to learn relevant 3Rs approaches if such are reported. This study investigates 3Rs-reporting in peer-reviewed preclinical animal research with at least one author affiliated to a Danish university. Using a systematic search and random sampling, we included 500 studies from 2009 and 2018. Reporting was low and improvement over time limited. A word search for 3R retrieved zero results in 2009 and 3.2% in 2018. Reporting on 3Rs-related sentences increased from 6.4% in 2009 to 18.4% in 2018, "reduction" increased from 2.4% to 8.0%, and "refinement" from 5.2% to 14.4%. Replacement was not reported. Reporting of the methodology was missing. For "reduction", methodology was mentioned in one study in 2009 and 11 studies in 2018, and for "refinement" in 9 and 21, respectively. Twenty-one studies stated compliance with ARRIVE-guidelines or similar without disclosure of details. Reporting of 3Rs approaches in preclinical publications is currently insufficient to guide researchers. Other strategies, e.g., education, interdisciplinary collaboration, and 3Rs funding initiatives, are needed.

From critical appraisal to risk of bias assessment: clarifying the terminology for study evaluation in JBI systematic reviews.

The foundations for critical appraisal of literature have largely progressed through the development of epidemiologic research methods and the use of research to inform medical teaching and practice. This practical application of research is referred to as evidence-based medicine and has delivered a standard for the health care profession where clinicians are equally as engaged in conducting scientific research as they are in the practice of delivering treatments. Evidence-based medicine, now referred to as evidence-based health care, has generally been operationalized through empirically supported treatments, whereby the choice of treatments is substantiated by scientific support, usually by means of an evidence synthesis. As evidence synthesis methodology has advanced, guidance for the critical appraisal of primary research has emphasized a distinction from the assessment of internal validity required for synthesized research. This assessment is conceptualized and branded in various ways in the literature, such as risk of bias, critical appraisal, study validity, methodological quality, and methodological limitations. This paper provides a discussion of the definitions and characteristics of these terms, concluding with a recommendation for JBI to adopt the term "risk of bias" assessment.

Residual insecticide surface treatment for preventing malaria: a systematic review protocol.

INTRODUCTION: Malaria presents a significant global public health burden, although substantial progress has been made, with vector control initiatives such as indoor residual surface spraying with insecticides and insecticide-treated nets. There now exists many different approaches to apply residual insecticide to indoor and outdoor surfaces in malaria-endemic settings, although no comprehensive systematic reviews exist evaluating these interventions. This manuscript outlines the protocol for a systematic review which aims to synthesise the best available evidence regarding full or partial indoor or outdoor residual insecticide surface treatment for preventing malaria. METHODS AND ANALYSIS: This review will comprehensively search the literature (both published and unpublished) for any studies investigating the effectiveness of residual insecticide surface treatment for malaria. Studies will be screened to meet the inclusion criteria by a minimum of two authors, followed by assessment of risk of bias (using appropriate risk-of-bias tools for randomised and non-randomised studies) and extraction of relevant information using structured forms by two independent authors. Meta-analysis will be carried out where possible for epidemiological outcomes such as malaria, anaemia, malaria-related mortality, all-cause mortality and adverse effects. Certainty in the evidence will be established with GRADE assessments. ETHICS AND DISSEMINATION: A full review report will be submitted to the Vector Control & Insecticide Resistance Unit, Global Malaria Program, WHO. A version of this report will be submitted for publication in an open access peer-reviewed journal. The report will inform the development of WHO recommendations regarding residual insecticide treatment for malaria. This systematic review does not require ethics approval as it is a review of primary studies. SYSTEMATIC REVIEW REGISTRATION: PROSPERO 293194.

Investigating different typologies for the synthesis of evidence: a scoping review protocol.

OBJECTIVE: The objective of this scoping review is to identify evidence synthesis types and previously proposed classification systems, typologies, or taxonomies that have guided evidence synthesis. INTRODUCTION: Evidence synthesis is a constantly evolving field. There is now a plethora of evidence synthesis approaches used across many different disciplines. Historically, there have been numerous attempts to organize the types and methods of evidence synthesis in the form of classification systems, typologies, or taxonomies. This scoping review will seek to identify all the available classification systems, typologies, or taxonomies; how they were developed; their characteristics; and the types of evidence syntheses included within them. INCLUSION CRITERIA: This scoping review will include discussion papers, commentaries, books, editorials, manuals, handbooks, and guidance from major organizations that describe multiple approaches to evidence synthesis in any discipline. METHODS: The Evidence Synthesis Taxonomy Initiative will support this scoping review. The search strategy will aim to locate both published and unpublished documents utilizing a three-step search strategy. An exploratory search of MEDLINE has identified keywords and MeSH terms. A second search of MEDLINE, Embase, CINAHL with Full Text, ERIC, Scopus, Compendex, and JSTOR will be conducted. The websites of relevant evidence synthesis organizations will be searched. Identified documents will be independently screened, selected, and extracted by two researchers, and the data will be presented in tables and summarized descriptively. DETAILS OF THIS REVIEW PROJECT ARE AVAILABLE AT: Open Science Framework https://osf.io/qwc27.

How should we handle predatory journals in evidence synthesis? A descriptive survey-based cross-sectional study of evidence synthesis experts.

Synthesizers of evidence are increasingly likely to encounter studies published in predatory journals during the evidence synthesis process. The evidence synthesis discipline is uniquely positioned to encounter novel concerns associated with predatory journals. The objective of this research was to explore the attitudes, opinions, and experiences of experts in the synthesis of evidence regarding predatory journals. Employing a descriptive survey-based cross-sectional study design, these experts were asked a series of questions regarding predatory journals to explore these attitudes, opinions, and experiences. Two hundred and sixty four evidence synthesis experts responded to this survey. Most respondents agreed with the definition of a predatory journal (86%), however several (19%) responded that this definition was difficult to apply practically. Many respondents believed that studies published in predatory journals are still eligible for inclusion into an evidence synthesis project. However, this was only after the study had been determined to be 'high-quality' (39%) or if the results were validated (13%). While many respondents could identify common characteristics of these journals, there was still hesitancy regarding the appropriate methods to follow when considering including these studies into an evidence synthesis project.