Interventions on gender equity in the workplace: a scoping review.

BACKGROUND: Various studies have demonstrated gender disparities in workplace settings and the need for further intervention. This study identifies and examines evidence from randomized controlled trials (RCTs) on interventions examining gender equity in workplace or volunteer settings. An additional aim was to determine whether interventions considered intersection of gender and other variables, including PROGRESS-Plus equity variables (e.g., race/ethnicity). METHODS: Scoping review conducted using the JBI guide. Literature was searched in MEDLINE, Embase, PsycINFO, CINAHL, Web of Science, ERIC, Index to Legal Periodicals and Books, PAIS Index, Policy Index File, and the Canadian Business & Current Affairs Database from inception to May 9, 2022, with an updated search on October 17, 2022. Results were reported using Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension to scoping reviews (PRISMA-ScR), Sex and Gender Equity in Research (SAGER) guidance, Strengthening the Integration of Intersectionality Theory in Health Inequality Analysis (SIITHIA) checklist, and Guidance for Reporting Involvement of Patients and the Public (GRIPP) version 2 checklist. All employment or volunteer sectors settings were included. Included interventions were designed to promote workplace gender equity that targeted: (a) individuals, (b) organizations, or (c) systems. Any comparator was eligible. Outcomes measures included any gender equity related outcome, whether it was measuring intervention effectiveness (as defined by included studies) or implementation. Data analyses were descriptive in nature. As recommended in the JBI guide to scoping reviews, only high-level content analysis was conducted to categorize the interventions, which were reported using a previously published framework. RESULTS: We screened 8855 citations, 803 grey literature sources, and 663 full-text articles, resulting in 24 unique RCTs and one companion report that met inclusion criteria. Most studies (91.7%) failed to report how they established sex or gender. Twenty-three of 24 (95.8%) studies reported at least one PROGRESS-Plus variable: typically sex or gender or occupation. Two RCTs (8.3%) identified a non-binary gender identity. None of the RCTs reported on relationships between gender and other characteristics (e.g., disability, age, etc.). We identified 24 gender equity promoting interventions in the workplace that were evaluated and categorized into one or more of the following themes: (i) quantifying gender impacts; (ii) behavioural or systemic changes; (iii) career flexibility; (iv) increased visibility, recognition, and representation; (v) creating opportunities for development, mentorship, and sponsorship; and (vi) financial support. Of these interventions, 20/24 (83.3%) had positive conclusion statements for their primary outcomes (e.g., improved academic productivity, increased self-esteem) across heterogeneous outcomes. CONCLUSIONS: There is a paucity of literature on interventions to promote workplace gender equity. While some interventions elicited positive conclusions across a variety of outcomes, standardized outcome measures considering specific contexts and cultures are required. Few PROGRESS-Plus items were reported. Non-binary gender identities and issues related to intersectionality were not adequately considered. Future research should provide consistent and contemporary definitions of gender and sex. TRIAL REGISTRATION: Open Science Framework https://osf.io/x8yae .

Embedding rapid reviews in health policy and systems decision-making: Impacts and lessons learned from four low- and middle-income countries.

BACKGROUND: Demand for rapid evidence-based syntheses to inform health policy and systems decision-making has increased worldwide, including in low- and middle-income countries (LMICs). To promote use of rapid syntheses in LMICs, the WHO's Alliance for Health Policy and Systems Research (AHPSR) created the Embedding Rapid Reviews in Health Systems Decision-Making (ERA) Initiative. Following a call for proposals, four LMICs were selected (Georgia, India, Malaysia and Zimbabwe) and supported for 1 year to embed rapid response platforms within a public institution with a health policy or systems decision-making mandate. METHODS: While the selected platforms had experience in health policy and systems research and evidence syntheses, platforms were less confident conducting rapid evidence syntheses. A technical assistance centre (TAC) was created from the outset to develop and lead a capacity-strengthening program for rapid syntheses, tailored to the platforms based on their original proposals and needs as assessed in a baseline questionnaire. The program included training in rapid synthesis methods, as well as generating synthesis demand, engaging knowledge users and ensuring knowledge uptake. Modalities included live training webinars, in-country workshops and support through phone, email and an online platform. LMICs provided regular updates on policy-makers' requests and the rapid products provided, as well as barriers, facilitators and impacts. Post-initiative, platforms were surveyed. RESULTS: Platforms provided rapid syntheses across a range of AHPSR themes, and successfully engaged national- and state-level policy-makers. Examples of substantial policy impact were observed, including for COVID-19. Although the post-initiative survey response rate was low, three quarters of those responding felt confident in their ability to conduct a rapid evidence synthesis. Lessons learned coalesced around three themes - the importance of context-specific expertise in conducting reviews, facilitating cross-platform learning, and planning for platform sustainability. CONCLUSIONS: The ERA initiative successfully established rapid response platforms in four LMICs. The short timeframe limited the number of rapid products produced, but there were examples of substantial impact and growing demand. We emphasize that LMICs can and should be involved not only in identifying and articulating needs but as co-designers in their own capacity-strengthening programs. More time is required to assess whether these platforms will be sustained for the long-term.

Quality improvement strategies to prevent falls in older adults: a systematic review and network meta-analysis.

BACKGROUND: Falls are a common occurrence and the most effective quality improvement (QI) strategies remain unclear. METHODS: We conducted a systematic review and network meta-analysis (NMA) to elucidate effective quality improvement (QI) strategies for falls prevention. Multiple databases were searched (inception-April 2017). We included randomised controlled trials (RCTs) of falls prevention QI strategies for participants aged ≥65 years. Two investigators screened titles and abstracts, full-text articles, conducted data abstraction and appraised risk of bias independently. RESULTS: A total of 126 RCTs including 84,307 participants were included after screening 10,650 titles and abstracts and 1210 full-text articles. NMA including 29 RCTs and 26,326 patients found that team changes was statistically superior in reducing the risk of injurious falls relative to usual care (odds ratio [OR] 0.57 [0.33 to 0.99]; absolute risk difference [ARD] -0.11 [95% CI, -0.18 to -0.002]). NMA for the outcome of number of fallers including 61 RCTs and 40 128 patients found that combined case management, patient reminders and staff education (OR 0.18 [0.07 to 0.47]; ARD -0.27 [95% CI, -0.33 to -0.15]) and combined case management and patient reminders (OR, 0.36 [0.13 to 0.97]; ARD -0.19 [95% CI, -0.30 to -0.01]) were both statistically superior compared to usual care. CONCLUSIONS: Team changes may reduce risk of injurious falls and a combination of case management, patient reminders, and staff education, as well as case management and patient reminders may reduce risk of falls. Our results can be tailored to decision-maker preferences and availability of resources. SYSTEMATIC REVIEW REGISTRATION: PROSPERO (CRD42013004151).

Utility of social media and crowd-intelligence data for pharmacovigilance: a scoping review.

BACKGROUND: A scoping review to characterize the literature on the use of conversations in social media as a potential source of data for detecting adverse events (AEs) related to health products. METHODS: Our specific research questions were (1) What social media listening platforms exist to detect adverse events related to health products, and what are their capabilities and characteristics? (2) What is the validity and reliability of data from social media for detecting these adverse events? MEDLINE, EMBASE, Cochrane Library, and relevant websites were searched from inception to May 2016. Any type of document (e.g., manuscripts, reports) that described the use of social media data for detecting health product AEs was included. Two reviewers independently screened citations and full-texts, and one reviewer and one verifier performed data abstraction. Descriptive synthesis was conducted. RESULTS: After screening 3631 citations and 321 full-texts, 70 unique documents with 7 companion reports available from 2001 to 2016 were included. Forty-six documents (66%) described an automated or semi-automated information extraction system to detect health product AEs from social media conversations (in the developmental phase). Seven pre-existing information extraction systems to mine social media data were identified in eight documents. Nineteen documents compared AEs reported in social media data with validated data and found consistent AE discovery in all except two documents. None of the documents reported the validity and reliability of the overall system, but some reported on the performance of individual steps in processing the data. The validity and reliability results were found for the following steps in the data processing pipeline: data de-identification (n = 1), concept identification (n = 3), concept normalization (n = 2), and relation extraction (n = 8). The methods varied widely, and some approaches yielded better results than others. CONCLUSIONS: Our results suggest that the use of social media conversations for pharmacovigilance is in its infancy. Although social media data has the potential to supplement data from regulatory agency databases; is able to capture less frequently reported AEs; and can identify AEs earlier than official alerts or regulatory changes, the utility and validity of the data source remains under-studied. TRIAL REGISTRATION: Open Science Framework ( https://osf.io/kv9hu/ ).

Comparisons of Interventions for Preventing Falls in Older Adults: A Systematic Review and Meta-analysis.

Importance: Falls result in substantial burden for patients and health care systems, and given the aging of the population worldwide, the incidence of falls continues to rise. Objective: To assess the potential effectiveness of interventions for preventing falls. Data Sources: MEDLINE, Embase, Cochrane Central Register of Controlled Trials, and Ageline databases from inception until April 2017. Reference lists of included studies were scanned. Study Selection: Randomized clinical trials (RCTs) of fall-prevention interventions for participants aged 65 years and older. Data Extraction and Synthesis: Pairs of reviewers independently screened the studies, abstracted data, and appraised risk of bias. Pairwise meta-analysis and network meta-analysis were conducted. Main Outcomes and Measures: Injurious falls and fall-related hospitalizations. Results: A total of 283 RCTs (159 910 participants; mean age, 78.1 years; 74% women) were included after screening of 10 650 titles and abstracts and 1210 full-text articles. Network meta-analysis (including 54 RCTs, 41 596 participants, 39 interventions plus usual care) suggested that the following interventions, when compared with usual care, were associated with reductions in injurious falls: exercise (odds ratio [OR], 0.51 [95% CI, 0.33 to 0.79]; absolute risk difference [ARD], -0.67 [95% CI, -1.10 to -0.24]); combined exercise and vision assessment and treatment (OR, 0.17 [95% CI, 0.07 to 0.38]; ARD, -1.79 [95% CI, -2.63 to -0.96]); combined exercise, vision assessment and treatment, and environmental assessment and modification (OR, 0.30 [95% CI, 0.13 to 0.70]; ARD, -1.19 [95% CI, -2.04 to -0.35]); and combined clinic-level quality improvement strategies (eg, case management), multifactorial assessment and treatment (eg, comprehensive geriatric assessment), calcium supplementation, and vitamin D supplementation (OR, 0.12 [95% CI, 0.03 to 0.55]; ARD, -2.08 [95% CI, -3.56 to -0.60]). Pairwise meta-analyses for fall-related hospitalizations (2 RCTs; 516 participants) showed no significant association between combined clinic- and patient-level quality improvement strategies and multifactorial assessment and treatment relative to usual care (OR, 0.78 [95% CI, 0.33 to 1.81]). Conclusions and Relevance: Exercise alone and various combinations of interventions were associated with lower risk of injurious falls compared with usual care. Choice of fall-prevention intervention may depend on patient and caregiver values and preferences.

Comparative efficacy of serotonin (5-HT3) receptor antagonists in patients undergoing surgery: a systematic review and network meta-analysis.

BACKGROUND: Serotonin (5-HT3) receptor antagonists are commonly used to decrease nausea and vomiting for surgery patients. We conducted a systematic review on the comparative efficacy of 5-HT3 receptor antagonists. METHODS: Searches were done in MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials to identify studies comparing 5-HT3 receptor antagonists with each other, placebo, and/or combined with other antiemetic agents for patients undergoing surgical procedures. Screening search results, data abstraction, and risk of bias assessment were conducted by two reviewers independently. Random-effects pairwise meta-analysis and network meta-analysis (NMA) were conducted. PROSPERO registry number: CRD42013003564. RESULTS: Overall, 450 studies and 80,410 patients were included after the screening of 7,608 citations and 1,014 full-text articles. Significantly fewer patients experienced nausea with any drug relative to placebo, except for ondansetron plus metoclopramide in a NMA including 195 RCTs and 24,230 patients. Significantly fewer patients experienced vomiting with any drug relative to placebo except for palonosetron plus dexamethasone in NMA including 238 RCTs and 12,781 patients. All agents resulted in significantly fewer patients with postoperative nausea and vomiting versus placebo in a NMA including 125 RCTs and 16,667 patients. CONCLUSIONS: Granisetron plus dexamethasone was often the most effective antiemetic, with the number needed to treat ranging from two to nine.

Comparative safety of serotonin (5-HT3) receptor antagonists in patients undergoing surgery: a systematic review and network meta-analysis.

BACKGROUND: Serotonin (5-HT3) receptor antagonists are commonly used to decrease nausea and vomiting for surgery patients, but these agents may be harmful. We conducted a systematic review on the comparative safety of 5-HT3 receptor antagonists. METHODS: Searches were done in MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials to identify studies comparing 5-HT3 receptor antagonists with each other, placebo, and/or other antiemetic agents for patients undergoing surgical procedures. Screening search results, data abstraction, and risk of bias assessment were conducted by two reviewers independently. Random-effects pairwise meta-analysis and network meta-analysis (NMA) were conducted. PROSPERO registry number: CRD42013003564. RESULTS: Overall, 120 studies and 27,787 patients were included after screening of 7,608 citations and 1,014 full-text articles. Significantly more patients receiving granisetron plus dexamethasone experienced an arrhythmia relative to placebo (odds ratio (OR) 2.96, 95 % confidence interval (CI) 1.11-7.94), ondansetron (OR 3.23, 95 % CI 1.17-8.95), dolasetron (OR 4.37, 95 % CI 1.51-12.62), tropisetron (OR 3.27, 95 % CI 1.02-10.43), and ondansetron plus dexamethasone (OR 5.75, 95 % CI 1.71-19.34) in a NMA including 31 randomized clinical trials (RCTs) and 6,623 patients of all ages. No statistically significant differences in delirium frequency were observed across all treatment comparisons in a NMA including 18 RCTs and 3,652 patients. CONCLUSION: Granisetron plus dexamethasone increases the risk of arrhythmia.

Trivalent and quadrivalent seasonal influenza vaccine in adults aged 60 and older: a systematic review and network meta-analysis.

OBJECTIVES: To compare the efficacy of influenza vaccines of any valency for adults 60 years and older. DESIGN AND SETTING: Systematic review with network meta-analysis (NMA) of randomised controlled trials (RCTs). MEDLINE, EMBASE, JBI Evidence-Based Practice (EBP) Database, PsycINFO, and Cochrane Evidence -Based Medicine database were searched from inception to 20 June 20, 2022. Two reviewers screened, abstracted, and appraised articles (Cochrane Risk of Bias (ROB) 2.0 tool) independently. We assessed certainty of findings using Confidence in Network Meta-Analysis and Grading of Recommendations, Assessment, Development and Evaluations approaches. We performed random-effects meta-analysis and network meta-analysis (NMA), and estimated odds ratios (ORs) for dichotomous outcomes and incidence rate ratios (IRRs) for count outcomes along with their corresponding 95% confidence intervals (CIs) and prediction intervals. PARTICIPANTS: Older adults (≥60 years old) receiving an influenza vaccine licensed in Canada or the USA (vs placebo, no vaccine, or any other licensed vaccine), at any dose. MAIN OUTCOME MEASURES: Laboratory-confirmed influenza (LCI) and influenza-like illness (ILI). Secondary outcomes were the number of vascular adverse events, hospitalisation for acute respiratory infection (ARI) and ILI, inpatient hospitalisation, emergency room (ER) visit for ILI, outpatient visit, and mortality, among others. RESULTS: We included 41 RCTs and 15 companion reports comprising 8 vaccine types and 206 032 participants. Vaccines may prevent LCI compared with placebo, with high-dose trivalent inactivated influenza vaccine (IIV3-HD) (NMA: 9 RCTs, 52 202 participants, OR 0.23, 95% confidence interval (CI) (0.11 to 0.51), low certainty of evidence) and recombinant influenza vaccine (RIV) (OR 0.25, 95%CI (0.08 to 0.73), low certainty of evidence) among the most efficacious vaccines. Standard dose trivalent IIV3 (IIV3-SD) may prevent ILI compared with placebo, but the result was imprecise (meta-analysis: 2 RCTs, 854 participants, OR 0.39, 95%CI (0.15 to 1.02), low certainty of evidence). Any HD was associated with prevention of ILI compared with placebo (NMA: 9 RCTs, 65 658 participants, OR 0.38, 95%CI (0.15 to 0.93)). Adjuvanted quadrivalent IIV (IIV4-Adj) may be associated with the least vascular adverse events, but the results were very uncertain (NMA: eight 8 RCTs, 57 677 participants, IRR 0.18, 95%CI (0.07 to 0.43), very low certainty of evidence). RIV on all-cause mortality may be comparable to placebo (NMA: 20 RCTs, 140 577 participants, OR 1.01, 95%CI (0.23 to 4.49), low certainty of evidence). CONCLUSIONS: This systematic review demonstrated efficacy associated with IIV3-HD and RIV vaccines in protecting older persons against LCI. RIV vaccine may reduce all-cause mortality when compared with other vaccines, but the evidence is uncertain. Differences in efficacy between influenza vaccines remain uncertain with very low to moderate certainty of evidence. PROSPERO REGISTRATION NUMBER: CRD42020177357.

Comparative-effectiveness research of COVID-19 treatment: a rapid scoping review.

OBJECTIVES: The COVID-19 pandemic has stimulated growing research on treatment options. We aim to provide an overview of the characteristics of studies evaluating COVID-19 treatment. DESIGN: Rapid scoping review DATA SOURCES: Medline, Embase and biorxiv/medrxiv from inception to 15 May 2021. SETTING: Hospital and community care. PARTICIPANTS: COVID-19 patients of all ages. INTERVENTIONS: COVID-19 treatment. RESULTS: The literature search identified 616 relevant primary studies of which 188 were randomised controlled trials and 299 relevant evidence syntheses. The studies and evidence syntheses were conducted in 51 and 39 countries, respectively.Most studies enrolled patients admitted to acute care hospitals (84%), included on average 169 participants, with an average age of 60 years, study duration of 28 days, number of effect outcomes of four and number of harm outcomes of one. The most common primary outcome was death (32%).The included studies evaluated 214 treatment options. The most common treatments were tocilizumab (11%), hydroxychloroquine (9%) and convalescent plasma (7%). The most common therapeutic categories were non-steroidal immunosuppressants (18%), steroids (15%) and antivirals (14%). The most common therapeutic categories involving multiple drugs were antimalarials/antibiotics (16%), steroids/non-steroidal immunosuppressants (9%) and antimalarials/antivirals/antivirals (7%). The most common treatments evaluated in systematic reviews were hydroxychloroquine (11%), remdesivir (8%), tocilizumab (7%) and steroids (7%).The evaluated treatment was in favour 50% and 36% of the evaluations, according to the conclusion of the authors of primary studies and evidence syntheses, respectively. CONCLUSIONS: This rapid scoping review characterised a growing body of comparative-effectiveness primary studies and evidence syntheses. The results suggest future studies should focus on children, elderly ≥65 years of age, patients with mild symptoms, outpatient treatment, multimechanism therapies, harms and active comparators. The results also suggest that future living evidence synthesis and network meta-analysis would provide additional information for decision-makers on managing COVID-19.

Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow.

BACKGROUND: Current text mining tools supporting abstract screening in systematic reviews are not widely used, in part because they lack sensitivity and precision. We set out to develop an accessible, semi-automated "workflow" to conduct abstract screening for systematic reviews and other knowledge synthesis methods. METHODS: We adopt widely recommended text-mining and machine-learning methods to (1) process title-abstracts into numerical training data; and (2) train a classification model to predict eligible abstracts. The predicted abstracts are screened by human reviewers for ("true") eligibility, and the newly eligible abstracts are used to identify similar abstracts, using near-neighbor methods, which are also screened. These abstracts, as well as their eligibility results, are used to update the classification model, and the above steps are iterated until no new eligible abstracts are identified. The workflow was implemented in R and evaluated using a systematic review of insulin formulations for type-1 diabetes (14,314 abstracts) and a scoping review of knowledge-synthesis methods (17,200 abstracts). Workflow performance was evaluated against the recommended practice of screening abstracts by 2 reviewers, independently. Standard measures were examined: sensitivity (inclusion of all truly eligible abstracts), specificity (exclusion of all truly ineligible abstracts), precision (inclusion of all truly eligible abstracts among all abstracts screened as eligible), F1-score (harmonic average of sensitivity and precision), and accuracy (correctly predicted eligible or ineligible abstracts). Workload reduction was measured as the hours the workflow saved, given only a subset of abstracts needed human screening. RESULTS: With respect to the systematic and scoping reviews respectively, the workflow attained 88%/89% sensitivity, 99%/99% specificity, 71%/72% precision, an F1-score of 79%/79%, 98%/97% accuracy, 63%/55% workload reduction, with 12%/11% fewer abstracts for full-text retrieval and screening, and 0%/1.5% missed studies in the completed reviews. CONCLUSION: The workflow was a sensitive, precise, and efficient alternative to the recommended practice of screening abstracts with 2 reviewers. All eligible studies were identified in the first case, while 6 studies (1.5%) were missed in the second that would likely not impact the review's conclusions. We have described the workflow in language accessible to reviewers with limited exposure to natural language processing and machine learning, and have made the code available to reviewers.

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies.

OBJECTIVES: To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). STUDY DESIGN AND SETTING: In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR. RESULTS: After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase. CONCLUSION: Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

A tutorial on pilot studies: the what, why and how.

Pilot studies for phase III trials - which are comparative randomized trials designed to provide preliminary evidence on the clinical efficacy of a drug or intervention - are routinely performed in many clinical areas. Also commonly know as "feasibility" or "vanguard" studies, they are designed to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibility of international collaboration or coordination for multicentre trials; to increase clinical experience with the study medication or intervention for the phase III trials. They are the best way to assess feasibility of a large, expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. The objective of this paper is to provide a detailed examination of the key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot study; 2) the relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and misconceptions about pilot studies; 4) the criteria for evaluating the success of a pilot study; 5) frequently asked questions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to report the results of pilot investigations using the CONSORT format.

Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol.

BACKGROUND: A new tool, "risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE)," was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). METHODS: Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC1 statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively. DISCUSSION: The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.

Inter-rater reliability and concurrent validity of ROBINS-I: protocol for a cross-sectional study.

BACKGROUND: The Cochrane Bias Methods Group recently developed the "Risk of Bias (ROB) in Non-randomized Studies of Interventions" (ROBINS-I) tool to assess ROB for non-randomized studies of interventions (NRSI). It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability of the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of ROBINS-I. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). METHODS: Reviewers from four participating centers will appraise the ROB of a sample of NRSI publications using the ROBINS-I tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSI publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC1 statistic. For concurrent validity, reviewers will appraise a sample of NRSI publications using both the New-castle Ottawa Scale (NOS) and ROBINS-I. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure the evaluator burden, we will assess the time taken to apply the ROBINS-I (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4 to manage and analyze study data, respectively. DISCUSSION: The quality of evidence from systematic reviews that include NRS depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of the ROBINS-I tool and how best to use it.

Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden.

OBJECTIVE: To assess the real-world interrater reliability (IRR), interconsensus reliability (ICR), and evaluator burden of the Risk of Bias (RoB) in Nonrandomized Studies (NRS) of Interventions (ROBINS-I), and the ROB Instrument for NRS of Exposures (ROB-NRSE) tools. STUDY DESIGN AND SETTING: A six-center cross-sectional study with seven reviewers (2 reviewer pairs) assessing the RoB using ROBINS-I (n = 44 NRS) or ROB-NRSE (n = 44 NRS). We used Gwet's AC1 statistic to calculate the IRR and ICR. To measure the evaluator burden, we assessed the total time taken to apply the tool and reach a consensus. RESULTS: For ROBINS-I, both IRR and ICR for individual domains ranged from poor to substantial agreement. IRR and ICR on overall RoB were poor. The evaluator burden was 48.45 min (95% CI 45.61 to 51.29). For ROB-NRSE, the IRR and ICR for the majority of domains were poor, while the rest ranged from fair to perfect agreement. IRR and ICR on overall RoB were slight and poor, respectively. The evaluator burden was 36.98 min (95% CI 34.80 to 39.16). CONCLUSIONS: We found both tools to have low reliability, although ROBINS-I was slightly higher. Measures to increase agreement between raters (e.g., detailed training, supportive guidance material) may improve reliability and decrease evaluator burden.

Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.

OBJECTIVES: The aim of the article was to identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews. STUDY DESIGN AND SETTING: A systematic review was conducted, searching MEDLINE, EMBASE, and the Cochrane Library from inception to September 1, 2016. Quality appraisal of included studies was undertaken using a modified Quality Assessment of Diagnostic Accuracy Studies 2, and key results on accuracy, reliability, efficiency of a methodology, or impact on results and conclusions were extracted. RESULTS: After screening 5,600 titles and abstracts and 245 full-text articles, 37 studies were included. For screening, studies supported the involvement of two independent experienced reviewers and the use of Google Translate when screening non-English articles. For data abstraction, studies supported involvement of experienced reviewers (especially for continuous outcomes) and two independent reviewers, use of dual monitors, graphical data extraction software, and contacting authors. For quality appraisal, studies supported intensive training, piloting quality assessment tools, providing decision rules for poorly reported studies, contacting authors, and using structured tools if different study designs are included. CONCLUSION: Few studies exist documenting common systematic review practices. Included studies support several systematic review practices. These results provide an updated evidence-base for current knowledge synthesis guidelines and methods requiring further research.

A retrospective comparison of systematic reviews with same-topic rapid reviews.

OBJECTIVE: To compare rapid reviews (RRs) to same-topic systematic reviews (SRs) for methods, studies included, and conclusions. STUDY DESIGN AND SETTING: A retrospective comparison of studies comparing RRs and SRs by searching four scoping reviews published between 2007 and 2016. Reports were included if literature searches were conducted within 24 months of each other and had common research questions. Reviews were compared for duration, studies included, population, intervention, comparisons, outcomes, study designs, quality, methods, and conclusions. RESULTS: Six studies containing 16 review pairs were included, covering nine topics. Overall, RRs used abbreviated methods more often: no search of grey literature, employing one reviewer to screen studies, engaging fewer experts, including fewer studies, and providing shorter reports, with poorer reporting quality and faster completion. Reviews reported similar conclusions, with two exceptions: one SR did not include a key study; separately, two RRs failed to highlight an association with early mortality identified by the SR. RRs tended to provide less detail and fewer considerations. CONCLUSION: RRs used several methodological shortcuts compared with SRs on the same topic. It was challenging to discern methodological differences because of retrospective assessment and substantial nonreporting, particularly for RRs.

Improving the conduct of systematic reviews: a process mining perspective.

OBJECTIVES: To illustrate the use of process mining concepts, techniques, and tools to improve the systematic review process. STUDY DESIGN AND SETTING: We simulated review activities and step-specific methods in the process for systematic reviews conducted by one research team over 1 year to generate an event log of activities, with start/end dates, reviewer assignment by expertise, and person-hours worked. Process mining techniques were applied to the event log to "discover" process models, which allowed visual display, animation, or replay of the simulated review activities. Summary statistics were calculated for person-time and timelines. We also analyzed the social networks of team interactions. RESULTS: The 12 simulated reviews included an average of 3,831 titles/abstracts (range: 1,565-6,368) and 20 studies (6-42). The average review completion time was 463 days (range: 289-629) (881 person-hours [range: 243-1,752]). The average person-hours per activity were study selection 26%, data collection 24%, report preparation 23%, and meta-analysis 17%. Social network analyses showed the organizational interaction of team members, including how they worked together to complete review tasks and to hand over tasks upon completion. CONCLUSION: Event log and process mining can be valuable tools for research teams interested in improving and modernizing the systematic review process.

Strategies to Prevent or Reduce Gender Bias in Peer Review of Research Grants: A Rapid Scoping Review.

OBJECTIVE: To review the literature on strategies implemented or identified to prevent or reduce gender bias in peer review of research grants. METHODS: Studies of any type of qualitative or quantitative design examining interventions to reduce or prevent gender bias during the peer review of health-related research grants were included. Electronic databases including MEDLINE, EMBASE, Education Resources Information Center (ERIC), PsycINFO, Joanna Briggs, the Cochrane Library, Evidence Based Medicine (EBM) Reviews, and the Campbell Library were searched from 2005 to April 2016. A search for grey (i.e., difficult to locate or unpublished) literature was conducted and experts in the field were consulted to identify additional potentially relevant articles. Two individuals screened titles and abstracts, full-text articles, and abstracted data with discrepancies resolved by a third person consistently. RESULTS: After screening 5524 citations and 170 full-text articles, one article evaluating gender-blinding of grant applications using an uncontrolled before-after study design was included. In this study, 891 applications for long-term fellowships in 2006 were included and 47% of the applicants were women. These were scored by 13 peer reviewers (38% were women). The intervention included eliminating references to gender from the applications, letters of recommendations, and interview reports that were sent to the committee members for evaluation. The proportion of successful applications led by women did not change with gender-blinding, although the number of successful applications that were led by men increased slightly. CONCLUSIONS: There is limited research on interventions to mitigate gender bias in the peer review of grants. Only one study was identified and no difference in the proportion of women who were successful in receiving grant funding was observed. Our results suggest that interventions to prevent gender bias should be adapted and tested in the context of grant peer review to determine if they will have an impact.