Transparency, openness, and reproducible research practices are frequently underused in health economic evaluations.

OBJECTIVES: To investigate the extent to which articles of economic evaluations of healthcare interventions indexed in MEDLINE incorporate research practices that promote transparency, openness, and reproducibility. STUDY DESIGN AND SETTING: We evaluated a random sample of health economic evaluations indexed in MEDLINE during 2019. We included articles written in English reporting an incremental cost-effectiveness ratio in terms of costs per life years gained, quality-adjusted life years, and/or disability-adjusted life years. Reproducible research practices, openness, and transparency in each article were extracted in duplicate. We explored whether reproducible research practices were associated with self-report use of a guideline. RESULTS: We included 200 studies published in 147 journals. Almost half were published as open access articles (n = 93; 47%). Most studies (n = 150; 75%) were model-based economic evaluations. In 109 (55%) studies, authors self-reported use a guideline (e.g., for study conduct or reporting). Few studies (n = 31; 16%) reported working from a protocol. In 112 (56%) studies, authors reported the data needed to recreate the incremental cost-effectiveness ratio for the base case analysis. This percentage was higher in studies using a guideline than studies not using a guideline (72/109 [66%] with guideline vs. 40/91 [44%] without guideline; risk ratio 1.50, 95% confidence interval 1.15-1.97). Only 10 (5%) studies mentioned access to raw data and analytic code for reanalyses. CONCLUSION: Transparency, openness, and reproducible research practices are frequently underused in health economic evaluations. This study provides baseline data to compare future progress in the field.

A Decade of Efforts to Add Value to Child Health Research Practices.

OBJECTIVE: To identify practices that add value to improve the design, conduct, and reporting of child health research and reduce research waste. STUDY DESIGN: In order to categorize the contributions of members of Standards for Research (StaR) in Child Health network, we developed a novel Child Health Improving Research Practices (CHIRP) framework comprised of 5 domains meant to counteract avoidable child health research waste and improve quality: 1) address research questions relevant to children, their families, clinicians, and researchers; 2) apply appropriate research design, conduct and analysis; 3) ensure efficient research oversight and regulation; 4) Provide accessible research protocols and reports; and 5) develop unbiased and usable research reports, including 17 responsible research practice recommendations. All child health research relevant publications by the 48 original StaR standards' authors over the last decade were identified, and main topic areas were categorized using this framework. RESULTS: A total of 247 publications were included in the final sample: 100 publications (41%) in domain 1 (3 recommendations), 77 publications (31%) in domain 2 (3), 35 publications (14%) in domain 3 (4), 20 publications (8%) in domain 4 (4), and 15 publications (6%) in domain 5 (3). We identified readily implementable "responsible" research practices to counter child health research waste and improve quality, especially in the areas of patients and families' engagement throughout the research process, developing Core Outcome Sets, and addressing ethics and regulatory oversight issues. CONCLUSION: While most of the practices are readily implementable, increased awareness of methodological issues and wider guideline uptake is needed to improve child health research. The CHIRP Framework can be used to guide responsible research practices that add value to child health research.

Health equity considerations in guideline development: a rapid scoping review.

BACKGROUND: Systematic guidance for considering health equity in guidelines is lacking. This scoping review aims to synthesize current best practices for integrating health equity into guideline development and the benefits or drawbacks of these practices. METHODS: We searched Ovid MEDLINE ALL and Embase Classic+Embase on the Ovid platform, CINAHL on EBSCO, and Web of Science (Core Collection) from 2010 to 2022. We searched grey literature from 2015 to 2022, using the Canadian Agency for Drugs and Technologies in Health Grey Matters checklist and searches of potentially relevant websites. Articles were screened independently by 1 reviewer. Proposed best practices, advantages and disadvantages, and tools were extracted independently by 1 reviewer and qualitatively synthesized based on the relevant steps of a comprehensive checklist covering the stages of guideline development. RESULTS: We included 26 articles that proposed best practices for incorporating health equity within the guideline development process. These practices were organized under different stages of the development process, including guideline planning, evidence review, guideline development and dissemination. Included studies provided best practices from guideline producers, articles discussing health equity in current guidelines, articles addressing strategies to increase equity in the guideline implementation process, and literature reviews of promising health equity practices. INTERPRETATION: Our scoping review identified best practices to incorporate health equity considerations at each phase of guideline development. Identified practices may be used to inform equity-promoting strategies with the guideline development process; however, guideline producers should carefully consider the advantages and disadvantages of best practices when integrating health equity.

Interventions to address potentially inappropriate prescriptions and over-the-counter medication use among adults 65 years and older in primary care settings: protocol for a systematic review.

PURPOSE: To inform recommendations by the Canadian Task Force on Preventive Health Care on potentially inappropriate prescribing and over-the-counter (OTC) medication use among adults aged 65 years and older in primary care settings. This protocol outlines the planned scope and methods for a systematic review of the benefits and harms and acceptability of interventions to reduce potentially inappropriate prescriptions and OTC medication use. METHODS: De novo systematic reviews will be conducted to synthesize the available evidence on (a) the benefits and harms of interventions to reduce potentially inappropriate prescriptions and OTC medications compared to no intervention, usual care, or non- or minimally active intervention among adults aged 65 years and older and (b) the acceptability of these interventions or attributes among patients. Outcomes of interest for the benefits and harms review are all-cause mortality, hospitalization, non-serious adverse drug reactions, quality of life, emergency department visits, injurious falls, medical visits, and the number of medications (and number of pills). Outcomes for the acceptability review are the preference for and relative importance of different interventions or their attributes. For the benefits and harms review, we will search MEDLINE, Embase, and Cochrane Central Register of Controlled Trials for randomized controlled trials. For the acceptability review, we will search MEDLINE, Embase, PsycInfo, Cochrane Central Register of Controlled Trials, and the NHS Economic Evaluation Database for experimental and observational studies with a comparator. Websites of relevant organizations, other grey literature sources, and reference lists of included studies and reviews will be searched. Title and abstract screening will be completed by two independent reviewers using the liberal accelerated approach. Full-text review, data extraction, risk of bias assessments, and GRADE (Grading of Recommendations Assessment, Development and Evaluation) will be completed independently by two reviewers, with any disagreements resolved by consensus or by consulting with a third reviewer. The GRADE approach will be used to assess the certainty of the evidence for outcomes. DISCUSSION: The results of this systematic review will be used by the Canadian Task Force on Preventive Health Care to inform their recommendation on potentially inappropriate prescribing and OTC medication use among adults aged 65 years and older. SYSTEMATIC REVIEW REGISTRATION: PROSPERO (KQ1: CRD42022302313; KQ2: CRD42022302324); Open Science Framework ( https://osf.io/urj4b/ ).

Data-sharing and re-analysis for main studies assessed by the European Medicines Agency-a cross-sectional study on European Public Assessment Reports.

BACKGROUND: Transparency and reproducibility are expected to be normative practices in clinical trials used for decision-making on marketing authorisations for new medicines. This registered report introduces a cross-sectional study aiming to assess inferential reproducibility for main trials assessed by the European Medicines Agency. METHODS: Two researchers independently identified all studies on new medicines, biosimilars and orphan medicines given approval by the European Commission between January 2017 and December 2019, categorised as 'main studies' in the European Public Assessment Reports (EPARs). Sixty-two of these studies were randomly sampled. One researcher retrieved the individual patient data (IPD) for these studies and prepared a dossier for each study, containing the IPD, the protocol and information on the conduct of the study. A second researcher who had no access to study reports used the dossier to run an independent re-analysis of each trial. All results of these re-analyses were reported in terms of each study's conclusions, p-values, effect sizes and changes from the initial protocol. A team of two researchers not involved in the re-analysis compared results of the re-analyses with published results of the trial. RESULTS: Two hundred ninety-two main studies in 173 EPARs were identified. Among the 62 studies randomly sampled, we received IPD for 10 trials. The median number of days between data request and data receipt was 253 [interquartile range 182-469]. For these ten trials, we identified 23 distinct primary outcomes for which the conclusions were reproduced in all re-analyses. Therefore, 10/62 trials (16% [95% confidence interval 8% to 28%]) were reproduced, as the 52 studies without available data were considered non-reproducible. There was no change from the original study protocol regarding the primary outcome in any of these ten studies. Spin was observed in the report of one study. CONCLUSIONS: Despite their results supporting decisions that affect millions of people's health across the European Union, most main studies used in EPARs lack transparency and their results are not reproducible for external researchers. Re-analyses of the few trials with available data showed very good inferential reproducibility. TRIAL REGISTRATION: https://osf.io/mcw3t/.

What difference might retractions make? An estimate of the potential epistemic cost of retractions on meta-analyses.

The extent to which a retraction might require revising previous scientific estimates and beliefs - which we define as the epistemic cost - is unknown. We collected a sample of 229 meta-analyses published between 2013 and 2016 that had cited a retracted study, assessed whether this study was included in the meta-analytic estimate and, if so, re-calculated the summary effect size without it. The majority (68% of N = 229) of retractions had occurred at least one year prior to the publication of the citing meta-analysis. In 53% of these avoidable citations, the retracted study was cited as a candidate for inclusion, and only in 34% of these meta-analyses (13% of total) the study was explicitly excluded because it had been retracted. Meta-analyses that included retracted studies were published in journals with significantly lower impact factor. Summary estimates without the retracted study were lower than the original if the retraction was due to issues with data or results and higher otherwise, but the effect was small. We conclude that meta-analyses have a problematically high probability of citing retracted articles and of including them in their pooled summaries, but the overall epistemic cost is contained.

The impact of gender on researchers' assessment: A randomized controlled trial.

OBJECTIVES: This randomized controlled trial aimed to test whether women or men would be preferred with identical curriculum vitae (CV); and the impact of the career stage in the evaluators' choice. STUDY DESIGN AND SETTING: A simulated post-doctoral process was carried forward to be assessed for judgment. Level 1 and 2 Brazilian fellow researchers in the field of Dentistry were invited to act as external reviewers in a post-doctoral process and were randomly assigned to receive a CV from a woman or a man. They were required to rate the CV from 0 to 10 in scientific contribution, leadership potential, ability to work in groups, and international experience. RESULTS: For all categories of CVs evaluated, CVs from men received higher scores compared to the CVs from women. Robust variance Poisson regressions demonstrated that men were more likely to receive higher scores in all categories, despite applicants' career stage. For example, CVs from men were nearly three quarters more likely to be seen as having leadership potential than equivalent CVs from women. CONCLUSION: Gender bias is powerfully prevalent in academia in the dentistry field, despite researchers' career stage. Actions like implicit bias training must be urgently implemented to avoid (or at least decrease) that more women are harmed.

Incorporating equity, diversity, and inclusiveness into the Hong Kong Principles.

In this response to Labib and Evans, authors of the Hong Kong Principles look forward to collaborating with those from the broad research integrity community to ensure that issues of equity, diversity and inclusion will become part of the ecosystem of research integrity.

The impact of gender on scientific writing: An observational study of grant proposals.

OBJECTIVES: This study aimed to determine whether there are differences in the language used in grant applications submitted to a Southern Brazil Research Support Foundation (FAPERGS) according to the gender, career stage, and the number of publications of applicants. STUDY DESIGN AND SETTING: This observational study also evaluated the relationship between gender, career stage, curriculum, and writing characteristics. Summaries of all research proposals in the biomedical field of FAPERGS during the years of 2013 and 2014 were evaluated according to six language patterns (Positive emotions, Negative emotions, Analytic thinking, Clout, Authenticity, and Emotional tone) defined by the LIWC software. Applicant's gender, career stage, and the number of publications were also collected. RESULTS: Three hundred and forty-four (344) grant proposals met the inclusion criteria and were included in the analysis. No statistical differences were observed in the language pattern used by different gender applicants. In the language used by successful and unsuccessful applicants, we only found a small difference for clout (score 54.5 for not funded and 56.5 for funded grants). However, the principal investigators of successful applications had a significantly higher number of papers published (mean number of papers: 104 versus 58.5). CONCLUSIONS: Gender bias appears to be a more complex problem than just the type of language used; the way society is organized causes several gender biases that may be reflected throughout the women's career.

Benefits and obstacles to cell therapy in neonates: The INCuBAToR (Innovative Neonatal Cellular Therapy for Bronchopulmonary Dysplasia: Accelerating Translation of Research).

Cell-based therapies hold promise to substantially curb complications from extreme preterm birth, the main cause of death in children below the age of 5 years. Exciting preclinical studies in experimental neonatal lung injury have provided the impetus for the initiation of early phase clinical trials in extreme preterm infants at risk of developing bronchopulmonary dysplasia. Clinical translation of promising therapies, however, is slow and often fails. In the adult population, results of clinical trials so far have not matched the enticing preclinical data. The neonatal field has experienced many hard-earned lessons with the implementation of oxygen therapy or postnatal steroids. Here we briefly summarize the preclinical data that have permitted the initiation of early phase clinical trials of cell-based therapies in extreme preterm infants and describe the INCuBAToR concept (Innovative Neonatal Cellular Therapy for Bronchopulmonary Dysplasia: Accelerating Translation of Research), an evidence-based approach to mitigate the risk of translating advanced therapies into this vulnerable patient population. The INCuBAToR addresses several of the shortcomings at the preclinical and the clinical stage that usually contribute to the failure of clinical translation through (a) systematic reviews of preclinical and clinical studies, (b) integrated knowledge transfer through engaging important stakeholders early on, (c) early economic evaluation to determine if a novel therapy is viable, and (d) retrospective and prospective studies to define and test ideal eligibility criteria to optimize clinical trial design. The INCuBAToR concept can be applied to any novel therapy in order to enhance the likelihood of success of clinical translation in a timely, transparent, rigorous, and evidence-based fashion.

Clinical relevance assessment of animal preclinical research (RAA) tool: development and explanation.

BACKGROUND: Only a small proportion of preclinical research (research performed in animal models prior to clinical trials in humans) translates into clinical benefit in humans. Possible reasons for the lack of translation of the results observed in preclinical research into human clinical benefit include the design, conduct, and reporting of preclinical studies. There is currently no formal domain-based assessment of the clinical relevance of preclinical research. To address this issue, we have developed a tool for the assessment of the clinical relevance of preclinical studies, with the intention of assessing the likelihood that therapeutic preclinical findings can be translated into improvement in the management of human diseases. METHODS: We searched the EQUATOR network for guidelines that describe the design, conduct, and reporting of preclinical research. We searched the references of these guidelines to identify further relevant publications and developed a set of domains and signalling questions. We then conducted a modified Delphi-consensus to refine and develop the tool. The Delphi panel members included specialists in evidence-based (preclinical) medicine specialists, methodologists, preclinical animal researchers, a veterinarian, and clinical researchers. A total of 20 Delphi-panel members completed the first round and 17 members from five countries completed all three rounds. RESULTS: This tool has eight domains (construct validity, external validity, risk of bias, experimental design and data analysis plan, reproducibility and replicability of methods and results in the same model, research integrity, and research transparency) and a total of 28 signalling questions and provides a framework for researchers, journal editors, grant funders, and regulatory authorities to assess the potential clinical relevance of preclinical animal research. CONCLUSION: We have developed a tool to assess the clinical relevance of preclinical studies. This tool is currently being piloted.

Funders' data-sharing policies in therapeutic research: A survey of commercial and non-commercial funders.

BACKGROUND: Funders are key players in supporting randomized controlled trial (RCT) data-sharing. This research aimed to describe commercial and non-commercial funders' data-sharing policies and to assess the compliance of funded RCTs with the existing data-sharing policies. METHODS AND FINDINGS: Funders of clinical research having funded at least one RCT in the years 2016 to 2018 were surveyed. All 78 eligible non-commercial funders retrieved from the Sherpa/Juliet Initiative website and a random sample of 100 commercial funders selected from pharmaceutical association member lists (LEEM, IFPMA, EFPIA) and the top 100 pharmaceutical companies in terms of drug sales were included. Thirty (out of 78; 38%) non-commercial funders had a data-sharing policy with eighteen (out of 30, 60%) making data-sharing mandatory and twelve (40%) encouraging data-sharing. Forty-one (out of 100; 41%) of commercial funders had a data-sharing policy. Among funders with a data-sharing policy, a survey of two random samples of 100 RCTs registered on Clinicaltrial.gov, data-sharing statements were present for seventy-seven (77%, 95% IC [67%-84%]) and eighty-one (81% [72% - 88%]) of RCTs funded by non-commercial and commercial funders respectively. Intention to share data was expressed in 12% [7%-20%] and 59% [49%- 69%] of RCTs funded by non-commercial and commercial funders respectively. CONCLUSIONS: This survey identified suboptimal performances of funders in setting up data-sharing policies. For those with a data-sharing policy, the implementation of the policy in study registration was limited for commercial funders and of concern for non-commercial funders. The limitations of the present study include its cross-sectional nature, since data-sharing policies are continuously changing. We call for a standardization of policies with a strong evaluation component to make sure that, when in place, these policies are effective.

Reporting of health equity considerations in cluster and individually randomized trials.

BACKGROUND: The randomized controlled trial (RCT) is considered the gold standard study design to inform decisions about the effectiveness of interventions. However, a common limitation is inadequate reporting of the applicability of the intervention and trial results for people who are "socially disadvantaged" and this can affect policy-makers' decisions. We previously developed a framework for identifying health-equity-relevant trials, along with a reporting guideline for transparent reporting. In this study, we provide a descriptive assessment of health-equity considerations in 200 randomly sampled equity-relevant trials. METHODS: We developed a search strategy to identify health-equity-relevant trials published between 2013 and 2015. We randomly sorted the 4316 records identified by the search and screened studies until 100 individually randomized (RCTs) and 100 cluster randomized controlled trials (CRTs) were identified. We developed and pilot-tested a data extraction form based on our initial work, to inform the development of our reporting guideline for equity-relevant randomized trials. RESULTS: In total, 39 trials (20%) were conducted in a low- and middle-income country and 157 trials (79%) in a high-income country focused on socially disadvantaged populations (78% CRTs, 79% RCTs). Seventy-four trials (37%) reported a subgroup analysis across a population characteristic associated with disadvantage (25% CRT, 49% RCTs), with 19% of included studies reporting subgroup analyses across sex, 9% across race/ethnicity/culture, and 4% across socioeconomic status. No subgroup analyses were reported for place of residence, occupation, religion, education, or social capital. One hundred and forty-one trials (71%) discussed the applicability of their results to one or more socially disadvantaged populations (68% of CRT, 73% of RCT). DISCUSSION: In this set of trials, selected for their relevance to health equity, data that were disaggregated for socially disadvantaged populations were rarely reported. We found that even when the data are available, opportunities to analyze health-equity considerations are frequently missed. The recently published equity extension of the Consolidated Reporting Standards for Randomized Trials (CONSORT-Equity) may help improve delineation of hypotheses related to socially disadvantaged populations, and transparency and completeness of reporting of health-equity considerations in RCTs. This study can serve as a baseline assessment of the reporting of equity considerations.

Reproducible research practices, openness and transparency in health economic evaluations: study protocol for a cross-sectional comparative analysis.

INTRODUCTION: There has been a growing awareness of the need for rigorously and transparent reported health research, to ensure the reproducibility of studies by future researchers. Health economic evaluations, the comparative analysis of alternative interventions in terms of their costs and consequences, have been promoted as an important tool to inform decision-making. The objective of this study will be to investigate the extent to which articles of economic evaluations of healthcare interventions indexed in MEDLINE incorporate research practices that promote transparency, openness and reproducibility. METHODS AND ANALYSIS: This is the study protocol for a cross-sectional comparative analysis. We registered the study protocol within the Open Science Framework (osf.io/gzaxr). We will evaluate a random sample of 600 cost-effectiveness analysis publications, a specific form of health economic evaluations, indexed in MEDLINE during 2012 (n=200), 2019 (n=200) and 2022 (n=200). We will include published papers written in English reporting an incremental cost-effectiveness ratio in terms of costs per life years gained, quality-adjusted life years and/or disability-adjusted life years. Screening and selection of articles will be conducted by at least two researchers. Reproducible research practices, openness and transparency in each article will be extracted using a standardised data extraction form by multiple researchers, with a 33% random sample (n=200) extracted in duplicate. Information on general, methodological and reproducibility items will be reported, stratified by year, citation of the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement and journal. Risk ratios with 95% CIs will be calculated to represent changes in reporting between 2012-2019 and 2019-2022. ETHICS AND DISSEMINATION: Due to the nature of the proposed study, no ethical approval will be required. All data will be deposited in a cross-disciplinary public repository. It is anticipated the study findings could be relevant to a variety of audiences. Study findings will be disseminated at scientific conferences and published in peer-reviewed journals.

Taking an integrated knowledge translation approach in research to develop the CONSORT-Equity 2017 reporting guideline: an observational study.

OBJECTIVE: We describe the use of an integrated knowledge translation (KT) approach in the development of the CONsolidated Standards Of Reporting Trials extension for equity ('CONSORT-Equity 2017'), and advisory board-research team members' ('the team') perceptions of the integrated KT process. DESIGN: This is an observational study to describe team processes and experience with a structured integrated KT approach to develop CONSORT-Equity 2017. Participant observation to describe team processes and a survey were used with the 38 team members. SETTING: Use of the CONSORT health research reporting guideline contributes to an evidence base for health systems decision-making, and CONSORT-Equity 2017 may improve reporting about health equity-relevant evidence. An integrated KT research approach engages knowledge users (those for whom the research is meant to be useful) with researchers to co-develop research evidence and is more likely to produce findings that are applied in practice or policy. PARTICIPANTS: Researchers adopted an integrated KT approach and invited knowledge users to form a team. RESULTS: An integrated KT approach was used in the development of CONSORT-Equity 2017 and structured replicable steps. The process for co-developing the reporting guideline involved two stages: (1) establishing guiding features for co-development and (2) research actions that supported the co-development of the reporting guideline. Stage 1 consisted of four steps: finding common ground, forming an advisory board, committing to ethical guidance and clarifying theoretical research assumptions. Bound by the stage 1 guiding features of an integrated KT approach, stage 2 consisted of five steps during which studies for consensus-based reporting guidelines were conducted. Of 38 team members, 25 (67.5%) completed a survey about their perceptions of the integrated KT approach. CONCLUSIONS: An integrated KT approach can be used to engage a team to co-develop reporting guidelines. Further study is needed to understand the use of an integrated KT approach in the development of reporting guidelines.