Assessing transferability in systematic reviews of health economic evaluations - a review of methodological guidance.

OBJECTIVE: For assessing cost-effectiveness, Health Technology Assessment (HTA) organisations may use primary economic evaluations (P-HEs) or Systematic Reviews of Health Economic evaluations (SR-HEs). A prerequisite for meaningful results of SR-HEs is that the results from existing P-HEs are transferable to the decision context (e.g, HTA jurisdiction). A particularly pertinent issue is the high variability of costs and resource needs across jurisdictions. Our objective was to review the methods documents of HTA organisations and compare their recommendations on considering transferability in SR-HE. METHODS: We systematically hand searched the webpages of 158 HTA organisations for relevant methods documents from 8th January to 31st March 2019. Two independent reviewers performed searches and selected documents according to pre-defined criteria. One reviewer extracted data in standardised and piloted tables and a second reviewer checked them for accuracy. We synthesised data using tabulations and in a narrative way. RESULTS: We identified 155 potentially relevant documents from 63 HTA organisations. Of these, 7 were included in the synthesis. The included organisations have different aims when preparing a SR-HE (e.g. to determine the need for conducting their own P-HE). The recommendations vary regarding the underlying terminology (e.g. transferability/generalisability), the assessment approaches (e.g. structure), the assessment criteria and the integration in the review process. CONCLUSION: Only few HTA organisations address the assessment of transferability in their methodological recommendations for SR-HEs. Transferability considerations are related to different purposes. The assessment concepts and criteria are heterogeneous. Developing standards to consider transferability in SR-HEs is desirable.

Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews.

BACKGROUND: Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews. METHODS: We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively. RESULTS: Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%). CONCLUSIONS: Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.

Development, testing and use of data extraction forms in systematic reviews: a review of methodological guidance.

BACKGROUND: Data extraction forms link systematic reviews with primary research and provide the foundation for appraising, analysing, summarising and interpreting a body of evidence. This makes their development, pilot testing and use a crucial part of the systematic reviews process. Several studies have shown that data extraction errors are frequent in systematic reviews, especially regarding outcome data. METHODS: We reviewed guidance on the development and pilot testing of data extraction forms and the data extraction process. We reviewed four types of sources: 1) methodological handbooks of systematic review organisations (SRO); 2) textbooks on conducting systematic reviews; 3) method documents from health technology assessment (HTA) agencies and 4) journal articles. HTA documents were retrieved in February 2019 and database searches conducted in December 2019. One author extracted the recommendations and a second author checked them for accuracy. Results are presented descriptively. RESULTS: Our analysis includes recommendations from 25 documents: 4 SRO handbooks, 11 textbooks, 5 HTA method documents and 5 journal articles. Across these sources the most common recommendations on form development are to use customized or adapted standardised extraction forms (14/25); provide detailed instructions on their use (10/25); ensure clear and consistent coding and response options (9/25); plan in advance which data are needed (9/25); obtain additional data if required (8/25); and link multiple reports of the same study (8/25). The most frequent recommendations on piloting extractions forms are that forms should be piloted on a sample of studies (18/25); and that data extractors should be trained in the use of the forms (7/25). The most frequent recommendations on data extraction are that extraction should be conducted by at least two people (17/25); that independent parallel extraction should be used (11/25); and that procedures to resolve disagreements between data extractors should be in place (14/25). CONCLUSIONS: Overall, our results suggest a lack of comprehensiveness of recommendations. This may be particularly problematic for less experienced reviewers. Limitations of our method are the scoping nature of the review and that we did not analyse internal documents of health technology agencies.

Words or numbers? Communicating risk of adverse effects in written consumer health information: a systematic review and meta-analysis.

BACKGROUND: Various types of framing can influence risk perceptions, which may have an impact on treatment decisions and adherence. One way of framing is the use of verbal terms in communicating the probabilities of treatment effects. We systematically reviewed the comparative effects of words versus numbers in communicating the probability of adverse effects to consumers in written health information. METHODS: Nine electronic databases were searched up to November 2012. Teams of two reviewers independently assessed studies. INCLUSION CRITERIA: randomised controlled trials; verbal versus numerical presentation; context: written consumer health information. RESULTS: Ten trials were included. Participants perceived probabilities presented in verbal terms as higher than in numeric terms: commonly used verbal descriptors systematically led to an overestimation of the absolute risk of adverse effects (Range of means: 3% - 54%). Numbers also led to an overestimation of probabilities, but the overestimation was smaller (2% - 20%). The difference in means ranged from 3.8% to 45.9%, with all but one comparison showing significant results. Use of numbers increased satisfaction with the information (MD: 0.48 [CI: 0.32 to 0.63], p < 0.00001, I2 = 0%) and likelihood of medication use (MD for very common side effects: 1.45 [CI: 0.78 to 2.11], p = 0.0001, I2 = 68%; MD for common side effects: 0.90 [CI: 0.61 to 1.19], p < 0.00001, I2 = 1%; MD for rare side effects: 0.39 [0.02 to 0.76], p = 0.04, I2 = not applicable). Outcomes were measured on a 6-point Likert scale, suggesting small to moderate effects. CONCLUSIONS: Verbal descriptors including "common", "uncommon" and "rare" lead to an overestimation of the probability of adverse effects compared to numerical information, if used as previously suggested by the European Commission. Numbers result in more accurate estimates and increase satisfaction and likelihood of medication use. Our review suggests that providers of consumer health information should quantify treatment effects numerically. Future research should focus on the impact of personal and contextual factors, use representative samples or be conducted in real life settings, measure behavioral outcomes and address whether benefit information can be described verbally.

[Advancing standards for the development of evidence-based health information for consumers: Needs and priorities among members of the Network for Evidence-based Medicine]. / Inhaltliche Weiterentwicklung von Standards zur Erstellung evidenzbasierter Gesundheitsinformationen: Eine Bedarfserhebung und Priorisierung im EbM-Netzwerk.

BACKGROUND: The Good Practice Guidelines for health information, Good Health Information Austria and Guideline Evidence-based Health Information are established resources for developing evidence-based health information in the German-speaking regions. The aim of this project was to capture challenges in applying these standards in practice, identify gaps and development needs and gain insights for their further development. METHODS: In December 2020, members of the working group for patient information and involvement of the Network for Evidence-based Medicine were invited to share their experiences and needs in applying German standards for evidence-based health information through an online survey focussing on open questions (part 1: needs assessment). The feedback was analysed using qualitative content analysis and presented in a workshop at the EbM Congress 2021 with the goal of specifying the feedback from the needs assessment and discussing ideas for the further development of the standards (part 2: specification). In the final step, a second survey was conducted in February 2023 to prioritize the identified topics by the working group members (part 3: prioritization). The results were analysed descriptively. RESULTS: Among the 41 participants, only 23% considered the standards to be sufficient, and only 55% found their application in the development of information to be easy or rather easy. The needs assessment and workshop (n=46) helped to identify various areas of action. With regard to the application of the standards, the following challenges were identified: lack of user orientation, content gaps, methods and risk communication. Gaps in the standards were identified regarding formats, content, and the connection to healthcare provision. For the advancement of the standards ideas for additional content, stakeholder involvement, and improvement of the usability of the standards were identified. In the prioritization survey, the topic areas, "presenting benefits and harms" and "content beyond treatments (in particular, diagnostics and prognosis)" were considered to be the most important (n=36). DISCUSSION: Among members of the working group for patient information and involvement of the German Network for Evidence-based Medicine, a high demand has been identified for the further development of standards for creating evidence-based health information. In addition to content development, the integration of existing documents and tools should also be considered, including products issued by other institutions. The success of advancing the standards also depends on improving their applicability - for example through an attractive online platform. The results are limited by the sample which only included members of the EbM Network's patient information and participation working group and a limited response rate. CONCLUSIONS: The needs assessment showed that the currently established standards and recommendations for the development of evidence-based health information in the German-speaking regions represent important cornerstones but need to be expanded to answer more practice-oriented questions. The challenges and proposed solutions stated by the participants can help further develop the standards. The prioritization can be used to set priorities for the development of the standards, guide the order of possible work packages and allocate resources.

The effects of modifying elements of written informed consent forms for elective surgical or invasive procedures: A systematic review.

OBJECTIVE: To study the effect of modifying content and design elements within written informed-consent-forms (ICF) for patients undergoing elective surgical or invasive procedures. METHODS: We included (quasi-)randomized trials in which a modified written ICF (e.g. visual aids) was compared to a standard written ICF. We searched PubMed, Web-of-Science and PsycINFO until 08/2021. Risk of Bias was assessed. The complexity of intervention was assessed using the Intervention Complexity Assessment Tool for Systematic Reviews. RESULTS: Eleven trials with 1091 participants were eligible. Effect sizes and levels of evidence varied from trivial to moderate andthere were contradictory findings for some outcomes. Providing patients with more informationin general or specific information on risks and complications mostly increased anxiety. The use of verbal risk presentation decreased anxiety and increased satisfaction.A lower readability level decreased anxiety and improved comprehension and knowledge. CONCLUSION: Our results suggest that providing more information and addressing certain types of risks have differential effects. While more information improved knowledge, it also increased anxiety. We did not find any or only insufficient evidence for many other possible ICF modifications. PRACTICE IMPLICATIONS: When developing ICFs the differential impact of different elements on patient important outcomes should be carefully considered.

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey.

OBJECTIVE: Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs. STUDY DESIGN AND SETTING: We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis. RESULTS: 162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%). CONCLUSION: Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.

Developing an integrated multilevel model of uncertainty in health care: a qualitative systematic review and thematic synthesis.

INTRODUCTION: Uncertainty is an inevitable part of healthcare and a source of confusion and challenge to decision-making. Several taxonomies of uncertainty have been developed, but mainly focus on decisions in clinical settings. Our goal was to develop a holistic model of uncertainty that can be applied to both clinical as well as public and global health scenarios. METHODS: We searched Medline, Embase, CINAHL, Scopus and Google scholar in March 2021 for literature reviews, qualitative studies and case studies related to classifications or models of uncertainty in healthcare. Empirical articles were assessed for study limitations using the Critical Appraisal Skills Programme (CASP) checklist. We synthesised the literature using a thematic analysis and developed a dynamic multilevel model of uncertainty. We sought patient input to assess relatability of the model and applied it to two case examples. RESULTS: We screened 4125 studies and included 15 empirical studies, 13 literature reviews and 5 case studies. We identified 77 codes and organised these into 26 descriptive and 11 analytical themes of uncertainty. The themes identified are global, public health, healthcare system, clinical, ethical, relational, personal, knowledge exchange, epistemic, aleatoric and parameter uncertainty. The themes were included in a model, which captures the macro, meso and microlevels and the inter-relatedness of uncertainty. We successfully piloted the model on one public health example and an environmental topic. The main limitations are that the research input into our model predominantly came from North America and Europe, and that we have not yet tested the model in a real-life setting. CONCLUSION: We developed a model that can comprehensively capture uncertainty in public and global health scenarios. It builds on models that focus solely on clinical settings by including social and political contexts and emphasising the dynamic interplay between different areas of uncertainty.

Communicating Uncertainty From Limitations in Quality of Evidence to the Public in Written Health Information: Protocol for a Web-Based Randomized Controlled Trial.

BACKGROUND: Uncertainty is integral to evidence-informed decision making and is of particular importance for preference-sensitive decisions. Communicating uncertainty to patients and the public has long been identified as a goal in the informed and shared decision-making movement. Despite this, there is little quantitative research on how uncertainty in health information is perceived by readers. OBJECTIVE: The objective of this study is to design an experiment to examine how different degrees of uncertainty (Q1) and different types of uncertainty (Q2) impact patients' perception of treatment effectiveness, the body of evidence, text quality, and hypothetical treatment intention. The experiment also examines whether there is an additive effect when multiple sources of uncertainty are communicated (Q3). METHODS: We developed 8 variations of a research summary set in a hypothetical scenario for a treatment decision in the context of tinnitus. These were modified only in the degree of uncertainty relating to the evidence of the presented treatment. We recruited members of the German public from a Web-based research panel and randomized them to one of 8 variations of the research summary to examine the 3 research questions. The trial was only open to the members of the research panel. The outcomes are perception of the effectiveness of the treatment (primary), certainty in the judgement of treatment effectiveness, perception of the body of evidence relating to the treatment, text quality, and decisional intention (secondary). Outcomes were self-assessed. We aimed to recruit 1500 participants to the trial. The recruitment and data collection was fully automated. Ethical approval was waivered by an ethics committee because of the negligible risk to participants. RESULTS: This protocol is retrospectively published in its original format. In the meantime, the trial was set up and the data collection was completed. Data collection was conducted in May 2018. A total of 1727 eligible panel members were enrolled. CONCLUSIONS: We aim to publish the results in a peer-reviewed journal by the end of 2019. In addition, results will be presented at conferences and disseminated among developers of guidance for the development of evidence-based health information and decision aids. TRIAL REGISTRATION: German Clinical Trials Register DRKS00015911; https://www.drks.de/drks_web/navigate.do? navigationId=trial.HTML&TRIAL_ID=DRKS00015911 (Archived by WebCite at http://www.webcitation.org/77zyZTGzk). INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/13425.

Quality ratings of reviews in overviews: a comparison of reviews with and without dual (co-)authorship.

BACKGROUND: Previous research shows that many authors of Cochrane overviews were also involved in some of the included systematic reviews (SRs). This type of dual (co-)authorship (DCA) may be a conflict of interest and a potential source of bias. Our objectives were to (1) additionally investigate DCA in non-Cochrane overviews; (2) investigate whether there is an association between DCA and quality assessments of SRs in Cochrane and non-Cochrane overviews. METHODS: We selected a sample of Cochrane (n = 20) and non-Cochrane (n = 78) overviews for analysis. We extracted data on the number of reviews affected by DCA and whether quality assessment of included reviews was conducted independently. Differences in mean quality scores between SRs with and without DCA were calculated in each overview. These differences were standardized (using the standardized mean difference (SMD)) and meta-analyzed using a random effects model. RESULTS: Forty out of 78 non-Cochrane overviews (51%) and 18 out of 20 Cochrane overviews (90%) had included at least one SR with DCA. For Cochrane overviews, a median of 5 [interquartile range (IQR) 2.5 to 7] SRs were affected by DCA (median of included reviews 10). For non-Cochrane overviews a median of 1 [IQR 0 to 2] of the included SRs were affected (median of included reviews 14). The meta-analysis showed a SMD of 0.58 (95% confidence interval (CI) 0.27 to 0.90) indicating higher quality scores in reviews with overlapping authors. The test for subgroup differences shows no evidence of a difference between Cochrane (SMD 0.44; 95% CI 0.07 to 0.81) and non-Cochrane overviews (SMD 0.62; 95% CI 0.06 to 1.17). CONCLUSIONS: Many authors of overviews also often have an authorship on one or more of the underlying reviews. Our analysis shows that, on average, authors of overviews give higher quality ratings to SRs in which they were involved themselves than to other SRs. Conflict of interest is one explanation, but there are several others such as reviewer expertise. Independent and blinded reassessments of the reviews would provide more robust evidence on potential bias arising from DCA.

Interventions for reducing self-stigma in people with mental illnesses: a systematic review of randomized controlled trials.

Background: Self-stigma occurs when people with mental illnesses internalize negative stereotypes and prejudices about their condition. It can reduce help-seeking behaviour and treatment adherence. The effectiveness of interventions aimed at reducing self-stigma in people with mental illness is systematically reviewed. Results are discussed in the context of a logic model of the broader social context of mental illness stigma. Methods: Medline, Embase, PsycINFO, ERIC, and CENTRAL were searched for randomized controlled trials in November 2013. Studies were assessed with the Cochrane risk of bias tool. Results: Five trials were eligible for inclusion, four of which provided data for statistical analyses. Four studies had a high risk of bias. The quality of evidence was very low for each set of interventions and outcomes. The interventions studied included various group based anti-stigma interventions and an anti-stigma booklet. The intensity and fidelity of most interventions was high. Two studies were considered to be sufficiently homogeneous to be pooled for the outcome self-stigma. The meta-analysis did not find a statistically significant effect (SMD [95% CI] at 3 months: -0.26 [-0.64, 0.12], I2=0%, n=108). None of the individual studies found sustainable effects on other outcomes, including recovery, help-seeking behaviour and self-stigma. Conclusions: The effectiveness of interventions against self-stigma is uncertain. Previous studies lacked statistical power, used questionable outcome measures and had a high risk of bias. Future studies should be based on robust methods and consider practical implications regarding intervention development (relevance, implementability, and placement in routine services).

Epidemiology and reporting characteristics of overviews of reviews of healthcare interventions published 2012-2016: protocol for a systematic review.

BACKGROUND: Overviews of systematic reviews (overviews) attempt to systematically retrieve and summarize the results of multiple systematic reviews (SRs) for a given condition or public health problem. Two prior descriptive analyses of overviews found substantial variation in the methodological approaches used in overviews, and deficiencies in reporting of key methodological steps. Since then, new methods have been developed so it is timely to update the prior descriptive analyses. The objectives are to: (1) investigate the epidemiological, descriptive, and reporting characteristics of a random sample of 100 overviews published from 2012 to 2016 and (2) compare these recently published overviews (2012-2016) to those published prior to 2012 (based on the prior descriptive analyses). METHODS: Medline, EMBASE, and CDSR will be searched for overviews published 2012-2016, using a validated search filter for overviews. Only overviews written in English will be included. All titles and abstracts will be screened by one review author; those deemed not relevant will be verified by a second person for exclusion. Full-texts will be assessed for inclusion by two reviewers independently. Of those deemed relevant, a random sample of 100 overviews will be selected for inclusion. Data extraction will be either performed by one reviewer with verification by a second reviewer or by one reviewer only depending on the complexity of the item. Discrepancies at any stage will be resolved by consensus or consulting a third person. Data will be extracted on the epidemiological, descriptive, and reporting characteristics of each overview. Data will be analyzed descriptively. When data are available for both time points (up to 2011 vs. 2012-2016), we will compare characteristics by calculating risk ratios or applying the Mann-Whitney test. DISCUSSION: Overviews are becoming increasingly valuable evidence syntheses, and the number of published overviews is increasing. However, former analyses found limitations in the conduct and reporting of overviews. This update of a recent sample of overviews will inform whether this has changed, while also identifying areas for further improvement. SYSTEMATIC REVIEW REGISTRATION: The review will not be registered in PROSPERO as it does not meet the eligibility criterion of dealing with health-related outcomes.

Most overviews of Cochrane reviews neglected potential biases from dual authorship.

OBJECTIVE: Some authors of Cochrane overviews have also (co-)authored one or more of the underlying reviews. We examined the extent of dual (co-)authorship in Cochrane overviews, how it is dealt with, and whether the issue is raised in protocols. STUDY DESIGN: The Cochrane Library was searched for overviews and protocols for overviews in September 2015. Data on dual (co-)authorship were extracted for each review into standard spreadsheets by one author and checked for accuracy by a second. RESULTS: Twenty overviews and 25 protocols were identified. The overviews included a median of 10 reviews (interquartile range [IQR]: 6-18.5). In 18/20 overviews (90%), at least one of the included reviews was affected by dual (co-)authorship. A median of 5 (IQR, 2.5-7) reviews per overview was affected by dual (co-)authorship. In 8/18 (44%) overviews with dual (co-)authorship, quality assessment was conducted independently. In 7/25 (28%) protocols, dual (co-)authorship was mentioned. CONCLUSION: Potential biases arising from dual (co-)authorship are often neglected in Cochrane overviews. We argue that authors of Cochrane overviews and Review Groups should pay more attention to the issue, to avoid bias and preserve the good reputation that Cochrane overviews will typically deserve.

[Overviews - status quo, potentials and perspectives]. / Overviews - Status Quo, Potentiale und Ausblick.

The increase in scientific literature has led reviewers to conduct overviews, hereby creating a new publication type. Overviews of reviews may have several advantages. For example, they can be conducted in a shorter time frame, they offer the opportunity of providing decision makers with a broader summary of the evidence, allowing comparison of multiple treatments, and they can be helpful in investigating discordant findings from multiple systematic reviews. However, there are some specific characteristics concerning methodology in conducting overviews. Questions that arise are how to deal with overlapping reviews, how the quality of the included reviews should be assessed, how discordant results should be dealt with, whether and how additional searches for current primary studies should be conducted. Overviews of reviews often lack methodological rigour. Methodological standards and reporting guidelines for overviews are needed to improve the quality of this new publication type.