Public participation in decisions about measures to manage the COVID-19 pandemic: a systematic review.

BACKGROUND: During the COVID-19 pandemic, governments and health authorities faced tough decisions about infection prevention and control measures such as social distancing, face masks and travel. Judgements underlying those decisions require democratic input, as well as expert input. The aim of this review is to inform decisions about how best to achieve public participation in decisions about public health and social interventions in the context of a pandemic or other public health emergencies. OBJECTIVES: To systematically review examples of public participation in decisions by governments and health authorities about how to control the COVID-19 pandemic. DESIGN: We searched Participedia and relevant databases in August 2022. Two authors reviewed titles and abstracts and one author screened publications promoted to full text. One author extracted data from included reports using a standard data-extraction form. A second author checked 10% of the extraction forms. We conducted a structured synthesis using framework analysis. RESULTS: We included 24 reports (18 from Participedia). Most took place in high-income countries (n=23), involved 'consulting' the public (n=17) and involved public meetings (usually online). Two initiatives reported explicit support for critical thinking. 11 initiatives were formally evaluated (only three reported impacts). Many initiatives did not contribute to a decision, and 17 initiatives did not include any explicit decision-making criteria. CONCLUSIONS: Decisions about how to manage the COVID-19 pandemic affected nearly everyone. While public participation in those decisions had the potential to improve the quality of the judgements and decisions that were made, build trust, improve adherence and help ensure transparency and accountability, few examples of such initiatives have been reported and most of those have not been formally evaluated. Identified initiatives did point out potential good practices related to online engagement, crowdsourcing and addressing potential power imbalance. Future research should address improved reporting of initiatives, explicit decision-making criteria, support for critical thinking, engagement of marginalised groups and decision-makers and communication with the public. PROSPERO REGISTRATION NUMBER: 358991.

The use of GRADE-CERQual in qualitative evidence synthesis: an evaluation of fidelity and reporting.

BACKGROUND: GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) is a methodological approach to systematically and transparently assess how much confidence decision makers can place in individual review findings from qualitative evidence syntheses. The number of reviews applying GRADE-CERQual is rapidly expanding in guideline and other decision-making contexts. The objectives of this evaluation were, firstly, to describe the uptake of GRADE-CERQual in qualitative evidence synthesis by review authors and, secondly, to assess both reporting of and fidelity to the approach. METHODS: The evaluation had two parts. Part 1 was a citation analysis and descriptive overview of the literature citing GRADE-CERQual. Authors worked together to code and chart the citations, first by title and abstract and second by full text. Part 2 was an assessment and analysis of fidelity to, and reporting of, the GRADE-CERQual approach in included reviews. We developed fidelity and reporting questions and answers based on the most recent guidance for GRADE-CERQual and then used NVivo12 to document assessments in a spreadsheet and code full-text PDF articles for any concerns that had been identified. Our assessments were exported to Excel and we applied count formulae to explore patterns in the data. We employed a qualitative content analysis approach in NVivo12 to sub-coding all the data illustrating concerns for each reporting and fidelity criteria. RESULTS: 233 studies have applied the GRADE-CERQual approach, with most (n = 225, 96.5%) in the field of health research. Many studies (n = 97/233, 41.6%) were excluded from full fidelity and reporting assessment because they demonstrated a serious misapplication of GRADE-CERQual, for example interpreting it as a quality appraisal tool for primary studies or reviews. For the remaining studies that applied GRADE-CERQual to assess confidence in review findings, the main areas of reporting concern involved terminology, labelling and completeness. Fidelity concerns were identified in more than half of all studies assessed. CONCLUSIONS: GRADE-CERQual is being used widely within qualitative evidence syntheses and there are common reporting and fidelity issues. Most of these are avoidable and we highlight these as gaps in knowledge and guidance for applying the GRADE-CERQual approach.

User experiences of structured stakeholder engagement to consider transferability: The TRANSFER approach.

Background: Systematic reviews are increasingly used to inform decision-making in health, education, social care and environmental protection. However, decision makers still experience barriers to using reviews, including not knowing how findings might translate to their own contexts, and lack of collaboration with systematic review authors. The TRANSFER approach is a novel method that aims to support review authors to systematically and transparently collaborate with stakeholders to consider context and the transferability of review findings from the beginning of the review process. Such collaboration is intended to improve the usefulness and relevance of review findings for decision makers. Objectives: We aim to explore the user experience of the TRANSFER approach conversation guide, and in doing so gain a better understanding of the role and perceived value of stakeholder engagement in systematic reviews for informed decision-making. Methods: We conducted four user tests of groups using the guide, organized around simulated meetings between review authors and stakeholders. Review authors led the meeting using the TRANSFER approach conversation guide. We audio-recorded and observed the meetings, collected feedback forms and conducted semi-structured interviews with review authors following the meeting. We analysed the data using framework analysis to examine the user experience of the TRANSFER approach conversation guide and of stakeholder engagement more generally. Results: Seventeen participants in four user groups participated in the user tests. Most participants were generally positive toward the structured approach using the conversation guide, and felt it would be useful in systematic review projects. We observed examples of misunderstanding of the terminology included in the guide, and received multiple suggestions for how to make the conversation guide more user friendly. We observed numerous challenges related to the hypothetical nature of a user test, including lack of familiarity with the review question/topic among participants and lack of preparation for the meeting. Conclusions: Review authors and stakeholders are positive toward using a structured approach to guide collaboration within the context of a systematic review. The TRANSFER conversation guide helps participants to discuss the review question and context in a structured way. Such structured collaboration could help to improve the usefulness and relevance of systematic reviews for decision making by improving the review question, inclusion criteria and consideration of transferability of review findings. The conversation guide needs to be modified to improve user experience. Further research is needed to explore stakeholder collaboration and the use of the TRANSFER conversation guide in systematic review processes.

The TRANSFER Approach for assessing the transferability of systematic review findings.

BACKGROUND: Systematic reviews are a key input to health and social welfare decisions. Studies included in systematic reviews often vary with respect to contextual factors that may impact on how transferable review findings are to the review context. However, many review authors do not consider the transferability of review findings until the end of the review process, for example when assessing confidence in the evidence using GRADE or GRADE-CERQual. This paper describes the TRANSFER Approach, a novel approach for supporting collaboration between review authors and stakeholders from the beginning of the review process to systematically and transparently consider factors that may influence the transferability of systematic review findings. METHODS: We developed the TRANSFER Approach in three stages: (1) discussions with stakeholders to identify current practices and needs regarding the use of methods to consider transferability, (2) systematic search for and mapping of 25 existing checklists related to transferability, and (3) using the results of stage two to develop a structured conversation format which was applied in three systematic review processes. RESULTS: None of the identified existing checklists related to transferability provided detailed guidance for review authors on how to assess transferability in systematic reviews, in collaboration with decision makers. The content analysis uncovered seven categories of factors to consider when discussing transferability. We used these to develop a structured conversation guide for discussing potential transferability factors with stakeholders at the beginning of the review process. In response to feedback and trial and error, the TRANSFER Approach has developed, expanding beyond the initial conversation guide, and is now made up of seven stages which are described in this article. CONCLUSIONS: The TRANSFER Approach supports review authors in collaborating with decision makers to ensure an informed consideration, from the beginning of the review process, of the transferability of the review findings to the review context. Further testing of TRANSFER is needed.

Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool.

BACKGROUND: Qualitative evidence synthesis is increasingly used alongside reviews of effectiveness to inform guidelines and other decisions. To support this use, the GRADE-CERQual approach was developed to assess and communicate the confidence we have in findings from reviews of qualitative research. One component of this approach requires an appraisal of the methodological limitations of studies contributing data to a review finding. Diverse critical appraisal tools for qualitative research are currently being used. However, it is unclear which tool is most appropriate for informing a GRADE-CERQual assessment of confidence. METHODOLOGY: We searched for tools that were explicitly intended for critically appraising the methodological quality of qualitative research. We searched the reference lists of existing methodological reviews for critical appraisal tools, and also conducted a systematic search in June 2016 for tools published in health science and social science databases. Two reviewers screened identified titles and abstracts, and then screened the full text of potentially relevant articles. One reviewer extracted data from each article and a second reviewer checked the extraction. We used a best-fit framework synthesis approach to code checklist criteria from each identified tool and to organise these into themes. RESULTS: We identified 102 critical appraisal tools: 71 tools had previously been included in methodological reviews, and 31 tools were identified from our systematic search. Almost half of the tools were published after 2010. Few authors described how their tool was developed, or why a new tool was needed. After coding all criteria, we developed a framework that included 22 themes. None of the tools included all 22 themes. Some themes were included in up to 95 of the tools. CONCLUSION: It is problematic that researchers continue to develop new tools without adequately examining the many tools that already exist. Furthermore, the plethora of tools, old and new, indicates a lack of consensus regarding the best tool to use, and an absence of empirical evidence about the most important criteria for assessing the methodological limitations of qualitative research, including in the context of use with GRADE-CERQual.

Perceptions and experiences of labour companionship: a qualitative evidence synthesis.

BACKGROUND: Labour companionship refers to support provided to a woman during labour and childbirth, and may be provided by a partner, family member, friend, doula or healthcare professional. A Cochrane systematic review of interventions by Bohren and colleagues, concluded that having a labour companion improves outcomes for women and babies. The presence of a labour companion is therefore regarded as an important aspect of improving quality of care during labour and childbirth; however implementation of the intervention is not universal. Implementation of labour companionship may be hampered by limited understanding of factors affecting successful implementation across contexts. OBJECTIVES: The objectives of the review were to describe and explore the perceptions and experiences of women, partners, community members, healthcare providers and administrators, and other key stakeholders regarding labour companionship; to identify factors affecting successful implementation and sustainability of labour companionship; and to explore how the findings of this review can enhance understanding of the related Cochrane systematic review of interventions. SEARCH METHODS: We searched MEDLINE, CINAHL, and POPLINE K4Health databases for eligible studies from inception to 9 September 2018. There were no language, date or geographic restrictions. SELECTION CRITERIA: We included studies that used qualitative methods for data collection and analysis; focused on women's, partners', family members', doulas', providers', or other relevant stakeholders' perceptions and experiences of labour companionship; and were from any type of health facility in any setting globally. DATA COLLECTION AND ANALYSIS: We used a thematic analysis approach for data extraction and synthesis, and assessed the confidence in the findings using the GRADE-CERQual approach. We used two approaches to integrate qualitative findings with the intervention review findings. We used a logic model to theorise links between elements of the intervention and health and well-being outcomes. We also used a matrix model to compare features of labour companionship identified as important in the qualitative evidence synthesis with the interventions included in the intervention review. MAIN RESULTS: We found 51 studies (52 papers), mostly from high-income countries and mostly describing women's perspectives. We assessed our level of confidence in each finding using the GRADE-CERQual approach. We had high or moderate confidence in many of our findings. Where we only had low or very low confidence in a finding, we have indicated this.Labour companions supported women in four different ways. Companions gave informational support by providing information about childbirth, bridging communication gaps between health workers and women, and facilitating non-pharmacological pain relief. Companions were advocates, which means they spoke up in support of the woman. Companions provided practical support, including encouraging women to move around, providing massage, and holding her hand. Finally, companions gave emotional support, using praise and reassurance to help women feel in control and confident, and providing a continuous physical presence.Women who wanted a companion present during labour and childbirth needed this person to be compassionate and trustworthy. Companionship helped women to have a positive birth experience. Women without a companion could perceive this as a negative birth experience. Women had mixed perspectives about wanting to have a male partner present (low confidence). Generally, men who were labour companions felt that their presence made a positive impact on both themselves (low confidence) and on the relationship with their partner and baby (low confidence), although some felt anxious witnessing labour pain (low confidence). Some male partners felt that they were not well integrated into the care team or decision-making.Doulas often met with women before birth to build rapport and manage expectations. Women could develop close bonds with their doulas (low confidence). Foreign-born women in high-income settings may appreciate support from community-based doulas to receive culturally-competent care (low confidence).Factors affecting implementation included health workers and women not recognising the benefits of companionship, lack of space and privacy, and fearing increased risk of infection (low confidence). Changing policies to allow companionship and addressing gaps between policy and practice were thought to be important (low confidence). Some providers were resistant to or not well trained on how to use companions, and this could lead to conflict. Lay companions were often not integrated into antenatal care, which may cause frustration (low confidence).We compared our findings from this synthesis to the companionship programmes/approaches assessed in Bohren's review of effectiveness. We found that most of these programmes did not appear to address these key features of labour companionship. AUTHORS' CONCLUSIONS: We have high or moderate confidence in the evidence contributing to several of these review findings. Further research, especially in low- and middle-income settings and with different cadres of healthcare providers, could strengthen the evidence for low- or very low-confidence findings. Ahead of implementation of labour companionship, researchers and programmers should consider factors that may affect implementation, including training content and timing for providers, women and companions; physical structure of the labour ward; specifying clear roles for companions and providers; integration of companions; and measuring the impact of companionship on women's experiences of care. Implementation research or studies conducted on labour companionship should include a qualitative component to evaluate the process and context of implementation, in order to better interpret results and share findings across contexts.

Systematic mapping of checklists for assessing transferability.

BACKGROUND: Systematic reviews of research evidence have become an expected basis for decisions about practice guidelines and policy decisions in the health and welfare sectors. Review authors define inclusion criteria to help them determine which studies to search for and include in their reviews. However, these studies may still vary in the extent to which they reflect the context of interest in the review question. While most review authors would agree that systematic reviews should be relevant and useful for decision makers, there appears to be few well known, if any, established methods for supporting review authors to assess the transferability of review findings to the context of interest in the review. With this systematic mapping and content analysis, we aim to identify whether there exists checklists to support review authors in considering transferability early in the systematic review process. The secondary aim was to develop a comprehensive list of factors that influence transferability as discussed in existing checklists. METHODS: We conducted a systematic mapping of checklists and performed a content analysis of the checklist criteria included in the identified checklists. In June 2016, we conducted a systematic search of eight databases to identify checklists to assess transferability of findings from primary or secondary research, without limitations related to publication type, status, language, or date. We also conducted a gray literature search and searched the EQUATOR repository of checklists for any relevant document. We used search terms such as modified versions of the terms "transferability," "applicability," "generalizability," etc. and "checklist," "guideline," "tool," "criteria," etc. We did not include papers that discussed transferability at a theoretical level or checklists to assess the transferability of guidelines to local contexts. RESULTS: Our search resulted in 11,752 titles which were screened independently by two review authors. The 101 articles which were considered potentially relevant were subsequently read by two authors, independently in full text and assessed for inclusion. We identified 31 relevant checklists. Six of these examined transferability of economic evaluations, and 25 examined transferability of primary or secondary research findings in health (n = 23) or social welfare (n = 2). The content analysis is based on the 25 health and social welfare checklists. We identified seven themes under which we grouped categories of checklist criteria: population, intervention, implementation context (immediate), comparison intervention, outcomes, environmental context, and researcher conduct. CONCLUSIONS: We identified a variety of checklists intended to support end users (researchers, review authors, practitioners, etc.) to assess transferability or related concepts. While four of these checklists are intended for use in systematic reviews of effectiveness, we found no checklists for qualitative evidence syntheses or for the field of social welfare practice or policy. Furthermore, none of the identified checklists for review authors included guidance to on how to assess transferability, or present assessments in a systematic review. The results of the content analysis can serve as the basis for developing a comprehensive list of factors to be used in an approach to support review authors in systematically and transparently considering transferability from the beginning of the review process.

Experiences of living with fetal alcohol spectrum disorders: a systematic review and synthesis of qualitative data.

AIM: To identify and assess available evidence from qualitative studies exploring experiences of individuals living with fetal alcohol spectrum disorders (FASD) or those living with a child with FASD, as well as experiences of interventions aimed at supporting individuals with FASD and their families. METHOD: A systematic literature search was conducted in six electronic databases: PubMed, Embase, Cochrane Library, CINAHL, PsycINFO, and Scopus. Included studies were analysed using manifest content analysis. Methodological limitations and confidence in the evidence were assessed using a modified version of the Critical Appraisal Skills Programme and the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in the Evidence from Reviews of Qualitative Research approach respectively. RESULTS: Findings from 18 studies show that individuals with FASD experience a variation of disabilities, ranging from somatic problems, high pain tolerance, destructive behaviour, hyperactivity, and aggressiveness, to social problems with friendship, school attendance, and maintenance of steady employment. Most studies reported parents' experiences with FASD; parenting was viewed as a lifelong engagement and that the whole family is isolated and burdened because of FASD. People with FASD feel that their difficulties affect their daily life in a limiting way and make them feel different from others. INTERPRETATION: From the perspective of primarily parents, individuals with FASD and their parents face many different difficulties, for which they need societal support. WHAT THIS PAPER ADDS: Individuals with fetal alcohol spectrum disorders (FASD) feel their difficulties make them different from others. Parents think of their parenting as a lifelong engagement. There is a shortage of studies investigating experiences of children with FASD. There is a shortage of studies investigating experiences of interventions given to individuals with FASD.

Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 7: understanding the potential impacts of dissemination bias.

BACKGROUND: The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on a probable fifth component, dissemination bias. Given its exploratory nature, we are not yet able to provide guidance on applying this potential component of the CERQual approach. Instead, we focus on how dissemination bias might be conceptualised in the context of qualitative research and the potential impact dissemination bias might have on an overall assessment of confidence in a review finding. We also set out a proposed research agenda in this area. METHODS: We developed this paper by gathering feedback from relevant research communities, searching MEDLINE and Web of Science to identify and characterise the existing literature discussing or assessing dissemination bias in qualitative research and its wider implications, developing consensus through project group meetings, and conducting an online survey of the extent, awareness and perceptions of dissemination bias in qualitative research. RESULTS: We have defined dissemination bias in qualitative research as a systematic distortion of the phenomenon of interest due to selective dissemination of studies or individual study findings. Dissemination bias is important for qualitative evidence syntheses as the selective dissemination of qualitative studies and/or study findings may distort our understanding of the phenomena that these syntheses aim to explore and thereby undermine our confidence in these findings. Dissemination bias has been extensively examined in the context of randomised controlled trials and systematic reviews of such studies. The effects of potential dissemination bias are formally considered, as publication bias, within the GRADE approach. However, the issue has received almost no attention in the context of qualitative research. Because of very limited understanding of dissemination bias and its potential impact on review findings in the context of qualitative evidence syntheses, this component is currently not included in the GRADE-CERQual approach. CONCLUSIONS: Further research is needed to establish the extent and impacts of dissemination bias in qualitative research and the extent to which dissemination bias needs to be taken into account when we assess how much confidence we have in findings from qualitative evidence syntheses.

Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 5: how to assess adequacy of data.

BACKGROUND: The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) working group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations; (2) coherence; (3) adequacy of data; and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual's adequacy of data component. METHODS: We developed the adequacy of data component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual adequacy of data component within several qualitative evidence syntheses before agreeing on the current definition and principles for application. RESULTS: When applying CERQual, we define adequacy of data as an overall determination of the degree of richness and the quantity of data supporting a review finding. In this paper, we describe the adequacy component and its rationale and offer guidance on how to assess data adequacy in the context of a review finding as part of the CERQual approach. This guidance outlines the information required to assess data adequacy, the steps that need to be taken to assess data adequacy, and examples of adequacy assessments. CONCLUSIONS: This paper provides guidance for review authors and others on undertaking an assessment of adequacy in the context of the CERQual approach. We approach assessments of data adequacy in terms of the richness and quantity of the data supporting each review finding, but do not offer fixed rules regarding what constitutes sufficiently rich data or an adequate quantity of data. Instead, we recommend that this assessment is made in relation to the nature of the finding. We expect the CERQual approach, and its individual components, to develop further as our experiences with the practical implementation of the approach increase.

Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 3: how to assess methodological limitations.

BACKGROUND: The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual's methodological limitations component. METHODS: We developed the methodological limitations component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual methodological limitations component within several qualitative evidence syntheses before agreeing on the current definition and principles for application. RESULTS: When applying CERQual, we define methodological limitations as the extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding. In this paper, we describe the methodological limitations component and its rationale and offer guidance on how to assess methodological limitations of a review finding as part of the CERQual approach. This guidance outlines the information required to assess methodological limitations component, the steps that need to be taken to assess methodological limitations of data contributing to a review finding and examples of methodological limitation assessments. CONCLUSIONS: This paper provides guidance for review authors and others on undertaking an assessment of methodological limitations in the context of the CERQual approach. More work is needed to determine which criteria critical appraisal tools should include when assessing methodological limitations. We currently recommend that whichever tool is used, review authors provide a transparent description of their assessments of methodological limitations in a review finding. We expect the CERQual approach and its individual components to develop further as our experiences with the practical implementation of the approach increase.

Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series.

The GRADE-CERQual ('Confidence in the Evidence from Reviews of Qualitative research') approach provides guidance for assessing how much confidence to place in findings from systematic reviews of qualitative research (or qualitative evidence syntheses). The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. Confidence in the evidence from qualitative evidence syntheses is an assessment of the extent to which a review finding is a reasonable representation of the phenomenon of interest. CERQual provides a systematic and transparent framework for assessing confidence in individual review findings, based on consideration of four components: (1) methodological limitations, (2) coherence, (3) adequacy of data, and (4) relevance. A fifth component, dissemination (or publication) bias, may also be important and is being explored. As with the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approach for effectiveness evidence, CERQual suggests summarising evidence in succinct, transparent, and informative Summary of Qualitative Findings tables. These tables are designed to communicate the review findings and the CERQual assessment of confidence in each finding. This article is the first of a seven-part series providing guidance on how to apply the CERQual approach. In this paper, we describe the rationale and conceptual basis for CERQual, the aims of the approach, how the approach was developed, and its main components. We also outline the purpose and structure of this series and discuss the growing role for qualitative evidence in decision-making. Papers 3, 4, 5, 6, and 7 in this series discuss each CERQual component, including the rationale for including the component in the approach, how the component is conceptualised, and how it should be assessed. Paper 2 discusses how to make an overall assessment of confidence in a review finding and how to create a Summary of Qualitative Findings table. The series is intended primarily for those undertaking qualitative evidence syntheses or using their findings in decision-making processes but is also relevant to guideline development agencies, primary qualitative researchers, and implementation scientists and practitioners.

Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 4: how to assess coherence.

BACKGROUND: The GRADE-CERQual (Grading of Recommendations Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE working group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) relevance, (3) coherence and (4) adequacy of data. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual's coherence component. METHODS: We developed the coherence component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual coherence component within several qualitative evidence syntheses before agreeing on the current definition and principles for application. RESULTS: When applying CERQual, we define coherence as how clear and cogent the fit is between the data from the primary studies and a review finding that synthesises that data. In this paper, we describe the coherence component and its rationale and offer guidance on how to assess coherence in the context of a review finding as part of the CERQual approach. This guidance outlines the information required to assess coherence, the steps that need to be taken to assess coherence and examples of coherence assessments. CONCLUSIONS: This paper provides guidance for review authors and others on undertaking an assessment of coherence in the context of the CERQual approach. We suggest that threats to coherence may arise when the data supporting a review finding are contradictory, ambiguous or incomplete or where competing theories exist that could be used to synthesise the data. We expect the CERQual approach, and its individual components, to develop further as our experiences with the practical implementation of the approach increase.

Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 6: how to assess relevance of the data.

BACKGROUND: The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual's relevance component. METHODS: We developed the relevance component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual relevance component within several qualitative evidence syntheses before agreeing on the current definition and principles for application. RESULTS: When applying CERQual, we define relevance as the extent to which the body of data from the primary studies supporting a review finding is applicable to the context (perspective or population, phenomenon of interest, setting) specified in the review question. In this paper, we describe the relevance component and its rationale and offer guidance on how to assess relevance in the context of a review finding. This guidance outlines the information required to assess relevance, the steps that need to be taken to assess relevance and examples of relevance assessments. CONCLUSIONS: This paper provides guidance for review authors and others on undertaking an assessment of relevance in the context of the CERQual approach. Assessing the relevance component requires consideration of potentially important contextual factors at an early stage in the review process. We expect the CERQual approach, and its individual components, to develop further as our experiences with the practical implementation of the approach increase.

Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table.

BACKGROUND: The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on making an overall assessment of confidence in a review finding and creating a CERQual Evidence Profile and a CERQual Summary of Qualitative Findings table. METHODS: We developed this guidance by examining the methods used by other GRADE approaches, gathering feedback from relevant research communities and developing consensus through project group meetings. We then piloted the guidance on several qualitative evidence syntheses before agreeing on the approach. RESULTS: Confidence in the evidence is an assessment of the extent to which a review finding is a reasonable representation of the phenomenon of interest. Creating a summary of each review finding and deciding whether or not CERQual should be used are important steps prior to assessing confidence. Confidence should be assessed for each review finding individually, based on the judgements made for each of the four CERQual components. Four levels are used to describe the overall assessment of confidence: high, moderate, low or very low. The overall CERQual assessment for each review finding should be explained in a CERQual Evidence Profile and Summary of Qualitative Findings table. CONCLUSIONS: Structuring and summarising review findings, assessing confidence in those findings using CERQual and creating a CERQual Evidence Profile and Summary of Qualitative Findings table should be essential components of undertaking qualitative evidence syntheses. This paper describes the end point of a CERQual assessment and should be read in conjunction with the other papers in the series that provide information on assessing individual CERQual components.

Effectiveness of interventions to reduce homelessness: a systematic review and meta-analysis.

This Campbell systematic review examines the effectiveness of interventions to reduce homelessness and increase residential stability for individuals who are homeless, or at risk of becoming homeless. Forty-three studies were included in the review, 37 of which are from the USA. Included interventions perform better than the usual services at reducing homelessness or improving housing stability in all comparisons. These interventions are: High intensity case managementHousing FirstCritical time interventionAbstinence-contingent housingNon-abstinence-contingent housing with high intensity case managementHousing vouchersResidential treatment These interventions seem to have similar beneficial effects, so it is unclear which of these is best with respect to reducing homelessness and increasing housing stability. Plain Language Summary: Interventions to reduce homelessness and improve housing stability are effective: There are large numbers of homeless people around the world. Interventions to address homelessness seem to be effective, though better quality evidence is required.What is this review about?: There are large numbers of homeless people around the world. Recent estimates are over 500,000 people in the USA, 100,000 in Australia and 30,000 in Sweden. Efforts to combat homelessness have been made on national levels as well as at local government levels.This review assesses the effectiveness of interventions combining housing and case management as a means to reduce homelessness and increase residential stability for individuals who are homeless, or at risk of becoming homeless.What studies are included?: Included studies were randomized controlled trials of interventions for individuals who were already, or at-risk of becoming, homeless, and which measured impact on homelessness or housing stability with follow-up of at least one year.A total of 43 studies were included. The majority of the studies (37) were conducted in the United States, with three from the United Kingdom and one each from Australia, Canada, and Denmark.What are the main findings of this review?: Included interventions perform better than the usual services at reducing homelessness or improving housing stability in all comparisons. These interventions are: High intensity case managementHousing FirstCritical time interventionAbstinence-contingent housingNon-abstinence-contingent housing with high intensity case managementHousing vouchersResidential treatment These interventions seem to have similar beneficial effects, so it is unclear which of these is best with respect to reducing homelessness and increasing housing stability.What do the findings of this review mean?: A range of housing programs and case management interventions appear to reduce homelessness and improve housing stability, compared to usual services.However, there is uncertainty in this finding as most the studies have risk of bias due to poor reporting, lack of blinding, or poor randomization or allocation concealment of participants. In addition to the general need for better conducted and reported studies, there are specific gaps in the research with respect to: 1) disadvantaged youth; 2) abstinence-contingent housing with case management or day treatment; 3) non-abstinence contingent housing comparing group vs independent living; 4) Housing First compared to interventions other than usual services, and; 5) studies outside of the USA.How up-to-date is this review?: The review authors searched for studies published up to January 2016. This Campbell systematic review was published in February 2018. Executive summary: Background: The United Nations Universal Declaration of Human Rights (Article 25) states that everyone has a right to housing. However, this right is far from being realized for many people worldwide. According to the United Nations High Commissioner for Refugees (UNHCR), there are approximately 100 million homeless people worldwide. The aim of this report is to contribute evidence to inform future decision making and practice for preventing and reducing homelessness.Objectives: To identify, appraise and summarize the evidence on the effectiveness of housing programs and case management to improve housing stability and reduce homelessness among people who are homeless or at-risk of becoming homeless.Search methods: We conducted a systematic review in accordance with the Norwegian Knowledge Centre's handbook. We systematically searched for literature in relevant databases and conducted a grey literature search which was last updated in January 2016.Selection criteria: Randomized controlled trials that included individuals who were already, or at-risk of becoming, homeless were included if they examined the effectiveness of relevant interventions on homelessness or housing stability. There were no limitations regarding language, country or length of homelessness. Two reviewers screened 2,918 abstracts and titles for inclusion. They read potentially relevant references in full, and included relevant studies in the review.Data collection and analysis: We pooled the results and conducted meta-analyses when possible. Our certainty in the primary outcomes was assessed using the Grading of Recommendations Assessment, Development, and Evaluation for effectiveness approach (GRADE).Results: We included 43 relevant studies (described in 78 publications) that examined the effectiveness of housing programs and/or case management services on homelessness and/or housing stability. The results are summarized below. Briefly, we found that the included interventions performed better than the usual services in all comparisons. However, certainty in the findings varied from very low to moderate. Most of the studies were assessed as having high risk of bias due to poor reporting, lack of blinding, or poor randomization and/or allocation concealment of participants.Case management: Case management is a process where clients are assigned case managers who assess, plan and facilitate access to health and social services necessary for the client's recovery. The intensity of these services can vary. One specific model is Critical time intervention, which is based on the same principles, but offered in three three-month periods that decrease in intensity.High intensity case management compared to usual services has generally more positive effects: It probably reduces the number of individuals who are homeless after 12-18 months by almost half (RR=0.59, 95%CI=0.41 to 0.87)(moderate certainty evidence); It may increase the number of people living in stable housing after 12-18 months and reduce the number of days an individual spends homeless (low certainty evidence), however; it may have no effect on the number of individuals who experience some homelessness during a two year period (low certainty evidence). When compared to low intensity case management, it may have little or no effect on time spent in stable housing (low certainty evidence).Critical time intervention compared to usual services may 1) have no effect on the number of people who experience homelessness, 2) lead to fewer days spent homeless, 3) lead to more days spent not homeless and, 4) reduce the amount of time it takes to move from shelter to independent housing (low certainty evidence).Abstinence-contingent housing programs: Abstinence-contingent housing is housing provided with the expectation that residents will remain sober. The results showed that abstinence-contingent housing may lead to fewer days spent homeless, compared with usual services (low certainty evidence).Non-abstinence-contingent housing programs: Non-abstinence-contingent housing is housing provided with no expectations regarding sobriety of residents. Housing First is the name of one specific non-abstinence-contingent housing program. When compared to usual services Housing First probably reduces the number of days spent homeless (MD=-62.5, 95%CI=-86.86 to -38.14) and increases the number of days in stable housing (MD=110.1, 95%CI=93.05 to 127.15) (moderate certainty evidence). In addition, it may increase the number of people placed in permanent housing after 20 months (low certainty evidence).Non-abstinence-contingent housing programs (not specified as Housing First) in combination with high intensity case management may reduce homelessness, compared to usual services (low certainty evidence). Group living arrangements may be better than individual apartments at reducing homelessness (low certainty evidence).Housing vouchers with case management: Housing vouchers is a housing allowance given to certain groups of people who qualify. The results showed that it mayreduce homelessness and improve housing stability, compared with usual services or case management (low certainty evidence).Residential treatment with case management: Residential treatment is a type of housing offered to clients who also need treatment for mental illness or substance abuse. We found that it mayreduce homelessness and improve housing stability, compared with usual services (low certainty evidence).Authors' conclusions: We found that a range of housing programs and case management interventions appear to reduce homelessness and improve housing stability, compared to usual services. The findings showed no indication of housing programs or case management resulting in poorer outcomes for homeless or at-risk individuals than usual services.Aside from a general need for better conducted and reported studies, there are specific gaps in the research. We identified research gaps concerning: 1)Disadvantaged youth; 2) Abstinence-contingent housing with case management or day treatment; 3) Non-abstinence contingent housing, specifically different living arrangements (group vs independent living); 4) Housing First compared to interventions other than usual services, and; 5) All interventions from contexts other than the USA.

Further exploration of dissemination bias in qualitative research required to facilitate assessment within qualitative evidence syntheses.

OBJECTIVES: To conceptualise and discuss dissemination bias in qualitative research. RESULTS: It is likely that the mechanisms leading to dissemination bias in quantitative research, including time lag, language, gray literature, and truncation bias also contribute to dissemination bias in qualitative research. These conceptual considerations have informed the development of a research agenda. CONCLUSION: Further exploration of dissemination bias in qualitative research is needed, including the extent of non-dissemination and related dissemination bias, and how to assess dissemination bias within qualitative evidence syntheses. We also need to consider the mechanisms through which dissemination bias in qualitative research could occur to explore approaches for reducing it.

Extent, Awareness and Perception of Dissemination Bias in Qualitative Research: An Explorative Survey.

BACKGROUND: Qualitative research findings are increasingly used to inform decision-making. Research has indicated that not all quantitative research on the effects of interventions is disseminated or published. The extent to which qualitative researchers also systematically underreport or fail to publish certain types of research findings, and the impact this may have, has received little attention. METHODS: A survey was delivered online to gather data regarding non-dissemination and dissemination bias in qualitative research. We invited relevant stakeholders through our professional networks, authors of qualitative research identified through a systematic literature search, and further via snowball sampling. RESULTS: 1032 people took part in the survey of whom 859 participants identified as researchers, 133 as editors and 682 as peer reviewers. 68.1% of the researchers said that they had conducted at least one qualitative study that they had not published in a peer-reviewed journal. The main reasons for non-dissemination were that a publication was still intended (35.7%), resource constraints (35.4%), and that the authors gave up after the paper was rejected by one or more journals (32.5%). A majority of the editors and peer reviewers "(strongly) agreed" that the main reasons for rejecting a manuscript of a qualitative study were inadequate study quality (59.5%; 68.5%) and inadequate reporting quality (59.1%; 57.5%). Of 800 respondents, 83.1% "(strongly) agreed" that non-dissemination and possible resulting dissemination bias might undermine the willingness of funders to support qualitative research. 72.6% and 71.2%, respectively, "(strongly) agreed" that non-dissemination might lead to inappropriate health policy and health care. CONCLUSIONS: The proportion of non-dissemination in qualitative research is substantial. Researchers, editors and peer reviewers play an important role in this. Non-dissemination and resulting dissemination bias may impact on health care research, practice and policy. More detailed investigations on patterns and causes of the non-dissemination of qualitative research are needed.

Internalized Homonegativity: A Systematic Mapping Review of Empirical Research.

Internalized homonegativity (IH) is an important variable affecting the wellbeing of lesbian, gay, and bisexual (LGB) persons. We included 201 studies in a systematic mapping review of IH. Most studies were conducted in North America and examined IH as a predictor of poor health. The primary focus of 14 studies was IH scale measurement, and, in total, these studies detailed nine distinct scales. Eighteen studies compared levels of IH in LGB populations, four described prevention programs, and one investigated IH using qualitative methods. Our review indicates that further research is needed, particularly qualitative research and ways to ameliorate IH.

Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).

Simon Lewin and colleagues present a methodology for increasing transparency and confidence in qualitative research synthesis.