Using systems perspectives in evidence synthesis: A methodological mapping review.

Reviewing complex interventions is challenging because they include many elements that can interact dynamically in a nonlinear manner. A systems perspective offers a way of thinking to help understand complex issues, but its application in evidence synthesis is not established. The aim of this project was to understand how and why systems perspectives have been applied in evidence synthesis. A methodological mapping review was conducted to identify papers using a systems perspective in evidence synthesis. A search was conducted in seven bibliographic databases and three search engines. A total of 101 papers (representing 98 reviews) met the eligibility criteria. Two categories of reviews were identified: (1) reviews using a "systems lens" to frame the topic, generate hypotheses, select studies, and guide the analysis and interpretation of findings (n = 76) and (2) reviews using systems methods to develop a systems model (n = 22). Several methods (e.g., systems dynamic modeling, soft systems approach) were identified, and they were used to identify, rank and select elements, analyze interactions, develop models, and forecast needs. The main reasons for using a systems perspective were to address complexity, view the problem as a whole, and understand the interrelationships between the elements. Several challenges for capturing the true nature and complexity of a problem were raised when performing these methods. This review is a useful starting point when designing evidence synthesis of complex interventions. It identifies different opportunities for applying a systems perspective in evidence synthesis, and highlights both commonplace and less familiar methods.

Why we habitually engage in null-hypothesis significance testing: A qualitative study.

BACKGROUND: Null Hypothesis Significance Testing (NHST) is the most familiar statistical procedure for making inferences about population effects. Important problems associated with this method have been addressed and various alternatives that overcome these problems have been developed. Despite its many well-documented drawbacks, NHST remains the prevailing method for drawing conclusions from data. Reasons for this have been insufficiently investigated. Therefore, the aim of our study was to explore the perceived barriers and facilitators related to the use of NHST and alternative statistical procedures among relevant stakeholders in the scientific system. METHODS: Individual semi-structured interviews and focus groups were conducted with junior and senior researchers, lecturers in statistics, editors of scientific journals and program leaders of funding agencies. During the focus groups, important themes that emerged from the interviews were discussed. Data analysis was performed using the constant comparison method, allowing emerging (sub)themes to be fully explored. A theory substantiating the prevailing use of NHST was developed based on the main themes and subthemes we identified. RESULTS: Twenty-nine interviews and six focus groups were conducted. Several interrelated facilitators and barriers associated with the use of NHST and alternative statistical procedures were identified. These factors were subsumed under three main themes: the scientific climate, scientific duty, and reactivity. As a result of the factors, most participants feel dependent in their actions upon others, have become reactive, and await action and initiatives from others. This may explain why NHST is still the standard and ubiquitously used by almost everyone involved. CONCLUSION: Our findings demonstrate how perceived barriers to shift away from NHST set a high threshold for actual behavioral change and create a circle of interdependency between stakeholders. By taking small steps it should be possible to decrease the scientific community's strong dependence on NHST and p-values.

Quantitizing findings from qualitative studies for integration in mixed methods reviewing.

In mixed methods reviewing, data from quantitative and qualitative studies are combined at the review level. One possible way to combine findings of quantitative and qualitative studies is to quantitize qualitative findings prior to their incorporation in a quantitative review. There are only a few examples of the quantification of qualitative findings within this context. This study adds to current research on mixed methods review methodology by reporting the pilot implementation of a new four-step quantitizing approach. We report how we extract and quantitize the strength of relationships found in qualitative studies by assigning correlations to vague quantifiers in text fragments. This article describes (a) how the analysis is prepared; (b) how vague quantifiers in text fragments are organized and transformed to numerical values; (c) how qualitative studies as a whole are assigned effect sizes; and (d) how the overall mean effects size and variance can be calculated. The pilot implementation shows how findings from 26 primary qualitative studies are transformed into mean effect sizes and corresponding variances.

Using the realist perspective to link theory from qualitative evidence synthesis to quantitative studies: Broadening the matrix approach.

BACKGROUND: This study describes an approach for the use of a specific type of qualitative evidence synthesis in the matrix approach, a mixed studies reviewing method. The matrix approach compares quantitative and qualitative data on the review level by juxtaposing concrete recommendations from the qualitative evidence synthesis against interventions in primary quantitative studies. However, types of qualitative evidence syntheses that are associated with theory building generate theoretical models instead of recommendations. Therefore, the output from these types of qualitative evidence syntheses cannot directly be used for the matrix approach but requires transformation. This approach allows for the transformation of these types of output. METHOD: The approach enables the inference of moderation effects instead of direct effects from the theoretical model developed in a qualitative evidence synthesis. Recommendations for practice are formulated on the basis of interactional relations inferred from the qualitative evidence synthesis. In doing so, we apply the realist perspective to model variables from the qualitative evidence synthesis according to the context-mechanism-outcome configuration. FINDINGS: A worked example shows that it is possible to identify recommendations from a theory-building qualitative evidence synthesis using the realist perspective. We created subsets of the interventions from primary quantitative studies based on whether they matched the recommendations or not and compared the weighted mean effect sizes of the subsets. The comparison shows a slight difference in effect sizes between the groups of studies. The study concludes that the approach enhances the applicability of the matrix approach.