Evaluating complex transformation.

PURPOSE: The purpose of this paper is to present a detailed case study of the evaluation strategies of a complex, multi-faceted response to a public health emergency: drug-related overdose deaths. It sets out the challenges of evaluating such a complex response and how they were overcome. It provides a pragmatic example of the rationale and issues faced to address the what, the why and particularly the how of the evaluation. DESIGN/METHODOLOGY/APPROACH: The case study overviews British Columbia's Provincial Response to the Overdose Public Health Emergency, and the aims and scope of its evaluation. It then outlines the conceptual approach taken to the evaluation, setting out key methodological challenges in evaluating large-scale, multi-level, multisectoral change. FINDINGS: The evaluation is developmental and summative, utilization focused and system informed. Defining the scope of the evaluation required a strong level of engagement with government leads, grantees and other evaluation stakeholders. Mixed method evaluation will be used to capture the complex pattern of relationships that have informed the overdose response. Working alongside people with drug use experience to both plan and inform the evaluation is critical to its success. ORIGINALITY/VALUE: This case study builds on a growing literature on evaluating large-scale and complex service transformation, providing a practical example of this.

Networks as systems.

Purpose The purpose of this paper is to present a case study of the World Health Organization's Global Healthcare Workforce Alliance (GHWA). Based on a commissioned evaluation of GHWA, it applies network theory and key concepts from systems thinking to explore network emergence, effectiveness, and evolution to over a ten-year period. The research was designed to provide high-level strategic guidance for further evolution of global governance in human resources for health (HRH). Design/methodology/approach Methods included a review of published literature on HRH governance and current practice in the field and an in-depth case study whose main data sources were relevant GHWA background documents and key informant interviews with GHWA leaders, staff, and stakeholders. Sampling was purposive and at a senior level, focusing on board members, executive directors, funders, and academics. Data were analyzed thematically with reference to systems theory and Shiffman's theory of network development. Findings Five key lessons emerged: effective management and leadership are critical; networks need to balance "tight" and "loose" approaches to their structure and processes; an active communication strategy is key to create and maintain support; the goals, priorities, and membership must be carefully focused; and the network needs to support shared measurement of progress on agreed-upon goals. Shiffman's middle-range network theory is a useful tool when guided by the principles of complex systems that illuminate dynamic situations and shifting interests as global alliances evolve. Research limitations/implications This study was implemented at the end of the ten-year funding cycle. A more continuous evaluation throughout the term would have provided richer understanding of issues. Experience and perspectives at the country level were not assessed. Practical implications Design and management of large, complex networks requires ongoing attention to key issues like leadership, and flexible structures and processes to accommodate the dynamic reality of these networks. Originality/value This case study builds on growing interest in the role of networks to foster large-scale change. The particular value rests on the longitudinal perspective on the evolution of a large, complex global network, and the use of theory to guide understanding.

Setting strategy for system change: using concept mapping to prioritise national action for chronic disease prevention.

BACKGROUND: Chronic diseases are a serious and urgent problem, requiring at-scale, multi-component, multi-stakeholder action and cooperation. Despite numerous national frameworks and agenda-setting documents to coordinate prevention efforts, Australia, like many countries internationally, is yet to substantively impact the burden from chronic disease. Improved evidence on effective strategies for the prevention of chronic disease is required. This research sought to articulate a priority set of important and feasible action domains to inform future discussion and debate regarding priority areas for chronic disease prevention policy and strategy. METHODS: Using concept mapping, a mixed-methods approach to making use of the best available tacit knowledge of recognised, diverse and well-experienced actors, and national actions to improve the prevention of chronic disease in Australia were identified and then mapped. Participants (ranging from 58 to 78 in the various stages of the research) included a national sample of academics, policymakers and practitioners. Data collection involved the generation and sorting of statements by participants. A series of visual representations of the data were then developed. RESULTS: A total of 95 statements were distilled into 12 clusters for action, namely Inter-Sectoral Partnerships; Systems Perspective/Action; Governance; Roles and Responsibilities; Evidence, Feedback and Learning; Funding and Incentive; Creating Demand; Primary Prevention; Social Determinants and Equity; Healthy Environments; Food and Nutrition; and Regulation and Policy. Specific areas for more immediate national action included refocusing the health system to prevention over cure, raising the profile of public health with health decision-makers, funding policy- and practice-relevant research, improving communication about prevention, learning from both global best-practice and domestic successes and failures, increasing the focus on primary prevention, and developing a long-term prevention strategy with an explicit funding commitment. CONCLUSIONS: Preventing chronic diseases and their risk factors will require at-scale, multi-component, multi-stakeholder action and cooperation. The concept mapping procedures used in this research have enabled the synthesis of views across different stakeholders, bringing both divergent and convergent perspectives to light, and collectively creating signals for where to prioritise national action. Previous national strategies for chronic disease prevention have not collated the tacit knowledge of diverse actors in the prevention of chronic disease in this structured way.

Scaling up complex interventions: insights from a realist synthesis.

Preventing chronic diseases, such as cancer, cardiovascular disease and diabetes, requires complex interventions, involving multi-component and multi-level efforts that are tailored to the contexts in which they are delivered. Despite an increasing number of complex interventions in public health, many fail to be 'scaled up'. This study aimed to increase understanding of how and under what conditions complex public health interventions may be scaled up to benefit more people and populations.A realist synthesis was conducted and discussed at an in-person workshop involving practitioners responsible for scaling up activities. Realist approaches view causality through the linkages between changes in contexts (C) that activate mechanisms (M), leading to specific outcomes (O) (CMO configurations). To focus this review, three cases of complex interventions that had been successfully scaled up were included: Vibrant Communities, Youth Build USA and Pathways to Education. A search strategy of published and grey literature related to each case was developed, involving searches of relevant databases and nominations from experts. Data extracted from included documents were classified according to CMO configurations within strategic themes. Findings were compared and contrasted with guidance from diffusion theory, and interpreted with knowledge users to identify practical implications and potential directions for future research.Four core mechanisms were identified, namely awareness, commitment, confidence and trust. These mechanisms were activated within two broad scaling up strategies, those of renewing and regenerating, and documenting success. Within each strategy, specific actions to change contexts included building partnerships, conducting evaluations, engaging political support and adapting funding models. These modified contexts triggered the identified mechanisms, leading to a range of scaling up outcomes, such as commitment of new communities, changes in relevant legislation, or agreements with new funding partners.This synthesis applies and advances theory, realist methods and the practice of scaling up complex interventions. Practitioners may benefit from a number of coordinated efforts, including conducting or commissioning evaluations at strategic moments, mobilising local and political support through relevant partnerships, and promoting ongoing knowledge exchange in peer learning networks. Action research studies guided by these findings, and studies on knowledge translation for realist syntheses are promising future directions.

Using systems thinking to support clinical system transformation.

Purpose - The British Columbia Ministry of Health's Clinical Care Management initiative was used as a case study to better understand large-scale change (LSC) within BC's health system. Using a complex system framework, the purpose of this paper is to examine mechanisms that enable and constrain the implementation of clinical guidelines across various clinical settings. Design/methodology/approach - Researchers applied a general model of complex adaptive systems plus two specific conceptual frameworks (realist evaluation and system dynamics mapping) to define and study enablers and constraints. Focus group sessions and interviews with clinicians, executives, managers and board members were validated through an online survey. Findings - The functional themes for managing large-scale clinical change included: creating a context to prepare clinicians for health system transformation initiatives; promoting shared clinical leadership; strengthening knowledge management, strategic communications and opportunities for networking; and clearing pathways through the complexity of a multilevel, dynamic system. Research limitations/implications - The action research methodology was designed to guide continuing improvement of implementation. A sample of initiatives was selected; it was not intended to compare and contrast facilitators and barriers across all initiatives and regions. Similarly, evaluating the results or process of guideline implementation was outside the scope; the methods were designed to enable conversations at multiple levels - policy, management and practice - about how to improve implementation. The study is best seen as a case study of LSC, offering a possible model for replication by others and a tool to shape further dialogue. Practical implications - Recommended action-oriented strategies included engaging local champions; supporting local adaptation for implementation of clinical guidelines; strengthening local teams to guide implementation; reducing change fatigue; ensuring adequate resources; providing consistent communication especially for front-line care providers; and supporting local teams to demonstrate the clinical value of the guidelines to their colleagues. Originality/value - Bringing a complex systems perspective to clinical guideline implementation resulted in a clear understanding of the challenges involved in LSC.

Sustaining organizational culture change in health systems.

PURPOSE: The questions addressed by this review are: first, what are the guiding principles underlying efforts to stimulate sustained cultural change; second, what are the mechanisms by which these principles operate; and, finally, what are the contextual factors that influence the likelihood of these principles being effective? The paper aims to discuss these issues. DESIGN/METHODOLOGY/APPROACH: The authors conducted a literature review informed by rapid realist review methodology that examined how interventions interact with contexts and mechanisms to influence the sustainability of cultural change. Reference and expert panelists assisted in refining the research questions, systematically searching published and grey literature, and helping to identify interactions between interventions, mechanisms and contexts. FINDINGS: Six guiding principles were identified: align vision and action; make incremental changes within a comprehensive transformation strategy; foster distributed leadership; promote staff engagement; create collaborative relationships; and continuously assess and learn from change. These principles interact with contextual elements such as local power distributions, pre-existing values and beliefs and readiness to engage. Mechanisms influencing how these principles sustain cultural change include activation of a shared sense of urgency and fostering flexible levels of engagement. PRACTICAL IMPLICATIONS: The principles identified in this review, along with the contexts and mechanisms that influence their effectiveness, are useful domains for policy and practice leaders to explore when grappling with cultural change. These principles are sufficiently broad to allow local flexibilities in adoption and application. ORIGINALITY/VALUE: This is the first study to adopt a realist approach for understanding how changes in organizational culture may be sustained. Through doing so, this review highlights the broad principles by which organizational action may be organized within enabling contextual settings.

Outcomes of Interorganizational Networks in Canada for Chronic Disease Prevention: Insights From a Concept Mapping Study, 2015.

INTRODUCTION: We conducted a mixed methods study from June 2014 to March 2015 to assess the perspectives of stakeholders in networks that adopt a population approach for chronic disease prevention (CDP). The purpose of the study was to identify important and feasible outcome measures for monitoring network performance. METHODS: Participants from CDP networks in Canada completed an online concept mapping exercise, which was followed by interviews with network stakeholders to further understand the findings. RESULTS: Nine concepts were considered important outcomes of CDP networks: enhanced learning, improved use of resources, enhanced or increased relationships, improved collaborative action, network cohesion, improved system outcomes, improved population health outcomes, improved practice and policy planning, and improved intersectoral engagement. Three themes emerged from participant interviews related to measurement of the identified concepts: the methodological difficulties in measuring network outcomes, the dynamic nature of network evolution and function and implications for outcome assessment, and the challenge of measuring multisectoral engagement in CDP networks. CONCLUSION: Results from this study provide initial insights into concepts that can be used to describe the outcomes of networks for CDP and may offer foundations for strengthening network outcome-monitoring strategies and methodologies.

Strengthening health systems in low-income countries by enhancing organizational capacities and improving institutions.

BACKGROUND: This paper argues that the global health agenda tends to privilege short-term global interests at the expense of long-term capacity building within national and community health systems. The Health Systems Strengthening (HSS) movement needs to focus on developing the capacity of local organizations and the institutions that influence how such organizations interact with local and international stakeholders. DISCUSSION: While institutions can enable organizations, they too often apply requirements to follow paths that can stifle learning and development. Global health actors have recognized the importance of supporting local organizations in HSS activities. However, this recognition has yet to translate adequately into actual policies to influence funding and practice. While there is not a single approach to HSS that can be uniformly applied to all contexts, several messages emerge from the experience of successful health systems presented in this paper using case studies through a complex adaptive systems lens. Two key messages deserve special attention: the need for donors and recipient organizations to work as equal partners, and the need for strong and diffuse leadership in low-income countries. An increasingly dynamic and interdependent post-Millennium Development Goals (post-MDG) world requires new ways of working to improve global health, underpinned by a complex adaptive systems lens and approaches that build local organizational capacity.

Improving the performance of interorganizational networks for preventing chronic disease: identifying and acting on research needs.

This article describes the role of interorganizational networks in chronic disease prevention and an action research agenda for promoting understanding and improvement. Through a model of engaged scholarship, leaders with expertise and experience in chronic disease prevention networks helped shape research directions focused on network value, governance, and evolution. The guiding principles for facilitating this research include applying existing knowledge, developing network-appropriate methods and measures, creating structural change, promoting an impact orientation, and fostering cultural change.

Networks to strengthen health systems for chronic disease prevention.

Interorganizational networks that harness the priorities, capacities, and skills of various agencies and individuals have emerged as useful approaches for strengthening preventive services in public health systems. We use examples from the Canadian Heart Health Initiative and Alberta's Primary Care Networks to illustrate characteristics of networks, describe the limitations of existing frameworks for assessing the performance of prevention-oriented networks, and propose a research agenda for guiding future efforts to improve the performance of these initiatives. Prevention-specific assessment strategies that capture relevant aspects of network performance need to be identified, and feedback mechanisms are needed that make better use of these data to drive change in network activities.

A time-responsive tool for informing policy making: rapid realist review.

BACKGROUND: A realist synthesis attempts to provide policy makers with a transferable theory that suggests a certain program is more or less likely to work in certain respects, for particular subjects, in specific kinds of situations. Yet realist reviews can require considerable and sustained investment over time, which does not always suit the time-sensitive demands of many policy decisions. 'Rapid Realist Review' methodology (RRR) has been developed as a tool for applying a realist approach to a knowledge synthesis process in order to produce a product that is useful to policy makers in responding to time-sensitive and/or emerging issues, while preserving the core elements of realist methodology. METHODS: Using examples from completed RRRs, we describe key features of the RRR methodology, the resources required, and the strengths and limitations of the process. All aspects of an RRR are guided by both a local reference group, and a group of content experts. Involvement of knowledge users and external experts ensures both the usability of the review products, as well as their links to current practice. RESULTS: RRRs have proven useful in providing evidence for and making explicit what is known on a given topic, as well as articulating where knowledge gaps may exist. From the RRRs completed to date, findings broadly adhere to four (often overlapping) classifications: guiding rules for policy-making; knowledge quantification (i.e., the amount of literature available that identifies context, mechanisms, and outcomes for a given topic); understanding tensions/paradoxes in the evidence base; and, reinforcing or refuting beliefs and decisions taken. CONCLUSIONS: 'Traditional' realist reviews and RRRs have some key differences, which allow policy makers to apply each type of methodology strategically to maximize its utility within a particular local constellation of history, goals, resources, politics and environment. In particular, the RRR methodology is explicitly designed to engage knowledge users and review stakeholders to define the research questions, and to streamline the review process. In addition, results are presented with a focus on context-specific explanations for what works within a particular set of parameters rather than producing explanations that are potentially transferrable across contexts and populations. For policy makers faced with making difficult decisions in short time frames for which there is sufficient (if limited) published/research and practice-based evidence available, RRR provides a practical, outcomes-focused knowledge synthesis method.

Doing the dance of culture change: complexity, evidence and leadership.

The challenge of culture change in hospitals must address three distinct but interwoven tensions: the need to shift paradigm and understand healthcare as a complex adaptive system; the challenge of knitting together the contributions of both evidence-based medicine and practice-based evidence; and the critical role of distributed, problem-focused leadership.The authors of the lead paper highlight five key issues in addressing this challenge: (1) the implementation of strategies like front-line ownership (FLO) in the context of macro-level social forces; (2) the central role of distributed leadership and its strengthening within the organization; (3) the need to attend to developing systems thinking skills at all levels; (4) the very significant challenge of how to scale up the labour-intensive change strategies within FLO, the role of "simple rules" and the potential for systems thinking tools such as concept mapping and dynamic modelling; and (5) the concurrent orchestration of not one culture change but three tensions in the challenge FLO represents to simpler versus complex adaptive systems, leadership and management and the balance between evidence-based medicine and practice-based evidence, at the clinical, organizational and macro-system levels.

Rethinking health systems strengthening: key systems thinking tools and strategies for transformational change.

While reaching consensus on future plans to address current global health challenges is far from easy, there is broad agreement that reductionist approaches that suggest a limited set of targeted interventions to improve health around the world are inadequate. We argue that a comprehensive systems perspective should guide health practice, education, research and policy. We propose key 'systems thinking' tools and strategies that have the potential for transformational change in health systems. Three overarching themes span these tools and strategies: collaboration across disciplines, sectors and organizations; ongoing, iterative learning; and transformational leadership. The proposed tools and strategies in this paper can be applied, in varying degrees, to every organization within health systems, from families and communities to national ministries of health. While our categorization is necessarily incomplete, this initial effort will provide a valuable contribution to the health systems strengthening debate, as the need for a more systemic, rigorous perspective in health has never been greater.

Large-system transformation in health care: a realist review.

CONTEXT: An evidence base that addresses issues of complexity and context is urgently needed for large-system transformation (LST) and health care reform. Fundamental conceptual and methodological challenges also must be addressed. The Saskatchewan Ministry of Health in Canada requested a six-month synthesis project to guide four major policy development and strategy initiatives focused on patient- and family-centered care, primary health care renewal, quality improvement, and surgical wait lists. The aims of the review were to analyze examples of successful and less successful transformation initiatives, to synthesize knowledge of the underlying mechanisms, to clarify the role of government, and to outline options for evaluation. METHODS: We used realist review, whose working assumption is that a particular intervention triggers particular mechanisms of change. Mechanisms may be more or less effective in producing their intended outcomes, depending on their interaction with various contextual factors. We explain the variations in outcome as the interplay between context and mechanisms. We nested this analytic approach in a macro framing of complex adaptive systems (CAS). FINDINGS: Our rapid realist review identified five "simple rules" of LST that were likely to enhance the success of the target initiatives: (1) blend designated leadership with distributed leadership; (2) establish feedback loops; (3) attend to history; (4) engage physicians; and (5) include patients and families. These principles play out differently in different contexts affecting human behavior (and thereby contributing to change) through a wide range of different mechanisms. CONCLUSIONS: Realist review methodology can be applied in combination with a complex system lens on published literature to produce a knowledge synthesis that informs a prospective change effort in large-system transformation. A collaborative process engaging both research producers and research users contributes to local applications of universal principles and mid-range theories, as well as to a more robust knowledge base for applied research. We conclude with suggestions for the future development of synthesis and evaluation methods.

System tools for system change.

BACKGROUND: Health system transformations are influenced by dynamic relationships within and between individuals and institutions, as well as political, educational and legislative factors. This article aims to promote awareness of five tools that recognise this complexity and that are proposed to have value for decision makers: concept mapping, social network analysis, system dynamics modelling, programme budgeting and marginal analysis, and the tools for knowledge management and translation. METHODS: The authors briefly describe the methodological approach of each tool, provide a commentary on the conditions in which these tools have been employed, and discuss their impact on the processes and outcomes of system transformation. An international advisory panel was convened based on a combination of experience, expertise and perspective. The panel assisted in synthesising the evidence relating to each tool and, in partnership with the authors, refined the interpretation of the role and value of each tool for system transformation. FINDINGS: The tools discussed may impact the structural and procedural outcomes of transformation as well as the values, behaviours and attitudes of people undergoing change. The techniques described provide those undertaking transformation with methods to negotiate clinical, academic, political, organisational and cultural perspectives, and recognise the pivotal role of context in transformation. CONCLUSIONS: This review offers a novel synthesis of how these tools may add value to decision making for health policy. The tools discussed, while not a panacea to the challenges of large system change, provide methods that acknowledge the complexity of the transformative challenge and present innovative paths to co-produced solutions.

It's a matter of values: partnership for innovative change.

We need new ways of thinking and of working in order to accommodate the complexity of the challenges in and urgent need for health system innovation and change. Solution seeking must begin with the convergence of two driving imperatives: the need to ground partnership in shared values and the need for systems thinking. The authors see three core value perspectives as central to partnerships for change: a patient- and family-centred social responsibility and equity paradigm, a commitment to changing outcomes and an evidence-informed strategy that integrates needs for research and knowledge translation. These imperatives can be expressed as a simple value stream: (1) articulate the shared values foundation of key partners; (2) express a common vision for changes needed; (3) develop a governance framework articulating roles, accountability and decision-making; (4) collaborate on an integrated intervention plan that takes complexity into account; and (5) ensure continuous improvement based on measured outcomes. The authors link this value stream to a six-point framework of guiding principles for innovation and implementation and discuss these six principles: values, systems, thinking, leadership, governance, learning networks and innovation research. Working partnerships among government, health services researchers and academic health science networks are essential if innovative change is to be implemented and sustained.

Mandatory public reporting: build it and who will come?

Rates of healthcare-associated infections (HAI) are being reported on an increasing number of public information websites in response to legislative mandates driven by consumer advocacy. This represents a new strategy to advance patient safety and quality of care by informing a broad audience about the relative performance of individual healthcare facilities. Unlike typical consumer health informatics products, the target audience and targeted health behaviors are less easily defined; further, the impact on providers to improve care is unknown relative to other incentives to improve. To address critical knowledge gaps facing all state agencies embarking on this new frontier, we found it essential and straightforward to recruit the assistance of university research faculty from a variety of disciplines. That interdisciplinary group was quickly able to define a 5-year applied evaluation research agenda spanning a progressive set of crucial questions.