Health system performance assessment: embedding resilience through performance intelligence.

Health systems around the world are facing challenges in achieving their goals. In the wake of the coronavirus disease pandemic, the need for resilient health systems has become even more apparent. This article argues that embedding resilience into health system performance assessment (HSPA) frameworks can be a valuable approach for improving health system performance. This perspective examines key challenges threatening health systems and makes a case for the continued relevance of HSPA by embedding resilience-related performance intelligence.

Status of the health information system in Ireland and its fitness to support health system performance assessment: a multimethod assessment based on stakeholder involvement.

BACKGROUND: Between 2019 and 2021, the first Irish health system performance assessment (HSPA) framework was developed. As routinely collected health data are necessary to continuously populate indicators of an HSPA framework, a purpose-driven assessment of the health information system (HIS) in Ireland and its fitness to support the implementation of an HSPA framework was conducted. This study reports on the status of the Irish HIS through a multimethod assessment based on continuous broad stakeholder involvement. METHODS: Between May and November 2020, over 50 informants were engaged in individual and group interviews and stakeholder consultation workshops as part of the HIS assessment process. Descriptive themes and high-level data availability heatmaps were derived from interview and workshop data using thematic analysis. Indicator "passports" for the HSPA framework were populated during stakeholder consultation workshops and analysed using univariate descriptive statistics. RESULTS: The HIS in Ireland was able to provide administrative, survey and registry-based data for public sector acute care services, focusing on structure, process and output metrics. Significant data availability gaps, most notably from primary care, private hospitals and community care, were reported, with little availability of electronic health record and people-reported data. Data on outcome metrics were mostly missing, as were linkage possibilities across datasets for care pathway monitoring. The COVID-19 pandemic highlighted the national HIS's shortcomings but also the capacity for rapid development and improvement. CONCLUSIONS: A tailor-made assessment of the HIS in Ireland, involving a broad set of relevant stakeholders, revealed strengths, weaknesses and areas for improvement in the Irish health data landscape. It also contributed to the development of a national HSPA framework and momentum to further strengthen data infrastructure and governance, while working towards a more data-driven and person-centred healthcare system. This work demonstrates the utility of an inclusive HIS assessment process and is applicable beyond Ireland, where this case study was conducted.

The current and potential uses of Electronic Medical Record (EMR) data for primary health care performance measurement in the Canadian context: a qualitative analysis.

BACKGROUND: Electronic Medical Records (EMRs) are a rich data source to measure and improve quality of care. As Canadian primary health care (PHC) EMRs mature, there is increasing potential use of EMR data for performance measurement. This study identifies and describes current uses of EMR data for performance measurement and considerations to further its potential in the Canadian context. METHODS: We applied a qualitative case study design and descriptive assessment in three phases, consulting multiple data sources including scientific and grey literature, system leaders (n = 41), and clinician/researchers (n = 20). Phases included a multimethod approach to identify initiatives using EMR data for performance measurement across Canadian jurisdictions; in-depth review of current initiatives identified from a healthcare performance intelligence lens; and triangulation and thematic analysis across data sources to explore considerations for advancing performance measurement uses of EMR data in the Canadian context. RESULTS: Six initiatives of EMR data use for performance measurement were identified: one multi-jurisdictional; five jurisdiction-specific in the provinces of British Columbia, Manitoba and Ontario. EMR data uses were predominately for micro-level PHC physician and team performance improvement, with some use for meso-level organization/network-wide improvement. Indicator sets varied in number, though shared emphasis on chronic disease management and prevention/screening and to a lesser extent medication management. Key considerations for governing, resourcing and implementing EMR data for performance measurement were identified. CONCLUSIONS: The extent of EMR data use for performance measurement varies across Canada. To further its potential, pan-Canadian data and privacy standards, performance intelligence competencies and renewed core PHC indicators should be prioritized. Experiences across countries, coupled with increasing momentum for performance measurement using real-world data, should be leveraged to avoid unnecessarily slow progress in Canada and abroad.

Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study.

BACKGROUND: Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration. OBJECTIVE: We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time. METHODS: We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point. RESULTS: Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards. CONCLUSIONS: Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

Features Constituting Actionable COVID-19 Dashboards: Descriptive Assessment and Expert Appraisal of 158 Public Web-Based COVID-19 Dashboards.

BACKGROUND: Since the outbreak of COVID-19, the development of dashboards as dynamic, visual tools for communicating COVID-19 data has surged worldwide. Dashboards can inform decision-making and support behavior change. To do so, they must be actionable. The features that constitute an actionable dashboard in the context of the COVID-19 pandemic have not been rigorously assessed. OBJECTIVE: The aim of this study is to explore the characteristics of public web-based COVID-19 dashboards by assessing their purpose and users ("why"), content and data ("what"), and analyses and displays ("how" they communicate COVID-19 data), and ultimately to appraise the common features of highly actionable dashboards. METHODS: We conducted a descriptive assessment and scoring using nominal group technique with an international panel of experts (n=17) on a global sample of COVID-19 dashboards in July 2020. The sequence of steps included multimethod sampling of dashboards; development and piloting of an assessment tool; data extraction and an initial round of actionability scoring; a workshop based on a preliminary analysis of the results; and reconsideration of actionability scores followed by joint determination of common features of highly actionable dashboards. We used descriptive statistics and thematic analysis to explore the findings by research question. RESULTS: A total of 158 dashboards from 53 countries were assessed. Dashboards were predominately developed by government authorities (100/158, 63.0%) and were national (93/158, 58.9%) in scope. We found that only 20 of the 158 dashboards (12.7%) stated both their primary purpose and intended audience. Nearly all dashboards reported epidemiological indicators (155/158, 98.1%), followed by health system management indicators (85/158, 53.8%), whereas indicators on social and economic impact and behavioral insights were the least reported (7/158, 4.4% and 2/158, 1.3%, respectively). Approximately a quarter of the dashboards (39/158, 24.7%) did not report their data sources. The dashboards predominately reported time trends and disaggregated data by two geographic levels and by age and sex. The dashboards used an average of 2.2 types of displays (SD 0.86); these were mostly graphs and maps, followed by tables. To support data interpretation, color-coding was common (93/158, 89.4%), although only one-fifth of the dashboards (31/158, 19.6%) included text explaining the quality and meaning of the data. In total, 20/158 dashboards (12.7%) were appraised as highly actionable, and seven common features were identified between them. Actionable COVID-19 dashboards (1) know their audience and information needs; (2) manage the type, volume, and flow of displayed information; (3) report data sources and methods clearly; (4) link time trends to policy decisions; (5) provide data that are "close to home"; (6) break down the population into relevant subgroups; and (7) use storytelling and visual cues. CONCLUSIONS: COVID-19 dashboards are diverse in the why, what, and how by which they communicate insights on the pandemic and support data-driven decision-making. To leverage their full potential, dashboard developers should consider adopting the seven actionability features identified.

Engaging citizens in the development of a health system performance assessment framework: a case study in Ireland.

BACKGROUND: The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework which should reflect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reflects on the added value of using a citizen panel in the development of an HSPA framework. METHODS: A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panellists received lay-language preparatory materials before the meeting. Panellists used a three-colour scheme to signal the importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups. RESULTS: The citizen panel generated 249 health performance measures ranging across 13 domains. Top-ranking domains to the citizen panel (people-centredness, coordination of care, and coverage) were less prioritized by the other panels; domains less prioritized by the citizen panel, such as accessibility, responsiveness, efficiency, and effectiveness, were of higher priority in the other panels. Citizen panellists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative. CONCLUSIONS: The priorities of the citizen panel were accounted for during all phases of developing the HSPA framework. This was possible by adopting an inclusive development process and by engaging citizens early on. Citizen engagement in HSPA development is essential for realizing citizen-driven healthcare system performance and generating trust and ownership in performance intelligence. Future research could expand the use of citizen panels to assess, monitor, and report on the performance of healthcare systems.

A problem-oriented systems approach to primary care system development: development and initial testing of the problem-oriented primary care system development record.

BACKGROUND: There is significant global policy interest related to enabling a data-driven approach for evidence-based primary care system development. This paper describes the development and initial testing of a prototype tool (the Problem-Oriented Primary Care System Development Record, or PCSDR) that enables a data-driven and contextualized approach to primary care system development. METHODS: The PCSDR is an electronic record that enables the systematic input, classification, structuring, storage, processing and analysis of different types of data related to the structure, function and performance of primary care systems over time. Data inputted into the PCSDR was coded using the WHO's PHC-IMPACT framework and classification system. The PCSDR's functionalities were tested by using a case study of primary care system development in Tajikistan. RESULTS: Tajikistan's case study demonstrated that the PCSDR is a potentially effective and conceptually-sound tool for the input, classification, structuring and storage of different data types from myriad sources. The PCSDR is therefore a basic data entry and data management system that enables query and analytics functions for health services research and evidence-based primary care system development functions. CONCLUSIONS: The PCSDR is a data system that enables a contextualized approach to evidence-based primary care system development. It represents a coherent and effective synthesis of the fields of primary care system development and performance assessment. The PCSDR enables analysts to leverage primary care performance assessment frameworks for a broad range of functions related to health systems analysis, improvement and the development of learning health systems.

Creating performance intelligence for primary health care strengthening in Europe.

BACKGROUND: Primary health care and its strengthening through performance measurement is essential for sustainably working towards universal health coverage. Existing performance frameworks and indicators to measure primary health care capture system functions like governance, financing and resourcing but to a lesser extent the function of services delivery and its heterogeneous nature. Moreover, most frameworks have weak links with routine information systems and national health priorities, especially in the context of high- and middle-income countries. This paper presents the development of a tool that responds to this context with the aim to create primary health care performance intelligence for the 53 countries of the WHO European Region. METHODS: The work builds-off of an existing systematic review on primary care and draws on priorities of current European health policies and available (inter)national information systems. Its development included: (i) reviewing and classifying features of primary care; (ii) constructing a set of tracer conditions; and (iii) mapping existing indicators in the framework resulting from (i). The analysis was validated through a series of reviews: in-person meetings with country-nominated focal points and primary care experts; at-distance expert reviews; and, preliminary testing with country informants. RESULTS: The resulting framework applies a performance continuum in the classical approach of structures-processes-outcomes spanning 6 domains - primary care structures, model of primary care, care contact, primary care outputs, health system outcomes, and health outcomes - that are further classified by 26 subdomains and 63 features of primary care. A care continuum was developed using a set of 12 tracer conditions. A total of 139 indicators were mapped to the classification, each with an identified data source to safeguard measurability. Individual indicator passports and a glossary of terms were developed to support the standardization of the findings. CONCLUSION: The resulting framework and suite of indicators, coined the Primary Health Care Impact, Performance and Capacity Tool (PHC-IMPACT), has the potential to be applied in Europe, closing the gap on existing data collection, analysis and use of performance intelligence for decision-making towards primary health care strengthening.

Optimising the secondary use of primary care prescribing data to improve quality of care: a qualitative analysis.

OBJECTIVES: To explore available data sources, secondary uses and key considerations for optimising the actionability of primary care prescribing data to improve quality of care in the Dutch context. DESIGN: An exploratory qualitative study was undertaken based on semi-structured interviews. We anchored our investigation around three tracer prescription types: antibiotics; benzodiazepines and opioids. Descriptive and explanatory themes were derived from interview data using thematic analysis. SETTING: Stakeholders were sampled from across the micro (clinical), meso (organisational) and macro (policy) contexts of the Dutch primary care system. PARTICIPANTS: The study involved 28 informants representing general practitioners (GPs), community pharmacists, regional chronic care networks (care groups), academia and research institutes, insurers, professional associations, electronic health record (EHR) vendors and national authorities. RESULTS: In the Netherlands, three main sources of data for improving prescribing in primary care are in use: clinical data in the EHRs of GP practices; pharmacy data in community pharmacy databases and claims data of insurers. While the secondary use of pharmacy and claims data is well-established across levels, the use of these data together with EHR data is limited. Important differences in the types of prescribing information needed by micro-meso-macro context are found, though the extent to which current indicators address these varies by prescription type. Five main themes were identified as areas for optimising data use: (1) measuring what matters, (2) increasing data linkages, (3) improving data quality, (4) facilitating data sharing and (5) optimising fit for use analysis. CONCLUSIONS: To make primary care prescribing data useful for improving quality, consolidated patient-specific data on the indication for a prescription and dispensed medicine, over time, is needed. In the Netherlands, the selection of indicators requires further prioritisation to better signal the appropriateness and long-term use of prescription drugs. Prioritising data linkages is critical towards more actionable use.

Describing and mapping scientific articles on alcohol globally for the period 2010-2021: a bibliometric analysis.

OBJECTIVES: To describe and map scientific literature related to alcohol consumption, its determinants, governance, harm and control policies by publication output, author affiliations, funding, countries of study and research themes. DESIGN: Bibliometric analysis using performance analysis and science mapping techniques. DATA SOURCES: Scientific articles. ELIGIBILITY CRITERIA: Indexed scientific articles published between 1 January 2010 and 31 December 2021 with an English abstract focused on alcohol consumption, its determinants, harms, governance and control policies. DATA EXTRACTION AND SYNTHESIS: Searches were run in Web of Science and PubMed. Performance metrics were analysed using descriptive statistics. Keywords were used for science mapping in a deductive approach to cluster articles by five main research themes. The 'policy response' theme was further analysed by six subthemes. RESULTS: 4553 articles were included in the analysis. Three out of four articles (3479/4553, 76.4%) were authored solely by authors affiliated with HIC institutions. One in five articles (906/4553, 19.9%) had at least one author affiliated to an institution from an upper-middle-income, middle-income or low-income country context. Governments, followed by research institutions, were the predominant funding source. Half (53.1%) studied a single country and, of these, 77.0% were high-income countries (HICs). Australia, USA and UK were the most studied countries, together accounting for 44.9% (975/2172) of country-specific articles. Thematically, 'consumption' was most studied, and 'alcohol determinants', least. 'Policy response' articles were predominately conducted in HIC contexts. CONCLUSIONS: Although the attributable harm of alcohol is known to affect more significantly lower-income and middle-income countries, scientific publications primarily report on HIC contexts by authors from HICs. Research themes reflect known cost-effective policy actions, though skewed towards HICs and a focus on consumption. The implementation of context-specific alcohol control policies requires addressing the determinants of the uneven geographical and thematic distribution of research.

The experiences of 33 national COVID-19 dashboard teams during the first year of the pandemic in the World Health Organization European Region: A qualitative study.

Background: Governments across the World Health Organization (WHO) European Region have prioritised dashboards for reporting COVID-19 data. The ubiquitous use of dashboards for public reporting is a novel phenomenon. Objective: This study explores the development of COVID-19 dashboards during the first year of the pandemic and identifies common barriers, enablers and lessons from the experiences of teams responsible for their development. Methods: We applied multiple methods to identify and recruit COVID-19 dashboard teams, using a purposive, quota sampling approach. Semi-structured group interviews were conducted from April to June 2021. Using elaborative coding and thematic analysis, we derived descriptive and explanatory themes from the interview data. A validation workshop was held with study participants in June 2021. Results: Eighty informants participated, representing 33 national COVID-19 dashboard teams across the WHO European Region. Most dashboards were launched swiftly during the first months of the pandemic, February to May 2020. The urgency, intense workload, limited human resources, data and privacy constraints and public scrutiny were common challenges in the initial development stage. Themes related to barriers or enablers were identified, pertaining to the pre-pandemic context, pandemic itself, people and processes and software, data and users. Lessons emerged around the themes of simplicity, trust, partnership, software and data and change. Conclusions: COVID-19 dashboards were developed in a learning-by-doing approach. The experiences of teams reveal that initial underpreparedness was offset by high-level political endorsement, the professionalism of teams, accelerated data improvements and immediate support with commercial software solutions. To leverage the full potential of dashboards for health data reporting, investments are needed at the team, national and pan-European levels.

Exploring the actionability of healthcare performance indicators for quality of care: a qualitative analysis of the literature, expert opinion and user experience.

BACKGROUND: This study explores the meaning of actionable healthcare performance indicators for quality of care-related decisions. To do so, we analyse the constructs of fitness for purpose and fitness for use across healthcare systems and in practice based on the literature, expert opinion and user experience. METHODS: A multiphase qualitative study was undertaken. Phases included a literature review, a first round of one-on-one interviews with a panel of academics and thought leaders in the field (n=16), and a second round of interviews with real-world users of performance indicators (n=16). Thematic analysis was conducted between phases in order to triangulate findings in a stepwise process. RESULTS: Common uses of healthcare performance indicators were differentiated within micro-meso-macro contexts of healthcare systems. Each purpose of use signals different decision-making tasks, and in effect information needs. An indicator's fitness for use can be appraised by three clusters of considerations: methodological, contextual and managerial. Methodological considerations gauge an indicator's perceived importance, engagement potential, interpretability, standardisation, feasibility of remedial actions, alignment to care models and sensitivity to change. Information infrastructure, system governance, workforce capacity and learning culture were found as enabling contextual considerations. Managerial considerations influencing an indicator's use in practice were found to span the selection of indicators, data collection, analysis, display of results and delivery of information to decision-makers. CONCLUSION: The actionability of a healthcare performance indicator should be appraised by its alignment with the intended purpose of use beyond aggregate healthcare system levels, in combination with the extent to which methodological, contextual and managerial fitness for use considerations are met. Striking a better balance between the importance weighted to an indicator's statistical merits and emphasis put to its fitness for purpose and use is needed for indicators that are ultimately actionable for quality of care-related decision-making.

Review of 128 quality of care mechanisms: A framework and mapping for health system stewards.

Health system stewards have the critical task to identify quality of care deficiencies and resolve underlying system limitations. Despite a growing evidence-base on the effectiveness of certain mechanisms for improving quality of care, frameworks to facilitate the oversight function of stewards and the use of mechanisms to improve outcomes remain underdeveloped. This review set out to catalogue a wide range of quality of care mechanisms and evidence on their effectiveness, and to map these in a framework along two dimensions: (i) governance subfunctions; and (ii) targets of quality of care mechanisms. To identify quality of care mechanisms, a series of searches were run in Health Systems Evidence and PubMed. Additional grey literature was reviewed. A total of 128 quality of care mechanisms were identified. For each mechanism, searches were carried out for systematic reviews on their effectiveness. These findings were mapped in the framework defined. The mapping illustrates the range and evidence for mechanisms varies and is more developed for some target areas such as the health workforce. Across the governance sub-functions, more mechanisms and with evidence of effectiveness are found for setting priorities and standards and organizing and monitoring for action. This framework can support system stewards to map the quality of care mechanisms used in their systems and to uncover opportunities for optimization backed by systems thinking.

Improving clinical practice in primary care for the prevention and control of noncommunicable diseases: a multi-actor approach to two regional pilot projects in Kazakhstan.

Improving access to quality services is integral to achieving better outcomes for noncommunicable diseases (NCDs). In Kazakhstan, like other countries with historically centralized governance models, key to improving quality is instilling a common and shared understanding of the roles and responsibilities in correspondence with the multifaceted nature of quality of care. This review details the experience of two pilot projects implemented in Kazakhstan's regions of Kyzylorda and Mangystau over a three-year period with the aim to improve clinical practice through a multi-actor, multi-intervention approach. Adopting a health system perspective, the pilots, by design, introduced interventions targeting four actors: policy-makers; health facility managers; health practitioners and patients. The review draws on the following sources of data: rapid baseline assessments; implementation plans, curriculums and other pilot-related material; a mid-way joint implementation meeting; intervention-specific evaluations; and a final external evaluation. The multi-actor, multi-intervention approach to the pilot projects showed some improvements to service outputs, in particular for cardiovascular disease (CVD) risk assessment and decreases in hospitalization rates for hypertension. The pilot projects also illustrated progress in working towards a shared understanding of the different roles of actors for improving quality of care, appreciating the complementarity of individual actors working towards improved population health and in establishing a culture of learning through the exchange of ideas and practices. The importance of responsibility across health system actors for outcomes is vital for the NCD agenda. This approach offers relevant policy lessons for similar centralized governance systems.

Health workforce governance: Processes, tools and actors towards a competent workforce for integrated health services delivery.

A competent health workforce is a vital resource for health services delivery, dictating the extent to which services are capable of responding to health needs. In the context of the changing health landscape, an integrated approach to service provision has taken precedence. For this, strengthening health workforce competencies is an imperative, and doing so in practice hinges on the oversight and steering function of governance. To aid health system stewards in their governing role, this review seeks to provide an overview of processes, tools and actors for strengthening health workforce competencies. It draws from a purposive and multidisciplinary review of literature, expert opinion and country initiatives across the WHO European Region's 53 Member States. Through our analysis, we observe distinct yet complementary roles can be differentiated between health services delivery and the health system. This understanding is a necessary prerequisite to gain deeper insight into the specificities for strengthening health workforce competencies in order for governance to rightly create the institutional environment called for to foster alignment. Differentiating between the contribution of health services and the health system in the strengthening of health workforce competencies is an important distinction for achieving and sustaining health improvement goals.

A review of health governance: definitions, dimensions and tools to govern.

BACKGROUND: The premise that good governance will ultimately lead to better health outcomes has been central to the proliferation of work in this area over the past decade. OBJECTIVE: To consolidate and align literature on governance by presenting an overview of efforts to define, describe and operationalize the health governance function. METHODS: A targeted review of governance literature. RESULTS: (1) A variety of terms have been assigned to precede health governance definitions. These terms commonly describe governance ideals (e.g. good, democratic) or characteristics of the organization of actors in governance arrangements (e.g. hierarchical, networked). (2) Dimensions of governance are defined from different perspectives and in varied combinations, capturing values, sub-functions and/or outcomes of governance. (3) Tools used to govern remain to be cataloged, however, measures can be aligned according to dimensions of governance or their ability to create specific relationships between actors. DISCUSSION: Resolving the conceptual confusion around health governance requires recognition for the differences in the premise and approaches taken to defining governance, as well as specifying core dimensions and aligning applicable tools. CONCLUSION: Despite a growing literature base, a concerted effort is needed for a more accessible understanding of health governance that is both practical at present and actionable for policy-makers.