Health system performance assessment: embedding resilience through performance intelligence.

Health systems around the world are facing challenges in achieving their goals. In the wake of the coronavirus disease pandemic, the need for resilient health systems has become even more apparent. This article argues that embedding resilience into health system performance assessment (HSPA) frameworks can be a valuable approach for improving health system performance. This perspective examines key challenges threatening health systems and makes a case for the continued relevance of HSPA by embedding resilience-related performance intelligence.

Status of the health information system in Ireland and its fitness to support health system performance assessment: a multimethod assessment based on stakeholder involvement.

BACKGROUND: Between 2019 and 2021, the first Irish health system performance assessment (HSPA) framework was developed. As routinely collected health data are necessary to continuously populate indicators of an HSPA framework, a purpose-driven assessment of the health information system (HIS) in Ireland and its fitness to support the implementation of an HSPA framework was conducted. This study reports on the status of the Irish HIS through a multimethod assessment based on continuous broad stakeholder involvement. METHODS: Between May and November 2020, over 50 informants were engaged in individual and group interviews and stakeholder consultation workshops as part of the HIS assessment process. Descriptive themes and high-level data availability heatmaps were derived from interview and workshop data using thematic analysis. Indicator "passports" for the HSPA framework were populated during stakeholder consultation workshops and analysed using univariate descriptive statistics. RESULTS: The HIS in Ireland was able to provide administrative, survey and registry-based data for public sector acute care services, focusing on structure, process and output metrics. Significant data availability gaps, most notably from primary care, private hospitals and community care, were reported, with little availability of electronic health record and people-reported data. Data on outcome metrics were mostly missing, as were linkage possibilities across datasets for care pathway monitoring. The COVID-19 pandemic highlighted the national HIS's shortcomings but also the capacity for rapid development and improvement. CONCLUSIONS: A tailor-made assessment of the HIS in Ireland, involving a broad set of relevant stakeholders, revealed strengths, weaknesses and areas for improvement in the Irish health data landscape. It also contributed to the development of a national HSPA framework and momentum to further strengthen data infrastructure and governance, while working towards a more data-driven and person-centred healthcare system. This work demonstrates the utility of an inclusive HIS assessment process and is applicable beyond Ireland, where this case study was conducted.

A Web Interface for Antibiotic Prescription Recommendations in Primary Care: User-Centered Design Approach.

BACKGROUND: Antibiotic misuse is a serious public health problem worldwide. National health authorities release clinical practice guidelines (CPGs) to guide general practitioners (GPs) in their choice of antibiotics. However, despite the large-scale dissemination of CPGs, GPs continue to prescribe antibiotics that are not recommended as first-line treatments. This nonadherence to recommendations may be due to GPs misunderstanding the CPGs. A web interface displaying antibiotic prescription recommendations and their justifications could help to improve the comprehensibility and readability of CPGs, thereby increasing the adoption of recommendations regarding antibiotic treatment. OBJECTIVE: This study aims to design and evaluate a web interface for antibiotic prescription displaying both the recommended antibiotics and their justifications in the form of antibiotic properties. METHODS: A web interface was designed according to the same principles as e-commerce interfaces and was assessed by 117 GPs. These GPs were asked to answer 17 questions relating to the usefulness, user-friendliness, and comprehensibility and readability of the interface, and their satisfaction with it. Responses were recorded on a 4-point Likert scale (ranging from "absolutely disagree" to "absolutely agree"). At the end of the evaluation, the GPs were allowed to provide optional, additional free comments. RESULTS: The antibiotic prescription web interface consists of three main sections: a clinical summary section, a filter section, and a recommended antibiotics section. The majority of GPs appreciated the clinical summary (90/117, 76.9%) and filter (98/117, 83.8%) sections, whereas 48.7% (57/117) of them reported difficulty reading some of the icons in the recommended antibiotics section. Overall, 82.9% (97/117) of GPs found the display of drug properties useful, and 65.8% (77/117) reported that the web interface improved their understanding of CPG recommendations. CONCLUSIONS: The web interface displaying antibiotic recommendations and their properties can help doctors understand the rationale underlying CPG recommendations regarding antibiotic treatment, but further improvements are required before its implementation into a clinical decision support system.

Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study.

BACKGROUND: Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration. OBJECTIVE: We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time. METHODS: We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point. RESULTS: Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards. CONCLUSIONS: Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

Features Constituting Actionable COVID-19 Dashboards: Descriptive Assessment and Expert Appraisal of 158 Public Web-Based COVID-19 Dashboards.

BACKGROUND: Since the outbreak of COVID-19, the development of dashboards as dynamic, visual tools for communicating COVID-19 data has surged worldwide. Dashboards can inform decision-making and support behavior change. To do so, they must be actionable. The features that constitute an actionable dashboard in the context of the COVID-19 pandemic have not been rigorously assessed. OBJECTIVE: The aim of this study is to explore the characteristics of public web-based COVID-19 dashboards by assessing their purpose and users ("why"), content and data ("what"), and analyses and displays ("how" they communicate COVID-19 data), and ultimately to appraise the common features of highly actionable dashboards. METHODS: We conducted a descriptive assessment and scoring using nominal group technique with an international panel of experts (n=17) on a global sample of COVID-19 dashboards in July 2020. The sequence of steps included multimethod sampling of dashboards; development and piloting of an assessment tool; data extraction and an initial round of actionability scoring; a workshop based on a preliminary analysis of the results; and reconsideration of actionability scores followed by joint determination of common features of highly actionable dashboards. We used descriptive statistics and thematic analysis to explore the findings by research question. RESULTS: A total of 158 dashboards from 53 countries were assessed. Dashboards were predominately developed by government authorities (100/158, 63.0%) and were national (93/158, 58.9%) in scope. We found that only 20 of the 158 dashboards (12.7%) stated both their primary purpose and intended audience. Nearly all dashboards reported epidemiological indicators (155/158, 98.1%), followed by health system management indicators (85/158, 53.8%), whereas indicators on social and economic impact and behavioral insights were the least reported (7/158, 4.4% and 2/158, 1.3%, respectively). Approximately a quarter of the dashboards (39/158, 24.7%) did not report their data sources. The dashboards predominately reported time trends and disaggregated data by two geographic levels and by age and sex. The dashboards used an average of 2.2 types of displays (SD 0.86); these were mostly graphs and maps, followed by tables. To support data interpretation, color-coding was common (93/158, 89.4%), although only one-fifth of the dashboards (31/158, 19.6%) included text explaining the quality and meaning of the data. In total, 20/158 dashboards (12.7%) were appraised as highly actionable, and seven common features were identified between them. Actionable COVID-19 dashboards (1) know their audience and information needs; (2) manage the type, volume, and flow of displayed information; (3) report data sources and methods clearly; (4) link time trends to policy decisions; (5) provide data that are "close to home"; (6) break down the population into relevant subgroups; and (7) use storytelling and visual cues. CONCLUSIONS: COVID-19 dashboards are diverse in the why, what, and how by which they communicate insights on the pandemic and support data-driven decision-making. To leverage their full potential, dashboard developers should consider adopting the seven actionability features identified.

Engaging citizens in the development of a health system performance assessment framework: a case study in Ireland.

BACKGROUND: The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework which should reflect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reflects on the added value of using a citizen panel in the development of an HSPA framework. METHODS: A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panellists received lay-language preparatory materials before the meeting. Panellists used a three-colour scheme to signal the importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups. RESULTS: The citizen panel generated 249 health performance measures ranging across 13 domains. Top-ranking domains to the citizen panel (people-centredness, coordination of care, and coverage) were less prioritized by the other panels; domains less prioritized by the citizen panel, such as accessibility, responsiveness, efficiency, and effectiveness, were of higher priority in the other panels. Citizen panellists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative. CONCLUSIONS: The priorities of the citizen panel were accounted for during all phases of developing the HSPA framework. This was possible by adopting an inclusive development process and by engaging citizens early on. Citizen engagement in HSPA development is essential for realizing citizen-driven healthcare system performance and generating trust and ownership in performance intelligence. Future research could expand the use of citizen panels to assess, monitor, and report on the performance of healthcare systems.

QUALITY ASSURANCE OF REGISTRIES FOR HEALTH TECHNOLOGY ASSESSMENT.

OBJECTIVES: The aim of this study was to identify guidelines and assessment tools used by health technology agencies for quality assurance of registries and investigate the current use of registry data by HTA organizations worldwide. METHODS: As part of a European Network for Health Technology Assessment Joint Action work package, we undertook a literature search and sent a questionnaire to all partner organizations on the work package and all organizations listed in the International Society for Pharmaco-economics and Outcomes Research directory. RESULTS: We identified thirteen relevant documents relating to quality assurance of registries. We received fifty-five responses from organizations representing twenty-one different countries, a response rate of 40.5 percent (43/110). Many agencies, particularly in Europe, are already drawing on a range of registries to provide data for their HTA. Less than half, however, use criteria or standards to assess the quality of registry data. Nearly all criteria or standards in use have been internally defined by organizations rather than referring to those produced by an external body. A comparison of internal and external standards identified consistency in several quality dimensions, which can be used as a starting point for the development of a standardized tool. CONCLUSION: The use of registry data is more prevalent than expected, strengthening the need for a standardized registry quality assessment tool. A user-friendly tool developed in conjunction with stakeholders will support the consistent application of approved quality standards, and reassure critics who have traditionally considered registry data to be unreliable.

Assessment of the eutrophication status at Mediterranean sub-basin scale, within the European Marine Strategy Framework Directive.

The aim of this work is to define harmonized reference conditions and assessment thresholds for selected criteria elements of the Marine Strategy Framework Directive (MSFD) Descriptor 5 (Eutrophication) in the Eastern Mediterranean Sea and to test if a tool for integrated assessment of the status of marine systems can be used as a common methodological approach. In this frame, we tested two statistical approaches in order to set threshold values for four criteria of Descriptor 5: nutrients, chlorophyll a, transparency and dissolved oxygen in the bottom waters. It is noteworthy that this work revealed the need to apply common procedures in data treatment and assessment evaluation. This is the first attempt to set common methods for the assessment of eutrophication in the Eastern Mediterranean, which is essential in marine environments, especially those shared by several countries. To this end, we have applied common criteria and metrics and established thresholds "Good" and "Moderate" for nutrients, chlorophyll a, transparency and dissolved oxygen in the bottom waters for the different Water Types of the Adriatic and Aegean Seas (I, II, IIIW, IIIE), based on datasets provided by Italy, Slovenia, Croatia and Greece. The selected criteria elements were common for all countries, providing a unified approach to GES assessment of two case study areas: the Adriatic Sea and the Saronikos Gulf. Dissolved Inorganic Nitrogen (DIN) threshold values of 15.6, 6.85, 1.61 and 2.11 µmol L-1 were set for the Water Types I, II, IIIW and IIIE, respectively. We also tested if an aggregation tool for GES assessment, such as Nested Environmental status Assessment Tool (NEAT), could be used as a common methodological approach. The comparison of NEAT with TRIX showed good comparability. In this end, NEAT can be used as a useful and much needed assessment tool for assessing eutrophication status of the marine.

Data-Driven Collaboration between Hospitals and Other Healthcare Organisations in Europe During the COVID-19 Pandemic: An Explanatory Sequential Mixed-Methods Study among Mid-Level Hospital Managers.

Introduction: Data and digital infrastructure drive collaboration and help develop integrated healthcare systems and services. COVID-19 induced changes to collaboration between healthcare organisations, which previously often happened in fragmented and competitive ways. New collaborative practices relied on data and were crucial in managing coordinated responses to the pandemic. In this study, we explored data-driven collaboration between European hospitals and other healthcare organisations in 2021 by identifying common themes, lessons learned and implications going forward. Methods: Study participants were recruited from an existing Europe-wide community of mid-level hospital managers. For data collection, we ran an online survey, conducted multi-case study interviews and organised webinars. Data were analysed using descriptive statistics, thematic analysis and cross-case synthesis. Results: Mid-level hospital managers from 18 European countries reported an increase in data exchange between healthcare organisations during the COVID-19 pandemic. Data-driven collaborative practices were goal-oriented and focused on the optimisation of hospitals' governance functions, innovation in organisational models and improvements to data infrastructure. This was often made possible by temporarily overcoming system complexities, which would otherwise hinder collaboration and innovation. Sustainability of these developments remains a challenge. Discussion: Mid-level hospital managers form a huge potential of reacting and collaborating when needed, including rapidly setting up novel partnerships and redefining established processes. Major post-COVID unmet medical needs are linked to hospital care provision, including diagnostic and therapeutic backlogs. Tackling these will require rethinking of the position of hospitals within healthcare systems, including their role in care integration. Conclusion: Learning from COVID-19-induced developments in data-driven collaboration between hospitals and other healthcare organisations is important to address systemic barriers, sustain resilience and further build transformative capacity to help build better integrated healthcare systems.

Enablers and barriers to the secondary use of health data in Europe: general data protection regulation perspective.

BACKGROUND: The General Data Protection Regulation is a regulation in EU law on data protection and privacy in the European Union. We aimed to provide an overview of the General Data Protection Regulation (GDPR) enablers and barriers to the secondary use of health data in Europe from the research we conducted in the Joint Action InfAct (Information for Action!) WP10 Assessing and piloting interoperability for public health policy, as well as to provide an example of a national-level case study on experiences with secondary use of health data and GDPR on an example of the Austrian COVID-19 data platform. METHODS: We have identified a number of European initiatives, projects and organizations that have dealt with cross-border health data sharing, linkage and management by desk research and we conducted 17 semi-structured in-depth interviews and analyzed the interview transcripts by framework analysis. RESULTS: GDPR was seen as an enabler to the secondary use of health data in Europe when it comes to user rights over their data, pre-existing laws regarding data privacy and data sharing, sharing anonymized statistics, developing new data analysis approaches, patients` trust towards dealing with their health data and transparency. GDPR was seen as a barrier to the secondary use of health data in Europe when it comes to identifiable and individual-level data, data sharing, time needed to complete the process, workload increase, differences with local legal legislations, different (and stricter) interpretations and access to data. CONCLUSION: The results of our analysis show that GDPR acts as both an enabler and a barrier for the secondary use of health data in Europe. More research is needed to better understand the effects of GDPR on the secondary use of health data which can serve as a basis for future changes in the regulation.

Use of Performance Data by Mid-Level Hospital Managers in Ontario: Results of a Province-Wide Survey and a Comparison with Hospital Managers in Europe.

This paper provides insights into the use of performance data by middle managerial staff in Ontario hospitals in 2019 and compares the results to a study conducted in Europe in the same year. A total of 236 managers working in 61 hospitals across Ontario provided responses to the survey. Compared to their European colleagues, Ontario respondents self-assessed using significantly more performance data for managerial decision making. The use of performance data in Ontario was mostly motivated by external accountability requirements, followed by internal quality improvement efforts. Ontario managers also reported accessibility, appropriateness and timeliness of data and human resources and engagement as the biggest barriers to further performance data utilization. Comparative studies, such as the one this paper is based on, provide the foundation for drawing lessons across jurisdictions. This paper also affirms the importance of hospital middle management in moving from quality assurance to quality improvement efforts and developing sustainable learning healthcare organizations and systems.

The experiences of 33 national COVID-19 dashboard teams during the first year of the pandemic in the World Health Organization European Region: A qualitative study.

Background: Governments across the World Health Organization (WHO) European Region have prioritised dashboards for reporting COVID-19 data. The ubiquitous use of dashboards for public reporting is a novel phenomenon. Objective: This study explores the development of COVID-19 dashboards during the first year of the pandemic and identifies common barriers, enablers and lessons from the experiences of teams responsible for their development. Methods: We applied multiple methods to identify and recruit COVID-19 dashboard teams, using a purposive, quota sampling approach. Semi-structured group interviews were conducted from April to June 2021. Using elaborative coding and thematic analysis, we derived descriptive and explanatory themes from the interview data. A validation workshop was held with study participants in June 2021. Results: Eighty informants participated, representing 33 national COVID-19 dashboard teams across the WHO European Region. Most dashboards were launched swiftly during the first months of the pandemic, February to May 2020. The urgency, intense workload, limited human resources, data and privacy constraints and public scrutiny were common challenges in the initial development stage. Themes related to barriers or enablers were identified, pertaining to the pre-pandemic context, pandemic itself, people and processes and software, data and users. Lessons emerged around the themes of simplicity, trust, partnership, software and data and change. Conclusions: COVID-19 dashboards were developed in a learning-by-doing approach. The experiences of teams reveal that initial underpreparedness was offset by high-level political endorsement, the professionalism of teams, accelerated data improvements and immediate support with commercial software solutions. To leverage the full potential of dashboards for health data reporting, investments are needed at the team, national and pan-European levels.

Why, what and how do European healthcare managers use performance data? Results of a survey and workshop among members of the European Hospital and Healthcare Federation.

Objective of this study was to better understand the use of performance data for evidence-based decision-making by managers in hospitals and other healthcare organisations in Europe in 2019. In order to explore why, what and how performance data is collected, reported and used, we conducted a cross-sectional study based on a self-reported online questionnaire and a follow-up interactive workshop. Our study population were participants of a pan-European professional Exchange Programme and their hosts (n = 125), mostly mid-level hospital managers. We found that a substantial amount of performance data is collected and reported, but could be utilised better for decision-making purposes. Motivation to collect and report performance data is equally internal and external, for improvement as well as for accountability purposes. Benchmarking between organisations is recognised as being important but is still underused. A plethora of different data sources are used, but more should be done on conceptualising, collecting, reporting and using patient-reported data. Managers working for privately owned organisations reported greater use of performance data than those working for public ones. Strategic levels of management use performance data more for justifying their decisions, while managers on operational and clinical levels use it more for day-to-day decision-making. Our study showed that, despite the substantial and increasing use of performance data for evidence-based management, there is room and need to further explore and expand its role in strategic decision-making and supporting a shift in healthcare from organisational accountability towards the model of learning organisations.

Addressing the Data Linking Challenges.

BACKGROUND: PARENT JA (cross-border Patient Registries iNiTiative Joint Action), a joint EU and Member States project, has conducted a research among EU patient registries aimed at gathering information on the registries' interoperability readiness. Leaning on the information and data collected through the previous PARENT JA research, this study aims to provide more detailed view into the registry holders' practical challenges with data linking. Since the studies which dealt with patient data exchange have often neglected the registry holders' performance of data exchange, we wanted to put a spotlight on various EU registry holders' practices and operations, aiming to detect their needs and concerns in the process of running an interoperable registry. The focus of this study was identifying the main practices and challenges in patient registries interoperability improvement. METHODS: The basis for this analysis were the data collected in the series of structured interviews. The size of the interview sample was 13 patient registries, each from a different EU country. The structured interview consisted of nine questions and was conducted in two parts: oral and written. The answers were analysed using open coding. RESULTS: Results are interpreted in the context of the six main themes that emerged through a comprehensive analysis. (1) Examples of data exchange: The most common reported data exchange practices were seen only as a way to achieve the most immediate needs and interests of the individual registries. (2) Awareness and use of international standards: International data and clinical standards were not widely used by the interviewed registries. (3) Use of data models and formats: In the area of data models and formats there is no universally used practice. (4) Data request protocols and procedures: Procedures and protocols varied, mostly depending on the national legal systems in which the patient registries operated. (5) Data security and integrity: Security of personal data was a universal concern for all registry holders that were interviewed; identifiable individual data was shared only in one case. (6) Opportunities and challenges of registry interoperability: most registry holders responded that their registries were well prepared for interoperability practices and that data exchange has never been their primary operative concern. CONCLUSIONS: Most of the difficulties regarding data linking were not necessarily associated with technical issues, which registry holders listed outright. Our analysis showed that the lack of interoperability came as a result of organizational or legal constraints that made the registries unable to process and conduct data linking quickly and effectively with other sources.

Self-Organizing Maps-based ocean currents forecasting system.

An ocean surface currents forecasting system, based on a Self-Organizing Maps (SOM) neural network algorithm, high-frequency (HF) ocean radar measurements and numerical weather prediction (NWP) products, has been developed for a coastal area of the northern Adriatic and compared with operational ROMS-derived surface currents. The two systems differ significantly in architecture and algorithms, being based on either unsupervised learning techniques or ocean physics. To compare performance of the two methods, their forecasting skills were tested on independent datasets. The SOM-based forecasting system has a slightly better forecasting skill, especially during strong wind conditions, with potential for further improvement when data sets of higher quality and longer duration are used for training.