Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework.

BACKGROUND: The science of measuring and reporting on the performance of healthcare systems is rapidly evolving. In the past decade, across many jurisdictions, organisations tasked with monitoring progress towards reform targets have broadened their purview to take a more system-functioning approach. Their aim is to bring clarity to performance assessment, using relevant and robust concepts - and avoiding reductionist measures - to build a whole-of-system view of performance. Existing performance frameworks are not fully aligned with these developments. METHODS: An eight stage process to develop a conceptual framework incorporated literature review, mapping, categorisation, integration, synthesis and validation of performance constructs that have been used by organisations and researchers in order to assess, reflect and report on healthcare performance. RESULTS: A total of 19 performance frameworks were identified and included in the review. Existing frameworks mostly adopted either a logic model (inputs, outputs and outcomes), a functional, or a goal-achievement approach. The mapping process identified 110 performance terms and concepts. These were integrated, synthesised and resynthesised to produce a framework that features 12 derived constructs reflecting combinations of patients' needs and expectations; healthcare resources and structures; receipt and experience of healthcare services; healthcare processes, functions and context; and healthcare outcomes. The 12 constructs gauge performance in terms of coverage, accessibility, appropriateness, effectiveness, safety, productivity, efficiency, impact, sustainability, resilience, adaptability and equity. They reflect four performance perspectives (patient, population, delivery organisation and system). CONCLUSIONS: Internationally, healthcare systems and researchers have used a variety of terms to categorise indicators of healthcare performance, however few frameworks are based on a theoretically-based conceptual underpinning. The proposed framework incorporates a manageable number of performance domains that together provide a comprehensive assessment, as well as conceptual and operational clarity and coherence that support multifaceted measurement systems for healthcare.

Envisioning the future of clinical analytics: a modified Delphi process in New South Wales, Australia.

BACKGROUND: Clinical analytics is a rapidly developing area of informatics and knowledge mobilisation which has huge potential to improve healthcare in the future. It is widely acknowledged to be a powerful mediator of clinical decision making, patient-centred care and organisational learning. As a result, healthcare systems require a strategic foundation for clinical analytics that is sufficiently directional to support meaningful change while flexible enough to allow for iteration and responsiveness to context as change occurs. METHODS: In New South Wales, the most populous state in Australia, the Clinical Analytics Working Group was charged with developing a five-year vision for the public health system. A modified Delphi process was undertaken to elicit expert views and to reach a consensus. The process included a combination of face-to-face workshops, traditional Delphi voting via email, and innovative, real-time iteration between text re-formulation and voting until consensus was reached. The six stage process engaged 35 experts - practising clinicians, patients and consumers, managers, policymakers, data scientists and academics. RESULTS: The process resulted in the production of 135 ideas that were subsequently synthesised into 23 agreed statements and encapsulated in a single page (456 word) narrative. CONCLUSION: The visioning process highlighted three key perspectives (clinicians, patients and managers) and the need for synchronous (during the clinical encounter) and asynchronous (outside the clinical encounter) clinical decision support and reflective practice tools; the use of new and multiple data sources and communication formats; and the role of research and education.

Barriers to accessing primary health care: comparing Australian experiences internationally.

Most highly developed economies have embarked on a process of primary health care (PHC) transformation. To provide evidence on how nations vary in terms of accessing PHC, the aim of this study is to describe the extent to which barriers to access were experienced by adults in Australia compared with other countries. Communities participating in an international research project on PHC access interventions were engaged to prioritise questions from the 2013 Commonwealth Fund International Health Policy Survey within a framework that conceptualises access across dimensions of approachability, acceptability, availability, affordability and appropriateness. Logistic regression models, with barriers to access as outcomes, found measures of availability to be a problematic dimension in Australia; 27% of adults experienced difficulties with out-of-hours access, which was higher than 5 of 10 comparator countries. Although less prevalent, affordability was also perceived as a substantial barrier; 16% of Australians said they had forgone health care due to cost in the previous year. After adjusting for age and health status, this barrier was more common in Australia than 7 of 10 countries. Findings of this integrated assessment of barriers to access offer insights for policymakers and researchers on Australia's international performance in this crucial PHC domain.

Acceptability and timing considerations when administering patient-reported outcome measures (PROMs) among people with chronic health conditions who are culturally and linguistically diverse (CALD): a qualitative study protocol.

INTRODUCTION: Patient-reported outcome measures (PROMs) are validated and standardised questionnaires that capture patients' own reports of their symptoms, functioning and well-being. PROMs can facilitate communication between patients and clinicians, reduce symptom burden, enhance quality of life and inform health service re-design. We aim to determine the acceptability of PROMs and the preferred timing of PROM completion in New South Wales (NSW) at the point of care, facilitated by the Health Outcomes and Patient Experiences (HOPE) platform. METHODS AND ANALYSIS: Semi-structured interviews with patients (~50-75, sampling across seven language groups and seven clinical cohorts), carers (~10-20) and clinicians (~18) enrolled in HOPE will be conducted via videoconference, telephone or in person. Participants will be asked questions about (1) what makes PROMs acceptable for use in chronic disease management (2) when patients would prefer to complete PROMs and when clinicians would like to use PROMs for clinical decision-making and (3) factors that impede the acceptability of PROMs for culturally and linguistically diverse patients. Interviews will be analysed using a reflexive thematic approach, guided by Normalisation Process Theory. ETHICS AND DISSEMINATION: Ethics approval has been obtained from the Sydney Local Health District Human Research Ethics Committee (SLHD HREC, Study Protocol #X24-0138). Results will be published in appropriate peer-reviewed journals, presented at conferences, disseminated to participants in the form of a plain language summary, and widely disseminated to consumer groups and professional stakeholders.

Horizon scanning, rapid reviews and living evidence to support decision-making: lessons from the work of the Critical Intelligence Unit in New South Wales, Australia during the COVID-19 pandemic.

The COVID-19 pandemic has seen an increase in rapidly disseminated scientific evidence and highlighted that traditional evidence synthesis methods, such as time and resource intensive systematic reviews, may not be successful in responding to rapidly evolving policy and practice needs. In New South Wales (NSW) Australia, the Critical Intelligence Unit (CIU) was established early in the pandemic and acted as an intermediary organisation. It brought together clinical, analytical, research, organisational and policy experts to provide timely and considered advice to decision-makers. This paper provides an overview of the functions, challenges and future implications of the CIU, particularly the Evidence Integration Team. Outputs from the Evidence Integration Team included a daily evidence digest, rapid evidence checks and living evidence tables. These products have been widely disseminated and used to inform policy decisions in NSW, making valuable impacts. Changes and innovations to evidence generation, synthesis and dissemination in response to the COVID-19 pandemic provide an opportunity to shift the way evidence is used in future. The experience and methods of the CIU have potential to be adapted and applied to the broader health system nationally and internationally.

Audit and feedback to reduce unwarranted clinical variation at scale: a realist study of implementation strategy mechanisms.

BACKGROUND: Unwarranted clinical variation in hospital care includes the underuse, overuse, or misuse of services. Audit and feedback is a common strategy to reduce unwarranted variation, but its effectiveness varies widely across contexts. We aimed to identify implementation strategies, mechanisms, and contextual circumstances contributing to the impact of audit and feedback on unwarranted clinical variation. METHODS: Realist study examining a state-wide value-based healthcare program implemented between 2017 and 2021 in New South Wales, Australia. Three initiatives within the program included audit and feedback to reduce unwarranted variation in inpatient care for different conditions. Multiple data sources were used to formulate the initial audit and feedback program theory: a systematic review, realist review, program document review, and informal discussions with key program stakeholders. Semi-structured interviews were then conducted with 56 participants to refute, refine, or confirm the initial program theories. Data were analysed retroductively using a context-mechanism-outcome framework for 11 transcripts which were coded into the audit and feedback program theory. The program theory was validated with three expert panels: senior health leaders (n = 19), Agency for Clinical Innovation (n = 11), and Ministry of Health (n = 21) staff. RESULTS: The program's audit and feedback implementation strategy operated through eight mechanistic processes. The strategy worked well when clinicians (1) felt ownership and buy-in, (2) could make sense of the information provided, (3) were motivated by social influence, and (4) accepted responsibility and accountability for proposed changes. The success of the strategy was constrained when the audit process led to (5) rationalising current practice instead of creating a learning opportunity, (6) perceptions of unfairness and concerns about data integrity, 7) development of improvement plans that were not followed, and (8) perceived intrusions on professional autonomy. CONCLUSIONS: Audit and feedback strategies may help reduce unwarranted clinical variation in care where there is engagement between auditors and local clinicians, meaningful audit indicators, clear improvement plans, and respect for clinical expertise. We contribute theoretical development for audit and feedback by proposing a Model for Audit and Feedback Implementation at Scale. Recommendations include limiting the number of audit indicators, involving clinical staff and local leaders in feedback, and providing opportunities for reflection.

Chartbook: shining a light on the quality of healthcare in Canada: what can be learned to catalyze improvements in healthcare quality?

This paper provides a reflection on the findings of Canada's first-ever chartbook on the quality of healthcare in Canada. Quality of Healthcare in Canada: A Chartbook was published in 2010 by the Canadian Health Services Research Foundation in partnership with the Canadian Institute for Health Information and the Canadian Patient Safety Institute, and with support from Statistics Canada. This paper, by the chartbook authors (Sutherland and Leatherman) and colleagues (Law, Verma and Petersen), presents selected key findings and lessons from the chartbook and aims to serve as a catalyst for ideas and discussion in the papers that follow. The chartbook identified a lack of common language and indicators on quality across Canada's provinces and territories, underscoring the need to create and coordinate core measures. The Canadian chartbook and this issue of Healthcare Papers provide an update on the existing quality measures and the state of healthcare quality in Canada, and create the opportunity for jurisdictions to learn from one another and to contemplate the steps required to improve quality across the country.

From data to practice change - exploring new territory for atlases of clinical variation.

Despite decades of atlas production and use within multiple healthcare systems, and consistent reporting of geographical differences in the utilisation of services, significant levels of clinical variation persist. Drawing on over forty years of combined experience using atlases of clinical variation, we reflect on why that might be the case and explore the role of atlases have played in efforts to reduce inappropriate overuse, underuse and misuse of healthcare services. We contend that atlases are useful but, on their own, are not enough to drive change in clinical practice and improvement in patient outcomes. Building on four conceptual models we have published since 2017, we argue that atlases, with their focus on measuring healthcare utilisation by residents in different geographies, generally fail to provide sufficient information and statistical analyses to truly assess the nature of the variation and support action for change. They seldom use structures such as hospitals or teams as the unit of analysis to understand variation; they rarely feature the key elements of healthcare performance which underlie variation; they are mostly silent about how to assess whether the variation measured is warranted or truly unwarranted; nor do they identify evidence-based levers for change. This means that a stark choice confronts producers of atlases - to either continue with the current model and more explicitly rely on other players to undertake work to complete the 'data to action' cycle that is necessary to secure improvement; or to refine their offering - including more sophisticated performance measurement approaches, nuanced guides for interpretation of any differences found, support for the selection and application of levers for change that align with local context, and provision of evidence-based options for implementation.

Indirect Impacts of COVID-19: A Case Study of Evidence, Advice and Representation From Consumer and Community Members in New South Wales Australia.

The COVID-19 pandemic continues to affect health care systems globally, and there is widespread concern about the indirect impacts of COVID-19. Indirect impacts are caused by missed or delayed health care-not as a direct consequence of COVID-19 infections. This study gathered experiences of, and perspectives on, the indirect impacts of COVID-19 for health consumers, patients, their families and carers, and the broader community in New South Wales, Australia. A series of semi-structured virtual group discussions were conducted with 33 health consumers and community members between August 24 and August 31, 2020. Data were analyzed using an inductive thematic analysis approach. The analysis identified 3 main themes: poor health outcomes for individuals; problems with how health care is designed and delivered; and increasing health inequality. This case study provides insight into the indirect impacts of COVID-19. Health systems can draw on the insights learned as a source of experiential evidence to help identify, monitor and respond to the indirect impacts of COVID-19.

Virtual care in end of life and palliative care: A rapid evidence check.

The purpose of this rapid evidence check was to identify virtual care modalities that are safe and effective in the delivery of end of life and palliative care. Thirty-three peer reviewed articles which were either review articles or interventional/evaluative studies presenting comparative data were identified through PubMed, Google, and Google Scholar searches. Extracted data was synthesized narratively and outcomes were categorised separately for patients, healthcare providers, caregivers and health system. Included studies reported on a wide range of virtual care modalities, including video consultation, mobile apps, videos, websites, telephone support, email and alert messages. Generally, studies reported similar or favourable quality of life outcomes to face-to-face palliative care, especially when virtual care was used as a supplement rather than a substitute for face-to-face care. Positive attitudes for perceived usefulness and helpfulness were reported by patients, caregivers and healthcare providers. Challenges identified related to technology limitations, trust, ethical concerns, administrative burden and evidence gaps. Overall, most studies found virtual care modalities to be safe and effective in end of life and palliative care with no detrimental adverse outcomes, when used as a supplement to face-to-face care.

Unwarranted clinical variation in health care: Definitions and proposal of an analytic framework.

RATIONALE, AIMS, AND OBJECTIVES: Unwarranted clinical variation is a topic of heightened interest in health care systems around the world. While there are many publications and reports on clinical variation, few studies are conceptually grounded in a theoretical model. This study describes the empirical foundations of the field and proposes an analytic framework. METHOD: Structured construct mapping of published empirical studies which explicitly address unwarranted clinical variation. RESULTS: A total of 190 studies were classified in terms of three key dimensions: perspective (assessing variation across geographical areas or across providers); criteria for assessment (measuring absolute variation against a standard, or relative variation within a comparator group); and object of analysis (using process, structure/resource, or outcome metrics). CONCLUSION: Consideration of the results of the mapping exercise-together with a review of adjustment, explanatory and stratification variables, and the factors associated with residual variation-informed the development of an analytic framework. This framework highlights the role that agency and motivation, evidence and judgement, and personal and organizational capacity play in clinical decision making and reveals key facets that distinguish warranted from unwarranted clinical variation. From a measurement perspective, it underlines the need for careful consideration of attribution, aggregation, models of care, and temporality in any assessment.

Assessing Performance in Health Care Using International Surveys: Are Patient and Clinician Perspectives Complementary or Substitutive.

BACKGROUND: Over the last decade, international surveys of patients and clinicians have been used to compare health care across countries. Findings from these surveys have been extensively used to create aggregate scores and rankings. OBJECTIVE: To assess the concordance of survey responses provided by patients and clinicians. METHODS: Analysis of 16 pairs of questions that focused on coordination, organizational factors, and patient-centered competencies from the Commonwealth Fund International Health Policy Survey of older adults (2014) and of primary care physicians (2015). Concordance was assessed by comparing absolute rates and relative rankings. RESULTS: In absolute terms, patients and clinicians gave differing responses for questions about coordination of care (patients were more positive) and provision of after-hours care (patients were less positive). In relative terms, country rankings were positively correlated for 5 of 16 question pairs (Spearman ρ > .6 and P < .05). CONCLUSION: Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients' and clinicians' perspectives before combining them to create aggregate assessments of performance.

Impact of COVID-19 on healthcare activity in NSW, Australia.

OBJECTIVE: COVID-19 has left no healthcare system untouched. In New South Wales (NSW), the most populous state in Australia, COVID-19 case numbers have to date been relatively low. However, that does not mean the state's healthcare system has been unaffected. Preparations to create sufficient capacity to accommodate a COVID-19 surge resulted in widespread deferment of scheduled medical procedures and appointments. Patterns of healthcare-seeking behaviour changed, with a reluctance to visit healthcare settings. The aim of this study was to quantify the changes in activity seen in the NSW health system in the first half of 2020. METHODS: Healthcare data were drawn from multiple sources, including primary care, ambulance, emergency departments and inpatient settings. Volumes of healthcare activity in 2020, overall, by urgency of cases and by reasons for care were compared with the figures for the same period in 2019. Changes in the modality of care provision were also examined. RESULTS: In March to June 2020, compared with the same period in 2019, primary care face-to-face consultations decreased by 22.1%, breast screening activity by 51.5%, ambulance incidents by 7.2%, emergency department visits by 13.9%, public hospital inpatient episodes by 14.3%, and public hospital planned surgical activity by 32.6%. CONCLUSION: There were substantial declines in a wide range of healthcare activities across the NSW health system between March and June 2020 due to the impact of the COVID-19 pandemic. Although activity levels were recovering by September 2020, they had not yet returned to 'normal'. The implications of these changes - and the indirect impact of COVID-19 - require further study.

Implementing large-system, value-based healthcare initiatives: a realist study protocol for seven natural experiments.

INTRODUCTION: Value-based healthcare delivery models have emerged to address the unprecedented pressure on long-term health system performance and sustainability and to respond to the changing needs and expectations of patients. Implementing and scaling the benefits from these care delivery models to achieve large-system transformation are challenging and require consideration of complexity and context. Realist studies enable researchers to explore factors beyond 'what works' towards more nuanced understanding of 'what tends to work for whom under which circumstances'. This research proposes a realist study of the implementation approach for seven large-system, value-based healthcare initiatives in New South Wales, Australia, to elucidate how different implementation strategies and processes stimulate the uptake, adoption, fidelity and adherence of initiatives to achieve sustainable impacts across a variety of contexts. METHODS AND ANALYSIS: This exploratory, sequential, mixed methods realist study followed RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) reporting standards for realist studies. Stage 1 will formulate initial programme theories from review of existing literature, analysis of programme documents and qualitative interviews with programme designers, implementation support staff and evaluators. Stage 2 envisages testing and refining these hypothesised programme theories through qualitative interviews with local hospital network staff running initiatives, and analyses of quantitative data from the programme evaluation, hospital administrative systems and an implementation outcome survey. Stage 3 proposes to produce generalisable middle-range theories by synthesising data from context-mechanism-outcome configurations across initiatives. Qualitative data will be analysed retroductively and quantitative data will be analysed to identify relationships between the implementation strategies and processes, and implementation and programme outcomes. Mixed methods triangulation will be performed. ETHICS AND DISSEMINATION: Ethical approval has been granted by Macquarie University (Project ID 23816) and Hunter New England (Project ID 2020/ETH02186) Human Research Ethics Committees. The findings will be published in peer-reviewed journals. Results will be fed back to partner organisations and roundtable discussions with other health jurisdictions will be held, to share learnings.

Lessons from evaluations of purchaser pay-for-performance programs: a review of the evidence.

There has been a growing interest in the use of financial incentives to encourage improvements in the quality of health care. Several articles have reviewed past studies of the impact of specific incentive arrangements, but these studies addressed small-scale experiments, making their findings arguably of limited relevance to current improvement efforts. In this article, the authors review evaluations of more recent pay-for-performance initiatives instituted by health plans or by provider organizations in cooperation with health plans. Findings show improvement in selected quality measures in most of these initiatives, but the contribution of financial incentives to that improvement is not clear; the incentives typically were implemented in conjunction with other quality improvement efforts, or there was not a convincing comparison group. However, the literature relating to purchaser pay-for-performance initiatives does underscore several important issues that deserve attention going forward that relate to the design and implementation of pay-for-performance initiatives.

Impact of targeted financial incentives on personal health behavior: a review of the literature.

Over the past decade, there has been a substantial increase in the use of financial incentives by private employers and public programs to encourage healthy behaviors, wellness activities, and use of preventive services. The research evidence regarding the effectiveness of this approach is reviewed, summarizing relevant findings from literature reviews and from recent evaluations. The article concludes that financial incentives, even relatively small incentives, can influence individuals' health-related behaviors. However, the findings regarding health promotion and wellness are based primarily on analyses of a limited number of private sector initiatives, whereas the evidence regarding preventive services is based on evaluations of initiatives sponsored predominantly by public programs and directed at low-income populations. In either case, there are several important limitations in the ability of the published findings to provide clear guidance for public program administrators or private purchasers seeking to design and implement effective incentive programs.