Multiple case study of processes used by hospitals to select performance indicators: do they align with best practices?

Several health policy institutes recommend reducing the number of indicators monitored by hospitals to better focus on indicators most relevant to local contexts. To determine which indicators are the most appropriate to eliminate, one must understand how indicator selection processes are undertaken. This study classifies hospital indicator selection processes and analyzes how they align with practices outlined in the 5-P Indicator Selection Process Framework. This qualitative, multiple case study examined indicator selection processes used by four large acute care hospitals in Ontario, Canada. Data were collected through 13 semistructured interviews and document analysis. A thematic analysis compared processes to the 5-P Indicator Selection Process Framework. Two types of hospital indicator selection processes were identified. Hospitals deployed most elements found within the 5-P Indicator Selection Process Framework including setting clear aims, having governance structures, considering indicators required by health agencies, and categorizing indicators into strategic themes. Framework elements largely absent included: adopting evidence-based selection criteria; incorporating finance and human resources indicators; considering if indicators measure structures, processes, or outcomes; and engaging a broader set of end users in the selection process. Hospitals have difficulty in balancing how to monitor government-mandated indicators with indicators more relevant to local operations. Hospitals often do not involve frontline managers in indicator selection processes. Not engaging frontline managers in selecting indicators may risk hospitals only choosing government-mandated indicators that are not reflective of frontline operations or valued by those managers accountable for improving unit-level performance.

Selecting Performance Indicators and Targets in Health Care: An International Scoping Review and Standardized Process Framework.

Objective: Health care organizations monitor hundreds of performance indicators. It is unclear what processes and criteria organizations use to identify the indicators they use, who is involved in these processes, how performance targets are set, and what the impacts of these processes are. The purpose of this study is to synthesize international approaches to indicator selection and develop a standardized process framework. Methods: Using the PubMed and Web of Science search engines, a scoping review of peer reviewed and grey literature following PRISMA-ScR guidelines was conducted to identify documents describing indicator selection processes used by health systems. English-language papers from 11 countries published from 2010 to 2020 were included. Papers were thematically analyzed to develop a standardized process framework. Results: The review included 33 peer-reviewed papers and 11 grey-literature documents. While there are common practices used in health care to select indicators, no single standardized process framework for indicator selection exists. Arbitrary or incomplete indicator selection processes risk over-measurement, lack of alignment with strategic and operational goals, lack of support by end-users, and paralyzed decision-making ability. By consolidating international practices, we developed the 5-P indicator selection process framework to mitigate process risks and support high-quality indicator selection processes. Conclusion: The 5-P indicator selection process framework consists of five domains and 17 elements, and offers health care agencies a practical structure they can use to design indicator selection processes. The framework also provides researchers with a basis by which the implementation of these processes may be evaluated.

Rethinking health policy student practicums through the application of the multiple streams framework: A case study.

Many university and college programs offer co-op placements or practicums as part of their curriculum, with the aim of providing real-world experience and opportunity for students to apply theory to practice. These practicums are not always grounded in the underlying management or policy theories the program teaches, instead they often focus on universal attributes such as task performance or general leadership. This case study describes how a University of Toronto Health Policy and Management student and an Executive from Ontario's Ministry of Health redesigned the student's practicum to be grounded in Kingdon's Multiple Stream Policy Framework. The case demonstrates how the theoretical framework was applied to enhance their weekly mentorship discussions, and organize the student's learnings relating to the Ministry's policy on hospital capacity during the COVID-19 pandemic by viewing the work through the framework's five streams.

What Factors Impact Implementation of Critical Incident Disclosure in Ontario Hospitals: A Multiple-Case Study.

Guidelines and legislation prescribe how hospitals should conduct critical incident disclosures with patients. However, variation in secondary disclosure implementation can occur. Using the Consolidated Framework for Implementation Research, this qualitative multiple-case study explored the factors that impact Ontario hospitals' secondary disclosure of critical incidents. The study concludes that while hospitals generally implement guidelines consistently, complex environments and differing professional backgrounds lead to variations. Consequently, hospitals should address timing delays, improve documentation and enhance support to clinicians who conduct the disclosures. Policy makers should consider the benefits and challenges of written disclosure, and offering patients a choice in the setting where disclosure occurs, as potential improvements.

An opportunity for improved engagement and transparency: A systematic review of renal dialysis cost effectiveness and discrete choice experiment studies.

Much attention is given to patient and provider engagement, cost, and quality. Nephrology is in a unique position to examine the intersection of these issues given kidney dialysis is delivered at a high cost to chronically ill patients. Annual dialysis treatments in Canada range from $56,000-$107,000 per patient dependent on modality. Economists quantify the preferred modality by calculating cost effectiveness through quality-adjusted life years or determining utilization through Discrete Choice Experiments (DCEs). Cost-effectiveness studies identify peritoneal dialysis as the most economical, yet it is the least used. Discrete choice experiments address patient preferences but rarely include cost attributes. This presents a unique paradigm: cost studies do not include patient or physician perspectives, and DCEs do not consider cost. This systematic review of dialysis cost-effectiveness studies and DCEs identifies an opportunity to increase engagement and transparency by involving all care partners in assessing quality and cost.

Twenty Years Later: Do We Have an Agreed-Upon Definition of Health Quality Management?

Since the Institute of Medicine's report in the USA in 1999 and the Baker-Norton report in Canada in 2004, academics have studied quality in areas such as governance, process improvement, measurement, patient engagement, physician and staff engagement and finance. Few, however, have offered a formal definition for their area or integrated each area's underlying assumptions to offer a shared definition of quality. This paper summarizes selected literature in each area, analyzes their contributions and compares common strands between each area to build a single integrated definition of health quality management which managers can apply in their health setting.

Yes Doctors, You Were Right. The Data Were Wrong: One Organization's Data Quality Journey.

In 2012, publicly released reports indicated that the health outcomes at St. Joseph's Health Centre, Toronto (SJHC), may not be of the same quality when compared with those at peer hospitals. This surprised the leaders within the organization given that SJHC had a sound reputation for quality and patient safety within the sector. As a result, SJHC's senior management and medical leadership identified clinical outcomes and data quality as items to be addressed within its enterprise risk management framework with a focus on the methods by which data were collected, coded and used by clinicians. The following article describes the approach SJHC used to improve the quality of its clinical data and how it changed physician participation in examining data designed to help inform and improve care.

Hospital on a page: standardizing data presentation to drive quality improvement.

Over the past five years, the Credit Valley Hospital (CVH) invested time and financial and human resources into performance measurement systems. In doing so, CVH launched a number of data tools including electronic scorecards and dashboards. However, the processes and accountability structures associated with the tools struggled to gain credibility with clinical and administrative leadership as the performance measurement system was primarily driven by the technology rather than a sound information strategy. Although a corporate-level scorecard was regularly updated, program-related scorecards and other measurement tools were only populated when programs reported to the board, at the time of accreditation or as a result of regulatory requirements. In addition, information contained in data reports was often presented in a manner that did not engage clinical and corporate decision-makers in the key issues of quality, access and sustainability. Following the release of its new strategic plan in 2009, CVH renewed its performance measurement framework and the methods by which it presented data so that the organization's strategic plan could be implemented and measured from the boardroom to the bedside. Long, complex spreadsheets were transformed into strategically designed, easy-to-understand, easy-to-access reports released in a standardized method in terms of format, media, content and timing. The following article describes the method CVH adopted to communicate the organization's performance and the role it played in enhancing the culture of quality and patient safety within the hospital.

From boardroom to bedside: how to define and measure hospital quality.

Following the release of its strategic plan, in which patient safety and quality were highlighted as key directions, St. Joseph's Healthcare Hamilton recognized the importance of engaging its board of trustees to achieve these goals. Following a collaborative retreat with senior management, medical staff leadership and professional practice leaders, the board enhanced its governance oversight on quality. By removing quality from the consent agenda, defining quality and selecting a series of "big dot" measures, the board has led the development of a culture of quality that cascades from the boardroom to the bedside. This article describes how the organization followed a systematic process to define quality and select big dot quality indicators.

Real numbers tell real stories in health services management.

In the words of one hospital manager, "hospital data is currently indigestible and alien to the average user." Drawing upon the experience of an academic hospital that, contrary to established practice, published real numbers alongside rates and ratios during a Clostridium difficile outbreak, the authors examined the pitfalls of publishing only abstract performance measures and the advantages of releasing real numbers to the public. This article identifies lessons for hospital board governance, media relations, employee communications, and citizen and patient engagement that are applicable across the healthcare industry in many countries. If healthcare is to be a caring industry, then care should be taken in the public reporting of data and information.

Perhaps I am one of the lucky ones.

I recently participated in the Ontario Hospital Association (OHA) webinar on quality and safety, in which I was asked to address the topic of engaging physicians in performance measurement, quality and safety. I am not a physician, but much of my work in healthcare has involved working with clinical leaders in these areas. At St. Joseph's Healthcare Hamilton, we started our quality and safety journey by creating a medical quality scorecard specifically designed for physicians. The card, written in physician-friendly versus business language, enabled physicians to drop the traditional business quadrants of finance and human resources and select four quadrants focused on clinical process and outcome indicators that matched their daily practice. Quality improvement initiatives resulting from the scorecard included the launch of a sepsis-management campaign in the emergency room and a new approach to neonatal safety.

Engaging physician leaders in performance measurement and quality.

As part of its 2007 Innovation Series, the Institute for Healthcare Improvement released a white paper, Engaging Physicians in a Shared Quality Agenda, in which it described the importance of medical staff leadership and the participation of these leaders in achieving a patient safety and quality culture. Following the release of its strategic plan to 2012, in which it highlighted patient safety and quality as key directions, St. Joseph's Healthcare Hamilton recognized the importance of engaging its medical staff to achieve these goals. To facilitate engagement, a medical quality scorecard was developed in collaboration with medical leaders with a view to enabling physicians to easily identify their program goals, measure their progress and transfer knowledge across the organization.