A comparison of two structured taxonomic strategies in capturing adverse events in U.S. hospitals.

OBJECTIVE: To compare the Agency for Healthcare Research and Quality's Quality and Safety Review System (QSRS) and the proposed triadic structure for the 11th version of the International Classification of Disease (ICD-11) in their ability to capture adverse events in U.S. hospitals. DATA SOURCES/STUDY SETTING: One thousand patient admissions between 2014 and 2016 from three general, acute care hospitals located in Maryland and Washington D.C. STUDY DESIGN: The admissions chosen for the study were a random sample from all three hospitals. DATA COLLECTION/EXTRACTION METHODS: All 1000 admissions were abstracted through QSRS by one set of Certified Coding Specialists and a different set of coders assigned the draft ICD-11 codes. Previously assigned ICD-10-CM codes for 230 of the admissions were also used. PRINCIPAL FINDINGS: We found less than 20 percent agreement between QSRS and ICD-11 in identifying the same adverse event. The likelihood of a mismatch between QSRS and ICD-11 was almost twice that of a match. The findings were similar to the agreement found between QSRS and ICD-10-CM in identifying the same adverse event. When coders were provided with a list of potential adverse events, the sensitivity and negative predictive value of ICD-11 improved. CONCLUSIONS: While ICD-11 may offer an efficient way of identifying adverse events, our analysis found that in its draft form, it has a limited ability to capture the same types of events as QSRS. Coders may require additional training on identifying adverse events in the chart if ICD-11 is going to prove its maximum benefit.

Opportunities to Improve Informed Consent with AHRQ Training Modules.

BACKGROUND: Informed consent is a process of communication between clinician and patient that results in the patient's decision about whether to undergo a specific intervention. However, patients often do not understand the risks, benefits, and alternatives, even after signing a consent form. METHODS: Mixed-methods pilot test of two Agency for Healthcare Research and Quality (AHRQ) informed consent training modules implemented in four hospitals. Methods included staff and patient surveys, interviews, site visits, and pre- and posttests of the modules. RESULTS: A low proportion of clinicians reported using teach-back (40.0%) or high-quality decision aids (55.0%). Patients reported limited use of best practices, including being asked to teach-back (58.4%), having other options described (54.9%), viewing decision aids (37.4%), and finding the form very easy to understand (66.8%). Content of the training modules aligned well with identified deficiencies. Barriers to completing the modules included staff turnover, competing demands, and lack of accountability. Facilitators included committed champions with available time, motivation, and release time for staff to take modules. Knowledge increased for leaders (p <0.05) and staff (p <0.001) who completed the training modules. Hospitals reported the effects of piloting the modules included fostering dialogue and identifying opportunities for improvements, identifying and rectifying policy ambiguity and noncompliance, reinforcing the use of interpreter services, and using modules' strategies and tools to improve informed consent. CONCLUSION: Many opportunities exist for hospitals to improve their informed consent practices. AHRQ's two training modules, have face validity, addressed demonstrated deficiencies in hospitals' informed consent policies and processes, and stimulated improvement activity in motivated hospitals.

Agency for Healthcare Research and Quality Evidence-based Practice Center methods provide guidance on prioritization and selection of harms in systematic reviews.

OBJECTIVES: Systematic reviews should provide balanced assessments of benefits and harms, while focusing on the most important outcomes. Selection of harms to be reviewed can be a challenge due to the potential for large numbers of diverse harms. STUDY DESIGN AND SETTING: A workgroup of methodologists from Evidence-based Practice Centers (EPCs) developed consensus-based guidance on selection and prioritization of harms in systematic reviews. Recommendations were informed by a literature scan, review of Evidence-based Practice Center reports, and interviews with experts in conducting reviews or assessing harms and persons representing organizations that commission or use systematic reviews. RESULTS: Ten recommendations were developed on selection and prioritization of harms, including routinely focusing on serious as well as less serious but frequent or bothersome harms; routinely engaging stakeholders and using literature searches and other data sources to identify important harms; using a prioritization process (formal or less formal) to inform selection decisions; and describing the methods used to select and prioritize harms. CONCLUSION: We provide preliminary guidance for a more structured approach to selection and prioritization of harms in systematic reviews.

Advancing perinatal patient safety through application of safety science principles using health IT.

BACKGROUND: The use of health information technology (IT) has been shown to promote patient safety in Labor and Delivery (L&D) units. The use of health IT to apply safety science principles (e.g., standardization) to L&D unit processes may further advance perinatal safety. METHODS: Semi-structured interviews were conducted with L&D units participating in the Agency for Healthcare Research and Quality's (AHRQ's) Safety Program for Perinatal Care (SPPC) to assess units' experience with program implementation. Analysis of interview transcripts was used to characterize the process and experience of using health IT for applying safety science principles to L&D unit processes. RESULTS: Forty-six L&D units from 10 states completed participation in SPPC program implementation; thirty-two (70%) reported the use of health IT as an enabling strategy for their local implementation. Health IT was used to improve standardization of processes, use of independent checks, and to facilitate learning from defects. L&D units standardized care processes through use of electronic health record (EHR)-based order sets and use of smart pumps and other technology to improve medication safety. Units also standardized EHR documentation, particularly related to electronic fetal monitoring (EFM) and shoulder dystocia. Cognitive aids and tools were integrated into EHR and care workflows to create independent checks such as checklists, risk assessments, and communication handoff tools. Units also used data from EHRs to monitor processes of care to learn from defects. Units experienced several challenges incorporating health IT, including obtaining organization approval, working with their busy IT departments, and retrieving standardized data from health IT systems. CONCLUSIONS: Use of health IT played an integral part in the planning and implementation of SPPC for participating L&D units. Use of health IT is an encouraging approach for incorporating safety science principles into care to improve perinatal safety and should be incorporated into materials to facilitate the implementation of perinatal safety initiatives.

Development and Validation of the Agency for Healthcare Research and Quality Measures of Potentially Preventable Emergency Department (ED) Visits: The ED Prevention Quality Indicators for General Health Conditions.

OBJECTIVE: To develop and validate rates of potentially preventable emergency department (ED) visits as indicators of community health. DATA SOURCES: Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project 2008-2010 State Inpatient Databases and State Emergency Department Databases. STUDY DESIGN: Empirical analyses and structured panel reviews. METHODS: Panels of 14-17 clinicians and end users evaluated a set of ED Prevention Quality Indicators (PQIs) using a Modified Delphi process. Empirical analyses included assessing variation in ED PQI rates across counties and sensitivity of those rates to county-level poverty, uninsurance, and density of primary care physicians (PCPs). PRINCIPAL FINDINGS: ED PQI rates varied widely across U.S. communities. Indicator rates were significantly associated with county-level poverty, median income, Medicaid insurance, and levels of uninsurance. A few indicators were significantly associated with PCP density, with higher rates in areas with greater density. A clinical and an end-user panel separately rated the indicators as having strong face validity for most uses evaluated. CONCLUSIONS: The ED PQIs have undergone initial validation as indicators of community health with potential for use in public reporting, population health improvement, and research.

Hospital-Level Factors Related to 30-Day Readmission Rates.

This study investigates the relationship between inpatient quality of care as measured by the Agency for Healthcare Research and Quality (AHRQ) patient safety indicator (PSI) composite and all-cause, hospital-wide, 30-day readmission rates. Discharge data from 4 statewide databases were analyzed. Linear, repeated-measures regressions were performed to predict hospital-level 30-day readmission rates. The mean readmission rate was 12.9%, and the mean PSI composite ratio was 0.95 among 524 hospitals with 2592 observations. In the hospital-level analysis, the risk-adjusted AHRQ PSI composite was not significantly associated with hospital 30-day readmission rate after controlling for hospital-level characteristics, patient case mix, and sociodemographics. Inpatient quality of care appears to have less influence on hospital readmission rates than do clinical and socioeconomic factors. However, these results suggest that a patient safety composite measure that includes postdischarge complications would provide more information to assist hospitals and communities in understanding the association between quality of care and readmission rates.

Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator for Postoperative Respiratory Failure (PSI 11) does not identify accurately patients who received unsafe care.

BACKGROUND: The Agency for Healthcare Research and Quality Patient Safety Indicator 11 is used to identify postoperative respiratory failure events and detect areas for quality improvement. This study examines the accuracy of Patient Safety Indicator 11 in identifying clinically valid patient safety events. METHODS: All cases flagged for Patient Safety Indicator 11 from July 2013 to July 2015 by Agency for Healthcare Research and Quality QI Version 4.5 including International Classification of Diseases-9 codes were evaluated. Code-confirmed cases underwent independent review by 2 physicians. Inpatient electronic medical records were used to identify clinical factors for postoperative respiratory failure in each case to determine if postoperative respiratory failure was a result of unsafe care. The clinical true-positive rate and positive predictive value were calculated. RESULTS: A total of 166 postoperative respiratory failure cases were reviewed; 51 were recoded and reversed due to coding or documentation errors; 115 cases met the Agency for Healthcare Research and Quality definition of postoperative respiratory failure. A total of 71 (61.7%) of the 115 cases were false positives and did not reflect unsafe care, while 44 cases were true positives with a positive predictive value of 38.3%. χ(2) analysis did not reveal an association between demographics, clinical characteristics, or operative procedure with true-positive cases. CONCLUSION: Administrative coding data for Agency for Healthcare Research and Quality Patient Safety Indicator 11 do not identify accurately patients who received unsafe care when taking into account unpreventable clinical factors causing postoperative respiratory failure. The use of Agency for Healthcare Research and Quality Patient Safety Indicator 11 as a hospital performance measure should be reconsidered until inclusion and exclusion criteria are revised.

Validity of the Agency for Health Care Research and Quality Patient Safety Indicators and the Centers for Medicare and Medicaid Hospital-acquired Conditions: A Systematic Review and Meta-Analysis.

BACKGROUND: The Agency for Health Care Research and Quality Patient Safety Indicators (PSIs) and Centers for Medicare and Medicaid Services Hospital-acquired Conditions (HACs) are increasingly being used for pay-for-performance and public reporting despite concerns over their validity. Given the potential for these measures to misinform patients, misclassify hospitals, and misapply financial and reputational harm to hospitals, these need to be rigorously evaluated. We performed a systematic review and meta-analysis to assess PSI and HAC measure validity. METHODS: We searched MEDLINE and the gray literature from January 1, 1990 through January 14, 2015 for studies that addressed the validity of the HAC measures and PSIs. Secondary outcomes included the effects of present on admission (POA) modifiers, and the most common reasons for discrepancies. We developed pooled results for measures evaluated by ≥3 studies. We propose a threshold of 80% for positive predictive value or sensitivity for pay-for-performance and public reporting suitability. RESULTS: Only 5 measures, Iatrogenic Pneumothorax (PSI 6/HAC 17), Central Line-associated Bloodstream Infections (PSI 7), Postoperative hemorrhage/hematoma (PSI 9), Postoperative deep vein thrombosis/pulmonary embolus (PSI 12), and Accidental Puncture/Laceration (PSI 15), had sufficient data for pooled meta-analysis. Only PSI 15 (Accidental Puncture and Laceration) met our proposed threshold for validity (positive predictive value only) but this result was weakened by considerable heterogeneity. Coding errors were the most common reasons for discrepancies between medical record review and administrative databases. POA modifiers may improve the validity of some measures. CONCLUSION: This systematic review finds that there is limited validity for the PSI and HAC measures when measured against the reference standard of a medical chart review. Their use, as they currently exist, for public reporting and pay-for-performance, should be publicly reevaluated in light of these findings.

Performance Measures in Neurosurgical Patient Care: Differing Applications of Patient Safety Indicators.

BACKGROUND: Patient Safety Indicators (PSIs) are administratively coded identifiers of potentially preventable adverse events. These indicators are used for multiple purposes, including benchmarking and quality improvement efforts. Baseline PSI evaluation in high-risk surgeries is fundamental to both purposes. OBJECTIVE: Determine PSI rates and their impact on other outcomes in patients undergoing cranial neurosurgery compared with other surgeries. RESEARCH DESIGN: The Agency for Healthcare Research and Quality (AHRQ) PSI software was used to flag adverse events and determine risk-adjusted rates (RAR). Regression models were built to assess the association between PSIs and important patient outcomes. SUBJECTS: We identified cranial neurosurgeries based on International Classification of Diseases, Ninth Revision, Clinical Modification codes in California, Florida, New York, Arkansas, and Mississippi State Inpatient Databases, AHRQ, 2010-2011. MEASURES: PSI development, 30-day all-cause readmission, length of stay, hospital costs, and inpatient mortality. RESULTS: A total of 48,424 neurosurgical patients were identified. Procedure indication was strongly associated with PSI development. The neurosurgical population had significantly higher RAR of most PSIs evaluated compared with other surgical patients. Development of a PSI was strongly associated with increased length of stay and hospital cost and, in certain PSIs, increased inpatient mortality and 30-day readmission. CONCLUSIONS: In this population-based study, certain accountability measures proposed for use as value-based payment modifiers show higher RAR in neurosurgery patients compared with other surgical patients and were subsequently associated with poor outcomes. Our results indicate that for quality improvement efforts, the current AHRQ risk-adjustment models should be viewed in clinically meaningful stratified subgroups: for profiling and pay-for-performance applications, additional factors should be included in the risk-adjustment models. Further evaluation of PSIs in additional high-risk surgeries is needed to better inform the use of these metrics.

Examining the validity of AHRQ's patient safety indicators (PSIs): is variation in PSI composite score related to hospital organizational factors?

Increasing use of Agency for Healthcare Research and Quality's Patient Safety Indicators (PSIs) for hospital performance measurement intensifies the need to critically assess their validity. Our study examined the extent to which variation in PSI composite score is related to differences in hospital organizational structures or processes (i.e., criterion validity). In site visits to three Veterans Health Administration hospitals with high and three with low PSI composite scores ("low performers" and "high performers," respectively), we interviewed a cross-section of hospital staff. We then coded interview transcripts for evidence in 13 safety-related domains and assessed variation across high and low performers. Evidence of leadership and coordination of work/communication (organizational process domains) was predominantly favorable for high performers only. Evidence in the other domains was either mixed, or there were insufficient data to rate the domains. While we found some evidence of criterion validity, the extent to which variation in PSI rates is related to differences in hospitals' organizational structures/processes needs further study.

Exploring relationships between patient safety culture and patients' assessments of hospital care.

OBJECTIVES: The purpose of this study was to examine relationships among 2 Agency for Healthcare Research and Quality measures of hospital patient safety and quality, which reflect different perspectives on hospital performance: the Hospital Survey on Patient Safety Culture (Hospital SOPS)--a hospital employee patient safety culture survey--and the Consumer Assessment of Healthcare Providers and Systems Hospital Survey (CAHPS Hospital Survey)--a survey of the experiences of adult inpatients with hospital care and services. Our hypothesis was that these 2 measures would be positively related. METHODS: We performed multiple regressions to examine the relationships between the Hospital SOPS measures and CAHPS Hospital Survey measures, controlling for hospital bed size and ownership. Analyses were conducted at the hospital level with each survey's measures using data from 73 hospitals that administered both surveys during similar periods. RESULTS: Higher overall Hospital SOPS composite average scores were associated with higher overall CAHPS Hospital Survey composite average scores (r = 0.41, P G 0.01). Twelve of 15 Hospital SOPS measures were positively related to the CAHPS Hospital Survey composite average score after controlling for bed size and ownership, with significant standardized regression coefficients ranging from 0.25 to 0.38. None of the Hospital SOPS measures were significantly correlated with either of the two single-item CAHPS Hospital Survey measures (hospital rating and willingness to recommend). CONCLUSIONS: This study found that hospitals where staff have more positive perceptions of patient safety culture tend to have more positive assessments of care from patients. This finding helps validate both surveys and suggests that improvements in patient safety culture may lead to improved patient experience with care. Further research is needed to determine the generalizability of these results to larger sets of hospitals, to hospital units, and to other settings of care.

Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicator Rates Among Veteran Dual Users.

This study compares rates of 11 Agency for Healthcare Research and Quality Patient Safety Indicators (PSIs) among 266 203 veteran dual users (ie, those with hospitalizations in both the Veterans Health Administration [VA] and the private sector through Medicare fee-for-service coverage) during 2002 to 2007. PSI risk-adjusted rates were calculated using the PSI software (version 3.1a). Rates of pressure ulcer, central venous catheter-related bloodstream infections, and postoperative sepsis, areas in which the VA has focused quality improvement efforts, were found to be significantly lower in the VA than in the private sector. VA had significantly higher rates for 7 of the remaining 8 PSIs, although the rates of only 2 PSIs (postoperative hemorrhage/hematoma and accidental puncture or laceration) remained higher in the VA after sensitivity analyses were conducted. A better understanding of system-level differences in coding practices and patient severity, poorly documented in administrative data, is needed before conclusions about differences in quality can be drawn.

The prevalence of patient safety indicators and hospital-acquired conditions in patients with ruptured cerebral aneurysms: establishing standard performance measures using the Nationwide Inpatient Sample database.

OBJECT: The Agency for Healthcare Research and Quality (AHRQ) patient safety indicators (PSIs) and the Centers for Medicare and Medicaid Services hospital-acquired conditions (HACs) are publicly reported metrics used to gauge the quality of health care provided by health care institutions. To better understand the prevalence of these events in hospitalized patients treated for ruptured cerebral aneurysms, the authors determined the incidence rates of PSIs and HACs among patients with a diagnosis of subarachnoid hemorrhage and procedure codes for either coiling or clipping in the Nationwide Inpatient Sample database. METHODS: The authors queried the Nationwide Inpatient Sample database, part of the AHRQ's Healthcare Cost and Utilization Project, for all hospitalizations between 2002 and 2010 involving coiling or clipping of ruptured cerebral aneurysms. The incidence rate of each PSI and HAC was determined by searching the hospital records for ICD-9 codes. The authors used the SAS statistical software package to calculate incidence rates and perform multivariate analyses to determine the effects of patient variables on the probability of developing each indicator. RESULTS: There were 62,972 patient admissions with a diagnosis code of subarachnoid hemorrhage between the years 2002 and 2010; 10,274 (16.3%) underwent clipping and 8248 (13.1%) underwent endovascular coiling. A total of 6547 PSI and HAC events occurred within the 10,274 patients treated with clipping; at least 1 PSI or HAC occurred in 47.9% of these patients. There were 5623 total PSI and HAC events among the 8248 patients treated with coils; at least 1 PSI or HAC occurred in 51.0% of coil-treated patients. Age, sex, comorbidities, hospital size, and hospital type had statistically significant associations with indicator occurrence. Compared with patients without events, those treated by either clipping or coiling and had at least 1 PSI during their hospitalization had significantly longer lengths of stay (p < 0.001), higher hospital costs (p < 0.001), and higher in-hospital mortality rates (p < 0.001). CONCLUSIONS: These results estimate baseline national rates of PSIs and HACs in patients treated for ruptured cerebral aneurysms. These data may be used to gauge individual institutional quality of care and patient safety metrics in comparison with national data.

Consideration of ICD-9 code-derived disease-specific safety indicators in CKD.

BACKGROUND AND OBJECTIVES: The Agency for Healthcare and Research Quality patient safety indicators track adverse safety events in hospitalized patients but overlook safety incidents specific to CKD. This study considers candidate CKD-pertinent patient safety indicators and compares them with the Agency for Healthcare and Research Quality patient safety indicators. DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS: Using a national Veterans Health Administration database of hospitalized veterans from fiscal year 2005, 247,160 hospitalized veterans with prehospitalization measures of renal function were retrospectively examined for proposed CKD patient safety indicators versus Agency for Healthcare and Research Quality patient safety indicators using International Classification of Diseases, Ninth Revision diagnosis codes. Candidate CKD-pertinent patient safety indicators included in-hospital acute kidney failure; in-hospital congestive heart failure (and related diagnostic codes); electrolyte disturbances; and medication errors, poisoning, and intoxication. Patients with a prehospital estimated GFR<60 ml/min per 1.73 m(2) (CKD group) were compared with a non-CKD group. For CKD patient safety indicators, hospitalizations were excluded if the admitting condition was a potential cause of the secondary condition. Regression methods were used to present adjusted rates in study groups of interest. RESULTS: The CKD patient safety indicators were generally more common than the Agency for Healthcare and Research Quality patient safety indicators in all groups, tended to occur in different patients than those patients who experienced Agency for Healthcare and Research Quality patient safety indicators, and were more common in the CKD group than the non-CKD group, except for hypoglycemia, hypokalemia, and hyponatremia. The adjusted composite CKD patient safety indicators rate (per 1000 patient-hospitalizations) was 398.0 (95% confidence interval, 391.2 to 405.0) for patients in the CKD group and 250.0 (95% confidence interval, 247.4 to 252.7) for patients in the non-CKD group. The prevalence ratio of CKD patient safety indicators to Agency for Healthcare and Research Quality patient safety indicators was 23.4 (95% confidence interval, 21.9 to 25.0). CONCLUSION: The candidate CKD patient safety indicators that occur in hospitalized patients are distinct from the Agency for Healthcare and Research Quality patient safety indicators and tend to be more common in CKD than non-CKD patients. These measures have the potential to serve as sentinel tools for identifying patients with CKD who warrant examination for disease-pertinent safety events.

Using present-on-admission coding to improve exclusion rules for quality metrics: the case of failure-to-rescue.

BACKGROUND: The Agency for Healthcare Research and Quality (AHRQ) patient safety indicator "death among surgical inpatients with serious treatable complications" (failure-to-rescue) uses rules to exclude complications presumed to be present-on-admission (POA). Like other administrative data-based quality measures, exclusion rules were developed with limited information on whether complications were POA. We examine whether the accuracy of failure-to-rescue exclusion rules can be improved with data with good POA indicators. METHODS: POA-coded data from 243,825 discharges from a large academic medical center were used to develop 3 failure-to-rescue exclusion rules. Data from 82,871 discharges from California hospitals screened for good POA coding practices was used as a validation sample. The AHRQ failure-to-rescue measure and 3 new measures based on alternative exclusion rules were compared on sensitivity, specificity, and C-statistics for prediction of POA status. Using data from the AHRQ HCUP National Inpatient Sample, the alternative specifications were tested for sensitivity to nurse staffing. RESULTS: The AHRQ exclusion rules had sensitivity of 18.5%, specificity 92.1%, and a C-statistic of 0.553. All POA-informed specifications of exclusion rules improved the C-statistic of the failure-to-rescue measure and its sensitivity, with modest losses of specificity. For all tested specifications, higher licensed hours and proportions of registered nurse were statistically significant and associated with lower risk of death. CONCLUSIONS: Failure-to-rescue is a robust quality measure, sensitive to nursing across alternative exclusion rule specifications. Despite expanded POA coding, exclusion-based rules are needed to analyze datasets not coded for POA, legacy datasets, and datasets with poor POA coding. POA-informed construction of exclusions significantly improves rules identifying POA complications.