How Accurate is the AHRQ Patient Safety Indicator for Hospital-Acquired Pressure Ulcer in a National Sample of Records?

In 2008, we conducted a retrospective cross-sectional study to determine the test characteristics of the Agency for Healthcare Research and Quality patient safety indicator (PSI) for hospital-acquired pressure ulcer (PU). We sampled 1,995 inpatient records that met PSI 3 criteria and 4,007 records assigned to 14 DRGs with the highest empirical rates of PSI 3, which did not meet PSI 3 criteria, from 32 U.S. academic hospitals. We estimated the positive predictive value (PPV), sensitivity, and specificity of PSI 3 using both the software version contemporary to the hospitalizations (v3.1) and an approximation of the current version (v4.4). Of records that met PSI 3 version 3.1 criteria, 572 (PPV 28.3%; 95% CI 23.6-32.9%) were true positive. PU that was present on admission (POA) accounted for 76% of the false-positive records. Estimated sensitivity was 48.2% (95% CI 41.0-55.3%) and specificity 71.4% (95% CI 68.3-74.5%). Reclassifying records based on reported POA information and PU stage to approximate version 4.4 of PSI 3 improved sensitivity (78.6%; 95% CI 62.7-94.5%) and specificity (98.0; 95% CI 97.1-98.9%). In conclusion, accounting for POA information and PU staging to approximate newer versions of the PSI software (v4.3) moderately improves validity.

Variation in academic medical centers' coding practices for postoperative respiratory complications: implications for the AHRQ postoperative respiratory failure Patient Safety Indicator.

BACKGROUND: The Agency for Healthcare Research and Quality Patient Safety Indicator (PSI) 11 uses International Classification of Disease, 9th Clinical Modification diagnosis code 518.81 ("Acute respiratory failure")-but not the closely related alternative, 518.5 ("Pulmonary insufficiency after trauma and surgery")-to detect cases of postoperative respiratory failure. We sought to determine whether hospitals vary in the use of 518.81 versus 518.5 and whether such variation correlates with coder beliefs. STUDY DESIGN: We conducted a cross-sectional analysis of administrative data from July 2009 through June 2010 for UHC (formerly University HealthSystem Consortium)-affiliated centers to assess the use of diagnosis codes 518.81 and 518.5 in PSI 11-eligible cases. We also surveyed coders at these centers to evaluate whether variation in the use of 518.81 versus 518.5 might be linked to coder beliefs. We asked survey respondents which diagnosis they would use for 2 ambiguous cases of postoperative pulmonary complications and how much they agreed with 6 statements about the coding process. RESULTS: UHC-affiliated centers demonstrated wide variation in the use of 518.81 and 518.5, ranging from 0 to 26 cases and 0 to 56 cases/1000 PSI 11-eligible hospitalizations, respectively. Of 56 survey respondents, 64% chose 518.81 and 30% chose 518.5 for a clinical scenario involving postoperative respiratory failure, but these responses were not associated with actual coding of 518.81 or 518.5 at the center level. Sixty-two percent of respondents agreed that they are constrained by the words that physicians use. Their self-reported likelihood of querying physicians to clarify the diagnosis was significantly associated with coding of 518.5 at the center level. CONCLUSIONS: The extent to which diagnosis code 518.81 is used relative to 518.5 varies considerably across centers, based on local coding practice, the specific wording of physician documentation, and coder-physician communication. To standardize the coding of postoperative respiratory failure, the 518.81 and 518.5 codes have recently been revised to make the available options clearer and mutually exclusive, which may improve the capacity of PSI 11 to discriminate true differences in quality of care.

Using the Agency for Healthcare Research and Quality patient safety indicators for targeting nursing quality improvement.

Quantifying the critical impact nurses have on the prevention and early recognition of potential complications and adverse events, such as those identified by the Agency for Healthcare Research and Quality (AHRQ) patient safety indicators (PSI), is becoming increasingly important. In this paper, we describe how the AHRQ PSI may be used to identify nursing-specific opportunities to improve care based on data from the national AHRQ PSI validation pilot project.

Designing an abstraction instrument: lessons from efforts to validate the AHRQ patient safety indicators.

BACKGROUND: The U.S. Agency for Healthcare Research and Quality (AHRQ) and other organizations have developed quality indicators based on hospital administrative data. Characteristics of effective abstraction instruments were identified for determining both the positive predictive value (PPV) of Patient Safety Indicators (PSIs) and the extent to which hospitals and clinicians could have prevented adverse events. METHODS: Through an iterative process involving nurse abstractors, physicians, and nurses with quality improvement experience, and health services researchers, 25 abstraction instruments were designed for 12 AHRQ provider-level morbidity PSIs. Data were analyzed from 13 of these instruments, and data are being collected using several more. FINDINGS: Common problems in designing the instruments included avoiding uninformative questions and premature termination of the abstraction process, anticipating misinterpretation of questions, allowing an appropriate range of response options; using clear terminology, optimizing the flow of the abstraction process, balancing the utility of data against abstractor burden, and recognizing the needs of end users, such as hospitals and quality improvement professionals and researchers, for the abstracted information. CONCLUSIONS: Designing medical record abstraction instruments for quality improvement research involves several potential pitfalls. Understanding how we addressed these challenges might help both investigators and users of outcome indicators to appreciate the strengths and limitations of outcome-based quality indicators and tools designed to validate or investigate such indicators within provider organizations.

Detection of postoperative respiratory failure: how predictive is the Agency for Healthcare Research and Quality's Patient Safety Indicator?

BACKGROUND: Patient Safety Indicator (PSI) 11, or postoperative respiratory failure, was developed by the US Agency for Healthcare Research and Quality to detect incident cases of respiratory failure after elective operations through use of ICD-9-CM diagnosis and procedure codes. We sought to determine the positive predictive value (PPV) of this indicator. STUDY DESIGN: We conducted a retrospective cross-sectional study, sampling consecutive cases that met PSI 11 criteria from 18 geographically diverse academic medical centers on or before June 30, 2007. Trained abstractors from each center reviewed medical records using a standard instrument. We assessed the PPV of the indicator (with 95% CI adjusted for clustering within centers) and conducted descriptive analyses of the cases. RESULTS: Of 609 cases that met PSI 11 criteria, 551 (90.5%; 95% CI, 86.5-94.4%) satisfied the technical criteria of the indicator and 507 (83.2%; 95% CI, 77.2-89.3%) represented true cases of postoperative respiratory failure from a clinical standpoint. The most frequent reasons for being falsely positive were nonelective hospitalization, prolonged intubation for airway protection, and insufficient evidence to support a diagnosis of acute respiratory failure. Fifty percent of true-positive cases involved substantial baseline comorbidities, and 23% resulted in death. CONCLUSIONS: Although PSI 11 predicts true postoperative respiratory failure with relatively high frequency, the indicator does not limit detection to preventable cases. The PPV of PSI 11 might be increased by excluding cases with a principal diagnosis suggestive of a nonelective hospitalization and those with head or neck procedures. Removing the diagnosis code criterion from the indicator might also increase PPV, but would decrease the number of true positive cases detected by 20%.

Evaluation of the predictive value of ICD-9-CM coded administrative data for venous thromboembolism in the United States.

OBJECTIVE: To determine the positive predictive value of International Classification of Disease, 9th Revision, Clinical Modification (ICD-9-CM) discharge codes for acute deep vein thrombosis or pulmonary embolism. MATERIALS AND METHODS: Retrospective review of 3456 cases hospitalized between 2005 and 2007 that had a discharge code for venous thromboembolism, using 3 sample populations: a single academic hospital, 33 University HealthSystem Consortium hospitals, and 35 community hospitals in a national Joint Commission study. Analysis was stratified by position of the code in the principal versus a secondary position. RESULTS: Among 1096 cases that had a thromboembolism code in the principal position the positive predictive value for any acute venous thrombosis was 95% (95%CI:93-97), whereas among 2360 cases that had a thromboembolism code in a secondary position the predictive value was lower, 75% (95%CI:71-80). The corresponding positive predictive values for lower extremity deep-vein thrombosis or pulmonary embolism were 91% (95%CI:86-95) and 50% (95%CI:41-58), respectively. More highly defined codes had higher predictive value. Among codes in a secondary position that were false positive, 22% (95%CI:16-27) had chronic/prior venous thrombosis, 15% (95%CI:10-19) had an upper extremity thrombosis, 6% (95%CI:4-8) had a superficial vein thrombosis, and 7% (95%CI:4-13) had no mention of any thrombosis. CONCLUSIONS: ICD-9-CM codes for venous thromboembolism had high predictive value when present in the principal position, and lower predictive value when in a secondary position. New thromboembolism codes that were added in 2009 that specify chronic thrombosis, upper extremity thrombosis and superficial venous thrombosis should reduce the frequency of false-positive thromboembolism codes.

Factors associated with increased healthcare worker influenza vaccination rates: results from a national survey of university hospitals and medical centers.

OBJECTIVE: To ascertain which components of healthcare worker (HCW) influenza vaccination programs are associated with higher vaccination rates. DESIGN: Survey. SETTING: University-affiliated hospitals. METHODS: Participating hospitals were surveyed with regard to their institutional HCW influenza vaccination program for the 2007-2008 influenza season. Topics assessed included vaccination adherence and availability, use of declination statements, education methods, accountability, and data reporting. Factors associated with higher vaccination rates were ascertained. RESULTS: Fifty hospitals representing 368,696 HCWs participated in the project. The median vaccination rate was 55.0% (range, 25.6%-80.6%); however, the types of HCWs targeted by vaccination programs varied. Programs with the following components had significantly higher vaccination rates: weekend provision of vaccine (58.8% in those with this feature vs 43.9% in those without; P = .01), train-the-trainer programs (59.5% vs 46.5%; P = .005), report of vaccination rates to administrators (57.2% vs 48.1%; P = .04) or to the board of trustees (63.9% vs 53.4%; P = .01), a letter sent to employees emphasizing the importance of vaccination (59.3% vs 47%; P = .01), and any form of visible leadership support (57.9% vs 36.9%; P = .01). Vaccination rates were not significantly different between facilities that did and those that did not require a signed declination form for HCWs who refused vaccination (56.9% vs 55.1%; P = .68), although the precise content of such statements varied. CONCLUSIONS: Vaccination programs that emphasized accountability to the highest levels of the organization, provided weekend access to vaccination, and used train-the-trainer programs had higher vaccination coverage. Of concern, the types of HCWs targeted by vaccination programs differed, and uniform definitions will be essential in the event of public reporting of vaccination rates.

A review of quality of care evaluation for the palliation of dyspnea.

Assessment and management of dyspnea has emerged as a priority topic for quality evaluation and improvement. Evaluating dyspnea quality of care requires valid, reliable, and responsive measures of the care provided to patients across settings and diseases. As part of an Agency for Healthcare Research and Quality Symposium, we reviewed quality of care measures for dyspnea by compiling quality measures identified in systematic searches and reviews. Systematic reviews identified only three existing quality measurement sets that included quality measures for dyspnea care. The existing dyspnea quality measures reported by retrospective evaluations of care assess only four aspects: dyspnea assessment within 48 hours of hospital admission, use of objective scales to rate dyspnea severity, identification of management plans, and evidence of dyspnea reduction. To begin to improve care, clinicians need to assess and regularly document patient's experiences of dyspnea. There is no consensus on how dyspnea should be characterized for quality measurement, and although over 40 tools exist to assess dyspnea, no rating scale or instrument is ideal for palliative care. The panel recommended that dyspnea assessment should include a measure of intensity and some inquiry into the associated bother or distress experienced by the patient. A simple question into the presence or absence of dyspnea would be unlikely to help guide therapy, as complete relief of dyspnea in advanced disease would not be anticipated. Additional knowledge gaps include standards for clinical dyspnea care, assessment in the cognitively impaired, and evaluation of effectiveness of dyspnea care for patients with advanced disease.

How valid is the ICD-9-CM based AHRQ patient safety indicator for postoperative venous thromboembolism?

BACKGROUND: Hospital administrative data are being used to identify patients with postoperative venous thromboembolism (VTE), either pulmonary embolism (PE) or deep-vein thrombosis (DVT). However, few studies have evaluated the accuracy of these ICD-9-CM codes across multiple hospitals. METHODS AND MATERIALS: The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator (PSI)-12 was used to identify cases with postoperative VTE in 80 hospitals that volunteered for either an AHRQ or University HealthSystem Consortium (UHC) validation project. Trained abstractors using a standardized tool and guidelines retrospectively verified all coded VTE events. RESULTS: In the combined samples, the positive predictive value of the set of prespecified VTE codes for any acute VTE at any time during the hospitalization was 451 of 573 = 79% (95% CI: 75%-82%). However, the positive predictive value for acute lower extremity DVT or PE diagnosed after an operation was 209 of 452 = 44% (95% CI: 37%-51%) in the UHC sample and 58 of 121 = 48% (95% CI: 42-67%) in the AHRQ sample. Fourteen percent of all cases had an acute upper extremity DVT, 6% had superficial vein thrombosis and 21% had no acute VTE, however, 61% of the latter had a documented prior/chronic VTE. In the UHC cohort, the sensitivity for any acute VTE was 95.5% (95% CI: 86.4%-100%); the specificity was 99.5% (95% CI: 99.4%-99.7%). CONCLUSION: Current PSI 12 criteria do not accurately identify patients with acute postoperative lower extremity DVT or PE. Modification of the ICD-9-CM codes and implementation of "present on admission" flags should improve the predictive value for clinically important VTE events.

Evaluation of hospital glycemic control at US academic medical centers.

OBJECTIVE: To evaluate contemporary hospital glycemic management in US academic medical centers. DESIGN: This retrospective cohort study was conducted on patients discharged from 37 academic medical centers between July 1 and September 30, 2004; 1,718 eligible adult patients met at least 1 of the inclusion criteria: 2 consecutive blood glucose readings >180 mg/dL within 24 hours, or insulin treatment at any time during hospitalization. We assessed 3 consecutive measurement days of glucose values, glycemic therapy, and additional clinical and laboratory characteristics. RESULTS: In this diverse cohort, 79% of patients had a prior diagnosis of diabetes, and 84.6% received insulin on the second measurement day. There was wide variation in hospital performance of recommended hospital diabetes care measures such as glycosylated hemoglobin (A1C) assessment (range, 3%-63%) and timely admission laboratory glucose measurement (range, 39%-97%). Median glucose was significantly lower for patients in the intensive care unit (ICU) compared to ward/intermediate care. ICU patients treated with intravenous insulin had significantly lower median glucose when compared to subcutaneous insulin. Only 25% of ICU patients on day 3 had estimated 6 AM glucose or=1 glucose measurement >or=180 mg/dL on measurement days 2 and 3. Severe hypoglycemia (<50 mg/dL) occurred in 2.8% of all patient days. CONCLUSIONS: Despite frequent insulin use, glucose control was suboptimal. Academic medical centers have opportunities to improve care to meet current American Diabetes Association hospital diabetes care standards.

Outcomes of an initial set of standardized performance measures for inpatient mental health.

BACKGROUND: In January 2006, the University HealthSystem Consortium (UHC) convened a committee of experts from academic health centers to identify an initial set of important standardized performance measures for inpatient psychiatric services and to evaluate the current state of performance in these measures at eight academic health centers. METHOD: The eight UHC academic medical centers completed a retrospective review of 20 inpatient psychiatric records on patients who were 18-65 years of age with a primary diagnosis of psychosis and a length of stay > or = 2 days. The performance measures, derived from practice standards and the consensus of an interdisciplinary committee of experts, focused on the processes of care, including screening, assessment, treatment, coordination, continuity, and safety. RESULTS: Although there was variability in organizational performance in a number of the psychiatric measures, some organizations demonstrated high levels of performance. Performance measures indicating the greatest improvement opportunities for organizations included notification of outpatient mental health provider of the psychiatric hospitalization within two days; collaboration with the outpatient mental health provider and/or primary care physician; and scheduling a follow-up appointment within seven days of discharge. DISCUSSION: This initial benchmarking project in mental health at academic health centers shows that there is a range of conformity to important processes of care in the inpatient mental health setting. The results of the notification, collaboration, and continuity measures in this study highlight national concerns regarding the lack of communication and collaboration between providers in the transition through the continuum of services. Future quality measurement projects in mental health services should integrate clinical process measures with outcome measures.

University HealthSystem Consortium quality performance benchmarking study of the insertion and care of central venous catheters.

We report data from an observational benchmarking study of adherence to recommended practices for insertion and maintenance of central venous catheters at a heterogeneous group of academic medical centers. These centers demonstrated a need for significant improvement in implementation and documentation of quality performance measures for the prevention of catheter-related bloodstream infections.

Failure to rescue: validation of an algorithm using administrative data.

BACKGROUND: Failure to rescue (FTR), the rate of death in patients suffering 1 of 6 in-hospital complications, is an Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator calculated from administrative data. OBJECTIVE: : We sought to assess the accuracy of the AHRQ FTR algorithm. METHODS: We undertook a retrospective chart review of 60 denominator cases of FTR identified by the algorithm at each of 40 University HealthSystem Consortium institutions. The primary outcome was the overall accuracy of the algorithm compared with chart review. We also assessed accuracy by complication type, patient characteristics, institution, service assignment, and mortality. RESULTS: Of 2354 cases, 1193 (50.7%) were accurately identified by the algorithm as having had at least one of the FTR-qualifying complications during hospitalization. Of the 3073 complications identified in these patients, 1497 (48.7%) were correctly flagged by the algorithm, 907 (29.5%) were present on admission, 419 (13.6%) were not confirmed by chart review, and 250 (8.1%) met a predefined complication-specific criterion for exclusion. The case accuracy rate varied significantly by institution (mean, 50.7%; range, 18.3-100%; P < 0.001), service assignment (surgical service, 62.9% vs. nonsurgical service, 42.9%; P < 0.001), and mortality (alive, 43.9% vs. dead, 67.5%; P < 0.001) but was not affected by patients' age, gender, race, or insurance status. CONCLUSIONS: As currently calculated from administrative data, the FTR algorithm misidentifies half of the cases on average, is least accurate for nonsurgical cases, and is widely variable across institutions. This indicator may be useful internally to flag possible cases of quality failure but has limitations for external institutional comparisons. Improvements in coding quality and consistency across institutions are needed.

Palliative care benchmarks from academic medical centers.

INTRODUCTION: Palliative care is growing in the United States but little is known about the quality of care delivered. OBJECTIVE: To benchmark the quality of palliative care in academic hospitals. DESIGN: Multicenter, cross-sectional, retrospective chart review conducted between October 1, 2002 and September 30, 2003. SETTING: Thirty-five University HealthSystem Consortium (UHC) academic hospitals across the United States. PARTICIPANTS: A total of 1596 patient records. INCLUSION CRITERIA: (1) adults, (2) high-mortality diagnoses: selected cancers, heart failure, human immunodeficiency virus (HIV), and respiratory conditions requiring ventilator support, (3) length of stay (LOS) more than 4 days, and (4) two prior admissions in the preceding 12 months. MAIN OUTCOME MEASURES: Compliance with 11 key performance measures (KPM) derived from practice standards, literature evidence, and input from a multidisciplinary expert committee. Analyses examined relationships between provision of the KPM and specific outcomes. RESULTS: Wide variability exists among academic hospitals in the provision of the KPM (0%-100%). The greater the compliance with KPM, the greater the improvement in quality outcomes, cost and LOS. Assessment of pain (96.1%) and dyspnea (90.2%) was high, but reduction of these symptoms was lower (73.3% and 77.2%). Documentation of prognosis (33.4%), psychosocial assessment (26.2%), communication with family/patient (46%), and timely planning for discharge disposition (53.4%) were low for this severely ill population (16.8% hospital mortality). Only 12.9% received a palliative care consultation. CONCLUSIONS: The study reveals significant opportunities for improvement in the effective delivery of palliative care. Care that met KPM was associated with improved quality, reduced costs and LOS. Institutions that benchmarked above 90% did so by integrating KPM into daily care processes and utilizing systematized triggers, forms and default pathways. The presence of a formalized palliative care program within a hospital system had a positive effect on the achievement of KPM, whether or not formal consultation occurred. Hospitals need to develop systematic methods to improve access to palliative care.

The relationship between evidence-based practices and survival in patients requiring prolonged mechanical ventilation in academic medical centers.

Studies suggest variable adoption of evidence-based practice guidelines. The authors hypothesized that compliance with guidelines for patients requiring mechanical ventilation would vary among academic medical centers and that this variation might be associated with survival. A total of 1463 intensive care unit cases receiving continuous mechanical ventilation for >96 hours were reviewed. The variation in mortality based on compliance with 6 evidence-based practices was determined, and the effect of each intervention was estimated using a logistic regression model. Compliance varied widely across the participating centers. A strong association with survival was seen for 2 of the 6 practices: sedation management and glycemic control (odds ratios for death of 0.30 and 0.46, respectively, each P < .01). Spontaneous breathing trials, deep venous thrombosis prophylaxis, semi-recumbent positioning, and stress ulcer prophylaxis were not associated with survival in the model. More consistent adoption of these practices represents an opportunity for academic medical centers and was associated with enhanced survival.