Identifying quality improvement intervention evaluations: is consensus achievable?

BACKGROUND: The diversity of quality improvement interventions (QIIs) has impeded the use of evidence review to advance quality improvement activities. An agreed-upon framework for identifying QII articles would facilitate evidence review and consensus around best practices. AIM: To adapt and test evidence review methods for identifying empirical QII evaluations that would be suitable for assessing QII effectiveness, impact or success. DESIGN: Literature search with measurement of multilevel inter-rater agreement and review of disagreement. METHODS: Ten journals (2005-2007) were searched electronically and the output was screened based on title and abstract. Three pairs of reviewers then independently rated 22 articles, randomly selected from the screened list. Kappa statistics and percentage agreement were assessed. 12 stakeholders in quality improvement, including QII experts and journal editors, rated and discussed publications about which reviewers disagreed. RESULTS: The level of agreement among reviewers for identifying empirical evaluations of QII development, implementation or results was 73% (with a paradoxically low kappa of 0.041). Discussion by raters and stakeholders regarding how to improve agreement focused on three controversial article selection issues: no data on patient health, provider behaviour or process of care outcomes; no evidence for adaptation of an intervention to a local context; and a design using only observational methods, as correlational analyses, with no comparison group. CONCLUSION: The level of reviewer agreement was only moderate. Reliable identification of relevant articles is an initial step in assessing published evidence. Advancement in quality improvement will depend on the theory- and consensus-based development and testing of a generalizable framework for identifying QII evaluations.

Finding order in heterogeneity: types of quality-improvement intervention publications.

BACKGROUND: Stakeholders in quality improvement agree on the need for augmenting and synthesising the scientific literature supporting it. The diversity of perspectives, approaches, and contexts critical to advancing quality improvement science, however, creates challenges. The paper explores the heterogeneity in clinical quality improvement intervention (QII) publications. METHODS: A preliminary classification framework was developed for QII articles, aiming for categories homogeneous enough to support coherent scientific discussion on QII reporting standards and facilitate systematic review. QII experts were asked to identify articles important to QII science. The framework was tested and revised by applying it to the article set. The final framework screened articles into (1) empirical literature on development and testing of QIIs; (2) QII stories, theories, and frameworks; (3) QII literature syntheses and meta-analyses; or (4) development and testing of QII-related tools. To achieve homogeneity, category (1) required division into (1a) development of QIIs; 1(b) history, documentation, or description of QIIs; or (1c) success, effectiveness or impact of QIIs. RESULTS: By discussing unique issues and established standards relevant to each category, QII stakeholders can advance QII practice and science, including the scope and conduct of systematic literature reviews.

Organizational systems used by California capitated medical groups and independent practice associations to increase cancer screening.

BACKGROUND: Patients in health maintenance organizations (HMOs) appear to have higher utilization of cancer screening tests than patients with fee-for-service insurance. METHODS: The authors surveyed the medical directors of 174 physician organizations in a California network model HMO to obtain information regarding their organizational structure, implementation of guidelines, and use of systems to increase cancer screening. RESULTS: The majority of independent practice associations (IPAs) and medical groups (IMGs) in this California HMO had guidelines and office systems aimed at improving cancer screening. These activities were reported more frequently for mammography and Papanicolaou (Pap) smears than for colorectal carcinoma screening. IMGs reported using flow sheets more often than IPAs. Chart audits were performed more frequently for mammography (48% for IMGs and 40% for IPAs) and Pap smears (45% and 40%, respectively) than for colorectal carcinoma screening (38% and 30%, respectively). Approximately 50% of IPAs and IMGs reported mailing reminders to patients for mammography and Pap smears, but only a few did so for colorectal carcinoma screening. Annual fecal occult blood testing was believed by most medical directors to be a reasonable strategy for managed care patients (86% of IPAs and 96% of IMGs); however, fewer believed that screening sigmoidoscopy for patients ages 50-70 years was a reasonable expectation (71% and 78%, respectively). CONCLUSIONS: The majority of IPAs and IMGs in this California HMO reported using both guidelines and office systems to improve cancer screening rates. Further research is needed to understand the effect of these systems, as well their complex interactions with competing incentives, on cancer screening in managed care patients.

Distress and perceived health: mechanisms of health decline.

Stress is a common experience in modern society, and it can affect both physical and mental health. Recognizing that not all stress is detrimental to health, this research examines the relationship between perceptions of distress and perceived health within a longitudinal framework. Using two waves of a nationally representative panel study, the National Health and Nutrition Examination Survey I (NHANES I), structural equation modeling revealed that distress leads to more negative health perceptions. In addition, perceived health was found to impact distress levels at the following wave suggesting a cycle of decline between distress and perceived health. Finally, perceived health was found to have predictive validity in determining future functional disability even when considering distress.

Health trajectories: long-term dynamics among black and white adults.

Disability has long been identified as a predictor of self-assessed health, but some studies suggest the opposite causal direction. The aim of this study is to examine the dynamic relationships between physical disability and assessments of health among Black and White adults while simultaneously considering changing morbidity. Research questions include: Do more negative health assessments lead to greater morbidity and physical disability? Do negative health assessments lead to a cycle of health decline over time? These questions were addressed for Black and White respondents over 15 years using data from three waves of the National Health and Nutrition Examination Survey I: Epidemiologic Follow-Up Study. Results from structural equation modeling reveal that self-assessed health predicts subsequent change in health, suggesting a cycle between health problems and negative health assessments for both White and Black adults. In addition, self-assessed health among African Americans declined at a faster rate than was the case for White adults.

Linking Medicare and national survey data.

Administrative records from the Medicare Program of the Health Care Financing Administration provide a valuable source of information for research on medical and public policy issues. This administrative database contains information on utilization of covered medical services, diagnoses, episodes of illness, and Medicare-covered costs of health care. Combining such data with information from national surveys on health status, demographics, and socioeconomic attributes substantially expands the scope of potential research questions that can be addressed. This article discusses the benefits and difficulties of linking Medicare administrative data with survey data and provides brief summaries of five national surveys of elderly U.S. citizens. These surveys can be valuable resources for examining the health status and life experiences of the Medicare population.

Double jeopardy, aging as leveler, or persistent health inequality? A longitudinal analysis of white and black Americans.

Longitudinal data from a 15-year national survey of adults are used to test the double jeopardy to health hypothesis as well as the alternative hypotheses that aging levels ethnic differences and that health inequalities persist across the life course. Findings show that African Americans began the study in poorer health and manifested higher mortality. Among survivors, young and middle-aged Black adults developed more serious illness and their subjective health declined more rapidly than their Whites counterparts during the first 10 years of the study. Among subjects with heart failure, Black people were also more likely than Whites to become more disabled over time. Depending upon the health measure considered and the analytic strategy applied, there was evidence for all three processes, but the weight of the evidence shows persistent health inequality over the life course. Indeed, evidence for the age-as-leveler thesis was eliminated by accounting for mortality during the study.

Double jeopardy to health hypothesis for African Americans: analysis and critique.

Considerable research on minority health has examined whether members of a minority group experience more rapid health declines than the White majority when both groups reach later life. Researchers have sought to determine if being both old and a member of a minority creates a double disadvantage to health. The primary purpose of this research is to test the double jeopardy to health hypothesis among Black and White Americans using data from a 15-year panel study of adults: National Health and Nutrition Examination Survey I: Epidemiological Follow-up Study (NHEFS). African Americans have poorer health at all three times on a variety of health status measures, but no evidence for double jeopardy was uncovered. There were important racial differences for change in health status but Black Americans of all ages -- including Black older adults -- suffered from growing disability and more negative ratings of health. Black adults are more likely to develop serious illness, and their ratings of health decline more rapidly than is the case for White respondents. While there is little support for the double jeopardy hypothesis as originally stated, there is overwhelming evidence to show that the health of Black Americans of all ages declines at a faster rate. The formulation of the double jeopardy hypothesis is critiqued on several points: ontogenetic fallacy, attribution of discrimination, and selective mortality.