Leveraging Patients' Creative Ideas for Innovation in Health Care.

Policy Points Patients' creative ideas may inform learning and innovation that improve patient-centered care. Routinely collected patient experience surveys provide an opportunity to invite patients to share their creative ideas for improvement. We develop and assess a methodological strategy that validates question wording designed to elicit creative ideas from patients. Health care organizations should consider how to report and use these data in health care delivery and quality improvement, and policymakers should consider promoting the use of narrative feedback to better understand and respond to patients' experiences. CONTEXT: Learning health systems (LHSs) have been promoted for a decade to achieve high-quality, patient-centered health care. Innovation driven by knowledge generated through day-to-day health care delivery, including patient insights, is critical to LHSs. However, the pace of translating patient insights into innovation is slow and effectiveness inadequate. This study aims to evaluate a method for systematically eliciting patients' creative ideas, examine the value of such ideas as a source of insight, and examine patients' creative ideas regarding how their experiences could be improved within the context of their own health systems. METHODS: The first stage of the study developed a survey and tested strategies for elicitation of patients' creative ideas with 600 patients from New York State. The second stage deployed the survey with the most generative open-ended question sequence within a health care system and involved analysis of 1,892 patients' responses, including 2,948 creative ideas. FINDINGS: Actionable, creative feedback was fostered by incorporating a request for transformative feedback into a sequence of narrative elicitation questions. Patients generate more actionable and creative ideas when explicitly invited to share such ideas, especially patients with negative health care experiences, those from minority racial/ethnic backgrounds, and those with chronic illness. The most frequently elicited creative ideas focused on solving challenges, proposing interventions, amplifying exceptional practices, and conveying hopes for the future. CONCLUSIONS: A valid and reliable method for eliciting creative ideas from patients can be deployed as part of routine patient experience surveys that include closed-ended survey items and open-ended narrative items in which patients share their experiences in their own words. The elicited creative ideas are promising for patient engagement and innovation efforts. This study highlights the benefits of engaging patients for quality improvement, offers a rigorously tested method for cultivating innovation using patient-generated knowledge, and outlines how creative ideas can enable organizational learning and innovation.

Assessing an innovative method to promote learning from patient narratives: Findings from a field experiment in ambulatory care.

OBJECTIVE: To assess whether an online interactive report designed to facilitate interpretation of patients' narrative feedback produces change in ambulatory staff learning, behavior at the individual staff and practice level, and patient experience survey scores. DATA SOURCES AND SETTING: We studied 22 ambulatory practice sites within an academic medical center using three primary data sources: 333 staff surveys; 20 in-depth interviews with practice leaders and staff; and 9551 modified CG-CAHPS patient experience surveys augmented by open-ended narrative elicitation questions. STUDY DESIGN: We conducted a cluster quasi-experimental study, comparing 12 intervention and 10 control sites. At control sites, narratives were delivered free-form to site administrators via email; at intervention sites, narratives were delivered online with interactive tools for interpretation, accompanied by user training. We assessed control-versus-intervention site differences in learning, behavior, and patient experience scores. DATA COLLECTION: Staff surveys and interviews were completed at intervention and control sites, 9 months after intervention launch. Patient surveys were collected beginning 4 months pre-launch through 9 months post-launch. We used control-versus-intervention and difference-in-difference analyses for survey data and thematic analysis for interview data. PRINCIPAL FINDINGS: Interviews suggested that the interface facilitated narrative interpretation and use for improvement. Staff survey analyses indicated enhanced learning from narratives at intervention sites (29% over control sites' mean of 3.19 out of 5 across eight domains, p < 0.001) and greater behavior change at staff and practice levels (31% and 21% over control sites' means of 3.35 and 3.39, p < 0.001, respectively). Patient experience scores for interactions with office staff and wait time information increased significantly at intervention sites, compared to control sites (3.7% and 8.2%, respectively); however, provider listening scores declined 3.3%. CONCLUSIONS: Patient narratives presented through structured feedback reporting methods can catalyze positive changes in staff learning, promote behavior change, and increase patient experience scores in domains of non-clinical interaction.

Learning from patients: The impact of using patients' narratives on patient experience scores.

BACKGROUND: Enthusiasm has grown about using patients' narratives-stories about care experiences in patients' own words-to advance organizations' learning about the care that they deliver and how to improve it, but studies confirming association have not been published. PURPOSE: We assessed whether primary care clinics that frequently share patients' narratives with their staff have higher patient experience survey scores. APPROACH: We conducted a 1-year study of 5,545 adult patients and 276 staff affiliated with nine clinics in one health system. We used multilevel models to analyze survey data from patients about their experiences and from staff about exposure to useful narratives. We examined staff confidence in own knowledge as a moderator because confidence can influence use of new information sources. RESULTS: Frequency of sharing useful narratives with staff was associated with patient experience scores for all measures, conditional on staff confidence in own knowledge ( p < .01). For operational measures (e.g., care coordination), increased sharing correlated with subsequently higher performance for more confident staff and lower performance or no difference for less confident staff, depending on measure. For relational measures (e.g., patient-provider communication), increased sharing correlated with higher scores for less confident staff and lower scores for more confident staff. CONCLUSION: Sharing narratives with staff frequently is associated with better patient experience survey scores, conditional on confidence in knowledge. PRACTICE IMPLICATIONS: Frequently sharing useful patient narratives should be encouraged as an organizational improvement strategy. However, organizations need to address how narrative feedback interacts with their staff's confidence to realize higher experience scores across domains.

Summary of the 2020 AHRQ research meeting on 'advancing methods of implementing and evaluating patient experience improvement using consumer assessment of healthcare providers and systems (CAHPS®) surveys'.

BACKGROUND: The Agency for Healthcare Research and Quality held a research meeting on using Consumer Assessment of Healthcare Providers and Systems (CAHPS®) data for quality improvement (QI) and evaluating such efforts. TOPICS COVERED.: Meeting addressed: 1)What has been learned about organizational factors/environment needed to improve patient experience? 2)How have organizations used data to improve patient experience? 3)What can evaluations using CAHPS data teach us about implementing successful programs to improve patient experience? KEY THEMES: Providers and stakeholders need to be engaged early and often, standardize QI processes, complement CAHPS data with other data, and compile dashboards of CAHPS scores to identify and track improvement. Rigorous study designs are valuable, but much can be learned and accomplished through practical organization-level studies.

A Rigorous Approach to Large-Scale Elicitation and Analysis of Patient Narratives.

Patient narratives have emerged as promising vehicles for making health care more responsive by helping clinicians to better understand their patients' expectations, perceptions, or concerns and encouraging consumers to engage with information about quality. A growing number of websites incorporate patients' comments. But existing comments have fragmentary content, fail to represent less vocal patients, and can be manipulated to "manage" providers' reputations. In this article, we offer the first empirical test of the proposition that patient narratives can be elicited rigorously and reliably using a five-question protocol that can be incorporated into large-scale patient experience surveys. We tested whether elicited narratives about outpatient care are complete (report all facets of patient experience), balanced (convey an accurate mix of positive and negative events), meaningful (have a coherent storyline), and representative (draw fulsome narratives from all relevant subsets of patients). The tested protocol is strong on balance and representativeness, more mixed on completeness and meaningfulness.

Americans' Growing Exposure To Clinician Quality Information: Insights And Implications.

For two decades, various initiatives have encouraged Americans to consider quality when choosing clinicians, both to enhance informed choice and to reduce disparities in access to high-quality providers. The literature portrays these efforts as largely ineffective. But this depiction overlooks two factors: the dramatic expansion since 2010 in the availability of patients' narratives about care and the growth of information seeking among consumers. Using surveys fielded in 2010, 2014, and 2015, we assessed the impact of these changes on consumers' awareness of quality information and sociodemographic differences. Public exposure to any quality information doubled between 2010 and 2015, while exposure to patient narratives and experience surveys tripled. Reflecting a greater propensity to seek quality metrics, minority consumers remained better informed than whites over time, albeit with differences across subgroups in the types of information encountered. An education-related gradient in quality awareness also emerged over the past decade. Public policy should respond to emerging trends in information exposure, establish standards for rigorous elicitation of narratives, and assist consumers' learning from a combination of narratives and quantified metrics on clinician quality.

What Words Convey: The Potential for Patient Narratives to Inform Quality Improvement.

Policy Points Narratives about patients' experiences with outpatient care are essential for quality improvement because they convey ample actionable information that both elaborates on existing domains within patient experience surveys and describes multiple additional domains that are important to patients. The content of narrative feedback from patients can potentially be translated to improved quality in multiple ways: clinicians can learn from their own patients, groups of clinicians can learn from the experience of their peers' patients, and health system administrators can identify and respond to patterns in patients' accounts that reflect systemic challenges to quality. Consistent investment by payers and providers is required to ensure that patient narratives are rigorously collected, analyzed fully, and effectively used for quality improvement. CONTEXT: For the past 25 years, health care providers and health system administrators have sought to improve care by surveying patients about their experiences. More recently, policymakers have acted to promote this learning by deploying financial incentives tied to survey scores. This article explores the potential of systematically elicited narratives about experiences with outpatient care to enrich quality improvement. METHODS: Narratives were collected from 348 patients recruited from a nationally representative Internet panel. Drawing from the literature on health services innovation, we developed a two-part coding schema that categorized narrative content in terms of (a) the aspects of care being described, and (b) the actionability of this information for clinicians, quality improvement staff, and health system administrators. Narratives were coded using this schema, with high levels of reliability among the coders. FINDINGS: The scope of outpatient narratives divides evenly among aspects of care currently measured by patient experience surveys (35% of content), aspects related to measured domains but not captured by existing survey questions (31%), and aspects of care that are omitted from surveys entirely (34%). Overall, the narrative data focused heavily on relational aspects of care (43%), elaborating on this aspect of experience well beyond what is captured with communication-related questions on existing surveys. Three-quarters of elicited narratives had some actionable content, and almost a third contained three or more separate actionable elements. CONCLUSIONS: In a health policy environment that incentivizes attention to patient experience, rigorously elicited narratives hold substantial promise for improving quality in general and patients' experiences with care in particular. They do so in two ways: by making concrete what went wrong or right in domains covered by existing surveys, and by expanding our view of what aspects of care matter to patients as articulated in their own words and thus how care can be made more patient-centered. Most narratives convey experiences that are potentially actionable by those committed to improving health care quality in outpatient settings.

Choosing Doctors Wisely: Can Assisted Choice Enhance Patients' Selection of Clinicians?

We conducted a simulated clinician-choice experiment, comparing choices and decision-making processes of participants (N = 688) randomized among four experimental arms: a conventional website reporting only quantitative performance information, a website reporting both qualitative (patient comments) and quantitative information, the second website augmented by a decision aid (labeling of patient comments), and the decision-aided website further augmented by the presence of a trained navigator. Introducing patient comments enhanced engagement with the quality information but led to a decline in decision quality, particularly the consistency of choices with consumers' stated preferences. Labeling comments helped erase the decline in decision quality, although the highest percentage of preference-congruent choices was seen in the navigator arm. Engagement with the quality information and satisfaction with choices available were likewise highest in the navigator arm. Findings held for high- and low-skilled decision makers. Thus, navigator assistance may be a promising strategy for equitably promoting higher quality choices in information-rich contexts.

Improving Patients' Choice of Clinician by Including Roll-up Measures in Public Healthcare Quality Reports: an Online Experiment.

BACKGROUND: Public reports on healthcare quality typically include complex data. To lower the cognitive burden of interpreting these data, some report designers create summary, or roll-up, measures combining multiple indicators of quality into one score. Little is known about how the availability of roll-ups affects clinician choice. OBJECTIVE: To determine how presenting quality scores at different levels of aggregation affects patients' clinician choices. DESIGN: We conducted a simulated clinician-choice experiment, randomizing participants to three versions of a public reporting website and comparing their clinician choices. One version aggregated all clinician-level quality measures into roll-ups, the second provided disaggregated (drill-down) scores only, and the third offered both roll-ups and drill-downs. PARTICIPANTS: Five hundred fifty panelists drawn from a probability-based Internet panel. MAIN MEASURES: We assessed the amount of effort participants exerted by tracking the length of time spent on the website and the number of concrete actions taken on the website (e.g., clicking items). We evaluated decision quality by measuring whether participants selected a clinician who performed more poorly than others and incongruence between participants' stated preferences for dimensions of quality and their chosen clinician's performance on those dimensions. KEY RESULTS: Participants seeing drill-downs alone (mean = 14.9) or with roll-ups (mean = 19.2) took more actions than those who saw roll-ups alone (mean = 10.5) (ps < 0.05). However, participants seeing only drill-downs made poorer choices than those who saw roll-ups alone or with drill-downs. More participants seeing drill-downs chose a clinician who was outperformed (36.3% versus 23.4% [roll-up] and 25.6% [drill-down + roll-up], ps < 0.05) and made choices incongruent with stated preferences (51.2% versus 45.6% [roll-up] and 47.5% [drill-down + roll-up], ps < 0.05). The distinction between roll-up and drill-down was somewhat stronger for sicker participants. CONCLUSIONS: Our results suggest that roll-ups in healthcare quality reports, alone or as a complement to drill-downs, can help patients make better decisions for themselves.

Using "roll-up" measures in healthcare quality reports: perspectives of report sponsors and national alliances.

OBJECTIVES: To understand the views of prominent organizations in the field of healthcare quality on the topic of reporting roll-up measures that combine indicators of multiple, often disparate, dimensions of care to consumers. STUDY DESIGN: This study used a semi-structured, qualitative interview design. METHODS: We conducted 30- to 60-minute semi-structured telephone interviews with representatives of 10 organizations that sponsor public healthcare quality reports and 3 national alliances representing multiple stakeholder groups. We conducted a thematic analysis of interview transcriptions to identify common issues and concerns related to reporting roll-up measures. RESULTS: Among sponsors reporting roll-up measures, current practices for calculating and reporting these measures are diverse. The main perceived benefit of reporting roll-up measures is that they simplify large amounts of complex information for consumers. The main concern is the potential for consumers to misunderstand the measures and what associated roll-up scores communicate about provider performance. Report sponsors and national alliances feel that more guidance and research on the methods for producing and reporting scores for roll-up measures are needed. CONCLUSIONS: The results of the interviews elucidate the need for research focused on construction and reporting of roll-up measures. Studies are needed to determine if roll-up measures are indeed perceived by consumers as being less complex and easier to understand.

CAHPS and Comments: How Closed-Ended Survey Questions and Narrative Accounts Interact in the Assessment of Patient Experience.

OBJECTIVES: To investigate whether content from patient narratives explains variation in patients' primary care provider (PCP) ratings beyond information from the closed-ended questions of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group Survey and whether the relative placement of closed- and open-ended survey questions affects either the content of narratives or the CAHPS composite scores. METHODS: Members of a standing Internet panel (N = 332) were randomly assigned to complete a CAHPS survey that was either preceded or followed by a set of open-ended questions about how well their PCP meets their expectations and how they relate to their PCP. RESULTS: Narrative content from healthier patients explained only an additional 2% beyond the variation in provider ratings explained by CAHPS composite measures. Among sicker patients, narrative content explained an additional 10% of the variation. The relative placement of closed- and open-ended questions had little impact on narratives or CAHPS scores. CONCLUSION: Incorporating a protocol for eliciting narratives into a patient experience survey results in minimal distortion of patient feedback. Narratives from sicker patients help explain variation in provider ratings.

Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives.

OBJECTIVE: To design a methodology for rigorously eliciting narratives about patients' experiences with clinical care that is potentially useful for public reporting and quality improvement. DATA SOURCES/STUDY SETTING: Two rounds of experimental data (N = 48 each) collected in 2013-2014, using a nationally representative Internet panel. STUDY DESIGN: Our study (1) articulates and operationalizes criteria for assessing narrative elicitation protocols; (2) establishes a "gold standard" for assessment of such protocols; and (3) creates and tests a protocol for narratives about outpatient treatment experiences. DATA COLLECTION/EXTRACTION METHODS: We randomized participants between telephone and web-based modalities and between protocols placed before and after a closed-ended survey. PRINCIPAL FINDINGS: Elicited narratives can be assessed relative to a gold standard using four criteria: (1) meaningfulness, (2) completeness, (3) whether the narrative accurately reflects the balance of positive and negative events, and (4) representativeness, which reflects the protocol's performance across respondent subgroups. We demonstrate that a five-question protocol that has been tested and refined yields three- to sixfold increases in completeness and four- to tenfold increases in meaningfulness, compared to a single open-ended question. It performs equally well for healthy and sick patients. CONCLUSIONS: Narrative elicitation protocols suitable for inclusion in extant patient experience surveys can be designed and tested against objective performance criteria, thus advancing the science of public reporting.

How Patient Comments Affect Consumers' Use of Physician Performance Measures.

BACKGROUND: Patients' comments about doctors are increasingly available on the internet. The effects of these anecdotal accounts on consumers' engagement with reports on doctor quality, use of more statistically reliable performance measures, and ability to choose doctors wisely are unknown. OBJECTIVE: To examine the effects of providing patient comments along with standardized performance information in a web-based public report. DESIGN: Participants were randomly assigned to view 1 of 6 versions of a website presenting comparative performance information on fictitious primary care doctors. Versions varied by the combination of information types [Consumer Assessment of Healthcare Providers and Systems (CAHPS), Healthcare Effectiveness Data and Information Set (HEDIS), and patient comments] and number of doctors. PARTICIPANTS: A random sample of working-age adults (N=848) from an online panel representing the noninstitutionalized population of the United States. MAIN MEASURES: Time spent and actions taken on the website, probing of standardized measures, and decision quality (chosen doctor rated highest on quantifiable metrics, chosen doctor not dominated by another choice). Secondary outcomes were perceived usefulness and trustworthiness of performance metrics and evaluations of the website. KEY RESULTS: Inclusion of patient comments increased time spent on the website by 35%-42% and actions taken (clicks) by 106%-117% compared with versions presenting only CAHPS and HEDIS measures (P<0.01). It also reduced participants' attention to standardized measures (eg, percentage of time probing HEDIS measures dropped by 67%, P<0.01). When patient comments were present, fewer participants chose the doctor scoring highest on standardized metrics (44%-49% vs. 61%-62%, P<0.01). CONCLUSIONS: Including patient comments in physician performance reports enhances consumers' engagement but reduces their attention to standardized measures and substantially increases suboptimal choices. More research is needed to explore whether integrated reporting strategies could leverage the positive effects of patient comments on consumer engagement without undermining consumers' use of other important metrics for informing choice among doctors.

Using Patient-Reported Information to Improve Clinical Practice.

OBJECTIVE: To assess what is known about the relationship between patient experience measures and incentives designed to improve care, and to identify how public policy and medical practices can promote patient-valued outcomes in health systems with strong financial incentives. DATA SOURCES/STUDY SETTING: Existing literature (gray and peer-reviewed) on measuring patient experience and patient-reported outcomes, identified from Medline and Cochrane databases; evaluations of pay-for-performance programs in the United States, Europe, and the Commonwealth countries. STUDY DESIGN/DATA COLLECTION: We analyzed (1) studies of pay-for-performance, to identify those including metrics for patient experience, and (2) studies of patient experience and of patient-reported outcomes to identify evidence of influence on clinical practice, whether through public reporting or private reporting to clinicians. PRINCIPAL FINDINGS: First, we identify four forms of "patient-reported information" (PRI), each with distinctive roles shaping clinical practice: (1) patient-reported outcomes measuring self-assessed physical and mental well-being, (2) surveys of patient experience with clinicians and staff, (3) narrative accounts describing encounters with clinicians in patients' own words, and (4) complaints/grievances signaling patients' distress when treatment or outcomes fall short of expectations. Because these forms vary in crucial ways, each must be distinctively measured, deployed, and linked with financial incentives. Second, although the literature linking incentives to patients experience is limited, implementing pay-for-performance systems appears to threaten certain patient-valued aspects of health care. But incentives can be made compatible with the outcomes patients value if: (a) a sufficient portion of incentives is tied to patient-reported outcomes and experiences, (b) incentivized forms of PRI are complemented by other forms of patient feedback, and (c) health care organizations assist clinicians to interpret and respond to PRI. Finally, we identify roles for the public and private sectors in financing PRI and orchestrating an appropriate balance among its four forms. CONCLUSIONS: Unless public policies are attentive to patients' perspectives, stronger financial incentives for clinicians can threaten aspects of care that patients most value. Certain policy parameters are already clear, but additional research is required to clarify how best to collect patient narratives in varied settings, how to report narratives to consumers in conjunction with quantified metrics, and how to promote a "culture of learning" at the practice level that incorporates patient feedback.

Complexity, public reporting, and choice of doctors: a look inside the blackest box of consumer behavior.

Health care consumers often make choices that are imperfectly informed and inconsistent with their expressed preferences. Past research suggests that these shortcomings become more pronounced as choices become more complex, through either additional options or more performance metrics. But it is unclear why this is true: Consumer choice remains a "black box" that research has scarcely illuminated. In this article, we identify four pathways through which complexity may impair consumer choice. We examine these pathways using data from an experiment in which consumers (hypothetically) selected a primary care physician. Some of the loss of decision quality accompanying more complex choice sets can be explained by consumers' skills and decision-making style, but even after accounting for these factors, complexity undermines the quality of decision making in ways that cannot be fully explained. We conclude by discussing implications for report designers, sponsors, and policy makers aspiring to promote consumer empowerment and health care quality.

Context-based strategies for engaging consumers with public reports about health care providers.

Efforts to engage consumers in the use of public reports on health care provider performance have met with limited success. Fostering greater engagement will require new approaches that provide consumers with relevant content at the time and in the context they need to make a decision of consequence. To this end, we identify three key factors influencing consumer engagement and show how they manifest in different ways and combinations for four particular choice contexts that appear to offer realistic opportunities for engagement. We analyze how these engagement factors play out differently in each choice context and suggest specific strategies that sponsors of public reports can use in each context. Cross-cutting lessons for report sponsors and policy makers include new media strategies such as a commitment to adaptive web-based reporting, new metrics with richer emotional content, and the use of navigators or advocates to assist consumers with interpreting reports.

Consumer response to patient experience measures in complex information environments.

BACKGROUND: As indicators of clinician quality proliferate, public reports increasingly include multiple metrics. This approach provides more complete performance information than did earlier reports but may challenge consumers' ability to understand and use complicated reports. OBJECTIVES: To assess the effects of report complexity on consumers' understanding and use of patient experience measures derived from the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) survey. RESEARCH DESIGN: In an Internet-based experiment, participants were asked to compare information on physician quality and choose a primary care doctor. Participants were randomly assigned to choice sets of varied complexity (CAHPS alone vs. CAHPS with other measures) and number of doctors. Participants completed surveys before and after this choice task. SUBJECTS: A total of 555 US residents, aged 25-64, who had Internet access through computer were recruited from an existing online panel. MEASURES: Recall seeing CAHPS measures; use of CAHPS measures for making choices; ratings of ease of use, usefulness and trustworthiness of CAHPS ratings; concerns about usefulness and trustworthiness. RESULTS: Participants presented with CAHPS information and other performance indicators relied less on CAHPS than did those presented with CAHPS information only, although they considered CAHPS information as valuable as did other respondents. Participants presented with smaller choice sets also judged CAHPS information as less easy to use when accompanied by other metrics than when it was presented alone.

Improving organizational climate for quality and quality of care: does membership in a collaborative help?

BACKGROUND: The lack of quality-oriented organizational climates is partly responsible for deficiencies in patient-centered care and poor quality more broadly. To improve their quality-oriented climates, several organizations have joined quality improvement collaboratives. The effectiveness of this approach is unknown. OBJECTIVE: To evaluate the impact of collaborative membership on organizational climate for quality and service quality. SUBJECTS: Twenty-one clinics, 4 of which participated in a collaborative sponsored by the Institute for Clinical Systems Improvement. RESEARCH DESIGN: Pre-post design. Preassessments occurred 2 months before the collaborative began in January 2009. Postassessments of service quality and climate occurred about 6 months and 1 year, respectively, after the collaborative ended in January 2010. We surveyed clinic employees (eg, physicians, nurses, receptionists, etc.) about the organizational climate and patients about service quality. MEASURES: Prioritization of quality care, high-quality staff relationships, and open communication as indicators of quality-oriented climate and timeliness of care, staff helpfulness, doctor-patient communication, rating of doctor, and willingness to recommend doctor's office as indicators of service quality. RESULTS: There was no significant effect of collaborative membership on quality-oriented climate and mixed effects on service quality. Doctors' ratings improved significantly more in intervention clinics than in control clinics, staff helpfulness improved less, and timeliness of care declined more. Ratings of doctor-patient communication and willingness to recommend doctor were not significantly different between intervention and comparison clinics. CONCLUSION: Membership in the collaborative provided no significant advantage for improving quality-oriented climate and had equivocal effects on service quality.