Evaluation of a protocol for eliciting narrative accounts of pediatric inpatient experiences of care.

OBJECTIVE: To evaluate the measurement properties of a set of six items designed to elicit narrative accounts of pediatric inpatient experience. DATA SOURCES: Data came from 163 participants recruited from a probability-based online panel of U.S. adults. Participants were family members of a child who had an overnight hospital stay in the past 12 months. STUDY DESIGN: Cross-sectional survey with follow-up phone interviews. DATA COLLECTION/EXTRACTION METHODS: Participants completed an online (n = 129) or phone (n = 34) survey about their child's hospitalization experience. The survey contained closed-ended items from the Child Hospital Consumer Assessment of Healthcare Providers and Systems (Child HCAHPS) survey, followed by the six narrative items. Approximately 2 weeks after completing the survey, 47 participants additionally completed a one-hour, semi-structured phone interview, the results of which served as a "gold standard" for evaluating the fidelity of narrative responses. Qualitative content analysis was used to code narrative and interview responses for domains of patient experience and actionability. PRINCIPAL FINDINGS: The average narrative was 248 words (SD = 319). Seventy-nine percent of narratives mentioned a topic included in the Child HCAHPS survey; 89% mentioned a topic not covered by that survey; and 75% included at least one detailed description of an actionable event. Overall, there was 66% correspondence between narrative and interview responses. Correspondence was higher on the phone than in the online condition (75% vs. 59%). CONCLUSIONS: Narratives elicited from rigorously designed multi-item sets can provide detailed, substantive information about pediatric inpatient experiences that hospitals could use to improve child and family experiences during pediatric hospitalization. They add context to closed-ended survey item responses and provide information about experiences of care important to children and families that are not included in quantitative surveys.

Summary of the 2020 AHRQ research meeting on 'advancing methods of implementing and evaluating patient experience improvement using consumer assessment of healthcare providers and systems (CAHPS®) surveys'.

BACKGROUND: The Agency for Healthcare Research and Quality held a research meeting on using Consumer Assessment of Healthcare Providers and Systems (CAHPS®) data for quality improvement (QI) and evaluating such efforts. TOPICS COVERED.: Meeting addressed: 1)What has been learned about organizational factors/environment needed to improve patient experience? 2)How have organizations used data to improve patient experience? 3)What can evaluations using CAHPS data teach us about implementing successful programs to improve patient experience? KEY THEMES: Providers and stakeholders need to be engaged early and often, standardize QI processes, complement CAHPS data with other data, and compile dashboards of CAHPS scores to identify and track improvement. Rigorous study designs are valuable, but much can be learned and accomplished through practical organization-level studies.

A Rigorous Approach to Large-Scale Elicitation and Analysis of Patient Narratives.

Patient narratives have emerged as promising vehicles for making health care more responsive by helping clinicians to better understand their patients' expectations, perceptions, or concerns and encouraging consumers to engage with information about quality. A growing number of websites incorporate patients' comments. But existing comments have fragmentary content, fail to represent less vocal patients, and can be manipulated to "manage" providers' reputations. In this article, we offer the first empirical test of the proposition that patient narratives can be elicited rigorously and reliably using a five-question protocol that can be incorporated into large-scale patient experience surveys. We tested whether elicited narratives about outpatient care are complete (report all facets of patient experience), balanced (convey an accurate mix of positive and negative events), meaningful (have a coherent storyline), and representative (draw fulsome narratives from all relevant subsets of patients). The tested protocol is strong on balance and representativeness, more mixed on completeness and meaningfulness.

Americans' Growing Exposure To Clinician Quality Information: Insights And Implications.

For two decades, various initiatives have encouraged Americans to consider quality when choosing clinicians, both to enhance informed choice and to reduce disparities in access to high-quality providers. The literature portrays these efforts as largely ineffective. But this depiction overlooks two factors: the dramatic expansion since 2010 in the availability of patients' narratives about care and the growth of information seeking among consumers. Using surveys fielded in 2010, 2014, and 2015, we assessed the impact of these changes on consumers' awareness of quality information and sociodemographic differences. Public exposure to any quality information doubled between 2010 and 2015, while exposure to patient narratives and experience surveys tripled. Reflecting a greater propensity to seek quality metrics, minority consumers remained better informed than whites over time, albeit with differences across subgroups in the types of information encountered. An education-related gradient in quality awareness also emerged over the past decade. Public policy should respond to emerging trends in information exposure, establish standards for rigorous elicitation of narratives, and assist consumers' learning from a combination of narratives and quantified metrics on clinician quality.

Choosing Doctors Wisely: Can Assisted Choice Enhance Patients' Selection of Clinicians?

We conducted a simulated clinician-choice experiment, comparing choices and decision-making processes of participants (N = 688) randomized among four experimental arms: a conventional website reporting only quantitative performance information, a website reporting both qualitative (patient comments) and quantitative information, the second website augmented by a decision aid (labeling of patient comments), and the decision-aided website further augmented by the presence of a trained navigator. Introducing patient comments enhanced engagement with the quality information but led to a decline in decision quality, particularly the consistency of choices with consumers' stated preferences. Labeling comments helped erase the decline in decision quality, although the highest percentage of preference-congruent choices was seen in the navigator arm. Engagement with the quality information and satisfaction with choices available were likewise highest in the navigator arm. Findings held for high- and low-skilled decision makers. Thus, navigator assistance may be a promising strategy for equitably promoting higher quality choices in information-rich contexts.

Improving Patients' Choice of Clinician by Including Roll-up Measures in Public Healthcare Quality Reports: an Online Experiment.

BACKGROUND: Public reports on healthcare quality typically include complex data. To lower the cognitive burden of interpreting these data, some report designers create summary, or roll-up, measures combining multiple indicators of quality into one score. Little is known about how the availability of roll-ups affects clinician choice. OBJECTIVE: To determine how presenting quality scores at different levels of aggregation affects patients' clinician choices. DESIGN: We conducted a simulated clinician-choice experiment, randomizing participants to three versions of a public reporting website and comparing their clinician choices. One version aggregated all clinician-level quality measures into roll-ups, the second provided disaggregated (drill-down) scores only, and the third offered both roll-ups and drill-downs. PARTICIPANTS: Five hundred fifty panelists drawn from a probability-based Internet panel. MAIN MEASURES: We assessed the amount of effort participants exerted by tracking the length of time spent on the website and the number of concrete actions taken on the website (e.g., clicking items). We evaluated decision quality by measuring whether participants selected a clinician who performed more poorly than others and incongruence between participants' stated preferences for dimensions of quality and their chosen clinician's performance on those dimensions. KEY RESULTS: Participants seeing drill-downs alone (mean = 14.9) or with roll-ups (mean = 19.2) took more actions than those who saw roll-ups alone (mean = 10.5) (ps < 0.05). However, participants seeing only drill-downs made poorer choices than those who saw roll-ups alone or with drill-downs. More participants seeing drill-downs chose a clinician who was outperformed (36.3% versus 23.4% [roll-up] and 25.6% [drill-down + roll-up], ps < 0.05) and made choices incongruent with stated preferences (51.2% versus 45.6% [roll-up] and 47.5% [drill-down + roll-up], ps < 0.05). The distinction between roll-up and drill-down was somewhat stronger for sicker participants. CONCLUSIONS: Our results suggest that roll-ups in healthcare quality reports, alone or as a complement to drill-downs, can help patients make better decisions for themselves.

Using "roll-up" measures in healthcare quality reports: perspectives of report sponsors and national alliances.

OBJECTIVES: To understand the views of prominent organizations in the field of healthcare quality on the topic of reporting roll-up measures that combine indicators of multiple, often disparate, dimensions of care to consumers. STUDY DESIGN: This study used a semi-structured, qualitative interview design. METHODS: We conducted 30- to 60-minute semi-structured telephone interviews with representatives of 10 organizations that sponsor public healthcare quality reports and 3 national alliances representing multiple stakeholder groups. We conducted a thematic analysis of interview transcriptions to identify common issues and concerns related to reporting roll-up measures. RESULTS: Among sponsors reporting roll-up measures, current practices for calculating and reporting these measures are diverse. The main perceived benefit of reporting roll-up measures is that they simplify large amounts of complex information for consumers. The main concern is the potential for consumers to misunderstand the measures and what associated roll-up scores communicate about provider performance. Report sponsors and national alliances feel that more guidance and research on the methods for producing and reporting scores for roll-up measures are needed. CONCLUSIONS: The results of the interviews elucidate the need for research focused on construction and reporting of roll-up measures. Studies are needed to determine if roll-up measures are indeed perceived by consumers as being less complex and easier to understand.

CAHPS and Comments: How Closed-Ended Survey Questions and Narrative Accounts Interact in the Assessment of Patient Experience.

OBJECTIVES: To investigate whether content from patient narratives explains variation in patients' primary care provider (PCP) ratings beyond information from the closed-ended questions of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group Survey and whether the relative placement of closed- and open-ended survey questions affects either the content of narratives or the CAHPS composite scores. METHODS: Members of a standing Internet panel (N = 332) were randomly assigned to complete a CAHPS survey that was either preceded or followed by a set of open-ended questions about how well their PCP meets their expectations and how they relate to their PCP. RESULTS: Narrative content from healthier patients explained only an additional 2% beyond the variation in provider ratings explained by CAHPS composite measures. Among sicker patients, narrative content explained an additional 10% of the variation. The relative placement of closed- and open-ended questions had little impact on narratives or CAHPS scores. CONCLUSION: Incorporating a protocol for eliciting narratives into a patient experience survey results in minimal distortion of patient feedback. Narratives from sicker patients help explain variation in provider ratings.

Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives.

OBJECTIVE: To design a methodology for rigorously eliciting narratives about patients' experiences with clinical care that is potentially useful for public reporting and quality improvement. DATA SOURCES/STUDY SETTING: Two rounds of experimental data (N = 48 each) collected in 2013-2014, using a nationally representative Internet panel. STUDY DESIGN: Our study (1) articulates and operationalizes criteria for assessing narrative elicitation protocols; (2) establishes a "gold standard" for assessment of such protocols; and (3) creates and tests a protocol for narratives about outpatient treatment experiences. DATA COLLECTION/EXTRACTION METHODS: We randomized participants between telephone and web-based modalities and between protocols placed before and after a closed-ended survey. PRINCIPAL FINDINGS: Elicited narratives can be assessed relative to a gold standard using four criteria: (1) meaningfulness, (2) completeness, (3) whether the narrative accurately reflects the balance of positive and negative events, and (4) representativeness, which reflects the protocol's performance across respondent subgroups. We demonstrate that a five-question protocol that has been tested and refined yields three- to sixfold increases in completeness and four- to tenfold increases in meaningfulness, compared to a single open-ended question. It performs equally well for healthy and sick patients. CONCLUSIONS: Narrative elicitation protocols suitable for inclusion in extant patient experience surveys can be designed and tested against objective performance criteria, thus advancing the science of public reporting.

How Patient Comments Affect Consumers' Use of Physician Performance Measures.

BACKGROUND: Patients' comments about doctors are increasingly available on the internet. The effects of these anecdotal accounts on consumers' engagement with reports on doctor quality, use of more statistically reliable performance measures, and ability to choose doctors wisely are unknown. OBJECTIVE: To examine the effects of providing patient comments along with standardized performance information in a web-based public report. DESIGN: Participants were randomly assigned to view 1 of 6 versions of a website presenting comparative performance information on fictitious primary care doctors. Versions varied by the combination of information types [Consumer Assessment of Healthcare Providers and Systems (CAHPS), Healthcare Effectiveness Data and Information Set (HEDIS), and patient comments] and number of doctors. PARTICIPANTS: A random sample of working-age adults (N=848) from an online panel representing the noninstitutionalized population of the United States. MAIN MEASURES: Time spent and actions taken on the website, probing of standardized measures, and decision quality (chosen doctor rated highest on quantifiable metrics, chosen doctor not dominated by another choice). Secondary outcomes were perceived usefulness and trustworthiness of performance metrics and evaluations of the website. KEY RESULTS: Inclusion of patient comments increased time spent on the website by 35%-42% and actions taken (clicks) by 106%-117% compared with versions presenting only CAHPS and HEDIS measures (P<0.01). It also reduced participants' attention to standardized measures (eg, percentage of time probing HEDIS measures dropped by 67%, P<0.01). When patient comments were present, fewer participants chose the doctor scoring highest on standardized metrics (44%-49% vs. 61%-62%, P<0.01). CONCLUSIONS: Including patient comments in physician performance reports enhances consumers' engagement but reduces their attention to standardized measures and substantially increases suboptimal choices. More research is needed to explore whether integrated reporting strategies could leverage the positive effects of patient comments on consumer engagement without undermining consumers' use of other important metrics for informing choice among doctors.

Examining the role of patient experience surveys in measuring health care quality.

Patient care experience surveys evaluate the degree to which care is patient-centered. This article reviews the literature on the association between patient experiences and other measures of health care quality. Research indicates that better patient care experiences are associated with higher levels of adherence to recommended prevention and treatment processes, better clinical outcomes, better patient safety within hospitals, and less health care utilization. Patient experience measures that are collected using psychometrically sound instruments, employing recommended sample sizes and adjustment procedures, and implemented according to standard protocols are intrinsically meaningful and are appropriate complements for clinical process and outcome measures in public reporting and pay-for-performance programs.

Complexity, public reporting, and choice of doctors: a look inside the blackest box of consumer behavior.

Health care consumers often make choices that are imperfectly informed and inconsistent with their expressed preferences. Past research suggests that these shortcomings become more pronounced as choices become more complex, through either additional options or more performance metrics. But it is unclear why this is true: Consumer choice remains a "black box" that research has scarcely illuminated. In this article, we identify four pathways through which complexity may impair consumer choice. We examine these pathways using data from an experiment in which consumers (hypothetically) selected a primary care physician. Some of the loss of decision quality accompanying more complex choice sets can be explained by consumers' skills and decision-making style, but even after accounting for these factors, complexity undermines the quality of decision making in ways that cannot be fully explained. We conclude by discussing implications for report designers, sponsors, and policy makers aspiring to promote consumer empowerment and health care quality.

Consumer response to patient experience measures in complex information environments.

BACKGROUND: As indicators of clinician quality proliferate, public reports increasingly include multiple metrics. This approach provides more complete performance information than did earlier reports but may challenge consumers' ability to understand and use complicated reports. OBJECTIVES: To assess the effects of report complexity on consumers' understanding and use of patient experience measures derived from the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) survey. RESEARCH DESIGN: In an Internet-based experiment, participants were asked to compare information on physician quality and choose a primary care doctor. Participants were randomly assigned to choice sets of varied complexity (CAHPS alone vs. CAHPS with other measures) and number of doctors. Participants completed surveys before and after this choice task. SUBJECTS: A total of 555 US residents, aged 25-64, who had Internet access through computer were recruited from an existing online panel. MEASURES: Recall seeing CAHPS measures; use of CAHPS measures for making choices; ratings of ease of use, usefulness and trustworthiness of CAHPS ratings; concerns about usefulness and trustworthiness. RESULTS: Participants presented with CAHPS information and other performance indicators relied less on CAHPS than did those presented with CAHPS information only, although they considered CAHPS information as valuable as did other respondents. Participants presented with smaller choice sets also judged CAHPS information as less easy to use when accompanied by other metrics than when it was presented alone.