U.S. flag

An official website of the Department of Health and Human Services

SHARE:

FacebookTwitterEmail AHRQPrintShare

CAHPS Patient Narrative Item Sets

The CAHPS Patient Narrative Item Sets—also referred to as Narrative Elicitation Protocols—are sets of open-ended questions that prompt survey respondents to tell a clear and detailed story about their healthcare experiences. Patients’ narratives provide a valuable complement to standardized survey scores, both to help clinicians understand what they can do to improve their care and to engage and inform patients about differences among providers. The CAHPS narrative items generate insights into the topics addressed by the survey’s measures as well as other important aspects of patient experience that may not be captured by closed-ended questions.1

Currently available:

Why Narrative Item Sets?

Over the last decade, there has been a tremendous growth in user-generated content about healthcare experiences. At the same time, Americans are increasingly seeking out online reviews of health care services. A study conducted by CAHPS researchers found that patient comments have become the form of physician quality information that Americans are most likely to see online.2 However, the unscientific way in which patients’ comments are commonly collected, whether by health care systems or provider review sites, tends to result in information that is neither representative of all patients’ experiences nor a full account of those experiences.

Collecting comments as part of a standardized patient experience survey is one way to address some of these concerns because the comments, like the survey data, are collected from a random sample of confirmed patients in a systematic and structured way. Thus, AHRQ has been funding the development and testing of narrative items sets to support health care organizations in eliciting short, salient narratives from patients about their experiences with care. The goal of this work has been to develop a method for collecting patient narratives that is as scientifically grounded and rigorous as the CAHPS closed-ended questions that are used to gather standardized data on patient experience.

Researchers affiliated with the Yale School of Public Health, RAND, and the University of Wisconsin-Madison have been working collaboratively to develop and test open-ended questions that meet the standards of and are compatible with CAHPS surveys. In the initial phase of the research, the team focused on specifying the characteristics of a “high-quality” patient narrative and developing strategies for measuring those characteristics. Drawing on the literature of narrative medicine, the development emphasized the following characteristics:

  • Completeness: The narratives create a full picture of the experiences that matter to each patient.
  • Balance: They reflect both positive and negative aspects in proportion to each patient's experiences.
  • Meaningfulness: They help the reader picture the patient’s experience with the clinician.
  • Representativeness: They include experiences from patients across a range of health status and socio-demographic characteristics.

For more information about this work and the methods used to develop the items, please refer to:

  • Grob R, Schlesinger M, Parker AM, et al. Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives. Health Serv Res 2016 Jun;51 Suppl 2:1248–72.
  • Schlesinger M, Grob R, Shaller D, et al. Taking Patients’ Narratives about Clinicians from Anecdote to Science. N Engl J Med 2015 Aug;373(7):675–679.

Using Comments from the CAHPS Narrative Items to Improve Care and Inform Consumers

Patient narratives can provide valuable information for both improving care and supporting consumer decision making. For providers, narratives can reveal what is driving CAHPS survey scores and what specific processes and behaviors can be addressed to improve patient experience. For consumers, narratives can communicate information beyond CAHPS scores regarding what a patient’s experience with a clinician and their office staff is like, and how that compares to the patient’s own values and preferences.3

Using Narratives to Inform Providers

Many health systems that are currently collecting open-ended responses as part of their patient surveys disseminate the verbatim comments to medical practice leaders and individual clinicians along with their survey scores.

The CAHPS team conducted research to identify and better understand useful ways of reporting patient narrative information to clinicians and administrative staff. In collaboration with New York-Presbyterian's Patient Experience Team and Ambulatory Care Network, the team assessed the feasibility, value, and use of the open-ended CAHPS items in selected ambulatory care practices. The research questions for this study included the following:

  • How feasible is the collection of narratives using the CAHPS Narrative Item Set in routine patient experience survey operations?
  • What is the added value of the Narrative Item Set compared to conventional open-ended questions?
  • How can narrative information be reported to practice leaders, clinicians, and administrative staff in ways that are easily understood and useful for improving patient experience?

Learn more: 

Learning from Patient Narratives Through Innovative Feedback Reporting Methods (Webcast) (May 31, 2023)

Learning from Patient Narratives Through Innovative Feedback Reporting Methods (Webcast)  (1:01:36)

The Power of Patient Stories for Improving the Patient Experience (Webcast) (May 12, 2022)  

The Power of Patient Stories for Improving the Patient Experience (Webcast) (1:00:26)

Using Narratives to Inform Consumers

The CAHPS team conducted an experiment to explore whether and how including patient narratives in consumer reports might enhance consumers’ understanding of standardized measures of quality, better engage consumers in health care decisionmaking, and more effectively convey patient-reported experiences. Findings from this research suggest that narratives have the potential to increase consumers’ attention to and engagement with reports on physician quality. But narratives can also reduce consumers’ attention to standardized measures and lead to suboptimal doctor choices based on those measures.

Learn more: Kanouse DE, Schlesinger M, Shaller D, Martino SC, Rybowski L. How patient comments affect consumers’ use of physician performance measures. Medical Care 2016 January; 54(1):24-31.

In a second round of experiments, the CAHPS team sought to determine whether tagging patient narratives with short labels to indicate their content helps consumers to better integrate narratives and quality scores when making decisions about physicians. The team also explored the impact of providing access to a navigator, an individual trained to assist with the decision-making process. As in its first round of experiments, the team found that the narratives enhanced engagement with information on physician quality but led to a decline in decision quality. Labeling comments helped mitigate the decline in decision quality, although consumers’ choices were most consistent with their stated preferences when a navigator was present. Engagement with the quality information and satisfaction with available choices were likewise highest when a navigator was present.

Learn more: Martino SC, Grob R, Davis S, et al. Choosing Doctors Wisely: Can Assisted Choice Enhance Patients’ Selection of Clinicians? Med Care Res Rev 2017 Nov 25. Epub.

Analyzing Patients’ Comments

While patients’ comments have great value when read verbatim, the collection of large volumes of narrative responses requires some type of processing to more efficiently extract the core meaning of the information. Two basic types of processing can be used: 

  • Qualitative analysis methods using human coders.
  • Natural language processing (NLP).

Qualitative Analysis Methods: The CAHPS team has developed and tested a qualitative approach to analyzing the content of patient narratives. This method was first used to code narrative data obtained with the 5-question Narrative Item Set developed for use with the CAHPS Clinician & Group Survey and other outpatient patient experience surveys4 and then adapted for the Child Hospital items. The analysis demonstrated how narratives can be coded to reveal specific themes and insights related to multiple domains of patient experience as well as the “actionability” of the narrative content. Content analysis identified many aspects of care not covered by the core domains of the closed-ended questions (such as emotional rapport with providers and interactions with other clinical staff such as nurses and medical assistants). The team also used the codes to assess the emotional valence of the narratives, i.e., whether experiences were positive or negative. Finally, the analysis assessed actionability by considering the who, what, when, where, and how of the experiences described in the narratives. 

Learn more about qualitative methods for analyzing patient narratives in “What Words Convey: The Potential for Narratives to Inform Quality Improvement” in The Milbank Quarterly, March 2019, Volume 97. 

Several health systems have used a similar approach to coding and labeling themes in their comments. However, at a large scale, this method may prove too costly or cumbersome to administer in a way that ensures consistent coding.

Natural Language Processing. Given the relatively labor-intensive nature of qualitative analysis methods, many health systems collecting large amounts of patient comments have opted to use some form of natural language processing (NLP) to analyze the narrative data. A number of patient survey vendors and technology firms now offer their clients NLP approaches to extract much of the same content as qualitative methods (including valence, themes, and actionability) in a more efficient manner. However, most of these methods use proprietary algorithms to process narrative data, making it difficult to validate how accurately they perform compared to human coding. 

A study conducted by members of the CAHPS team to assess the promise of NLP methods concluded that using a combination of machine and human coding analyses may help to leverage the strengths of both approaches. 

1. Martino SC, Shaller D, Schlesinger M, et al. CAHPS and Comments: How Closed-Ended Survey Questions and Narrative Accounts Interact in the Assessment of Patient Experience. J Patient Exp 2017 Mar;4(1):37-45.

2. Schlesinger MJ, Rybowski L, Shaller D, et al. Americans' growing exposure to clinician quality information: Insights and implications. Health Aff (Millwood) 2019 Mar, 38(3): 374-382. https://pubmed.ncbi.nlm.nih.gov/30830827.

3. Grob R, Schlesinger M, Barre LR, et al. What words convey: The potential for patient narratives to inform quality improvement. Milbank Q 2019 Mar, 97(1): 176-227. https://pubmed.ncbi.nlm.nih.gov/30883954

4. The CAHPS team’s method involved the development of a coding scheme for identifying key content domains. (The coding scheme used in testing is available upon request to the CAHPS Help Line at cahps1@westat.com.) Responses to the five narrative questions were aggregated to create a single narrative. Trained researchers then coded the narrative on several dimensions, including the overall valence of the narrative and the number of times certain aspects of care were mentioned, and counted the number of words used. Reliability was established through agreement among the trained coders.  

Page last reviewed June 2024
Page originally created December 2016

Internet Citation: CAHPS Patient Narrative Item Sets. Content last reviewed June 2024. Agency for Healthcare Research and Quality, Rockville, MD.
https://www.ahrq.gov/cahps/surveys-guidance/item-sets/elicitation/index.html

Click to copy citation