5. Impacts of Federally Funded HSR and PCR
Health services and primary care in the United States are complex, multilevel, and layered systems in which the process of change is not always well understood, and effecting positive change often requires leveraging multiple strategies. Understanding how HSR and PCR can and have had impact on these systems is important for assessing the contributions of federally funded research to the fields of HSR and PCR and for informing the prioritization of research gaps. We asked TEP and interview participants to describe the types of impact that HSR and PCR can have and to discuss challenges in defining and assessing this impact. Through discussions with study participants, we learned of examples of federally funded HSR and PCR studies that had clear impact, as well as the many challenges in measuring and achieving that impact.
This chapter is organized in three sections. First, we describe different types of research impact based on both discussions with study participants and existing frameworks of impact. Next, based on feedback from the study participants, we describe challenges to assessing research impact; and, finally, we describe challenges to achieving impact of HSR and PCR.
The findings presented in this chapter do not represent a complete catalog of the impact of federally funded HSR and PCR, which would have been beyond the scope of this study. However, the examples provided throughout this chapter demonstrate key types and mechanisms of impact, which are useful for understanding and assessing the impact of HSR and PCR portfolios.
Defining Types of Research Impact
Study participants emphasized various kinds of impact generated by HSR and PCR. Many of these correspond to types of research impact that have been specified in existing frameworks for HSR and PCR (Buykx et al., 2012; Kuruvilla et al., 2006; Centers for Disease Control and Prevention, 2017a; Donovan and Hanney, 2011). We synthesized these impact frameworks and themes from the TEP and interview discussions to identify the following six categories of HSR and PCR impact:
- Scientific impact. Knowledge generation, research methods, capacity building, training, and leadership.
- Professional knowledge and practice impact. Education and training curricula, professional certification, and clinical guidelines.
- Health care systems and services impact. Quality of care and service delivery, improved information and health services management, cost containment and effectiveness, resource allocation, and health workforce.
- Policy impact. Information base for political and executive decisionmaking, influence and involvement in decisionmaking processes.
- Patient impact. Patient and family experience of care; health literacy; health knowledge, attitudes, behavior, and outcomes.
- Societal impact. Population health, trust in health care, economic productivity, employment benefits and income.
It is unlikely for individual research projects to generate impact across all the above categories, and, as study participants pointed out, the impacts of specific projects may not always occur in a linear order. For example, a project may have scientific impact and generate new knowledge, which might in turn have an impact on policy, but not on clinical guidelines or health care systems and services. Further, because these impact categories are quite broad, it will likely be necessary to identify subcategories within these categories as well as to identify metrics that can be used to measure impact; thus, we expect that the combination of impacts from any study or line of research could vary in multiple ways.
But, as explained in several of the cited research impact frameworks, the types of impacts identified represent a progression from interim research impacts (i.e., on scientific knowledge, professional practice, health care systems, and policy) to direct outcomes for patients and wider outcomes for society. As such, the categories represent a general guiding framework for assessing the accumulated impact of portfolios of research within an agency or that span agencies.
Below we describe the six categories of impact and illustrative HSR and PCR examples of each. We then provide case illustrations of two research areas—HAIs and PCMH (related to HSR and PCR, respectively) that describe the accumulation of research impacts through various coordinated efforts across agency portfolios over time.
Scientific Impact
Study participants described research impact—as it is traditionally conceived by the research communities—through indicators of knowledge generation. For example, members of the HSR TEP suggested a “Hall of Fame” measure for assessing impact that includes publication in high-impact journals, number of citations, and altmetrics scores (Azer and Azer, 2019), as well as citation in influential reports, such as those produced by the National Academy of Medicine. Others focused on development of a research “method or some way of [conducting] services research that would change the future of HSR.”
Scientific impact in the areas of HSR and PCR may include not only knowledge generation, but research methods and capacity training as well. For primary care, scientific impact has often taken the form of foundational knowledge that laid the groundwork for future practice transformation and model creation. Multiple state-level delivery systems leaders, PCR TEP members, and other interview participants identified various research streams with scientific impact, including Bodenheimer’s work on the building blocks of effective primary care (Bodenheimer et al., 2014), ecological studies from Starfield on the benefits of primary care being greater than the sum of its parts (Starfield, Shi, and Macinko, 2005b), Milstein’s work on aspects of high-performing primary care (Simon et al., 2017), and Stange’s research on the direct observation of primary care that laid the groundwork for understanding its complexity (American Academy of Family Physicians, undated).
Professional Knowledge and Practice Impact
Multiple HSR TEP, research, and other stakeholders emphasized the professional and practical impact of research that is seen when research findings are incorporated into clinical guidelines as a conduit toward change in clinical practice:
The thing that I think about is impact on guidelines, right? Do professional societies, you know, are they interested and is there uptake of the evidence by professional societies or other decisionmakers or payers?
PCR TEP members and a delivery leader interview participant identified the Systolic Blood Pressure Intervention Trial (NHLBI, undated) and the Action to Control Cardiovascular Risk in Diabetes studies (Action to Control Cardiovascular Risk in Diabetes, 2019) as examples of HSR projects directly leading to changes in guidelines and ultimately clinical practice. The Women’s Health Study was highlighted by the same delivery systems leader as having had an impact on the use of estrogen therapy in practice. Researchers also highlighted important use of research in changing guidelines, though more often pointing to a body of work rather than a specific study (e.g., use of research by CDC’s Advisory Committee on Immunization Practices; clinical guidance on management of strokes, myocardial infarctions, blood pressure, and cholesterol).
Professional societies were mentioned by several TEP members as an important route for impact, first via clinical guidelines and then ultimately by changes to practice. One HSR TEP member noted the potential impact of research on professional knowledge and practice through the inclusion of research findings in professional education curricula and training:
“Something is the gold standard if it’s the way that we train medical students, residents, nurse practitioners, nursing students . . . are the findings from an HSR study or an HSR program, to what extent are those embedded in training?” However, this potential is not always realized as, for example, “medical school faculty don’t teach it well [or] don’t teach it all.”
Health Care Systems and Services Impact
A range of stakeholders emphasized changes to health care systems and services as a critical impact of HSR and PCR. These stakeholders included not only health care delivery leaders, but federal, state-level, researcher, and other stakeholders as well. At the most granular level, changes to health care systems and services might mean thinking about the impact of research in terms of practice models. A number of studies or avenues of research were cited as having particular impact in the area of practice models and practice transformation, especially some related to primary care practice. Both an employer/payer and a delivery system leader stakeholder noted the impact of microsimulation modeling studies that helped build the business case for practice transformation. Finally, the continued development and penetration of primary care medical homes were cited as indicators of high impact by researchers, state-level representatives, and others.
Stakeholders also noted the importance of understanding the level to which health care systems have been informed by research, resulting in changes in systems structures and programs and in approaches used by those directly involved in the delivery of care. In the words of a state hospital association stakeholder:
[A]s a health services researcher, I want to be able to answer how does this improve access, how does this improve the quality of care, and how does this reduce the cost of care? So when I’m doing this work in that space, those are the three questions that I better be able to answer. And when it’s translated down to implementation, that’s where you really ultimately see [the impact].
Similarly, a member of the HSR TEP focused on uptake and use of evidence-informed practices as a measure of research impact on health care systems: “Rather than just the effectiveness of an intervention itself, look at the effectiveness of the uptake and the effectiveness to which it’s been adopted or actually changed system practices.” This could also be described as “the degree to which the finding [becomes] ‘hard wired’ into practice.” This change could occur literally, such as when a clinical pathway is incorporated into an EHR order set that clinicians use, but could also refer to various changes in organizational policies, processes, and norms of care that occur indirectly. Other impacts in this vein, as suggested by a state-level stakeholder, include reduction in provider burden and increase in the efficiency of medical practices.
To achieve the uptake and use of HSR and PCR findings, one federal stakeholder pointed to the need for research evidence to reach “implementers,” including but not limited to health systems administrators, quality improvement professionals, and clinicians. At the same time, delivery system leaders highlighted the importance of researchers producing evidence on new practice models and innovations that health care systems are trying to implement or are considering implementing. In the words of one delivery systems leader, both these goals are attainable through close collaboration between researchers and health care system stakeholders:
You have to work with health service providers and health system people and managers and implementers, . . . they hold some of the impact keys as well as the researcher, right? So the trick is how do you measure your contribution in terms of that meaningful impact? I think you have to do it in terms of does health services research result in beneficial changes done by managers or health system people, right? You actually see changes in decisions that are made as a result of any research findings that you’re producing.
A federal research leader described how his agency has changed its approach to study proposals and design to encourage researchers to build in linkages to health service uptake and impact earlier in the development of projects:
We’re refining our applications to spell out more clearly what they see the next steps of their proposal to be. We’re replacing what used to be a fairly bland dissemination and implementation section with something more specific about what would really be needed to convince a stakeholder to actually take any action based on the research. So is the study going to be sufficient, or would you have to take a single center study, to study it in a multicenter study and learn more about the implementation? Would you have to do a budget impact analysis? But to force them to think in ways they don’t always think about what the obstacles might be for the research making a difference.
A representative of a state-level safety net association further highlighted thinking about impact in terms of building relationships between health care delivery organizations and research collaborators: “It’s something that increases knowledge, gives us an opportunity to learn new things, and allows for development of relationships so that then there’s a resource later on. So in some of these research things that we’ve worked on with [the state university], it’s built relationships with people that are then resources for us when we need some other information.” Likewise, a member of the HSR TEP stressed “partnership with industry or industry uptake” as a type of impact.
Policy Impact
Policy impact—meaning changes to policy with the goal of ultimately improving health care—was identified as an important aspect of impact by both TEP and interview participants. A representative of a state-level improvement organization emphasized the importance of policymaker awareness, with a first step being to “turn up the volume about the need to pay attention.” Federal stakeholders and TEP members also highlighted the influence of research on both legislative and regulatory policy. As one federal stakeholder commented, “We’re trying to work with our contractors around keeping in mind that the audience is not always the research community. It’s the policymaking community, so we need to be able to have multiple ways of communicating.”
A member of the HSR TEP encouraged keeping a “broad” view of policy in considering the impact of research, including a “Supreme Court or any court decision,” as well as federal program and private policymaking bodies, such as “CMS conditions of participation . . . the Joint Commission, for instance, if they change something based on HSR, or some national patient safety goal.” TEP members and federal and other policy stakeholders also pointed to the role of research in developing quality measures that are subsequently incorporated into federal quality reporting and value-based payment programs, such as the Medicare Advantage Star rating system.
Many interview participants and TEP members cited examples of impact at the federal or state level that resulted either directly or indirectly from HSR, such as aspects of the Affordable Care Act that were inspired by work in HSR. The TEP also noted that the funding and development of alternative payment models have embedded a vehicle for change into the health care system.
Two other interview participants noted that continued work in these areas, including evaluation, has had impact as well.
A variety of examples of policy impact were at the state level. TEP participants and interview participants cited projects that influenced state-level initiatives, such as the Alternative Quality Contracts in Massachusetts, which was informed by work on quality measurement, pay for performance, capitation, and global payments. Another state-level representative highlighted research used by policymakers in Maine to expand Medicaid coverage, while yet another identified the use of research on postpartum depression to push state legislatures to implement voluntary home visits for newborns and their mothers in their state.
Patient Impact
TEP and interview participants also focused on HSR’s and PCR’s impact on patients, as one HSR TEP member commented, “The patient-related impact is separate and should be counted.” Several researchers and TEP members described this as impact that was “meaningful” to patients. One researcher specified impact in terms of an increase in patient knowledge, which leads patients to become better consumers of health care. Interview participants from varied backgrounds, including consumer, federal, and research stakeholders, conceptualized impact on patients more broadly, citing improvement in patients’ general health, patient-centered outcomes such as quality of life, and patient safety.
In the area of PCR, one researcher interview participant cited recent NIH-funded work by Levine and colleagues that found that patients with any source of primary care reported significantly better health care access and experience (Levine, Landon, and Linder, 2019). In addition, AHRQ’s patient safety portfolio (discussed in more detail later in this chapter) was repeatedly noted as having had impact as a body of work that ultimately translates into safer and better care for patients.
Societal Impact
In terms of societal impact, study participants focused mostly on population health. As one health care delivery leader stated, “Ultimately, impact should be measured in improved health.” Other health care delivery and state public health and hospital leaders talked about the “Triple” or “Quadruple Aim” in terms of how other health care system impacts result in improved population health:
We’re funding health research. So we actually have to produce health, right? So the impact is about health. It is about the triple aim or the quadruple aim. It’s actually how do you produce health in a way that’s affordable, in a way that . . . it’s a reasonably pleasant experience for people, and that is doable by a workforce.
A member of the HSR TEP also pointed to broad impact of HSR and PCR on “not just [the] patient, but the general public, . . . because there are lot of issues about trust in the health care system, the people who may or may not even see the doctors, that is needed to influence health in a positive way.” One specific example in the area of primary care and societal impact was cited by a researcher interview participant who highlighted NIH-funded work showing that a greater supply of primary care physicians is associated with improved mortality (even as the supply of primary care physicians declined per capita over the same ten-year period).
Case Illustrations of Impact in HSR and PCR
In the next section, we describe two major examples in HSR and PCR that illustrate the accumulation of different types of research impact delineated above, often involving coordinated efforts, across agency portfolios over time. The choice of the two case illustrations—HAIs for HSR, and PCMH for PCR—were informed by themes that emerged consistently from both the TEPs and interviews.
Healthcare–Associated Infections
HAIs, particularly those developed during hospitalization, represent a major patient safety concern with significant effects on health at both the public and personal levels. For years, before HAIs were publicly recognized as a major and ultimately preventable problem in health care, the epidemiological and infection control communities had been working on this issue (Mendel et al., 2011). These communities generated scientific impact by developing the basic scientific understanding on the microbes and pathology associated with HAIs, assessing the current public health surveillance systems and methodologies for monitoring HAI rates, and creating a host of guidelines and prevention practices, which became the first HAI prevention toolkits to have professional knowledge and practice impact.
In the early 2000s, two other historic forces drew attention to HAIs: (1) public awareness campaigns spearheaded by consumer groups advocating action to address the growing HAI epidemic and, (2) the patient safety movement to reduce patient harms experienced during receipt of health care. These forces culminated in a 2008 report by the GAO (Government Accountability Office, 2008) and subsequent congressional hearings on the need for federal coordination that catalyzed an interagency HAI Action Plan and set the stage for changes in policy, funding, and stakeholder engagement.
Within the realm of HAIs, central line–associated bloodstream infections (CLABSIs) provide a clear example of how federal agencies and other entities can collaborate to fund research with the potential for impact at all levels of health care. As noted above, there was a solid scientific evidence base indicating that CLABSIs could be prevented, including a landmark study from Pronovost et al. (2006). In addition to its clinical guidelines on CLABSI prevention, CDC developed a measure for CLABSIs that could be collected through the National Healthcare Safety Network (Centers for Disease Control and Prevention, 2017b) (scientific impact, professional knowledge and practice impact). Meanwhile, AHRQ—working in collaboration with private philanthropic entities—funded the Comprehensive Unit-Based Safety Program (CUSP)—a strategic framework for safety improvement that integrates communication, teamwork, and leadership with evidence-based clinical prevention practices—to address CLABSIs, as well as other HAIs. The program was cited as a “measurable national success story in quality improvement” (Pronovost, Marsteller, and Goeschel, 2011). Between 2001 and 2009, CLABSIs dropped by 63 percent (Centers for Disease Control and Prevention, 2011). Empirical research found that more than 1,000 intensive care units across the country reduced CLABSI rates by a total of 41 percent when they used tools based on CUSP to improve patient safety (health care systems and services impact) (Health Research & Educational Trust, 2012).
The results from the CUSP program, and other patient safety work funded by AHRQ, would later inform CMS policy around value-based payments for CLABSIs (policy impact) (Centers for Medicare and Medicaid Services, 2019b). Currently AHRQ, CMS, and other stakeholders are working together to ensure patient and societal impact through the Partnership for Patients to improve the quality, safety, and affordability of health care through the reduction of Hospital-Acquired Conditions (including HAIs) (Centers for Medicare and Medicaid Services, undated-c). In 2019, the National Scorecard on Rates of Hospital-Acquired Conditions estimated that from 2014 to 2017 hospital-acquired conditions fell by 13 percent, saving about 20,500 lives and about $7.7 billion in healthcare costs (Agency for Healthcare Research and Quality, 2019c).
Patient-Centered Medical Home
The PCMH is a model of primary care that attempts to operationalize the core functions and attributes of primary health care (i.e., being comprehensive, patient-centered, coordinated, accessible, and focused on quality and safety) (Agency for Healthcare Research and Quality, undated-a). This model is based on findings from a large body of research in primary care, including prior work defining primary care (World Health Organization, 1978; Institute of Medicine Committee on the Future of Primary Care, 1996; Starfield, Shi, and Macinko, 2005a; Green et al., 2004) and precursor models (Bodenheimer, Wagner, and Grumbach, 2002; Murray, 2005; Ad Hoc Task Force on Definition of the Medical Home, 1992; Grumbach and Bodenheimer, 2004; The Robert Graham Center, 2007). In 2007, four medical professional associations in primary care developed and endorsed the Joint Principles of the Patient-Centered Medical Home (American Academy of Family Physicians, 2007).
In 2010, findings from the country’s first national demonstration of a PCMH model, consisting of 36 family practices over two years, provided the first national-level evidence documenting the successes and challenges of PCMH implementation (Stange et al., 2010). This privately sponsored study spurred further diffusion of the model (health care systems and services impact) and research into PCMH implementation and outcomes (scientific impact).
Since then, there have been significant investments of federally funded PCR into PCMH implementation and outcomes. AHRQ has funded a variety of PCMH research, including three major grant initiatives “to understand the challenges faced by primary care practices as they transform into PCMHs and to help create an infrastructure to assist them with this transformation”—the first on primary care transformation, the second on infrastructure to support transformation, and the third on costs of supporting transformation (Agency for Healthcare Research and Quality, 2015). The agency has also served as a clearinghouse for dissemination of PCMH resources, including definitions of the PCMH model, tools for implementation and practice facilitation, evidence and evaluation findings, guidance on evaluation methods, and a searchable database for additional resources (Agency for Healthcare Research and Quality, undated-h) (scientific impact, health care systems and services impact).
AHRQ additionally established a federal interagency workgroup, the Federal PCMH Collaborative, for agencies across the federal government to share lessons and research findings on PCMH implementation as well as coordinate federal research, dissemination, and implementation efforts. The wide range of these efforts conducted by other federal agencies has included the VHA’s and DoD’s large-scale implementation of the PCMH model in their respective health care delivery systems (U.S. Department of Veterans Affairs, 2019b; U.S. Department of Veterans Affairs, 2011), HRSA’s technical assistance and funding programs to support PCMH adoption among FQHCs (Health Resources and Services Administration, 2018), CMS’s CPC and other alternative payment models for PCMH-based care (Centers for Medicare and Medicaid Services, 2019a, 2019b), and OMB’s integration of the PCMH into employee benefit plans. As an example of federal collaboration on PCMH efforts, a federal interview participant described how the workgroup facilitated AHRQ’s engagement with CMS to help inform the development of the CPC payment demonstration and evaluation (scientific impact, health care systems and policy impact).
Over the past decade, PCMH-influenced models of care have proliferated in private practices across the country (Jackson et al., 2013; Edwards et al., 2014; Nielsen, 2016). This proliferation has been encouraged by formal PCMH certification programs for primary care practices offered by accreditation organizations (Burton, 2012), as well as the wave of federal, state, and multi-payer payment reform initiatives that often incorporate incentives for PCMH models of care and certification (e.g., Centers for Medicare and Medicaid Services, 2019a; Primary Care Collaborative, 2019b; Primary Care Collaborative, 2019a) (policy impact, health care systems and services impact).
Studies documenting PCMH transformation and outcomes—many of which have been federally funded—have likewise multiplied (Agency for Healthcare Research and Quality, undated-g). Findings indicate that transforming primary care is challenging (Martsolf et al., 2015; Crabtree et al., 2010). There is evidence that practices of transformation vary in how they put the concepts of PCMH into practice (Rittenhouse et al., 2013; Solberg, 2011; Tomoaia-Cotisel et al., 2016), and that implementation can improve outcomes such as cost, quality, and patient satisfaction (Friedberg et al., 2014, 2015; Strickland et al., 2011; Rosenthal, 2008; Shi et al., 2017; Farrell et al., 2015) (health care systems and services impact, patient impact). Federally funded PCR has also helped to improve and develop methods for studying PCMHs (Miller et al., 2013; Tomoaia-Cotisel et al., 2013; Devers, 2013) (scientific impact).
Beyond primary care service delivery systems, educational institutions have been increasingly focused on strengthening “interprofessional practice and education” (Nester, 2016) (professional knowledge and practice impact) in part because of team-based care at the core of the PCMH model. Medical home-type implementation and certification have begun to be adapted into specialty settings as well (National Committee for Quality Assurance, 2017) (health care systems and services impact, policy impact).
Challenges in Assessing the Impact of HSR and PCR
In addition to describing types of impact, study participants were asked to discuss challenges in assessing the impact of HSR and PCR. Three themes that emerged from these discussions were the difficulty in attributing impact, the lengthy time delay to observing impact, and the difficulty of measuring some types of impact.
Difficulty in Attributing Impact
Many TEP participants noted that the complexity of health care systems and the complex ways in which research findings are translated and implemented in such applied fields as HSR and PCR make it difficult to attribute impact to any one study. As one PCR TEP participant explained:
If you think about a lot of health services research, it ends up being a cumulative body of evidence that might influence things, as opposed to saying this one study. . . . [I]n general, in health services research, we end up accumulating evidence that ultimately will have an impact from the design limitation and evolution of programs and policies. But it’s hard to attribute something to one particular spot, or one particular study. So that part is incredibly challenging.
Thus, as one state-level insurer representative noted, although it is important for health service organizations, such as health plans, to ground changes in specific research evidence, the best evidence is rarely provided by the findings of a single study but rather by a body of studies.
Patient safety was identified by multiple interviewees (including researchers and federal stakeholders) and the HSR TEP as an important and impactful area of work that could not be tied to any one particular research study or project. Although there were identifiable landmark demonstration projects—such as Pronovost’s and colleague’s studies on preventing CLABSIs and other HAIs using the CUSP change methodology, as discussed above (Pronovost, Marsteller and Goeschel, 2011)—study participants noted significant strides in patient safety improvement over recent years that were difficult to pinpoint to a specific research project or newly demonstrated practice.
Likewise, although individual studies may have proximal impacts on health care research, policy, and practice, multiple TEP participants and interview participants, including federal and other policy stakeholders, agreed that it is most useful to assess impacts at the level of research portfolios in order to evaluate the results of research investments and needs for further funding. Another federal stakeholder observed the role of evidence review and synthesis in aggregating, assessing, and amplifying research findings across studies; doing so quickly and efficiently, given health care systems’ rapidly evolving needs for new evidence is important for shortening the time to impact for HSR and PCR (Guise, Savitz, and Friedman, 2018).
Time Delay to Achieving Impact
HSR TEP participants, as well as multiple interview participants (including state-level stakeholders, delivery systems leaders, and others), noted that impacts of federally funded HSR and PCR are often realized after various time delays, making it challenging to assess the impact of a particular research project. One reason given for how a delay can affect the assessment of impact is that the consequences of the intervention may take longer to manifest than the life of the project. A delivery leader interview participant cited practice transformation evaluations as a specific area where this challenge has manifested:
They tend to report results of two years of work or three years of work and that’s really too short a period of time to document the benefits of transformation. It’s really a five- or eight-year project and we don’t have the robust federal funding to do rigorous drilldowns into the impact of transformation.
Another interview participant noted that research findings are often implemented slowly and/or gradually, lengthening the time required to observe an impact. The same participant mentioned that the dissemination and impact of more comprehensive primary care practice models have been delayed due to the slow pace of change in payment policy.
Study participants also discussed ways in which the context for implementation sometimes changes before research findings are broadly implemented. For example, a consumer interview participant commented that research findings are occasionally out of date by the time guidelines are released.
Delays can affect all types of impact including: professional and practice impact, policy impact, patient impact, and societal impact. One researcher interview participant mentioned that improvements in social phenomena such as stigma are especially prone to delayed impact and thus challenging to study.
Difficulty in Measuring Impacts
Some types of impact are relatively easy to measure and assess. For example, data on traditional scientific impact measures (e.g., numbers of citations, journal impact scores) are readily available. Similarly, study participants noted the vast amount of publicly available data on health outcomes collected and provided by CDC for the public at large, as well as by CMS and the VHA on the large populations they serve. However, similar types of standardized sets of measurement and data infrastructure are not available to describe the impacts of scientific knowledge generation on health outcomes, particularly for health systems and services impact. As observed in the HSR TEP, the uptake, adoption, and use in practice of new evidence-based practices are “harder to assess” and, even if measured, the data are typically “proprietary” to specific health care organizations.
Challenges in Achieving Impact of HSR and PCR
In addition to challenges in assessing impact, study participants reported challenges and barriers to achieving impact of HSR and PCR. Two general challenges raised were the lack of investment in high-risk research and disconnects between research and implementation. Similar issues emerged in the discussion of gaps in research approaches (Chapter 6).
Lack of Investment in High-Risk Research
Some interview participants mentioned that the lack of investment in high-risk studies can dampen impact; particularly as such studies may be those with potential for the highest impact. Such studies might be similar to those funded by the NIH Pioneer Awards (which, to date, have typically focused on basic science), which are projects at high risk of poor or null results but that also have potential for high reward in terms of the gain in knowledge and practice. One policy expert interview participant noted that
a lot of the investment is probably not on the highest-risk, highest-potential value, highest-impact studies. They suffer from the same things that folks have leveraged against other NIH funding that you just do safe studies that are very incremental that don’t really contribute that much to care improvement, but they add a little bit of added knowledge or some nuance.
Disconnects Between Research and Implementation
Multiple interview participants stated that research impact was limited because it is partially determined by the dissemination and implementation of research results, yet there is a disconnect between research and implementation. This issue was brought up by delivery system leaders, researchers, and state- and federal-level policymakers. One policy expert interviewed stated that lack of investment in infrastructure and practice change means this topic is understudied, which results in difficulty in putting research into practice. A delivery leader on the HSR TEP stated: “Our biggest challenge is how do we get our clinical teams to adopt new practices? . . . [There are] subtleties to the information you present to them.”
Sometimes the failure to translate research results into practice was due to a disconnect between statistical significance and clinically meaningful significance. One delivery leader noted that the size of a statistically significant effect should also be taken into account in considering whether or not to adopt an intervention, such as this example related to diabetes control: “Patients had statistically significant reduction in A1c over the course of a year, but it was by 0.1 . . . that’s not clinically significant.”
Multiple interview participants also noted that the incentives are not well aligned between researchers performing research and organizations implementing research, and, as a result, the two groups often do not communicate well with each other. Academic-based research in particular was viewed as frequently being driven by interests different from what may be needed in practice:
The ecosystem that supports that environment is a very particular ecosystem that requires a very well-defined set of actors and processes. So it requires individuals who are pursuing academic advancement that need federal grant support typically, in order to achieve academic advancement, they need to write papers. They need to get grants funded. And that is a very different environment from the environment around care being actually delivered to real people, real time and how that care can be improved.
At the same time, a state-level interview participant noted that policymakers often are less interested in hearing about research evidence and more about stories of success, which may be at odds with the results produced by the research community. Specifically, policymakers may make decisions based more on the experience of others: “Policymakers want to be aware—they kind of move in packs. So if they’ve heard that one state has done this, they will try it themselves.”
Implementation issues may also relate to the business case for adopting evidence-based interventions among health care stakeholders. Three HSR TEP members noted that challenges arise when benefits and costs accrue to different entities:
You know, it’s easy to show that delivering some service now will prevent money later. . . . But the entity providing it now and incurring the cost is one thing. It may be the same entity down the road, but it’s different people. So the savings won’t accrue to that entity, or it may be a different entity down the road, and so now you’re talking about, you know, why does that initial entity not implement the results of this wonderful study. Well, yeah, they’re going to lose their shirts if they do.
In interviews, consumer stakeholders noted that the disconnect between research and practice implementation could to some degree be due to lack of engagement by patients and frontline providers in both HSR and PCR, resulting in research that is less relevant and usable for change:
I think a lot of the health services research could be more effective and more impactful if the funders were to engage the patient and consumer community as decisionmakers in what gets funded. And that I think more patients and consumers and caregivers need to be engaged as partners in the research that is funded, because I have seen a lot of research that answers research questions that matter to researchers. And I think that it’s—it doesn’t always serve . . . it doesn’t always answer the questions that matter most to patients.
Lastly, TEP and interview participants identified certain limitations on agencies’ ability to facilitate the impact of the research they sponsor. One federal agency representative noted that agencies have a very specific “lever” that they can pull, which may limit the potential impact of sponsored research, especially if an agency encounters difficulties coordinating with other agencies on a topic. Researchers from both interviews and the HSR TEP noted that contracted research has the benefit of quicker turnaround but may not necessarily be published and available to the wider field through standard channels. A few HSR TEP members and interview participants (both federal and state-level) noted that agencies are sometimes reluctant to promote findings that are perceived to be politically sensitive. Moreover, as noted by a federal interview participant, political decisionmaking processes beyond the influence of agencies frequently determine whether research findings are translated into policy change and impact. Finally, a HSR policy interview participant as well as several HSR TEP members perceived that agencies, and AHRQ specifically, could improve their efforts to inform policymakers of both the implications as well as the impact of their funded research.
Chapter Summary
Health services and primary care in the United States are complex, multilevel, and layered systems in which the process of change is not always well understood, and effecting positive change often requires leveraging multiple strategies. Understanding how HSR and PCR can and have had impact on these systems is important for assessing the contributions of federally funded research to the fields of HSR and PCR and for informing the prioritization of research gaps.
Based on existing frameworks of research impact as well as discussions with TEP and interview participants, we identified six categories of impact, which we illustrated with examples of federally funded HSR and PCR: (1) scientific impact, (2) professional knowledge and practice impact, (3) health care systems and services impact, (4) policy impact, (5) patient impact, and (6) societal impact. It is unlikely for a single project to generate impact across all categories, and the impacts of specific projects may not always occur in a linear order. However, the types of impacts identified represent a progression from interim research impacts (i.e., on scientific knowledge, professional practice, health care systems, and policy) to direct outcomes for patients and wider outcomes for society. As such, they represent a general guiding framework for assessing the accumulated impact of portfolios of research within an agency or that span agencies.
In addition to identifying types of impact, study participants noted important challenges to assessing impact, including the difficulty of tracing the accumulation of impacts across specific projects or sets of projects within a portfolio, especially when research is funded by multiple agencies. Impact may take time to accumulate and be realized, which further complicates attribution to specific projects or sources of funding. Moreover, many types of impact are by their nature difficult to systematically measure. Study participants also called attention to challenges in achieving HSR and PCR impact. These barriers included a lack of investment in high-risk studies and various disconnects between research and implementation.
