AHRQ Challenge on Integrating Healthcare System Data With Systematic Review Findings

The Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services (HHS), is announcing a challenge competition to explore the resources and infrastructure needed to integrate real-world evidence from healthcare systems into systematic review findings.

• About the Challenge
• How to Enter the Challenge
• Review Process

Announcements

Challenge Award Winners

Winner: Mayo Clinic

The team addressed the evidence gap identified in a 2023 AHRQ systematic review with regard to the comparative effectiveness of different modalities of partial breast irradiation (PBI) for breast cancer treatment. Three PBI modalities were evaluated: 1) proton radiation therapy, 2) photon radiation therapy, and 3) applicator-based brachytherapy. The team emulated a clinical trial using electronic health records from a multistate large tertiary health system and used propensity score matching to balance confounders. The three PBI modalities showed similar rates of ipsilateral breast recurrence and overall survival. Although the body of evidence remains small and requires longer followup, the current study is the largest to date and represents improvement in the precision of estimates and in strength of evidence. Team: Hassan Murad, Zhen Wang, Dean Shumway, Kimberly Corbin, Robert Mutter.

Runner-up: Kaiser Research

The team addressed the inadequate evidence in a 2022 AHRQ systematic review assessing the accuracy of screening for suicide risk in children and adolescents. Data from their large integrated health system was evaluated for the accuracy of self-report questionnaires and computed risk-prediction scores for prediction of suicidal behaviors. Results indicate that responses to PHQ-9 item 9 can accurately identify adolescents at increased risk for self-harm over 3 to 6 months following an outpatient visit. Risk scores computed from health records data had slightly better performance than PHQ-9 item 9 scores among mental health specialty visits, but poorer performance among general medical visits with a recorded mental health diagnosis or primary care visits without a mental health diagnosis. Extending screening to primary care visits without mental health diagnosis adds little to identify adolescents with subsequent self-harm events. Team: Gregory Simon, Christine Stewart, Julie Richards.

Honorable Mention: ECRI-U Penn

The research team assessed the impact of the DKA clinical pathway on length of stay, inpatient mortality, and readmissions to fill evidence gaps identified in the 2021 AHRQ report, Interventions to Decrease Hospital Length of Stay. Although results were limited due to lack of reliable clinical data accessed via routinely available informatics databases, there was an insightful description of risks, barriers and challenges encountered, limitations in accessing granular data in structured form, and the difficulties the team had in finding a data source that met the needs of the proposed analysis. Team: Emilia Flores, Shazia Siddiuqe, Nikhil Mull, Michael Harhay, Sameh Saleh, Ilya Ivlev, Johnathan Treadwell.

Phase 1 Finalists

The Phase 1 proposals of the following five finalists were selected as eligible to progress to Phase 2 of the Challenge. A brief description of the analyses they propose to submit is noted below.

ECRI-Penn: The research team plans to use real-world data to understand the impact of Penn Medicine's clinical pathway for diabetic ketoacidosis on patients' length of stay, readmissions, and mortality. Since the evidence from the 2021 AHRQ report, Interventions to Decrease Hospital Length of Stay, consisted primarily of non-US studies, this analysis of real-world data will help the team better understand the effect and applicability of the pathways at their healthcare system and the context in which they have been developed and implemented. Team: Emilia Flores, Shazia Siddique, Nikhil Mull, Michael Harhay, Sameh Saleh, Ilya Ivlev, Johnathan Treadwell.

Kaiser Permanente Washington: To address the inadequate evidence in a 2022 AHRQ systematic review assessing the accuracy of screening for suicide risk in children and adolescents, the research team will leverage data from their large integrated health system and evaluate the accuracy of self-report questionnaires and computed risk-prediction scores for prediction of suicidal behaviors. Team: Gregory Simon, Christine Stewart, Julie Richards.

Mayo Clinic: The team will address the evidence gap identified in a 2023 AHRQ systematic review with regards to the comparative effectiveness of different modalities of partial breast irradiation (PBI) for breast cancer treatment. The team will analyze data collected from the three health system locations to track the health outcomes of women with early-stage breast cancer undergoing PBI, based on their tumor characteristics and ductal carcinoma in-situ status. Team: Hassan Murad, Zhen Wang, Dean Shumway, Kimberly Corbin, Robert Mutter.

OCHIN: The researchers will evaluate the use of telehealth for counseling women of reproductive age on contraception and sexually transmitted infections, with data from a national network of community health centers. They will also examine the equitable uptake of telehealth by comparing its use in populations that experience health disparities. The goal is to strengthen the certainty of findings from the 2022 AHRQ systematic review on Telehealth for Women's Preventive Services and identify intervention opportunities to maximize the public health impact. Team: Rose Goueth, Brenda McGrath, Suparna Navale.

Vanderbilt University: Team members will address an evidence gap identified in the 2021 AHRQ evidence report, Management of High-Need, High-Cost Patients: A "Best Fit" Framework Synthesis, Realist Review, and Systematic Review. They will use data from electronic health records and attempt to differentiate modifiable from unmodifiable hospital utilization in the high-cost, high-need (HCHN) patient population based on the interventions they received. The team hopes that other health systems will be able to use the insights from this study to refine their own processes and criteria for identifying HCHN patients most likely to benefit from program interventions. Team: Scott Lee, Benjamin French, Francis Balucan, Michael McCann, Eduard Vasilevskis.

About the Challenge

This healthcare systems data can augment study findings synthesized in systematic reviews in several ways, including by filling evidence gaps identified in the systematic review to strengthen the available evidence and by examining the applicability of systematic review findings to real-world populations, including population subgroups not examined in published studies. Ultimately, this will help AHRQ understand if and how sources of data and information outside of traditional systematic reviews, particularly from healthcare systems, could be used alongside systematic reviews to improve clinical decision making, healthcare delivery, and patient outcomes.

Challenge Goal

The AHRQ Evidence-based Practice Center (EPC) Program is interested in learning how analysis of real-world data collected by healthcare systems can be used in conjunction with findings from an AHRQ systematic review to inform healthcare decision-making in the context of a specific local setting. 

This challenge aims to explore and determine the feasibility, resources, and infrastructure needed to incorporate unpublished healthcare system data into systematic review findings. Ideally, these data will enable the healthcare system to decide which practices to incorporate locally and how to overcome barriers to implement the evidence to improve clinical practice, healthcare system operations, and, potentially, health outcomes.

Background

AHRQ’s Evidence-based Practice Center (EPC) Program produces systematic reviews which synthesize information from the peer-reviewed literature and provide the state of the science on available healthcare technologies and healthcare delivery strategies. Creating these reviews is stakeholder-driven, methodologically rigorous, and transparent. Reviews are used to inform healthcare decisions such as clinical practice guidelines and national coverage determinations by Medicare. AHRQ also supports healthcare systems in their efforts to improve the quality of care and optimize patient outcomes, and systematic reviews are scoped to address issues of priority to healthcare systems. Yet, due to limitations in the literature base, EPC systematic reviews may be inconclusive or only represent a narrow patient population, making it difficult to generalize or implement the findings within heterogeneous healthcare systems.  

Systematic reviews may also lack contextual details that can inform successful implementation. Improving healthcare delivery (and thus patient outcomes) often entails addressing issues beyond the benefits or harms of an intervention, traditionally the objective of a systematic review. Traditional reviews may not explain gaps in uptake or use of a clinical service and questions about how best to implement a given clinical service (e.g., details around implementation of a service or intervention).

A recent EPC methods report articulates specific scenarios with examples where healthcare system data may most effectively complement systematic reviews (i.e., to improve the strength, applicability, and implementation of evidence). For example, investigators at the Mayo clinic found that published evidence on outcomes following total pancreatectomy was sparse, so they supplemented a meta-analysis of published studies with their unpublished healthcare system data, which more than doubled the sample size and improved the strength of evidence available. 

In another instance, secondary analyses of Veterans Affairs (VA) data confirmed the applicability to the VA of findings from a published systematic review.

The recent EPC methods report also outlines important limitations and considerations when using unpublished healthcare system data alongside systematic reviews, such as relevant limitations in information and data quality. However, the report  did not address the necessary resources, skills, partnerships, and processes required to utilize healthcare system data alongside systematic reviews to strengthen the actionability of systematic reviews.

This Challenge, therefore, invites applicants to conduct an analysis of healthcare system data to supplement an existing AHRQ EPC Program systematic review. This will help AHRQ understand if and how sources of data and information outside of traditional systematic reviews, particularly from healthcare systems, could be used alongside systematic reviews to improve decision making, healthcare delivery, and patient outcomes.

Challenge Design

• All evidence reports (systematic reviews, rapid reviews, and technical briefs) published on the AHRQ website since January 2018 may be considered for this Challenge.
• Healthcare systems and other provider groups interested in implementing evidence at their sites may apply. 
• An organization may choose to address no more than two systematic reviews, submitting a separate proposal for each review.
• Teams should have expertise in the clinical topic, evidence-based practice, data analysis, and quality improvement.

This Challenge consists of two phases:

Phase 1: Proposal

Elicit written proposals on a topic for which an AHRQ evidence report (systematic review, rapid review, or technical brief) has been published since January 2018 and that is relevant to a dilemma in the applicant’s healthcare system. Each proposal is to be written in the form of a narrative that:

1. explicitly states the rationale (e.g., addressing evidence gaps to strengthen available evidence, examining the applicability of findings to real-world patients) to complement conclusions of an EPC report with an analysis of health system data, including a discussion of possible limitations of the analysis.
2. develops and describes an analytic plan for the use of healthcare system data (EHR data from an individual healthcare system or networks of healthcare systems [for example, PCORnet, Epic's Cosmos, etc.]).
3. provides an approach for decision-making based on the data analysis and evidence report results and an evaluation of the decision-making process and results.
4. describes potential challenges, barriers, and strategies to complete the analysis successfully.
5. lists team members, their roles, areas of expertise, and hours on the project.

Each proposal will be considered and evaluated on its own merit.

A total of 5 proposals will be selected as winners for Phase 1.

Phase 2: Analyses

Healthcare systems selected as winners in Phase 1 will be invited to provide a written narrative that:

1. includes a complete analysis of internal real-world data and appraisal of the analysis for risk of bias using a proper tool such as the ROBINS I.
2. specifies how the findings from unpublished data support, refute, and otherwise complement findings from the published evidence examined in the systematic review. If the unpublished data conflict with the AHRQ review’s conclusions, discuss possible reasons for the discrepancy (e.g., challenges with internal validity of healthcare system data analysis related to study design and methods used, or challenges with  external validity concerning population subgroups [gender, race/ethnicity, multimorbidity] examined in the healthcare system data).
3. describes how the unpublished data informed decision making (e.g., adapt, adopt, abandon, prioritize).
4. reports on solutions to any barriers encountered to using healthcare system data alongside a published evidence review, including barriers to access to healthcare system data, interoperability of data sources, and data analysis.
5. briefly describes potential approaches to implement or deimplement the evidence, including the use of clinical advisories, clinical pathways, clinical decision support, or any other method. The plan should describe anticipated risks and barriers and strategies for successful implementation. Decisions made against uptake should be justified.

Timeline and Prize Amounts

AHRQ is hosting this challenge as a two-phase competition. All costs associated with developing and submitting proposals and conducting real-world data analysis of real-world data will be the responsibility of the Challenge participant. Cash prizes will be awarded only after the projects are evaluated and determined acceptable at the end of Phase 2.

Timeline

• September 26, 2022—Challenge launch.
• January 23, 2023, by 7:00 p.m. ET—Submissions for Phase 1 (written proposals) are due. AHRQ will complete the review of the proposals within 6-8 weeks of closing the announcement.
• March 30, 2023—AHRQ will announce the Phase 1 winners. Phase 2 of the Challenge will commence once the Phase 1 winners are announced and notified by March 31, 2023. The AHRQ team will schedule a live, virtual technical assistance webinar with all winners of Phase 1 to discuss scope of content, accessibility/compliance with Section 508, and address questions that the winners may have.
• March 31, 2023—Phase 2 participants will have 120 calendar days from notification to create and submit their analyses as described in their proposal(s). The deadline to submit the analysis is July 31, 2023, by 7:00 p.m. ET.
• Fall 2023—The final winners of Phase 2 of the competition will be announced in October 2023.

Prize Amounts

The top five entries in Phase 1 will be invited to participate in Phase 2. Upon completion of Phase 2, each of the top five entries will each receive cash awards of $50,000. Additionally, the Phase 2 first-place winner will be awarded an additional $150,000, and the second-place winner will be awarded an additional $100,000. The winner and runner up from Phase 2 will be posted on the AHRQ website.

Participants in Phase 2 may be disqualified if their submitted analyses deviate from their winning proposals.