How Will We Measure the Impact of the Innovation?
Why This Matters
The only way to know if an innovation is an improvement is to measure its impact. Done correctly, measurement will let you monitor the implementation process, early results, and ultimate outcomes of the innovation. Remember the adage, “What gets measured gets done.”
Key Questions to Consider
- How do we evaluate the innovation?
- What measures should we use?
Question 1. How do we evaluate the innovation?
Tracking progress and measuring impact of an innovation requires time and effort. The first step is to plan out your evaluation strategy.
- Articulate your evaluation goals. Think about how you want to use the information (e.g., accountability, improvement, expansion decisions, public relations) and who the audience will be.
- Formulate your evaluation questions. Decide what you (and your audiences) will want to know in the end.
- Decide who will perform the evaluation. If staff members do not have the necessary skills and resources to conduct the evaluation, consider how you might augment in-house expertise.
- Consider the data sources available to you and how much effort you are willing to expend on collecting new data.
- Select the measures that reflect your organizational priorities.
- Estimate how long it will take to get results.
The Innovation Network offers a variety of tools, including a workbook to plan an evaluation of implementation and outcomes. You can find it at http://www.innonet.org/client_docs/File/evaluation_plan_workbook.pdf (If you don’t have the software to open this PDF, download free Adobe Acrobat Reader® software.)
The W.K. Kellogg Foundation logic model handbook includes a section on posing evaluation questions. See p. 35 at http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx.
Question 2. What measures should we use?
Developing a measurement set requires balancing priorities. On one hand, you want a measurement set that is parsimonious. Too many measures can be burdensome and confusing. However, you also want a measurement set that answers the questions stakeholders will ask and can provide concrete feedback to implementers.
Consider the following when selecting measures:
- Which measures will be meaningful to stakeholders? Which measures will answer your evaluation questions?
- Are the measures valid and reliable?
- Are the measures well specified (e.g., define numerator and denominator, periodicity of data collection)?
- Is the innovation likely to affect the measures during the measurement period?
- Does your measurement set include an appropriate mix of structural, process, outcome, workforce, financial, access, and patient experience of care measures?
- Can you measure intermediary as well as long-term effects? If the impact of the innovation won’t be measurable for a long time, are there surrogate or intermediate measures you can use?
- Will collecting data for measurement be burdensome?
- Are there data or measures already being collected that would be appropriate?
- Can your current data systems and resources capture the measures?
- How will you measure change over time?
- Can you obtain baseline data?
- Will you be able to track changes at appropriate intervals?
- Are there benchmarks with which you can compare your performance?
The Outcome Measurement Guide, developed by the Sharon Martin Community Health Trust Fund, describes outcome measurement and potential indicators. You can find it at http://www.smartfund.ca/docs/smart_outcomes_guide.pdf.
You can find a library of potential measures at the Agency for Healthcare Research and Quality’s National Quality Measures Clearinghouse: http://www.qualitymeasures.ahrq.gov/
The Institute for Healthcare Improvement has an Improvement Tracker, an interactive tool to help monitor the impact of innovations, at http://app.ihi.org/Workspace/tracker/.
Can We Try the Innovation First?
Why This Matters
Innovations that can be tried on a limited basis are adopted and assimilated more easily (Plsek, 2003; Rogers, 2003; Yetton, Sharma, and Southon, 1999). Organizations unwilling to commit to full-scale adoption may find pilot testing an innovation is an acceptable risk. A pilot test may serve as a small test of change on which you can base the larger adoption decision. It can also provide an opportunity to assess fit and the need to adapt an innovation before applying it more broadly. Although pilot testing can be extremely valuable, it is not always feasible; some innovations are not suitable for piloting.
Key Questions to Consider
- Can we try the innovation for a limited time?
- Can we try the innovation on a small scale?
- Can we phase the innovation into the organization?
Question 1. Can we try the innovation for a limited time?
The ability to try the innovation for a limited time before deciding whether to continue depends largely on the nature of the innovation. Questions to consider include the following:
- Can the innovation be evaluated after a trial period?
- How long a trial period is necessary to see the benefits?
- Are the startup costs of a trial period prohibitive?
- How will staff, patients, and other stakeholders react to discontinuing the innovation after a trial period?
- Will discontinuation be disruptive to operations?
The decisionmaking culture at Clinica Campesina is one of trying. Please refer to Section 1.3 of the case studies in the Appendix to find out how they decided to try out the group visit model rather than adopt it irreversibly.
Question 2. Can we try the innovation on a small scale?
Some innovations are designed for specific components or services within a health care organization, while others may be applied more broadly. Review both the nature of the innovation and the nature of the organization when considering a pilot test on a small scale before broader adoption. Questions to consider include the following:
- Is the innovation designed to affect the entire organization, or can it be tried by a smaller component?
- Are economies of scale lost when the innovation is implemented on a small scale?
- How will we measure the results of the pilot?
- Can we generalize from the results of the pilot?
- Will piloting the innovation cause confusion or be disruptive?
- If one component of the organization is selected for a pilot test, how will other components react to not being selected?
- How will we demonstrate the value of the innovation to ensure wider acceptance beyond the pilot test site?
Golisano Children’s Hospital’s champions and decisionmakers considered whether it would be feasible to implement family-centered rounds on a small scale initially. Please refer to Section 3.2 of the case studies in the Appendix to find out how they decided to proceed.
Learn about how to conduct small tests of change through Plan, Do, Study, Act (PDSA) cycles and access the Institute for Healthcare Improvement’s PDSA Worksheet at http://www.ihi.org/knowledge/Pages/HowtoImprove/default.aspx.
Question 3. Can we phase the innovation into the organization?
A careful plan for phasing in an innovation across an organization can build support and promote success. This approach requires several considerations:
- Can the innovation be segmented and implemented over a period of time?
- Can momentum for implementing the innovation be maintained during the phase-in process?
- Does phasing in the innovation increase comfort with the change among key stakeholders?
- What is the likely impact of a prolonged transition period on productivity?
- What is the likely impact of a prolonged transition period on staff support or opposition?
When Group Health Cooperative implemented its depression guideline by deploying a simple screening tool, adoption was slow because physicians did not think that they had time to use the tool. As the tool was introduced, one or two clinicians in each module would volunteer to try it out. Other clinicians in the module observed that it was possible to do, and the model remains in place to this day (http://www.thepermanentejournal.org/files/Spring2005/diffusing.pdf).
How Will We Implement the Innovation?
Why This Matters
No one wants to adopt an innovation that cannot be implemented successfully or cannot be sustained. It pays to think ahead about how you will manage the change process and how you will sustain the innovation once it is implemented.
Key Questions to Consider
- How will we manage change?
- How will we monitor and evaluate implementation?
- How will we sustain the innovation?
Question 1. How will we manage change?
Unmanaged change can result in poor morale, loss of trust in management, and lower productivity. Change that is managed skillfully can strengthen organizational resilience and readiness to adopt innovations in the future. Consider developing a change-management plan that delineates the decisionmaking and implementation process. The plan should address the following:
- Preparation: Explain why the innovation is needed; create a motivating vision that will inspire staff.
- Change management team: Designate who will be responsible for coordinating and implementing the innovation.
- Communications: Define stakeholders (e.g., patients and families, clinical staff, support and administrative staff, unions, middle and upper management, board members) and use a variety of channels to communicate with them.
- Policies and procedures: Articulate a process for changing policies and procedures as needed to support the innovation.
- Building buy-in: Involve participants in the process, coaching, training, and using champions and change agents to build buy-in and manage resistance.
- Monitoring: Develop performance measures, regularly assess performance, and institute corrective action as necessary to keep the innovation on track.
- Reinforce change: Consider performance incentives, morale boosters, and celebrations of success.
The Free Management Library offers several tips and resources for effectively managing the communication, delegation, and planning of the organizational change process: http://managementhelp.org/organizationalchange/index.htm
George Mason University offers an overview of leading change in a continuous quality improvement model. It describes the key role of leadership in change, and outlines necessary elements and clear action steps. http://gunston.gmu.edu/healthscience/708/LeadingChange.asp
The overall guide is here: http://gunston.gmu.edu/healthscience/708/default.asp.
The Department of Veterans Affairs Health Services Research and Development Service provides an organizational change primer containing lessons learned about managing organizational change, as well as pitfalls to avoid: http://www.hsrd.research.va.gov/publications/internal/organizational_change_primer.pdf.
The Score Association offers 5 Tips on Preparing for Change at http://www.score.org/resources/5-tips-preparing-change.
Anticipating potential challenges and developing strategies to overcome them is an important aspect of change management. Resistance to change in varying degrees is common. Many people fear uncertainty, and some may not perceive the need for change. Some staff members may oppose how the change will occur.
Imperial College London has developed guidance on conducting a stakeholder analysis. Learn how to develop an Influence/Interest Grid at http://changingminds.org/disciplines/change_management/stakeholder_change/interest_influence.htm.
A table highlighting six layers of resistance to change is available on the Focused Performance Web site at http://www.focusedperformance.com/articles/resistance.html.
To get staff buy-in for family-centered rounds, decisionmakers at Golisano Children’s Hospital rolled out the decisionmaking process as shared consensus building. For information about how they obtained staff support for this innovation, please refer to Section 3.2 of the case studies in the Appendix.
Though there were naysayers along the way, the visibility of the results from Six Sigma process improvements in the Mt. Carmel Health System created traction for this innovation. Please refer to Section 2.3 of the case studies in the Appendix for more information about how this system overcame challenges they faced in adoption and implementation.
Question 2. How will we monitor and evaluate implementation?
A formative evaluation monitors the implementation process and the progress toward goals, providing information for midcourse corrections. Formative evaluations answer such questions as the following:
- Was the innovation implemented as intended?
- What adaptations to the innovation were made?
- What were the unanticipated impacts of the innovation?
- What corrective action can we take?
- Why did the innovation have the impact it had?
- What influenced its success or failure?
- How can we improve?
Formative evaluations must be done quickly if they are to provide feedback in time to be useful for short-term adjustments.
The Department of Veterans Affairs QUERI Implementation Guide provides an overview of formative evaluation at http://www.queri.research.va.gov/implementation/.
Question 3. How will we sustain the innovation?
You know that energy and resources will be needed to get the innovation off the ground. Be sure to think ahead about maintaining the innovation as well. Otherwise, as enthusiasm for the innovation ebbs or trained staff leave, the organization may slide back into its old patterns. If you are not reasonably confident that you can sustain the innovation, investment in an innovation may not be warranted.
Think about where you want your organization to be in a few years. Ask yourself:
- What does the innovation look like in a steady state?
- How can we institutionalize the innovation?
- When does change management leave off and plain old management kick in?
- How realistic is it to expect the innovation to substitute for any current activities?
- How much staff time and other resources does the innovation require in the maintenance phase (e.g., “booster” training and training of new staff)?
- How can we keep staff engaged?
- How will we celebrate success?
- How will we spread the innovation to other parts of the organization?
A well-rounded approach to program planning can play a pivotal role in achieving intended results from an innovation. A planning checklist, such as the RE-AIM planning tool, can be useful in identifying key issues to consider when evaluating the potential sustainability of a candidate innovation. The Maintenance section of the checklist inquires about anticipated challenges to long-term success, plans for sustainability, stakeholder commitment to the innovation in the long run, and the extent to which the innovation will be integrated into the organization as part of standard practice.
The RE-AIM planning tool is a checklist to facilitate planning related to improving reach, effectiveness, adoption, implementation, and maintenance: http://www.re-aim.org/forms/re-aim_planning_tool_and_adaptation.pdf
The Institute for Healthcare Improvement discusses how to measure the spread of an innovation at http://www.ihi.org/knowledge/Pages/Measures/RateofSpread.aspx.
Within a few years of implementing Six Sigma, Mt. Carmel Health System staff had developed a systematic approach to the selection of projects and each member of the senior management team had a set of guides assigned to him or her. Please refer to Section 2.3 of the case studies in the Appendix for a description of how this innovation was incorporated into this health system’s regular operating procedures.
