Measurement

Once the problem or challenge has been identified, measurement will play an important role in defining the magnitude of the problem, assessing the short-term reactions to the TeamSTEPPS training, and ultimately demonstrating improvements that follow TeamSTEPPS implementation.

Measurement is the process of classifying based on the characteristics of an object or action. Objects or actions are sometimes classified into distinct groups (e.g., physician, nurse, pharmacist), and sometimes the grouping is based on the level the object or action falls into (e.g., rapid, normal, or slow heart rate; number of hours of sleep).

Measurement plays an important role in each implementation phase:

• Pre-implementation: Measures are used to assess readiness and to compile baseline data.
• Initial implementation: Measures are used to assess reactions to trainings; they are also used to assess progress on targeted processes (for example, are staff beginning to use TeamSTEPPS tools?). 
• Continued implementation and sustainability: Measures are used to assess process and outcome changes, including improvements in culture, team behaviors, and targeted clinical outcomes.

Across these phases, all measures can be viewed as fitting into one of four general categories proposed by Kirkpatrick.¹ While Kirkpatrick's focus was on how to measure learning, the four levels encompass all the measure types needed to plan, train for, and implement TeamSTEPPS in an organization.

Multi-Level Approach to Measurement

Levels I and II will be relevant for initial training. Whether students liked the training and appear to have learned TeamSTEPPS concepts and tools are examples of level I and II measures.

• A course evaluation form (PDF, 354 KB) can provide information about trainees' reactions to the training. Other, organization-specific training evaluation forms can be used instead or as well. An evaluation form can tell something as simple as whether the participants liked the course and whether they found it valuable. 
• The Teamwork Attitudes Questionnaire (T-TAQ) (PDF, 193 KB) is a 30-item self-report tool that measures attitudes toward team structure, leadership, situation monitoring, mutual support, and communication. Sometimes the questionnaire is administered before and after trainings. While it's important to know whether attitudes have changed, short-term attitude change is not a substitute for actual changes in behaviors or outcomes.
• Learning Benchmarks (PDF, 247 KB) can be used to see whether factual knowledge improved through training. The curriculum includes a test of overall TeamSTEPPS concepts. If you teach only a single module or concept, you can choose items from this list or write a few questions of your own.
• Listening to participants is a less formal, but equally critical, way to assess learning. Asking training participants to describe what they've learned can help determine whether they fully understood the concepts or tools they were taught. If they provide inaccurate explanations, future trainings can be adjusted to avoid any misconceptions that might have surfaced in training conversations.

Level III: Needs assessments and culture measures include elements of both level II and level III measures, as the measures assess both feelings about the organization and organizational behaviors.

• The Team Performance Observation Tool (PDF, 286 KB)   can be used to assess the use of recommended behaviors either before or after a training or intervention. As needed, modify this tool based on the specifics of individual units or organizations. Observing what staff do in practice often has considerable value. If training does not change behavior, it was not successful. This change will probably not be immediate and almost always requires practice and positive reinforcement. An excellent opportunity to use this tool is during low- and high-tech clinical and tabletop simulations.
• If you don't see changes in behavior, ask staff why. If they don’t fully understand a key concept or tool, more or better training may be the solution. Taking time to understand why training isn't changing behaviors shows that key leaders are listening, want to see behavior change, and are acting to address underlying issues that may surface in conversations with staff. It is easy to administer a questionnaire. Observing staff may make some people uncomfortable, which is why simulations are a great opportunity to use this tool. Informal conversations that surface causes and potential solutions will model the type of teamwork TeamSTEPPS seeks to foster.
• The Teamwork Perceptions Questionnaire (TPQ) (PDF, 209 KB) can be used to learn staff perceptions of key TeamSTEPPS concepts either before or after training. Sometimes training can sensitize trainees to inappropriate behaviors, resulting in short-term reductions in ratings. Therefore, direct observation or other direct measures of process or outcome change are helpful to use in addition to the Perceptions Questionnaire. The questionnaire can be used for baseline perceptions and to guide the TeamSTEPPS Implementation Plan.
• Other process measures may make a stronger case for TeamSTEPPS than any of the measures noted above. If teaching TeamSTEPPS to fix specific problems, ask:
  • What behaviors do we want to change?
  • How will we know whether those behaviors have changed?
• To encourage better communication with patients or among team members, you may want to use standard AHRQ Consumer Assessment of Healthcare Providers and Systems (CAHPS) questions to assess perceptions of communication by patients. Results from an employee safety culture survey can depict how communication is perceived. Online customer ratings may provide useful insights about some behavioral changes you are attempting to make. Organizations measure many things; if existing organizational data can be leveraged to assess behavior change, the results will be perceived as more credible and data collection will be less burdensome. 

Level IV: While measurements at levels I and II can provide short-term reactions to trainings and adjust for effectiveness, a business case for TeamSTEPPS will require measures of behavior and results. These may include measures of errors or near-misses, staff turnover, or clinical outcomes targeted by your intervention.

Results fall into two general categories. If you have implemented TeamSTEPPS to address a specific safety or care quality issue, the most important results should be whether relevant clinical processes and outcomes have changed. If you are trying to improve the overall culture to prioritize patient partnerships, communication, and patient safety among leadership and staff, you may focus on changes in culture.

Clinical Processes and Outcomes

• Examples of process and outcome measures include length of patient wait time, time to intubate, medication administration delays, compliance with preventive screenings, number of misdiagnoses, number of structured handoffs used, or other processes or outcomes directly linked to communication and teamwork. Your organization may already capture the measures that relate to the problem you were trying to address. Using existing measures is preferable to creating new ones, as existing measures may be less burdensome to collect and more credible to your organization.
  
  When using existing measures, be sure to:
  • Confirm that the areas included in the measurement match the areas where TeamSTEPPS was implemented. If data are for the entire hospital but you only implemented TeamSTEPPS in a specific unit, you’re not likely to see large changes in the overall results. Work with the data team to see whether results can be provided for the areas where TeamSTEPPS was implemented.
  • Ensure measures aren’t heavily affected by factors that improved teamwork and communication cannot change. If delays in administering medications are caused by a shortage of staff in the in-house pharmacy, improved teamwork may not shorten the delays. The better matched your TeamSTEPPS intervention and your process measures are, the more clearly you’ll be able to see whether TeamSTEPPS has made a difference.
  • Account for timing. Changing processes takes time, and outcome measurements sometimes have delays before they are available. Be sure that the data to evaluate process and outcome changes come from the period that follows the TeamSTEPPS implementation.

Culture Change

• You can assess your organization’s safety culture by using one of the safety culture surveys AHRQ has developed for hospitals, medical offices, nursing homes, and pharmacies. Getting a high completion rate from staff members is important for all these measures because staff who do not respond may have different views of the safety culture than respondents. If you use a safety culture survey in a small organization or for a unit with a limited number of staff, a high response rate is even more imperative.
• While staff perceptions of the safety culture are important, patient perceptions are also vital to track. You can use data from patient experience surveys or information from patient or family advisory groups to see how patients perceive an organization’s commitment to safe and high-quality care. Additional information on the range of surveys available for use in different healthcare contexts is available at AHRQ's CAHPS website.

Present Slide 12, "Measurement." Use this slide to introduce the topic of measurement and note that it is key to all organizational improvement initiatives.

In discussing Slide 13, "Define a Measurable Goal or Problem You Plan To Address," it’s important to stress that TeamSTEPPS always should have a defined goal or be introduced in response to a clearly identified problem you seek to address. If the goals are unclear, any accomplishments through TeamSTEPPS may not be possible to realize. Also explain that effective measurement requires clarity about the problem and what observable changes are expected.

Present Slide 14, "Measurement Is Foundational to Implementation." As you introduce the concept of measurement, note that it’s an important topic throughout the implementation process. It is being discussed as a pre-implementation topic because quantifying your problem at the beginning is important and so is measuring your readiness to implement TeamSTEPPS.

Present Slide 15, "How To Measure," noting that all measures likely to be used during a TeamSTEPPS implementation can be classified based on a widely used grouping of measures developed by Kirkpatrick and used in adult education. Explain that the first two levels are primarily used to assess reactions and learning after trainings, while behavior changes and results are what make TeamSTEPPS worthwhile. Stress the need for TeamSTEPPS to ultimately change results or outcomes within your organization.

Present Slide 16, "TeamSTEPPS Measurement Option Overview"; use this slide to illustrate types of measures available on the TeamSTEPPS and other AHRQ websites that can be used to evaluate TeamSTEPPS trainings and outcomes.

In displaying Slide 17, "Level III: Behavior Measures," briefly note the types of level III measures included on the website.

Present Slide 18, "Level IV: Results." Briefly note the types of level IV measures included on the website. It's important to emphasize that behavior and results measures are most important to the success of your overall implementation. If members of your class intend to use specific tools, you may want to spend more time discussing them. Have the class share outcome or process measures that they would like to use TeamSTEPPS to improve.

In discussing Slide 19, "AHRQ Patient Safety Indicators," you may (if it’s relevant) want to note that the AHRQ PSIs are useful tools for tracking outcome changes.

Present Slide 20, "Accurately and Efficiently Measuring Outcomes." Because outcomes are often neglected during TeamSTEPPS implementation, discuss each bullet on this slide. In addition, you may want to discuss these insights with the Change Team your organization has formed to lead the TeamSTEPPS implementation, to aid them as they select measures and begin to evaluate whether targeted outcomes are improving.

Present Slide 21, "Evaluating TeamSTEPPS Exercise." If you have time, the exercise on this slide is a useful way to consolidate and reinforce the discussion of measurement. Ask the class questions such as:

• When would you be most concerned about measuring reactions and learning?: (after the trainings)
• What process and outcome measures would be of most concern?: handwashing (process), surgical site infection rates (outcomes) 
• What would be important to consider when gathering data on these process and outcome measures?: (Were the data from the units participating; were there data before and after the implementation; how long after the training were you expecting the changes in processes and outcomes to occur?)

Note

1. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick's Four Levels of Training Evaluation. Alexandria, VA: Association for Talent Development, 2016.