Evaluating Your TeamSTEPPS Intervention To Improve the Safety of LEP Patients

Contents

Introduction
    Why evaluate?
    Who should evaluate?
    How do you evaluate?
Basic Evaluation Checklist
Survey Instruments and Metrics
    Readiness Assessment Survey
    Course Evaluation
    Learning Outcomes Assessment
    Unit Staff Pretraining Behavior Survey
    Posttraining Behavior Survey
    Process Evaluation
Basic Analysis Plan
 

Introduction

 Why evaluate?

• To make sure the time is right to implement the intervention.
• To document your efforts.
• To see if it worked.
• To make it work even better.
• To help you tell your story to staff members and senior leaders.

 Who should evaluate?

The change team should assign someone to lead the evaluation task. This person should ideally have some experience with evaluation, but a novice can follow the simple evaluation checklist and analysis plan provided below. The change team should ideally be a group of five or six people with the expertise, credibility, and motivation to drive a successful TeamSTEPPS initiative.

 How do you evaluate?

This guide provides a basic step-by-step evaluation plan for the LEP Patient Safety module, including metrics and an analysis plan. We include metrics at four levels: trainees' immediate reactions; learning; behaviors that changed because of the training; and results (i.e., patient outcomes).^1,2 The higher you go in the levels, the more rigorous your evaluation will be. This guide also explains how to conduct a basic process evaluation.

1. Kilpatrick DL. Evaluating training programs. San Francisco: Berrett-Koehler Publishers; 1998.
2. Salas E. Patient safety training evaluations: reflections on level 4 and more. Agency for Healthcare Research and Quality Annual Meeting, Rockville, MD, September 2009.

Return to Contents

Basic Evaluation Checklist

Before you commit to a TeamSTEPPS intervention

• Complete the readiness assessment questionnaire.
• Decide what result you want to achieve.

At least 2 weeks before the training

• Consider on which levels you will evaluate (1, 2, 3, 4?).
• Customize the metrics to your information needs, if desired.
• If doing a level 4 evaluation:
  • Collect baseline data. This can be the patient survey provided below or other metrics that reflect the result you want to achieve.

During the training session

• Right before you start:
  • Administer pretraining behavior survey.
  • Administer learning outcomes survey.
• Right after you finish:
  • Administer training participant satisfaction survey.
  • Repeat the learning outcomes survey.

6-8 weeks after the training

• Conduct posttraining behavior survey.
• Collect level 4 data (patient survey or other metrics of your choosing).

When you've got your data

• Analyze the data:
  • Frequency counts.
  • Bar graphs.
  • Means.
  • Pre-post differences in scores.
  • Statistical tests (if possible).
• Tell your story:
  • Quantitative results.
  • Process evaluation.
  • Conclusion, next steps, recommendations.

Return to Contents

Survey Instruments and Metrics

Readiness Assessment Survey

Is Your Organization Ready for This TeamSTEPPS Module?

These questions can help you assess your institution's readiness to implement the TeamSTEPPS module to improve safety for patients with limited English proficiency (LEP) You may find it helpful to have a colleague review your responses or to answer the questions with a larger group (e.g., senior leaders, including patient safety officers, leaders of the units where you plan to implement the module, leaders in interpreter services, motivated frontline staff).

Does your institution have patients with limited English proficiency (LEP)?

[__]  Yes

[__]  No

Are key leaders in your institution committed to providing excellent care to all patients regardless of race, ethnicity, country of origin, or language spoken?

[__]  Yes

[__]  No

Does your institution have a system for staff to access an interpreter for LEP patients?

[__]  Yes

[__]  No

Will your institution allow time for at least two persons to attend training as master-trainers (30 minutes of prework, 4.5 hours of training, plus travel), and customize the course (4 to 8 hours prep time)?

[__]  Yes

[__]  No

Will your institution allow time for all staff in at least one unit to attend training, along with some of the hospital's interpreters (1.5 hours)?

[__]  Yes

[__]  No

If needed, would key leaders in your institution consider making system changes or allocating additional resources to improve care for LEP patients?

[__]  Yes

[__]  No

If you answered "no" to any of the questions above, your institution may not be ready to implement the TeamSTEPPS module to improve the safety of LEP patients. Please refer to the accompanying hospital guide for steps you and your institution can take to increase your organization's readiness.

Return to Contents

Course Evaluation

Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS)

Location: ____________________________________ Date: ________________________________________

Response scale: 1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = strongly agree.

Module: Enhancing Safety for Patients With Limited English Proficiency (LEP)
Instructor Name:	Strongly disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Strongly agree
Please rate how strongly you agree or disagree with each of the following statements by circling the number corresponding to your answer
1. The speaker was knowledgeable, organized, and effective in his/her presentation	1	2	3	4	5
2. The teaching methods and aids were used effectively	1	2	3	4	5
3. I improved my understanding of patient safety risks to LEP patients	1	2	3	4	5
4. I learned the process to assemble the most effective care team for LEP patients	1	2	3	4	5
5. I learned how to identify and raise patient communication issues due to language or cultural barriers	1	2	3	4	5

6. What in this training did you find MOST useful?

7. What in this training did you find LEAST useful?

8. What could be done to improve this training?

9. How do you plan to use what you learned in this training?

Return to Contents

Learning Outcomes Assessment

For each of the following questions, circle the letter next to the one best answer.

1. A 22-year-old male patient, Alejandro, arrives at the emergency department (ED) by ambulance in a coma, accompanied by several family members. The family members speak Spanish and Alejandro's mother, Juanita, speaks some English. The ED triage nurse, Jim, says he will call an interpreter, but Juanita says, "No, I speak English." Jim should:
   1. Drop the issue so as not to further offend the patient's mother.
   2. Drop the issue, but make a note in the record that an interpreter may be needed later.
   3. Ask the patient's mother to sign a waiver.
   4. Acknowledge that her English is quite good but say it's the hospital's policy to call an interpreter to avoid misunderstandings.
2. What does the acronym "CUS" stand for in the TeamSTEPPS patient safety system?
   1. Create Understanding and Safety.
   2. Concerned, Unify, Standardize.
   3. Correct, Undo, Study.
   4. Concerned, Uncomfortable, Safety.
3. Maria, age 20, is scheduling a knee surgery appointment for her mother, Rigoberta, a 54-year-old Guatemalan patient who does not speak English. The receptionist, Carolyn, knows that one of the nurses, Betty, speaks Spanish. When she schedules Rigoberta's appointment, Carolyn should:
   1. Make sure Betty is scheduled to do Rigoberta's intake assessment.
   2. Schedule a Spanish-speaking interpreter for Rigoberta's appointment.
   3. Ask Maria what language her mother speaks best, and arrange for an interpreter of that language.
   4. Ask Maria to come to the appointment with her mother so she can interpret.
4. Through the interpreter, Nurse Arroyo tells the patient, Mr. Chin: "Take this medication once every 12 hours, with food." The patient looks confused. The interpreter should:
   1. Say to the nurse, "Let me check back with you. This medicine is to be taken after eating twice a day, once in the morning and once at night, is that correct?" and interpret the nurse's response.
   2. Say to the patient, "Mr. Chin, let me check back with you. Do you understand how to take your medicine?" and interpret his response to Nurse Arroyo.
   3. Say to the patient, "Mr. Chin, let me check back with you. That means you should take the medicine after eating twice a day, once in the morning and once in the evening after eating."
   4. Say to the nurse, "Let me check back with you. Do you think Mr. Chin understood how to take his medicine?"
5. Nurse Angelis is preparing to discharge Mr. Lo, a 65-year-old Hmong patient who was hospitalized for asthma. She explains to Mr. Lo how to use a single-dose dry powder inhaler at home. The interpreter, Mr. Khang, interprets Nurse Angelis's instructions. The patient nods. Nurse Angelis states that Mr. Lo is ready to be discharged. Mr. Khang worries that Mr. Lo may not have fully understood Nurse Angelis's instructions. Mr. Khang's responsibility is to...
   1. Interpret the nurse's and patient's words to one another without adding or subtracting anything.
   2. Interrupt the conversation to state his concern.
   3. Report the incident to his supervisor.
   4. Follow Mr. Lo out of the room and offer to provide a more detailed explanation.

Answer key:

1.   D
2.   D
3.   C
4.   A
5.   B

Return to Contents

Unit Staff Pretraining Behavior Survey

As part of our quality improvement activities, we are conducting a brief survey about the care we provide for patients with limited English proficiency (LEP). Please answer the questions below truthfully to the best of your abilities and return the survey to ___________________________. Your answers will be confidential. We appreciate your time and honesty.

S2. Are you an interpreter?

1 [__] Yes → Do not complete this questionnaire 
2 [__] No

1. What languages do you speak very well?
   1. English

      1 [__] Yes
      2 [__] No

   2. Spanish

      1 [__] Yes
      2 [__] No

   3. Other languages (list here) _________________________________

      ______________________________________________________

      ______________________________________________________

2. About what percentage of your patients do not speak any of those languages very well?

______________%

3. For those patients who do not speak of any of those languages very well, about what percentage of your encounters take place in the presence of a trained interpreter?

______________%

4. What comments do you have about why you do or do not use interpreters to communicate with LEP patients whose language you do not speak very well?

The next questions focus on some things we will cover in the TeamSTEPPS training on LEP patient safety. We’re just checking to see if you’ve used these tools before.

How many times in the past 2 months have you used the following tools with LEP patients?

5. Assertion or advocacy on behalf of an LEP patient (for example, to get an interpreter to the encounter or to make sure the doctor or nurse explained things more clearly)

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

6. Expressing concern or discomfort about communication involving LEP patients when there is a patient safety issue

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

7. Briefs that include the interpreter (a brief is a short meeting of the care team before seeing the patient where the team leader describes team roles, goals, plans, and risks)

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

8. Creating psychological safety, such as saying, “Please let me know if anything I say is not clear.” The interpreter can do this for the patient, and other members of the care team can do this for the interpreter.

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

9. Check-back with the patient (repeating out loud what the patient said to check your understanding)

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

Posttraining Behavior Survey

S1. Did you take the TeamSTEPPS training on improving safety for patients with Limited English Proficiency (LEP)?

1 [__] Yes
2 [__] No

S2. Are you an interpreter?

1 [__] Yes → Skip to Question 4
2 [__] No

1. What languages do you speak very well?
   1. English

      1 [__] Yes
      2 [__] No

   2. Spanish

      1 [__] Yes
      2 [__] No

   3. Other languages (list here) _________________________________

      ______________________________________________________

      ______________________________________________________

2. About what percentage of your patients do not speak any of those languages very well?

______________%

3. For those patients who do not speak of any of those languages very well, about what percentage of your encounters have taken place in the presence of a trained interpreter since the TeamSTEPPS training on improving safety for LEP patients?

______________%

4. What comments do you have about why you do or do not use interpreters to communicate with LEP patients whose language you do not speak very well?

______________________________________________________

______________________________________________________

Since the TeamSTEPPS training on LEP patient safety, how many times have you used the following tools with LEP patients?

5. Assertion or advocacy on behalf of an LEP patient (for example, to get an interpreter to the encounter or to make sure the doctor or nurse explained things more clearly).

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

6. Expressing concern or discomfort about communication involving LEP patients when there is a patient safety issue.

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

7. Briefs that include the interpreter (a brief is a short meeting of the care team before seeing the patient where the team leader describes team roles, goals, plans, and risks).

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

8. Creating psychological safety, such as saying, "Please let me know if anything I say is not clear." The interpreter can do this for the patient, and other members of the care team can do this for the interpreter.

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

9. Check-back with the patient (repeating out loud what the patient said to check your understanding).

1 [__] Never
2 [__] Once
3 [__] Once to 5 times
4 [__] More than 5 times

The next questions focus on your experience of implementing these tools.

10. How easy is it to incorporate TeamSTEPPS tools into your daily routines?

1 [__] Very easy
2 [__] Easy
3 [__] Hard
4 [__] Very hard

11. What would make it easier to incorporate TeamSTEPPS tools into your daily routines?

 

 

12. What additional training would you find helpful to improve safety for LEP patients?

 

 

Patient Outcome Survey¹ 

Screening Question

What is your preferred language?

  1 [__] English → If English, end questions
  2 [__] [Insert language 2]
  3 [__] [Insert language 3]
  4 [__] [Insert language 4]
  5 [__] Other
 

1. How well do you speak English?
1 [__] Very well
2 [__] Well
3 [__] Not well
4 [__] Not at all

2. How well do you understand English?
1 [__] Very well
2 [__] Well
3 [__] Not well
4 [__] Not at all

If 1 and 2 = very well, end questions

---

1 This survey instrument is based on a new HCAHPS supplemental item set that was under development as of September 2012. If you will be implementing a patient outcomes survey, please check the CAHPS Web site at www.ahrq.gov.

---

3. During this hospital stay, how often did hospital staff speak to you in your preferred language?
1 [__] Never
2 [__] Sometimes
3 [__] Usually
4 [__] Always

4. During this hospital stay, how often did hospital staff explain things in a way you could understand?
1 [__] Never
2 [__] Sometimes
3 [__] Usually
4 [__] Always → If 3. and 4.=Always, Go to Question 11

5. An interpreter is someone who helps you talk with others who do not speak your language. Interpreters can include hospital staff or telephone interpreters. During this hospital stay, was there any time when you needed an interpreter?
1 [__] Yes
2 [__] No

6. During this hospital stay, did hospital staff let you know that an interpreter was available free of charge?
1 [__] Yes
2 [__] No

7. During this hospital stay, how often did you use an interpreter provided by the hospital to help you talk with hospital staff?
1 [__] Never
2 [__] Sometimes
3 [__] Usually
4 [__] Always

8. During this hospital stay, did you use a child younger than 18 to help you talk with hospital staff?
1 [__] Yes
2 [__] No
 

9. During this hospital stay, how often did you use a friend or family member as an interpreter when you talked with hospital staff?
1 [__] Never → If Never, Go to Question 11
2 [__] Sometimes
3 [__] Usually
4 [__] Always

10. During this hospital stay, did you use friends or family members as interpreters because that was what you preferred?
1 [__] Yes
2 [__] No
 

 11. During this hospital stay, did hospital staff tell you how to take care of yourself at home?
1 [__] Yes
2 [__] No → If No, Go to Question 13
 

12. Was this information easy to understand?
1 [__] Yes
2 [__] No
 

13. During this hospital stay, did you get instructions in writing about how to take care
of yourself at home?
1 [__] Yes
2 [__] No → If No, Go to Question 17

14. Were the written instructions easy to understand?
1 [__] Yes
2 [__] No

15. Did you need instructions in a language other than English?
1 [__] Yes
2 [__] No → If No, Go to Question 17

16. Were the instructions available in your preferred language?
1 [__] Yes
2 [__] No

Overall Rating of Hospital

Please answer the following questions about your stay at the hospital named on the cover. Do not include any other hospital stays in your answer.

 17. Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible, what number would you use to rate this hospital during your stay?
0 [__]  0 Worst hospital possible
1 [__]  1
2 [__]  2
3 [__]  3
4 [__]  4
5 [__]  5
6 [__]  6
7 [__]  7
8 [__]  8
9 [__]  9
10 [__]  10 Best hospital possible

18. Would you recommend this hospital to your friends and family?
1 [__] Definitely no
2 [__] Probably no
3 [__] Probably yes
4 [__] Definitely yes

Return to Contents

Process Evaluation

Documenting your process can help you tell your story. In turn, this can help you justify your time investment, explain why the intervention is working or not working, and identify any changes you may need to make to improve your intervention's outcomes. You can use the template below as a model.

Institution name: _______________________________________________________________________

Unit(s) where intervention will be/ has been implemented: _________________________________

Name of person(s) completing this process evaluation: _____________________________________

Position/change team role: ______________________________________________________________

Time period evaluated:    Start: _________________    End: __________________

INPUT

Briefly describe the resources that your institution has invested in this TeamSTEPPS intervention. Include time spent on training of trainers, staff time to attend the training, and any equipment and supplies that your institution may have provided, such as markers and flipcharts. Bullet-points are sufficient.

ACTIVITIES

Briefly describe what you did to implement this TeamSTEPPS module. This may include trainings and/or process improvement interventions. Bullet-points are sufficient.

OUTPUT

Use this space to quantify what you have done. For example, how many training sessions did you complete? How many people did you train? Bullet-points are sufficient.

EXPERIENCE

In this space, share your experience implementing this module and tell your story.

• How did you pick the units that would get the intervention?
• What made it easy or hard?
• How much support did you get from leadership in patient safety, language services, nursing, or other areas?
• Was it easy or hard to engage certain types of team members and why?
• Did other quality improvement initiatives interfere with or enhance your TeamSTEPPS intervention?
• What else was going on in the hospital or in the broader environment at the time that may have affected your results?
• How will changes be sustained?
• Were any hospital or unit policies changed as a result of the module?

Return to Contents

Basic Analysis Plan

The goal of the analysis is to assess what participants thought of the training (at level 1) and what changed as a result of the training (levels 2, 3, and 4). Once you've collected your data and entered it into a database (e.g., Access®) or spreadsheet (e.g., Excel®), you are ready to analyze.

A basic analysis plan might include:

• Tabulating frequencies for each survey item.
• Creating bar charts (go to Figures 1 and 2, below, for examples).
• Creating tables comparing survey responses before and after.
• Comparing average scores before and after the training on learning outcomes and selected patient outcomes.
• Writing a process evaluation report.

 

Figure 1. Use of CUS words by OB-GYN staff and interpreters in the past month

 

Figure 2. How often patients needed an interpreter but did not get one

You can also run some statistical tests to see if the changes you see are significant or so small they could be due to chance. There are free Web sites where you can run the tests just by entering your data and pressing a button, such as http://www.graphpad.com/quickcalcs/index.cfm.

• " Use a t-test to assess pre- and post-test differences in:
  • Average scores on learning outcomes.
  • Q3 on behavior surveys (% of LEP patient encounters that take place with an interpreter present).
  • Any continuous variables where you want to assess the difference in means before and after the training.
  Graphpad offers free t-tests here: http://www.graphpad.com/quickcalcs/ttest1.cfm.
• Use a binomial test to compare pre- and post-test training results on any variable that has only two response categories (e.g., yes or no). Graphpad offers free binomial tests here: http://www.graphpad.com/quickcalcs/binomial1.cfm.
  • Use pretest proportions as your "hypothetical probability" for this test. This means you are testing the hypothesis that your proportions are the same as before and that any variation is so small it is likely due to chance.
  • If you have a variable with more than two categories (e.g., Definitely no/Probably no/Probably yes/Definitely yes), you can recode it into two categories (e.g., yes or no) before running the binomial test.

This basic evaluation will show any changes after the training, but you won't know for sure that the training caused the change. Be sure to use your process evaluation to identify anything else that may have caused or contributed to the change.

You can also do a more rigorous evaluation to rule out alternative explanations for the results that you see. For example, you can use a comparison group or a randomized controlled trial. Be sure to work with a trained evaluator if you do.