Pediatric Measure Set Creation
Step 1: Narrow the Field of Measures
Figure C-1 provides an overview of the measure field narrowing process. Details of each step are provided below.
Inclusions
Table C-1 lists the set of 40 measures that meet the following two inclusion criteria:
- Measures that are applicable to primary care facilities or that are not setting specific.
- Measures that are applicable to children or that are not age specific or for which patient age is not applicable.
Exclusions
In total, 10 measures were excluded because they are not focused entirely on coordination performed by a primary care provider or practice. Specific reasons for the exclusions were:
- Measures assessing coordination by providers or entities other than primary care: 8, 56.
- Measures assessing coordination at the level of the health care system, not primary care practices: 14, 51.
- Measures of discharge from inpatient setting: 9a, 57, 58, 59, 60.
- Measure 43 focuses on general attitudes towards the relative roles of physicians and nurses and specifies a hospital context for some items. Although there may be linkages between attitudes about collaboration and care coordination activities, we are unaware of any evidence base to inform use of these for assessing the quality of care coordination, therefore, this measure is excluded as not reflective of primary care coordination processes.
An additional 4 measures were excluded because they are disease-specific:
- Measure 21 focuses on patients with diabetes.
- Measure 49 focuses on patients with schizophrenia.
- Measure 54 focuses on patients with cardiac conditions or who have undergone cardiac procedures, including acute myocardial infarction, chronic stable angina and cardiac valve surgery.
- Measure 61 focuses on patients with melanoma.
While not age-specific, these 4 measures are also not generally applicable to a pediatric population.
Feasibility and Degree of Focus on Care Coordination
Next, we assessed the feasibility and degree of focus on care coordination for the 26 remaining measures. Table C-2 reports the priority rankings based on this assessment.
Table C-3 lists the set of 26 measures that remain for consideration during the Set Assembly process (Step 2) after applying the exclusion criteria and priority groupings. Twenty measures in Priority Groups 1 and 2 will be considered first in assembling a measure set. A further 6 measures from Priority Group 3 and 4 may be considered if none of the Priority Group 1 and 2 measures are appropriate.
Step 2: Identify the Most Comprehensive Measures
Table C-4 maps measures from priority groups 1 and 2 to the Atlas activity domains. Note that in assessing domain gaps, measures that do not map to the Communicate domain are still considered to be comprehensive if they instead map to both the Interpersonal Communication and Information Transfer sub-domains.
Patient/Family Perspective
Measures with Broadest Domain Coverage:
- The sole Priority Group 1 measure (Measure 45) does not offer comprehensive domain coverage (maps to only 6 of 11 activity domains).
- From Priority Group 2, measure 11a covers all Atlas activity domains and sub-domains except Facilitate Transitions Across Settings and Measure 17a covers all Atlas activity domains and sub-domains except Facilitate Transitions as Coordination Needs Change, and Links to Community Resources.
- Measure 64 is lower priority (Group 2) and has weaker domain coverage (7 of 11 activity domains).
Measure for Detailed Review:
- Measure 11a (Family-Centered Care Self-Assessment Tool—Family Version).
- Measure 17a (Primary Care Assessment Tool—Child Edition).
Health Care Professional Perspective
Measures with Broadest Domain Coverage:
- Measure 5 covers all Atlas activity domains except the sub-domain Facilitate Transitions as Coordination Needs Change. Although it does not map to Interpersonal Communication, this is not considered a gap because it does map to Communicate. However, it is important to note that this mapping is based on possible response choices from an audit tool rather than a survey-based measure.
- No other high priority (Priority Group 1) measures cover more than 6 of 11 Atlas activity domains.
- Of the Priority Group 2 measures, 11b covers all Atlas activity domains except the sub-domain Facilitate Transitions Across Settings and 17d covers all activity domains and sub-domains except Establish Accountability or Negotiate Responsibility, Facilitate Transitions as Coordination Needs Change, and Create a Proactive Plan of Care.
Measure for Detailed Review:
- Measure 5 (Care Coordination Measurement Tool).
- Measure 11b (Family-Centered Care Self-Assessment Tool—Provider Version).
- Measure 17d (Primary Care Assessment Tool—Provider Edition).
System Representative Perspective
Measures with Broadest Domain Coverage:
- Measure 16a covers all Atlas activity domains except Monitor, Follow-up and Respond to Change.
- Alternatively, 17c covers all Atlas activity domains and sub-domains except Establish Accountability or Negotiate Responsibility, Facilitate Transitions as Coordination Needs Change, and Create a Proactive Plan of Care.
Measure for Detailed Review:
- Measures 16a (Medical Home Index).
- Measure 17c (Primary Care Assessment Tool—Facility Edition).
Step 3: Conduct Detailed Assessment and Select Final Measures
For those measures selected to undergo detailed review in the previous step, we conducted a cited reference search to identify additional sources pertaining to the measure, evaluated any available evidence regarding reliability or validity of the measure, and assessed any additional information on feasibility, past applications, measurement purpose, and focus on care coordination.
Patient/Family Perspective
Table C-5 summarizes the reliability and validity information for the pediatric patient/family perspective measures.
Measure 11a—Family-Centered Care Self-Assessment Tool—Family Version1,2
Validity and Reliability
- The instrument was developed based on 10 components of family-centered care within a framework for partnership between families and professionals. No detailed testing information was described in the sources identified.
Further Information on Feasibility
- No further information available. The survey contains 98 items.
Past or Suggested Uses
- This measure is designed for quality improvement purposes.
Unit of Analysis in Past Applications
- No information available on analysis of past applications.
Focus on Care Coordination
- 90 of 98 total instrument items mapped to a care coordination domain (92%).
- Due to lack of psychometric testing, the measure developers do not recommend combining responses across items to develop mean or aggregate scores for groups of conceptually-related instrument items. Rather, they recommend reporting percent responses to individual items only. Therefore, while the instrument should be administered as a whole, responses on items most relevant to care coordination may be reported separately.
Depth of Domain Focus
- This measure has at least three items for every Atlas activity domain except Communicate (but ≥3 items each for Interpersonal Communication and Information Transfer sub-domains).
Measure 17a—Primary Care Assessment Tool—Child Edition (PCAT-CE)3-6
Validity and Reliability
- Test-retest Reliability. Though absolute agreement was modest for many items (range across all items 37% to 39%), results using a pi coefficient that adjusts for prevalence and variation were adequate (>0.50) for all included items.
- Internal Consistency Reliability. The alpha coefficient was >0.70 for all but 2 scales. The Longitudinality-relationship scale was revised due to a low alpha (alpha=0.40). The First Contact Accessibility scale was very near the threshold typically considered as adequate reliability (alpha=0.68).
- Factor Analysis. A principal components factor analysis yielded 5 separate factors. These corresponded to the instrument's 5 scales: (1) Longitudinality-relationship; (2) First Contact Accessibility; (3) Comprehensiveness of Services Available; (4) Comprehensiveness of Services provided; (5) Coordination. Only factors with adequate loadings on a hypothesized factor were retained in the final instrument.
- Content Validity. A panel of 9 content experts rated appropriateness and representative-ness of domains included in the measure. Agreement among experts was strong (>75%) for most items. Agreement was lowest for items included in the Longitudinality-relationship scale (range 11% to 100%).
Further Information on Feasibility
- Among a sample of 450 survey respondents, average time to complete the telephone survey was 25 minutes.
- The survey may be administered by telephone or through self-assessment. No information is available on typical self-assessment completion times.
Past or Suggested Uses
- This measure is designed for accountability purposes. It has also been used for research purposes.
Unit of Analysis in Past Applications
- Patient.
- Family
- Physician.
- Practice.
Focus on Care Coordination
- 86 of 115 total items map to a care coordination domain (75%).
- The instrument includes a sub-domain for coordination and another sub-domain for coordination related to information systems. The user guide contains information for calculating scores for each of these sub-domains, as well as for an overall primary care score.
- Although the instrument should be administered as a whole, scores for the coordination sub-domains may provide useful information in understanding the measure results most relevant to care coordination. Note that many items in other sub-domains map to care coordination domains, even if the focus of the sub-domain overall is not on coordination.
Depth of Domain Focus
- PCAT-CE has at least three items for each Atlas activity domain, except: Communicate (but ≥3 items each for Interpersonal Communication and Information Transfer sub-domains), and Establish Accountability (n=1).
Recommendations for Final Measure Set:
- The PCAT-CE has been much more thoroughly tested than the Family-Centered Care Self-Assessment Tool—Family Version. Overall, it has been shown to have adequate reliability and validity. In addition, it was designed for use as an accountability measure, though we have not identified published sources of such use. Furthermore, though lengthy and focused on many aspects of primary care beyond care coordination, it does include sub-domains particular to coordination which will aid in interpretation and reporting of results for coordination measurement purposes. Therefore, we recommend the PCAT-CE for inclusion in the measure set.
- If validity or reliability is of less interest, the Family-Centered Care Self-Assessment Tool—Family Version may be considered as an alternative.
Table C-6 shows Recommendations for Care Coordination Accountability Measures for Primary Care Practice.
Health Care Professional Perspective
Measures from this perspective were considered for use as quality improvement tools that may be used to guide improvement initiatives in response to the accountability measure set (patient/family perspective) results. Given the reliance on health care professional self-assessment for all measures from the health care professional perspective, no measures from this perspective were selected for use as accountability tools. Table C-7 summarizes the reliability and validity information for the pediatric health care professional perspective measures.
Measure 5—Care Coordination Measurement Tool7,8
Validity and Reliability
- No validity or reliability testing reported.
- Later use notes that the measure was adapted from earlier instrument and pilot tested, but details are not provided.
Further Information on Feasibility
- The measure was designed for use by clinicians, but there is no report of its impact on clinical work flow or completion rates.
- In one study, use of the instrument required 2-hour training sessions prior to data collection and ongoing technical support throughout data collection period. Generally, practice personnel were able to incorporate accurate use of CCMT into clinical workflow after 1 week of using the instrument. No quantitative assessment of feasibility was provided.
Past or Suggested Uses
- This measure has been used for research.
Unit of Analysis in Past Applications
- Patient encounters.
Focus on Care Coordination
- Domain mapping was based on response choices available from an audit tool. As such, it is not useful to report the percent of items that map to care coordination activity domains. However, the tool as a whole was designed to document care coordination processes.
Depth of Domain Focus
- Domain mapping was based on response choices available from an audit tool. As such, it is not useful to report the percent of items that map to care coordination domains.
Measure 11b—Family-Centered Care Self-Assessment Tool—Provider Version1,2
Validity and Reliability
- The instrument was developed based on 10 components of family-centered care within a framework for partnership between families and professionals. No detailed testing information was described in the sources identified.
Further Information on Feasibility
- No further information available. The survey contains 98 items.
Past or Suggested Uses
- This measure is designed for quality improvement purposes.
Unit of Analysis in Past Applications
- No information available on analysis of past applications.
Focus on Care Coordination
- 88 of 105 total instrument items mapped to a care coordination domain (84%).
- Due to lack of psychometric testing, the measure developers do not recommend combining responses across items to develop mean or aggregate scores for groups of conceptually-related instrument items. Rather, they recommend reporting percent responses to individual items only. Therefore, while the instrument should be administered as a whole, responses to items most relevant to care coordination may be reported separately.
Depth of Domain Focus
- This measure has at least three items for every Atlas activity domain except the sub-domain: Facilitate Transitions Across Settings (n=0).
Measure 17d—Primary Care Assessment Tool—Provider Edition (PCAT-PE)3-6
Validity and Reliability
- Indirect. No validity or reliability testing of this version is reported, but it is very similar to the PCAT-CE, which has undergone validity and reliability testing. That testing found adequate test-retest and internal consistency reliability for all items and factor analyses identified five separate factors which correspond to the PCAT-CE instrument sub-scales. Agreement among a panel of content experts was generally good about the appropriateness and representative-ness of domains included in the PCAT-CE, on which the PCAT-PE is based.
Further Information on Feasibility
- The instrument includes a sub-domain for coordination and another sub-domain for coordination related to information systems. The user guide contains information for calculating scores for each of these sub-domains, as well as for an overall primary care score.
- Although the instrument should be administered as a whole, scores for the coordination sub-domains may provide useful information in understanding the measure results most relevant to care coordination. Note that many items in other sub-domains map to care coordination domains, even if the focus of the sub-domain overall is not on coordination.
Focus on Care Coordination
- 114 of 153 total instrument items map to a care coordination domain (75%).
Past or Suggested Uses
- This measure is designed for accountability purposes. It has also been used for research.
Unit of Analysis in Past Applications
- Physician.
- Practice.
Depth of Domain Focus
- PCAT-PE has at least three items for all but five Atlas activity domains or sub-domains: Establish Accountability (n=0); Communicate (but ≥3 items each for Interpersonal Communication and Information Transfer sub-domains); Facilitate Transitions Across Settings (n=2); Facilitate Transitions as Coordination Needs Change (n=0); and Create a Proactive Plan of Care (n=0).
Recommendations for Final Measure Set:
- No validity or reliability testing has been performed for any of the measures under consideration. Indirect evidence of validity and reliability is available for the PCAT-PE, which is based on the validated PCAT-CE instrument, but given that the two instruments are designed for different users (parents of pediatric patients vs. health care providers), we recommend caution in extrapolating validity and reliability information.
- If comprehensive measurement is of chief interest, consider the Family-Centered Care Self-Assessment Tool—Provider Version, which provides comprehensive assessment of care coordination, with three or more items mapped to all but one Atlas activity sub-domain (Facilitate Transitions Across Settings).
- If consistency with the pediatric patient/family perspective measure is desirable, consider the Primary Care Assessment Tool—Provider Edition (PCAT-PE), which is closely related to the PCAT-CE, which will improve comparability of responses across the two instruments. Results of the provider version survey may be particularly useful in guiding improvement initiatives that aim to improve performance on the related child version of the survey that is recommended for accountability purposes.
- The Care Coordination Measurement Tool (CCMT) offers an alternative to survey-based measurement. It uses an audit technique to collect detailed information about coordination activities carried out by health care professionals. Although the tool still relies on self-reporting by health care professionals, if methods of auditing or validating self-reported information were developed, the CCMT might be useful as an accountability measure. However, note that validity and reliability remain unknown at this time. The tool may be useful for quality improvement purposes by providing detailed information about the types of coordination-related activities performed in a clinic.
System Representative Perspective
Table C-8 summarizes the reliability and validity information for the system representative perspective measures.
Measure 16a—Medical Home Index (MHI)9,10
Validity and Reliability
- Inter-rater reliability assessment suggested acceptable reliability between assessment by outside observers (study site visitors) and practice staff (self-assessment).
- Internal consistency reliability was strong for the total score (Cronbach's alpha >0.70).
- Content validity was assessed by experts in medical home concept. Instrument revised following review.
- Construct validity—univariate or bivariate. Practice MHI score was not correlated with family satisfaction with care. In bivariate analyses, higher overall practice MHI score (indicating better adherence to the medical home model) was significantly associated with lower hospitalization rates. No significant correlations were observed between overall score and ED visits or the ratio of primary to specialty care visits.
- Construct validity—multivariate. In multivariate analyses controlling for patients' chronic conditions, overall MHI practice score was significantly associated with hospitalization rates. Rate of emergency department (ED) visits was significantly associated with the care coordination and chronic condition management sub-domain scores, but not the overall practice MHI score. The care coordination domain was also associated with lower hospitalization rates. No significant associations were found between overall MHI score or any sub-domain scores and the ratio of primary to specialty care visits.
Further Information on Feasibility
- Typical completion time for the Medical Home Index is 30-45 minutes, including time needed for both practice representatives to reach consensus.
Past or Suggested Uses
- This measure has been used for quality improvement and research purposes.
- Use for accountability or recognition purposes was also suggested by the measure developer, but we are not aware of any instances in which the measure has been used in this way. The reliance on self-assessment may limit the usefulness for accountability purposes.
Unit of Analysis in Past Applications
- Primary care practice.
Focus on Care Coordination
- All 25 instrument items map to a care coordination domain.
- The instrument includes a sub-domain for care coordination which may be scored separately from the total instrument score. However, given the relevance of all items to care coordination, the total score is also highly relevant for care coordination measurement.
Depth of Domain Focus
- The MHI has at least 3 items for only 5 activity domains. Those domains or sub-domains with <3 items are: Establish Accountability (n=2); Interpersonal Communication (n=0, but 4 items address the Communicate domain); Information Transfer (n=2); Facilitate Transitions Across Settings (n=2); Facilitate Transitions as Coordination Needs Change (n=1); Monitor, Follow-up and Respond to Change (n=0); Support Self-Management Goals (n=1). This lower domain density largely reflects the brevity of the instrument, as all 25 survey items mapped to one of the framework domains.
Measure 17c—Primary Care Assessment Tool—Facility Edition (PCAT-FE)3,11
Validity and Reliability
- Indirect. No validity or reliability testing of this version is reported, but it is very similar to the PCAT-AE and PCAT-CE, which have undergone validity and reliability testing.
Focus on Care Coordination
- 114 of 153 total instrument items map to a care coordination domain (75%).
Past or Suggested Uses
- This measure is designed for accountability purposes.
Unit of Analysis in Past Applications
- No information available on analysis of past applications of the instrument.
Further Information on Feasibility
- The instrument includes a sub-domain for coordination and another sub-domain for coordination related to information systems. The user guide contains information for calculating scores for each of these sub-domains, as well as for an overall primary care score.
- Although the instrument should be administered as a whole, scores for the coordination sub-domains may provide useful information in understanding the measure results most relevant to care coordination. Note that many items in other sub-domains map to care coordination domains, even if the focus of the sub-domain overall is not on coordination.
Depth of Domain Focus
- PCAT-FE has at least three items for all but five Atlas activity domains or sub-domains: Establish Accountability (n=0); Communicate (but ≥3 items each for Interpersonal Communication and Information Transfer sub-domains); Facilitate Transitions Across Settings (n=1); Facilitate Transitions as Coordination Needs Change (n=0); and Create a Proactive Plan of Care (n=0).
Recommendations for Final Measure Set:
- If validity or reliability is of chief interest, consider the Medical Home Index (MHI). This measure has strong validity and reliability and is designed and has been used for quality improvement purposes.
- If feasibility is of chief interest, consider the Medical Home Index (MHI), which is limited to 25 items and has reported a typical completion time of 30 to 45 minutes. A 10-item short version of this survey is also available (Atlas measure # 16b) which offers reduced measurement burden, although it offers less comprehensive domain coverage.
- If in-depth measurement is of chief interest, consider the Primary Care Assessment Tool—Facility Expanded Edition (PCAT-FE). Although this measure does not cover every Atlas activity domain, it does include at least three items for most domains, offering a more detailed assessment of those coordination mechanisms than the briefer MHI. The additional detail provided by in-depth measurement may be particularly useful for guiding improvement initiatives.
- If consistency with the pediatric patient/family perspective measure is desirable, consider the Primary Care Assessment Tool—Facility Expanded Edition (PCAT-FE). This survey is closely related to the PCAT-CE, which will improve comparability of responses across the two measures. Results of the provider version survey may be particularly useful in guiding improvement initiatives that aim to improve performance on the related child version of the survey that is recommended for accountability purposes.
Adult Measure Set Creation
Step 1: Narrow the Field of Measures
Figure C-2 summarizes the process of narrowing the field of measures for the adult measure set. Details of each step are provided below.
3.2.1.1 Inclusions
Table C-9 lists the set of measures that meet the following three inclusion criteria:
- Measures that are applicable to primary care facilities or that are not setting specific.
- Measures that are applicable to adults or that are not age specific or for which patient age is not applicable.
- Measures that are applicable to patients with chronic conditions or that are not condition specific or where patient condition is not applicable. This inclusion criterion was added to further limit the field of measures given the large number the remained when using only the age and setting criteria. We chose to focus on patients with chronic conditions because they have some of the greatest needs for care coordination and such measures would be widely applicable given the prevalence of chronic disease.
Exclusions
In total, 22 measures were excluded for the following reasons:
- Measures that are disease-specific were excluded because they would not be widely applicable: 21, 26, 52, and 54.
- Measures that are not applicable for evaluating care coordination as performed by primary care practices or providers: 9a, 9b, 32, 38a, 38b, 38c, 38d, 38e, 38f, 39, 43, 57, 58, 59, 60 and 62.
- Measure that is applicable only to older adults and therefore limits overall use in a general population: 2.
- Measure that is applicable only to women and therefore limits overall use in a general population: 33.
Feasibility and Degree of Focus on Care Coordination
Next, we assessed feasibility and degree of focus on care coordination for these 28 measures. Table C-10 reports the priority rankings based on this assessment.
Table C-11 lists the set of 24 measures that remain for consideration during Step 2 (Priority Groups 1 and 2) after applying the exclusion criteria and priority groupings. Measures in Group 1 will be considered first for inclusion in the final measure set, followed by measures in Group 2. Only if no adequate measures are identified from groups 1 and 2, measures in Group 3 or Group 4 would be considered.
Step 2: Identify the Most Comprehensive Measures
Table C-12 maps domain and perspective for the measures in Priority Groups 1 and 2 that will be considered in step 2.
Patient/Family Perspective
Measures with Broadest Domain Coverage:
- Measure 6 (Client Perceptions of Coordination Questionnaire) covers all Atlas activity domains except Facilitate Transitions (both sub-domains), and Links to Community Resources.
- Measure 37 covers all domains and sub-domains except Facilitate Transitions as Coordination Needs Change, Links to Community Resources, and Align Resources with Patient and Population Needs.
- Among Priority Group 2, Measure 17b (Primary Care Assessment Tool—Adult Version) offers the broadest domain coverage, mapping to all Atlas activity domains and sub-domains except Facilitate Transitions as Coordination Needs Change, Create a Proactive Plan of Care, and Links to Community Resources.
- No measures map to the Facilitate Transitions as Coordination Needs Change sub-domain. This is a true measurement gap.
Measures for Detailed Review:
- Measure 6 (Client Perception of Coordination Questionnaire).
- Measure 37 (Patient Perceptions of Care).
- Measure 17b (Primary Care Assessment Tool—Adult Version).
Health Care Professional Perspective
Measures with Broadest Domain Coverage:
- Only Measure 17d (Primary Care Assessment Tool—Provider Edition) has broad domain coverage. No other measures map to more than six Atlas activity domains. None of the measures in priority groups 3 or 4 are from the health care professional perspective.
Measures for Detailed Review:
- Measure 17d (Primary Care Assessment Tool—Provider Edition). The strengths and weaknesses of this measure will be assessed, as it is the only measure under consideration.
System Representative Perspective
Measures with Broadest Domain Coverage:
- Measure 1 (Assessment of Chronic Illness Care) covers all Atlas activity domains except Facilitate Transitions (both sub-domains).
- Measure 16a (Medical Home Index—Long Version) covers all Atlas activity domains except Monitor, Follow-up and Respond to Change.
Measures for Detailed Review:
- Measure 1 (Assessment of Chronic Illness Care).
- Measure 16a (Medical Home Index—Long Version).
Step 3: Conduct Detailed Measure Assessment and Select Final Measures
For those measures selected to undergo detailed review in the previous step, we conducted a cited reference search to identify additional sources pertaining to the measure, evaluated any available evidence regarding reliability or validity of the measure, and assessed any additional information on feasibility, past applications, measurement purpose, and focus on care coordination.
Patient/Family Perspective
Table C-13 summarizes the reliability and validity evidence for adult patient/family perspective measures.
Measure 6 Client Perception of Coordination Questionnaire (CPCQ)12
Validity and Reliability
- Internal Consistency Reliability. Overall Cronbach's alpha was 0.92, suggesting high internal consistency. Four of six sub-scales had acceptable Cronbach's alpha (>0.70), but the authors note that poor internal consistency of the two remaining scales (client comprehension and client capacity) raise concerns about the reliability of the overall scale score. All but two individual items were statistically significantly correlated with the global CPCQ score. The uncorrelated items focused on decision-making preferences.
- Factor Analysis. Principal components analysis suggested six sub-scales. These generally fit with the hypothesized five survey components, with two sub-scales distinguishable for the first component (global quality and quality of specific aspects of care).
- Construct Validity. Mean scores for all items were very similar between two test groups (general population and participants in a care coordination trial), as expected.
- Construct Validity. As expected, respondents with chronic pain reported less coordinated care (65.5% well coordinated among chronic pain group vs. 83.9% well coordinated among patients without chronic pain). At the level of individual survey items, patients with chronic pain also reported less coordinated care for all but three survey items.
Further Information on Feasibility
- No information available on typical completion times.
- Completion rates for sections in the measure instrument suggest good feasibility, but the authors note that some items were not applicable to a substantial percent of respondents (i.e., no recent test, no change in needs). Missing responses were most prevalent on the portion of the instrument that focuses on a single provider (range 8.1% to 10.3% missing).
Past or Suggested Uses
- This measure has been used for research purposes.
Unit of Analysis in Past Applications
- No information available on analysis of past applications.
Focus on Care Coordination
- The instrument is designed to focus on care coordination.
- 23 of 31 total instrument items map to a care coordination domain (74%).
- No total or sub-domain scores are calculated for the measure; results are reported for individual items in the instrument.
Depth of Domain Focus
- The CPCQ has at least three items for only two Atlas activity domains: Communicate and Support Self-management Goals. It has no items mapped to the following Atlas activity domains: Facilitate Transitions (both Across Settings and as Coordination Needs Change sub-domains), and Links to Community Resources. This lower domain density partially reflects the brevity of the instrument, as 23 of 31 survey items mapped to one of the Atlas framework domains.
Measure 37—Patient Perceptions of Care (PPOC)13,14
Validity and Reliability
- Indirect. This measure is based on components of the 1998 VA National Outpatient Customer Satisfaction Survey, conducted by the VA National Performance Data Resource Center. Similar items have been used in the Veterans Satisfaction Survey. However, no information is available on the validity or reliability of the original survey, nor the validity of selecting items from that survey to create this measure.
Further Information on Feasibility
- The survey contains 40 items. No further information available.
Past or Suggested Uses
- This measure is has been used for research purposes. The VA survey on which it is based is used for quality improvement purposes.
Unit of Analysis in Past Applications
- Practice.
- Hospital.
Focus on Care Coordination
- 26 of 40 total instrument items mapped to a care coordination domain (65%).
- The instrument includes two sub-scales related to care coordination: overall coordination of care and coordination of care at a visit. Scores for these sub-scales may be reported separately, although psychometric testing of sub-scales is not reported.
Depth of Domain Focus
- The PPOC has at least three items for six Atlas activity domains or sub-domains. Those with <3 items are: Facilitate Transitions Across Settings (n=1); Facilitate Transitions as Coordination Needs Change (n=0); Create a Proactive Plan of Care (n=1); Support Self-Management Goals (n=2); Link to Community Resources (n=0) and Align Resources with Patient and Population Needs (n=0).
Measure 17b—Primary Care Assessment Tool—Adult Edition (PCAT-AE)11,15
Validity and Reliability
- Internal Consistency Reliability. Only those items with item-total correlations >0.30 were retained in the final instrument. The range of correlations in the final instrument was 0.34 to 0.91. Cronbach's alpha for sub-scales >0.70 for all but one sub-scale (First Contact- Utilization, alpha=0.64).
- Factor Analysis. Seven factors were identified, in accordance with the hypothesized conceptual model. One factor (First Contact-Accessibility) retained only four of 12 items, suggesting this concept may not be well measured by the instrument. The authors suggest that users review the appropriateness of the items from this scale before determining whether they are applicable to their particular setting of interest.
- Indirect. The adult version of the PCAT is based on the child edition, which has been previously validated.
Further Information on Feasibility
- The measure was reported to take approximately 40 minutes to complete in a sample of 890 U.S. patients.
- The measure developers report that the survey can be self-administered (as was the case in published use of the instrument) or completed in person or by phone with the help of an interviewer. They note that a high school reading level is required for self-administration.
Past or Suggested Uses
- This measure is designed and has been used for accountability purposes. The authors suggest use for research purposes, as well.
Unit of Analysis in Past Applications
- Payer.
Focus on Care Coordination
- 80 of 131 total instrument items map to a care coordination domain (61%).
- The instrument includes a sub-domain for coordination and another sub-domain for coordination related to information systems. The user guide contains information for calculating scores for each of these sub-domains, as well as for an overall primary care score.
- Although the instrument should be administered as a whole, scores for the coordination sub-domains may provide useful information in understanding the measure results most relevant to care coordination. Note that many items in other sub-domains map to care coordination domains, even if the focus of the sub-domain overall is not on coordination.
Depth of Domain Focus
- The PCAT-AE has at least three items for each Atlas activity domain or sub-domain, except: Establish Accountability (n=1); Communicate (but ≥3 items each for Interpersonal Communication and Information Transfer sub-domains); Facilitate Transitions as Coordination Needs Change (n=0); and Create a Proactive Plan of Care (n=0).
Recommendations for Final Measure Set:
- The Client Perceptions of Coordination Questionnaire (Measure 6) has the strongest validity and reliability, an important consideration for measurement for accountability purposes. It also offers comprehensive domain coverage and a strong focus on care coordination. Although no information is available about typical completion times, its relatively short length (31 items) suggests good feasibility. Therefore, we recommend the CPCQ for inclusion in the measure set.
- The Primary Care Assessment Tool—Adult Edition (Measure 17b) may be a useful alternative to Measure 6 for those who are interested in consistency with the Pediatric measure set, or if understanding transitions of care is of particular interest. (Measure 17b maps to the Facilitate Transitions Across Settings sub-domain, while Measure 6 does not).
Health Care Professional Perspective
Table C-14 summarizes the reliability and validity evidence for adult health care professional perspective measures.
Measure 17d—Primary Care Assessment Tool—Provider Edition (PCAT-PE)3-6,16
Validity and Reliability
- Indirect. No validity or reliability testing of this version is reported, but it is very similar to the PCAT-CE, which has undergone validity and reliability testing. That testing found adequate test-retest and internal consistency reliability for all items and factor analyses identified five separate factors which correspond to the PCAT-CE instrument sub-scales. Agreement among a panel of content experts was generally good about the appropriateness and representative-ness of domains included in the PCAT-CE, on which the PCAT-PE is based.
Further Information on Feasibility
- The instrument includes a sub-domain for coordination and another sub-domain for coordination related to information systems. The user guide contains information for calculating scores for each of these sub-domains, as well as for an overall primary care score.
- Although the instrument should be administered as a whole, scores for the coordination sub-domains may provide useful information in understanding the measure results most relevant to care coordination. Note that many items in other sub-domains map to care coordination domains, even if the focus of the sub-domain overall is not on coordination.
Past or Suggested Uses
- This measure is designed for accountability purposes. It has also been used for research.
Unit of Analysis in Past Applications
- Physician.
- Practice.
Focus on Care Coordination
- 114 of 153 total instrument items map to a care coordination domain (75%).
Depth of Domain Focus
- The PCAT-PE has at least three items for all but five Atlas activity domains or sub-domains: Establish Accountability (n=0); Communicate (but ≥3 items each for Interpersonal Communication and Information Transfer sub-domains); Facilitate Transitions Across Settings (n=2); Facilitate Transitions as Coordination Needs Change (n=0); and Create a Proactive Plan of Care (n=0).
Recommendations for Final Measure Set:
- Only the adult primary care measure (PCAT-PE) maps to more than six Atlas activity domains from the health care professional perspective. This measure may be considered for quality improvement purposes, but it is important to note that only indirect evidence of validity and reliability is available for this measure, which is based on the validated PCAT-CE instrument. Given that the child and provider versions of the PCAT are designed for different users (parents of pediatric patients vs. health care providers for adults), we recommend caution in extrapolating validity and reliability information.
System Representative Perspective
Table C-15 summarizes the reliability and validity evidence for adult system representative perspective measures.
Measure 1—Assessment of Chronic Illness Care (ACIC)17-20
Validity and Reliability
- Construct Validity—Bivariate. The instrument was tested in 108 organizational teams implementing 13-month long quality improvement collaboratives in health care systems across the U.S. Paired t-tests were used to evaluate the sensitivity of the ACIC to detect system improvements. Testing revealed that all six subscale scores were responsive to system improvements made by care teams, as assessed by faculty raters.
- Convergent Validity. In the same study noted above, a significant positive relationship between differences in self-reported ACIC scores and a RAND measure of the presence of chronic care model components in care program implementation was found.
- Construct Validity—Multivariate. In a study of patients with type II diabetes from 20 primary care clinics in Texas, one study found that attributable risk but not absolute risk of fatal or non-fatal coronary heart disease was inversely associated with ACIC score.18 Attributable risk is the excess risk estimated to be due to poor control of modifiable risk factors. A 1-point increase in ACIC score, indicating better adherence to the Chronic Care Model, was associated with a 16% relative decrease in attributable risk (95%confidence interval [CI] 5% to 26%).18
- Construct Validity—Multivariate. In a related study, the authors found that for every 1-unit increase in ACIC score patients' most recent hemoglobin A1c value decreased by 0.07 among a sample of 538 type II diabetes patients.19 They reported that the relationship between ACIC score and HbA1c values was strongest among patients who did not adhere to exercise recommendations; adherence to diet recommendations did not change the relationship between ACIC and HbA1c when controlling for exercise adherence.19
- Construct Validity—Bivariate and Multivariate. In a large multi-specialty medical group, seventeen primary care clinics found that ACIC scores improved after implementation of the Chronic Care Model in their clinics, as did scores on composite outcomes measures assessing cholesterol (LDL) and hemoglobin A1c values among diabetic patients and assessing LDL levels and cardiac events in patients with coronary heart disease. The authors noted that although the overall ACIC score increased over the implementation period, the score for the support for self-management component did not, suggesting that changes in outcomes were not associated with improvements in self-management support. A significant change was observed post-intervention in the community linkages sub-scale of the ACIC. Correlations between sub-scales of the ACIC and the outcomes measures did not show any significant relationship between either the self-management support or community linkage sub-scales and any outcomes, however. Results were similar using multivariate regression.20
Focus on Care Coordination
- 25 of 34 total instrument items map to a care coordination domain (74%).
Further Information on Feasibility
- The measure contains 34 items.
- No information is available on typical completion times.
Past or Suggested Uses
- This measure has been used for quality improvement and research purposes.
Unit of Analysis in Past Applications
- Practice.
- Geographic region (non-U.S.).
Depth of Domain Focus
- The ACIC has at least three items for six Atlas activity domains or sub-domains. Those with <3 items are: Establish Accountability (n=2); Communicate (n=2); Interpersonal Communication (n=0); Facilitate Transitions Across Settings (n=0); Facilitate Transitions as Coordination Needs Change (n=0); and Create a Proactive Plan of Care (n=1).
Measure 16a—Medical Home Index (MHI)9,10
Validity and Reliability
- Inter-rater reliability assessment suggested acceptable reliability between assessment by outside observers (study site visitors) and practice staff (self-assessment).
- Internal consistency reliability was strong for the total score (Cronbach's alpha >0.70).
- Content validity was assessed by experts in the medical home concept. Instrument revised following review.
- Construct Validity—univariate or bivariate. Practice MHI score was not correlated with family satisfaction with care. In bivariate analyses, higher overall practice MHI score (indicating better adherence to the medical home model) was significantly associated with lower hospitalization rates. No significant correlations were observed between overall score and ED visits or the ratio of primary to specialty care visits.
- Construct Validity—multivariate. In multivariate analyses controlling for patients' chronic conditions, overall MHI practice score was significantly associated with hospitalization rates. Rate of emergency department (ED) visits was significantly associated with the care coordination and chronic condition management sub-domain scores, but not the overall practice MHI score. The care coordination domain was also associated with lower hospitalization rates. No significant associations were found between overall MHI score or any sub-domain scores and the ratio of primary to specialty care visits.
Further Information on Feasibility
- Typical completion time for the Medical Home Index is 30-45 minutes, including time needed for both practice representatives to reach consensus.
Past or Suggested Uses
- This measure has been used for quality improvement and research purposes.
- Use for accountability or recognition purposes was also suggested by the measure developer, but we are not aware of any instances in which the measure has been used in this way. The reliance on self-assessment may limit the usefulness for accountability purposes.
Unit of Analysis in Past Applications
- Primary care practice.
Focus on Care Coordination
- All 25 instrument items map to an Atlas care coordination domain (100%).
- The instrument includes a sub-domain for care coordination which may be scored separately from the total instrument score. However, given the relevance of all items to care coordination, the total score is also highly relevant for care coordination measurement.
Depth of Domain Focus
- The MHI has at least three items for only five Atlas activity domains or sub-domains. Those with <3 items are: Establish Accountability (n=2); Interpersonal Communication (n=0, but four items for Communicate domain); Information Transfer (n=2); Facilitate Transitions Across Settings (n=2); Facilitate Transitions as Coordination Needs Change (n=1); Monitor, Follow-up and Respond to Change (n=0); and Support Self-Management Goals (n=1). This lower domain density largely reflects the brevity of the instrument, as all 25 survey items mapped to one of the framework domains.
Recommendations for Final Measure Set:
- If focus on care coordination is of chief interest: consider the Medical Home Index, for which all items map to a care coordination domain. Therefore, both the total score and care coordination sub-scale of the MHI are of use in care coordination assessment.
- If validity is of chief interest: consider the Assessment of Chronic Illness Care (ACIC), which has very strong evidence supporting validity, although reliability is unknown. Note that the MHI does also have evidence of validity, although not as robust as the ACIC.
- If feasibility is of chief interest: also consider the MHI, which is reported to take 30 to 45 minutes to complete and is shorter in length than the ACIC. Also note that an abbreviated version (10 items) of the MHI is available (Measure 16b), although it offers narrower domain coverage.
Table C-16 shows companion measure options.
Appendix C References
- Family Centered Care Self-Assessment Tool—Family Version. October 2008. (Accessed 17 September 2010, at http://www.familyvoices.org).
- National Center for Family-Centered Care. Family-Centetred Care for Children with Special Health Care Needs. Bethesda, MD; 1989.
- Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA. Measuring consumer experiences with primary care. Pediatrics 2000;105:998-1003.
- Haggerty JL, Pineault R, Beaulieu MD, et al. Room for improvement—Patients' experiences of primary care in Quebec before major reforms. Canadian Family Physician 2007;53:1057-U42.
- Haggerty JL, Pineault R, Beaulieu MD, et al. Practice features associated with patient-reported accessibility, continuity, and coordination of primary health care. Ann Fam Med 2008;6:116-23.
- Stevens GD, Shi LY. Racial and ethnic disparities in the primary care experiences of children: A review of the literature. Medical Care Research and Review 2003;60:3-30.
- Antonelli RC, Antonelli DM. Providing a medical home: the cost of care coordination services in a community-based, general pediatric practice. Pediatrics 2004;113:1522-8.
- Antonelli RC, Stille CJ, Antonelli DM. Care coordination for children and youth with special health care needs: a descriptive, multisite study of activities, personnel costs, and outcomes. Pediatrics 2008;122:e209-16.
- Cooley WC, McAllister JW, Sherrieb K, Clark RE. The Medical Home Index: Development and validation of a new practice-level measure of implementation of the Medical Home model. Ambulatory Pediatrics 2003;3:173-80.
- Cooley WC, McAllister JW, Sherrieb K, Kuhlthau K. Improved Outcomes Associated With Medical Home Implementation in Pediatric Primary Care. Pediatrics 2009;124:358-64.
- Shi L. Validating the Adult Primary Care Assessment Tool. J Fam Pract 2001;50:161.
- McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of health care. Int J Qual Health Care 2003;15:309-18.
- Borowsky SJ, Nelson DB, Fortney JC, Hedeen AN, Bradley JL, Chapko MK. VA community-based outpatient clinics: performance measures based on patient perceptions of care. Med Care 2002;40:578-86.
- Flach SD, McCoy KD, Vaughn TE, Ward MM, Bootsmiller BJ, Doebbeling BN. Does patient-centered care improve provision of preventive services? J Gen Intern Med 2004;19:1019-26.
- Shi L, Starfield B, Xu J, Politzer R, Regan J. Primary care quality: community health center and health maintenance organization. South Med J 2003;96:787-95.
- Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider perceptions of the quality of primary care: implications for managed care. J Fam Pract 1998;46:216-26.
- Bonomi AE, Wagner EH, Glasgow RE, VonKorff M. Assessment of chronic illness care (ACIC): a practical tool to measure quality improvement. Health Serv Res 2002;37:791-820.
- Parchman ML, Pugh JA, Wang CP, Romero RL. Glucose control, self-care behaviors, and the presence of the chronic care model in primary care clinics. Diabetes Care 2007;30:2849-54.
- Parchman ML, Zeber JE, Romero RR, Pugh JA. Risk of coronary artery disease in type 2 diabetes and the delivery of care consistent with the chronic care model in primary care settings: a STARNet study. Med Care 2007;45:1129-34.
- Solberg LI, Crain AL, Sperl-Hillen JM, Hroscikoski MC, Engebretson KI, O'Connor PJ. Care quality and implementation of the chronic care model: a quantitative study. Ann Fam Med 2006;4:310-6.
AHRQ Publication No. 12-0019-EF