CC BY-NC-ND
0Measuring Community Programs and Jb Policies in the Healthy Communities
Stephen B. Fawcett, PhD,1 Vicki L. Collie-Akers, PhD,1 Jerry A. Schultz, PhD,1 Melinda Kelley, PhD2
Childhood obesity is a challenging public health issue facing communities throughout the U.S. Local efforts are believed to be essential to assuring environments that support physical activity and healthy food/beverage consumption among children and their families. However, little is known about how broadly and intensively communities are implementing combinations of programs and policies that address childhood nutrition, physical activity, and weight control. The Healthy Communities Study is a nationwide scientific study in diverse communities to identify characteristics of communities and programs that may be associated with childhood obesity. Data collection occurred in 2013-2015; data analysis will be completed in 2016. As part of the Healthy Communities Study, researchers designed a measurement system to assess the number and scope of community programs and policies and to examine possible associations between calculated "intensity" scores for these programs and policies and behavioral and outcome measures related to healthy weight among children. This report describes the protocol used to capture and code instances of community programs and policies, to characterize attributes of community programs and policies related to study hypotheses, and to calculate the intensity of combinations of community programs and policies (i.e., using the attributes of change strategy, duration, and reach).
(Am J Prev Med 2015;49(4):636-641) © 2015 American Journal of Preventive Medicine. All rights reserved. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Introduction
Childhood obesity is a critical public health challenge facing communities.1-3 To accelerate progress in obesity prevention, communities throughout the U.S. have engaged—to varying degrees —in creating environments that support healthy nutrition, physical activity, and healthy weight.4 These activities include efforts to:
1. increase healthy food/beverage choices (e.g., through programs to improve access to healthy foods in stores and school lunches); and
2. increase physical activity (e.g., through changes in bike lanes, walking trails, and school policies to assure more time for youth to be physically active).4,5
From the 1Work Group for Community Health and Development, University of Kansas, Lawrence, Kansas; and 2National Institute on Aging, Bethesda, Maryland
Address correspondence to: Stephen B. Fawcett, PhD, Work Group for Community Health and Development, University of Kansas, 1000 Sunny-side Dr., Lawrence Kansas 66045. E-mail: sfawcett@ku.edu. 0749-3797/$36.00
http://dx.doi.org/10.1016/j.amepre.2015.06.027
Yet little is known about the scope and intensity of efforts by communities to implement multiple programs and policies relevant to childhood obesity. To accelerate health promotion efforts, an understanding of how community efforts occur under real-world conditions is needed, as well as learning whether combinations of community programs and policies (CPPs) are associated with children's diet and physical activity behaviors and healthy weight outcomes.6
Acquiring this information requires a measurement system for documenting the implementation of multiple CPPs, in different sectors, over time.6 Because not all CPPs are equally influential, even under optimal implementation conditions, it also requires accounting for differential levels of influence and estimating the intervention strength of CPPs, singly and in combination.7
This report outlines the community measurement system used in the Healthy Communities Study (HCS). The HCS was initiated by the NIH, and this protocol is part of a research design developed by Battelle Memorial Institute, NIH, and university partners. The goal of the HCS is to examine associations between characteristics of CPPs and diet, physical activity, and BMI among children. The overall research design and other components
636 Am J Prev Med 2015;49(4):636-641 © 2015 American Journal of Preventive Medicine. All rights reserved. This is an open
access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
are described more fully in other papers in this supplement. This protocol was used to capture and characterize CPPs occurring in 130 diverse communities throughout the U.S. This report describes the community measurement methods used in the HCS.
Methods
Context for Protocol Development
A core research question in the HCS is whether an association exists between CPPs and outcomes related to childhood obesity. The study required a systematic method7 to capture, code, characterize, and calculate the intensity of community efforts to prevent childhood obesity. The Work Group for Community Health and Development at the University of Kansas (KU Work Group) led the development of the community measurement protocol based on its prior experience in community measure-ment.7-10
A Community Measurement Subcommittee of the HCS was formed to oversee development of the protocol. Members included representatives from NIH, CDC, Battelle Memorial Institute, and the university partners on the study. The aim of this community measurement approach was to identify CPPs and characterize them sufficiently to explore the main hypotheses of the study; those focused on determining if specific combinations of CPPs are associated with BMI, diet, and physical activity among children.
Overview and Development of Protocol
Drawing on community measurement methods developed by the KU Work Group,7'8 the protocol was designed to capture and characterize the multiple and varied community programs and policies emerging over time in diverse communities.
For this study, CPPs were defined as the presence of a program, policy, or other activity in the community related to improved nutrition, increased physical activity, or weight control (preventing childhood obesity). Data collection occurred in 2013-2015; data analysis will be completed in 2016. Retrospective collection of CPP data allowed for CPPs to be documented for a longer study period (2003-2015). Types of CPPs included:
1. programs (e.g., nutrition program in a youth organization);
2. policies (e.g., new physical activity requirement in a school district); and
3. environmental changes (e.g., expanded bike path).
To be coded as an instance of a CPP, the activity had to be related to the goals of the study, occur during the study period, directly or indirectly address children aged 4-15 years, and be implemented in or affect the high school catchment area (that defined the community).
Two primary methods were used to capture instances of CPPs. Initially, community liaisons (CLs) conducted interviews with key informants (KIs; mean, 12 per community) drawn from multiple sectors. KIs were individuals identified as having knowledge of community efforts in their organizations and other settings and sectors (e.g., schools, government). Priority roles of KIs from targeted sectors included school principals, leads of health
coalitions, health department and hospital representatives, parks and recreation staff, and staff of other governmental and community-based organizations. Then, document abstraction was used to capture instances of CPPs from archived reports, web searches, and other written sources identified through searches and information from KIs.
Not all community programs and policies are likely to have an equivalent effect on health behaviors and related obesity outcomes, even in an ideal implementation scenario. Moreover, the implementation of each program and policy is likely to vary across communities, as a result of differences in local infrastructure, funding, staff availability, and other factors.
"Intensity" scoring provides a way to take the potential influence of these activities into account and to estimate the "dose" of intervention actually delivered in each community. This scoring approach also allows for an examination of the association between different amounts and kinds of CPPs with key outcome variables (i.e., dietary intake, physical activity, and BMI) at different times in the study period. In a complex system, an intensity score for a given CPP should take into account multiple attributes, such as reach or change strategy, as well as real-world circumstances that influence associations between programs and policies and changes in childhood obesity.
Accordingly, the team defined the construct of intensity7 (of CPPs) to reflect three characteristics:
1. behavioral intervention strategy (e.g., modifying access was weighted higher than providing information);
2. duration (e.g., an ongoing policy was weighted higher than a 1-day event); and
3. reach (e.g., a program reaching 20% of the population was rated higher than one reaching 2%).7,11-13
To capture the dynamic unfolding of comprehensive community efforts over time, a composite intensity score was computed for each community for each year of the study.
The community measurement protocol was approved by the HCS Executive Committee, IRBs of Battelle Memorial Institute and the University of Kansas, the Observational Study Monitoring Board, and the Office of Management and Budget.
Steps in Implementing the Community Measurement Protocol
A four-step protocol was implemented.
1. Capture community programs and policies using KI interviews and document abstraction. CLs were trained by the KU team to use the community measurement protocol to conduct structured interviews with KIs; the training included workshops, feedback on performance in scoring, and practice to mastery. CLs reviewed community information sources to locate potential KIs within the specific priority roles (e.g., school principal, staff in parks and recreation). Using phone calls and written communications, they communicated the types of information sought and confirmed whether the person was appropriate before requesting an interview time via phone or in person. They also requested any key documents (e.g., final reports, reports to funders) that could help identify relevant CPPs.
Using a structured KI interview, CLs asked about instances and attributes of CPPs that occurred in or affected the defined community over a 10-year period. The interview protocol sought to obtain:
1. a list and description of specific CPPs that the KI's organization implemented during the study period;
2. the timing of the activity (e.g., year of onset, offset); and
3. a detailed characterization of each CPP (e.g., strategy used, behavioral objective addressed).
The focus was on obtaining information on the complete set of CPPs implemented in each community and their full characterization, including information to examine the primary study hypotheses. Initial open-ended questions probed for CPPs (e.g., What specific programs (policies) were implemented by your organization to...) and information about who did what. This information was used by researchers to help characterize CPPs after the KI interview.
Follow-up questions probed for more specific information, and these were followed by response options. After capturing CPPs, the KI interview also sought information to help characterize factors that facilitated and restrained community efforts. Questions and responses were prompted through a Federal Information Security Management Act-compliant Information Management System maintained by Battelle.
In addition, document abstraction was used to capture other instances of CPPs and to help clarify or confirm information reported in KI interviews. Research staff obtained and reviewed documents (e.g., annual program reports, reports to funders) and searched for web-based information about community activities. The information was used to capture candidate CPPs that were then scored using a codebook and scoring instructions.
2. Code instances of community programs and policies.
Coding of activities captured through KI interviews and document abstraction was conducted by CLs and Battelle staff, with training and technical support provided by the KU team based on prior research.7-9 Supports for systematic scoring included a codebook with definitions, examples and non-examples, and scoring instructions that defined an instance of a CPP.
For a program or policy to be scored as a CPP, it had to meet all of the following criteria:
1. It occurred (e.g., was not only planned).
2. It was a program, policy, or other change to the environment (e.g., walking trail) that existed in the community during the study period.
3. It was related to nutrition, physical activity, or weight control/ prevention of childhood obesity.
4. It targeted or benefited children aged 4-15 years.
5. It occurred in or benefited children in the defined community.
The Battelle team scored each documented activity as an instance (or not) of a CPP. The KU team ensured data quality by independently scoring a randomly selected set of identified CPPs. The reliability standard used was an interobserver agreement of > 80%; lower levels of agreement triggered re-training and certification of coders.
3. Characterize community programs and policies for key attributes. Once captured and coded, instances of CPPs were further characterized for attributes used in intensity scoring (see below) as well as other HCS hypotheses (e.g., behavioral objective addressed) and aspects of interest (e.g., sector in which implemented).
The CL posed specific questions during the KI interview to help characterize CPPs. For instance, to characterize the behavioral objective addressed by the CPP, CLs asked: What were the key behavioral objectives of the community program or policy? What behaviors of children were expected to change? These questions were followed by response options; for example, those for behavioral objectives related to physical activity included increase walking or biking to/from school and increase exposure to physical education, among others.
To characterize CPPs for intensity scoring, each CPP was coded for three specific attributes:
1. behavioral intervention strategy used (i.e., providing information and enhancing skills; enhancing services and support; modifying access, barriers, and opportunities; changing consequences; or modifying policies and broader conditions);
2. duration (i.e., description of the event as a one-time occurrence, occurring more than once, or being ongoing); and
3. reach (i.e., what proportion—high, medium, or low—of the total priority population was involved in or experienced the program or policy).
Each category was assigned a numerical value based on its relative strength for the attribute.7
4. Calculate intensity scores for community programs and policies. Utilizing the scores collected through the characterization of CPPs described above, the study team rated each dimension on a scale of 0 (minimum) to 1 (maximum), and summed the results to obtain a single intensity score7 for each program and policy. Scores were then summed for all CPPs in place for each year of the study period.
Table 1 describes the categories and assigned values for weighting the potential impact of a particular CPP. The values across all three domains were summed to create a single overall intensity score. Using this approach, the CPPs documented in the HCS could range in score from 0.3 (weakest and potentially of less influence on longer-term outcomes) to 3.0 (strongest and potentially of greater influence). The formula used to calculate intensity scores was: Strategy value + Duration value + Reach value.
Initial data analyses for intensity scoring will be computed using a formula that weights all elements equally; subsequent analyses will test variations of weighting that aim to capture the complex interactions among CPPs.
Table 2 provides two examples of CPPs observed in participating communities, organized by primary goal addressed, type of characterization, and intensity score (using methods described above).
Assuring Quality of Community Measurement
Several mechanisms were used to ensure the quality of data collected using the community measurement protocol. First, the KU team provided training for Battelle staff and CLs in capturing, coding, and characterizing instances of CPPs. Second, the KU team conducted direct
Fawcett et al / Am J Prev Med 2015;49(4):636-641 639
Table 1. Calculating the Intensity Score for Documented Community Programs and Policies Using Three Factors7
Characterization and weight for intensity scoring
Dimension Scoring rubric for characterizing intensity of documented community programs/policies (1=highest intensity; 0=lowest intensity) Examples
Type of behavioral intervention strategy High (1.0) - Modifying policies and systems; Changing consequences; Modifying access, opportunities, and barriers Medium (0.55) - Enhancing services and support Low (0.1) - Providing information and enhancing skills Competitive pricing for food choices Establishing a community garden or walking path Providing peer support for physical activity Providing an educational program for physical activity
Duration High (1.0) - Ongoing (i.e., throughout the designated year of the project/ study period A new bike bath continuously available
Medium (0.55) - Occurring more than once during that year A program that has 16 sessions over the year
Low (0.1) - One-time event A local health fair or 5K
Reach High (1.0) - >21% of the population to benefit/exposed to the CPP Medium (0.55) - 6%-20% of the population exposed to the CPP Low (0.1) - 0%-5% of the population exposed to the CPP A citywide pedestrian policy having an impact on all residents A policy implemented at a few schools in the district A program engaging fewer than 50 children
CPP, community programs/policies.
observations of KI interviews (via phone) and scored those sessions for compliance with the protocol. The minimum score required for all observations was 80% compliance with the protocol; CLs who received lower than the minimum score received refresher training, feedback, and additional monitoring. Third, weekly meetings of the HCS Quality Assurance and Quality Control Committee reviewed reports about the distribution of KIs across sectors and the degree of completeness of the data. Fourth, the Battelle and KU staff shared responsibility for scoring documented activities as CPPs. Independently, each party used descriptions of the activity and responses to key questions to indicate whether each CPP met all criteria for inclusion. Fifth, the KU team also scored all elements of the variables used in the intensity score to assure reliable reporting and coding by the KIs and CLs.
Discussion
This community measurement protocol was designed for the HCS, a large, multi-community study with the
primary aim of examining whether there is an association between CPPs and behaviors and healthy weight outcomes of children in participating communities. Thus, the overall goal of the community measurement system was to identify CPPs related to promoting healthy nutrition and physical activity of children and to characterize these sufficiently to estimate their intensity. Intensity scores were computed for each discrete CPP, and for all CPPs in a community, over a 10-year study period. Employing a mixed-methods approach, the protocol used KI interviews and document abstraction—as well as systematic coding and characterization—to permit computations of intensity scores for overall community efforts.
A large-scale, retrospective study of this kind poses challenges for community measurement. First, implementing the protocol required extensive logistical support
Table 2. Examples of Scoring for Community Programs and Policies
Community/program policy (goal addressed) Duration Reach Behavioral intervention strategy used Intensity scorea
Created walking path/greenway to connect neighborhoods and schools (physical activity) Ongoing High Modifying access, barriers, and opportunities 1.0
Provided an educational seminar to parents attending elementary school Parent Teacher Association meeting about how to promote healthy eating among children (healthy eating) One-time event Low Providing information and enhancing skills 0.10
a1=highest intensity; 0=lowest intensity
given the time, people, and other resources required to complete more than 1,500 interviews with KIs in 130 communities. Training and assuring quality performance of CLs required extensive initial training in the protocol and monitoring and feedback for quality assurance.
Second, time and resource constraints for KIs and CLs made it difficult to guarantee a complete capture of CPPs that occurred during the study period. Based on pilot testing, the protocol was simplified and refined to gather basic information about CPPs earlier in the KI interviews. Although there is likely a bias toward remembering longer-lasting programs/policies (compared with something of a short duration), data from initial communities showed that KIs could recall community programs/policies as far back as 10 years. If there is bias, it is assumed that it is systematic and can also be managed analytically. Although KIs reported that implementation of CPPs had occurred, the absence of a direct measure of the actual level of implementation is a limitation of this study.
Third, data cleaning was challenging because it required review and refinement of captured information for clarity and completeness, sometimes splitting reported clusters of multiple programs and policies into discrete CPPs. It also required harmonizing data from different KIs and document abstractions into descriptions of activities suitable for coding and characterization.
The HCS community measurement protocol has a number of strengths. First, it helps fill a key gap noted in prior IOM reports on childhood obesity,1,4,6 that is, the need to systematically unravel and operationalize the "complex web of influences" on physical activity and nutrition that may affect population-level outcomes related to childhood obesity. This community measurement protocol7 offers a replicable and standardized approach for capturing and characterizing the multiple and varied childhood obesity prevention programs/ policies adopted by local communities over time.
Second, intensity score data from the community measurement system, when integrated with behavioral and outcome measures, will permit analyses of possible associations between community efforts and related child behavior change and healthy weight outcomes. Systematic and consistent measurement across communities will allow for first-ever examinations of such relationships. When further validated, this intensity scoring approach could be useful for predicting the potential collective impact of comprehensive efforts to prevent childhood obesity, as well as for associating specific components of community efforts with beneficial outcomes.
Finally, this community measurement protocol permits the first-ever profile of comprehensive, community efforts to prevent childhood obesity in a diverse sample
of U.S. communities. Such data will be vital to developing a better understanding of how a community's adoption and implementation of combinations of programs and policies may lead to improved health for all its children.
The Healthy Communities Study was funded with federal funds from the National Heart, Lung, and Blood Institute, in collaboration with the Eunice Kennedy Shriver National Institute of Child Health and Development, National Institute of Diabetes and Digestive and Kidney Disorders, National Cancer Institute, and the NIH Office of Behavioral and Social Sciences Research; DHHS, under Contract No. HHSN268201000041C.
The DHHS NIH National Heart, Lung, and Blood Institute contract (HHSN268201000041C) to Battelle Memorial Institute funded this research; with subcontracts to scientific partners at the University of South Carolina, University of California-Berkeley, and the University of Kansas. The Community Measurement Subcommittee of the Healthy Communities Study—including representatives from the National Cancer Institute, National Institute of Child Health and Human Development, CDC, and Robert Wood Johnson Foundation—contributed to the development of the community measurement approach. More detail on the context, methods, and emerging findings from the HCS can be found elsewhere in this issue and at the HCS website hosted by the National Heart, Lung, and Blood Institute. Copies of the codebooks and scoring instructions used in the community measurement protocol are available from the Healthy Communities Study/National Heart, Lung, and Blood Institute website and from the University of Kansas team that leads that aspect of the study: http://communityhealth.ku.edu/.
No financial disclosures were reported by the authors of this paper.
References
1. IOM. Preventing Childhood Obesity: Health in the Balance. Washington, DC: The National Academies Press; 2005.
2. Ogden CL, Carroll MD, Kit BK, Flegal KM. Prevalence of childhood and adult obesity in the United States, 2011-2012. JAMA. 2014;311(8): 806-814. http://dx.doi.org/10.1001/jama.2014.732.
3. National Center for Health Statistics. Health, United States, 2011: With Special Features on Socioeconomic Status and Health. Hyattsville, MD; DHHS; 2012.
4. IOM. Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation. Washington, DC: The National Academies Press; 2012.
5. Collie-Akers VL, Fawcett SB. Preventing childhood obesity through collaborative public health action in communities. In: Jelalian E, Steele R, eds. Handbook of Child and Adolescent Obesity. New York: Springer Publishers, 2008.
6. IOM. Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress. Washington, DC: The National Academies Press; 2013.
7. Collie-Akers VL, Fawcett SB, Schultz JA. Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica. 2013;34(6):422-428.
8. Fawcett SB, Sterling T, Paine-Andrews AL, et al. Evaluating community efforts to prevent cardiovascular diseases. Atlanta, GA: CDC, National Center for Chronic Disease Prevention and Health Promotion; 1995.
9. Fawcett SB, Schultz JA. Using the Community Tool Box's online documentation system to support participatory evaluation ofcommunity health initiatives. In: Minkler M, Wallerstein N, eds. Community-based Participatory Research for Health. San Francisco: Jossey-Bass, 2008.
10. Collie-Akers VL, Schultz JA, Carson VL, Fawcett SB. Evaluating mobilization of neighborhood and faith organizations to reduce risk for health disparities within a REACH 2010 coalition in Kansas City, Missouri. Health Promot Pract. 2009;10(2):118S-127S. http://dx.doi. org/10.1177/1524839908331271.
11. Cheadle A, Rauzon S, Spring R, et al. Kaiser Permanente's community health initiative in northern California: evaluation findings and lessons learned. Am J Health Promot. 2012;27(2):e59-e68. http://dx.doi.org/ 10.4278/ajhp.111222-QUAN-462.
12. Glasgow R, Vogt T, Boles S. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322-1327. http://dx.doi.org/10.2105/AJPH.89.9.1322.
13. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006;21(5):688-694. http: //dx.doi.org/10.1093/her/cyl081.
