Measurement and evaluation practices of factors that contribute to effective health promotion collaboration functioning: A scoping review Academic research paper on "Health sciences"

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage www.elsevier.com/locate/evalprogplan

EVALUATION

and PROGRAM PLANNING

Measurement and evaluation practices of factors that contribute to effective health promotion collaboration functioning: A scoping review ^

Sean Stolp3'*, Joan L. Bottorffa,g, Cherisse L. Seatona, Margaret Jones-Brickerb, John L. Oliffec, Steven T. Johnsond, Sally Erreye, Kerensa Medhurstf, Sonia Lamonte

CrossMark

a Institute for Healthy Living and Chronic Disease Prevention, University of British Columbia, Kelowna, Canada

Canadian Cancer Society, British Columbia & Yukon Division, Prince George, Canada c School of Nursing, Faculty of Applied Science, University of British Columbia, Vancouver, Canada d Centre for Nursing and Health Studies, Athabasca University, Athabasca, Alberta, Canada e Prevention Programs, BC Cancer Agency, Canada f BC Cancer Agency, Canada

g Faculty of Health Sciences, Australian Catholic University, Melbourne, Australia

ARTICLE INFO

ABSTRACT

Article history:

Received 20 December 2015

Received in revised form 14 November 2016

Accepted 21 November 2016

Available online 22 November 2016

Keywords:

Collaboration

Health promotion

Scoping review

Measurement

Leadership

The purpose of this scoping review was to identify promising factors that underpin effective health promotion collaborations, measurement approaches, and evaluation practices. Measurement approaches and evaluation practices employed in 14 English-language articles published between January 2001 and October 2015 were considered. Data extraction included research design, health focus of the collaboration, factors being evaluated, how factors were conceptualized and measured, and outcome measures. Studies were methodologically diverse employing either quantitative methods (n = 9), mixed methods (n = 4), or qualitative methods (n = 1).

In total, these 14 studies examined 113 factors, 88 of which were only measured once. Leadership was the most commonly studied factor but was conceptualized differently across studies. Six factors were significantly associated with outcome measures across studies; leadership (n = 3), gender (n = 2), trust (n = 2), length of the collaboration (n = 2), budget (n = 2) and changes in organizational model (n = 2). Since factors were often conceptualized differently, drawing conclusions about their impact on collaborative functioning remains difficult. The use of reliable and validated tools would strengthen evaluation of health promotion collaborations and would support and enhance the effectiveness of collaboration. © 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND

license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction

The use of multi-stakeholdercollaborations to address complex social and cultural health disparities is becoming more common. Indeed, a need for collaborative work has been highlighted as early as 1986, with the Ottawa Charter for Health Promotion (World Health Organization, 1986) and remains an essential element in promoting health and health equity (World Health Organization, 2014). As a result, the collective benefit of multi-organizational collaborations to aid population and public health promotion are frequently described in the literature (Gillies, 1998; Graham &

* Corresponding author. E-mail address: sean.stolp@ubc.ca (S. Stolp).

Spengler, 2009; Kania & Kramer, 2011). However, bringing together organizations with varying structures, goals, and resources to achieve a shared collaborative goal can be challenging (Ansari & Weiss, 2006). In health promotion, stakeholders have drawn on multiple sources of literature to determine what factors, or components, need to be present for a collaboration to function effectively (Axelsson & Axelsson, 2006). However, there remains little consensus on what the most important factors are and how each contributes to effective collaborative processes and their potential impact on outcomes of the collaborative work (i.e., the effect of the health promotion initiative on people's health). Previous reviews (Foster-Fishman, Berkowitz, Lounsbury, Jacob-son, & Allen, 2001 ; Roussos & Fawcett, 2000) have identified factors associated with either collaborative functioning or community and population-level outcomes, yet these have not

http://dx.doi.org/10.1016/j.evalprogplan.2016.11.013

0149-7189/© 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

focused on measurement or evaluation of the factors. The objectives of this scoping review were to: a) identify and compare promising factors that contribute to effective health promotion collaboration, along with measurement approaches; and, b) make recommendations for strengthening assessments of population and public health promotion collaborations.

2. Methods

Based on the exploratory nature of the research objectives, a scoping review was conducted. A scoping review is a method of knowledge synthesis that addresses exploratory questions aimed at mapping the extent, range, and nature of research activity by systematically searching, selecting, and synthesizing existing knowledge (Arksey & O'Malley, 2005). In addition to identifying if a systematic review is feasible or needed, scoping reviews are also undertaken to provide a narrative summary of evidence, identify gaps, and offer conclusions about the state of research activity in a particular area. The scoping review was conducted in accordance with PRISMA Guidelines (Moher, Liberati, Tetzlaff, & Altman, 2009), an evidence-based guide for reporting reviews of the empirical literature.

2.1. Search strategies

Searches were conducted in the MEDLINE, CINAHL, PsychINFO and Academic Search Complete databases to retrieve peer-reviewed, empirical, English-language articles published between January 2001 and October 2015. Search phrases were a combination of nine terms ((Partnership OR Alliance OR Collabor* OR "Health collaboration") AND (Organization OR Agency) AND ("Health promotion" OR Prevention OR "Community development")).

2.2. Study selection

Articles were screened to evaluate whether they met the following inclusion criteria: 1) published in English, 2) addressed health promotion, and 3) formally evaluated the process of collaboration including the impact of specific factors on effective collaboration. We limited the search to partnerships that involved organizations in the public or non-profit sector. Despite growing interest in public-private partnerships to promote health, we excluded these types of partnerships because of the wide variation in the use of the term "partnerships" to label various types of interaction between government and industry, and the fact that evidence remains scarce about the effectiveness of these partnerships (Hernandez-Aguade & Zaragoza, 2016). Articles were also excluded if they focused on collaboration within a single agency (multi-department).

The search resulted in a total of3516 articles which yielded 2471 articles after duplicates were removed using RefWorks, a citation management program. A title and abstract review was conducted to exclude articles that did not meet the eligibility criteria. In total, 433 articles were identified for further assessment, and the full texts of these articles were reviewed. After excluding articles that did not meet the aforementioned criteria, 14 articles were retained for analyses. A flow diagram summarizing article inclusion/ exclusion is provided in Fig. 1.

2.3. Data extraction and quality assessment

Relevant information from each article was extracted, including research design, health focus of the collaboration, factors being evaluated, how the factors were conceptualized and measured, outcome measures, and major findings. Data extraction for each

Fig. 1. Article Selection Procedure.

article was completed by authors, and areas of disagreement were resolved through discussion. The quantitative studies were evaluated on a scale from 0 to 10 using the Liverpool Quality Assessment Tool (Pope, 2015). The tool was chosen for its ability to assess a wide range of different study methodologies (Voss & Rehfuess, 2013). The qualitative studies were evaluated on a scale form 0-11 using an adapted version of a quality assessment tool used in previous public health systematic reviews (Harden, Brunton, Fletcher, & Oakley, 2009; Puzzolo, Stanistreet, Pope, Bruce, & Rehfuess, 2013). For mixed methods studies, quality assessment tools were assigned according to the dominant method used by the study. Scores were standardized as a percentage for comparative purposes. Quality assessment scores ranged from 33.3 -100% with an average score of 70.4%.

3. Results

The 14 articles included in this review involved studies undertaken in three different countries, the USA (n=12), Ireland (n = 1) and Holland (n=1). The majority of studies focused on a single collaboration involving two or more collaborative partners (n=11); the remaining three studies focused on collaborations with 21, 40 and 99 partners. The health focus of the collaborations under study varied greatly and included youth and childhood health promotion (e.g., substance abuse or teen pregnancy prevention) (n = 5), undefined health promotion (n = 2), and community health promotion (n = 2). Five studies focused on unique collaborations with health promotion efforts targeting: response to the H1N1 influenza virus, prevention of type 2 diabetes, tobacco reduction, HIV prevention, and prevention of lead exposure.

The majority of studies selected for this review were cross-sectional (n=12) and two were longitudinal in design. Studies utilized quantitative methods (n = 9), mixed methods (n = 4) and qualitative methods (n = 1) to assess the contribution of specific factors to effective collaboration. Of note, five studies employed network analysis (n = 5).

3.1. Study participants

In the majority of studies (n = 6), the responses of an individual were used to represent a collaborating organization (number of study participants: 6-599). This was followed by studies in which participants responded as individual members of an organizational collaboration (n = 4; number of study participants: 106-168). In three studies participant responses were aggregated to the collaborative level (number of study participants: 99-337).

3.2. Factors contributing to collaborative effectiveness

Researchers examined a wide range of factors as independent variables in assessing their contribution to the effectiveness of collaborative processes. On average the number of factors under investigation per study was 8.4 (range = 1-20). The 14 studies examined the contribution of 101 unique factors, or independent variables, to collaboration functioning. Of the 101 unique factors

identified, 88 were only measured in one study. These factors were allocated to four groups: 1) characteristics of the collaboration (32 factors); 2) community/organization characteristics (29 factors); 3) measures relevant to network analysis (13 factors); and, 4) collaborative actions (14 factors).

Of the 101 factors, 13 were examined in more than one study although conceptualization and measurement of these factors varied for 7 of these (see Table 1). Leadership was the most commonly included factor, and was evaluated in five studies but definitions and measures of leadership differed. Factors that were examined as independent variables and conceptualized across studies in the same manner were: network density (n = 4), network centralization (n = 3), network links through sharing information (n = 2), goals and structure of an organization (n = 2), programming provided by organizations (n = 2) and length of participation in the collaboration (n = 2).

Five studies reported on the reliability of the measures used (Cramm, Phaff, & Nieboer, 2013; Feinberg, Chilenski, Greenberg,

Table 1

Factors contributing to collaboration effectiveness with differing conceptualizations across studies.

Article

Conceptualization of the Factor

Measure Source

Leadership Kramer et al. (2005)

Brown et al. (2008) Jones and Barry (2011b)

Cramm et al. (2013)

Perkins et al. (2011)

Benefits associated with collaboration participation Provan et al. (2005)

Kramer et al. (2005)

Feinberg et al. (2007) Trust

Provan et al. (2005)

Jones and Barry (2011b)

Retrum et al. (2013)

Skills and Resources Feinberg et al. (2007)

Jones and Barry (2011b)

Cramm et al. (2013)

Administration and management of partnership Jones and Barry (2011b)

Cramm et al. (2013)

Efficiency

Jones and Barry (2011b) Cramm et al. (2013) Budget

Zahner(2005) Kun et al. (2013)

Defined by presence of a steering or executive committee, an identifiable non-paid collaboration chair, paid staff in leadership roles, and the number of directors during the life of the project Positional or referred leadership

Defined as capabilities for fixing public problems in a shared-power world

Defined by an example item: "inspiring or motivating people and taking responsibility"

Extent to which team leadership encouraged collaborative processes and displayed effective interpersonal and leadership behaviors

Survey questions designed by local evaluators

Descriptive measure of the organization structure 11 items, 5 point scale, previously developed (Weiss, Anderson, & Lasker, 2002)

4 items, 5 point scale from the validated Partnerships Self-Assessment Tool (Cramm, Strating, & Nieboer, 2011)

8 item scale, previously developed (Kegler, Steckler, McLeroy, & Malek, 1998)

Benefits in relation to collaboration involvement Perceived benefits of participation in the collaboration

The balance of benefits to costs associated with participation in a collaboration

Quality of relationship among partnering agencies based on level of trust

Assessed two components: trust and mistrust. Focused on trusting (openness and sharing) and trustworthiness (support and acceptance)

Trust within a collaborative, based upon organization reliability, support of the mission and openness to discussion

Skills and resources team members bring to the team process

Boundary-spanning skills, example items include "ability to work effectively with the community" and "ability to see new opportunities for the partnership" Non-financial resources including data and information (statistical data, information, skills, expertise)

Communicating effectively, coordinating activities, managing grants and funds, orientating new partners and evaluating the impact of the partnership on health Evaluating the progress and impact of the partnership and organizing partnership activities, including meetings and projects

Conceptualized as administration and management efficiency

Example item: "how well the partnership uses the partners' financial resources."

Whether the partnership had a budget

How funding contributed to strengthening partnership

11 items, 4 point scale based on previous (Provan, Nakama, Veazie, Teufel-Shone, & Huddleston, 2003) Survey questions (unknown structure) designed by local evaluators

8 item investigator developed scale; reliability reported

1 item, 4 point scale based on previous (Provan & Milward, 1995; Provan et al., 2003) 14 items, 5 point scale previously developed (Jones, 2008)

3 items, unknown source

13 items, 5 point scale developed by investigators, reliability reported

14 items, 5 point scale, investigator developed based upon literature review, reliability reported

4 items, 5 point scale from the validated Partnerships Self-Assessment Tool (Cramm et al., 2011)

8 item, 5 point scale previously developed (Weiss et al., 2002)

4 items, 5 point scale from the validated Partnerships Self-Assessment Tool developed (Cramm et al., 2011)

3 item, 5 point scale previously developed (Weiss et al., 2002)

3 items, 5 point scale from the validated Partnerships Self-Assessment Tool (Cramm et al., 2011)

Dichotomous descriptive measure Open-ended question

Table 2

Effective collaboration and significantly related factors.

Measure of Effective Collaboration

Method of Assessment

Factors Significantly Related to Effective Collaboration

Prevention Collaboration (CF) Brown et al. (2008)

Team Functioning (CF)

■ Focus on work

■ Team leadership

■ Team culture

■ Team tension

Feinberg et al. (2002)

Partnership synergy (CF) Jones and Barry (2011b)

Strength of partnership (CF) Kun et al. (2013)

Network centrality (CF) Leischow et al. (2010)

Success in achieving collaboration among service providers (CF) Parrish et al. (2013)

Sustainability planning (CF) Perkins et al. (2011)

9 items, based upon literature review, assessing collaboration regarding prevention-specific activities within communities (a = 0.91)

Focus on work: 5 items, adapted (Moos and Insel, 1981); example item: "People pay a lot of attention to getting work done" (a = 0.72) Team leadership: 8 items, adapted (Kegler et al., 1998); assesses the degree to which team leadership encourages input and consensus, along with promotes a friendly work-environment (a = 0.85) Team culture: 8 items, adapted (Kegler et al., 1998); assesses the team atmosphere. (a = 0.89) Team tension: 1 item, 4 point Likert scale, unknown source; assesses the degree of conflict and tension in the collaboration

■ Two previously developed scales (Jones and Barry, 2011a; Weiss et al., 2002): 9 item, 5 point Likert scale and 8 item, 5 point Likert scale; conceptualized as "the degree to which the partnership combines the complementary strengths, perspectives, values and resources of all the partners in the search for better solutions. A partnership that has maximized synergy has achieved the full potential of collaboration." (combined a = 0.91, pattern coefficients 0.696 - 0.833)

■ 1 item, unknown source, rating "the strength of partnership with each organization" (minimal, average, good and excellent)

Of 11 factors examined, one was significant: Females

reported higher levels of prevention collaboration than

Of 14 factors examined, 8 were significantly related to

team functioning:

■ Community poverty was negatively associated with team leadership (r= -0.59) and team culture (r=-0.51)

■ Substance use norms was associated with team tension (r = -0.49) after controlling for community poverty

■ School collaboration and value of prevention were negatively associated with team tension (r= -0.57, and r = -0.45, respectively)

■ Community collaboration was associated with team culture (r=0.38)

■ Team member acceptance of adolescent alcohol use was negatively associated with team focus on work (r = -0.62) and culture (r = -0.49, after controlling for community poverty)

■ Net benefits of participation correlated with team focus on work (r= 0.54), and team culture (r= 0.64, after controlling for community poverty)

■ Collaboration with the prevention coordinator was correlated with team leadership (r=0.53)

Of 8 factors examined, 3 were significant:

■ Trust/mistrust (b = 0.59)

■ Leadership (b =0.63)

■ Efficiency predicted partnership synergy (b =

Only 1 factor examined and was significant: Funding increased partnership strength

Calculated by measuring contact between network members; unknown source

Of 7 factors examined, 2 were significant:

■ Member job rank was significantly associated with member centrality

■ More barriers were cited by organizations central in the network

■ Content analysis of qualitative responses, unknown Of 3 factors examined, 3 were consistently said to be source negatively related to the outcome measure:

■ lack of communication

■ lack of input

■ lack of investment contributed to low success in achieving collaboration

Willingness to collaborate (CF) Pinto (2013)

■ Team expertise: 4 items assessed the extent to which local team had necessary expertise to fulfill the goal of sustaining the collaboration (a = 0.69)

■ Team confidence (a = 0.52): 2 items assessed the degree to which local teams are interested in sustaining programs (r = 0.52)

■ Team funding plans: 4 items assessed how thoroughly teams had made plans to secure funds for programs beyond the grant (a = 0.44)

■ Leadership: 3 items assessed the extent to which team leadership encouraged collaborative processes and displayed effective interpersonal and leadership behaviors (a = 0.65)

■ 1 item, 5-point scale, assessing willingness to Of 7 factors examined, 4 significantly predicted engage in a collaborative project, unknown source willingness to collaborate including:

Of 8 factors examined, 3 were significant:

■ New member integration was associated with team expertise (r = 0.59), team confidence (r = 0.53) and funding plans (r=0.74)

■ Member participation associated with team expertise (r=0.62) and funding plans (r=0.60)

■ Team enablers associated with team expertise (r=0.64) and team funding plans (r =0.48)

■ Attitudes towards researchers availability (b = 0.07)

■ Perceptions of the benefits of research (b = 0.08)

Table 2 (Continued)

Measure of Effective Collaboration

Method of Assessment

Factors Significantly Related to Effective Collaboration

Network links (CF) Provan et al. (2005)

Number of linkages in network (CF) Density scores (CF) Intensity of collaboration (CF) Singer and Kegler (2004)

Shared information link, dichotomous Shared resources link, dichotomous

■ Joint projects link, dichotomous

■ Client patient/referrals link, dichotomous

■ Any links: respondents indicated the organizations they linked with and the type of link (i.e., sharing information); dichotomous

■ Number of linkages in network

■ Density scores: the number of linkages present in the network/total number of possible links

■ Intensity of collaboration: participants indicated frequency of interaction with other organizations (no relationship, one to four interactions/year)

■ Agency preparedness (b = 0.10)

■ Gender (women were significantly more likely than men to collaborate) (b = 0.37)

Of 7 factors examined, 3 were significantly related to

network links:

■ Reputation was correlated with shared information links (r=0.68), shared resources (r=0.53), joint projects (r=0.72), referrals (r = 0.78) and any link (r =0.82)

■ Perceived benefit was correlated with shared resources (r=0.53)

■ Trust was correlated with joint projects (r=0.51)

Of 9 factors examined, 3 were significantly related to

nature of linkages in collaborative network:

■ Organizations with a program and with staff committed to lead had more linkages

■ Density scores are highest at the lower intensity of collaboration and weaken progressively as the intensity of collaboration moved towards formal relationships among partners

Sustainability of partnership implemented programmes (CO) Cramm et al. (2013)

9 item instrument developed by Slaghuis, Strating, Bal, Of 5 factors examined, 3 were significant in predicting & Nieboer (2011); example items include: "the new sustainability of partnership program:

practice is regarded as the standard way to work" and "all colleagues involved in the new work practice are knowledgeable about it." (a= .84, validity referenced)

■ Leadership (b = 0.32)

■ Non-financial resources (b = 0.25)

■ Partnership synergy (controlling for all other partnership functioning variables) (b = 0.39)

Perceptions of achieving of interim outcomes (CO) Kramer et al. (2005)

7 items, related to interim collaborative project goals of bringing successful teen pregnancy prevention strategies, such as "greater organizational strength and coordination focused on teen pregnancy and other youth development issues." (a unreported)

Of 13 factors examined, 5 were significantly associated

with perceived achievement of interim outcomes:

■ Projects with a catchment wide collaboration were associated with better perceived interim outcomes

■ Having a steering committee was associated with better perceived interim outcomes

■ Changes in the organizational model over time was negatively correlated to interim outcomes

■ Leadership: paid staff were more likely to achieve interim outcomes

■ Younger collaborations less likely to have achieved interim outcomes

Value that partners bring to each collaborative (CF) Trust within a collaborative (CF) Resources (CO) Outcomes achieved (CO) Most important outcome achieved (CO) Retrum et al. (2013)

Organizational Collaborativeness Index Score (CF) Implementation of partnership plans (CO) How successful implemented plans were (CO) Zahner(2005)

Value that partners bring to collaborative: 3 items, assesses perceptions of values brought to collaborative, unknown source

Trust within a collaborative: 3 items, unknown source

Resources, chosen from a closed list (Number reported, diversity reported) Outcomes achieved and most important outcome achieved: respondents selected project outcomes from a list (new policy development, education campaigns, reduced health disparities, etc.) and then indicate the most important outcome

4 items, 5 point Likert; conceptualized as an organizational characteristic that has the potential to affect the likelihood of using partnership strategies and effectiveness in doing so, Implementation of partnership plans, dichotomous (Weiss, Miller, & Lasker, 2001) Success of plans: dichotomized to very/generally successful or very/somewhat unsuccessful

Of 3 factors examined, 3 were significant:

■ Collaboratives with higher density perceive partners as more valuable (b =0.53), are more trusted (b = 0.50) and more likely to have rare resources (b = 0.36)

■ Collaboratives with less breadth had greater agreement on most important project outcome achieved (b = 0.40) and higher total resources contributed (b = 0.24)

■ More de-centralized collaboratives are associated with a greater number of reported project outcomes (b = 0.40)

Of 22 factors examined, 9 were significant:

■ Number of partner types (OR= 1.29), budget (OR= 2.15), financial contribution (OR = 2.94), having a budget (OR = 2.15), length of partnership (OR = 8.29), conducting community events (OR = 6.86), public information dissemination (OR = 2.06), and coordination of direct services predicted successful implementation of plans

■ Government mandate (OR = 0.2) and system redesign (OR = 0.37) negatively predicted implementation of plans

CF = Collaboration Functioning variable. CO = Collaboration Outcome variable.

Spoth, & Redmond, 2007; Jones & Barry 2011b; Perkins et al., 2011; Pinto, 2013). In two studies (Kramer et al., 2005; Zahner, 2005), the measures section was insufficiently detailed to determine if the reporting of reliability and validity of the measures used was necessary.

3.3. Measuring effective collaboration and contributing factors

The 14 studies reported on 25 different measures of effective health promotion collaborations (average = 2.4; range 1-6), with no duplication of measures across studies. The measures of effective collaboration represented in the 14 studies were grouped into two types: 1) outcomes specifically related to collaborative functioning (e.g., strength of partnership) and, 2) outcomes related to the focus of the collaborative work (e.g., the impact of the health promotion initiative). Ten studies examined predictors of collaborative functioning, 2 studies examined predictors of collaborative project outcomes, and 2 studies included both. In Table 2, variables related to effective collaboration, their associated measures, and factors that were found to be significantly related these outcomes are presented. Across the 14 studies, five factors were examined in more than one study and found to be statistically significantly related to their respective outcome variable(s): leadership (n = 3), gender (n = 2), trust (n = 2), length of time the collaboration existed (n = 2), budget (n = 2), and changes in the organizational model (n = 2).

4. Discussion

The objective of this scoping review was to identify promising factors that underpin effective health promotion collaborations, as well as the measurement approaches and evaluation practices used to identify these factors. In the 14 studies included in this review a wide range of factors were evaluated for their role in contributing to the effectiveness a health promotion collaboration; however, few factors were assessed in more than one study. This lack of repeated factors is in contrast to a recent review of descriptive studies conducted to identify factors that contributed to collaboration effectiveness (Seaton et al., 2016). In this review, a shared vision, effective leadership, member characteristics, organizational commitment, availability of resources, clear roles and responsibilities, trusting relationships, and engaging the target population were identified as important by many of the 25 studies reviewed. In the present scoping review, many of these same factors were evaluated and found to be related to collaborative outcomes. These findings provide a foundation for further exploration of factors that if in place would improve a collabo-rative's chances of success and allow for the evaluation of a collaborative's efforts to be assessed not only by its working outcomes, but also on how well the collaboration itself functioned.

Two factors that stand out in this scoping review as holding particular significance for multi-organizational collaborations in health promotion are leadership and trust. While findings corroborate previous work where leadership was identified as the most commonly measured collaborative factor (Jones, 2008), leadership was also the factor most often empirically related to health promotion outcomes achieved through the collaborative work. This was evident despite the fact that leadership was conceptualized differently across each of the five studies in which this factor appeared. In these studies, for example, leadership was measured by examining whether collaboration leaders were paid staff, the extent to which leaders were inspiring or motivating, and the extent to which leaders displayed effective interpersonal and leadership behaviors. Explorations using these varying conceptualizations of leadership highlight the need to further examine

which components of leadership are needed and/or most important for a collaborative to function well.

Similarly, trust has previously been found to be the most important factor involved in successful collaborative functioning and is thought to be so critical that it is almost impossible to offset its absence (Child & Faulkner, 1998). Across the included studies trust was evaluated three times, but was also conceptualized and operationalized differently in each study. Trust was measured using a 1 item scale assessing "level of trust", a 14 item scale assessing openness, sharing, and trustworthiness, and a 3 item scale assessing trust based upon organization reliability, support of the collaborative mission, and openness to discussion.

Despite the variation in conceptualization and measurement, it is interesting that both trust and leadership were often found to significantly contribute to inter-organization health promotion collaborative functioning, signaling the potential importance of these factors. However, a more consistent approach to conceptualizing and measuring these factors is needed to verify these findings and understand their effect at different stages of collaborative work.

Contributing to the varying conceptualization of factors is the finding that very few studies used previously validated measures. The development and use of reliable and validated tools would strengthen evaluation of health promotion collaborations and advance our understanding of ways to support and enhance the effectiveness of collaborative efforts. The lack of reliable and validated tools has been previously noted (Ansari & Weiss, 2006). Only two studies (Cramm et al., 2013; Jones & Barry, 2011b) reported or referenced the reliability and validity of their measures. Although in these two studies conceptualizations of leadership, efficiency, and partnership synergy differed. It should be noted that other valid measurement tools to assess collaborative functioning in health promotion do exist in the literature (Leurs, Mur-Veeman, van der Sar, Schaalma, & de Vries, 2008), yet they were not used in the studies reviewed. A more consistent effort by researchers to use measures conceptualized and operationalized in the same manner will aid the evaluation of collaborative health promotion.

5. Limitations

This scoping review examined literature related to collaborations focused on health promotion that involved organizations in the public or non-profit sector. As such, relevant collaboration-based findings from other fields are not represented here. The included studies were also limited to those that empirically evaluated the impact of specific factors on effective collaboration. It should also be noted that the objective of this study was to identify promising factors that underpin effective health promotion collaborations, yet, collaborating effectively is not a guarantee of a successful health promotion initiative as there are many other factors to consider that are specific to the initiative itself.

6. Lessons learned

The findings drawn from the current scoping review indicate that despite the growing number of health promotion based collaborations, the number of studies specifically measuring factors and their contribution to the effective functioning of these collaborations should continue to be explored. Further investigation will allow for more consistent conceptualization of factors, such as leadership, and in turn, allow for a more accurate depiction of how these factors contribute to collaboration functioning. The use and promotion of standardized tools would be of great benefit to multi-organizational collaborations focusing on population and public health.

Acknowledgment

This research was supported by the Canadian Cancer Society (Grant # 701259-00).

References

Ansari, W. E., & Weiss, E. S. (2006). Quality of research on community partnerships: Developing the evidence base. Health Education Research, 21(2), 175-180. http:// dx.doi.org/10.1093/her/cyh051.

Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32.

Axelsson, R., & Axelsson, S. B. (2006). Integration and collaboration in public health-a conceptual framework. International Journal of Health Planning and Management, 21(1), 75.

Brown, E. C., Hawkins, J. D., Arthur, M. W., Abbott, R. D., & Van Horn, M. L. (2008). Multilevel analysis of a measure of community prevention collaboration. American Journal of Community Psychology, 41(1-2), 115-126. http://dx.doi.org/ 10.1007/s10464-007-9154-855.

Child, J. (1998). Strategies of co-operation. managing alliances, networks, and joint ventures. Oxford: Oxford University Press.

Cramm, J. M., Strating, M. M., & Nieboer, A. P. (2011). Development and validation of a short version of the partnership self-assessment tool (PSAT) among professionals in dutch disease-management partnerships. BMC Research Notes, 4,. http://dx.doi.org/10.1186/1756-0500-4-224 [224-0500-4-224].

Cramm, J. M., Phaff, S., & Nieboer, A. P. (2013). The role of partnership functioning and synergy in achieving sustainability of innovative programmes in community care. Health & Social Care in the Community, 21(2), 209-215.

Feinberg, M. E., Greenberg, M. T., Osgood, D. W., Anderson, A., & Babinski, L. (2002). The effects of training community leaders in prevention science: Communities that care in pennsylvania. Evaluation and Program Planning, 25(3), 245-259.

Feinberg, M. E., Chilenski, S. M., Greenberg, M. T., Spoth, R. L., & Redmond, C. (2007). Community and team member factors that influence the operations phase of local prevention teams: The PROSPER project. Prevention Science: The Official Journal of the Society for Prevention Research, 8(3), 214-226.

Foster-Fishman, P. G., Berkowitz, S. L., Lounsbury, D. W., Jacobson, S., & Allen, N. A. (2001). Building collaborative capacity in community coalitions: A review and integrative framework. American Journal of Community Psychology, 29(2), 241261.

Gillies, P. (1998). Effectiveness of alliances and partnerships for health promotion. Health Promotion International, 13(2), 99-120.

Graham, G. N., & Spengler, R. F. (2009). Collaborating to end health disparities in our lifetime. American Journal of Public Health, 99(11), 1930-1932. http://dx.doi.org/ 10.2105/AJPH.2009.167908.

Harden, A., Brunton, G., Fletcher, A., & Oakley, A. (2009). Teenage pregnancy and social disadvantage: Systematic review integrating controlled trials and qualitative studies. BMJ (Clinical Research Ed.), 339, b4254. http://dx.doi.org/ 10.1136/bmj.b4254.

Hernandez-Aguade, I., & Zaragoza, G. A. (2016). Support of public-private partnerships in health promotion and conflicts of interest. BMJ Open, 6, e009342. http://dx.doi.org/10.1136/bmjopen-2015-009342.

Jones, J., & Barry, M. M. (2011a). Developing a scale to measure synergy in health promotion partnerships. Global Health Promotion, 18(2), 36-44. http://dx.doi. org/10.1177/1757975911404762.

Jones, J., & Barry, M. M. (2011b). Exploring the relationship between synergy and partnership functioning factors in health promotion partnerships. Health Promotion International, 26(4), 408-420. http://dx.doi.org/10.1093/heapro/ dar002.

Jones, J. (2008). Health Promotion Partnerships: An analysis of the factors that

contribute to successful partnership functioning. Galway: National University of Ireland [Unpublished phd thesis].

Kania, J., & Kramer, M. (2011). Collective impact. Stanford Social Innovation Review36-41.

Kegler, M. C., Steckler, A., McLeroy, K., & Malek, S. H. (1998). Factors that contribute to effective community health promotion coalitions: A study of 10 project ASSIST coalitions in north carolina. american stop smoking intervention study for cancer prevention. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 25(3), 338-353.

Kramer, J. S., Philliber, S., Brindis, C. D., Kamin, S. L., Chadwick, A. E., Revels, M. L., et al. (2005). Coalition models: Lessons learned from the CDC's community coalition partnership programs for the prevention of teen pregnancy. The Journal of Adolescent Health: Official Publication of the Society for Adolescent Medicine, 37(Suppl. 3), S20-S30 S1054-139X(05)00248-X [pii].

Kun, K. E., Zimmerman, J., Rose, D. A., & Rubel, S. (2013). State, territorial, and local health departments' reporting of partnership strength before and after the H1N1 response. Prehospital and Disaster Medicine, 28(6), 580-585. http://dx.doi. org/10.1017/S1049023x13009011.

Leischow, S. J., Luke, D. A., Mueller, N., Harris, J. K., Ponder, P., Marcus, S., et al. (2010). Mapping U.S. government tobacco control leadership: Networked for success? Nicotine & Tobacco Research: Official Journal of the Society for Research on Nicotine and Tobacco, 12(9), 888-894. http://dx.doi.org/10.1093/ntr/ntq112.

Leurs, M., Mur-Veeman, I., van der Sar, R., Schaalma, H., & de Vries, N. (2008). Diagnosis of sustainable collaboration in health promotion-a case study. BMC Public Health, 8(1), 382.

Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269.

Moos, R., & Insel, P. (1981). Manual for work environment scale. Palo Alto CA Consulting.

Parrish, D. E., Harris, D., & Pritzker, S. (2013). Assessment of a service provider self-study method to promote interorganizational and community collaboration. Social Work, 58(4), 354-364.

Perkins, D. F., Feinberg, M. E., Greenberg, M. T., Johnson, L. E., Chilenski, S. M., Mincemoyer, C. C., et al. (2011). Team factors that predict to sustainability indicators for community-based prevention teams. Evaluation and Program Planning, 34(3), 283-29 .

Pinto, R. M. (2013). What makes or breaks provider-researcher collaborations in HIV research? A mixed method analysis of providers' willingness to partner. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 40(2), 223-230. http://dx.doi.org/10.1177/1090198112447616.

Pope, D. (2015). Personal communication..

Provan, K. G., & Milward, H. B. (1995). A preliminary theory of interorganizational network effectiveness: A comparative study of four community mental health systems. Administrative Science Quarterly 1-33.

Provan, K. G., Nakama, L., Veazie, M. A., Teufel-Shone, N. I., & Huddleston, C. (2003). Building community capacity around chronic disease services through a collaborative interorganizational network. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 30(6), 646-662. http://dx.doi.org/10.1177/1090198103255366.

Provan, K. G., Harvey, J., & de Zapien, J. G. (2005). Network structure and attitudes toward collaboration in a community partnership for diabetes control on the US-mexican border. Journal of Health Organization and Management, 19(6), 504518.

Puzzolo, E., Stanistreet, D., Pope, D., Bruce, N., & Rehfuess, E. (2013). Factors influencing the large-scale uptake by households of cleaner and more efficient household energy technologies..

Retrum, J. H., Chapman, C. L., & Varda, D. M. (2013). Implications of network structure on public health collaboratives. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 40(Suppl. 1), 13S-23S. http://dx.doi.org/10.1177/1090198113492759.

Roussos, S. T., & Fawcett, S. B. (2000). A review of collaborative partnerships as a strategy for improving community health. Annual Review of Public Health, 21, 369-402.

Seaton, C. L., Holm, N., Bottorff, J. L., Jones-Bricker, M., Errey, S., Caperchione, C., et al. (2016). Factors that impact the success of inter-agency health promotion collaborations: A Scoping Review [In review]

Singer, H. H., & Kegler, M. C. (2004). Assessing interorganizational networks as a dimension of community capacity: Illustrations from a community intervention to prevent lead poisoning. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 31(6), 808-821.

Slaghuis, S. S., Strating, M. M., Bal, R. A., & Nieboer, A. P. (2011). A framework and a measurement instrument for sustainability of work practices in long-term care. BMC Health Services Research, 11,. http://dx.doi.org/10.1186/1472-6963-11-314 [314-6963-11-314].

Voss, P. H., & Rehfuess, E. A. (2013). Quality appraisal in systematic reviews of public health interventions: An empirical study on the impact of choice of tool on meta-analysis. Journal of Epidemiology and Community Health, 67(1), 98-104.

Weiss, E., Miller, R., & Lasker, R. D. (2001). Findings from the national study of partnership functioning: Report to the partnerships that participated Center for the Advancement of Collaborative Strategies in Health, Division of Public Health. New York Academy of Medicine.

Weiss, E. S., Anderson, R. M., & Lasker, R. D. (2002). Making the most of

collaboration: Exploring the relationship between partnership synergy and partnership functioning. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 29(6), 683-698.

World Health Organization (1986). Ottawa charter for health promotion. [Accessed July 12, 2016 at: http://www.who.int/healthpromotion/conferences/previous/ ottawa/en/].

World Health Organization (2014). Health in all policies: Framework for country action. [Helsinki, Finland. Accessed July 12, 2016 at: http://www.who.int/ cardiovascular_diseases/140120HPRHiAPFramework.pdf].

Zahner, S. J. (2005). Local public health system partnerships. Public Health Reports, 120(1), 76-83.