Scholarly article on topic 'Improving the reporting of public health intervention research: advancing TREND and CONSORT'

Improving the reporting of public health intervention research: advancing TREND and CONSORT Academic research paper on "Economics and business"

0
0
Share paper
Academic journal
Journal of Public Health
OECD Field of science
Keywords
{""}

Academic research paper on topic "Improving the reporting of public health intervention research: advancing TREND and CONSORT"

Journal of Public Health | Vol. 30, No. 1, pp. 103-109 | doi:10.1093/pubmed/fdm082 | Advance Access Publication 19 January 2008

Improving the reporting of public health intervention research: advancing TREND and CONSORT

Rebecca Armstrong1,2, Elizabeth Waters1,2, Laurence Moore3, Elisha Riggs2, Luis Gabriel Cuervo4, Pisake Lumbiganon5, Penelope Hawe6

Cochrane Health Promotion and Public Health Field, VicHealth, PO Box 154, Carlton South, VIC 3053, Melbourne, Australia

2The McCaughey Centre: the VicHealth Centre for the Promotion of Mental Health and Community Wellbeing, School of Population Health, University of Melbourne, Melbourne, Australia

3Cardiff Institute of Society, Health and Ethics, Cardiff University, Cardiff, Wales, UK

4Pan American Health Organization (WHO/PAHO), Washington DC, USA

5Clinical Epidemiology Unit, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand

6Population Health Intervention Research Centre, University of Calgary, AB, Canada

Address correspondence to Rebecca Armstrong, E-mail: rarmstrong@vichealth.vic.gov.au

ABSTRACT

Background Evidence-based public health decision-making depends on high quality and transparent accounts of what interventions are effective, for whom, how and at what cost. Improving the quality of reporting of randomized and non-randomized study designs through the CONSORTand TREND statements has had a marked impact on the quality of study designs. However, public health users of systematic reviews have been concerned with the paucity of synthesized information on context, development and rationale, implementation processes and sustainability factors.

Methods This paper examines the existing reporting frameworks for research against information sought by users of systematic reviews of public health interventions and suggests additional items that should be considered in future recommendations on the reporting of public health interventions.

Results Intervention model, theoretical and ethical considerations, study design choice, integrity of intervention/process evaluation, context, differential effects and inequalities and sustainability are often overlooked in reports of public health interventions.

Conclusion Population health policy makers need synthesized, detailed and high quality a priori accounts of effective interventions in order to make better progress in tackling population morbidities and inequalities. Adding simple criteria to reporting standards will significantly improve the quality and usefulness of published evidence and increase its impact on public health program planning.

Keywords public health, primary research, evidence, editorial policies

Introduction

For many years, primary research has been supported by statements or guidelines aimed at improving the quality of reporting. This has resulted in significant improvements in reports of intervention studies, particularly for clinical randomized controlled trials (RCTs). However, there remain many deficiencies in these frameworks for the reporting of complex public health interventions. This has implications for those who conduct systematic reviews of public health interventions, those who conduct implementation evaluations or health systems research and those who use these forms of evidence to inform their decision-making. This paper examines the existing reporting frameworks against information sought by users of systematic reviews of public health interventions, and suggests key additional items that

should be considered in future recommendations on the reporting of public health interventions by researchers and publishers. As the demand for high quality, transparent secondary research increases amongst health care policy makers

Rebecca Armstrong, Senior Research Fellow

Elizabeth Waters, Professorial Fellow, Public Health and Health Equity Laurence Moore, Professor and Director Elisha Riggs, Research Fellow

Luis Gabriel Cuervo, MD, Unit Chief, Research Promotion and Development Pisake Lumbiganon, Professor in Ob & Gyn

Penelope Hawe, Professor, Markin Chair in Health and Society & Alberta Heritage, Foundation for Medical Research Health Scientist

The views expressed in this publication do not necessarily represent those of the organisations within which authors are employed.

© The Author 2008, Published by Oxford University Press on behalf of Faculty of Public Health. All rights reserved

and other end-users it is an auspicious time to address these

issues.

Value of CONSORT and TREND in improving the reporting of primary research

The Consolidated Standards of Reporting Trials (CONSORT) statement was first developed in 1996 (and updated in 2001)4'5 with the aim of improving the quality of reporting of RCTs. It requires authors of reports and journal articles to follow a checklist of 22 items that should be reported. CONSORT has now been adopted by more than 150 medical, clinical and psychological journals (www. consort-statement.org). The need for an alternate statement to improve the reporting of non-randomized studies was subsequently identified in 2004. This resulted in the development of the Transparent Reporting of Evaluations with Non-randomized Designs (TREND) Statement.6 The TREND statement is potentially very important for disciplines such as public health where debate continues about the extent to which RCTs are feasible or appropriate.7'8

A number of other guidelines have also been published to support the systematic review process including QUORUM9 (recently renamed PRISMA) (reporting systematic reviews of RCTs), STROBE10 (reporting of observational studies) and MOOSE11 (reporting systematic reviews of observational studies). The National Knowledge Service in the UK has funded EQUATOR which will act as an umbrella organization to coordinate work with the mutual interest of 'improving the quality of research publications and of research itself'.12

CONSORT and TREND were primarily conceived to increase the ability of research consumers to judge the internal validity of published studies, and thereby gain an appreciation of potential sources of bias that may have had some influence on the trial result. Key threats to internal validity that were often hidden in reports of trials were subversion of randomization, blinding and response bias, which are each covered by items in CONSORT. The statement has been associated with improvements in the quality of reporting of primary research: a systematic review conducted in 2006 found that journals using CONSORT had significantly better reporting of method of sequence generation (risk ratio [RR], 1.67; 95% CI, 1.19-2.33), allocation concealment (RR, 1.66; 95% CI, 1.372.00) and overall number of CONSORT items than those not using CONSORT (standardized mean difference, 0.83; 95% CI, 0.46-1.19).13 CONSORT appeared to have had less effect on items including reporting of participant flow, and

blinding of participants or data analysts.13 Where studies

compared journals pre- and post-adoption of CONSORT an improvement was found in the description of the method of sequence generation (RR, 2.78; 95% CI, 1.78-4.33), participant flow (RR, 8.06; 95% CI, 4.10-15.83) and total CONSORT items (standardized mean difference, 3.67 items; 95% CI, 2.09-5.25) all improved after the adoption of CONSORT.13 This review is currently being updated.14

However, CONSORT was less concerned with other aspects of the conduct and reporting of intervention studies which can be equally critical to maximize the appropriateness of inferences to be drawn from such research. Within the context of randomized trials of drugs and other well-defined interventions, concerns regarding pre-specification of primary outcomes and planned sub-group analyses have been better addressed through the publication and registration of trial protocols. In the context of more complex interventions, where key concerns are to unpack the 'black box' of the inter-vention15 and to make clear assessments about the external validity of the study16 challenges remain in the reporting of randomized and non-randomized trials and therefore in the interpretation and synthesis of the evidence they provide.

Further, unlike the CONSORT statement which has seen widespread support, the TREND statement appears to be only formally supported by 15 journals. These are primarily in areas of HIV AIDS, behavioural medicine or psychology, but supporters also include the BMJ and the American Journal of Public Health, which published the statement. It is not known whether other journals informally support TREND or why the uptake has been slower than for CONSORT.

Reporting frameworks can guide primary public health research

Although statements such as CONSORT and TREND have been established to improve the reporting of research, they have also provided researchers with a framework by which to design the primary research itself. Relevant items relate primarily to the mechanics of the scientific conduct of the study; what was implemented and what results occurred and how might these be interpreted. However, although CONSORT and TREND have usefully outlined the key methodological components of randomized and non-randomized trials that need to be reported, additional information is required by users of research to determine generalizability, applicability and transferability. Public health interventions are inherently complex and as such the interpretation of findings is frequently dependent on a range of contextual factors, irrespective of the study design.

The authors of the TREND statement have encouraged feedback on their work and some useful ideas have been subsequently published: Dzewaltowski et a/.17 have suggested an increase in focus on factors that impact on external validity. This would include a particular focus on the impact of setting-level factors and intervention staff, skills, intervention costs and long-term outcomes that provide information on whether effects are sustained. Kirkwood8 has suggested the authors reflect on the applicability of the items that are based heavily on those outlined in CONSORT and suggests additional guidance for development of interventions, process and confounding variables and evaluation design. Further, one obviously useful modification to the TREND statement, in order to increase its usability by public health, health promotion, practitioners and researchers, would be to expand its focus beyond HIV/ AIDS.

In 2001, core information needs required of systematic reviews of public health interventions for public health decision-making were described18'19 by users of evidence within global public health policy and programs. The subsequent application of these guidelines in systematic reviews of public health interventions has demonstrated that many sought after dimensions have not been included in either the conduct or reporting of the primary research studies.20 These dimensions have been mapped against the key criteria of CONSORT and TREND to identify the missing components in the existing statements, and used to formulate the suggestions outlined below that are presented in order of the paper section in which they should be reported.

Reporting INTRODUCTION/BACKGROUND

Intervention model and theoretical considerations

Far too often, public health interventions are tested for effectiveness without such a model being established and/or reported, and this leads to poorly specified systematic reviews in which substantially different interventions are lumped together due to similarities in key superficial aspects such as intervention form, setting and/or target health behaviour.19 Thus, for example, it has been concluded that school based smoking education is ineffective21 and should be abandoned, even though there are promising approaches which need to be identified, developed and evaluated.22

Another area of research where interventions are often complex is health services research, where integrated care pathways can be broken down into critical components.23 Public health intervention research needs to identify similar ways of describing the intervention-process-outcome

pathway. It is important to have a clear model which sets out how the intervention is intended to bring about change in the intended outcomes, and reports of public health intervention studies should publish such a model as a requirement to allow intelligent consumption and synthesis. In the evaluation field this is known as the logic model.24,25 Ideally, the model should distinguish core and peripheral components, and emphasize the relative importance of form and function26 so that the essence of the intervention is captured allowing the integrity of the intervention to be maintained in future implementation.

This section should also provide some detail on the development and piloting of the intervention, and present any theoretical or empirical support for the intervention model and the way it is being operationalized in the intervention. There should also be discussion of potential negative outcomes of the intervention (including the timing and measurement used to assess them27), and any particular ethical concerns that there might be.28

Reporting METHODS

Study design choice

Authors should justify the choice of study design, particularly where the chosen design has been reached as a compromise between the requirements of research rigour and internal validity on the one hand, and the complexity and constraints of the research question and research setting on the other. It may be possible to identify a consistent way of identifying key strengths and weaknesses of the chosen study design, relating particularly to internal validity issues and of describing the justification for accepting design limitations. Key components of study design that should be justified are the research design, operationalization of the design including group allocation and choice of counterfactual, choice of outcome measures and measurement methods.

Reporting RESULTS

Integrity of intervention/process evaluation

The integrity (or fidelity) of the intervention is the degree to which the intervention is implemented as planned.28 Dane and Schneider29 identify aspects of integrity/fidelity as adherence, exposure, quality of delivery, participant responsiveness and program differentiation. All five dimensions should be measured in order to provide a comprehensive picture of intervention integrity.29 CONSORT does not explicitly mention fidelity and while the TREND statement recommends including issues of fidelity in the interpretation

of results, an additional guidance is required to describe how this should be achieved. This is particularly important given that only a limited number of studies disentangle the factors that ensure successful outcomes, characterize the failure to achieve success or attempt to document the steps involved in achieving successful implementation of complex interventions.30'31 Process evaluations provide a mechanism to collect and report on intervention fidelity15'28 and when reported should include information on the intensity, duration and reach of intervention components' and how these varied by sub-group. Increased access to existing checklists and resources that support the reporting of process evaluations would be beneficial.

Context, differential effects and multi-level processes

The success or failure of a complex public health intervention is often highly dependent on the social, economic and political context in which it is developed and operated.32 Knowing this type of information provides important clues about the applicability and transferability of interventions to other populations or organizational contexts. It also helps to highlight the critical success factors and the elements that are more likely to lead to program failure. Hawe33 has outlined a range of important contextual information: aspects of the host organization and staff, aspects of the system, characteristics of the target population and inter-organizational networks. These factors should be considered in reporting of public health interventions. Although these data are often collected, they are often then excluded from publications. It would be useful to highlight within publications where this information has been collected but not reported by referring readers to a website or full report.

It is well established through perspectives such as McLeroy's socio-ecological model framework34 and the WHO Ottawa Charter35 that interventions do not work in isolation and require multi-level processes. Effectiveness will vary importantly with reference to context: what works in one setting or population group would not necessarily work in another.32,36 Thus reports need to include a description of multi-level processes and a strong set of data to contextualize the study (external validity). They should also report subgroup effects by key demographic and policy factors such as SES, ethnicity and gender so that pooled analysis can be done without needing to obtain original datasets. Counter to established orthodoxy in the analysis and reporting of effectiveness trials, there may be some value in undertaking and reporting sub-group analyses even where there is insufficient statistical power within the study. This would allow pooled analysis of

these sub-group effects to be undertaken in systematic reviews. This is particularly important in considering the differential effect of interventions by socio-economic status to assess the magnitude and direction of the effects of an intervention on health inequalities.37'38 Neither CONSORT nor TREND specifically address issues pertaining to inequalities.

Sustainability

Effective public health programs or interventions should attempt to assess the degree to which they would be sustainable beyond the life of the program. In acknowledging this, it is also important to highlight that assessing the sustainability of a program and or its effects is an additional research question requiring supplementary research design development and study conduct. Shediac-Rizkallah and Bone39 present a useful framework to address sustainability. In this framework, the key aspects of program sustainability are defined as maintenance of health benefits from the program, institutionalization of a program within an organization and capacity building in the community. In reporting, authors should consider the economic and political variables, the strength of the organization, the degree to which activities are integrated into existing programs and policies and the degree of community involvement/participation.40

Reporting DISCUSSION

Many of the sections above make recommendations for reporting complex public health interventions that should also be considered in the discussion section. We would highlight specifically the opportunity that this section provides for authors to fully discuss limitations of study design and honestly appraise weaknesses. It also allows authors to interpret and discuss any sub-group effects or interactions with context that seem important. As a conclusion to the report, authors should reflect on the original model used and make statements about what worked well and what not so well so that future research or implementation can learn and further develop. An important aspect of this model could include unanticipated effects of the intervention that the investigators became aware of during the course of the study.41 Although such effects cannot be interpreted in the same way as those outcomes hypothesized and planned for in study design, they are frequently important both for future intervention development and also for the planning of future evaluation studies to capture wider potential harms and benefits, and more broadly scope the potential intervention effects.27,42

Can it possibly fit in journal word limits?

A strength of CONSORT is that it identifies the key elements required in reports of RCTs and as such has assisted authors in producing concise papers that meet the increasingly tight word length requirements of journals.14 A potential problem for the inclusion of the additional items suggested above is that it will be impossible to adequately report these items without substantial additional text, which traditional printed journals may find difficult to accommodate. Further work will be required to identify efficient, elegant and concise ways of presenting the additional data required, which may, for example, include diagrams of the hypothesized pathway from intervention to intended outcome, or describing the intervention components in a separate text box which does not contribute to overall word length.

Other possibilities may be that online editions of journals offer more space or act as channels to publish supporting documentation covering details of the suggested items. Or more radically, in parallel with established practice in clinical research, there needs to be increased funding and more opportunities for publication of intervention development research, covering phases 1—3 of the MRC Framework43 such that this work is peer-reviewed and published as a precursor to large scale effectiveness studies.

Conclusion

Guidelines for the reporting of public health intervention research are not currently available to the extent to which they are required. Given the need to increasingly understand what works, for whom, why, when and at what cost, standards for reporting on public health interventions, aligned with registries and evaluation frameworks,44 have the potential to make a significant impact on the capacity to better understand these dimensions for the benefit of public health policy.

The CONSORT and TREND statements have been important in improving the quality of published RCTs and non-randomized evaluations of interventions. This paper has suggested key elements which are additionally needed in the reporting of complex public health interventions. It is hoped that further development of these suggestions will lead to improved conduct and reporting of primary intervention studies, enrichment of systematic reviews of public health interventions and therefore greater support for informed decision-making.

Development of comprehensive guidelines will require the establishment of an international taskforce. A range of practitioners, policy-makers and researchers across different countries and organizational contexts will need to be

members of this taskforce. This will ensure that the guidelines and recommendations meet the needs of public health researchers, practitioners and funders more broadly. If we are to 'improve the quality of research publications and of research itself'12 then this is a necessary step forward.

New public health relevant Cochrane reviews

and protocols from issues 3 and 4, 2007

Reviews

• Abstinence-only programs for HIV infection prevention in high-income countries.

• Diet or exercise, or both, for weight reduction in women after childbirth.

• Home-based support for disadvantaged adult mothers.

• Home-based support for disadvantaged teenage mothers.

• Exercise for improving balance in older people.

• Interventions for the prevention of nutritional rickets in term born children.

• Interventions in the workplace to support breastfeeding for women in employment.

• Interventions to reduce harm from continued tobacco use.

• Parenting interventions for the prevention of unintentional injuries in childhood.

• School-based education programs for the prevention of child sexual abuse.

• Traditional birth attendant training for improving health behaviours and pregnancy outcomes.

Protocols

• Dietary advice in pregnancy for preventing gestational diabetes mellitus.

• Harm reduction interventions to prevent HIV/AIDS transmission in involuntary detainees.

• Indoor residual spraying for preventing malaria.

• Maternity waiting facilities for improving maternal and neonatal outcome in low-resource countries.

• Mobile phone-based interventions for smoking cessation.

• Routine iron supplementation for preventing or treating iron-deficiency anaemia in children in malaria-endemic areas.

• Routine vs. voluntary HIV testing for increasing HIV testing rates.

• Salt fluoridation for preventing dental caries.

• Social norms interventions to reduce alcohol misuse in University or College students.

• Vitamin C for preventing and treating tetanus.

• Zinc supplements for preventing otitis media.

Funding

The Cochrane Health Promotion and Public Health Field acknowledges the Victorian Health Promotion Foundation (VicHealth) and UK Department of Health for their financial support. Professor Waters is supported by a VicHealth Public Health Research Fellowship.

References

1 Ministers of Health of the Americas in Panama City. Health Agenda for the Americas 2008-2017. 2007 http://www.paho.org/English/ DD/PIN/Health_Agenda.pdf (cited 2007).

2 World Health Organization. Fifty-Eighth World Health Assembly: Ministerial Summit on Health Research (item A58/22). 2005, http://www who.int/gb/ebwha/pdf_files/WHA58/A58_22-en.pdf (cited 2007).

3 World Health Organization. Fifty-Eighth World Health Assembly: Ministerial Summit on Health Research (Resolution A58/34). 2005 http:// wwwwho.int/rpc/meetings/58th_WHA_resolution.pdf (cited 2007).

4 Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001;285(15):1987-91.

5 Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357(9263):1191 —4.

6 Des Jarlais DC, Lyles C, Crepaz N. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health 2004;94(3):361-6.

7 Victora CG, Habicht JP Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health 2004;94(3):400-5.

8 Kirkwood B. Making public health interventions more evidence based. BMJ 2004;328(7446):966—7.

9 Moher D, Cook DJ, Eastwood S et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999;354(9193):1896-900.

10 von Elm E, Altman DG, Egger M et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 2007;335(7624):806—8.

11 Stroup DF, Berlin JA, Morton SC et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283(15):2008—12.

12 National Knowledge Service. EQUATOR—Enhancing the Quality and Transparency of Health Research. 2007. http://www.nks.nhs.uk/equator. asp (cited 2007).

13 Plint AC, Moher D, Morrison A et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006;185(5):263-7.

14 CONSORT Evidence of Improvement in Reporting Quality. 2007. http:// www.consort-statement.org/index.aspx?o=1218

15 Wight D, Obasi A. Unpacking the Black Box: the importance of process data to explain outcomes. In: Stephenson J, Imrie J, Bonell C (eds). Effective Sexual Health interventions: Issues in Experimental Evaluation. Oxford: Oxford University Press, 2003:151—66.

16 Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312(7040):1215-8.

17 Dzewaltowski DA, Estabrooks PA, Klesges LM et al. TREND: an important step, but not enough. Am J Public Health 2004;94(9):1474.

18 Waters E, Doyle J, Jackson N. Evidence-based public health: improving the relevance of Cochrane Collaboration systematic reviews to global public health priorities. J Public Health Med 2003;25(3):263—6.

19 Jackson N, Waters E. Criteria for the systematic review of health promotion and public health interventions. Health Promot int 2005;20(4):367—74.

20 Summerbell CD, Waters E, Edmunds LD et al. Interventions for preventing obesity in children. Cochrane database of Syst Rev (Online). 2005;3:pCD001871.

21 Thomas R, Perera R. School-based programmes for preventing smoking. Cochrane Database Syst Rev (Online) 2006;3:pCD001293.

22 Audrey S, Cordall K, Moore L et al. The development and implementation of an intensive, peer-led intervention aimed at changing the smoking behaviour of secondary school students using their established social networks. Health EducJ 2004;63:266-84.

23 Campbell H, Hotchkiss R, Bradshaw N et al. Integrated care pathways. BMJ 1998;316(7125):133-7.

24 McLaughlin J, Jordan G. Logic models: a tool for telling your programs' performance story. EvalProgram Plan 1999;22:65-72.

25 Kellog WK. Foundation. Logic Model Development Guide: Using Logic Models to Bring Together Planning, Evaluation and Action. Battle Creek, MI: WK. Kellog Foundation, 2001.

26 Hawe P Shiell A, Riley T. Complex interventions: how "out of control" can a randomised controlled trial be? BMJ 2004; 328(7455):1561-3.

27 Cuervo LG, Clarke M. Balancing benefits and harms in health care. BMJ 2003;327(7406):65-6.

28 Jackson N, Waters E, Guidelines for Systematic Reviews in Health Promotion and Public Health Taskforce. Guidelines for Systematic Reviews in Health Promotion and Public Health. Melbourne: Cochrane Health Promotion and Public Health Field, 2005.

29 Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clin

Psychol Rev 1998;18(1):23-45.

30 Steckler A, Linnan L (eds). Process Evaluation for Public Health interventions and Research. USA: Jossey-Bass, 2002.

31 Green J, Tones K. Towards a secure evidence base for health promotion. J Public Health Med 1999;21(2):133-9.

32 Frommer M, Rychetnik L. From evidence-based medicine to evidence-based public health. In: Lin V, Gibson B (eds). Evidence-Based Health Policy: Problems and Possibilities. Melbourne: Oxford University Press, 2003.

33 Hawe P, Shiell A, Riley T et al. Methods for exploring implementation variation and local context within a cluster randomised

community intervention trial. J Epidemiol Commun Health 2004; 58(9):788—93.

34 McLeroy KR, Bibeau D, Steckler A et al. An ecological perspective on health promotion programs. Health Educ Q 1988; 15(4):351 —77.

35 World Health Organisation. Ottawa Charter for Health Promotion. Ottawa: World Health Organisation, 1986.

36 Goodman K. Ethics and Evidence-Based Medicine; Fallibility and Responsibility in Clinical Science. Cambridge: Cambridge University Press, 2003.

37 World Health Organisation. World Report on Knowledge for Better Health. Geneva: World Health Organisation, 2004.

38 Correa-Velez I, Gifford S. When the right to be counted doesn't count: the politics and challenges of researching the health of asylum seekers. Critical Public Health 2007;17(3):259 —67.

39 Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and

future directions for research, practice and policy. Health Educ Res 1998;13(1):87—108.

40 Bossert TJ. Can they get along without us? Sustainability of donor-supported health projects in Central America and Africa. Soc Sci Med 1990;30(9):1015-23.

41 Rychetnik L, Hawe P, Waters E et al. A glossary for evidence based public health. J Epidemiol Commun Health 2004;58(7):538-45.

42 Ioannidis JP, Evans SJ, Gotzsche PC et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141(10):781 -8.

43 Campbell M, Fitzpatrick R, Haines A et al. Framework for design and evaluation of complex interventions to improve health. BMJ

2000;321(7262):694-6.

44 Waters E, Priest N, Armstrong R et al. The role of a prospective public health intervention study register in building public health evidence: proposal for content and use. J Public Health 2007;29(3):322-7.