Scholarly article on topic 'Effective in-service training design and delivery: evidence from an integrative literature review'

Effective in-service training design and delivery: evidence from an integrative literature review Academic research paper on "Educational sciences"

Share paper
Academic journal
Human Resources for Health
OECD Field of science

Academic research paper on topic "Effective in-service training design and delivery: evidence from an integrative literature review"


Effective in-service training design and delivery: evidence from an integrative literature review

Julia Bluestone1*1, Peter Johnson11, Judith Fullerton2, Catherine Carr1+, Jessica Alderman3 and James BonTempo1

Background: In-service training represents a significant financial investment for supporting continued competence of the health care workforce. An integrative review of the education and training literature was conducted to identify effective training approaches for health worker continuing professional education (CPE) and what evidence exists of outcomes derived from CPE.

Methods: A literature review was conducted from multiple databases including PubMed, the Cochrane Library and Cumulative Index to Nursing and Allied Health Literature (CINAHL) between May and June 2011. The initial review of titles and abstracts produced 244 results. Articles selected for analysis after two quality reviews consisted of systematic reviews, randomized controlled trials (RCTs) and programme evaluations published in peer-reviewed journals from 2000 to 2011 in the English language. The articles analysed included 37 systematic reviews and 32 RCTs. The research questions focused on the evidence supporting educational techniques, frequency, setting and media used to deliver instruction for continuing health professional education.

Results: The evidence suggests the use of multiple techniques that allow for interaction and enable learners to process and apply information. Case-based learning, clinical simulations, practice and feedback are identified as effective educational techniques. Didactic techniques that involve passive instruction, such as reading or lecture, have been found to have little or no impact on learning outcomes. Repetitive interventions, rather than single interventions, were shown to be superior for learning outcomes. Settings similar to the workplace improved skill acquisition and performance. Computer-based learning can be equally or more effective than live instruction and more cost efficient if effective techniques are used. Effective techniques can lead to improvements in knowledge and skill outcomes and clinical practice behaviours, but there is less evidence directly linking CPE to improved clinical outcomes. Very limited quality data are available from low- to middle-income countries.

Conclusions: Educational techniques are critical to learning outcomes. Targeted, repetitive interventions can result in better learning outcomes. Setting should be selected to support relevant and realistic practice and increase efficiency. Media should be selected based on the potential to support effective educational techniques and efficiency of instruction. CPE can lead to improved learning outcomes if effective techniques are used. Limited data indicate that there may also be an effect on improving clinical practice behaviours. The research agenda calls for well-constructed evaluations of culturally appropriate combinations of technique, setting, frequency and media, developed for and tested among all levels of health workers in low- and middle-income countries.

Keywords: In-service training, Continuing professional education, Continuing medical education, Continuing professional development

* Correspondence: tEqualcontributors

1Jhpiego Corporation, 1615 Thames Street, Baltimore, MD 21231, USA Full list of author information is available at the end of the article


Bio Med Central

© 2013 Bluestone et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.Org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


The need to increase the effectiveness and efficiency of both pre-service education and continuing professional education (CPE) (in-service training) for the health workforce has never been greater. Decreasing global resources and a pervasive critical shortage of skilled health workers are paralleled by an explosion in the increase of and access to information. Universities and educational institutions are rapidly integrating different approaches for learning that move beyond the classroom [1]. The opportunities exist both in initial health professional education and CPE to expand education and training approaches beyond classroom-based settings.

An integrative review was designed to identify and review the evidence addressing best practices in the design and delivery of in-service training interventions. The use of an integrative review expands the variety of research designs that can be incorporated within a review's inclusion criteria and allows the incorporation of both qualitative and quantitative information [2]. Five questions were formulated based on a conceptual model of CPE developed by the Johns Hopkins University Evidence-Based Practice Center (JHU EPC) for an earlier systematic review of continuing medical education (CME) [3]. We asked whether: 1. particular educational techniques, 2. frequency of instruction (single or repetitive), 3. setting where instruction occurs, or 4. media used to deliver the instruction make a difference in learning outcomes; and, 5. if there was any evidence regarding the desired outcomes, such as improvements in knowledge, skills or changes in clinical practice behaviours, which could be derived from CPE, using any mixture of technique, media or frequency.


Inclusion/exclusion criteria

Articles were included in this review if they addressed any type of health worker pre-service or CPE event, and included an analysis of the short-term evaluation and/or assessment of the longer-term outcomes of the training. We included only those articles published in English language literature. These criteria gave priority to articles that used higher-order research methods, specifically meta-analyses or systematic reviews and evaluations that employed experimental designs. Articles excluded from analysis were observational studies, qualitative studies, editorial commentary, letters and book chapters.

Search strategy

A research assistant searched the electronic, peer-reviewed literature between May and June 2011. The search was conducted on studies published in the English language from 2000 to 2011. Multiple databases including PubMed, the Cochrane Library and Cumulative Index to Nursing and Allied Health Literature (CINAHL) were utilized in

the search. Medical subject headings (MeSH) and key search terms are presented below in Table 1.

Study type, quality assessment and grade

An initial review of titles and abstracts produced 244 results. We identified the strongest studies available, using a range of criteria tailored to the review methodology. Initial selection criteria were developed by a panel of experts. Grading and inclusion criteria are presented in Table 2. The grading criteria were adapted from the Oxford Centre for Evidence-Based Medicine (OCEMB) levels of evidence model [4]. Grading of studies included within systematic reviews was reported by authors of those reviews and was not further assessed in this integrative review. Therefore, reference to quality of studies in our report refers to those a priori judgments. Only tier 1 articles (grades 1 and 2) were included in our analysis.

After prioritization of the articles, 163 tier 1 articles were assessed by a senior public health professional to determine topical relevance, study type and grade. A total of 61 tier 1 studies were selected to be included in the analysis following this second review. An additional hand search of the reference lists cited in published studies was conducted for topics that were underrepresented, specifically on the frequency and setting of educational activities. This search added eight articles for a total of 69 studies, including 37 systematic reviews and 32 randomized controlled trials (RCTs), see inclusion process for articles included in analysis, Figure 1.

A data extraction spreadsheet was developed, following the model offered in the Best Evidence in Medical Education (BEME) group series [5] and the conceptual model and definition of terms offered by Marinopoulos et al. in the JHU EPC earlier review of CME [3]. Categorization decisions were necessary in cases when the use of terminology was inconsistent with the Marinopoulos et al. definitions of terms for CPE [3]. For example, an article that analysed 'distance learning as a technique and used the computer as the medium to deliver an interactive e-learning course was coded and categorized as an 'interactive' technique delivered via 'computer' as the medium of instruction. See illustration of categorization terminology in panels A, B, and C, Figure 2, for an illustration of how terminology was used to categorize and organize articles for analysis.


Selected articles that best represent common findings and outcomes (effects) of CPE are discussed in the results and discussion sections; the related tables present all the articles analysed and categorized for that topic, and each article is included only once. Relevant information obtained from educational psychology literature is referenced in the discussion.

Table 1 Medical subject headings (MeSH) and key search terms

Group-based education Facility-based education On-the-job education Group-based training Facility-based training On-the-job training Point-of-care training Mobile technologies

Asynchronous distance learning Synchronous distance learning Online learning Distance learning Continuing medicaleducation Continuing nursing education

Nursing education Medical education Teaching methods Health care professionals Education methods Continuing education methods Nursing education methods Medical education methods


The articles or studies that specifically addressed educational techniques are summarized in Table 3. Technique refers to the educational methods used in the instruction. Technique descriptions are based on the Marinopoulos et al. definitions of terms [6] and reflect the approaches defined in the articles analysed.

Case-based: use of created or actual clinical cases that present materials and questions

Though case-based learning was not specifically compared with other techniques in the literature reviewed, it was often noted as a method in articles that discussed interactive techniques. Case-based learning was also noted as a technique used for computer-delivered CPE courses. Triola et al. compared types of media utilized for case-based learning and found positive learning outcomes both with the use of a live standardized patient and a computer-based virtual patient [7].

Didactic/lecture: presenting knowledge content; facilitator determines content, organization and pace Lecture was often referred to in the literature as traditional instruction, lecture-based or didactic teaching. Didactic instruction was not found to be an effective educational technique compared with other methods. Two studies [8,9] found no statistical difference in learning outcomes, and three studies found didactic to be less effective than other techniques [10-12]. Reynolds et al. compared didactic

Table 2 Grading criteria

Design Type of groups

Meta-analysis or systematic review Experimental



instruction with simulation. The study was limited by small sample size (n = 50), but still demonstrated that the simulation group had a significantly higher mean post-test score (P <0.01) and overall higher learner satisfaction [12].

Several systemic reviews that compared didactic instruction to a wide variety of teaching approaches also identified didactic instruction as a less effective educational technique [13-15].

Feedback: providing information to the learner about performance

Multiple articles identified feedback as important for outcomes [16-18]. Herbert et al. compared individualized feedback in the form of a graphic (a prescribing portrait based on personal history of drug-prescribing practices) to small group discussion of the same material and found that both the feedback and the live, interactive session were somewhat effective at changing physician's prescribing behaviours [16]. The Issenberg et al. systematic review of simulation identified practice and feedback as key for effective skill development [17]. A Cochrane review of the evidence to support CPE suggested the importance of feedback and instructor interaction in improving learning outcomes [18].

Games: competitive game with preset rules

The use of games as an instructional technology was addressed in one rigorous systematic review. The authors found only a limited number of studies, which were of low

Literature grade Tier

Between subjects (experimentaland control) 2

Within subjects (crossover) 2

Non-equivalent controlgroup 3

Repeated measures 3

Comparison group 4

Pre-test/post-test 4

Post-test only 5

NA not applicable.

Journal Database

Titles and Abstract Review

Excluded for data quality control

First Quality Review

133 30

Grade One Grade Two

Excluded for data quality control

Second Quality Review

First Tier

Added after hand review

Additional "Time and Setting" Hand Search

69 First Tier

Systematic Reviews

Randomized Control Trials

Figure 1 Inclusion process for articles included in the analysis.

Figure 2 Illustration of categorization terminology in panels a-c.

to moderate methodological quality and offered inconsistent results. Three of the five RCTs included in the review suggested that educational games could have a positive effect on increasing medical student knowledge and that they include interaction and allow for feedback [19].

Interactive: provide for interaction between the learner and facilitator

Five articles specifically compared interactive CPE to other educational techniques. De Lorenzo and Abbot found interactive techniques to be moderately superior for knowledge outcomes than didactic lecture [10]. Two other studies found interactive techniques were more effective when feedback from chart audits was added to the intervention [16,20].

Three systematic reviews and one meta-analysis specifically noted the importance of learner interactivity or engagement in learning in achieving positive learning outcomes [21-24] (refer to summary of articles focused on outcomes).

Point-of-care (POC): information provided as needed, at the point of clinical care

Two articles and one systematic review specifically addressed point-of-care (POC) as a technique. The systematic review included three studies and concluded that while the findings were weak, they did indicate that POC led to improved knowledge and confidence [25]. In an examination of media, Leung et al. determined that handheld devices were more effective than print-based, POC support, although outcome measures were self-reported behaviours [26]. You et al. found improved performance on a procedure among surgical residents who


Study design


Ak¡ E et al. 2010

Systematic review: five articles reviewed to determine the effectiveness of educational gaming on learning

Mostly medical students

J et al. 2010 Systematic review: 45 articles included for

review, only three related to POC support, included qualitative and quantitative data

Nurses in developing countries

Bruppacher H et al. 2010

Prospective, single-blinded RCT to determine if simulation or interactive techniques are better for teaching weaning a patient from anaesthesia

Anaesthesiology trainees, post-graduate year 4

I = 10, C = 10

Country: China

Daniels K et al. 2010 Prospective RCT to determine if simulation Residents and labour and is more effective than didactic in obstetric delivery nurses emergency management _ _

Country: USA

De Lorenzo R and RCT to determine if the adult learning Army medic students

Abbott C 2004 model improves student learning in terms _ _ C-69

of cognitive performance and perception n_ ' ~ ' ~

of proficiency in military medic training Country: USA

Harder BN 2010 Systematic review: 23 articles reviewed to Health professionals

evaluate the use of clinical simulation in health care education


Key findings

Technique: educational games Media: multiple Frequency: NR Technique: didactic vs POC Media: computer-based vs live Frequency: NR

Technique: simulation vs interactive

Media: live

Frequency: single

ntervention group received simulation-based training; control group received an interactive seminar.

Technique: simulation vs interactive

Media: live Frequency: single

ntervention group received simulation-based training; control group received an interactive seminar.

Technique: interactive vs didactic Media: live

Frequency: single

ntervention group emphasized the principles of adult learning including small group interactive approach, self-directed study, multimedia didactics and intensive integrated practice of psychomotor skills; control group received a traditional, lecture-based course.

Technique: simulation

Media: multiple

Frequency: single

Findings in three of the five RCTs suggested but did not confirm a positive effect of the games on medical students' knowledge.

POC findings: studies were weak but indicated knowledge improved and increased rapport in trusting personal judgment.

The simulation group scored significantly higher than the seminar group at both post-test and retention test. Clinica decision-making/psychomotor skills can be acquired via simulation.

Simulation-trained teams had superior performance scores when tested in a labour and delivery drill. In an academic training programme, didactic and simulation-trained groups showed equal results on written test scores.

The adult learning model offered only a modest improvement in cognitive evaluation scores over traditional teaching. Additionally, students in the traditional teaching model assessed themselves as proficient more frequently than instructors, whereas instructor and student perception of proficiency were more closely matched in the adult learning model.

Inconclusive evidence about the use of simulation due to a low number of studies. However, the use of simulation, as opposed to other education and training methods (motor skills laboratory sessions with task trainers, computer-based instruction and lecture classes), increased students' clinica skills in the majority of studies.

Herbert C et al. 2004

Issenberg S et al. 2005

Lamb D 2007

Laprise R et al. 2009

Lin C et al. 2010

RCT to assess the impact of individualized feedback and live, interactive group education on prescriptive practices


Systematic review: 109 studies reviewed to determine the use of high-fidelity medicalsimulations that lead to most effective learning

Cluster randomized trialof 122 general practitioners to determine if chart audits and feedback reminders after a CME event lead to better adherence to clinical guidelines

RCT to determine if peer-tutored, PBL is preferable to didactic-based instruction for teaching nursing ethics

Technique: audit and feedback vs interactive plus audit and feedback vs interactive session only vs nothing

11 =48, audit and feedback only; Media: live

12 = 47, interactive module only;

13 = 49, interactive plus audit and feedback; C = 56, nothing

4,394 charts reviewed Country: Canada Health professionals

Literature review: nine articles reviewed to determine effectiveness of experiential (focused on simulations) learning

Health professionals


Nursing students I = 72, C = 70 Country: Taiwan

Frequency: single

Technique: simulation Media: multiple

Frequency: both single and multiple

Technique: simulation Media: multiple

Frequency: both single and multiple

n = 122, I = 61, C = 61

Chart audit of 2,344 consenting patient charts

Country: Canada

Technique: audit and feedback plus interactive vs interactive only

Media: live

Frequency: single vs multiple

Intervention group and control group received the same CME intervention, a 2-hour live, interactive workshop. The intervention group also received six monthly follow-up visits from a nurse that included chart screening, audits and feedback, and a print-based checklist distribution and print summary of expert recommendations.

Technique: PBL vs didactic

Media: live

Frequency: single

Increase in prescribing preference for correct drug class in module and "prescribing portraits" (graphic comparisons between individual, group and evidence based prescribing practices) group. Evidence-based educationalinterventions combining personalized prescribing feedback with interactive group discussion can lead to modest but meaningfulchanges in physician prescribing.

The weight of the best available evidence suggests that high-fidelity medical simulations facilitate learning under the right conditions. These conditions include: providing feedback, repetitive practice, curriculum integration, range of difficulty, multiple learning strategies, capture clinical variation, controlled environment, individualized learning, defined outcomes and simulator validity.

None of the studies showed conclusively that simulated learning improves patient outcome; however, evidence suggests human patient simulators to be advantageous over other modalities. They have been proven to be at least as effective as traditionalteaching by didactic methods. Both human patient simulators (models) and computer-simulations may be effective.

This study demonstrated significantly improved adherence in the intervention group using chart audits vs CME alone. The magnitude of the difference observed between the two groups in absolute pre-post intervention change is consistent with previous studies on the effectiveness of chart prompting in preventive care.

Peer-tutored, PBL was shown to be more effective than conventional lecture-type teaching. Peer-tutored, PBL has the potential to enhance the efficacy of teaching nursing

McGaghie W et al. 2009a Systematic review: nine of the JHU EPC Health professionals systematic review articles reviewed to determine the effectiveness of simulation methods in medical education outside ofCME

Merien A et al. 2010 Systematic review: eight articles reviewed Health professionals

to determine the effectiveness of team-based training for obstetric care

Murad MH et al. 2010 Systematic review: 59 articles (enrolled Health professionals 8,011 learners) reviewed to determine effectiveness of self-directed learning

Perry M et al. 2011 Systematic review: six articles representing Health professionals

five studies were reviewed to determine the effect of educational interventions in primary dementia care

Reynolds A et al. 2010

RCT to compare students' knowledge using either simulation or didactic lecture

Midwifery students I = 26, C = 24 Country: Portugal

ntervention group received PBL technique; control group received didactic-based instruction.

Technique: simulation Media: multiple

Frequency: both single and multiple

Technique: team-based Media: live Frequency: NR

Technique: self-directed Media: multiple Frequency: NR

Technique: multiple Media: multiple

Frequency: both single and multiple

Technique: simulation vs didactic Media: live Frequency: single

ntervention group received simulation-based training; control group received didactic lectures with print visuals.

ethics in situations in which there are personnel and resource constraints.

Due to a low number of studies, evidence on simulation methods is inconclusive. However, the direction of evidence points to the effectiveness of simulation training, especially for psychomotor and communication skills. Data analysis revealed a highly significant dose-response' relationship among practice and achievement, with more practice producing higher outcome gains.

Due to a low number of studies, evidence on teamwork training in simulation is inconclusive. However, introduction of multidisciplinary teamwork training with integrated acute obstetric training interventions in a simulation setting is potentially effective in the prevention of errors, thus improving patient safety in acute obstetric emergencies.

Moderate-quality evidence suggests that self-directed learning in health professions education is associated with moderate improvement in the knowledge domain compared with traditional teaching methods, and may be as effective in the skills and attitudes domains.

nteractive workshops and decision support systems led to increased detection rates. Evidence shows moderate improvements in knowledge and techniques that required active participation tended to improve detection rates.

A significantly higher short-term reinforcement of knowledge and greater learner satisfaction was obtained using simulation-based training compared to image-based lectures when teaching routine management of normal delivery and resolution of shoulder dystocia to midwives in training.

OJ tO n>

Smits P et al. 2003 RCT to compare effectiveness of PBL vs Post-graduate medical trainees

= 59, C = 59

Country: the Netherlands

didactic for management of mental 1-59 C - 59

health problems - ' -

Steadman R et al. 2006 RCT to determine if simulation is better 4th year medical students than PBL for teaching assessment and management skills - ' -

Country: USA

Sturm L et al. 2008 Systematic review: 11 articles reviewed to Surgeons

determine if skills acquired by simulation-based training transfer to the operative setting

Werb S and Matear D 2004 Systematic review: three systematic Allied health professionals

reviews and nine original research articles reviewed to examine evidence-based clinical teaching and faculty continuing education

White M et al. 2004 RCT to investigate effectiveness of PBL vs Physicians

didactic for asthma management

= 23, C = 29 Country: Canada

Young J and Ward J 2002 Randomized trial to determine the effect Family physicians of self-directed (distance) learning on

knowledge, attitudes and practices related - ' -

to smoking cessation Country: Australia

Technique: PBL vs didactic Media: live Frequency: single

ntervention group received PBL technique; control group received didactic-based instruction.

Technique: simulation vs PBL Media: live Frequency: single

ntervention group received simulation-based teaching; control group received PBL.

Technique: simulation

Media: multiple

Frequency: both single and multiple

Technique: PBL Media: multiple

Frequency: both single and multiple

Technique: PBL vs didactic Media: live Frequency: single

ntervention group received PBL technique; control group received didactic-based instruction.

Technique: self-directed vs reading Media: print Frequency: single

ntervention group received a self-directed learning module; control group received guidelines only.

The study found that both PBL and didactic-based instruction were effective, but had no statistical difference. The PBL programme appeared to be more effective than the lecture-based programme in improving performance, but received less favourable evaluations.

Simulation-based teaching was superior to PBL for the acquisition of critical assessment and management skills.

Due to limited quality and methodology and a lack of relevant studies, a weak conclusion can be made supporting the transfer of skills developed in simulation to the operative setting. Evidence from one study showed better performance for participants who received simulation-based training before undergoing patient-based assessment than their counterparts who did not receive previous simulation training.

PBL and evidence-based health care interventions were effective in increasing students' knowledge of medical topics and their ability to search, evaluate and appraise medical literature. Dental students in a PBL curriculum, emphasizing evidence-based practices, scored higher on the Nationa Dentistry Boards, Part I, than students in traditional curricula.

There was no significant difference in knowledge gained or satisfaction with the facilitator between the PBL group and the lecture-based group. The PBL group rated the educational value higher than the didactic group.

Modest changes from baseline to post-test for both the distance learning group and self-directed group suggest a lack of significant evidence to support a distance or self-directed approach to address changes in practice.

Yuan H et al. 2008

Zurovac D et al. 2011

Systematic review: 10 studies reviewed to Nursing students determine the evidence to support PBL

Cluster RCT at 107 ruralhealth facilities to Health professionals

determine if text-message reminders would improve provider adherence to national malaria treatment guidelines

119 health workers

Case-management practices were assessed for 2,269 children who needed treatment

I = 1,157, C= 1,112

Country: Kenya

Technique: PBL Media: multiple

Frequency: both single and multiple

Technique: reminders Media: mobile phone Frequency: repetitive

Intervention group received repetitive text messages over a 6-month period; control group received nothing.

Inconclusive evidence to support PBL. While severalstudies showed increased reported self-confidence in ability to make decisions, and several showed increased skills in critical thinking questions from the PBL group, overallfindings were inconclusive due to a lack of quality studies.

The use of mobile technology showed significant improvement in case management practice for paediatric malaria cases among physicians with repetitive text-message reminders compared to control group.

aJHU EPC systematic review. C Control, CME Continuing medical education, I Intervention, JHU EPC Johns Hopkins University Evidence-Based Practice Center, NR Not reported, PBL problem-based learning, POC point-of-care, RCT randomized controlled trial.

received POC mentoring via a video using a mobile device, compared with those who received only didactic instruction [27].

Problem-based learning (PBL): present a case, assign information-seeking tasks and answer questions about the case; can be facilitated or non-facilitated

Four articles specifically compared problem-based learning (PBL) to other methods. One study identified PBL as slightly better [11], and two studies indicated it to be relatively equal to didactic instruction [8,9]. A systematic review of 10 studies on PBL reported inconclusive evidence to support the approach, although several studies reported increased critical thinking skills and confidence in making decisions [28].

Reminders: provision of reminders

The Zurovac et al. study conducted in Kenya found that using mobile devices for repetitive reminders resulted in significant improvement in health care provider's case management of paediatric malaria, and these gains were retained over a 6-month period [29]. Intention-to-treat analysis showed that correct management improved by 23.7% (95% confidence interval (CI) 7.6 to 40.0, P <0.01) immediately after intervention and by 24.5% (95% CI 8.1 to 41.0, P <0.01) 6 months later, compared with the control group [29]. Reminders were also noted as an effective technique by two of the systematic reviews [13,14].

Self-directed: completed independently by the learner based on learning needs

This term was difficult to extract for analysis due to widely varying terminology. Some authors used the term 'distance learning', and some used it to define the medium of delivery, rather than technique. This analysis specifically discusses articles that were consistent with the description for self-directed learning, even if the authors used different terminology.

A recent systematic review identified that moderate-quality evidence suggests a slight increase in knowledge domain compared with traditional teaching, but notes that this may be due to the increased exposure to content [30]. One RCT found modest improvements in knowledge using a self-directed approach, but noted it was less effective at impacting attitudes or readiness to change [31].

Multiple studies focused on use of the computer as the medium to deliver instruction and noted that self-directed instruction was equally (or more) effective as instructor-led didactic or interactive instruction and potentially more efficient.

Simulation may include models, devices, standardized patients, virtual environments, social or clinical situations that simulate problems, events or conditions experienced in professional encounters [17]. Simulation was noted as an

effective technique for promotion of learning outcomes across the systematic reviews, particularly for the development of psychomotor and clinical decision-making skills. The systematic reviews all highlighted inconclusive and weak methodology in the studies reviewed, but noted sufficient evidence existed to support simulation as useful for psychomotor and communication skill development [32-34] and to facilitate learning [35]. The systematic review by Lamb suggests that patient simulators, whether computer or anatomic models, are one of the more effective forms of simulations [36].

Outcomes of the four separate RCTs indicated simulation was better than the techniques to which they were compared, including interactive [37,38], didactic [12] and problem-based approaches [35]. A study by Daniels et al. found that although knowledge outcomes were similar between the interactive and simulation groups, the simulation team performance in a labour and delivery clinical drill was significantly higher for both shoulder dystocia (11.75 versus 6.88, P <0.01) and eclampsia (13.25 versus 11.38, P = 0.032) at 1 month post-intervention [38].

Simulation was also found to be useful for identifying additional learning gaps, such as a drill on the task of mixing magnesium sulfate for administration [39]. A systematic review focused on resuscitation training identified simulation as an effective technique, regardless of media or setting used to deliver it [40].

Team-based: providing interventions for teams that provide care together

Articles discussed here focused on the technique of providing training to co-workers engaged as learning teams. One systematic review of eight studies found that there is limited and inconclusive evidence to support team-based training [41]. Two of the articles reporting on the same CPE study did not identify any improvements in performance or knowledge acquisition with the addition of using a team-based approach [39,42].


This review included consideration of frequency, comparing single versus repetitive exposure. The findings regarding frequency are summarized in Table 4.

The three articles focused on frequency all support the use of repetitive interventions. These studies evaluated repetition using the Spaced Education platform (now called Qstream), an Internet-based medium that uses repeated questions and targeted feedback. The evidence from these three articles demonstrated that repetitive, time-spaced education exposures resulted in better knowledge outcomes, better retention and better clinical decisions compared with single interventions and live instruction [43-45].

The use of repetitive or multiple exposures is supported in other systematic reviews of the literature, as well as one


Study design


Kerfoot BP et al. 2007

RCT to determine if spacing principles can improve acquisition and retention of medical knowledge

Five cohorts with 76 to 80 urology residents in each cohort

Of 537 participants, 400 (74%) completed the online staggered tests and 515 (96%) completed the n-Service Examination

Cohort 1 = bolus, single intervention; Cohort 2 = multiple, spaced intervention

Country: USA and Canada

Kerfoot BP et al. 2009 RCT to determine if Spaced Education Urologists and urology residents

is an effective form of CME _ . .

Completed by 71% or urologists

and 83% of residents

Cohort 1 =80 urologists, 160 residents, completed by 196; Cohort 2 = 80 urologists, 160 residents, completed by 182

Country: USA (March to July 2007)


Key findings

Frequency: multiple vs single Technique: self-directed

Media: Internet-based

Conclusive evidence to support repetitive, spaced education in online learning, since residents in the spaced education cohort demonstrated significantly greater online test scores than those in the bolus cohort. The scores for the spaced cohort remained stable with no overtime, while test scores in the bolus cohort demonstrated a significant ¡near decrease.

Cohort 1 received bolus education of 96 study questions (June 2005); Cohort 2 received daily emails over 27 weeks (June to December 2005), each with one to two questions in spaced pattern. In November 2005, all participants completed the urology exam. Participants were randomized to five cohorts and completed a 32-item online test at staggered time points (1 to 14 weeks) after completion of Spaced Education.

Frequency: multiple vs single Technique: self-directed

Media: Internet-based

Conclusive evidence to support the use of ISE programmes. Knowledge scores of ISE intervention were statistically significantly higher than those of the control bolus method.

A total of 160 urologists and 320 urology residents were randomized to one of two cohorts. Participants were stratified by training level (urologist in practice vs resident) and urology training year (residents only) and were block randomized (block size = 8) to one of two cohorts. Participants in Cohort 1 received the 3-cycle ISE course on the HP CPGs, with 24 control items on the SIA CPGs in cycle 3. Participants in Cohort 2 received the 3-cycle ISE course on SIA CPGs, with 24 control items on HP CPGs in cycle 3. The trial was structured in this manner to allow the topic-specific learning gains from the ISE courses to be identified in cycle 3. Since the 24 items are

presented simultaneously to both cohorts in ^

cycle 3, the learning gains of physicians who (q

had completed two cycles of the ISE ™

programme could be directly compared with those physicians who were presented with the

material for the first time (controls). ct>

Kerfoot BP et al. 2010

RCT to determine if Spaced Education can effect knowledge transfer and the ability to make diagnostic decisions

Urology residents Frequency: multiple vs single

Cohort 1 = 164; Cohort 2 = 194 Technique: self-directed

Country: USA (June 2007 to June 2008) Media: Internet-based

Transfer and retention of diagnostic skills between Spaced Education vs bolus, WBT

Allresidents were sent both spaced education and WBT, but the set of topics delivered by each method varied by cohort. Residents in Cohort 1 received three cycles of spaced education on prostate-testis (weeks 1 to 4, 5 to 8, and 13 to 16) and three WBT modules on bladder-kidney (weeks 14 to 16). Residents in Cohort 2 received three cycles of spaced education on bladder-kidney (weeks 1 to 4, 5 to 8, and 13 to 16) and three WBT modules on prostate-testis (weeks 14 to 16). The spaced education items were delivered each weekday through emails containing one question/answer, and the spaced education materialwas distributed in three cycles or repetitions to take advantage of the spacing effect. The WBT used the identicalcontent and delivery system, with the questions aggregated into three 20-question modules delivered through separate emails in week 14. The trialwas specifically structured to ensure that within a given set of topics (bladder-kidney or prostate-testis) the only difference between intervention cohorts was the spacing of content.

Conclusive evidence to support spaced, web-based education compared to WBT. Spaced education demonstrated a statically significant increase in knowledge and long-term retention of knowledge compared with bolus web-based modules that delivered the same content of histopathology diagnostic skills.

CME Continuing medical education, CPG Clinical practice guideline, HP Haematuria and priapism, ISE Interactive spaced education, RCT Randomized controlled trial, SIA Staghorn calculi, infertility, and antibiotic use, WBT Web-based teaching.

RCT conducted in Kenya that used repeated text reminders and resulted in a significant improvement in adherence to malaria treatment protocols [29].


Setting is the physical location within which the instruction occurs. We identified three articles that looked specifically at the training setting. The findings regarding setting are summarized in Table 5. Two of them stemmed from the same intervention. Crofts et al. specifically addressed the impact of setting and technique (team-based training) on knowledge acquisition and found no significant difference in the post-score based on the setting [42]. A systematic review of eight articles evaluating the effectiveness of team-based training for obstetric care did not find significant differences in learning outcomes between a simulation centre and a clinical setting [41].

Coomarasamy and Khan conducted a systematic review and compared classroom or stand-alone versus clinically integrated teaching for evidence-based medicine (EBM). Their review identified that classroom teaching improved knowledge, but not skills, attitudes or behaviour outcomes; whereas clinically integrated teaching improved all outcomes [46]. This finding was supported by the Hamilton systematic review of CPE, which suggests that teaching in a clinical setting or simulation setting is more effective (Table 1), as well as the Raza et al. systematic review of 23 studies to evaluate stand-alone versus clinically integrated teaching. This review suggested that clinically integrated teaching improved skills, attitudes and behaviour, not just knowledge [18].

Media refers to the means used to deliver the curriculum. The majority of RCTs compared self-paced or individual instruction delivered via computer versus live, group-based instruction. The findings regarding media are summarized in Table 6.

Live versus computer-based

Live instruction was found to be somewhat effective at improving knowledge, but less so for changing clinical practice behaviours. When comparing live to computer-based instruction, a frequent finding was that computer-based instruction led to either equal or slightly better knowledge performance on post-tests than live instruction. One of the few to identify a significant difference in outcomes, Harrington and Walker found the computer-based group outperformed the instructor-led group on the knowledge post-test and that participants in the computer-based group, on average, spent less time completing the training than participants in the instructor-led group [47].

Systematic reviews indicate that the evidence supports the use of computer-delivered instruction for knowledge

and attitudes; however, insufficient evidence exists to support its use in the attempt to change practice behaviours. The Raza Cochrane systematic review identified 16 randomized trials that evaluated the effectiveness of Internet-based education used to deliver CPE to practicing health care professionals. Six studies showed a positive change in participants' knowledge, and three studies showed a change in practice in comparison with traditional formats [18]. One systematic review noted the importance of interactivity, independent of media, in achieving an impact on clinical practice behaviours [48].


One article assessed the use of animations against audio instructions in cardiopulmonary resuscitation (CPR) using a mobile phone and found the group that had audiovisual animations performed better than the group that received live instruction over the phone in performing CPR; however, neither group was able to perform the psychomotor skill correctly [49]. Leung et al. found providing POC decision support via a mobile device resulted in slightly better self-reporting on outcome measures compared with print-based job aids, but that both the print and mobile groups showed improvements in use of evidence-based decision-making [26].

The systematic review of print-based materials conducted by Farmer et al. did not find sufficient evidence to support the use of print media to change clinical practice behaviours [50]. A comparison of the use of print-based guidelines to a live, interactive workshop indicated that those who completed live instruction were slightly better able to identify patients at high risk of an asthma attack. However, neither intervention resulted in changed practice behaviours related to treatment plans [51].

Multiple systematic reviews caution against the use of print only media, concluding that live instruction is preferable to print only. Another consistent theme was support for the use of multimedia in CPE interventions.


Outcomes are the consequences of a training intervention. This literature review focuses on changes in knowledge, attitudes, psychomotor, clinical decision-making or communication skills, and effects on practice behaviours and clinical outcomes. All of the articles that focused on outcomes were systematic reviews of the literature and are summarized in Table 7.

The weight of the evidence across several studies indicated that CPE could effectively address knowledge outcomes, although several studies used weaker methodological approaches. Specifically, computer-based instruction was found to be equally or more effective than

Table 5 Summary of articles focused on setting


Study design



Key findings

Coomarasamy A and Khan K 2004 (link to the follow-up study, Raza A et al. 2009)

Crofts J et al. 2007

Systematic review: 23 articles reviewed to determine the effect of stand-alone compared to clinically integrated teaching in EBM

Prospective RCT to explore if knowledge acquisition is influenced by training setting or teamwork training

Post-graduate physicians, allied health professionals

Senior doctors, junior and senior midwives

Totalof 140 participants; interdisciplinary teams of four or six in four blocks

Country: UK

Ellis D et al.

Same study design as Crofts et al. 2007

Same participants as Crofts et al. 2007

Technique: multiple, focus on case-based Media: live

Frequency: both single and multiple

Technique: team-based vs interactive

Media: live

Frequency: single

I1 = 1-day interactive at hospital (no team-based training); I2 = 1-day interactive at simulation centre (no team-based training); I3 = 2-day team training at hospital; I4 = 2-day interactive in simulation centre

Main outcome measured by a 185 multiple-choice questionnaire completed 3 weeks before and 3 weeks after the training intervention.

Same intervention as Crofts et al. 2007

Sufficient evidence to support the use of clinically integrated teaching over stand-alone education. While stand-alone teaching improved knowledge, there were no improvements in skills, attitudes or behaviours, whereas clinically integrated teaching showed improvements in knowledge, skills, attitude and behaviour.

Statistical evidence supported the use of live, multi-professional, obstetric emergency training to increase midwives' and doctors' knowledge of obstetric emergency management. However, neither the location of training either in a simulation centre or in localhospitals, nor the inclusion of teamwork training, made any significant difference to the acquisition of knowledge in obstetric emergencies.

Statisticalevidence to support the use of live, eclampsia training to increase providers' performance rate for completion of basic tasks. Neither the location (simulation centre or in localhospitals), nor the inclusion of teamwork training made any significant difference to the performance results for basic task completion.

EBM Evidence-based medicine, I Intervention.


Study design


Augestad K and Lindsetmo R

Systematic review: 51 articles reviewed to determine usefulness of videoconferencing as a clinical and educational too


Country: Norway and developed countries

Bloomfield J et al. 2010 RCT to test if the theory and skill of Nursing students

handwashing can be taught more „,., ,

n — 242 1 — 113 C

effectively when taught using

computer-assisted learning compared Country: UK

to conventional face-to-face teaching

Bradley P et al. 2005

Prospective RCT and qualitative evaluation to compare self-directed, computer-based learning to traditional, live, interactive education techniques

Medical students I = 85, C = 90 Country: Norway

Choa et al

Single-blinded, cluster randomized tria to compare the effectiveness of audiovisual animated CPR instruction with audio, dispatcher-assisted instruction in participants with no previous CPR training; both via mobile phones

Allied health professionals, hospital employees

= 44, C = 41

Country: Korea

Chili S et al. 2009 Experimental research design with two Nurses

groups, one pre-test and two post-tests, C-40

to determine the effectiveness of _ ' _

computer-based interactive instruction Country: Taiwan vs video didactic instruction


Key findings

Media: video Technique: multiple Frequency: NR

Media: computer-based vs live Techniques: multiple Frequency: single

ntervention group received theory via computer-based module; control group via instructor-led. The objectives and content were the same, both groups included practice opportunities.

Technique: self-directed vs interactive

Media: computer-based vs live Frequency: single

ntervention group received self-directed, computer-based modules on EBM; contra group received live, interactive sessions.

Media: mobile, audiovisual animation vs audio instructions from live dispatcher

Technique: POC

Frequency: single

ntervention group used mobile phone application with audiovisual animation instructions for CPR; control group received audio guidance from a live dispatcher.

Media: computer-based vs video

Technique: self-directed interactive vs didactic

Frequency: single

ntervention group received computer-based, interactive educational module; control group watched a video of a lecture.

Review discussed primarily observational data on the use of videoconferencing for provision of lecture, mentoring and POC support for emergencies or trauma settings. Methodology of studies is weak, but shows promise for providing POC and mentoring to rural settings from specialists in other geographical areas.

The computer-assisted learning module was an effective strategy for teaching both theory and practice of handwashing to nursing students and was found to be at least as effective as conventional, face-to-face teaching methods. However, this finding must be interpreted with caution in light of sample size and attrition rates.

There were no differences in outcomes for the computer-based group compared to the live, interactive group in knowledge acquisition, critical appraisal skills or attitudes toward EBM. This trial and its accompanying qualitative evaluation suggest that self-directed, computer-assisted learning may be an alternative format for teaching EBM.

Audiovisual animated CPR instruction via mobile phone resulted in better scores in checklist assessment and time interva compliance in participants without CPR skill compared to those who received CPR instructions from a dispatcher. However, the accuracy of important psychomotor skill measures was unsatisfactory in both groups.

nteractive, computer-assisted instruction increased student assessment correctness compared to video didactic instruction for in-service neurological nursing education after statistical adjustments for length of experience.

Curran Vand Fleet L 2005

Farmer A et al. 2008

Fordis M et al. 2005

Hadley J et al. 2010

Harrington S and Walker B 2004

Systematic review to evaluate the nature and characteristics of the web-based CME, based on Kirkpatrick levels of evaluation; 86 studies were identified, majority were descriptive

Systematic review: 23 studies reviewed to determine the usefulness of print-based materials in practice behaviours or clinicalpractice outcomes

RCT to determine if Internet-based CME can produce changes comparable to those produced via live, small group, interactive CME with respect to physician knowledge and behaviours that have an impact on patient care


Media: Internet

Technique: multiple

Frequency: both single and multiple

Health care professionals


n = 97; I =49, randomly assigned Internet-based over 2 weeks; C1 =44, single, live, interactive session; C2 = 18, from same sites received nothing

Country: USA

Media: print Technique: didactic Frequency: single

Media: Internet-based vs live, interactive Technique: self-directed vs interactive Frequency: single

Intervention group received Internet-based modules over 2 weeks; one control group received a live, interactive session and the other controlgroup received nothing.

Cluster RCT to evaluate the educational effectiveness of a clinically integrated e-learning course for teaching basic EBM among post-graduate medical trainees compared to a traditional lecture-based course of equivalent content

RCT to determine effectiveness of computer-based training compared with the traditional, instructor-led format

Post-graduate medicaltrainees, Media: Internet vs live

Techniques: multiple Frequency: single


Seven clusters of 237 I = 88, C = 72 Country: UK


n = 1,294, I = 670, C = 624 Country: USA

Intervention group received clinicalintegrated, e-learning course on EBM; controlgroup received live, didactic-based course.

Media: computer-based vs live Technique: didactic vs self-directed Frequency: single

Intervention group received self-directed, computer-based instruction; controlgroup received instructor-led, live instruction. Both groups had the same objectives and content.

Inconclusive evidence to identify the most effective characteristics of web-based CME due to a lack of studies focusing on performance change. Findings suggest web-based CME is effective in enhancing knowledge and attitudes. Severalstudies suggest interactive CME that requires participant activity and the chance to practice skills can effect changes in practice behaviours.

Insufficient information to support the effectiveness of print-based educational materials compared to other interventions. Print materials may have a beneficialeffect on process outcomes compared to no intervention, but not on clinicalpractice outcomes.

Internet-based CME can produce objectively measured changes in behaviour as well as sustained gains in knowledge that are comparable or superior to those realized from an effective, live, group-based activity. The Internet-based intervention was associated with a significant increase in the percentage of high-risk patients treated with pharmacotherapeutics according to guidelines compared to the live, group-based control group.

An e-learning course in EBM was as effective in improving knowledge as a standard lecture-based course. There was no statistically significant difference in knowledge of participants in the e-learning course compared to the lecture-based course. The benefits of an e-learning approach include standardization of teaching materials and it is a potentialcost-effective alternative to standard, lecture-based teaching.

The computer-based group significantly outperformed the instructor-led group on the knowledge sub-test at post-test (gain of 28% vs 26%). Participants reported linked, computer-based learning and researchers noted potentialfor efficiencies and cost reduction.

Horiuchi S et al. 2009

RCT compared web-based to live instruction

Kemper K et al. 2006

Leung G et al. 2003

Liaw S et al. 2008

Rabol L et al. 2010

Nurses or midwives

n = 93; C = 45, web-based; I =48, live

Country: Japan

National randomized 2x2 factorial trial Health professionals

RCT to compare the effectiveness of mobile, POC support vs print-based job aids

n = 1,267; completion rate = 62%; Group 1 = 318; Group 2 = 318; Group 3 = 318; Group 4 = 313

Country: USA

Cluster randomized trialto determine the effectiveness of locally adapted practice guidelines and education about paediatric asthma management, delivered to generalpractitioners using interactive, smallgroup workshops

Systematic review: 18 studies reviewed to determine outcomes of live, classroom-based, multi-professional team training

4th year medical students

n = 169; I = 54; C/pocket card = 55; C/nothing = 55

Country: China


n = 29, randomly assigned; I = 1 live, interactive plus guidelines; C/guidelines only= 18; C/nothing = 15

Country: Australia

Health professionals

Media: Internet vs live Techniques: multiple Frequency: single

Intervention group received web-based instruction; controlgroup received didactic live instruction.

Media: Internet

Technique: self-directed

Frequency: single

Group 1: four modules delivered weekly over 10 weeks by email(drip-push); Group 2: modules accessible on web site with four reminders weekly for 10 weeks (drip-pull); Group 3: 40 modules delivered within 4 days by email(bolus-push); and Group 4: 40 modules available on the Internet with one emailinforming participants of availability (bolus-pull).

Media: mobile vs print

Technique: POC

Frequency: single

Intervention group given PDA devices with clinicaldecision support tools; one control group was given a pocket card containing guidelines and the other control group received no intervention.

Media: live vs print only

Technique: interactive vs reading

Frequency: single

Intervention group received live, interactive sessions plus guidelines; controlgroups received guidelines only and no intervention.

Media: live

Technique: multiple

Frequency: single

No significant differences in knowledge were observed between the web-based and face-to-face group. However, the web-based instruction was rated as more flexible and affordable and had a lower drop-out rate than the face-to-face programme.

There were statistically significant improvements in knowledge, confidence and communication scores after the course for each of the Internet-based delivery methods, with no significant differences in any of the three outcomes by delivery strategy. Outcomes were better for those who paid for continuing education credit.

Both the PDA and pocket card groups showed improvements in scores for personal application and current use of EBM. The PDA group showed slightly higher scores in allfive outcomes, whereas those for the pocket card group were not appreciably different from the previous rotation.

Using interactive smallgroup workshops to disseminate locally adapted guidelines was associated with improvement in general practitioners' knowledge and confidence to manage asthma compared to receiving guidelines alone in the controlarm, but did not change their self-reported provision of written action plans.

Although most studies had weak design methods, findings from the 18 studies concluded that team-based training led to positive participant evaluation, knowledge gain and behaviour change. However, the impact on clinicaloutcomes was limited.

Sulaiman N et al. 2010

Triola M et al. 2006

Turner M et al. 2006

Wutoh R et al. 2004

You J et al. 2009

Same study design as Liaw S et al. 2008 for CPE intervention, but used questionnaires to determine any impact on completing written action plans or patient outcomes

RCT to compare effectiveness of virtual patients to live, standardized patients for improving clinicalskills and knowledge

Randomized, controlled, crossover trial to compare efficacy, student preference and cost of web-based, virtualpatient vs live, standardized patient

411 patient surveys from patients of three arms utilized in Liaw, S., et al. 2008 at baseline; 341 at follow-up

Country: Australia

Health professionals I = 23, C = 32 Country: USA

See Liaw S et al. 2008

2nd year medical students I = 25, C = 24 Country: USA

Systematic review: 16 articles reviewed to determine the effect of Internet-based CME interventions on physician performance and health care outcomes

Prospective, randomized study to investigate usefulness of video via mobile device as an instruction tool


Surgical residents I = 24, C = 25 Country: South Korea

Media: virtual patient vs live patient Technique: case-based Frequency: single

Intervention group received two live, standardized patient cases and two virtual patient cases; controlgroup received four standardized patient cases.

Media: virtualpatient vs live patient

Technique: case-based

Frequency: single

Intervention group received web-based instruction for one topic, then standardized patient for another topic. This was reversed for the second cohort, or controlgroup, standardized patient first followed by web-based instruction.

Media: Internet

Technique: multiple

Frequency: both single and multiple

Media: mobile videoconferencing/feedback

Technique: live, interactive with and without mobile POC feedback using video

Frequency: single

Both intervention groups had a didactic session, performed a thoracentesis on a manikin while using video on a mobile phone and received feedback from a live instructor; controlgroup did not receive any video-aided guidance.

The interactive, smallgroup workshops failed to translate into increased ownership of written action plans, improved control of asthma or improved quality of life, compared to receiving guidelines alone or control intervention.

Improvements in diagnostic abilities were equivalent in groups who experienced cases either live or virtually. There was no subjective difference perceived by learners. Using virtual cases has the potentialfor cost efficiencies.

There was no statisticaldifference in learning outcomes between the web-based and standardized patient; however, students preferred the standardized patient format. Start-up costs were comparable, but the ongoing costs of the web-based format were less expensive, suggesting that web-based teaching may be a viable strategy.

Results demonstrate that Internet-based CME are just as effective in imparting knowledge as traditionalformats of CME. However, there is a lack of quality studies to conclude significant positive changes in practice behaviour and additionalstudies are needed.

The overallsuccess rate for performing needle thoracocentesis was significantly higher for the mobile phone video intervention compared to the controlgroup without aided instruction. Participants also rated the mobile phone intervention with significantly higher scores for instrument difficulty and procedure satisfaction.

C Control, CME Continuing medical education, CPR Cardiopulmonary resuscitation, EBM Evidence-based medicine, I Intervention, NR Not reported, PDA Personal digital assistant. POC Point-of-care, RCT Randomized controlled trial.


Study design



Key findings

Alvarez M and Agra Y 2006

Berkhof M et al. 2010

Bloom B 2005

Bordage G et al. 2009a

Davis D and Galbraith R 2009a

Systematic review: 18 articles reviewed to determine educationalinterventions in palliative care and their impact on practice behaviours

Physicians and other allied health professionals

Systematic review: 12 systematic reviews reviewed to determine effective educationaltechniques to teach communication to physicians


Systematic review: 26 articles (all systemic reviews or meta-analyses) reviewed to examine effectiveness of current CME tools and techniques in changing physician clinicalpractices and improving patient health outcomes

Systematic review: 29 articles reviewed to determine if CME leads to an increase in physician knowledge

Systematic review: 105 studies reviewed to determine impact of CME on practice behaviours


Health professionals

Practice behaviours Technique: multiple Media: live

Frequency: both single and multiple

Physicians and health professionals

Communication skills

Technique: multiple

Media: multiple

Frequency: both single and multiple

Practice behaviours and clinicalpractice outcomes

Technique: multiple

Media: multiple

Frequency: both single and multiple


Technique: multiple

Media: multiple

Frequency: both single and multiple

Practice behaviours

Technique: multiple

Media: live

Due to a lack of quality studies, there are insufficient data to conclude about the impact of palliative care interventions on primary care physician practice performance. Although improvements in knowledge, some attitudes and provider satisfaction were demonstrated, there were no significant effects reported on practice behaviours. Didactic education alone was found to be ineffective. Interventions involving multiple techniques, reminders and feedback were found to be more effective at changing behaviours.

Sufficient evidence from 12 systematic reviews to recommend training programmes last at least 1 day, are learner-centred and focus on practicing skills. The best training strategies within the programmes included role-play, feedback and smallgroup discussions. Training programmes should include active, practice-oriented strategies. Oral presentations on communication skills, modeling and written information should only be used as supportive strategies.

Sufficient evidence to conclude that interactive techniques (audit/feedback, academic detailing/ outreach, reminders) are the most effective CME methods impacting practice outcomes and behaviours, while clinicalguidelines and opinion leaders are less effective. Didactic presentations and distributing printed information had little to no effect on physician practice.

Despite low quality of evidence presented in the literature, there is sufficient evidence to confirm an increase in physician knowledge with the use of multimedia, multiple instructional techniques and multiple exposures in CME.

Sufficient evidence to support the use of single, live or multimedia and multiple educational techniques as effective CME methods in changing physician performance. Recommend multiple exposures over single exposures.

Forsetlund L et al. 2009

Gysels M et al. 2005

Hamilton R 2005

Marinopoulos S et al. 2007a

Systematic review: 81 articles reviewed to determine the effect of educational meetings on practice behaviours and clinicalpractice outcomes

Systematic review: 16 articles reviewed to evaluate effective educationaltechniques for teaching communication skills

Systematic review: 24 articles reviewed to determine how to enhance retention of knowledge and skills during and after resuscitation training

Systematic review: from 68,000 citations, 136 studies and nine systematic reviews were identified and reviewed

More than 11,000 health professionals

Health professionals

Health professionals

Health professionals and allied health professionals

Mansouri M and Lockyer J 2007

Meta-analysis: 31 studies reviewed to determine the impact of CME on knowledge, skills and clinicalpractice outcomes

Mostly physicians

Practice behaviours and clinicalpractice outcomes

Technique: multiple

Media: live

Frequency: single

Communication skills

Technique: multiple

Media: multiple

Frequency: both single and multiple

Knowledge, skills

Technique: multiple

Media: multiple

Frequency: both single and multiple

Knowledge, skills, practice behaviours and clinical practice outcomes

Technique: multiple

Media: live

Frequency: both single and multiple

Knowledge, skills, practice behaviours and clinical practice outcomes

Technique: multiple

Media: live

Sufficient evidence to conclude that educational meetings alone or combined with other interventions can have a small improvement on professional practice and health care outcomes, but no effect on changing complex behaviours. Previous reviews found that interactive workshops resulted in moderate improvements, whereas didactic sessions did not.

Sufficient evidence to recommend communication training programmes that are learner-centred, carried out over a long period of time, and combine didactic theoretical components with practicalrehearsaland constructive feedback.

Sufficient evidence to recommend in-hospital simulation to teach resuscitation training for nurses in clinicalareas in addition to remedial training and the availability of resuscitation equipment for self-study. Video self-instruction has been shown to improve competence in resuscitation.

Firm conclusions are not possible due to the overalllow quality of the literature. Despite this, the literature supported the concept that CME was effective at the acquisition and retention of knowledge, attitudes, skills, behaviours and clinicaloutcomes. Common themes included that live media was more effective than print, multimedia was more effective than single media interventions, multiple exposures were more effective than a single exposure, and simulation methods are effective in the dissemination of psychomotor and procedural skills.

Sufficient information to conclude that the impact of CME on physician performance and patient outcome is small, but has a medium effect on knowledge and a larger effect when the interventions are interactive, use multiple methods and are designed for a smallgroup of physicians from a single discipline.

Mazmanian P et al. 2009a

Moores L et al. 2009a

Nestelet al. 2011

O'NeilK et al.

Rampatige R et al. 2009

Raza A et al. 2009 (follow-up study to Coomarasamy A and Khan KS 2004)

Systematic review: 37 articles reviewed to determine the impact of CME on clinical practice outcomes

Systematic review: 136 articles and nine systematic reviews were reviewed to evaluate what makes CME effective

Systematic review: 81 articles retrieved to summarize the best evidence related to use of simulation for learning

Physicians, nurse-practitioners, nurses, allied health professionals

Health professionals

Health professionals

Systematic review: from the 136 studies identified in the systematic review, 15 articles, 12 addressing physician application of knowledge and three addressing psychomotor skills, were identified and reviewed

Systematic review: 476 articles selected for inclusion. Section A relates to CPE in general (sections B and C are not relevant); nine studies were reviewed to determine effect of CME on practice behaviours and clinicalpractice outcomes

Systematic review: Cochrane review of 36 studies reviewed to determine evidence to support effective CME

Health professionals and allied health professionals

Health professionals

Health professionals

Clinicalpractice outcomes

Technique: multiple

Media: multiple

Frequency: both single and multiple


Technique: multiple

Media: multiple

Frequency: both single and multiple

Psychomotor skills

Technique: multiple

Media: multiple

Frequency: both single and multiple

Knowledge, psychomotor skills

Technique: multiple

Media: multiple

Frequency: both single and multiple

Practice behaviours and clinicalpractice outcomes

Technique: multiple

Media: multiple

Frequency: both single and multiple


Technique: multiple

Media: multiple

Due to low quality of evidence, there is no firm conclusion on the impact of CME on clinical practice outcomes; however, multimedia, multiple techniques of instruction and multiple exposures to content are suggested to meet instructionalobjectives intended to improve clinicaloutcomes.

Significant evidence to support the use of CME interventions that use multimedia in instruction, multiple instruction techniques and frequency of exposure, to have a positive effect on knowledge, psychomotor skills, practice performance and clinical outcomes. The use of print media alone is not recommended.

Sufficient evidence is available to conclude that use of simulation leads to improved knowledge and skill. Studies with low-quality evidence suggest a transfer of skills to the clinicalsetting. Instructionaldesign and educationaltheory, contextualization, transferability, accessibility and scalability must allbe considered in simulation-based education programmes.

Sufficient evidence to support CME as effective in improving physician application of knowledge. Multiple exposures and longer durations of CME are recommended to optimize educationaloutcomes. Quality of evidence does not address to psychomotor skilldevelopment.

Interactive and practice enabling strategies are more useful than print-based and educational meetings. Multiple education efforts combined with good feedback/interaction between educators and learners are most effective. Opinion leaders and outreach visits shown to be effective.

Evidence from 16 randomized trials support interactive and clinically integrated learning sessions and interactive classroom teaching as second choice for an effective form of CME. Review demonstrated that interactive workshops improved knowledge and practice behaviours.

Satterlee W et al. 2008

Williams J et al. 2008

Systematic review: nine articles reviewed to determine impact of CME on clinical practice outcomes

Health professionals

Thomson O'Brien MA et al. 2001

Systematic review: 32 articles reviewed to determine the effect of educational meetings on practice behaviours and clinicalpractice outcomes

Systematic review: nine studies reviewed to evaluate if disaster training improves knowledge and skills

Health professionals

n = 2,9

Health professionals and allied health professionals

Clinical practice outcomes

Technique: multiple

Media: multiple

Frequency: both single and multiple

Practice behaviours and clinical practice outcomes

Technique: multiple

Media: live

Frequency: single

Knowledge, skills

Technique: multiple

Media: multiple

Frequency: both single and multiple

Combined didactic presentations and interactive workshops and combined didactic presentations were more effective than traditional didactic presentations alone. The use of multiple interventions over an extended period increased effectiveness. Targeted education should focus on changing a behaviour that is simple, since effect size is inversely proportional to the complexity of the behaviour.

Moderate data quality suggests interactive workshops can result in moderately large changes in professionalpractice. Didactic sessions alone are unlikely to change professional practice.

Insufficient data quality exists to report on the impact of disaster response training interventions on knowledge and skills. Data suggest that both computer-based and live instruction may increase knowledge.

aJHU EPC systematic review. CME Continuing medical education, JHU EPC Johns Hopkins University Evidence-Based Practice Center.

re 2 NJ

live instruction for addressing knowledge, while multiple repetitive exposures leads to better knowledge gains than a single exposure. Games can also contribute to knowledge if designed as interactive learning experiences that stimulate higher thinking through analysis, synthesis or evaluation.

No studies or systematic reviews looked only at attitudes, but CPE that includes clinical integration, simulations and feedback may help address attitudes. The JHU EPC group systematic review evaluation of the short- and long-term effects of CPE on physician attitudes reviewed 26 studies and, despite the heterogeneity of the studies, identified trends supporting the use of multimedia and multiple exposures for addressing attitudes [6].

Several systematic reviews looked specifically at skills, concluding that there is weak but sufficient evidence to suggest that psychomotor skills can be addressed with CPE interventions that include simulations, practice with feedback and/or clinical integration. 'Dose-response' or providing sufficient practice and feedback was identified as important for skill-related outcomes. Other RCTs suggest clinically integrated education for supporting skill development. Choa et al. found that neither the audio mentoring via mobile nor animated graphics via mobile resulted in the desired psychomotor skills, reinforcing the need for practice and feedback for psychomotor skill development identified in other studies [49].

Two systematic reviews focused on communication skills and found techniques that include behaviour modeling, practice and feedback, longer duration or more practice opportunities were more effective [52,53]. Evidence suggests that development of communication skills requires interactive techniques that include practice-oriented strategies and feedback, and limit lecture and print-based materials to supportive strategies only.

Findings also suggest that simulation, PBL, multiple exposures and clinically integrated CPE can improve critical thinking skills. Mobile-based POC support was found to be more useful in the development of critical thinking than print-based job aids.

Several systematic reviews specifically looked at CPE, practice behaviours and the behaviours of the provider. These studies found, despite reportedly weak evidence, that interactive techniques that involved feedback, interaction with the educator, longer durations, multiple exposures, multimedia, multiple techniques and reminders may influence practice behaviours.

A targeted review of 37 articles from the JHU EPC review on the impact of CPE on clinical practice outcomes drew no firm conclusions, but multiple exposures, multimedia and multiple techniques were recommended to improve potential outcomes [6]. Interaction and feedback were found to be more useful than print or educational meetings (systematic review of nine articles) [24], but print-based unsolicited materials were not found to be effective [50].

The systematic review of live, classroom-based, multiprofessional training conducted by Rabal et al. found 'the impact on clinical outcomes is limited' [54].


The heterogeneity of study designs included in this review limits the interpretations that can be drawn. However, there is remarkable similarity between the information from studies included in this review and similar discussions published in the educational psychology literature. We believe that there is sufficient evidence to support efforts to implement and evaluate the combinations of training techniques, frequency, settings and media included in this discussion.

Avoid educational techniques that provide a passive transfer of information, such as lecture and reading, and select techniques that engage the learner in mental processing, for example, case studies, simulation and other interactive strategies. This recommendation is reinforced in educational psychology literature [55]. There is sufficient evidence to endorse the use of simulation as a preferred educational technique, notably for psychomotor, communication or critical thinking skills. Given the lack of evidence for didactic methods, selecting interactive, effective educational techniques remains the critical point to consider when designing CPE interventions.

Self-directed learning was also found to be an effective strategy, but requires the use of interactive techniques that engage the learner. Self-directed learning has the additional advantage of allowing learners to study at their own pace, select times convenient for them and tailor learning to their specific needs.

Limited evidence was found to support team-based learning or the provision of training in work teams. There is a need for further study in this area, given the value of engaging teams that are in the same place at the same time in an in-service training intervention. This finding is especially relevant for emergency skills that require the collaboration and cooperation of a team.

Repetitive exposure is supported in the literature. When possible, replace single-event frequency with targeted, repetitive training that provides reinforcement of important messages, opportunities to practice skills and mechanisms for fostering interaction. Recommendations drawn from the educational psychology literature that address the issue of cognitive overload [56] suggest targeting information to essentials and repetition.

Select the setting based on its ability to deliver effective educational techniques, be similar to the work environment and allow for practice and feedback. In this time of crisis, workplace learning that reduces absenteeism and supports individualized learning is critical. Conclusions from literature in educational psychology reinforce the importance of 'situating' learning to make the experience as similar to the workplace as possible [57].

Certain common themes emerged from the many articles that commented on the role of media in CPE effectiveness. A number of systematic reviews suggest the use of multimedia in CPE. It is important to note that the studies that found similar knowledge outcomes between computer-based and live instruction stated that both utilized interactive techniques, possibly indicating the effectiveness was due to the technique rather than the media through which it was delivered. While the data on use of mobile technology to deliver CPE were limited, the study by Zurovac et al. indicated the potential power of mobile technology to improve provider adherence to clinical protocols [29]. Currently, there is unprecedented access to basic mobile technology and increasing access to lower-cost tablets and computers. The use of these devices to deliver effective techniques warrants exploration and evaluation, particularly in low- and middle-income countries.

CPE can positively impact desired learning outcomes if effective techniques are used. There are, however, very limited and weak data that directly link CPE to improved clinical practice outcomes. There are also limited data that link CPE to improved clinical practice behaviours, which may influence the strength of the linkage to outcomes.


The following limitations apply to the methodology that we selected for this study. An integrative review of the literature was selected because the majority of published studies of education and training in low- and middle-resource countries did not meet the parameters required of a more rigorous systematic review or meta-analysis. The major limitation of integrative reviews is the potential for bias from their inclusion of non-peer-reviewed information or lower-quality studies. The inclusion of articles representing a range of rigor in their research design restricts the degree of confidence that can be placed on interpretations drawn by the authors of those articles, with the exception of original articles that explicitly discussed quality (such as systematic reviews). This review did not make an additional attempt to reanalyse or combine primary data.

Therefore, for purpose of this article, we also graded all articles and included only tier 1 articles in the analysis. This resulted in restriction of information on certain topics for this report, although a wider range of information is available.

We faced an additional limitation in that many articles included in the review were neither fully transparent nor consistent with terminology definitions used in other reports. This is due in part to the fact that we went beyond the bio-medical literature, to include studies conducted in the education and educational psychology literature, as was appropriate to the integrative review methodology. Certain topics were underdeveloped in the literature, which limits

the interpretation that can be drawn on these topics. Other topics are addressed in studies conducted using lower-tier research methodologies (for example observational and/or qualitative studies) that were not included in this article. In addition, the overwhelming majority of studies focused on health professionals in developed or middle-income countries. There were very few articles of sufficient rigor conducted in low- and middle-income countries. This limits what we can say regarding the application of these findings among health workers of a lower educational level and in lower-resourced communities.


In-service training has been and will remain a significant investment in developing and maintaining essential competencies required for optimal public health in all global service settings. Regrettably, in spite of major investments, we have limited evidence about the effectiveness of the techniques commonly applied across countries, regardless of level of resource.

Nevertheless, all in-service training, wherever delivered, must be evidence-based. As stated in Bloom's systematic review, 'Didactic techniques and providing printed materials alone clustered in the range of no to low effects, whereas all interactive programmes exhibited mostly moderate to high beneficial effect. ... The most commonly used techniques, thus, generally were found to have the least benefit' [14]. The profusion of mobile technology and increased access to technology present an opportunity to deliver in-service training in many new ways. Given current gaps in high-quality evidence from low- and middle-income countries, the future educational research agenda must include well-constructed evaluations of effective, cost-effective and culturally appropriate combinations of technique, setting, frequency and media, developed for and tested among all levels of health workers in low- and middle-income countries.


BEME: Best Evidence in MedicalEducation; CI: Confidence interval; CINAHL: Cumulative Index to Nursing and Allied Health Literature; CME: Continuing medicaleducation; CPE: Continuing professionaleducation; CPR: Cardiopulmonary resuscitation; EBM: Evidence-based medicine; JHU EPC: Johns Hopkins University Evidence-Based Practice Center; MeSH: Medicalsubject headings; OCEMB: Oxford Centre for Evidence-Based Medicine; PBL: Problem-based learning; POC: Point-of-care; RCT: Randomized controlled trial.

Competing interests

The authors declare they have no competing interests. Authors' contributions

JB performed article reviews for inclusion, synthesized data and served as primary author of the analysis and manuscript. PJ conceived the study, participated in its design and coordination, and provided significant input into the manuscript. JF provided guidance on the literature review process, grading and categorizing criteria, and quality review of selected articles, and participated actively as an author of the manuscript. CC and JBT contributed to writing of the manuscript. JA searched the literature, performed initial

review and coding, and contributed to selected sections of the manuscript. Allauthors read and approved the finalmanuscript.


We thank the Jhpiego Corporation for support for this research. We thank Dana Lewison, Alisha Horowitz, Rachel Rivas D'Agostino and Trudy Conley for their support in editing and formatting the manuscript. We also thank Spyridon S Marinopoulos, MD, MBA, from the Johns Hopkins University Schoolof Medicine, for his initialinput into the study and links to relevant resources. The findings, interpretations and conclusions expressed in this paper are those of the authors and not necessarily those of the Jhpiego Corporation.

Author details

Jhpiego Corporation, 1615 Thames Street, Baltimore, MD 21231, USA. independent Consultant, San Diego, CA, USA. 3Research Assistant, Baltimore, MD, USA.

Received: 8 August 2012 Accepted: 2 May 2013 Published: 1 October 2013


1. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, Fineberg H, Garcia P, Ke Y, Kelley P, Kistnasamy B, Meleis A, Naylor D, Pablos-Mendez A, Reddy S, Scrimshaw S, Sepulveda J, Serwadda D, Zurayk H: Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010, 376(9756):1923-1958.

2. Wittemore R, Kanfl K: The integrative review: updated methodology. In J Adv Nursing 2005, 52(5):546-553.

3. Marinopoulos SS, Baumann MH: American College of Chest Physicians Health and Science Policy Committee: Methods and definition of terms: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009, 135(3 Suppl):17S-28S.

4. Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence Working Group: The Oxford 2011 Levels of Evidence. Oxford: Centre for Evidence Based Medicine; 2011.

5. Best Evidence in Medical Education (BEME): Appendix 1-BEME Coding Sheet; 2011. 204%20Figs%20&%20Appendices/BEME4_Appx1.pdf.

6. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum R, Bass EB: Effectiveness of continuing medical education. Evidence Report/Technology Assessment, No. 149, (Prepared by the Johns Hopkins Evidence-based Practice Center, under Contract No.290-02-0018) AHRQ Publication No. -7-E006. Agency for Healthcare Research and Quality: Rockville, MD; January 2007.

7. Triola M, Feldman H, Kalet AL, Zabar S, Kachur EK, Gillespie C, Anderson M, Griesser C, Lipkin M: A randomized trial of teaching clinical skills using virtual and live standardized patients. J Gen Intern Med 2006, 21 (5):424-429.

8. Smits PB, De Buisonje CD, Verbeek JH, Van Dijk FJ, Metz JC, Ten Cate OJ: Problem-based learning versus lecture-based learning in postgraduate medical education. Scand J Work Environ Health 2003, 29(4):280-287.

9. White M, Michaud G, Pachev G, Lirenman D, Kolenc A, FitzGerald JM: Randomized trial of problem-based versus didactic seminars for disseminating evidence-based guidelines on asthma management to primary care physicians. J ContinEduc Health Prof 2004, 24(4):237-243.

10. De Lorenzo RA, Abbott CA: Effectiveness of an adult-learning, self-directed model compared with traditional lecture-based teaching methods in out-of-hospital training. Acad Emerg Med 2004,11 (1)33-37.

11. Lin CF, Lu MS, Chung CC, Yang CM: A comparison of problem-based learning and conventional teaching in nursing ethics education. Nurs Ethics 2010, 17(3):373-382.

12. Reynolds A, Ayres-de-Campos D, Pereira-Cavaleiro A, Ferreira-Bastos L: Simulation for teaching normal delivery and shoulder dystocia to midwives in training. Educ Health (Abingdon) 2010, 23(3):405.

13. Alvarez MP, Agra Y: Systematic review of educational interventions in palliative care for primary care physicians. Palliat Med 2006, 20(7):673-683.

14. Bloom BS: Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. Int J Technol Assess Health Care 2005, 21 (3):380-385.

Satterlee WG, Eggers RG, Grimes DA: Effective medical education: insights from the Cochrane Library. Obstet Gynecol Surv 2008, 63(5):329-333. Herbert CP, Wright JM, Maclure M, Wakefield J, Dormuth C, Brett-MacLean P, Legare J, Premi J: Better Prescribing Project: a randomized controlled trial of the impact of case-based educational modules and personal prescribing feedback on prescribing for hypertension in primary care. Fam Pract 2004, 21 (5):575-581.

Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ: Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach 2005, 27(1):10-28. Raza A, Coomarasamy A, Khan KS: Best evidence continuous medical education. Arch Gynecol Obstet 2009, 280(4):683-687. Akl EA, Pretorius RW, Sackett K, Erdley WS, Bhoopathi PS, Alfarah Z, Schunemann HJ: The effect of educational games on medical students' learning outcomes: a systematic review: BEME Guide No 14. Med Teach 2010, 32(1):16-27.

Laprise R, Thivierge R, Gosselin G, Bujas-Bobanovic M, Vandal S, Paquette D, Luneau M, Julien P, Goulet S, Desaulniers J, Maltais P: Improved cardiovascular prevention using best CME practices: a randomized trial.

J Contin Educ Health Prof 2009, 29(1):16-31.

Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, Davis D, Odgaard-Jensen J, Oxman AD: Continuing education meetings and workshops: effects on professional practice and health care outcomes.

Cochrane Database Syst Rev 2009, 2, CD003030.

Mansouri M, Lockyer J: A meta-analysis of continuing medical education

effectiveness. J Contin Educ Health Prof 2007, 27(1):6-15.

Perry M, Draskovic I, Lucassen P, Vernooij-Dassen M, Van Achterberg T,

Rikkert MO: Effects of educational interventions on primary dementia

care: A systematic review. Int J Geriatr Psychiatr 2011, 26(1):1-11.

Rampatige R, Dunt D, Doyle C, Day S, Van Dort P: The effect of continuing

professional education on health care outcomes: lessons for dementia

care. Int Psychogeriatr 2009, 21(Suppl 1):S34-S43.

Blaya JA, Fraser HS, Holt B: E-health technologies show promise in

developing countries. Health Aff (Millwood) 2010, 29(2):244-251.

Leung GM, Johnston JM, Tin KY, Wong IO, Ho LM, Lam WW, Lam TH:

Randomised controlled trial of clinical decision support tools to improve

learning of evidence-based medicine in medical students. BMJ 2003,


You JS, Park S, Chung SP, Park JW: Usefulness of a mobile phone with video telephony in identifying the correct landmark for performing needle thoracocentesis. Emerg Med J 2009, 26(3):177-179. Yuan H, Williams BA, Fan L: A systematic review of selected evidence on developing nursing students' critical thinking through problem-based learning. Nurse Educ Today 2008, 28(6):657-663.

Zurovac D, Sudoi RK, Akhwale WS, Ndiritu M, Hamer DH, Rowe AK, Snow RW: The effect of mobile phone text-message reminders on Kenyan health workers' adherence to malaria treatment guidelines: a cluster randomised trial. Lancet 2011, 378(9793):795-803. Murad MH, Coto-Yglesias F, Varkey P, Prokop LJ, Murad AL: The effectiveness of self-directed learning in health professions education: a systematic review. Med Educ 2010, 44(11):1,057-1,068. Young JM, Ward J: Can distance learning improve smoking cessation advice in family practice? A randomized trial. J Contin Educ Health Prof 2002,22(2):84-93. Harder BN: Use of simulation in teaching and learning in health sciences: a systematic review. J Nurs Educ 2010, 49(1):23-28. McGaghie WC, Siddall VJ, Mazmanian PE, Myers J: American College of Chest Physicians Health and Science Policy Committee: Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009, 135(3 Suppl):62S-68S. Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ: A systematic review of skills transfer after surgical simulation training. Ann Surg 2008, 248(2):166-179.

Steadman RH, Coates WC, Huang YM, Matevosian R, Larmon BR, McCullough L, Ariel D: Simulation-based training is superior to problem-based learning for the acquisition of critical assessment and management skills. Crit Care Med 2006, 34(1):151-157.

Lamb D: Could simulated emergency procedures practised in a static environment improve the clinical performance of a Critical Care Air Support Team (CCAST)? A literature review. Intensive Crit Care Nurs 2007, 23(1)33-42.

Bruppacher HR, Alam SK, LeBlanc VR, Latter D, Naik VN, Savoldelli GL, Mazer CD, Kurrek MM, Joo HS: Simulation-based training improves physicians' performance in patient care in high-stakes clinical setting of cardiac surgery. Anesthesiology 2010, 112(4):985-992. Daniels K, Arafeh J, Clark A, Waller S, Druzin M, Chueh J: Prospective randomized trial of simulation versus didactic teaching for obstetrical emergencies. Simul Healthc 2010, 5(1):40-45.

Ellis D, Crofts JF, Hunt LP, Read M, Fox R, James M: Hospital, simulation center, and teamwork training for eclampsia management: a randomized controlled trial. Obstet Gynecol 2008,111 (3):723-731. Hamilton R: Nurse's knowledge and skill retention following cardiopulmonary resuscitation training: a review of the literature. JAdv Nurs 2005, 51 (3):288-297.

Merien AE, Van de Ven J, Mol BW, Houterman S, Oei SG: Multidisciplinary team training in a simulation setting for acute obstetric emergencies: a systematic review. Obstet Gynecol 2010,115(5)1,021 -1,031. Crofts JF, Ellis D, Draycott TJ, Winter C, Hunt LP, Akande VA: Change in knowledge of midwives and obstetricians following obstetric emergency training: a randomised controlled trial of local hospital, simulation centre and teamwork training. BJOG 2007, 114(12):1,534-1,541. Kerfoot BP, Fu Y, Baker H, Connelly D, Ritchey ML, Genega EM: Online spaced education generates transfer and improves long-term retention of diagnostic skills: a randomized controlled trial. J Am Coll Surg 2010,211 (3)331 -337. Kerfoot BP, Kearney MC, Connelly D, Ritchey ML: Interactive spaced education to assess and improve knowledge of clinical practice guidelines: a randomized controlled trial. Ann Surg 2009, 249(5):744-749. Kerfoot BP, Baker HE, Koch MO, Connelly D, Joseph DB, Ritchey ML: Randomized, controlled trial of spaced education to urology residents in the United States and Canada. J Urol 2007, 177(4):1,481-1,487. Coomarasamy A, Khan KS: What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ 2004, 329(7473):1,017.

Harrington SS, Walker BL: The effects of computer-based training on immediate and residual learning of nursing facility staff. J Contin Educ Nurs 2004, 35(4):154-163. quiz 186-7.

Curran VR, Fleet L: A review of evaluation outcomes of web-based

continuing medical education. Med Educ 2005, 39(6):561 -567.

Choa M, Park I, Chung HS, Yoo SK, Shim H, Kim S: The effectiveness of

cardiopulmonary resuscitation instruction: animation versus dispatcher

through a cellular phone. Resuscitation 2008, 77(1):87-94.

Farmer AP, Legare F, Turcot L, Grimshaw J, Harvey E, McGowan JL, Wolf F:

Printed educational materials: effects on professional practice and health

care outcomes. Cochrane Database Syst Rev 2008, 3, CD004398.

Liaw ST, Sulaiman ND, Barton CA, Chondros P, Harris CA, Sawyer S, Dharmage

SC: An interactive workshop plus locally adapted guidelines can improve

general practitioners asthma management and knowledge: a cluster

randomised trial in the Australian setting. BMC Fam Pract 2008,9:22.

Gysels M, Richardson A, Higginson IJ: Communication training for health

professionals who care for patients with cancer: a systematic review of

training methods. Support Care Canc 2005, 13(6):356-366.

Berkhof M, Van Rijssen HJ, Schellart AJ, Anema JR, Van der Beek AJ: Effective

training strategies for teaching communication skills to physicians: an

overview of systematic reviews. Patient Educ Couns 2011,84(2):152-162.

Rabol LI, Ostergaard D, Mogensen T: Outcomes of classroom-based team

training interventions for multiprofessional hospital staff: A systematic

review. Qual Saf Health Care 2010, 19(6):e27.

Woolfolk AE: Educational Psychology. 11th edition. Upper Saddle River, NJ: Pearson/Merrill/Prentice Hall; 2009.

Mayer RE: Applying the science of learning to medical education.

Med Educ 2010, 44(6):543-549.

Gott S, Lesgold A: Competence in the workplace: how cognitive performance models and situated instruction can accelerate skill acquisition. In Advances in Instructional Psychology, Volume 4. Edited by Glaser R. Hillsdale, NJ: Erlbaum; 2000.


Cite this article as: Bluestone et al.: Effective in-service training design and delivery: evidence from an integrative literature review. Human Resources for Health 2013 11:51.

Submit your next manuscript to BioMed Central and take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at

I Central