Scholarly article on topic 'Flipping Radiology Education Right Side Up'

Flipping Radiology Education Right Side Up Academic research paper on "Clinical medicine"

CC BY-NC-ND
0
0
Share paper
Academic journal
Academic Radiology
OECD Field of science
Keywords
{"Flipped classroom" / "flipped learning" / neuroimaging / clerkship / imaging / "task value" / "achievement emotions"}

Abstract of research paper on Clinical medicine, author of scientific article — Erin E. O'Connor, Jessica Fried, Nancy McNulty, Pallav Shah, Jeffery P. Hogg, et al.

Rationale and Objectives In flipped learning, medical students independently learn facts and concepts outside the classroom, and then participate in interactive classes to learn to apply these facts. Although there are recent calls for medical education reform using flipped learning, little has been published on its effectiveness. Our study compares the effects of flipped learning to traditional didactic instruction on students' academic achievement, task value, and achievement emotions. Materials and Methods At three institutions, we alternated flipped learning with traditional didactic lectures during radiology clerkships, with 175 medical students completing a pretest on general diagnostic imaging knowledge to assess baseline cohort comparability. Following instruction, posttests and survey examinations of task value and achievement emotions were administered. Linear mixed effects analysis was used to examine the relationship between test scores and instruction type. Survey responses were modeled using ordinal category logistic regression. Instructor surveys were also collected. Results There were no baseline differences in test scores. Mean posttest minus pretest scores were 10.5% higher in the flipped learning group than in the didactic instruction group (P = 0.013). Assessment of task value and achievement emotions showed greater task value, increased enjoyment, and decreased boredom with flipped learning (all P < 0.01). All instructors preferred the flipped learning condition. Conclusions Flipped learning was associated with increased academic achievement, greater task value, and more positive achievement emotions when compared to traditional didactic instruction. Further investigation of flipped learning methods in radiology education is needed to determine whether flipped learning improves long-term retention of knowledge, academic success, and patient care.

Academic research paper on topic "Flipping Radiology Education Right Side Up"

ARTICLE IN PRESS

Original Investigation

Flipping Radiology Education Right

Side Up

Erin E. O'Connor, MD, Jessica Fried, BA, Nancy McNulty, MD, Pallav Shah, MD, Jeffery P. Hogg, MD, FACR, Petra Lewis, MBBS, FACR, Thomas Zeffiro, MD, PhD,

Vikas Agarwal, MD, Sravanthi Reddy, MD

Rationale and Objectives: In flipped learning, medical students independently learn facts and concepts outside the classroom, and then participate in interactive classes to learn to apply these facts. Although there are recent calls for medical education reform using flipped learning, little has been published on its effectiveness. Our study compares the effects of flipped learning to traditional didactic instruction on students' academic achievement, task value, and achievement emotions.

Materials and Methods: At three institutions, we alternated flipped learning with traditional didactic lectures during radiology clerkships, with 175 medical students completing a pretest on general diagnostic imaging knowledge to assess baseline cohort comparability. Following instruction, posttests and survey examinations of task value and achievement emotions were administered. Linear mixed effects analysis was used to examine the relationship between test scores and instruction type. Survey responses were modeled using ordinal category logistic regression. Instructor surveys were also collected.

Results: There were no baseline differences in test scores. Mean posttest minus pretest scores were 10.5% higher in the flipped learning group than in the didactic instruction group (P = 0.013). Assessment of task value and achievement emotions showed greater task value, increased enjoyment, and decreased boredom with flipped learning (all P < 0.01). All instructors preferred the flipped learning condition.

Conclusions: Flipped learning was associated with increased academic achievement, greater task value, and more positive achievement emotions when compared to traditional didactic instruction. Further investigation of flipped learning methods in radiology education is needed to determine whether flipped learning improves long-term retention of knowledge, academic success, and patient care.

Key Words: Flipped classroom; flipped learning; neuroimaging; clerkship; imaging; task value; achievement emotions.

© 2016 The Association of University Radiologists. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

INTRODUCTION

Medical students must master an extraordinarily large knowledge base and associated technical vocabulary in a very short time period. To this end, undergraduate radiology education has been largely dominated by didactic teaching methods designed to expeditiously deliver large volumes of information with a minimum student/instructor interaction (1). However, traditional large group lectures may not be ideal for the facilitation of development of the types of knowledge discovery and problem-

Acad Radiol 2016; ■:■■-■■

From the Department of Radiology, Temple University School of Medicine, 3401 N. Broad Street, Philadelphia, PA 19140 (E.E.O., P.S.); Department of Radiology, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire (J.F., N.MN., P.L.); Robert C. Byrd Health Sciences Center, Department of Radiology, West Virginia University School of Medicine, Morgantown, West Virginia (J.P.H.); Department of Psychology, Temple University, Philadelphia, Pennsylvania (T.Z.); Department of Radiology, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania (V.A.); University of Southern California Keck School of Medicine, Los Angeles, California (S.R.). Received January 21, 2016; revised February 23, 2016; accepted February 24, 2016. Address correspondence to: E.E.O. e-mail: oconnoe@temple.edu

© 2016 The Association of University Radiologists. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). http://dx.doi.org/10.1016/j.acra.2016.02.011

solving skills required in radiology and other types of medical practice (2,3).

To address this problem, there is growing interest in exploring complementary medical instructional approaches that would more efficiently fill in existing knowledge gaps, foster application of knowledge stores, promote higher order thinking, and better prepare students for the challenges of clinical decision-making encountered in patient care contexts (4,5). Although many medical schools have begun to integrate problem- and team-based learning into their curricula, the transition to these interactive learning methods has been slow, and the use of didactic lectures still predominates (6). It is difficult to assess the learning outcomes associated with the ongoing slow reform in undergraduate medical education over the past decade, because medical educators have historically adopted varying definitions of what constitutes a problem-based learning curriculum, and not all have adopted the criteria advocated by Barrows, who developed the first problem-based learning curriculum at McMasters University (7). Moreover, estimates of the efficacy of problem-based learning in medical curricula vary (7,9,10).

Concerning the rate at which clerkship education is changing to incorporate more problem-based learning approaches, an informal survey of the institutions participating in this study

revealed that didactics are still widely used, occupying between 50% and 95% of the available clerkship class time, perhaps because of both the relative efficiency of lectures in transmitting information to large groups and the faculty reluctance to adopt new teaching methods. In radiology instruction during the clinical years in medical education, 50% of institutions report exclusive use of lectures and textbooks (8). Concerns have also been raised regarding the economic viability of problem-based curricula for medical school class sizes greater than 100 because of the extensive resources needed to operate a fully problem-based curriculum (9,10). We suspect that lectures constitute the primary formal education method in medical clerkships in most US institutions and that it is common for clerkship students to receive didactic instruction by attending lectures designed primarily for resident staff.

The flipped classroom pedagogical approach encourages students to work independently to learn basic facts and concepts outside the classroom through varied methods, including reading, completing online education modules, and watching recorded lectures (11—15). This self-paced fact learning is supplemented by more dynamic, interactive classes in which the educator engages students in activities designed to develop skills related to application of these facts and concepts (16). The key features of this blended learning method involve both the use of class time to foster interactive application of knowledge and a shift in the educator's role from primarily transmitting facts to one facilitating deeper learning (17,18). The flipped classroom framework can be contrasted with more traditional instructional approaches in which classroom time consists of didactic lectures, and work outside the classroom involves either additional reading or working on problem sets before or after the didactic lecture. A key aspect of the flipped classroom involves moving basic material learning to self-paced sessions before class, thereby preserving class time for heightened student/instructor interaction and consolidation of the basic material.

An extension of the original flipped classroom idea, called flipped learning, has additional features, including an even more active and collaborative learning environment, additional options for self-paced learning, possibilities for peer-to-peer teaching, and provision of feedback to students for successful performance (19,20). In this arrangement, a particular advantage to instructors is the opportunity to identify concepts and skills that present difficulties for individual students, allowing adaptation of the teaching session to promote mastery of the more challenging skills and more strongly reinforce learning of basic facts and concepts (13).

Flipped classroom and flipped learning methods have been widely used in grade school, college, and graduate business education (19,21—25). Although not employed extensively in medical education, flipped approaches are being considered by many medical school curriculum reform efforts (26). Employing them seems sensible in a digitally oriented world with rapidly increasing biomedical knowledge, whose mastery may require more efficient instructional methods. Flipped learning in medicine allows students to apply recently acquired

domain knowledge to problem-solving scenarios that simulate the clinical decision processes commonly required in patient care. This experiential and situated learning has high face validity with respect to physician training, in contrast to lectures that simply convey facts. Although we agree with the need to reimagine how medical education is delivered, very little has been published on the effectiveness of flipped learning approaches in medical education settings. It seems prudent to explore the effectiveness and student acceptability of this proposed reform before implementing such a major transformation in the delivery of undergraduate medical education.

While exploring the efficacy of flipped learning in fact and skill learning, it is also important to examine the emotional factors that influence learning in medical environments. Medical students' motivational beliefs and achievement emotions play a significant role in their academic achievement (27), perhaps by encouraging the types of more intense student participation that strongly correlate with academic performance (28,29). Although cognitive factors, including academic achievement and standardized test scores, receive strong emphasis in medical education, they are often of limited value in predicting future clinical performance (27,30). Medical students are consistently reminded that the process of learning is more important than the grades received. Even so, the National Residency Matching Program fosters an emphasis on quantitative performance data to discern levels of achievement, paradoxically emphasizing the weight of grades and standardized examination results. This situation should compel us to more closely examine personal factors that shape learning and performance in medical training, as their assessment may contribute to better prediction of an individual's class performance. To our knowledge, there are no previous studies examining how flipped learning influences emotional and experiential factors. We hypothesized that a flipped learning approach, centered on clinical diagnosis and management scenarios, will facilitate mastery of the radiology clerkship curriculum, strengthen motivation beliefs, and elicit more positive achievement emotions. Our experimental design compared the effects of flipped learning to didactic instruction on medical students' academic achievement, task value, and achievement emotions in a radiology clerkship course. We believe that comparing the outcomes associated with flipped learning and didactic instruction is an important first step, as it provides a relevant contrast between two operationally defined teaching methods, in the context of ongoing calls for replacement of didactics with flipped learning (6,26).

METHODS

A prospective cohort study was conducted from January 2014 to April 2015 at three medical schools, with Institutional Review Board approval at each site. We attempted to include a geographically diverse breadth of schools to account for possible regional influences. All participants were third or fourth year medical students enrolled in a 4-week radiology clerkship or radiology elective at one of the three participating

institutions. There were a total of four instructors at the three sites. At two of the sites, one instructor was responsible for two neuroimaging educational sessions. At the third site, there were two instructors, each of whom was responsible for presenting one of the two neuroimaging topics covered and presented their assigned topic in both the didactic lecture and the flipped learning workshop formats. We chose to contrast flipped learning with traditional didactic instruction because instructional lectures are still widely used in medical curricula, and the two approaches could be operationally defined to allow clean experimental comparison.

Before the start of the study, two traditional didactic lectures and two flipped learning workshops, both focusing on the same neuroimaging content, were designed by two of the authors who are fellowship-trained, practicing neuroradiologists who direct the radiology course at their institutions. Although the learning objectives and curriculum covered by the traditional didactic lectures and flipped learning workshops were identical, the flipped learning workshops used clinical scenarios designed to engage students in the application of factual knowledge. The lectures and flipped learning workshops underwent peer review by three different radiologists. Two of the reviewers subspecialized in neuroradiology, and the other had extensive experience in undergraduate medical education. The reviewers critiqued both the didactic lectures and the flipped learning workshops and evaluated whether both curricula covered the stated learning objectives and had similar content. The didactic lectures and flipped learning workshops were edited based on the reviews and then sent back to the reviewers for final approval.

Guidelines on how faculty should deliver the didactic lectures and administer the flipped learning sessions were developed before the start of the study to increase the uniformity of the material presented across institutions. Each traditional didactic lecture and flipped learning workshop came with a script for the instructor to follow. In addition, all authors attended a 9-hour training session. Didactic lectures and flipped learning sessions covered the same content and had the same learning objectives.

Flipped learning educators distinguish between flipped classrooms and flipped learning, as these terms are not interchangeable. Some educators may already supplement their classes by assigning students texts, online tutorials, or supplemental videos before or after class. However, to implement flipped learning, educators must integrate the following additional components into their practice: a flexible teaching environment, a learning culture, intentional content, and a professional educator (16). With these characteristics, flexible learning environments provide variation in learning modes, possibilities for students to interact and reflect on their learning experience, and opportunities to monitor student progress and make adjustments as needed (16).

To provide a flexible learning environment for our flipped learning cohorts, the medical students completed a self-paced online tutorial before attending an interactive workshop that posed questions to medical students based on common

clinical scenarios. The medical students would then analyze the associated images and summarize the relevant imaging findings. Although content and presentation guidelines were provided to facilitators conducting the workshops, instructors were able to adapt individual workshops to devote more time on teaching concepts that particular medical students found more challenging and devote less time on concepts that students had already mastered.

A culture of learning shifts emphasis from the educator to the learner (16). When students actively participate in knowledge application, the resultant learning should hold greater personal meaning and students have opportunities to explore content in greater depth. To incorporate this pillar into our flipped learning classes, educators facilitated a workshop that asked medical students to answer questions and solve problems during class time. Particular topics could be explored in greater detail where needed or desired. The flipped learning workshops used clinical scenarios to simulate clinical decision-making with emphasis on the role of imaging, providing a source of personal meaning for student physicians. This arrangement contrasts with traditional didactic instruction that uses an educator-centered model, with the instructor serving as the primary source for fact dissemination, and students not given opportunities in class to develop clinical decision-making skills.

Emphasizing intentional content requires that educators decide what material students should explore on their own and what student-centered active learning strategies they will use in class to facilitate synthesis of knowledge (16). For this study, we employed online neuroimaging tutorials that medical students would be required to complete before attending class and designed the associated flipped learning workshops. The flipped learning workshops employed active learning strategies that required application and synthesis of knowledge acquired during the online tutorials. Questions were posed to learners in slide format with a single question per slide, with answers discussed in the class to promote discussion. In the clinical scenarios used for problem solving, students were asked to highlight and discuss imaging findings presented on slides, then explaining how the imaging findings would affect clinical care. Although the authors designed the didactic lectures to cover content and learning objectives identical to the flipped learning workshops, the didactic lectures did not use active learning strategies.

The presence of a professional educator plays an essential role in observing and providing immediate feedback to learners in flipped learning classes (16). The flipped learning arm of our study incorporated this pillar through the facilitators' discussion of correct and incorrect answers during class time.

To control for the amount of learning time spent by each cohort, the didactic classes also completed the assigned online tutorials before attending lectures. However, we did not incorporate any of the four pillars of flipped learning into their instruction.

All components of the investigation (Fig 1) were completed within the first 2 weeks of the 4-week clerkship and

Traditional didactic cohort

Pre-test on general imaging knowledge

Completion of online tutorial on neuroimaging of trauma

Didactic lecture on neuroimaging of trauma

Completion of online tutorial on stroke and headache neuroimaging

Didactic lecture on neuroimaging of stroke and headache

Survey assessment of task value and achievement emotions

Post-test on neuroimaging

Flipped learning

cohort

Pre-test on general imaging knowledge

Completion of online tutorial on neuroimaging of trauma

Interactive workshop on neuroimaging of trauma

Completion of online tutorial on neuroimaging of stroke and headache

Interactive workshop on neuroimaging of stroke and headache

Survey assessment of task value and achievement emotions

Post-test on neuroimaging

Figure 1. The flowchart demonstrates the chronological order of the investigation for each cohort.

there were no other neuroimaging teaching sessions scheduled during that time. Student assignment to flipped learning (intervention group) or traditional didactic lectures (control group) alternated with each block of the clerkship, with 175 (69%) of the 253 radiology students consenting to the study and completing all requirements for inclusion. There were 103 subjects out of 139 in the traditional didactic instruction cohort and 72 subjects out of 112 in the flipped learning cohort, representing 27% and 36% drop-out rates, respectively. Although student participation in the study was voluntary, all components of the study, with the exception of the survey, were course requirements. The pretest and posttest scores did not count toward the students' final grade.

To assess baseline comparability between the cohorts, on the first day of the course, all students completed a pretest comprising 20 validated questions on general imaging knowledge selected from the Radiology ExamWeb databank (31) (ExamWeb, St. Helena, CA). Radiology ExamWeb is a national, web-based question item database and examination system.

Subjects in both the control and the intervention groups were required to complete a peer-reviewed, online, virtual patient, neuroradiology educational tutorial before attending the traditional didactic lecture or flipped learning workshop. The tutorials were from the Case-based Online Radiology Education collection (www.med-u.org/core), for which the participating institutions had subscriptions. One of the online

tutorials focused on trauma neuroimaging. The average time taken by students in the didactic cohort to complete this tutorial was 46 minutes and the average time taken by students in the flipped learning cohort was 48 minutes. The other online tutorial focused on neuroimaging of stroke and headache, for which the average time taken for completion was 61 minutes for the didactic cohort and 62 minutes for the flipped learning cohort. Timely completion of the tutorials was a criterion for inclusion in the study and was monitored.

Instructors were blinded as to which students enrolled in the study. The control group received two 65- to 70-minute didactic lectures that encompassed the content of the online tutorial. Subjects in the control group were asked to hold their questions until the end of the lecture, with the final 10 minutes of the session used to answer questions. Subjects in the intervention group took part in flipped learning workshops that required application and synthesis of knowledge acquired from the online tutorials using the methods described previously. The flipped learning workshops lasted 70— 75 minutes. The range of time between the two neuroimaging sessions for both cohorts was 0—11 days.

A 19-item electronic survey was administered after the second neuroimaging didactic lecture or flipped classroom workshop to examine students' task value and achievement emotions associated with the neuroimaging instruction (Appendix). Task value is defined as students' subjective judgments of how interesting, important, and useful an educational activity is to them (32). Course task value was measured by adapting questions from a previous educational study (33). The six-item task value subscale assessed how interesting, important, and useful the presented material was to the participants. Students' achievement emotions are defined as emotions tied directly to achievement activities or achievement outcomes (34). Achievement emotions related to the course were measured using a shortened version of the class-related emotions section of the Achievement Emotions Questionnaire (35). Minor changes were made to wording used in the original subscales to reflect the undergraduate medical education context of our investigation. Both adapted subscales were modeled after the previous work of Artino et al. (27). Student survey responses concerning task value and achievement emotions associated with sessions were collected using 5-point Likert items and managed using REDCap electronic data capture tools (36). Likert scale responses were formed by summing responses within each of the five categories: task value, enjoyment, anxiety, boredom, and two miscellaneous questions. We then compared the task value and achievement emotions between the two cohorts using ordinal categorical regression.

A posttest, consisting of 30, multiple-choice, validated examination questions drawn from the Radiology ExamWeb data bank was administered within 3 days of completion of the second neuroimaging didactic lecture/flipped classroom workshop. The posttest questions assessed knowledge of the neuroimaging content covered and were based on the key

learning objectives of the two online virtual patient neuroimaging cases.

The pretest and posttest items had been previously validated by analysis of P and point biserial coefficients so that P values were >0.35 and <0.95 and point biserial coefficient value was >0.10. Items had been previously deployed in national examinations with >30 student respondents and reviewed by the Radiology ExamWeb Committee (Alliance of Medical Student Educators in Radiology). The pretest and posttest scores did not count toward students' final grade for the course.

Data Collection and Analysis

A survey, incorporating two subscales assessing task value and student achievement emotion, was administered via REDCap (36). Data were then screened for accuracy and missing values. Examination score data were imported from Radiology ExamWeb and then screened for accuracy and missing values.

We used R 3.2.2 (R Core Team, 2015) and the lmer function to perform a linear-mixed effects regression analysis of the relationship between test score and instruction type. As fixed effects, we entered test (pre vs. post instruction test) and instruction type and their interaction into the model. As random effects, we included intercepts for subjects and institutions. Visual inspection of residual plots did not reveal any obvious deviations from homoscedasticity (variance homogeneity) or normality. Parameter-specific P values and approximate degrees of freedom were obtained using the Satterthwaite approximation.

Survey responses to individual questions were first examined using Likert plots (37) (Figs 2 and 3). Next, responses assessing task value or emotional content were then summed within each category and modeled using an ordinal logistic response model with a cumulative link function (vglm function).

Instructor surveys regarding their experience using traditional didactic instruction and flipped classroom instruction were also administered via REDCap with Likert items that ranged from strongly disagree to strongly agree (Table 2). Free response questions that inquired about aspects of each instructional technique that promoted or hindered learning were also included in the survey.

RESULTS

The mean pretest score was 53.4%. There was no statistically significant difference in the pretest scores between the two cohorts (P = 0.42). The mean test score increase averaged across both instruction types was 28.5% (P < 0.001). The effect of instruction type is seen in the interaction between pre- and post test scores and instruction type (P = 0.013), reflecting a 5.36% greater score increase in the flipped learning than in the didactic instruction cohort (Fig 2a). There was no significant variation in the effects of instruction type among institutions (Fig 2b).

Figure 2. (a) Effect of instruction type on test scores. The box plots show changes in median pretest and posttest scores in the traditional didactic instruction and flipped learning cohorts. Gray boxes represent pretest scores and white boxes represent posttest scores. The horizontal line is the median; the box shows the interquartile range and the notch displays a 95% confidence interval around the median. (b) Effect of site on test scores. The box plots show variation in the effects of instruction type among the three participating institutions, labeled 1,2, and 3. Gray boxes represent pretest scores and white boxes represent posttest scores. The horizontal line is the median; the box shows the interquartile range and the notch displays a 95% confidence interval around the median.

An analysis of variance with dropout number as the response variable and class type and institution as factors did not reveal a statistically significant effect of either class type or institution or their interaction.

Survey results for the entire group revealed a largely consistent pattern of responses within each of the five question categories (Fig 3a). Visual inspection of the associated Likert plots shows differences between the flipped learning cohort and the didactic lecture cohorts (Fig 3b). The results of ordinal multinomial response models confirmed that the flipped classroom group experienced higher task value (P < 0.001), less boredom (P < 0.001), and had greater enjoyment (P < 0.001). There was no statistically significant difference in anxiety experienced by the two cohorts (P = 0.579). There was no statistically significant effect of sex on task value, enjoyment, anxiety, or boredom (P = 0354, P = 0.546, P = 0.516, P = 0.368, respectively) (Table 1).

Figure 3. (a) Each student answered one of five levels of agreement or disagreement with each of 19 questions. Survey results revealed a largely consistent pattern of responses within each of the five question types. Answers are shown averaged for both cohorts with responses grouped by categories to demonstrate the polarity of the responses: TV, task value; AE-E, achievement emotions—enjoyment; AE-A, achievement emotions—anxiety; AE-B, achievement emotions—boredom. Each panel of the plot shows an ordered grouping of the questions into the category listed in its left strip label. The frequency distribution of many of the responses is skewed, motivating treatment of the data with ordinal categorical modeling. The number of students answering each question is shown as row count totals. (b) The student survey results are shown here with the didactic and flipped group survey responses in separate columns, showing differences in the response frequency distributions in the two experimental conditions. The survey questions are listed to the left of the response distributions in the order presented.

TABLE 1. Student Survey Results for Didactic and Flipped Classroom Types

TV-SUM

AE-E-SUM

AE-A-SUM

AE-B-SUM

Didactic Strongly Disagree Disagree Neutral Agree

Strongly Agree Flipped Strongly Disagree Disagree Neutral Agree

Strongly Agree Coefficients Estimate P

2 23 106 303 184

0 2 42 175 213

-1.09 0.00040

0 19 88 194 111

0 3 28 118 139

-1.15 0.00022

70 177

97 55 13

98 50 72 11

-0.17 0.59

48 91 72 80 18

44 108 37 26 1

0.90 0.0032

Summed survey responses are grouped by categories. AE-A, achievement emotions—anxiety; AE-B, achievement emotions—boredom; AE-E, achievement emotions—enjoyment; TV, task value. Coefficient estimates and their associated P values are shown for the didactic vs flipped condition contrast.

Instructor survey responses are listed in Table 2. All of the instructors preferred the flipped learning condition, with three of the four strongly favoring it as the preferred teaching style.

DISCUSSION

Critical thinking and complex reasoning skills are at the core of medical education (38). The traditional didactic approaches often used in medical education do not foster such skills (4). We tested the effects of flipped learning in radiology clerkships to explore whether it would better facilitate fact learning and the development of clinical decision-making skills. Flipped learning provides medical students with opportunities to develop self-directed learning skills, still allowing possibilities for solidifying acquired knowledge and concepts through facilitator-led interactive workshops (39). Flipped learning uses modern instructional technologies for pre-class learning and encourages participation in more interactive activities such as problem solving, discussions, and debates during class time (12—16,40). Videos, online educational modules, and recorded lectures are ideal media for a

flipped learning approach, providing students with self-study materials to be covered at their convenience, and freeing educators to use limited class time more efficiently and effectively (41). Our study demonstrates how the interactive nature of flipped learning is effective in developing problemsolving and radiology clinical decision-making skills. It also clearly demonstrates the positive impact interactive learning has on emotional factors that influence learning. Our flipped learning model gave students experience with clinical cases and compelled them to apply the facts previously learned via self-study to patient care contexts. Although students receiving didactic instruction may also learn facts independently before or during instruction, the application of these facts generally receives little attention. In addition, the traditional didactic classroom environment lacks flexibility, as instructors are unable to adapt the session to meet individual students' needs. Exercises performed in an interactive environment empower students to envision how their acquired knowledge can influence subsequent patient care, giving the exercises personal meaning for physicians in training. The traditional didactic lecture format may make it more difficult for students to see how disseminated information will affect patient care, and thus may have less associated personal meaning.

TABLE 2. Instructor Survey Responses

Instructor 1

Instructor 2

Easy to give a traditional didactic lecture.

Easy to give a flipped classroom workshop.

Students found it easy to learn in a traditional didactic lecture.

Students found it easy to learn in a flipped classroom workshop.

Satisfied with the instruction I gave in the traditional lecture format.

Satisfied with the instruction I gave in the flipped classroom workshop format.

The traditional didactic lectures promoted learning.

Aspects of the traditional didactic lectures that promoted learning

The flipped classroom workshops promoted learning.

Aspects of the flipped classroom workshops that promoted learning.

The traditional didactic lectures

hindered learning. Aspects of traditional didactic lectures that hindered learning.

The flipped classroom workshops hindered learning.

Aspects of the flipped classroom workshops that hindered learning.

Overall preference using traditional didactic lectures versus flipped classroom workshops for instruction.

Disagree

Strongly Agree

Disagree

Disagree

Strongly agree

Neutral

The information was there

Strongly agree

Easier to pay attention. Information seems more easily absorbed.

Neutral

Mind may wander sometimes

Strongly disagree

If the student did not complete the assigned homework, he/she was lost.

Greatly prefer flipped classroom workshops

Strongly Agree

Neutral

Strongly agree

Neutral

Instructor determines pace and thus can cover all material. Structured and sequential approach

Able to gauge the students' level of understanding and dispel any misconceptions. Students more comfortable asking questions.

Strongly disagree

Students less engaged. Instructor unable to gauge the students' depth of knowledge until the end

Neutral

Difficult to pace. Variations in knowledge base may have slowed the pace of instruction

Greatly prefer flipped classroom workshops

Instructor 3

Instructor 4

Disagree Agree Disagree Agree

Strongly disagree Strongly agree

Disagree

Neutral

Strongly agree

Neutral

Predictable content. Material presented clearly.

Thoroughness and consistency among different groups of students.

Strongly agree

Ability to alter teaching when student misconceptions are identified. Students are asked to apply their knowledge.

Neutral

Inability of students to ask questions or clarify understanding during the didactics.

Strongly disagree

Somewhat prefer flipped classroom workshops

Strongly agree

Require higher order cognitive skills. Varied amount of time spent on concepts depending on which concepts the particular group of students found more challenging.

Disagree

Time wasted discussing concepts already mastered, leaving less time to spend on more difficult concepts.

Disagree

Some students feel intimidated by requirement to participate

Greatly prefer flipped classroom workshops

There is evidence that engaging students in active learning exercises improves learning outcomes, motivation, and attitudes (42—44). Advances in instructional technology make online learning convenient and may also improve educational outcomes. A meta-analysis published by the Department of Education in 2010 concluded that "on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction," with larger effects seen if the online learning was combined with face-to-face instruction (45). This meta-analysis encompassed research literature on career technology, higher education, medical training, corporate training, military training, and K—12 education. Flipped learning takes advantage of technical advances to teach basic cognitive skills to students in a format that meets the style and expectations of this generation of learners, freeing instructors to use class time to foster higher order cognitive skills through active learning exercises (41). In the setting of clinical clerkships, the flipped learning approach can be used to foster the development of cognitive skills required in applying learned facts to clinical reasoning problems.

Prior nursing and pharmacy education studies comparing the use of a flipped classroom to traditional didactic lectures have shown improvements in both students' standardized test scores and positive student perceptions (25—27). Belfi et al. showed that flipped classroom instruction improved posttest performance compared to independent learning and traditional didactic lectures given to medical students (46). However, the topics taught and the instructors were different among the comparison groups, making it difficult to isolate the effects of instruction.

Factors that lead to an individual physician's career success are not well understood (47). Although medical education tends to focus on cognitive factors, academic performance has only a small effect on postgraduate medical competence (30). The influence of emotional factors such as motivation and achievement emotions has only recently received attention from medical educators (27,48,49). Task value has been shown to be a significant predictor of students' use of self-regulated learning and academic achievement in traditional school settings (50).

Subjects who participated in the flipped learning cohorts had greater interest in learning, increased enjoyment, and higher task value than the traditional didactic instruction cohorts. These positive effects of flipped learning on medical students' motivational beliefs and achievement emotions can enhance academic performance (19). Flipped learning may increase medical students' task value and positive achievement emotions for multiple reasons. First, students are expected to prepare in advance so that they can participate in exercises and contribute to discussions. This responsibility encourages students to prepare by completing assignments before attending instructional sessions. Second, exploring clinical scenarios in a flipped learning context allows medical students to develop skills needed to apply learned facts to clinical decision-making. We are encouraged that despite the added level of responsibility for medical students participating in flipped learn-

ing, they do not seem to experience more anxiety. We attribute the increased enjoyment and decreased boredom experienced by medical students in the flipped learning environments to their discovery of how recently acquired knowledge relates to patient care.

As medical knowledge and practice continue to rapidly expand and change, it is imperative for radiologists to acquire and sharpen skills using self-directed learning (1). The flipped learning approach provides medical students with the opportunity to develop self-directed learning skills while also providing opportunities to solidify already acquired knowledge and concepts through active learning strategies. Technology is often a key component of flipped learning, which is used to facilitate both independent and classroom learning activities (51). The heavily visual nature ofradiology knowledge makes videos, online educational modules, and recorded lectures ideal media for flipped learning approaches, providing students with engaging self-study materials, thereby enabling educators to use their limited class time more efficiently.

Parallels can be drawn between the use of clinical scenarios in flipped learning to develop medical problemsolving skills and the process of applying previously learned facts to make sound clinical decisions. Our conceptual framework for this study is that medical students must first build a foundation of factual knowledge and its associated large and complex technical vocabulary to subsequently begin to develop clinical problem-solving skills. That is, before students can practice clinical medicine, they must learn diagnostic and treatment skills needed to effectively apply the facts they have learned. We propose using flipped learning during radiology clinical clerkships as an approach to managing a structured transition from activities mostly involving fact learning to activities involving application of medical knowledge in patient care settings.

This study has some limitations. First, we focused on two neuroimaging teaching sessions that took place in the first 2 weeks of a 4-week course. The survey assessment of task value and achievement emotions may have been confounded by effects of concurrent non-neuroimaging teaching sessions. Second were differences in instructional time. Flipped learning workshops typically lasted 5—10 minutes longer than their traditional didactic counterpart. It is possible that this difference could have affected student performance, task value, and achievement emotions. Third was the inability to control for differences in stylistic approach by instructors. Although a script was provided to guide the instructor through both the traditional didactic lectures and the flipped learning workshops, there was likely variability among the instructors at different sites and even among the individual sessions given by each instructor. Fourth, variations in class size, ranging from 3 to 12 students per block, could have had an effect on both instruction and learning. Fifth, although we demonstrated that a flipped learning approach results in better student achievement for a radiology clerkship curriculum, the use of this instructional approach in other medical education domains remains a topic for future investigation. Finally, the experi-

mental design we employed did not allow isolation of the specific factors causing the superior achievement and more positive achievement emotions in the flipped learning group.

Although flipped learning is a growing trend at all educational levels over the last decade, greatly affecting the dynamic of how instruction is delivered, few studies have examined if it really improves educational outcomes or the students' experience. Our study demonstrates that using the flipped learning approach to educate medical students in radiology improves student academic performance compared to the traditional didactic lecture format. Flipped learning also has a positive influence on the emotional factors associated with learning, possibly resulting from the reinforcement students experience when successfully applying the knowledge base collected in the first 2 years of medical school to realistic diagnostic and treatment problems. Medical educators can have a positive effect on educational outcomes by judicious choice of educational materials and methods. Our results motivate further investigations concerning whether the use of flipped learning pedagogy improves long-term retention of knowledge, eventual academic success, and patient care outcomes.

ACKNOWLEDGMENTS

This research was supported by a grant from MedU. The content is solely the responsibility of the authors and does not necessarily represent the official view of MedU.

The authors acknowledge Harprit Bedi, MD, Carl Furhman, MD, and Beverly Hershey, MD, for their assistance in reviewing the educational materials created for use in this study. The authors express their gratitude to Adriana Cassini, Tracy E. Frazee, and Sharon Mitchell for their administrative support, and Dr. Anthony Artino and Dr. Norman Berman for their support and advice on this project.

REFERENCES

1. Spencer JA, Jordan RK. Learner centred approaches in medical education. BMJ 1999;318:1280-1283.

2. Ashley EA. Medical education—beyond tomorrow? The new doctor— Asclepiad or Logiatros? Med Educ 2000; 34:455-459.

3. Freeman S, Eddy SL, McDonough M, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A 2014; 111:8410-8415.

4. Irby DM, Cooke M, O'Brien BC. Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med 2010; 85:220-227.

5. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010; 376:1923-1958.

6. Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med 2013; 88:1407-1410.

7. Neville AJ. Problem-based learning and medical education forty years on. A review of its effects on knowledge and clinical performance. Med Princ Pract 2009; 18:1-9.

8. Straus CM, Webb EM, Kondo KL, et al. Medical student radiology education: summary and recommendations from a national survey of medical school and radiology department leadership. J Am Coll Radiol 2014; 11:606-610.

9. Colliver JA. Effectiveness of problem-based learning curricula: research and theory. Acad Med 2000; 75:259-266.

10. Nandl PL, Chan JN, Chan CP, etal. Undergraduate medical education: comparison of problem-based learning and conventional teaching. Hong Kong Med J 2000; 6:301-306.

11. Bergmann J, Overmyer J, Wilie B. The Flipped Class: Myths vs. Reality." Available at: http://www.thedailyriff.com/articles/the-flipped-class -conversation-689.php, pp 1-4. Accessed March 29, 2016.

12. Davies RS, Dean DL, Ball N. Flipping the classroom and instructional technology integration in a college-level information systems spreadsheet course. Educ Technol Res Dev 2013; 61:563-580.

13. Foertsch J, Moses G, Strikwerda J, et al. Reversing the lecture/homework paradigm using eTEACH Web-based streaming video software. J Eng Educ 2002; 91:267.

14. Fulton K. Upside down and inside out: flip your classroom to improve student learning. Learn Lead Technol 2012; 39:12.

15. Hughes H. Introduction to flipping the college classroom. 2012; 24342438. Available at: https://www.editlib.org/p/41097/?nl. Accessed November 25, 2015.

16. Sams A, Bergmann J, Daniels K, et al. "The Definition of Flipped Learning." Available at: http://www.flippedlearning.org/definition. Accessed March 29, 2016.

17. Sharma N, Lau CS, Doherty I, et al. How we flipped the medical classroom. Med Teach 2015; 37:327-330.

18. Abeysekera L, Dawson P. Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research. High Educ Res Dev 2015; 34:1-14.

19. Strayer JF. How learning in an inverted classroom influences cooperation, innovation and task orientation. Learn Environ Res 2012; 15:171193.

20. Bergmann J, Sams A. Flip your classroom: reach every student in every class every day. Eugene, OR: International Society for Technology in Education, 2012. 100 p.

21. Deslauriers L, Schelew E, Wieman C. Improved learning in a large-enrollment physics class. Science 2011; 332:862-864.

22. Bhattacharyya R. 'Flipped' classroom teaching clicks with B-Schools in India." Available at: http://articles.economictimes.indiatimes.com/2013 -09-03/news/41726825_1_isb-indian-school-classroom. Accessed October 15, 2015.

23. Lage MJ, Platt GJ, Treglia M. Inverting the Classroom: A Gateway to Creating an Inclusive Learning Environment." Journal of Economic Education, pp 31-43. Available at: http://www.academia.edu/340051/Inverting_the _Classroom_A_Gateway_to_Creating_An_Inclusive_Learning_Environment. Accessed October 15, 2015.

24. Available at: http://www.albany.edu/news/44951.php. Accessed October 15, 2015.

25. Available at: http://www.uq.edu.au/teach/flipped-classroom/docs /cs-seaboyer.pdf. Accessed October 15, 2015.

26. Prober CG, Heath C. Lecture halls without lectures—a proposal for medical education. N Engl J Med 2012; 366:1657-1659.

27. Artino AR, La Rochelle JS, Durning SJ. Second-year medical students' motivational beliefs, emotions, and achievement. Med Educ 2010; 44:12031212.

28. Millis RM, Dyson S, Cannon D. Association of classroom participation and examination performance in a first-year medical school course. Adv Physiol Educ 2009; 33:139-143.

29. Stegers-Jager KM, Cohen-Schotanus J, Themmen APN. Motivation, learning strategies, participation and medical school performance. Med Educ 2012; 46:678-688.

30. Ferguson E, James D, O'Hehir F, et al. Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree. BMJ 2003; 326:429-432.

31. Lewis PJ, Chen JY, Lin DJ, et al. Radiology ExamWeb: development and implementation of a national web-based examination system for medical students in radiology. Acad Radiol 2013; 20:290-296.

32. Eccles JS, Wigfield A. Motivational beliefs, values, and goals. Annu Rev Psychol 2002; 53:109-132.

33. Artino AR, McCoach DB. Development and initial validation of the online learning value and self-efficacy scale. J Educ Comput Res 2008; 38:279303.

34. Pekrun R. The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ Psychol Rev 2006; 18:315-341.

35. Pekrun R, Goetz T, Frenzel AC, et al. Measuring emotions in students' learning and performance: the Achievement Emotions Questionnaire (AEQ). Contemp Educ Psychol 2011; 36:36-48.

36. Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42:377-381.

37. Heiberger RM, Robbins NB. Design of diverging stacked bar charts for Likert scales and other applications. Journal of Statistical Software 2014; 57:1-32.

38. Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new "public interest". Acad Med 2010; 85:S56-S65.

39. Khanova J, Roth MT, Rodgers JE, et al. Student experiences across multiple flipped courses in a single curriculum. Med Educ 2015; 49:10381048.

40. Kim MK, Kim SM, Khera O, et al. The experience of three flipped classrooms in an urban university: an exploration of design principles. Internet High Educ 2014; 22:37-50.

41. O'Flaherty J, Phillips C. The use of flipped classrooms in higher education: a scoping review. Internet High Educ 2015; 25:85-95.

42. Freeman S, O'Connor E, Parks JW, et al. Prescribed active learning increases performance in introductory biology. CBE Life Sci Educ 2007; 6:132-139.

43. Bransford JD, Brown AL, Cocking RR. How people learn: brain, mind, experience, and school. Expanded edition. Washington, DC: National Academies Press, 2000.

44. Bonwell CC, Eison JA. Active learning: creating excitement in the classroom. 1991 ASHE-ERIC higher education reports. ERIC clearinghouse

on higher education. Washington, DC: The George Washington University. 2016 http://files.eric.ed.gov/fulltext/ED336049.pdf.

45. Means B, Toyama Y, Murphy R, et al. "Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies." October 2014. Available at: http://www2.ed .gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf. Accessed September 1, 2015.

46. Belfi LM, Bartolotta RJ, Giambrone AE, et al. "Flipping" the introductory clerkship in radiology: impact on medical student performance and perceptions. Acad Radiol 2015; 22:794-801.

47. Hamdy H, Prasad K, Anderson MB, et al. BEME systematic review: predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach 2006; 28:103-116.

48. Artino AR, Jr, Hemmer PA, Durning SJ. Using self-regulated learning theory to understand the beliefs, emotions, and behaviors of struggling medical students. Acad Med 2011; 86(10 suppl):S35-S38.

49. Artino AR, Holmboe ES, Durning SJ. Control-value theory: using achievement emotions to improve understanding of motivation, learning, and performance in medical education: AMEE Guide No. 64. Med Teach 2012; 34:e148-e160.

50. Pintrich PR. The role of motivation in promoting and sustaining self-regulated learning. Int J Educ Res 1999; 31:459-470.

51. Bishop JL, Verleger MA. The Flipped Classrom: A Survey of the Research" June, 2013. Available at: http://www.studiesuccesho .nl/wp-content/uploads/2014/04/flipped-classroom-artikel.pdf. Accessed November 25, 2015.

APPENDIX

1. It was important for me to perform well in these sessions. Strongly Disagree

Disagree Neutral

Strongly Agree

2. The sessions provided a great deal of practical information. Strongly Disagree

Disagree

Neutral

Strongly Agree

3. I was very interested in the content of thes e sessions. Strongly Disagree

Disagree

Neutral

Strongly Agree

4. Attending these sessions moved me closer to attaining my career goals.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

5. It was important for me to learn the material in these sessions.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

6. The knowledge I gained by attending these sessions can be applied in many different situations.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

7. I understand the material taught in these sessions. Strongly Disagree

Disagree

Neutral

Strongly Agree

8. I enjoy the sessions. Strongly Disagree Disagree

Neutral Agree

Strongly Agree

9. I am excited about the material taught in these sessions. Strongly Disagree

Disagree

Neutral Agree

Strongly Agree

10. I am happy I understand the material taught in these sessions.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

11. My enjoyment of the sessions makes me want to learn the material.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

12. I worry whether I will be able to understand the session material.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

13. I worry whether I am sufficiently prepared for the sessions.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

14. I worry whether the sessions' demands might be too great. Strongly Disagree

Disagree

Neutral

Strongly Agree

15. I worry whether I am sufficiently prepared for the examination on the material taught in these sessions.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

16. I am bored in the sessions. Strongly Disagree Disagree

Neutral Agree

Strongly Agree

17. My mind tends to wander in the sessions. Strongly Disagree

Disagree

Neutral

Strongly Agree

18. I often think about what else I would rather be doing during the sessions.

Strongly Disagree Disagree Neutral Agree

Strongly Agree

19. I felt intimidated during the question and answer segments of these sessions.

Strongly Disagree

Disagree

Neutral

Strongly Agree

Questions 1—6 examine task value. Questions 8—18 examine academic achievement emotions: 8—11 focus on enjoyment; 12—15 focus on anxiety; and 16—18 focus on boredom. Questions 7 and 19 are nonvalidated questions used to focus on students' perception of whether they understood the material covered and their level of comfort in asking questions during the sessions.