Scholarly article on topic 'On the Use and Misuse of Lectures in Higher Education'

On the Use and Misuse of Lectures in Higher Education Academic research paper on "Educational sciences"

CC BY-NC-ND
0
0
Share paper
Academic journal
Health Professions Education
OECD Field of science
Keywords
{Lectures / "Active learning" / "Peer discussion" / Constructivism}

Abstract of research paper on Educational sciences, author of scientific article — Henk G. Schmidt, Stephanie L. Wagener, Guus A.C.M. Smeets, Lianne M. Keemink, Henk T. van der Molen

Abstract Background The purpose of this article was to review the literature on lecture effectiveness and to suggest avenues for improvements. Methods Selective literature review with an emphasis on active learning in the classroom setting. Results Conventional lectures are effective only to a limited extent in attaining important curriculum objectives. They do not promote critical thinking; student attendance tends to be low and so is cognitive engagement; furthermore, the idea that lectures should and can cover all essential subject matter is false. Moreover, empirical literature on what students actually learn from lectures is lacking. A most fundamental problem of lectures is that they are based on the information transmission fallacy, the idea that students learn just by being told. The paper proposes an alternative approach to lecturing based on studies in teaching the natural sciences: active learning in the classroom. This approach has four key elements: (a) an initial individual learning attempt by students to master important concepts or ideas, (b) the presentation of a relevant problem by the teacher in the classroom setting, (c) elaborative activities of individual students or small groups of peers to come up with solutions to the problem, and (d) feedback of the teacher. Conclusion The available evidence suggests that active learning in the classroom setting supports and fosters learning to a much larger extent than conventional large-group teaching.

Academic research paper on topic "On the Use and Misuse of Lectures in Higher Education"

Available online at www.sciencedirect.com

crossMark ScienceDirect

Health Professions Education 1 (2015) 12-18

www.elsevier.com/locate/hpe

On the Use and Misuse of Lectures in Higher Education

Henk G. Schmidt*, Stephanie L. Wagener, Guus A.C.M. Smeets, Lianne M. Keemink,

Henk T. van der Molen

Institute of Psychology, Erasmus University Rotterdam, the Netherlands Available online 8 December 2015

Abstract

Background: The purpose of this article was to review the literature on lecture effectiveness and to suggest avenues for improvements.

Methods: Selective literature review with an emphasis on active learning in the classroom setting.

Results: Conventional lectures are effective only to a limited extent in attaining important curriculum objectives. They do not promote critical thinking; student attendance tends to be low and so is cognitive engagement; furthermore, the idea that lectures should and can cover all essential subject matter is false. Moreover, empirical literature on what students actually learn from lectures is lacking. A most fundamental problem of lectures is that they are based on the information transmission fallacy, the idea that students learn just by being told. The paper proposes an alternative approach to lecturing based on studies in teaching the natural sciences: active learning in the classroom. This approach has four key elements: (a) an initial individual learning attempt by students to master important concepts or ideas, (b) the presentation of a relevant problem by the teacher in the classroom setting, (c) elaborative activities of individual students or small groups of peers to come up with solutions to the problem, and (d) feedback of the teacher.

Conclusion: The available evidence suggests that active learning in the classroom setting supports and fosters learning to a much larger extent than conventional large-group teaching.

© 2015 King Saud bin Abdulaziz University for Health Sciences. Production and Hosting by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Keywords: Lectures; Active learning; Peer discussion; Constructivism

1. Introduction

Lecturing is the most employed tool for information transmission in higher education. A cursory look at Dutch medical education demonstrates that around 70% of the teaching activities in which instructors engage

Peer review under the responsibility of King Saud bin Abdulaziz University for Health Sciences.

*Corresponding author. Tel.: + 316 2079 5919. E-mail address: schmidt@fsw.eur.nl (H.G. Schmidt).

consists of lecturing to students.1 A teacher resides in front of a class, talks, and shows slides while students listen and take notes. Lecturing is part of a long tradition, probably even dating from the dawn of mankind. In the early days, information was shared through verbal transmission exclusively. Anthropologists argue that story telling used to be the most employed and most successful instrument for cultural transmission. It is based on the assumption that when you tell somebody something, and that person shows interest in what you are telling him, he will eventually

http://dx.doi.org/10.1016/j.hpe.2015.11.010

2452-3011/© 2015 King Saud bin Abdulaziz University for Health Sciences. Production and Hosting by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

remember it, use it, and convey it to a next generation. In early era education, before books became cheap and widely available, the teacher had no other means than simply telling students what he knew. He would literally read from his own notes, enabling students to write down everything he dictated, thereby more or less creating their own books. In the English university system a teacher is still called a "reader," or a "lecturer" ("legere," passivum: "lectus" is Latin for reading). Lectures help when student wrestle with particular difficult topics. A teacher who has the ability to explain concepts in a simple yet effective way can be a big help. Lectures can also be quite engaging if delivered by a charismatic teacher. Most of us still remember that one teacher who changed our perspective of the world. Finally, lectures have survived as a cost-effective way to instruct large numbers of students.

2. Shortcomings of lectures

2.1. Lack of student engagement

Lecturing is not without its critics however. Bligh,2 for instance suggests that lectures are generally poor if one wishes to promote critical thinking in students. Kelly and colleagues,3 observing student engagement in three types of classrooms, found engagement to be the lowest in the lecture theater. Attendance of lectures tends to be low.4 Typically less than half of the students show up, even if those who attend score higher on examinations than those absent.5 In surveys, students indicate that an important reason to attend is to get an impression of what will be asked during the examination; acquiring knowledge is usually rated as less important. Some teachers seem to reinforce such attitudes by focusing during tests exclusively on information shared with students through PowerPoint slides. The result is that students do not consult textbooks in their area. Their knowledge of the domain necessarily must become superficial and abbreviated; stuff for examinations rather than for life.

Then there is the issue of attention span. Some say that students are unable to attend to lectures for more than 15 min at a time. Although others challenge this point of view, there is a broadly shared opinion among teachers that students are less and less able to stay engaged with a lecture for longer periods of time. This expresses itself in off-task behaviors such as chatting with each other, being engaged in answering emails, consulting Facebook, twittering, etc. The teacher is forced more and more to act as a police officer restoring order than as an inspiring sage on the stage.

Charisma, an antidote to noisy lecture theaters, is usually in short supply among university teachers. Finally, lecture time is limited. A teacher cannot, in sufficient detail, discuss all subject matter to be learned. He has to summarize topics, focus on what he sees as essential, describe in approximate fashion, or select topics at the expense of others. Therefore, teachers often feel that they have insufficiently covered the material taught.

2.2. Lack of learning from lectures

But the really important question is: do students learn from lectures? And if so: what do they learn? The question of how effective lectures are in transmitting knowledge has found surprisingly few answers. This is somewhat unexpected given their ubiquitous presence in higher education and the importance attached to the quality of that kind of education. What information is available is old and suggests that lectures are about as effective in transmitting knowledge as other forms of teaching.2,6 However, most of the studies summarized by these authors involved comparisons between combinations of methods, for instance, lectures plus independent reading versus discussion plus reading. In addition, these studies employed natural classrooms rather than randomly assigned groups, leaving room for all kinds of confounds. A more relevant test of the effectiveness of lectures would be to have exactly the same information presented to students in two different teaching formats for exactly the same amount of time. From the literature only one such study could be retrieved, and a fairly old one at that. Corey 7 compared the learning of two groups of students who either attended a 25-minute lecture or studied for the same amount of time the text as delivered by the teacher. The latter group remembered significantly more information on a subsequent immediate test. In a yet unpublished experiment involving two randomly assigned groups, Arshad and colleagues 8 reached a similar conclusion. The authors suggest that there are at least four reasons why studying from text is superior to listening to a lecturer. The first is that students can read the text at their own pace, whereas during a lecture the teacher determines the pace with which the material is presented and has to be processed. Second, a text enables the rereading of materials too difficult or too complex to understand immediately. Third, while studying a text, the student can rehearse some of the materials or elaborate on them to improve memory. And fourth, while engaging in elaborative activities, there is no risk

that a lecturer will interfere with these activities by presenting new information.

Curriculum-level studies concerning the effect of lectures on learning are virtually absent. One exception is a study by Schmidt et al.1 entitled "Learning more by being taught less." They computed the number of lecture hours per week and the number of self-study hours per week of ten generations of students entering all eight medical schools in the Netherlands between 1989 and 1998. They also looked at graduation rates of these more than 14,000 medical students and at how much time they needed to complete their training. The investigators found that the more lectures student had to attend during their years in medical education, the lower the graduation rates of these students and the more time they needed to graduate. On the other hand: the more time for self-study was available, the higher the graduation rates and the shorter the time needed to graduate. The differences were by no means trivial. Graduation rates ranged from 91% for the curriculum with the fewest lectures per week to 77% for the curriculum with the most extensive lecture schedule. Study duration ranged from an average of 6.86 years for the lecture-leanest medical curriculum to 7.67 years, for the program with the largest number of lectures per week.9 Their explanation for these findings was that lectures do not add much to learning, but rather can be in the way of learning. Students having more time for self-study are in a better position to prepare themselves for examinations and, hence, do better on them and in larger numbers. A restriction to their conclusion was that the three medical schools with the leanest lecture schedules all employed small-group teaching instead. This type of teaching encourages students to study regularly and not postpone working until the examinations come close.

2.3. Why are lectures only partially effective?

A conclusion from the foregoing may be that conventional lectures in higher education do not add much to student learning and sometimes may even be detrimental to their learning. Why is this so? We believe that lecturing falls prey to what we would call the "information transmission fallacy." Implicit to conventional lecturing is the idea that information can be directly transmitted from one person to another. The other person then stores the information as communicated by the sender, and what is transmitted is remembered, provided the receiver pays attention. This is a misconception because the human mind does not work as a receiver. Students have to do something with

the information to enable them to remember and use it in the future. They have to be able to elaborate upon the information using their prior knowledge, to rephrase the information in their own words, to discuss the information with other students or with the teacher, to explain what is learned to others, to apply the information to a problem. All these activities help students storing the information in memory for long-term use. This is because our memory is constructive. We have to use what we already know about a topic to construct meaning for new information about that topic and have to use the resulting cognitive representations in a variety of settings in order to reinforce and stabilize them. This perspective on how people learn is called constructivism.10 Lectures only partially give room for this kind of active learning.

3. Solutions

3.1. How can we make better use of lectures?

In the last twenty years educators have sought to enrich lectures such that they enable the kind of cognitive activities sketched above. In particular in the natural sciences (physics, biology, mathematics) attempts have been made to improve on learning. An instructive example is the experience of Eric Mazur, a professor of applied physics at Harvard University. Mazur was quite contented with teaching freshman physics courses until he discovered that his students developed little conceptual understanding of central ideas in physics, despite of the fact that they became quite good at applying problem-solving algorithms. His discovery came after he administered the Force Concept Inventory, a standardized test designed to examine students' understanding of Newton's laws of mechanics. Mazur discovered that his students, the best in the country, were doing much poorer on this test than he expected.11 Therefore, he decided to change his teaching approach. He uses what he calls "peer instruction" (PI) as part of his lectures. He describes his approach as follows:

"A class taught with PI is divided into a series of short presentations, each focused on a central point and followed by a related conceptual question, which probes students' understanding of the ideas just presented. Students are given one or two minutes to formulate individual answers and report their answers to the instructor. Students then discuss their answers with others sitting around them; the instructor urges students to try to convince each other of the correctness of their own answer by explaining the underlying

reasoning. During the discussion, which typically lasts two to four minutes, the instructor moves around the room listening. Finally, the instructor calls an end to the discussion, polls students for their answers again, which may have changed based on the discussion, explains the answer, and moves on to the next topic.12

He and his coworkers found that the posttest-pretest gains on the Force Concept Inventory and two other physics tests more than doubled after the introduction of PI in the classroom. Other researchers report similar results 13-15 Hake, 14 for instance, reports on studies conducted in 62 introductory physics classes involving more than six thousand students. Forty-eight courses made use of what he calls "interactive engagement," which he defined as methods "designed at least in part to promote conceptual understanding through interactive engagement of students in heads-on (always) and hands-on (usually) activities which yield immediate feedback through discussion with peers and/or instructors (p.65)." 14 Fourteen could be classified as traditional, lecture-based courses. All of these courses made use of tests such as the Force Concept Inventory and similar standardized tests aimed at conceptual understanding of Newtonian mechanics. The interactive engagement courses demonstrated average normalized achievement gains (gains over pretest performance) of almost two standard deviations over the traditional courses. For the uninformed reader: this is a huge effect in statistical terms.

Finally a study by Deslauriers et al. 4 published in Science in 2011, deserves attention. It involved an experimental study, in which almost six hundred students were subdivided in two large lecture classes. The two classes were equal in terms of previous achievement, attendance to lectures, and engagement with the subject matter. One group was taught by a highly experienced teacher; the other group by a trained but inexperienced postdoctoral fellow. The experienced teacher taught the subject—electromagnetic waves—in the usual ways, employing Powerpoint to address content, adding in class-demonstrations and example problems, while students were required to do preread-ing before each class. The inexperienced instructor did not lecture. Rather, he presented the students with a series of challenging clicker questionsnote 1 and tasks

1note Clicker questions are part of an interactive technology that enables teachers to present problems to students and immediately collect and view the responses of the class as a whole. Students use remote transmitters or their own mobile phones to respond to the questions, which often have the form of a multiple choice. Software then tabulates the answers for presentation to the class as a whole (See for instance http://www.cmu.edu/teaching/clickers/pdfs/clickers-pedagogicalvalue.pdf.)

that require the students to practice physicist-like reasoning and problem solving during class time while being provided with frequent feedback. According to the authors, the goal of these activities was to have the students spend all their time in class engaged in thinking scientifically in the form of making and testing predictions and arguments about the relevant topics, solving problems, and critiquing their own reasoning and that of others. At the beginning of each class, the students were asked to form groups of two. After a clicker question was shown to the class, the students discussed the question within their groups and submitted their answer. When the voting was complete, the instructor showed the results and gave feedback. The small-group tasks were questions that required a written response. Students worked in the same groups but submitted individual answers at the end of each class for participation credit.

The results of this experiment were quite astonishing. While attendance to, and engagement in, the lectures remained the same for the control group, both indicators of commitment increased dramatically in the experimental group. Attendance grew from 57 to 75%, and engagement-as measured by trained observers using a standard protocol-increased from 45 to 85%. More importantly, students in the experimental group learned on average twice as much about electromagnetic waves as the students in the control group.

3.2. Various approaches to active learning during lectures

The previous review suggests that active learning in the classroom setting can take various forms. These forms can differ from each other on at least four dimensions: (1) when and how knowledge is acquired; (2) the format of the problems presented to students; (3) the nature of the learning activities required from students during the lecture; and (4) the extent of teacher feedback.

1. All approaches to active learning during lectures assume that students first acquire knowledge individually, either through pre-reading or through short presentations by the lecturer in-class. In the study by Deslauriers et al. 4 students were asked to study 4-5 pages of relevant materials prior to coming to the lecture. A possible shortcoming of this latter approach is that many students tend not to prepare themselves well. They wait for the teacher to tell them, because they tend to be victim of the information transmission fallacy as well. An alternative is proposed by Mazur 11 He introduces central ideas through short and

focused presentations in the classroom not requiring any prior preparation from the students and not lasting more than a few minutes each, before engaging students in an active-learning exercise. A third possibility is to have students individually study focused texts while in class. Of course, texts or short presentations can easily be replaced by internet-provided materials such as short lectures by international experts, demonstrations, or simulations. 2. The format in which problems are presented is crucial to the success of the active-learning approach. The problems should be engaging and seen as interesting, and relevant to the purpose of the course. Some teachers field-test their problems on a few students before presenting them to the class as a whole. In its simplest form it consists of a factual question with a couple of possible answers, one of which is the correct one; a multiple choice question in other words. This is in fact the most used format because it is easy to construct and enables the clicker technology to be applied. An example, taken from an upcoming PhD-thesis by University of Cape Town researcher Elmi Badenhorst, is this:

Air moves out of the lungs because:

A. the gas pressure in the lungs is less than the outside pressure.

B. the volume of the lungs decreases with expiration.

C. contraction of the diaphragm decreases the volume of the pleural cavity.

More interesting are problems that encourage students to apply what they have learned to new situations. Such problems support the transfer of knowledge, a much sought-after but not easily attained goal of higher education. A second example in which the transfer of physiology knowledge is the focus (from the same thesis):

A person is brought to the Emergency care because he is stabbed in the chest during a Saturday night drunken fight. The intrapleural space is punctured. His lung collapses because:

A. The pressure in the intrapleural space equals atmospheric pressure.

B. The negative pressure in the intrapleural space has been lost.

C. The external intercostal muscles are damaged.

D. The patient does not breathe as deeply because of pain.

Third, in medical education, diagnosis and treatment are central to the training of doctors. The difficulty of diagnosis is to distinguish between different possible diseases that have symptoms in common. Think of the many diseases that share jaundice. Jaundice can be caused by acute inflammation of the liver, inflammation of the bile duct, obstruction of the bile, hemolytic anemia, Gilbert's syndrome, or cholestasis, among others. Presenting a number of jaundice cases, each with a set of possible diagnoses, would force students to think about and discuss how each of these diseases would cause the symptoms described in the case (the pathophysiology of the diseases), and how these possible diagnoses can be distinguished.

A more demanding type of problem is the open-ended problem; not so much in terms of its production as well because the clicker technology cannot be applied and the teacher has to look for other ways to collect various responses. However, an open-ended problem has an important advantage over a multiple-choice version. It relies on recall of knowledge rather than on recognition and does not constrain thinking and reasoning processes to the same extent as the multiple-choice version. We prefer open-ended problems therefore. Our approach is to collect various responses from students after they have discussed the problem, moving among the audience with a microphone, before discussing the problem ourselves. An alternative is to have the problem discussed in the open-ended version and only afterwards present on screen several possible answers from which the students may choose. This combines the advantages of the open-ended approach with those of the clicker approach.

Essential is to seek variation in the kind of ways in which problems are presented. The internet can play an important role here again. Students tend to get bored from active learning as well. Attempting to find other ways to present a problem, and in particular looking for real-life presentations also enables students to transfer their theoretical knowledge to the real world where it counts.

3. The standard way to involve students in the active learning exercise is to have the audience subdivide itself in groups of two, three, or four. When the problem is presented students are requested to think individually about a possible solutions first, make some notes etc. This may take two or three minutes. Then, they are allowed to discuss their solutions with their peers. They are encouraged to defend their own solutions against their peers but also to listen closely to the arguments of their colleagues.

The whole exercise is about the arguments that are produced and about the thinking that produces them. Here elaboration based on one's own prior knowledge takes place and these elaborations are confronted with ideas of others. Most authors reviewed in this article agree that it is this clash of ideas that produces the deeper understanding of the topic at hand observed in the empirical studies. Small-group discussion is the most important but not the only way to reach this goal. Some teachers ask students to write, while in the lecture theater, a short response to the problem individually. These responses are subsequently assessed for credit. Other teachers would present a short, ten-item test and then check who of the students have the highest score. A fourth possibility is to have four students with different opinions defend their solution in front of the class and let the students vote for the best answer. A fifth possibility is to use pleasant competition among students as an engine for involvement. Students are asked to stand rather than to sit. Then they are presented with a number of true/false questions. Everyone who gives chooses the wrong answer has to sit down until only a few "winners" are left. Anything goes, as long as students are encouraged to engage themselves with the subject. And what has been said about the nature of the problem applies here as well: variation in approach prevents boredom.

4. Teacher feedback provided after the solutions have been defended, the votes cast, or the answers collected, should be short, not more than a few minutes, and should be well prepared. Again, the use of other media would help here. There are excellent videos available on the Internet on almost any topic. Since students have already thought and talked about the concept or idea central to the problem at hand, the feedback phase provides also an opportunity for the lecturer to further elaborate if useful.

4. Discussion

Lecturing is by far the most used didactic instrument in teaching students in higher education. The purpose of this paper was to address some of the shortcomings of this pedagogy and to suggest ways to make them more effective. Lectures seem to fall short in a number of ways: (a) lectures are poor at promoting critical thinking, (b) students attend lectures in limited numbers, and (c) while present engage themselves only to a limited extent, (d) students loose interest in the subject

matter and display all kinds of off-task behaviors, (e) students tend to concentrate their learning activities based on the lecture notes rather than the more extensive book, (f) the idea that lectures can cover most subject-matter is false, and (g) the empirical literature on what students learn from lectures is limited. The most fundamental problem of lectures is that they tend to be based on the information transmission fallacy. This is the idea that what is taught by the teacher is remembered by the student. In reality however, students do not store information as taught. They store their interpretation of what is lectured. Memory is constructive and students have to do something with what they learn in order to remember and use. Constructive activities that foster learning and remembering are: recalling and rephrasing what is learned in one' s own words, writing an account of what is learned again in one's own words, discussing subject-matter with peers, presenting subject matter to others, etc. These constructive activities, aimed at improving memory and transfer, are the core of attempts to make lectures more effective. Our paper described various approaches of active learning in the classroom, that have four elements in common: (a) an initial individual learning attempt by students to master important concepts of ideas, (b) the presentation of a relevant problem by the teacher in the classroom setting, (c) elaborative activities of individual students or small groups of students to come up with solutions to the problem, and (d) feedback of the teacher. The available evidence suggests that these activities massively foster student learning.

The reader may have deduced from the foregoing that we are of the opinion that conventional lectures have no use. This is a not entirely accurate deduction. We believe that books and other resources that can be studied at one's own pace in the taciturnity of the library or at home, generally do a better job in conveying content than a lecturer from his notes. In particular if the goal of learning is to recall information for later application and therefore requires detailed scrutiny by the learner, self-study is superior to listening and making notes.

However, the lecture theater is a good place to tell stories. A discipline does not only consist of facts and theories. It also entails a way to view and approach the world: a collection of beliefs, attitudes, a common ethic, and research practices that its contributors share; a particular history and particular stories about the scientists who made that history. In addition, science as described in books and articles, and science as actually carried out are not entirely the same thing. Articles

describe only the end product of a scientific endeavor and do so in a static and formal way. Students however deserve to hear the whole story; the story of how the researcher developed a particular hypothesis, the story of the difficulties the researcher encountered, and his or her emotions when a cherished hypothesis turned out to be false. Who can tell these stories better than the researcher him- or herself? These narratives should be told and the lecture is a good place to do just that, in particular if the lecturer knows how to tell a good story.

References

1. Schmidt HG, Cohen-Schotanus J, Van der Molen, HT, et al. Learning more by being taught less: A "time-for-self-study" theory explaining curricular effects on graduation rate and study duration. Higher Educ. 2010;60(3):287-300.

2. Bligh DA. What's the Use of Lectures?. Harmondsworth, UK: Penguin; 1972.

3. Kelly PA, Haidet P, Schneider V, Searle N, Seidel CL, Richards BF. A comparison of in-class learner engagement across lecture, problem-based learning, and team learning using the STROBE classroom observation tool. Teach. Learn. Med. 2005;17(2): 112-118.

4. Deslauriers L, Schelew E, Wieman C. Improved Learning in a Large-Enrollment Physics Class. Science 2011;332(6031): 862-864.

5. Van Berkel H, Schmidt HG. On the additional value of lectures in a problem-based curriculum. Educ. Health 2005;18(1):45-61.

6. Dubin R, Taveggia TC. The Teaching-Learning Paradox: A Comparative Analysis of College Teaching Methods. Eugene, Ore: University of Oregon. Center for the Advanced Study of Educational Administration; 1968.

7. Corey SM. Learning from lectures vs. learning from readings. J. Educ. Psychol. 1934;25(6):459-470.

8. Arshad A, Mamede S, Schmidt HG. Comparing knowledge acquisition based on lecture versus text. Riyadh, Saudi Arabia: King Saud bin Abdulaziz University for Health Sciences; 2014.

9. Schmidt HG, Cohen-Schotanus J, Arends LR. Impact of problem-based, active learning on graduation rates for 10 generations of Dutch medical students. Med. Educ. 2009;43(3): 211-218.

10. Bransford JD, Brown AL, Cocking RR. How People Learn Brain, Mind, Experience, and School. Washington, DC: National Academy Press; 2004.

11. Powell K. Spare me the lecture. Nature 2003;425:234-236.

12. Crouch CH, Mazur E. Peer instruction: Ten years of experience and results. Am. J. Phys. 2001;69(9):970-977.

13. Green J. Replacing lectures in conventional university courses by text-based flexible learning can be a rewarding experience for the lecturer and students. Biochem. Mol. Biol. Educ. 2005;33(3): 205-207.

14. Hake RR. Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Am. J. Phys. 1998;66(1):64-74.

15. Huxham M. Learning in Lectures: Do'Interactive Windows' Help? Act. Learn. High. Educ. J. Inst. Learn. Teach. 2005;6(1): 17-31.