Scholarly article on topic 'Technology-Enhanced Formative Assessment: A Research-Based Pedagogy for Teaching Science with Classroom Response Technology'

Technology-Enhanced Formative Assessment: A Research-Based Pedagogy for Teaching Science with Classroom Response Technology Academic research paper on "Educational sciences"

Share paper
OECD Field of science

Academic research paper on topic "Technology-Enhanced Formative Assessment: A Research-Based Pedagogy for Teaching Science with Classroom Response Technology"

J Sci Educ Technol (2009) 18:146-162 DOI 10.1007/s10956-008-9140-4

Technology-Enhanced Formative Assessment: A Research-Based Pedagogy for Teaching Science with Classroom Response Technology

Ian D. Beatty • William J. Gerace

Published online: 7 January 2009

© The Author(s) 2008. This article is published with open access at

Abstract Classroom response systems (CRSs) are a promising instructional technology, but most literature on CRS use fails to distinguish between technology and pedagogy, to define and justify a pedagogical perspective, or to discriminate between pedagogies. Technology-enhanced formative assessment (TEFA) is our pedagogy for CRS-based science instruction, informed by experience and by several traditions of educational research. In TEFA, four principles enjoin the practice of question-driven instruction, dialogical discourse, formative assessment, and metalevel communication. These are enacted via the question cycle, an iterative pattern of CRS-based questioning that can serve multiple instructional needs. TEFA should improve CRS use and help teachers ''bridge the gap'' between educational research findings and practical, flexible classroom strategies for science instruction.

Keywords Classroom response system • Pedagogy • Educational technology • Formative assessment

A classroom response system (CRS) is technology that helps an instructor pose questions and poll students' answers during class. It consists of a set of input devices for students, communicating in some way with software

I. D. Beatty • W. J. Gerace

Scientific Reasoning Research Institute, University of Massachusetts, Amherst, MA, USA

Present Address:

I. D. Beatty (El) • W. J. Gerace

Department of Physics & Astronomy, University of North Carolina, 321 Petty Bldg., Greensboro, NC 27402-6170, USA e-mail:

running on the instructor's computer. After the instructor has posed a question, students can key their responses into their input devices, and the software collects the responses, aggregates them, and displays to the class a bar chart showing the number of students selecting each response (Abrahamson 2006; Banks 2006; Beatty 2004; Fies and Marshall 2006). Several modern commercial CRSs exist, generally similar in their core functionality (Burnstein and Lederman 2003). They use simple handheld keypads— colloquially called ''clickers''—as student input devices, sending data to the instructor's computer via infrared or radio-frequency signals. All CRSs permit students to select a response to a multiple-choice question, and some also permit numeric or free-text answers. Individual student responses remain anonymous to the class, though most systems allow an instructor to look them up. Some CRSs provide additional capabilities and convenience features. Common synonyms for CRS include ''classroom communication system,'' ''audience response system,'' ''student response system,'' ''voting machine,'' and, colloquially, ''clicker'' system.

CRS use has grown dramatically over the last 15 years, and is rapidly becoming mainstream in US universities. According to Abrahamson (2006):

Today, at almost every university in the USA, somewhere a faculty member in at least one discipline is using a response system in their teaching... Amazingly, these generally somewhat primitive tools are used in just about every discipline taught. Arguably, not since the overhead projector, has a piece of technology received such widespread acceptance as an aid to classroom teaching. (p. 2)

First adopted by innovative ''bleeding edge'' teachers, CRSs are now an officially supported instructional

technology at many universities. Clickers are often sold at the campus bookstore. As of Spring 2004, approximately 8,000 clickers were in use at the University of Massachusetts at Amherst, and 6,000 at the University of Colorado Boulder (Duncan 2005). CRS adoption in K-12 classrooms is more difficult to gauge; manufacturers claim many customers, but the market is so huge that penetration is still tiny. In early 2005, Abrahamson (2006) conducted a web search and found over 3,000 K-12 schools using CRSs.

Given the popularity and widely claimed promise of CRS technology, we submit that how it is being employed and how it could most effectively be employed are vital research questions. Unfortunately, little work has been done to answer them. The literature on CRSs tends to fall into three general (and often overlapping) categories: introductions to the technology, with advice for new or potential adopters (e.g., Caldwell 2007; Duncan 2006; Herreid 2006; Johnson and McLeod 2004); reports of individual efforts to teach with CRSs, largely anecdotal, sometimes buttressed with limited data (e.g., Barnett 2006; Burnstein and Lederman 2001; Draper and Brown 2004; Herreid 2006); and compilations of recommendations and ''best practices'' (e.g., Caldwell 2007; Duncan 2005; Garner et al. 2008; Wieman et al. 2008). Almost all of the literature conflates technology with pedagogy, treating ''CRS use'' as if it were a pedagogy and forgetting that like any tool, a CRS may be used in many possible ways for many possible ends. In a 2006 review of CRS literature, Fies and Marshall acknowledge this when they argue that ''Missing from current CRS research reports are... CRS use in connection with diverse pedagogical approaches'' (p. 106).

To the extent that some reports do explicitly identify a pedagogical approach, they fail to articulate their theoretical frame or situate themselves within the larger educational research literature. In a 2004 review of CRS literature, Roschelle et al. (2004b) said:

None of the available studies rises to the present specification of ''scientifically based research'' that would allow inferences about causal relationships or that could form the basis for estimating the magnitude of the effect. Our review found that existing research does not connect with the larger research base in education or psychology, which could be used to create an explanatory theory or model. (p. 3)

In our opinion, even these insightful reviewers are not as careful as they could be to unpack the role of pedagogy in CRS-based instruction. Fies and Marshall asserted that ''Missing from current CRS research reports are. Tightly controlled comparisons in which the only difference is the use, or lack of use, of a CRS'' (p. 106). This strikes us as similar to asking whether a house made with a nail-gun is

better than a house made with a hammer, given identical blueprints and materials. We argue that tools should be evaluated on their affordances, whereas approaches and methodologies should be evaluated on their student impacts. In other words, don't ask what the learning gain from CRS use is; ask what pedagogical approaches a CRS can aid or enable or magnify, and what the learning impacts of those various approaches are. Even though such a separation may not always be possible in practice (e.g., when a particular approach is not practicable without a CRS), we believe that maintaining the conceptual distinction helps one to identify and articulate choices and perspectives that might be critical to identifying causal factors.

Meanwhile, Roschelle et al. (2004a) indiscriminately lumped various authors' use of CRSs and other ''classroom networks'' together into one approach, which they label ''classroom aggregation technology for activating and assessing learning and your students' thinking'' (CATAA LYST), ''a title that refers both to the enabling technology and the pedagogy it supports'' (p. 8). Note their use of the singular case for ''the pedagogy.'' We believe that very significant differences in pedagogical philosophy, methods, and objectives exist between various practitioners' CRS-based instruction, and illuminating these will be critical to understanding and optimizing the benefits CRSs can offer.

We are aware of only three separate efforts to present and justify an explicit, coherent pedagogy for CRS-based teaching. One is Eric Mazur's Peer Instruction (Mazur 1997), so widely known that CRS use is sometimes casually called ''peer instruction.'' Mazur suggested regularly inserting CRS-administered ConcepTests—multiple-choice conceptual questions about the material being taught—into the lesson at strategic junctures. If a significant number of students answer incorrectly, the class is asked to discuss the question among themselves and then answer again. Mazur found that this methodology increases student engagement, improves learning, provides the instructor with feedback about student understanding, and promotes knowledge ''diffusion'' between students. Quantitative evidence, primarily from pre/post testing with the Physics Force Concept Inventory (Hestenes et al. 1992), supports his assertion that Peer Instruction improves student understanding (Crouch and Mazur 2001; Fagen et al. 2002; Mazur 1997).

Independently of Mazur, the University of Massachusetts Physics Education Research Group (UMPERG) developed a superficially similar CRS pedagogy. At first unnamed (Dufresne et al. 1996; Wenk et al. 1997), they later referred to it as Assessing-to-Learn (A2L; Dufresne et al. 2000) or Question-Driven Instruction (Beatty et al. 2006). Dufresne et al. (1996) defined four broad educational objectives of the A2L approach:

(1) Students should know and understand definitions, terminology, facts, concepts, principles, operations, and procedures; (2) Students should be able to communicate what they know to others; (3) Students should know how to apply what they have learned to analyze situations and solve problems, extending this ability to increasingly complex situations; and (4) Students should develop the ability to evaluate critically the usefulness of various problem-solving approaches. We do not take for granted that students will acquire or enhance these habits of mind working independently outside of class. (p. 12)

Dufresne et al. (2000) later elaborated on these goals, identifying 12 specific ''beneficial habits of mind'' that A2L aims to help students develop, as well as five ''stages of cognitive development'' A2L practitioners should address in order to help students assemble well-structured, robust, transferable knowledge structures and problemsolving strategies. They also explicitly connected A2L with formative assessment, arguing that the practice of A2L ''informs teachers about what students think; it informs students what their classmates think; it informs individuals what they themselves think'' (p. 11). The primary mechanism by which A2L addresses these goals is a ''question cycle,'' an iterative pattern of having students read or hear a question, think about it alone and/or discuss it in small groups, enter responses, view the chart of response counts, present and discuss arguments for various choices, and then listen to an apropos micro-lecture or other ''closure'' to the cycle. An important difference between Peer Instruction and A2L is that Mazur's ConcepTests are intended for intermittent insertion within more traditional instruction in order to enhance and guide that instruction, whereas the A2L question cycle is intended as the basic structure of class activity and engine of learning, with ''microlectures'' or other direct instruction inserted when needed and motivated by the questions and discussion.

More recently, the The Ohio State University's Physics Education Research Group has been developing a methodology for CRS use focused on sets of related questions working together to target specific instructional objectives (Reay et al. 2006, 2008). They argued that ''Single questions provide limited assessment of whether students are able to make desired connections and transfer their understanding across contexts'' (Reay et al. 2008, p. 171). They acknowledged that the general idea of using coordinated question sets as a coherent instructional unit is not new, but have significantly elaborated the strategy by proposing and extensively testing two specific design patterns for such sets. One pattern, called easy-hard-hard, is a series of three questions about the same concept. The first question is an easy ''warm-up'' designed to build

confidence, and typically requires little discussion. The second question is difficult: it pushes the limits of students' understanding of the concept, and is intended to elicit a broad spectrum of answers and lead to extensive discussion. The third question is also difficult, in the same way as the second, but with different surface features. It reveals to students and the instructor whether students have learned what they were intended to from the second question by checking whether they can transfer it to a different context. The group has developed a second question-set methodology that they call rapid-fire, in which a series of moderately difficult questions present one concept in a variety of contexts. By comparing pre/post test results and exam performance between a traditional, non-CRS class and an otherwise equivalent class that regularly used a CRS with question sets carefully crafted according to these patterns, they found that ''students using voting machines and discussing solutions with each other during voting achieved a small but significant gain in conceptual learning,'' and that ''using voting machines reduces the gap between male and female student performances on tests'' (Reay et al. 2008, p. 178).

Of these three published pedagogies for teaching with a CRS, only A2L is explicitly linked to a research-based pedagogical perspective (Dufresne et al. 1996), and one can argue that many of the details of A2L were insufficiently defended. Noting a general lack of connection between publications about CRS use and the broader knowledge base in educational research, Roschelle et al. (2004a, b) attempted to isolate the significant features of CRS-based instruction reported in the literature, and then connect those to established constructs from educational research. While this is helpful for framing future analysis of CRS-based instruction, it does not fill the need for systematic efforts to define, ground, justify, and thoroughly explicate coherent pedagogies for teaching with a CRS.

We do not mean to imply that the existing literature on CRS-based instruction is without merit. On the contrary, it has played a crucial role in facilitating explosive growth in CRS use. It has also documented that use, leading Roschelle et al. (2004b) to assert that ''This body of evidence, taken together, is suggestive of a real and important phenomenon at hand'' (p. 3). However, we believe that additional progress in CRS-based instruction will come by turning the critical lens of scholarly research to the pedagogies such instruction involves, with a fine eye towards differences between them. The first step is for practitioners, evaluators, and evangelists of CRS-based instruction to articulate their pedagogical perspectives and methods, in as much detail as possible. Publishing such pedagogies will benefit researchers, instructors, and teacher professional development experts, as well as technologists working on future CRS systems.

To that end, this paper defines and details technology-enhanced formative assessment (TEFA), a rich and flexible pedagogical approach we have developed for teaching science and mathematics with the aid of a CRS. TEFA has evolved from, and supersedes, A2L. We attempt to ground TEFA solidly and broadly in literature from multiple complementary educational research traditions; to identify core, guiding principles for robustness across contexts and for resistance to "shallow" implementations; to provide a concrete classroom activity pattern by which TEFA can be realized and benefits reaped by even relatively novice practitioners; and to explore the multiplicity of ways it can be employed in the classroom. Our target audience is educational researchers, teacher professional development staff, curriculum developers, and others interested in a highly theoretical discussion of pedagogy. Practical advice, concrete examples, and discussion of teachers' learning of TEFA have been and will continue to be addressed in other writings.

Grounding TEFA

TEFA is both theoretically and empirically grounded. It is theoretically grounded in several educational research traditions and perspectives, and empirically grounded in the combined experience we and our close colleagues have accumulated through many years of practicing, mentoring, studying, and reflecting upon CRS-based instruction. It was conceived, grown, and refined over time in the crucible of an ongoing interaction between practice, research, and encounter with the findings and thoughts of other researchers and instructors. Thus, TEFA can neither be purely deduced as a necessary result of accepted pedagogical theory, nor purely defended on the strength of experimental findings. Instead, we submit it for consideration as a comprehensive, parsimonious, and productive pedagogical vision: a vision that is consistent with established thinking in educational research, and is credible because of our history and because of its consonance with the perspectives of other CRS researchers and reflective users.

Empirical Basis

Classtalk was the first modern, commercially available CRS (Abrahamson 2006). In 1993, we and our colleagues in the UMass Physics Education Research Group (UMPERG) began using Classtalk at UMass. Shortly thereafter, instructors from other departments began using our Classtalk system with our support and mentoring. From 1994 to 1997, UMPERG spearheaded a multi-university research project to refine and test Classtalk in collaboration

with the system's creator, Better Education Inc.1 As part of this project, we applied then-current knowledge from physics education research in order to develop pedagogical methods and curriculum for teaching university physics with a CRS (Dufresne et al. 1996; Mestre et al. 1997; Wenk et al. 1997). In 1999, the UMass Amherst Department of Biology received funding from the Pew Foundation to redesign their entire introductory biology course sequence, including making CRS use (first Classtalk, later Interwrite PRS) integral to lectures (Phillis 2005). The project was highly successful, and inspired a subsequent UMass Amherst project to introduce CRS use to large lecture courses in five new academic departments and to establish Interwrite PRS as an officially supported learning technology on campus. Members of UMPERG have continued to advise and mentor UMass Amherst instructors in the use of a CRS, both informally and through programs like the PRS Best Practice Fellows working group (Garner et al. 2008).

In a subsequent project from 1998 to 2003, we and our UMPERG colleagues introduced CRSs to high school physics teachers and helped them develop sound pedagogical practices and effective curriculum for use with it.2 During this time we continued to extend, refine, and formalize our pedagogical approach to CRS-based instruction (now named Assessing-to-Learn, A2L), and we learned that it can be as effective in a high school classroom as a university lecture hall (Dufresne and Gerace 2004; Dufresne et al. 2000, 2001; Leonard et al. 2001). We also began studying and building models of the teacher learning process and the difficulties teachers have in adopting CRS technology and the A2L approach.

In an ongoing project that began in 2005, we and our colleagues are conducting intensive, sustained, on-site professional development (PD) programs with over 40 middle- and high-school teachers from three school districts, with the aim of helping them master the evolving CRS-based pedagogical approach that we now call technology-enhanced formative assessment (TEFA).3 This PD program serves as a context for longitudinal research on inservice science teachers' learning and pedagogical change. In support of the project's PD component and in response to research findings, we continue to refine and articulate the TEFA pedagogy. Preliminary findings from the project indicate that TEFA can be highly effective, even transformative, for secondary school science and mathematics instruction (Beatty et al. 2008). Although TEFA is unique,

1 TTECCS: Transforming Technical Education with a Classroom Communication System, U.S. National Science Foundation grant DUE-9453881.

2 A2L: Assessing-to-Learn Physics, U.S. National Science Foundation grant ESI-9730438.

3 TLT: Teacher Learning of Technology-Enhanced Formative Assessment, US National Science Foundation grant TPC-0456124.

it shares common elements with many other ways of using a CRS that have been described in the literature (see reviews in Fies and Marshall 2006; Roschelle et al. 2004a, b). Thus, the general instructional efficacy reported for CRS use can be interpreted as additional, albeit indirect and partial, evidence in support of TEFA.

Theoretical Basis

TEFA is rich and multifaceted, and addresses many aspects of teaching science. Our perspective, therefore, incorporates multiple research traditions. For insight into the cognitive dimension of learning, we turn to constructivism, the conceptual change tradition, and recent thinking about how knowledge is accessed. For insight into the social dimension, we turn to the sociocultural tradition and work on the role of discourse in science teaching. For insight into the psychological dimension, we turn to work on student attitudes, motivation, and self-regulation.

The Cognitive Dimension

Teaching science largely means developing and refining students' understanding of the concepts of science; consequently, constructivism and the conceptual change research tradition are central to our perspective. The term ''constructivism'' has different meanings to different people; we subscribe to radical constructivism, as developed independently by Heinz von Foerster (1981) and Ernst von Glasersfeld (1981). To us, the epistemological content of that outlook most relevant here can be concisely summa-rized—though perhaps oversimplified—by four premises (articulated in Gerace 1992): (a) knowledge is constructed, not transmitted; (b) the construction of knowledge requires purposeful and effortful activity by the learner; (c) prior knowledge impacts the learning process; and (d) initial understanding is local, not global. These premises align with the fundamental insights into the conceptual learning of science shared by the majority of cognitive perspectives (as summarized in Scott et al. 2007).

Following Posner et al. (1982), we see learning science as a complex process of growth and reorganization of an individual student's conceptual ecology. The development of scientific understanding requires integrated change to an interlocking set of ideas, and pre-existing beliefs must be explicitly addressed. Since experts and novices differ in the organization, not just the extent, of their knowledge (Chi and Glaser 1981; Glaser 1992; Larkin 1979), a significant focus of science instruction must be helping students to use, reflect upon, and appropriately structure and restructure their knowledge.

Recent research suggests that when students seem to exhibit missing knowledge, they frequently possess the

requisite knowledge elements but fail to activate them in response to the context at hand (Dufresne et al. 2005; Hammer et al. 2004; Redish 2003; Scherr 2007). We are therefore concerned with the nature and span of the contexts in which students explore new science knowledge, and value learning experiences in which students must search their accumulated knowledge, weighing alternatives and strategizing an approach. Furthermore, what knowledge students even attempt to invoke while problemsolving, or seek to develop while learning, is strongly constrained by their epistemological framing of a situation (Elby 2001; Hammer and Elby 2003). A person's frame of the moment is their answer to the questions ''What is it that's going on here, and what should I be trying to do?'' Consequently, we see attending to how students frame their participation in learning and interacting with their framing process as vital and foundational to effective instruction.

The Social Dimension

Within the conceptual change tradition, the primary role of classroom discourse is to direct students' thinking and provide material to think about. The sociocultural research tradition identifies other crucial roles for language in science instruction. Carlsen (2007), extending Sutton's (1998) work, articulated three distinct ways of conceptualizing language in science and science teaching: as a system for transmitting information, as an interpretive system for making sense of experience, and as a tool for participation in communities of practice. Effective instruction should help students develop facility with all three.

According to Bakhtin (summarized in Wertsch 1991, pp. 93-118), learning science or mathematics involves developing fluency in the social language of the discipline: the language, concepts, norms, and genres for communication used by the discipline's practitioners. The assemblage of social languages that a person knows comprises a ''toolkit'' of ways of knowing and thinking. The nature of scientific social languages is qualitatively different from the nature of everyday social languages, in both ontological and epistemological ways (Carlsen 2007; Mortimer and Scott 2003); science social languages can view phenomena very differently from the ways everyday ones do, leading to what researchers in the cognitive change tradition would call ''misconceptions'' or ''preconceptions'' as well as to a disconnect between real life and ''what we learn in the classroom.'' To learn science, students need scaffolded practice speaking its social languages in order to develop fluency, and they need help recognizing and resolving conflicts between alternative social languages.

Lemke (1990) asserted that ''learning science means learning to talk science'' (p. 1). He sees the content of science curricula as thematic patterns, networks of

semantic relationships between words and their corresponding meanings, which are learned through discourse experiences. Seen from this point of view, learning the ideas of science absolutely requires dialogue about those ideas, including opportunities for students to try putting together words that suit their tentative, vague initial sense of the thematic patterns.

For Vygotsky (1987), ideas are first encountered and ''rehearsed'' through communication on the social plane. As an individual reflects upon and makes sense of these, the social tools for communication become internalized to the internal plane and provide the means for individual thinking. This suggests that a major role for talk in the science classroom is to make available conceptual and linguistic tools for thinking scientifically and give students space to experiment with and internalize their use.

Based on such thinking as well as their own research, Mortimer and Scott (2003) have provided a framework for analyzing classroom discourse. They suggest two dimensions for characterizing the communicative approach taken by the teacher during an episode of talk: a continuum from non-interactive to interactive describing the diversity of participants, and a continuum from authoritative to dialogic describing the diversity of ideas. According to Scott and Mortimer (2006), ''dialogic interactions are notably absent from science classrooms around the world'' (p. 2). This is unfortunate, they claimed, because ''any sequence of science lessons, which has as its learning goal the meaningful understanding of scientific conceptual knowledge, must entail both authoritative and dialogic passages of interaction'' (p. 2). Thus, guided by work from the sociocultural research tradition, we take the practice of genuinely dialogic discourse to be vital for effective science instruction.

The Attitudinal Dimension

Understanding what motivates students to learn and how to shape their learning behaviors is also important, so we turn to research on student attitudes, motivations, and self-regulation. Koballa and Glynn (2007) identified four general theoretical orientations within the research literature on student motivation: behavioral, which focuses on incentive and reinforcement; humanistic, which focuses on students' personal growth and desire to self-actualize; cognitive, which focuses on students' goals, plans, expectations, and attributions; and social, which focuses on students' identities and interpersonal relationships. The cognitive, humanistic, and social orientations all inform our perspective to some extent, but the cognitive orientation is dominant, in that it grounds our thinking about the mechanisms of motivation and behavior.

Most research about student attitudes in science education has focused on attitudes towards science. We are

more urgently interested in attitudes towards learning, instruction, and classroom behavior, so that we may constructively influence how students frame and participate in classroom activity. However, we expect the underlying psychological dynamics to be largely the same. The theory of reasoned action (Koballa and Glynn 2007) posits that beliefs determine attitudes and attitudes shape behavior. This means that in order to influence students' classroom behavior—for example, to change the ways they ponder questions or participate in discussion—we should seek to elicit, interact with, and influence their underlying beliefs.

According to Koballa and Glynn, a key construct for understanding student motivation is intrinsic motivation. ''Motivation to perform an activity for its own sake is intrinsic, whereas motivation to perform it as a means to an end is extrinsic'' (p. 89). Their literature synthesis found five factors that influence students' degree of intrinsic motivation: teacher expectations, goal-directed behavior, self-determination, self-regulation, and self-efficacy. A synthesis of research on self-regulation in academic learning by Schunk and Ertmer (2000) cited evidence that students' development of self-regulated learning could be improved by directing them to focus on process goals rather than performance goals, and by explicitly modeling and teaching strategies for self-regulation. Thus, we stress forming and communicating positive and constructive teacher expectations; providing students with the opportunity, scaffolding, and reinforcement to grow in the other four factors; focusing on process goals; and explicitly communicating about strategies for self-regulation.

Synthesis of Research on Effective Learning Environments

How People Learn (Bransford 1999) is a landmark synthesis of research on learning and instruction, commissioned by the US National Research Council. The authors argued that effective learning environments should be student-centered, knowledge-centered, assessment-centered, and community-centered (ch. 6). A student-centered learning environment treats students as individuals, coaching them from their varied initial states to the intended learning goal by whatever unique trajectory each requires, taking into account their initial knowledge and perceptions, their culture, their language use, and their ongoing and very personal process of sense-making. ''The term also fits the concept of 'diagnostic teaching': attempting to discover what students think in relation to the problems on hand, discussing their misconceptions sensitively, and giving them situations to go on thinking about which will enable them to readjust their ideas'' (pp. 133-134). ''Learner-centered teachers also respect the language practices of their students because they provide a basis for further learning'' (p. 135).

A knowledge-centered learning environment treats knowledge not as a collection of ideas, facts, and skills, but rather as a rich, interconnected structure that must be organized and refined as it is expanded. ''Knowledge-centered environments also include an emphasis on sense-making—on helping students become metacognitive by expecting new information to make sense and asking for clarification when it doesn't'' (p. 137).

An assessment-centered learning environment weaves formative assessment deeply into the fabric of instruction, providing continual, detailed feedback to guide students' learning and instructors' teaching. ''Given the goal of learning with understanding, assessments and feedback must focus on understanding, and not only on memory for procedures or facts'' (p. 140). In a nod to CRSs (as well as other instructional technologies), the authors note that ''Teachers have limited time to assess students' performances and provide feedback, but new advances in technology can help solve this problem'' (p. 142).

A community-centered learning environment recognizes that students belong to communities of co-learners at the course, program, institution, and society levels, and promotes constructive interaction between individuals to further learning. ''At the level of classrooms and schools, learning seems to be enhanced by social norms that value the search for understanding and allow students (and teachers) the freedom to make mistakes in order to learn'' (p. 145).

Taken together, these four qualities form an analytical framework or lens that can help us evaluate instructional environments for their alignment with education research. They do not, however, prescribe what instructors should do to produce these qualities; for that, we need a pedagogy. With TEFA, we seek to construct a pedagogy that provides teachers with one way (out of many conceivable ways) of building a learning environment that possesses all four qualities in abundance. We wish to bridge the gap between the four qualities framework and the decisions a classroom teacher must make on a daily basis.

Defining TEFA

Technology-Enhanced Formative Assessment (TEFA) is our pedagogy for teaching science with a classroom response system. TEFA is an extension, refinement, and formalization of the Assessing-to-Learn pedagogy described above. We have four criteria for constructing our pedagogy:

1. It should be concrete and directly implementable. That is, it should indicate specific actions and practices that teachers can do in their classrooms in order to foster a desirable learning environment.

2. It should be principle-based. That is, it should not be an assortment of rules, recipes, tips, or ''best practices,'' but rather a coherent system organized around and growing from a few general, flexible core principles. These principles should be cast as imperative statements about what a teacher ought to do, not as declarative statements about learning and teaching or about the standards that instruction ought to meet.

3. It should realize extant wisdom. That is, it should connect to and be consonant with research findings about learning, teaching, and the qualities of effective learning environments.

4. It should work. That is, a significant fraction of teachers who attempt to implement it in real-world circumstances should have successful, reinforcing experiences.

In this section we describe TEFA, hopefully in sufficient detail for a reader to verify that it fulfills the first three of our criteria. Addressing the fourth criterion is beyond the scope of this paper, and will be left to existing and forthcoming publications (e.g., Beatty et al. 2008). Here we will identify the instructional goals that TEFA has been designed to help a teacher accomplish, introduce TEFA's four core principles and connect them to our literature synthesis, present the primary mechanism by which these principles are enacted in the classroom with the help of a CRS, and discuss the spectrum of roles and ends this mechanism can serve within instruction.

Instructional Goals of TEFA

TEFA has been crafted for two general purposes: to help students develop expertise in science content, and to help prepare students for future learning (Bransford and Schwartz 1999; Schwartz and Martin 2004). We believe these goals go hand-in-hand. In alignment with the conceptual change literature, we seek to help students grow contextually robust, transferable conceptual ecologies that are thoroughly reconciled with their experiences, perceptions, and prior understandings. In alignment with the sociocultural learning literature, we try to accomplish this largely by engaging students in extensive dialogical discourse about scientific ideas and their applications, set within the context of rich and challenging questions and problems. In alignment with the student motivation literature, we aim to explicitly confront students' beliefs and attitudes, communicate high teacher expectations, and scaffold self-directed, self-regulated learning habits. We endeavor to help students recognize and seek well-structured knowledge, to participate in productive modes of discourse, and to attentively self-regulate their learning, in order to both facilitate deep learning of the content at hand and develop long-lasting and beneficial learning habits.

The Four Principles of TEFA

At the heart of TEFA lie four key principles. An instructor's implementation of TEFA is of low fidelity unless it aligns well with these principles; mechanically following TEFA's patterns and methods is insufficient. The four principles are:

1. Motivate and focus student learning with question-driven instruction.

2. Develop students' understanding and scientific fluency with dialogical discourse.

3. Inform and adjust teaching and learning decisions with formative assessment.

4. Help students develop metacognitive skills and cooperate in the learning process with meta-level communication.

Together, these four principles realize the instructional implications of the research summarized earlier.

Question-Driven Instruction

The first TEFA principle is ''Motivate and focus student learning with question-driven instruction'' (QDI). QDI positions learning within the context of students' encounter with questions—often conceptually rich, meaty, messy, challenging ones—to provide context, motivation, and direction to students' sense-making efforts. (We define ''question'' broadly enough to include, for example, a ''problem'' to be solved.) Questions are used to set up fertile learning situations and to catalyze learning, not just to assess previous instruction or gather data to inform future instruction. Wiliam (2007) has called this ''assessment AS learning,'' to distinguish it from summative assessment (''assessment OF learning'') and formative assessment (''assessment FOR learning''). Questions should drive the instruction itself, occurring throughout an instructional sequence: at the outset, in order to motivate it and create a fertile context for students to refer to during subsequent learning; during the development of new material, in order to explore it and connect it to other material; during the elaboration of material, in order to challenge the limits of students' understanding, find gaps and flush out misunderstandings, and interconnect ideas with increasing sophistication; and at the end, to support students' self-assessment and develop strategic thinking skills. ''Direct instruction'' is inserted into and after questions when students are primed to attend to it and process it most productively. QDI aligns well with active learning (Bonwell and Eison 1991; McNeal and D'Avanzo 1997) and with problem based learning (Chin and Chia 2004).

According to How People Learn (Bransford et al. 1999), ''Ideas are best introduced when students see a need or a

reason for their use—this helps them see relevant uses of knowledge to make sense of what they are learning'' (p. 139). The principle of QDI is motivated by many of the research threads we identified earlier, but literature in what we have termed the cognitive dimension speaks to it most strongly. Radical constructivism and the conceptual change tradition paint a picture of learning as an effortful and protracted struggle to construct, extend, and reorganize a complex, interlinked conceptual ecology that can successfully support a varied range of thinking tasks. The idea of learning as sense-making is central to this picture, and sense-making is inherently a process of working out answers to questions that challenge one's ability to explain something. Carefully chosen questions can call students' attention to gaps in their understanding, raise dilemmas for them to wrestle with, and challenge the limits of their context-dependent knowledge. The limitations of knowledge are revealed only when it is applied. The importance of confronting or building on students' prior knowledge and pre-existing beliefs suggests that one should elicit those early on, in order to address them during subsequent learning. Questions provide a mechanism for so doing. Also, the conceptual change tradition emphasizes the importance of having students reflect upon and structure their knowledge, and appropriate questions can direct such reflective thinking.

The research on when and how students access their knowledge, and how their epistemological framing of tasks affects this, highlights the importance of context for learning and instruction. If learning and using knowledge are both heavily shaped by the context of the moment, we must attend carefully to the contexts we create for students. Putting questions ''up front'' sets a context, and a range of questions can be employed to deliberately vary context in order to develop robust, less context-bound, more trans-ferrable knowledge. Furthermore, placing learning in the context of question-answering helps to frame knowledge as ''stuff to answer questions with'' and learning as ''figuring out how to answer questions,'' which pushes against students' deeply trained inclination to frame knowledge as ''facts and procedures to remember'' and learning as ''storing what I'm told so I can recall it later.''

In addition, QDI is motivated by work in the social dimension. One of the roles identified for language in science is as ''an interpretive system for making sense of experience'' (Carlsen 2007, p. 68). As we argued above, sense-making is inherently question-driven. Vygotsky's (1987) description of language-mediated learning as assimilating social tools for communication to the internal plane, where they provide means for individual thinking, implies that something is being discussed on the social plane and thought about on the individual plane; appropriate questions motivate and create context for such

discussion and thinking. Similarly, Lemke's (1990) assertion that learning science means learning to talk science implies that the thematic patterns constituting scientific knowledge are being expressed in the process of applying them to something, which almost always means answering a question. In general, any view that holds discourse as central to learning can motivate QDI, since questions are the obvious means to drive discourse.

QDI is also motivated by the attitudinal dimension, albeit indirectly. In keeping with the theory of reasoned action, we want to influence students' attributions about science and learning. Using QDI implicitly but strongly frames science as a process of figuring out answers rather than of amassing facts and skills, and simultaneously frames learning as a process of sense-making and searching for answers. To support students' development of intrinsic motivation, extensive wrestling with questions can provide objects for goal-directed behavior; can provide space for the exercise of self-determination (e.g., in the selection of strategies and in deciding whether to engage); and can scaffold the development of self-regulation. It supports the development of self-efficacy by allowing students to wrestle with a task they do not initially know how to complete, but can eventually succeed at.

Note that suitable questions must be chosen in order for QDI to fulfill the various roles indicated above. Questions must often be difficult: not necessarily computationally or procedurally difficult, but deeply challenging to students' understandings. To help students construct, extend, and reorganize a complex, interlinked conceptual ecology, questions must often draw together multiple ideas. To help develop robust access to knowledge across contexts, ideas must be revisited in multiple contexts. To support scientific discourse and language practice, questions should be disputable (as opposed to ''you know it or you don't'' styles), and should generally be qualitative in order to facilitate linguistically rich discussion.

Dialogical Discourse

The second TEFA principle is ''Develop students' understanding and scientific fluency with dialogical discourse'' (DD). By this, we mean arranging to have students participate extensively in whole-class and small-group discussions wherein multiple perceptions, ways of thinking, and possible conclusions are articulated, explored, challenged, compared, and resolved. DD within TEFA is intended to have several effects: to clarify thought through the process of articulation and externalization; to expose students to different points of view and lines of thinking; to promote analysis and resolution of disagreements; to supply stimuli, context, and tools for individual sense-making; and to provide practice speaking the social language

of science. The lion's share of student learning in TEFA usually happens during DD, so orchestrating high-quality discourse—interactive, dialogical, and thematically rich—is a top priority for teaching with TEFA.

DD is strongly and directly motivated by the social dimension of our literature synthesis. All authors sharing the sociocultural view of learning agree that talk must be central to the process of learning science. Furthermore, such talk must often be dialogical. One of Carlsen's (2007) three roles for language in science teaching is as a way of making sense of experience; given that the sense-making endeavor is inherently personal and idiosyncratic, discourse must accommodate different students' processes of thinking, eliciting, exploring, and reconciling. Carlsen also identified language as a tool for participating in communities of practice, and discourse within scientific communities is heavily dialogical. In Bakhtin's view (Wertsch 1991), learning science means developing fluency in the social languages of science, and developing fluency in a language requires space and safety to try constructing one's own utterances—one's own scientific thoughts—imperfectly at first, with just enough support to keep going. Worrying about using the language correctly and simultaneously expressing ''correct'' thoughts is too demanding. Additionally, to help students recognize and resolve conflicts between scientific and everyday social languages, both must be voiced, and various ways of speaking about a situation elicited and juxtaposed.

The cognitive dimension of our literature synthesis also points to the need for DD. Constructivism and the conceptual change tradition argue that learning must address the beliefs and knowledge already in students' minds, which implies that a teacher should try to draw these out, getting students to articulate and explore them. This motivates the practice of DD, given that students enter with different ideas and follow idiosyncratic learning trajectories. In fact, any time students must learn to make choices—about what strategy to use for a problem, how to structure a set of ideas, or how to frame an activity—they benefit from hearing and discussing multiple possible choices and ways of making the choice.

The attitudinal dimension also indicates the importance of DD. Influencing students' attributions is easier if we can elicit and examine their various beliefs and perceptions, taking them seriously without necessarily agreeing. We can enhance their sense of self-determination by hearing and respecting their priorities and goals. Also, we can implicitly communicate our expectations regarding students' active engagement in and critical thinking about science by deeply integrating DD into teaching.

DD helps create a learning environment that meets all four of the desirable qualities identified by How People Learn. By drawing out and examining students' individual

thoughts, it helps make instruction student-centered. By stressing sense-making and drawing attention to different possible ways of thinking and metacognitive choices, it helps make instruction knowledge-centered. By making students' thinking visible to teachers, it helps instruction be assessment-centered. Also, by modeling and supporting a dynamic of collaborative sense-making and peer co-learning, it helps make instruction community-centered.

Formative Assessment

The third TEFA principle is ''Inform and adjust teaching and learning decisions with formative assessment'' (FA). As defined by Black et al. (2002),

Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting students' learning. Such assessment becomes ''formative assessment'' when the evidence is actually used to adapt the teaching work to meet learning needs. (p. 1)

The efficacy of formative assessment is strongly and directly supported by empirical results (Bell and Cowie 2001; Black and Wiliam 1998b, 2005; Bransford et al. 1999; Sadler 1989). According to Black and Wiliam (1998a), ''innovations which include strengthening the practice of formative assessment produce significant, and often substantial, learning gains'' across ages, school subjects, and countries—gains ''larger than most of those found for educational interventions'' (p. 140). FA is particularly beneficial for traditionally ''low achieving'' students, with potential to help narrow the achievement gap between students from different socioeconomic strata (Black 1998; Stiggins 2002). FA can elicit richer classroom discourse and help students become more engaged and motivated (Gallagher 2000), can help students become aware of the limits of their understanding and the actions they can take to progress (Ramaprasad 1983; Sadler 1989), and can catalyze significant teacher learning (Black et al. 2002; Bransford et al. 1999). In ''real-time'' FA, learning and instruction are almost continuously monitored and adjusted (Stiggins 2002).

The need for FA also follows from our literature synthesis. According to our cognitive dimension, students' initial beliefs and knowledge are idiosyncratic, as are their learning trajectories; a teacher needs detailed and current information about a student's thinking and state of understanding to efficiently facilitate the learning process, and this is obtained through FA. (We use the term ''agile teaching'' to mean fine-tuning instruction—and occasionally making large changes—on a minute-to-minute basis, guided by real-time FA.) In particular, the idea that students can ''possess'' some piece of knowledge but not have

access to it when they need it suggests that a teacher ought to ferret out more than just what a student ''knows''; he or she should also practice FA by exploring the circumstantial associations, links, and frames involved in students' ability to access their knowledge. Students, too, can benefit from FA that helps them understand the shortcomings in their own conceptual ecologies and points them towards productive learning activity.

From the attitudinal dimension, the idea that a teacher should interact with and influence students' attributions suggests the use of FA to elicit students' beliefs, expectations, perceptions, values, and the like. Also, for students to develop habits and skills of self-regulatory learning, they must learn to practice their own self-directed FA; a teacher can model and scaffold this with explicit FA practices in conjunction with meta-level communication (see below). Self-efficacy is built on evidence of personal capacity and the growth of mastery, and FA that informs students can provide such evidence.

How People Learn speaks strongly in support of FA. It argues for student-centered classrooms, and focusing on students as individuals requires guidance from FA. It argues for using students' language practices as a basis for further learning, implying that a teacher should ''go onto students' turf'' in discussion; learning that turf requires FA. When it argues that effective classrooms are assessment-centered, it explicitly refers to FA.

Meta-Level Communication

The fourth TEFA principle is ''Help students develop metacognitive skills and cooperate in the learning process with meta-level communication'' (MLC). Three categories of MLC are significant to TEFA: meta-narrative, meta-cognitive talk, and metacommunication. (Metadiscourse, which simply means ''talk about talk,'' is a more general concept from semantic analysis that overlaps with these three; c.f. Lemke 1990.) We can say that most of the talk in a science class is discourse about the science content (or about administrative issues), but MLC is discourse about learning the content. TEFA employs MLC to (a) improve learning by increasing the efficiency of the instructional process, and to (b) improve the learner by promoting and scaffolding student development of more productive learning beliefs, attitudes, and behaviors. These two outcomes address TEFA's twin goals of helping students develop expertise in the science subject being taught and helping to prepare them for future learning.

Meta-narrative is communication about the purpose, design, and unfolding of the course from a higher perspective. Its purpose is to help students become informed participants: to make them more consciously aware of what is going on in the class and why it is happening, so that

they may frame their activity appropriately, focus on the most salient aspects, and actively seek the right kind of realizations. Metacognitive talk is communication about thinking, learning, knowledge, and related cognitive and epistemological issues. Its purpose is to improve students' understanding of these things so that they become more self-aware as learners and can make wiser choices about their learning actions. Metacommunication is communication about communication. Its purpose is to refine communication in the classroom and help students participate more consciously and efficiently in it. Seen through the lens of constructivism, communication is inherently error-prone: the recipient of a message attaches his or her own meanings to the words, and constructs an interpretation of the message that may or may not align well with the sender's intention. This difficulty is exacerbated in teaching, when one party (the student) is new to much of the language being used as well as to the ideas it is being used to convey. It follows that awareness of the potential for miscommu-nication and proactive monitoring of message fidelity— such as asking or saying something in multiple ways, actively considering alternative interpretations of a message, and ''closing the loop'' by seeking a return message consistent with the intended interpretation—improve communication efficiency. The more active students become about aggressively improving communication, the more effective instruction can be (see discussion in Gerace 1992).

These three categories of MLC overlap. A statement about the instructor's purpose in saying or asking something may be both meta-narrative and metacommunication. Similarly, explanation of the purpose of some part of the course in terms of its role in the learning process could be both meta-narrative and metacognitive talk. Often, we find that different kinds of MLC chain together, with a meta-narrative or metacommunicative comment blossoming into an extended metacognitive discussion on an underlying issue.

As motivated by the attitudinal dimension of our literature synthesis, we use MLC as a tool for impacting students' beliefs, attitudes, and motivation, and thus for altering their behavior patterns. We do not ''train'' students to engage in instructional activities the way we wish; instead, we invite them, attempting to make them consciously aware of their choices for learning activity and the ramifications those choices can have, thus enhancing self-determination and inviting self-regulatory learning. We suggest and encourage self-regulatory strategies, thus scaffolding the development of self-regulation. We indicate ways to improve success at communication and learning, thus enhancing self-efficacy. We elicit and discuss students' personal goals and how learning behavior and outcomes can connect to them, supporting goal-directed behavior. We explicitly communicate and discuss teacher expectations, and we challenge beliefs

about teaching, learning, and science by suggesting and defending alternative interpretations, thus influencing attitudes and behavior.

MLC also addresses considerations raised by the cognitive and attitudinal dimension of our literature synthesis. We can warn students of the need to identify, challenge, and integrate their prior knowledge and pre-existing beliefs, and we can advise them on fruitful strategies for structuring their knowledge through metacognitive talk. We can metacommunicate to arm students against conflicts between everyday and science social languages. We can help students select appropriate epistemological frames for their participation through meta-narrative on the nature and purpose of activities, including the purposes of QDI questions, DD discussions, and FA ''testing.'' How People Learn includes, in its criteria for a knowledge-centered learning environment, helping students develop metacog-nitive capacity; and, in its criteria for a community-centered one, developing norms that value the search for understanding and the freedom to make mistakes. Both of these are supported by the practice of MLC.

For explicit MLC to be effective, of course, it must be reinforced rather than undercut by the implicit meta-mes-sages contained in a teacher's actions, in his or her other statements, and in curriculum and activities. For example, if we tell students that we are more concerned with the cogency of their reasoning than with the correctness of their answer choice, we can reinforce that meta-level statement through QDI questions with multiple reasonable and defensible answers. We could easily undercut it by, for example, expressing happiness when students provide a particular answer we are seeking, or by engaging in ''IRE'' evaluative questioning during class discussion.

Synergy Between the Principles

TEFA's four principles are not a collection of independent pedagogical exhortations that we happen to espouse. They interlock and reinforce each other. In the absence of any one, the other principles become less potent and more difficult to implement, and TEFA is stunted or unravels. For example, successful QDI requires tuning questions to students' zone of proximal development (Vygotsky 1978) and scaffolding their efforts just enough to help them succeed without bypassing the confusion, struggle, and conflict essential to sense-making; FA helps a teacher gather information to tune and scaffold successfully. FA requires gathering data about what students are thinking, and why; QDI and DD provide complementary data sources. DD requires a context and focus, and engages students more when they have at least provisionally committed to some position; QDI arranges these. QDI, DD, and FA are all aided by students' active, well-intentioned,

well-informed cooperation; MLC helps cultivate that. The four principles enhance each other synergistically in many ways both obvious and subtle.

The Question Cycle

The four principles of TEFA are general and flexible by design, so that they may productively guide instruction in almost any circumstance. To help teachers implement the principles, TEFA specifies a particular, tangible pattern of classroom activity called the question cycle (introduced in Dufresne et al. 1996). It is an iterative cycle of question posing, answering, and discussing, aided by CRS technology, that forms a scaffold for structuring whole-class interaction. The essential phases of the cycle are:

1. Pose a question or problem to the students, generally challenging, often multifaceted. (In TEFA, we do not teach first and then ask questions about what was taught; we ask questions first, and use them as a context for sense-making and direct instruction.)

2. Have students wrestle with the question—alone, in small groups, or both in succession—and decide upon a response.

3. Use a CRS to collect responses (even from students who are uncertain) and display a chart of the aggregated responses.

4. Elicit from students as many different reasons and justifications for the chosen responses as possible, without revealing which (if any) is (or are) correct. In the process, draw out students' reasoning and vocabulary, expose them to each others' ideas, and make implicit assumptions explicit.

5. Develop a student-dominated discussion of the assumptions, perceptions, ideas, and arguments involved. Help students formulate their ideas and practice talking science, find out why they think what they do, and gently increase their understanding. (In practice, phases 4 and 5 usually blend together.)

6. Provide a summary, micro-lecture, meta-level comments, segue to another question, or whatever other closure seems warranted, informed by the detailed data just obtained on students' thinking. (The class should now be well primed to receive the message, appreciate its relevance, and integrate it with other knowledge.)

Questions can build upon each other or function together as sets in order to develop students' understanding. Demonstrations, a second answer-collecting round after step 5, and other elaborations may be inserted as appropriate. We find that iterating through this cycle three or four times in 50-60 minutes of TEFA instruction is usually optimal; a higher rate does not give students enough time to really engage, ponder, articulate, listen, resolve, and practice speaking.

The TEFA question cycle is flexible and rich enough to be a regular, perhaps dominant, part of science instruction. It does not, however, address every learning goal. Most teachers will need to include complementary course components such as pre-class reading (for initial exposure to ideas), post-class homework (for more intensive problemsolving work and skills practice), group projects (for extended explorations of contexts and situations), and laboratory exercises (for hands-on learning opportunities and experience ''doing'' science). TEFA is aimed at ''whole-class instruction,'' which complements other components of a course. It is a pedagogy primarily designed for the portion of class time in which the teacher works with the whole class to help them make sense of new content material. It is not intended to supplant seat-work, group projects, laboratory work or other modes of non-frontal teaching.

The principle of QDI is realized in the question cycle by the placement of question posing at the beginning of the cycle to motivate and contextualize all that follows. The ''closure'' phase, where most direct teaching will occur, is placed at the end. The principle of DD is realized in the question cycle through both the small-group and whole-class discussion phases. Beginning whole-class discussion by identifying different answers that have been selected and eliciting arguments or explanations for them, before scrutinizing the validity of any response, is a strategy for increasing dialogicity. FA, including real-time FA, is realized in the question cycle in multiple ways. Students learn about what they do and don't understand by their ability to answer the posed questions and by how their responses compare to their peers'. They learn more about the extent of their own understanding in the process of trying to articulate it coherently and convince others, and yet more when the instructor provides apropos, prescriptive feedback as part of the closure phase. Teachers learn about their students' understanding, perceptions, assumptions, and reasoning from the chart of question responses, and in more detail from what students say as they defend, explore, and contrast their ideas. In our experience, teachers are frequently surprised by what TEFA reveals about their students' thinking. MLC is not explicitly included in the question cycle, but should be integrated liberally both by design and spontaneously when a suitable occasion arises. The final, closure phase of the cycle often presents a natural opportunity to ''go meta.''

The Role of Technology in TEFA

Nothing about TEFA requires a CRS, at least not in principle. However, using the technology to assist with TEFA offers several benefits that enhance, and in some contexts make possible, what TEFA prescribes. One crucial feature of CRSs is that they simultaneously provide anonymity and

accountability: students can be (formally or socially) held accountable for answering questions, but the actual answer each student has chosen is not revealed to other students and is not immediately obvious to the teacher (Roschelle et al. 2004a). True anonymity that students have faith in is difficult to assure by raised hands (even with heads down), color-coded cards, or other means. An equivalent level of anonymity can be achieved via paper forms, but this is too slow for real-time FA.

A CRS also supports collecting answers from all students in a class, rather than just the few who speak up or are called upon. This means that all students can benefit from the cognitive act of choosing and committing to an answer. It also means the teacher gains better data about students' thinking. ''Increased student engagement and participation'' is one of the two most commonly reported findings of CRS implementation studies (Fies and Marshall 2006). We conjecture that beyond merely gathering responses, CRS technology provides a benefit in that the act of pushing a physical button and definitively submitting an answer, with no waffling or qualification possible (until the discussion phase, of course), amplifies the psychological benefit of having students choose sides. Once students have committed to an answer, whether or not they are confident in it, we believe they attend to subsequent discussion and resolution of the matter in a different and more attentive way. The answer they have selected is now ''their'' answer, for good or ill, and they want to see how it fares.

The CRS chart showing the distribution of students' answers also adds value to the process. It is not just a way to find out how many picked which answer; as Roschelle et al. (2004a) note, it is also a ''high contrast display that drive[s] productive discourse'' (p. 28). It makes differences in students' positions starkly obvious. One glance strongly conveys whether the class is in agreement (a single peak), generally undecided (a uniform or random spread), or highly polarized (two distinct peaks). It also serves as an inscription (a point of focus and a reference for discussion and thinking; Forman and Ansell 2002) for subsequent discussion. The chart communicates the same information as a list of numbers, but in a way that is more forceful and easier to digest.

Finally, a CRS can record the data of students' individual and collective responses for subsequent analysis. This supports additional FA by helping a teacher to diagnose class-wide or individual student needs, or to self-evaluate his or her own instruction.

Uses for TEFA Questions

TEFA questions and the question cycle can serve various possible roles within instruction. A teacher can further

one or more general objectives with a question, including learning about students' knowledge and thinking; helping students become more aware of their own and each others' knowledge and thinking; preparing a fertile context for subsequent instruction; catalyzing small-group discussion and peer learning; provoking, motivating, grounding, and shaping whole-class discussion of a topic; and precipitating student insights and realizations. The following list provides examples of some more specific instructional purposes that TEFA questions and the question cycle can be put to during the course of instruction.

1. Status check: During instruction, poll students for their self-reported degree of confidence in their understanding of a topic.

2. Exit poll: At the close of a class session, poll students to find out which of several concepts covered that day they most want to spend more time on.

3. Assess prior knowledge: Elicit what students already know, think, believe, or perceive about a topic or idea before addressing it in class.

4. Provoke thinking: Ask a provocative and interesting, but inviting, question to ''open up'' a new topic or subject, get students engaged and thinking about it, and provide context and shared experience for subsequent learning.

5. Elicit a misconception: Lead students to manifest a specific common misconception or belief that may hinder their learning, so that it may be articulated, examined, and dispatched.

6. Exercise a cognitive skill: Drive students to engage in a specific type of cognitive activity or exercise a specific habit of mind (c.f. Dufresne et al. 2000) such as seeking alternative representations, comparing and contrasting two situations, categorizing and classifying cases, or strategizing a solution.

7. Build conceptual structure: Hone, link, or extend a concept by challenging students to identify its limits of applicability, differentiate it from a similar concept, recognize a relationship with another concept, or apply it in a new context.

8. Stimulate discussion: Provoke dialogical whole-class discussion with a highly disputable question having multiple reasonable or defensible—but not obviously correct—answers.

9. Induce cognitive conflict: Create a teachable moment by deliberately bringing students to the realization that two of their beliefs, perceptions, ideas, interpretations, or models conflict.

10. Anticipate a demonstration: Ask students to predict the outcome of a demonstration or experiment, and commit to that prediction, so that they will be attentive

to the important aspects, and will learn more when their prediction is either confirmed or disconfirmed.

11. Test capability: Determine whether students have developed the capacity to answer a particular kind of question.

12. Demonstrate success: Build students' confidence and help them to recognize their own progress by posing a question that most can answer successfully now, but could not have at a previous time.

13. Review: Pose a series of rapid questions with minimal discussion, to remind students of a body of material already covered and to help both students and teacher gauge how well students understand it.

This list is not exhaustive, but should serve to demonstrate TEFA's richness and flexibility, and to help teachers avoid getting ''stuck in a rut'' and repeatedly using a CRS in the same ways.


In this article, we have argued for the importance of distinguishing between the technology and pedagogy of classroom response system use, of differentiating the pedagogies described or implied in the CRS literature, and of articulating, defending, and elaborating specific pedagogies for teaching with a CRS. As a step in that direction, we have described in detail our CRS-based pedagogy, technology-enhanced formative assessment (TEFA). We presented the research-based theoretical perspective on learning and teaching that motivates it, articulated its instructional objectives, defined the four principles at its core, and explained the question cycle by which it is implemented in the classroom. We also indicated some of the many roles which the question cycle can serve within instruction.

We adopted a bottom-up approach to our presentation of TEFA, building from the literature synthesis through core principles and classroom enactment to possible applications. The resulting pedagogy ought, in principle, to be consistent with empirical findings on the effectiveness of CRS-based instruction. In their review of CRS literature, Roschelle et al. (2004a) identified four main, unifying constructs that they suggest focusing on in order to connect CRS-based instruction to the broader education research literature. These constructs are: (a) ''formative assessment,'' (b) ''driving discussion by important conceptual contrasts,'' (c) ''shifting to mastery-oriented motivational incentives,'' and (d) ''harnessing diversity for generativity'' (pp. 26-27). They do not claim that CRS-based instruction necessarily does these things, but rather that these constructs are useful to understand what can, and often does, happen when a CRS is used in class.

We find good agreement, though not one-to-one correspondence, between their four constructs and our four principles. Their first construct, ''formative assessment,'' maps directly to our formative assessment principle. Their second and fourth constructs, ''driving discussion by important conceptual contrasts'' and ''harnessing diversity for generativity,'' both align with our stress on dialogicity in our principle of dialogical discourse, and also with our assertion that the questions used for question-driven instruction should frequently be disputable, ambiguous, and have multiple defensible answers. Their third construct, ''shifting to mastery-oriented motivational incentives,'' identifies part of our motivation for formulating the principles of question-driven instruction and meta-level communication. We believe that the agreement between TEFA's principles and these four empirically derived constructs lends credibility to TEFA, and simultaneously suggests that our exposition of TEFA can help meet Ros-chelle et al. call for ''aligning the existing research that narrow focuses on classroom networks to well-established findings of broader educational research through [our proposed] four main constructs'' (p. 1, abstract).

TEFA and its exposition in this article are significant for two reasons. One, as argued above, is to advance research into CRS-based instruction and development of effective CRS-using pedagogical approaches. The other is to help bridge a gap that we perceive between the output of educational research and the daily needs of science teachers. Teachers have difficulty extracting concrete guidance from much of the educational research literature. On one hand, studies conducted in specific instructional contexts can be difficult and risky to derive general implications from. On the other hand, broad syntheses—such as How People Learn, with its four qualities of effective learning envi-ronments—usually say more about the results teachers should strive for than how exactly they can achieve those results. Lists of ''best practices'' can be useful as starting points or tweaks, but without a coherent underlying framework, they are rarely transformative for a teacher. In our extensive professional development work with in-service teachers, we have found that specific examples and case studies, abstract theory, and best practice tips can all be useful up to a point, but that teachers often struggle to apply them and to deeply integrate them into daily practice.

We have attempted to solve this problem by formulating TEFA in four layers: a research-based perspective on teaching and learning; four principles following from that perspective, cast as imperative statements about what teachers should try to do; an explicit structure for classroom activity that enacts the four principles in a concrete, easy-to-follow way; and many specific ways that the question cycle can be applied to meet various needs in the complex task of teaching science (only briefly alluded to in

this article). In particular, we stress the importance of building our pedagogy on a few core principles, and urge others who articulate, analyze, and promulgate pedagogies to do the same.

The work of defining, defending, and evangelizing TEFA is far from done. The attentive reader may have noted that while TEFA's principles are consistent with and perhaps implied by the educational research base, many aspects of the edifice are not directly established by empirical evidence. In particular, little research exists to directly support our assertions about the efficacy of metalevel communication (MLC) for influencing students' attitudes. Koballa and Glynn (2007, p. 85) listed several different kinds of attitude change interventions he encountered during a literature survey on attitudinal and motivational constructs in science learning, and nothing like MLC is included in his list. Our own experiences as teachers and teacher mentors, upon which this perspective is largely based, are extensive but anecdotal. We aspire to further develop and test our assertions about MLC in the future.

More ambitiously, a large-scale controlled study of TEFA's student learning impacts is needed to solidly establish the efficacy of the pedagogy as a whole. Conducting such a study would of course be quite challenging, for all the usual reasons that make gold-standard evaluations of instructional approaches difficult, but it would be possible. At present, however, we lack some crucial knowledge needed to craft such a study. Most notably, we lack sufficient knowledge about how teachers learn to practice TEFA, how to efficiently and reliably support their learning and foster skilled practice of TEFA, and how to define and reliably and efficiently gauge implementation fidelity for a pedagogy so complex and deep. Without good measures of implementation fidelity, distinguishing between ''failure of TEFA'' and ''failure to implement TEFA'' is impossible. Recognizing this, we and our colleagues are currently conducting a longitudinal, mixed-methods, case-based study of how secondary science teachers adopt and adapt TEFA. To date, the study has involved over 40 teachers from six schools in three school districts. It has been designed to lay the groundwork for a scaling study on TEFA student learning impacts by developing a better understanding of teachers' TEFA learning trajectories, by designing and documenting an effective TEFA professional development program, and by devising suitable instrumentation and data collection methods.

We do not expect that an understanding of TEFA at the level presented in this article is sufficient to enable most teachers to successfully implement the pedagogy. For one thing, the nature and quality of the QDI questions used to anchor iterations of the question cycle are crucial to the success of the endeavor, and we have said little here about

engineering effective questions. For another, conducting class in the manner TEFA suggests—including high-quality dialogical discourse and pervasive meta-level communication—requires a skill-set many teachers have not developed, and a role in the classroom that many are not accustomed to. We have addressed, and will continue to address, such aspects of TEFA in other writings (e.g., Beatty et al. 2006). However, we hope this article can begin a critical conversation about pedagogies for CRS-based instruction, and can be of at least some help in encouraging and improving CRS use in science classrooms.

Acknowledgments Preparation of this manuscript and development of the ideas described have been supported in part by US National Science Foundation grants DUE-9453881, ESI-9730438, and TPC-0456124. Any opinions, findings, conclusions, or recommendations expressed are those of the authors and do not necessarily reflect the views of the NSF.

Open Access This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.


Abrahamson LA (2006) A brief history of networked classrooms: effects, cases, pedagogy, and implications. In: Banks DA (ed) Audience response systems in higher education: applications and cases. Idea Group Inc, Hershey, pp 1-25 Banks DA (ed) (2006) Audience response systems in higher

education: applications and cases. Idea Group Inc, Hershey Barnett J (2006) Implementation of personal response units in very large lecture classes: student perceptions. Australas J Educ Technol 22(4):474-494 Beatty ID (2004) Transforming student learning with classroom communication systems (ECAR Research Bulletin ERB0403). Educause center for applied research. Retrieved 14 October 2008 Beatty ID, Gerace WJ, Leonard WJ, Dufresne RJ (2006a) Designing effective questions for classroom response system teaching. Am J Phys 74(1):31-39 Beatty ID, Leonard WJ, Gerace WJ, Dufresne RJ (2006b) Question driven instruction: teaching science (well) with an audience response system. In: Banks DA (ed) Audience response systems in higher education: applications and cases. Idea Group Inc, Hershey, pp 96-115 Beatty ID, Feldman A, Leonard WJ, Gerace WJ, St Cyr K, Lee H et al (2008) Teacher learning of technology-enhanced formative assessment. Paper Presented at the Annual Conference of the National Association for Research in Science Teaching, Baltimore

Bell B, Cowie B (2001) The characteristics of formative assessment

in science education. Sci Educ 85(5):536-553 Black P (1998) Formative assessment: raising standards inside the

classroom. Sch Sci Rev 80(291):39-46 Black P, Wiliam D (1998a) Assessment and classroom learning.

Assess Edu Princ Policy Pract 5(1):7-74 Black P, Wiliam D (1998b) Inside the black box: raising standards through classroom assessment. Phi Delta Kappan 80(2):139-148

Black P, Wiliam D (2005) Developing a theory of formative assessment. In: Gardner J (ed) Assessment and learning. Sage Books, London, pp 81-100 Black P, Harrison C, Lee C, Marshall B, Wiliam D (2002) Working inside the black box: assessment for learning in the classroom. King's College, London Bonwell CC, Eison JA (1991) Active learning: creating excitement in the classroom (ASHE-ERIC Higher Education Reports ED336049). ERIC Clearinghouse on Higher Education, Washington, DC

Bransford JD, Schwartz D (1999) Rethinking transfer: a simple

proposal with multiple implications. Rev Res Educ 24:61-100 Bransford JD, Brown AL, Cocking RR (1999) How people learn: brain, mind, experience, and school. National Academy Press, Washington, DC

Burnstein RA, Lederman LM (2001) Using wireless keypads in

lecture classes. Phys Teach 39(1):8-11 Burnstein RA, Lederman LM (2003) Comparison of different

commercial wireless keypad systems. Phys Teach 41:272-275 Caldwell JE (2007) Clickers in the large classroom: current research

and best-practices tips. CBE Life Sci Educ 6(1):9-20 Carlsen WS (2007) Language and science learning. In: Abell SK, Lederman NG (eds) Handbook of research on science education. Lawrence Erlbaum Associates, Mahwah, pp 57-74 Chi MTH, Glaser R (1981) The measurement of expertise: analysis of the development of knowledge and skills as a basis for assessing achievement. In: Baker EL, Quellmalz ES (eds) Design, analysis and policy in testing. Sage Publications, Beverly Hills, pp 37-47 Chin C, Chia L (2004) Problem-based learning: using students' questions to drive knowledge construction. Sci Educ 88(5):707-727

Crouch C, Mazur E (2001) Peer instruction: ten years of experience

and results. Am J Phys 69(9):970 Draper SW, Brown MI (2004) Increasing interactivity in lectures using an electronic voting system. J Comput Assist Learn 20(2):81

Dufresne RJ, Gerace WJ (2004) Assessing-to-learn: formative

assessment in physics instruction. Phys Teach 42(6):109-116 Dufresne RJ, Gerace WJ, Leonard WJ, Mestre JP, Wenk L (1996) Classtalk: a classroom communication system for active learning. J Comput High Educ 7(2):3-47 Dufresne RJ, Gerace WJ, Mestre JP, Leonard WJ (2000) ASK-IT/ A2L: assessing student knowledge with instructional technology (Tech. Rep. dufresne-2000ask). University of Massachusetts Amherst Scientific Reasoning Research Institute Dufresne RJ, Gerace WJ, Leonard WJ, Mestre JP (2001) Creating an

item for in-class formative assessment. Interact Classr 1(3) Dufresne RJ, Thaden-Koch T, Gerace WJ, Leonard WJ (2005) Knowledge representation and coordination in the transfer process. In: Mestre JP (ed) Transfer of learning from a modern multidisciplinary perspective. Information Age Publishing, Greenwich, pp 89-119 Duncan D (2005) Clickers in the classroom: how to enhance science teaching using classroom response systems. Addison-Wesley, New York

Duncan D (2006) Clickers: a new teaching aid with exceptional

promise. Astron Educ Rev 5(1):70-88 Elby A (2001) Helping physics students learn about learning. Am J

Phys Suppl 69(S1):S54-S64 Fagen AP, Crouch CH, Mazur E (2002) Peer instruction: results from

a range of classrooms. Phys Teach 40(4):206-207 Fies C, Marshall J (2006) Classroom response systems: a review of

the literature. J Sci Educ Technol 15(1):101-109 Forman EA, Ansell E (2002) Orchestrating the multiple voices and inscriptions of a mathematics classroom. J Learn Sci 11(2-3): 251-274

Gallagher JJ (2000) Teaching for understanding and application of

science knowledge. Sch Sci Math 100(6):310-318 Garner A, Shih MY, Rogers R, Hart D (eds) (2008) Personal response system (PRS): a handbook for UMass faculty. University of Massachusetts Amherst Center for Teaching, Center for Educational Software Development, and Provost's Office, Amherst. 2008.pdf. Retrieved 9 October 2008 Gerace WJ (1992) Contributions from cognitive research to mathematics and science education. In: Grayson D (ed) Workshop on research in science and mathematics education. University of Natal, Durban, pp 25-44 Glaser R (1992) Expert knowledge and processes of thinking. In: Halpern D (ed) Enhancing thinking skills in the sciences and mathematics. Lawrence Erlbaum Associates, Hillsdale, pp 63-75 Hammer D, Elby A (2003) Tapping epistemological resources for

learning physics. J Learn Sci 12(1):53-90 Hammer D, Elby A, Scherr RE, Redish EF (2004) Resources, framing, and transfer. In: Mestre JP (ed) Transfer of learning: research and perspective. Information Age Publishing, Greenwich, pp 89-119

Herreid CF (2006) ''Clicker'' cases: introducing case study teaching

into large classrooms. J Coll Sci Teach 36(2):43-47 Hestenes D, Wells M, Swackhamer G (1992) Force concept

inventory. Phys Teach 30(3):141-158 Johnson D, McLeod S (2004) Get answers: using student response systems to see students' thinking. Learn Lead Technol 32(4):18-23

Koballa TR, Glynn SM (2007) Attitudinal and motivational constructs in science learning. In: Abell SK, Lederman NG (eds) Handbook of research on science education. Lawrence Erlbaum Associates, Mahwah, pp 75-102 Larkin JH (1979) Information processing models in science instruction. In: Lochhead J, Clement J (eds) Cognitive process instruction. Lawrence Erlbaum Associates, Hillsdale, pp 109-118 Lemke JL (1990) Talking science: language, learning, and values.

Ablex Publishing, Stamford Leonard WJ, Gerace WJ, Dufresne RJ (2001) Questions first (Q1st): the challenges, benefits, drawbacks, and results of asking students questions prior to formal instruction. Proceedings of the 2001 Physics Education Research Conference, pp 41-44 Mazur E (1997) Peer instruction: a user's manual. Prentice Hall,

Upper Saddle River McNeal A, D'Avanzo C (1997) Student-active science: models of innovation in undergraduate science teaching. Saunders College Publishing, Orlando Mestre JP, Gerace WJ, Dufresne RJ, Leonard WJ (1997) Promoting active learning in large classes using a classroom communication system. In: Redish EF, Rigden JS (eds) The changing role of physics departments in modern universities: Proceedings of the international conference on undergraduate physics education. American Institute of Physics, Woodbury, pp 1019-1036 Mortimer EF, Scott PH (2003) Meaning making in secondary science

classrooms. Open University Press, Maidenhead Phillis R (2005) University of Massachusetts Amherst: introductory biology. The National Center for Academic Transformation web site: Retrieved 9 October 2008 Posner GJ, Strike KA, Hewson PW, Gertzog WA (1982) Accommodation of a scientific conception: toward a theory of conceptual change. Sci Educ 66(2):211-227 Ramaprasad A (1983) On the definition of feedback. Behav Sci 28(1):4-13

Reay NW, Bao L, Warnakulasooriya R, Baugh G (2006) Toward the effective use of voting machines in physics lectures. Am J Phys 73(6):554-558

Reay NW, Li P, Bao L (2008) Testing a new voting machine question

methodology. Am J Phys 76(2):171-178 Redish EF (2003) A theoretical framework for physics education research: modeling student thinking. In: Vicentinni M, Redish EF (eds) Proceedings of the Varenna summer school, ''Enrico Fermi'' Course CLVI. IOS Press, Amsterdam Roschelle J, Abrahamson LA, Penuel WR (2004a) Integrating classroom network technology and learning theory to improve classroom science learning: a literature synthesis. Paper Presented at the Annual Meeting of the American Educational Research Association, San Diego Roschelle J, Penuel WR, Abrahamson LA (2004b) Classroom response and communication systems: research review and theory. Paper Presented at the Annual Meeting of the American Educational Research Association, San Diego Sadler R (1989) Formative assessment and the design of instructional

systems. Instr Sci 18(2):119-144 Scherr RE (2007) Modeling student thinking: an example from

special relativity. Am J Phys 75(3):272-280 Schunk DH, Ertmer PA (2000) Self-regulation and academic learning. In: Boekaerts M, Pintrich PR, Zeidner M (eds) Handbook of self-regulation. Academic Press, San Diego, pp 631-650 Schwartz DL, Martin T (2004) Inventing to prepare for future learning: the hidden efficiency of encouraging original student production in statistics instruction. Cognit and Instr 22(2):129-184 Scott PH, Mortimer EF (2006) The tension between authoritative and dialogic discourse: a fundamental characteristic of meaning making interactions in high school science lessons. Sci Educ 90(4):605-631

Scott P, Asoki H, Leach J (2007) Student conceptions and conceptual learning in science. In: Abell SK, Lederman NG (eds) Handbook

of research on science education. Lawrence Erlbaum Associates, Mahwah, pp 31-56 Stiggins RJ (2002) Assessment crisis: the absence of assessment FOR

learning. Phi Delta Kappan 83(10):758-765 Sutton C (1998) New perspectives on language in science. In: Fraser BJ, Tobin KG (eds) International handbook of science education. Kluwer Academic, Dordrecht, pp 27-38 von Foerster H (1981) Observing systems. Intersystems Publications, Seaside

von Glasersfeld E (1981) An introduction to radical constructivism.

In: Watzlawick P (ed) The invented reality. Norton, New York Vygotsky LS (1978) The development of higher psychological

processes. Harvard University Press, Cambridge Vygotsky LS (1987) Thinking and speech. In: Rieber RW, Carton AS (eds) The collected works of L. S. Vygotsky. Plenum Press, New York

Wenk L, Dufresne RJ, Gerace WJ, Leonard WJ, Mestre JP (1997) Technology-assisted active learning in large lectures. In: McNeal A, D'Avanzo C (eds) Student-active science: models of innovation in undergraduate science teaching. Saunders College Publishing, Orlando, pp 431-451 Wertsch JV (1991) Voices of the mind: a sociocultural approach to

mediated action. Harvard University Press, Cambridge Wieman CE, Perkins K, Gilbert S, Benay F, Kennedy S, Semsar K et al (2008) Clicker resource guide. University of Colorado Science Education Initiative & University of British Columbia Carl Wieman Science Education Initiative, Vancouver Wiliam D (2007) Keeping learning on track: classroom assessment and the regulation of learning. In: Lester FK (ed) Second handbook of mathematics teaching and learning. Information Age Publishing, Greenwich, pp 1051-1098