Scholarly article on topic 'Triangulating Graduate Instructor Learning in FL Teaching Methods: Questionnaires, Concept Maps, and Reflective Teaching Journals'

Triangulating Graduate Instructor Learning in FL Teaching Methods: Questionnaires, Concept Maps, and Reflective Teaching Journals Academic research paper on "Educational sciences"

CC BY-NC-ND
0
0
Share paper
OECD Field of science
Keywords
{"graduate language instructors" / "teacher cognition" / "FL teaching methods" / "content learning"}

Abstract of research paper on Educational sciences, author of scientific article — Laura Gurzynski-Weiss

Abstract Graduate students who teach languages in university settings often receive their training in an intensive, semester-long teaching methodology course. Measuring learning in this context is often a challenge, as graduate students enter with varying levels of knowledge and experience and are often expected to simultaneously learn and apply this knowledge to the courses they are teaching. While common means of measuring instructor learning via cognitive change has been documented for more experienced language instructors (cf. Borg, 2006) [1], graduate instructors are an often-overlooked population (cf. Gurzynski- Weiss, 2013; in press) [2] [3]. In hopes of encouraging robust research in this critical area, this paper details three instruments that can be utilized to examine graduate instructor cognition and evidence of learning in a methods course: questionnaires, concept maps, and reflective teaching journals. Design and implementation techniques of each instrument, along with strengths and limitations, are discussed. Illustrative examples from an on-going research project are presented to demonstrate how each instrument can elicit evidence of graduate instructor cognitive change (e.g., learning), and be utilized in other graduate instructor teaching methods courses.

Academic research paper on topic "Triangulating Graduate Instructor Learning in FL Teaching Methods: Questionnaires, Concept Maps, and Reflective Teaching Journals"

Available online at www.sciencedirect.com

ScienceDirect

Procedia - Social and Behavioral Sciences 141 (2014) 171 - 181

WCLTA 2013

Triangulating graduate instructor learning in FL teaching methods: Questionnaires, concept maps, and reflective teaching journals

Laura Gurzynski-Weiss

Indiana University, 1020 East Kirkwood Avenue, Ballantine Hall 876, Bloomington, IN 47405, United States

Abstract

Graduate students who teach languages in university settings often receive their training in an intensive, semester-long teaching methodology course. Measuring learning in this context is often a challenge, as graduate students enter with varying levels of knowledge and experience and are often expected to simultaneously learn and apply this knowledge to the courses they are teaching. While common means of measuring instructor learning via cognitive change has been documented for more experienced language instructors (cf. Borg, 2006) [1], graduate instructors are an often-overlooked population (cf. Gurzynski-Weiss, 2013; in press) [2] [3]. In hopes of encouraging robust research in this critical area, this paper details three instruments that can be utilized to examine graduate instructor cognition and evidence of learning in a methods course: questionnaires, concept maps, and reflective teaching journals. Design and implementation techniques of each instrument, along with strengths and limitations, are discussed. Illustrative examples from an on-going research project are presented to demonstrate how each instrument can elicit evidence of graduate instructor cognitive change (e.g., learning), and be utilized in other graduate instructor teaching methods courses.

© 2014 Publishedby ElsevierLtd.This isanopen access article under the CC BY-NC-ND license (http://creativecommons.Org/licenses/by-nc-nd/3.0/).

Selection and peer-review under responsibility of the Organizing Committee of WCLTA 2013. Keywords: graduate language instructors, teacher cognition, FL teaching methods, content learning

1. Introduction

Measuring graduate instructor learning in a foreign language teaching methods course is a complicated task. In this context, graduate students are asked to learn information, to reflect on and process it, and apply this knowledge to their practice, i.e., language classes that they are often teaching at the same time (many for the first time) while enrolled in a methodology course. As these graduate student instructors bring varying levels of knowledge and teaching experience to the course, assignment and course grades *

Corresponding Author: Laura Gurzynski-Weiss. Tel.: +0-000-000-0000 E-mail address: lauraweiss@gmail.com

1877-0428 © 2014 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

Selection and peer-review under responsibility of the Organizing Committee of WCLTA 2013. doi: 10.1016/j.sbspro.2014.05.031

do not necessarily reflect what has been learned in their methodology course during the semester.

One common way of measuring learning, namely, via examinations of teacher cognition, has been documented for more experienced and full-time language instructors (cf. Borg, 2006) [1]. Graduate instructors, however, are an often-overlooked population (cf. Gurzynski-Weiss, 2013; in press) [2] [3]. This is particularly surprising given the fact that these instructors make up the bulk of the teaching staff in language departments, at least at research-oriented universities in the United States. For example, graduate instructors typically teach lower-level language classes, while full-time faculty and professors teach upper-level content courses.

In hopes of encouraging robust research in this critical area, this paper reports on ways to examine graduate instructor cognitive processing via three instruments: questionnaires, concept maps, and reflective teaching journals. All materials presented here are currently used as part of a yearly graduate-level foreign language teaching methods course designed and taught by the author, and can be adapted to measure cognitive change in other methods courses.

The article is divided as follows: First, teacher cognition research as a means of measuring learning is presented, focusing specifically on the overlooked population of graduate student instructors of foreign languages and the importance of examining this group of teachers. Second, design and implementation techniques of each instrument are discussed in detail, including key strengths and limitations of each when used in isolation as well when used in tandem. Illustrative examples from an on-going research project are presented throughout to demonstrate how each instrument can elicit evidence of cognitive change (e.g., teacher learning), and be utilized by others. The article concludes with practical suggestions for instructors and researchers interested in adapting these materials to measure graduate instructor learning in their own methodology courses.

1.1. Teacher cognition as a measure of learning

It has been argued that initial evidence of learning often can be seen in cognitive reports before it is manifested in behaviour (Kowalski & Webster, 2008) [4]. This is particularly the case for changes to long-held positions such as one's teaching philosophy, which is based largely on beliefs formed over time via one's own experiences as a language learner and as an instructor. For this reason, examining a teacher's change in cognition is more effective than measuring their learning (i.e., knowledge gained in a methodology class) via a test or quiz. Teacher cognition can be summarized as what an instructor "knows, believes, and thinks" (Borg, 2003; 81) [5]. While more traditional assessments offer the ability to examine knowledge in an objective, closed-point way, examining teacher cognition provides a richer, more detailed picture of knowledge. Most importantly for the purposes in the current article, examining cognition can provide details on how the instructor applies information in practical ways, such as an instructor's knowledge about second language acquisition in relation to their in-class feedback provision (Gurzynski-Weiss, 2010) [6], or with respect to how and why instructors deviate from their lesson plans (Richards, 1998) [7]. Doing so can also inform methodology instructors, so that they can better cater their courses to graduate instructor needs.

Teacher cognition research can examine (1) instructor's cognitive processes at a given moment in time (Gurzynski-Weiss, 2010) [6], (2) how cognition can change over time as a result of teaching experience (Gurzynski-Weiss, in press) [3], and (3) most importantly for the current study, how cognition can change a result of a brief training workshop (Mackey, Polio, & McDonough, 2004) [8] or semester-long course (McDonough, 2006) [9]. In the latter contexts, studies examining teacher cognition in relationship to education have largely been limited to separate examinations of pre- or in-service instructors (Peacock, 2001) [10] or comparisons of instructors with more or less teaching experience in the same study (Gatbonton, 2008) [11]. While these populations of instructors are undoubtedly important, current research traditions, whether intentionally or not, seem to be ignoring a key population: graduate student instructors.

1.2. Graduate student instructors of languages

Graduate student instructors are a critical population to examine in terms of cognitive change, particularly those instructors who are teaching foreign languages in university settings. At large research universities in the United States, in particular, the beginning and intermediate language classes— oftentimes the extent of university-wide requirements for foreign languages—are commonly taught exclusively by graduate instructors (Gurzynski-Weiss, 2013; in press) [2] [3]. These graduate instructors are often teaching for the first time, and at minimum have a one-semester language teaching methodology course, frequently given the same semester they begin teaching, or the semester immediately prior. This teaching methods course, for many graduate instructors, serves as the only guaranteed source of dedicated time and study of the nature of foreign language learning and teaching in a classroom setting. For this reason, it is paramount to ensure that these foreign language teaching methodology courses do in fact result in learning.

Unlike other content courses where a simple assessment could suffice, graduate instructors often enter a teaching methods course with varying levels of knowledge, especially if they have prior teaching or tutoring experience, or education in applied linguistics. Whether or not they have experience teaching a language, graduate instructors all have had experience being a language learner. Research has shown that, at least at the beginning years of teaching, instructors rely on their previous learning experiences to inform their teaching beliefs about what constitutes good language teaching, either as a reaction to what they did not like as language learners, or as a way of clinging to something comforting that 'worked' for them (Gurzynski-Weiss, in press; Johnson, 1994) [3] [12]. Despite these inherent challenges with measuring graduate student instructor cognitive change as evidence of learning, it is an endeavour important for stakeholders in language programs, for these instructors' foreign language students, for the instructors teaching the methods courses, and also for the graduate instructors themselves.

2. Organizing questions

To this end, the current article examined how three instruments may be utilized to investigate graduate instructor learning during a teaching methodology course. The questions motivating the current article were as follows: (1) How could graduate instructor learning be uncovered via questionnaires, concept maps and reflective teaching journals? And (2) how could this information be utilized to improve learning opportunities in a teaching methods course? This article will focus on how to utilize these techniques to measure graduate instructor cognitive change, and provide examples on how to code, interpret and utilize data that is collected.

3. Instruments

In the paragraphs that follow, the three instruments—questionnaires, concept maps, and reflective teaching journals—are presented in detail, including considerations in design, implementation and evaluation. Strengths and limitations of each instrument and illustrative examples from an on-going research project are discussed throughout to demonstrate how each instrument can elicit evidence of cognitive change (e.g., learning) in this specific teaching methods context, and be adapted for use by others. All instruments presented in this article can be downloaded and adapted from www.iris-database.org.

3.1 Questionnaires

The first type of instrument presented as a means to elicit and measure evidence of cognitive change in this article is questionnaires; an accessible, familiar tool to elicit specific information from instructors. When designed well, questionnaires can be easily used for robust data collection, and for comparing knowledge at various points in time. They can also require a minimal amount of time invested by both the creator/administrator and the responder. For a detailed account of questionnaire design and administration, see Dornyei and Taguchi (2010) [13]. Given that questionnaires can include myriad options of question types, the researcher/instructor has limitless options for creating questions that will elicit the data needed to answer the questions at hand. To ensure the most robust data collection, it is recommended that questionnaires contain both closed-answer questions, such as multiple choice or ranked answers (e.g., Likert-scale) that guarantee a respondent will choose between specific selections, as well as open-ended questions where participants can enter in their own responses or clarify their answers to closed questions. When using a scale where participants are asked to rank their agreement with a statement, it is important to provide an explanation of the value of each ranking to ensure that all participants who select a '2' on a scale of one to five, for example, are using that ranking in the same way. Questionnaires must be written in prose that is accessible to the participant population, in a language that is comfortable for them to read and respond to, and should not be too long, especially when being collected on a voluntary basis. These instruments, just as any other method of data collection, should be pilot tested with several people from the target population to ensure that the questions are clearly written, and that the instrument does in fact elicit the information for which it was designed. In the case of questionnaires examining learning in a teaching methods course, an instructor could pass their questionnaire to a volunteer graduate student from their university who will not be in their class, or to a colleague or two, to ensure that the questionnaire is understandable.

There are several websites available that provide questionnaire creation, collection and analysis services. As an example, the author of this article utilizes SurveyMonkey.com, chosen for its ease of use, storage, and the simple quantitative analysis tools available. For those less familiar with questionnaire design, the website also provides sample questionnaire templates, and allows for the creation of a customizable URL that links the questionnaire to any email or website of choice, such as an OnCourse or Blackboard site. SSL encryption is also included to ensure privacy for the data, the survey link, and the survey pages.

The author of this article uses two questionnaires in her teaching methods course: a topic familiarity questionnaire and a belief questionnaire. They will be explained to illustrate how questionnaires can be examined with respect to graduate instructor cognitive change. Both the topic familiarity and belief questionnaires are given once at the beginning of the semester, and once at the end. The topic familiarity questionnaire is very brief, less than ten questions, and focuses on graduate student instructors' familiarity with various teaching topics, their teaching experience and their concerns about teaching in this new context, what they were most interested in learning in the course, and how the course fit into their long-term academic and research goals. When administered at the beginning of the semester, this questionnaire contains questions targeting instructors' previous teaching experience and largest concerns for teaching that semester, whereas the questionnaire given at the end of the semester asks which aspects of the course were the most effective for graduate instructor learning (and why), and which components could be improved in the future. This questionnaire is anonymous to encourage the most honest responses possible. A screen shot of a sample question from this topic familiarity questionnaire can be seen in Figure 1.

4. How familiar/comfoliable are you with the following topics?

Very familiar/comfortable Vaguely familiar/comfortable Unfamiliar

Communicative language teaching (CLT)

Task-based language teaching (TBLT)

Lesson planning

Creating tasks and activities

Providing feedback

Teaching vocabulary

Teaching grammar

Figure 1. Sample question from the topic familiarity questionnaire

The belief questionnaire used by this author is longer, more than 35 questions, and focuses on graduate instructor beliefs regarding their role(s) as the instructor in the classroom, what a good language learning task looks like, how tasks should be sequenced in a lesson plan, as well as instructors' view on how languages are learned and what variables within a classroom setting can mediate these learning opportunities. The belief questionnaire is not anonymous and asks for instructors' identifying information to allow for specific within-instructor comparisons. In this example context, identical versions of the belief questionnaire are given at the beginning and end of the semester. See Figure 2 for a screen shot of sample questions from this questionnaire.

7. How do you believe adults learn best in the FL classroom?

8. In your own words, how would you define language "learning"?

Figure 2. Sample questions from the belief questionnaire

When questionnaires contain both closed and open-ended questions, evidence of learning can be examined both quantitatively and qualitatively. First, when examining data from questionnaires, all questions, regardless of type, must be grouped according to topic or component of learning. For the topic familiarity questionnaire described above, the author could zoom in on, for example, learning related to the topic of task-based language teaching, and compare the average ranking of familiarity regarding task-based language teaching at the beginning and end of the semester, or the percentage of instructors rating this item as "very familiar." Finding out which course topics improve in terms of instructor-ranked familiarity provides valuable quantitative information regarding the strength of each unit in the course. In addition, it allows graduate instructors to visually rate their own knowledge, a comparison which can be considered to be evidence of learning. For example, if there was little improvement in the familiarity and comfort of grammar teaching compared to familiarity gains reported with respect to vocabulary, the researcher/instructor could examine the differences between the lesson materials and practices for the topics and reflect on what could be done to improve a particular lesson in the future, such as providing a reading with more practical teaching examples, presenting the material with more accessible terminology or additional scaffolding, or providing more comprehensive lecture before in-class discussion and practice, etc. These quantitative analyses can be performed with simple percentages provided by the website utilized, by hand, or by utilizing a statistical package such as SPSS to examine if the changes

reported by graduate instructors are statistically significant.

When examining open-ended questions, the researcher/instructor can gain insight into common themes present within each sub-topic. These qualitative data can be looked at in isolation, or can be completed in relation to the above-mentioned quantitative data to further explain those trends and provide a more holistic understanding. For example, for the open-ended question of what a graduate instructor sees as their main role(s) as a language instructor, the general idea of the language instructor as a 'facilitator' might come out as the most common theme at the beginning of the semester across questionnaires, although the exact terminology may differ greatly between participants, such as 'helper,' 'assistant,' 'leader,' etc. In the questionnaire at the end of the semester, the researcher/instructor may look for common themes related to the answer of this question, and see what terms are used by graduate instructors to describe their roles, as well as how and which instructors adjusted their terminology, etc. Qualitative analyses can likewise be done by hand by several iterations of analysis by the researcher/instructor, or can be done with qualitative analysis software.

Both anonymous and non-anonymous questions allow for group trend comparisons. Non-anonymous questions allow for individual comparison, which can be followed up with an interview with the graduate student instructor to more thoroughly contextualize their individual responses. In light of the variation of previous knowledge that these graduate instructors bring to a methods course, examining both group and individual data is advisable to ensure that variation in graduate instructor learning (tied specifically to the course) can be identified.

3.2 Concept maps

The second type of instrument presented for measuring graduate instructor learning in a methods course is concept maps (Farrell, 2008) [14]. Concept maps are a specific way of eliciting an individual's knowledge about how concepts relate to each other, as well as which factor(s) are seen as more important to the respondent. These instruments most often have a word or question in the centre or top of the page, and respondents are asked to diagram their understanding of a concept, their answer to the question, or diagram topics they see as related to the prompt. They can be another fruitful means to gain insight -visual and through text - of how graduate student instructors' cognition and understanding changes over time.

In the foreign language methods course taught by the author of this article, for example, graduate instructors are provided with concept maps on the first and the last day of class. On the first day, the name of the course is introduced, followed by introductions of the students and professor. The concept maps are then passed out; prior to any discussion of course content to ensure graduate instructors report their knowledge without any additional influence. The question provided in the centre of the concept map in a bubble is the organizing question of the course: "What factors contribute to/complicate adult foreign language learning in a classroom setting?" A few lines are drawn out from the bubble, as seen in Figure 3, and students are encouraged to add factors (via their own extra 'bubbles') however they see fit.

Figure 3. Concept map illustration from the author's methods course (inspired by Farrell, 2008) [14]

In this example context, students write their names on the concept maps, discuss common themes in a smaller group, share their commonalities and differences with the entire class, and turn in the concept maps to the instructor. They complete this same task once again at the end of the semester, after which they are given back their original concept map for additional comparison and discussion.

One benefit of concept maps not afforded by questionnaires is the fact that concept maps allow graduate instructors to respond freely with minimal researcher/instructor bias within the instrument. By not providing pre-determined categories, such as learner individual differences (e.g., motivation, anxiety, prior language experience), or logistical concerns (e.g., classroom space, amount of instructional hours, department requirements, etc.), the graduate instructor respondents are free to express their knowledge as it is organized in their individual cognitive system. Furthermore, having students complete the same task at the end of the course allows them to see how their conception of adult foreign language learning and teaching has or has not changed. This tool also provides valuable information to the course instructor. The instructor or class together, can create a visual on how the entire class's concept map changed over the course of the semester.

Evidence of learning in these concept maps can be examined via qualitative comparisons between individual graduate instructor concept maps (at the beginning and end of the semester), as well as via comparisons of group trends found throughout the concept maps (at both points in the semester). In the author's example context, data from the concept maps are being entered into a qualitative data analysis program, NVivo, which allows facilitated open coding of written and visual data forms (cf. Baralt, 2011 for a guide on how to use NVivo) [15]. Factors cited in each concept map, as well as patterns of how these factors are organized by students, are compared from the beginning and end of the semester. Differences identified can also be cross-compared with the course content to see which areas of knowledge changed the most, and why. Patterns of knowledge that became more target-like (as operationalized by the methodology instructor, in line with the course goals) can then be utilized as examples for improving areas of the course where cognitive change was not as evident. For example, if one of the main goals of a teaching methods course is for graduate instructors to consider both learner-internal factors (such as learning styles), as well as contextual factors (such as the teaching philosophy of the language department), the researcher/instructor would specifically look for that inclusion throughout the concept maps collected at the end of the semester. Should that data not be present, the instructor could identify course components that are present, and look to those lessons (along with triangulated data from

the other two instruments discussed in this article) for the reasoning behind these cognitive reports.

3.3 Reflective teaching journals

The third and final data source presented in this article, as a means for examining graduate instructor learning in a methods course, is reflective teaching journals. These instruments, as their name suggests, are journals that target instructors' reflective thought processes with respect to aspects of their teaching (cf. Richards & Lockhart, 1996) [16]. These journals can be directed via specific prompts or questions that instructors write about, or, they can be left open so that instructors journal about whatever is most important or relevant for their teaching at a given time. Likewise, instructors may be asked to journal a specific amount, or according to a specific frequency. In order to elicit instructors' genuine reflections, it is paramount that the journal space feels safe and respected. For graduate courses where students may have diverse native languages, it is important for the course organizer to consider allowing graduate instructors to respond in the language they feel most comfortable, in order to encourage the most free response and in-depth reflection.

In the methods course discussed in this article, each week graduate student instructors are asked to reflect on and journal about a topic that relates to the week's readings and class discussions. These reflections were private the first year the class was taught, written online in the course website, and with reading privileges restricted to the individual student and the professor. Since this initial year, following graduate student feedback, the journal entries are now posted on the course website with the privacy settings open to the rest of the class. An additional component has also been added—per student request—requiring graduate instructors to not only reflect and post on the topic, but also to comment on at least one colleague's post per week. Prompts include reflections on theory, such as "Discuss which view(s) of second language acquisition most accurately accounts for language learning in a classroom setting. How will this shape your instruction and your expectations for your students' learning?" as well as more practical applications such as "Operationalize your students' current language level(s) and plan ways to more effectively measure their level and/or ways to push them to the next level. What did you discover and what will you adjust based on these readings, discussions and analysis?" Students are frequently given two prompts to choose from to encourage thoughtful reflections and address different learning styles and interests (e.g., students focusing on linguistics have preferred more theoretical questions while literature-focused students have requested more practical applications). A screen shot of how these prompts are disseminated can be seen in Figure 4.

Announcement Subject Saved By Modified Date Beginning Date Ending Date Groups

Message

Option It 1: Choose one of the common Issues ¡1 classroom management below and articulate briefly how you plan to effectively deal with fand/or how you have effectively dealt with In the past) the Issue If It arises unexpectedly In class.

♦ lack of learner autonomy (always expectlng/relylng on you rather than on themselves)

♦ spaclngflut or sleeping In class

♦ frequent absences/taidlness

♦ food and cell phone disruptions

♦ disrespectful behavior towards you or another student

♦ refusal to participate

♦ too much chit-chat

Option ttZ: How do you lesson plan? Do you have a process you always fdlcw? What are the factors you take into account when you lesson plan? Why? What are the aspects of lesson planning you would like to mprove on? How do you plan on improving this aspect of lesson planning? Be specific.

Reflective teaching journal- due September 6th Laura Gurzynski-Welsa Sep 4, 2013 11:17 am

Figure 4. Partial screen shot of a reflective teaching journal prompt

To uncover evidence of learning in the reflective teaching journals, posts and comments are currently being qualitatively analysed according to each unit within the course (e.g., vocabulary, feedback, etc.), as well as for patterns within each students' posting. For this purely qualitative instrument, an approach such as grounded theory may be an appropriate way of examining the data (Glaser & Strauss, 1967) [17]. In grounded theory, like many qualitative analyses, the data is approached from the bottom-up via several iterations, or examinations. In this context, the author's first iteration of the journal entries began at the end, with the final journal entry of the semester. These prompt asked learners to compare the two drafts of their teaching philosophy statement as well as in-class patterns at the beginning and end of the semester, which provided an ideal orientation to specific entries that, would provide the most promising evidence of language learning, as well as which posts could be analysed for the opposite case. For example, one learner cited major changes related to class work, stating, "After the class on learning strategies, I really found myself working to include training on strategy use in my class, which I had never done before." The learner went on to explain exactly what was different in their approach. After this initial iteration examining a specific reflective teaching journal entry, additional entries can be systematically examined and compared across students (e.g., all journal entries focusing on grammar teaching) and within students (e.g., how did a particular graduate instructor begin talking about their role in the classroom as the teacher, and how has his or her view changed (if there is change)?). Reflective teaching journal entries can be examined in relation to the goals set out for each unit in the course, and in assignments. In this example context, graduate instructor comments to each other's reflective teaching journals will also be examined for additional evidence of cognitive change and learning.

4. Discussion and pedagogical implications for this context

Thus far this article has focused on three instruments—questionnaires, concept maps, and reflective teaching journals—in isolation. However, these techniques can serve as measures of graduate instructor learning in complimentary ways, as well. Arguably, when utilized to measure graduate instructor learning in a methods course, these instruments are strongest when triangulated in a mixed-methods approach. The questionnaires, because of their overall emphasis on close-ended questions (at least in the example context), provide an opportunity to collect specific information that can be compared quantitatively, even statistically. The open-ended questions on the questionnaires provide contextual information and guidance on how to interpret the numerical data gleaned from this technique, which can be anonymous or not anonymous, depending on the needs of the researcher/instructor. The concept maps, on the other hand, provide a much more holistic view of how knowledge and concepts are structured relationally in the mind of the graduate instructor. These instruments provide information regarding the hierarchical structure of how, in the case of the example context, factors within the foreign language classroom are more or less influential according to each graduate student instructor. Finally, the reflective teaching journals provide the most comprehensive view of graduate instructor cognitive processes and change, allowing instructors to address each topic within the course in relation to their own idiosyncratic classroom. This instrument allows for evidence of application of knowledge, whereas the concept maps encourage the presentation of concepts in a more objective, theoretical way. Used in tandem, these three instruments provide a holistic look at several facets of graduate instructor cognitive change and learning within a semester-long methods course.

These instruments, and their use in examining evidence of learning, can simultaneously serve both the researcher/instructor, in her efforts to increase learning opportunities in a methods course, as well as the graduate instructors who are equally if not more invested in the success of this course. As the article discusses early on, these graduate student instructors most often have a one-semester teaching methods course as their only opportunity to formally develop as a language instructor; this may especially be the

case for graduate students of foreign language literature and culture. In utilizing these instruments, and making the graduate instructors part of this iterative process, perhaps one of the greatest contributions of this type of research will be how we train our graduate instructors to become reflective language instructors who can measure learning in challenging contexts, and adjust their own practices accordingly.

5. Guidelines for utilizing these techniques to measure learning

In this section, guidelines are provided for utilizing these three techniques to robustly measure graduate instructor learning in a teaching methodology course. The readers should note that these instruments reflect the teaching philosophy of this particular researcher/instructor and her views of how learning occurs. They should therefore be adapted in accordance with other researcher/instructor's own philosophies and contexts. Adaptations of the instruments presented in this article are encouraged, as is further sharing via www.iris-database.org.

1. Operationalize what learning means in your context. Is it being able to explain or apply certain concepts? Recount details from specific studies?

2. Identify the challenges to measuring learning in your context. Do you also have graduate student instructors who enter with varying levels of experience and knowledge? A cross-listed course with both undergraduate and graduate students?

3. Identify the instruments you have already in your class. Before you create new instruments, what do you already have that can be examined? What are the strengths and limitations of each?

4. Triangulate with a mixed-methods approach. Do you have both closed and open questions that allow for different types of analyses? If not, consider adapting additional instruments that will allow learners to demonstrate their learning in varied ways.

6. Conclusion

This article presented ways of measuring graduate student instructor learning in a semester-long foreign language teaching methodology course via three data sources: questionnaires, concept maps and instructors' reflective teaching journal entries. Examples of how these instruments can be used in isolation as well as in relation to each other reveal the strengths and limitations of each instrument and their ability to uncover graduate instructor cognitive change (e.g., learning) in a semester-long methods course. These methods will be utilized in an on-going research project examining graduate instructor learning, and will inform the author's future adaptations of her foreign language teaching methodology course. It is this researcher's hope that these methods will be utilized in other language teaching methods contexts as well, especially in addressing the too-often overlooked area of graduate instructor training and learning.

Acknowledgements

The author would like to thank the graduate students with whom she is fortunate to work. Their passion for teaching and learning inspires and validates the challenge to strive for continuous improvement as an instructor. She would also like to thank Melissa Baralt for providing valuable feedback on an earlier

version of this paper.

References

Borg, S. (2006). Teacher cognition and language education: Research and practice. New York: Continuum.

Gurzynski-Weiss, L. (2013). Instructor characteristics and classroom-based second language acquisition of Spanish. In K.L. Geeslin (Ed.), The handbook of second language Spanish (pp. 530-546). Wiley Blackwell.

Gurzynski-Weiss, L. (in press). Graduate instructor in-class cognition and feedback provision over time. To appear in R.T. Miller, K.I. Martin, C.M. Eddington, A. Henery, N.M. Miguel, A. Tseng, A. Tuninetti, & D. Walter (Eds.), Selected proceedings of the 2012 second language research forum.

Kowalski, R.M., & Westen, D. (2008). Psychology (5th edition). Hoboken, NJ: John Wiley & Sons.

Borg, S. (2003). Teacher cognition in language teaching: A review of research on what language teachers think, know, believe, and do. Language Teaching 36, 81-109.

Gurzynski-Weiss, L. (2010). Factors influencing oral corrective feedback provision in the Spanish foreign language classroom: investigating instructor native/non-native speaker status, second language acquisition education, and teaching experience. PhD dissertation, Georgetown University. Washington, DC.

Richards, J.C. (1998). What's the use of lesson plans? In J.C. Richards (Ed.), Beyond training (pp. 10321). Cambridge: Cambridge University Press.

Mackey, A., Polio, C., & McDonough, K. (2004). The relationship between experience, education and teachers' use of incidental focus-on-form techniques. Language Teaching Research, 8(3), 301-27.

McDonough, K. (2006). Action research and the professional development of graduate teaching assistants. The Modern Language Journal 90(1), 33-47.

Peacock, M. (2001). Pre-service ESL teachers' beliefs about second language learning: A longitudinal study. System 29, 177-93.

Gatbonton, E. (2008). Looking beyond teachers' classroom behaviour: Novice and experienced ESL teachers' pedagogical knowledge. Language Teaching Research 12, 161-82.

Johnson, K.E. (1994). The emerging beliefs an instructional practices of preservice English as a second language teachers. Teaching and Teacher Education 10, 439-52.

Dornyei, Z., with Taguchi, T. (2010). Questionnaires in second language research: Construction, administration and processing (2nd edition). New York: Lawrence Erlbaum.

Farrell, T.S.C. (2008). Reflecting language teaching: From research to practice. New York: Continuum.

Baralt, M. (2011). Coding qualitative data. In A. Mackey & S.M. Gass (Eds.), Research methods in second language acquisition: A practical guide. Mahwah, New Jersey: Wiley-Blackwell, 222-44.

Richards, J.C., & Lockhart, C. (1996). Reflective teaching in language classrooms. New York: Cambridge.

Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine.