Appraisal on Bloom's Separation in Final Examination Question of Engineering Mathematics Courses using Rasch Measurement Model Academic research paper on "Educational sciences"

Available online at www.sciencedirect.com

SciVerse ScienceDirect PrOC6d ¡0

Social and Behavioral Sciences

Procedia - Social and Behavioral Sciences 60 (2012) 172 - 178 —

UKM Teaching and Learning Congress 2011

Appraisal on Bloom's Separation in Final Examination Question of Engineering Mathematics Courses using Rasch Measurement Model

Izamarlina Asshaari3*, Haliza Othman, Hafizah Bahaludina, Nur Arzilah Ismaila,

Zulkifli Mohd Nopiahb

aCentre for Engineering Education Research, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia bUnit of Fundamental Engineering Studies, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia

Abstract

This paper studied and analyzed the separation of Bloom's cognitive level in the final exam questions (items) for KKKQ2114 Mathematics Engineering III (Differential Equation) course using Rasch Mesurement Model. The purpose of this study is to use Rasch Measurement Model as a tool to assists lecturers to develop and monitor the distribution and separation of items across content and task domain based on Bloom's Taxonomy (cognitive skill levels). The construction of the item and responses, as well as the process and content of the item contribute to the classification of an item by Bloom's cognitive level. This study revealed that the separation and the distribution of the final exam questions for KKKQ2114 should be revised because there is a large gap in each cognitive level of Bloom's. The level of difficulty for each cognitive skill also need to be reviewed and adapted to CO and PO of the course.

© 2011Published byElsevierLtd. Selection and/orpeer reviewedunderresponsibilityofthe UKMTeachingand LearningCongress 2011

Keywords: Rasch Measurement Model; blooms' taxonomy; cognitive level; course learning outcome

1. Main text

Mathematical knowledge plays an important role in supporting a large number of engineering courses and subsequently, it is vital for engineering students to embrace a strong mathematical problem solving abilities that can keep their motivation for reasonable progress of their engineering programs. The objective of teaching mathematics to engineering students is to find the right balance between practical applications of mathematical equations and in-depth understanding of living situation (Sazhin, 1998). On the other hand, the impact of teaching mathematical thinking skills on an engineer will enable them to use mathematics in their practice (Cardella, 2008).

Zainuri et al. (2009) and Othman et al. (2010) studied the Mathematics Pre-Test performance of engineering students at Faculty of Engineering and Built Environment (FKAB), Universiti Kebangsaan Malaysia (UKM) and the results agreed with the findings found by Lawson (2003), that there has been a significant decline in many

* Corresponding author. Tel.: +6-03-8921-6681; fax: +6-03-8921-6960. E-mail address: emiey@vlsi.eng.ukm.my.

ELSEVIER

1877-0428 © 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and Learning Congress 2011 doi:10.1016/j.sbspro.2012.09.364

mathematical skills regarded by higher education as critical for those whose taking degree courses with significant mathematical content. According to Ball et al. (2001), one of the difficulties in teaching mathematics is that the students do not understand the importance and usefulness of mathematical problem solving until they put it into practice. Thus, FKAB has taken several measures to improve students' performance and achievement, such as, introducing Problem Based Learning (PBL) and Cooperative Learning (CL) as an alternative teaching method in order to improve students' ability in learning and also revise CLO for all engineering courses. However, according to Sun et al. (2009), students' understanding on the course content is more important and has long been supported by educators. A suitable assessment tools in teaching and learning process is required to measure students' understanding fairly and equally. Hence, all the courses in the faculty are designed to meet CLO, which is constructed to measure generic, cognitive skills and students' performance, based on Bloom's Taxonomy (Cognitive Skill Level).

Ghulman & Masodi (2009) mentioned that students' performance mostly depends on how the students carry out tasks such as series of tests or quizzes, final examination and assignments. Good tests or quizzes, final examination and assignments must provide the same level of cognitive thinking skills to all students on what they have learned. A well organized and developed exam questions, according to Bloom's cognitive thinking skill, contribute to the increase in students' performance. Thus, in the process of constructing examination questions, it is crucial to fairly distribute exam questions based on Bloom's Cognitive Separation. In this paper, the final exam questions (items) for KKKQ2114 course was studied and analysed for its Bloom's Cognitive separation.

This study used Rasch Measurement Model in analysing Bloom's Cognitive Separation for final exam question of KKKQ2114 course. Ghulman & Masodi (2009) mentioned that Rasch Measurement Model is useful with its predictive feature to overcome missing data. Meanwhile, Aziz et al. (2007) stated that Person and Items Distribution Map (PIDM) can give a precise overview of the student's achievement on a linear scale of measurement. Rashid et al. (2007) and Masodi et al. (2010) used Rasch Measurement Model in evaluation of learning outcome for the Electrical Engineering Program in UKM and Engineering Education Research, respectively.

This paper focuses on the analysis on the separation of Bloom's cognitive level in the distribution of final exam questions of KKKQ2114 course using Rasch Measurement Model. It is part of the study to help and enhance students' ability in solving mathematics problems. Therefore, the increase in students' performance in Engineering Mathematics courses at FKAB, UKM can be improved significantly.

2. Methodology

The study was based on the final exam questions of second year students of FKAB, UKM. Data from 218 students from four different engineering departments, namely: Department of Civil and Structural Engineering (JKAS), Department of Mechanical and Materials Engineering (JKMB), Department of Chemical and Process Engineering (JKKP) and Department of Electricals, Electronics and System Engineering (JKEES); were collected. There are 30 questions, including the sub-questions; from the final examination questions of KKKQ2114 course studied. The final consists of three parts, which are Part A, Part B and Part C where students are required to answers all questions in Part A and B, while Part C is an optional question.

The course outcomes and the Bloom's Taxonomy cognitive expected from students to achieve for KKKQ2114 course are shown in Table 1 and 2. Blooms' Taxonomy cognitive thinking levels define into six domains from simplest to complex; Level-1: Knowledge, 2: Understand, 3: Apply, 4: Analyse, 5: Evaluate, and 6: Synthesis are all measurable.

For KKKQ2114 course, the students were expected to develop Level 1-4; i.e. understand and apply knowledge acquired to analyse situations requiring them to provide the appropriate solutions. Table 2 shows the topics and Blooms' Taxonomy domain assessed for each questions.

Izamarlina Asshaari et al. /Procedia - Social and Behavioral Sciences 60 (2012) 172 - 178 Table 1. Course outcomes and blooms' Taxonomy Domain for KKKQ2114

No. Course Outcomes

Blooms' Taxonomy

Understand the basic concepts of differential equations and their solutions. Comprehensive

Able to solve first and second order ordinary differential equations. Application

Able to determine the Laplace transforms and the inverse Laplace transforms of Analysis elementary functions.

Able to build and solve a differential equations model of problems involving half-life, Application mixing problem, spring-mass system and electric circuits.

Able to determine the Fourier series, integrals and transforms of simple functions. Application

Know the types of partial differential equations and their applications in engineering. Knowledge

Table 2. Topics and Blooms' Taxonomy Domain assessed for each examination question

Qs. Entry No. Learning Topic Blooms' Taxonomy

1a A01_C Definition and Terminology Comprehension (C)

1b A02_K Solution curve Knowledge (K)

1c A03_K Solution curve Knowledge (K)

2ai A04_P Homogeneous equation Application (P)

2aii A05_P Homogeneous equation Application (P)

2bi A06_C Variations of parameter Comprehension (C)

2bii A07_P Variations of parameter Application (P)

3ai A08_K Laplace Transforms Knowledge (K)

3aii A09_P Laplace Transforms Application (P)

3b A10_P Inverse Laplace Transforms Application (P)

4a A11_P Series Solution Application (P)

4bi A12_C Fourier Series Comprehension (C)

4bii A13_C Fourier Series Comprehension (C)

4c A14_C Heat Equation Comprehension (C)

a B15_K Definition and Terminology Knowledge (K)

b B16_P Homogeneous Equation Application (P)

c B17_A Particular Solution using Undetermined Coefficient General Solution for RLC circuit Analysis (A)

d B18_A Initial Value Problem Analysis (A)

e B19_P Steady State Solution for RLC circuit Application (P)

f B20_A Solution for RLC Analysis (A)

1a C21_P Population Growth Application (P)

1b C22_A Limiting value of Population Growth Analysis (A)

2a C23_P Damping Force Application (P)

2b C24_C Equation of Motion for Spring Mass Comprehension (C)

2c C25_P Equilibrium Position Application (P)

3a C26_C Inverse Laplace Comprehension (C)

3bi C27_C Unit Step Function Comprehension (C)

3bii C28_P Unit Step Function Application (P)

3ci C29_A RLC circuit in Laplace Analysis (A)

3cii C30_P Unit Step Function in RLC circuit Application (P)

The questions are entered as entry number as shown in Table 2. The students are labelled as gender number, ethnic and department; i.e., student number 1 is a Malay male from Department of Civil and Structural Engineering, thus, M00111, and student number 30 is a Chinese female from Department of Electricals, Electronics and System Engineering, thus, F03024. The item are labelled as Question No., Taxonomy Bloom Domain and Learning Topic, thus for entry number 1, the item is QA01_C (Table 2).

The final exam results were compiled. Since these raw score have different total marks for each question, a standardization method is used. The formula for the standardization is given below:

z _ xj - mm Xj (1)

max xj

where i = the ith students (i = 1, 2, ... , 218), j = the jth questions (J = 1, 2, ..., 30), zj = standardized marks for ith student and jth question, xij = marks for ith student and jth question, min Xj = minimum marks for jth question, and max Xj = maximum marks for jth question.

Responses from the students' exam results were analysed using rating scale in which the students were rated according to their achievement. From (1),

zv x10 = A (2)

Then, A is classified correspond to the rating scale as in Table 3.

Table 3. Marks (A) and correspond rating scale

Marks (A) 0-1.49 1.50-3.49 3.50-6.49 6.50-8.49 8.50-10.00

Rating Scale 1 2 3 4 5

This grade rating is tabulated in Excel*prn format. This numerical coding is necessary for further evaluation of the students' achievement using Winstep software version 3.68.0. The analysis outputs obtained from the Winstep were analyzed.

3. Data Analysis and Discussion

The summarised statistics of the result are given in Figure 1, which summarises the persons and items involved in this study. In the Rasch Measurement Model, persons represent the students, and the items represent the questions asked. The summary statistics contain information of the mean, standard deviation, and maximum and minimum values for both persons and items, where the maximum and minimum of the person and item spread are reflected in the Standard Deviation (SD) in the Person Item Distribution Map (PIDM) in Figure 2. This is called the distribution of the students, and the questions are based on the logit ruler.

The PIDM shows a better picture on how the person correlates to the respective items. It can give a clearer view of the person's ability and relevant item difficulty. For this study, the items were the main concern. The higher ranks of the items indicate the item is more difficult and vice versa.

Figure 1 reveals a good reliability of Cronbach a with 0.86. It shows that a good spread of 3.07logit, where MaxPerson = 0.85 and MinPerson= -2.22 and Person Reliability=0.86. The major finding in Figure 1 is the Person Mean, ^Person = -0.41 logit which is lower than the threshold value, ^Item= 0. These values show that the Students were found to be lower than the expected performance with two groups of student separation (G = 2.44). It also

gives a good summary with item separation, G=6.60 and a very high reliability=0.98. It has a good item spread of 2.60logit with SD,=0.54.

summary of 218 measured Persons

mean 5. D. max.

67. 2 17. 2 108.0

27. 5 1. 2

28. О

MODEL INFIT OUTFIT

MEA5URE ERROR MN5Q ZSTD MNSQ ZSTD

-.41 .18 1.04 .0 1.02 .0

. 53 . 04 . 38 1.3 .42 1. 2

. 85 .41 2.46 3. 5 2. 95 4. 7

-2. 22 .15 .44 -3.1 .42 -2.4

REAL RMSE .20 ADJ.SD MODEL RMSE .18 ADJ.SD S.E. OF Person MEAN = .04

.49 I SEPARATION 2.44 ¡Person RELIABILITY .86 1 .50 1 SEPARATTÖN—У.ЬУ 'Person RELTABTLTTV . ИИ

VALID RESPONSES: 91.6%

PfrfTftn RAW .yORR-TO-MFftiri/RF TORRFI ATIflN = .97 fflnnrftXl'mnTF. due to missing data) £RONBACH ALPHA (KR-2Q) Person RAW SCORE RELIABILITY - . 86 ¡(approximate due to missing data)

summary of 30 measured Items

RAW MODEL INFIT OUTFIT

SCORE COUNT MEA5URE ERROR MN5Q ZSTD MNSQ Z5TD

MEAN 488. 3 199. 6 00 .07 1.02 -.4 1.01 -. 3

S. D. 183. 8 44.1 Ь4 .02 .43 4.4 .40 3.4

MAX. 987.0 218.0 83 .15 2.00 8. 3 2.01 8. 8

MIN. 80.0 40.0 -1 // .06 .41 -8.0 . 51 -5. 6

REAL RMSE . 08 ADJ. SD . 53 1 SEPARATION 6. 60 1 Item RELIABILITY . 981

MODEL RM5E .07 ADJ . 5D . 53 5EPAÜAH5N У.Л Item RkLIAUILI1V . УВ

S. E. OF item MEAN = .10

umean=.ooo uscale=1.000

item raw score-to-measure correlation = -.80 (approximate due to missing data) 5988 data points. log-likelihood chi-square: 14883.21 with 5738 d.f. p=. 0000

Figure 1. Summary statistics

""" IIIIII I

»»• »I '«■■J*

kd 7 I' c2i i

- ' S3 VERY DTFFTCULT

DTFFTCULT

MODERATE

VERY EASY

Figure 2. Person-Item Distribution Map (PIDM)

From Figure 2, only 26.61% (N=58) of the students were found to be above the ^Item whilst 73.39% (N=160) students are below the ^Item. Overall, 19 questions are difficult for students to answer where question QA10 (Application) is the most difficult question while QB15 (Knowledge) is the easiest question for the students as shown in Figure 2.

As in Figure 2, exam questions were categorised into five different categories. The categories are 'very difficult', 'difficult', 'moderate', and 'easy' to 'very easy'. It was also noted that there is a huge gap indicating very easy questions denoted by ( < ► ), between QA02 (Knowledge) and QB15 (Knowledge). The most difficult item in comprehension level; QA06, QA12 are noted by the gap against QA13 and QA14; application; QA10, QC30 and QC25 are noted by the gap against QA05, QA07, and QC28; and analysis level; QC29 is noted by the gap against QB20.

Lowest cognitive domain, which is knowledge, is considered as easy. From Figure 2, the maximum logit for knowledge cognitive level is 0.37logit; i.e. items QA03 and QA08, whilst item QA06 is the highest item for comprehension level with 0.65logit. For higher cognitive domain, which is application level, the highest item is QA10, with 0.83logit, whereas QC29 is the highest item for analysis level with 0.56logit.

Also in Figure 2, the lowest item is QB15 with -1.77logit. This item is belonged to knowledge cognitive level and categorised as the most and easy question. As for comprehension level, the lowest item is QC26 with -0.37 and categorised as moderate item. QB16 and QB17 represent application and analysis level respectively, and these items is categorised as easy question with -1.10logit and -0.50logit respectively.

As extension of the result in Figure 2, Table 4 summarises the measurement for each level of Blooms' Taxonomy. Bloom's define cognitive learning levels into six domains from the simplest to complex; knowledge, comprehension, application, analysis, evaluation and synthesis. For KQ2114, the students were expected to develop Level 1 - 4; i.e. understand and apply knowledge acquired to analyse situations requiring them to provide the appropriate solutions.

Table 4. Summary statistics for each Blooms' level

Level ^"■""•»■x. Measurement Blooms' Knowledge Comprehension Application Analysis

N 4 8 13 5

Mean -0.33 0.13 -0.01 0.10

S.D 0.86 0.34 0.52 0.36

Max 0.37 0.65 0.83 0.56

Min -1.77 -0.37 -1.10 -0.50

As in Table 4, the spread for knowledge cognitive level is 2.14logit with ^Knowledge = -0.33logit while for comprehension cognitive level, the spread of 1.02logit with ^Comprehension = 0.13logit, . As for application level, it has good item spread of 1.93logit with ^Application = -0.01logit. However, for analysis cognitive level, it has poor item spread of 1.06logit with ^Analysis = 0.10logit.

4. Conclusion

Rasch Measurement Model obviously is an effective tool in measuring the separation of Blooms' cognitive level for the questions (item) of KKKQ2114 Engineering Mathematics III (DE). It gives the profiling of the questions constructed and makes good prediction regarding the constructed questions. Results show that some final examination questions need to be revised and reviewed, including some changes on the level of difficulty for each question (based on Bloom's Taxonomy) and the number of questions in each level of cognitive skills have to be distributed uniformly. In addition, the coverage of the questions must base on the content learned by the students and should meet the expectation of CLO. Thus, the questions were measured on the ability of student learning and subsequently, contribute to the improvement and enhancing engineering students' academic performance.

Using Rasch Measurement Model, the result more accurately classified the questions according to their Blooms' Taxonomy level. It enables each item to be evaluated discretely. It also accurately classified the students according to their observed achievements.

Acknowledgements

We would like to thank UKM for providing the research grant (UKM-PTS-2011-020). References

Aziz, A. A., Mohamed, A., Arshad, N. H., Zakaria, S., & Masodi, M. S. (2007). Appraisal on Course Learning Outcomes using Rasch Measurement: A case study in Information Technology Education. International Journal of Systems Application, Engineering and Development, 164-171.

Ball, D. L., Lubienski, S., & Mewborn, D. (2001). Research on teaching mathematics: The unsolved problem of teachers' mathematical knowledge. In V. Richardson (Ed.), Handbook of research on teaching (4th ed.). New York: Macmillan

Cardella, M. E. (2008). Which Mathematics should we teach engineering students? An empirically grounded case for a broad notion of Mathematical Thinking. Teaching Mathematics and Its Applications, 27(3), 150-159.

Ghulman, H. A. & Masodi, M.S. (2009). Modern measurement paradigm in Engineering Education: Easier to read and better analysis using Rasch-based approach. 2009 International Conference on Engineering Education (ICEED 2009), December 7-8, 2009, Kuala Lumpur, Malaysia.

Lawson, D. (2003). Changes in student entry competencies 1991-2001, Teaching Mathematics and Its Application, 22(4), 171-175.

Masodi, S., Aziz, A. A., Rodzo'an, N. A., Omar, M. Z., Zaharim, A. And Basri, H. (2010). Easier learning outcomes analysis using rasch model in engineering education research. EDUCATI0N'10 Proceedings of the 7th WSEAS International Conference on Engineering Education, 442-447.

Othman, H., Ariff, F. H. M., Ismail, N. A., Asshaari, I., Zainuri, N. A, Razali, N., & Nopiah, Z. M. (2010). Engineering students' performance in mathematical courses: The case study of Faculty of Engineering & Built Environment, Universiti Kebangsaan Malaysia. Proceeding of The 1st Regional Conference on Applied and Engineering Mathematics Proceedings (RCAEM), Eastern & Oriental Hotel, Pulau Pinang, Malaysia.

Rashid, A. R., Zaharim, A. & Masodi, S. (2007). Application of Rasch Measurement in evaluation of Learning Outcome: A case study in Electrical Engineering. Regional Conference on Engineering Mathematics, Mechanics, Manufacturing & Architecture (EMARC) 2007, 151165.

Sazhin, S. S. (1998). Teaching mathematics to engineering students, International Journal Engineering Education, 14(2), 145-152.

Sun, Y. & Chen, W. (2009). The relationship between teaching comprehensibility and instructional time vs. students' achievement in rational numbers. The Journal of Human Resource and Adult Learning, 5(2), 99-107.

Zainuri, N. A., Nopiah, Z. M, Razali, N., Asshaari, I. & Othman, H. (2009). The study on the weaknesses of Mathematical Foundation in the First Year Engineering Students, UKM, Prosiding Seminar Pendidikan Kejuruteraan & Alam Bina (PeKA09), 226-233.