Application of Rasch Model in Measuring Students’ Performance in Civil Engineering Design II Course Academic research paper on "Educational sciences"

Available online at www.sciencedirect.com

Procedía

Social and Behavioral Sciences

ELSEVIER

Procedia - Social and Behavioral Sciences 56 (2012) 59 - 66

International Conference on Teaching and Learning in Higher Education (ICTLHE 2012) in

conjunction with RCEE & RHED 2012

Application of Rasch Model in Measuring Students' Performance In Civil Engineering Design II Course

Siti Aminah Osmana' b*, Syahdatul Isnain Naamb, Othman Jaafarb,

b b Wan Hamidon Wan Badaruzzaman & Riza Atiq Abdullah O.K. Rahmat

aCentre for Engineering Education Research, Universiti Kebangsaan Malaysia, Bangi, 43600, Selangor, Malaysia

bDepartment of Civil & Structural Engineering, Faculty of Engineering & Built Environment, Universiti Kebangsaan Malaysia, Bangi,

_Selangor, Malaysia_

This paper describes a measurement model used to measure the students' performances in the Department of Civil and Structural Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia (UKM). The assessment model was developed based on students mark entries together with Rasch Measurement Model; it can be used to measure the students' performances in term of course outcomes (COs) for the Civil Engineering Design II Course (KH 4253).This assessment was conducted to all 64 final year students in the Department of Civil and Structural Engineering (JKAS), Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia that have registered for the course in second semester of session 2010/2011. This study shows that Rasch Model can precisely classify and tabulates the students; i.e. Person and Items on a Distribution Map (PIDM) according to their achievements. Comparative analysis against the conventional distribution marks shows that Rasch Measurement Model was found to give almost the same results on the students' achievement and reveals the true degree of learning abilities of the students even with small number of sampling unit.

© 2012Publishedby Elsevier Ltd.Selectionand/orpeer-reviewunder responsibility of CentreofEngineeringEducation, Universiti TeknologiMalaysia

Keywords: Rasch Measurement Model; Course Outcomes; Bloom Taxonomy Level; Civil Engineering Design Course

1. Introduction

The Outcome Based Education (OBE) learning process was implemented by Engineering Accreditation Council (EAC) to all engineering programme at the Higher Learning Institutions (IPTA) in Malaysia since 2004

* Corresponding author. Tel.: +603-8921-6221; fax: +603-8921-6147 E-mail address: saminah@eng.ukm.my

Abstract

1877-0428 © 2012 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of Centre of Engineering Education, Universiti

Teknologi Malaysia

doi:10.1016/j.sbspro.2012.09.632

with the purpose that the degree produced by Malaysian IPTA are recognized by the Washington Accord (WA) (Siti Aminah et al. 2011b). Through OBE approach, the students' performance in carrying out tasks such as exam, quizzes, tutorial, group project, laboratory and presentation can be measured. The measurement and assessment in OBE must always be monitored and these can only be achieved if the mapping of Program Outcome (PO) and Course Outcome (CO) is well designed.

In OBE learning process, the POs of the programme must be formulated first by the faculty or department after considering the EAC guidelines as well as adopting some ABET criteria (Shahrir et al. 2008). Then followed by the COs in which must also be formulated or designed according to the syllabus of the course offered in the faculty or department. Once the POs and COs have been identified the assessment and measurement can be carried out, hence the students' achievement on the expected knowledge with particular skills can be obtained at the end of the semester. However, it was quite difficult to measure the actual performance of each COs since there was no specific method or tool that can accurately measure it (Kamsuriah et al. 2011). Therefore, modern measurement method as practiced using item response theory with a focus on Rasch measurement model was introduced as a new measurement model in measuring COs performance of each students (Kamsuriah et al. 2011).

Rasch Model is a new measurement method that uses data from the students' assessment and transforms it into 'logit' scale thus transform the assessment outcome into a linear correlation with equal interval (Rozeha et al. 2007). In Rasch, it produced a reliable repeatable measurement instrument instead of establishing the 'best fit line' (Azrilah et al. 2008). The results were then evaluated whether it have been accurately assessed and later will be used by the lecturer as guidance in improving the teaching method (Rozeha et al. 2007). The results from the Rasch analysis will provide the lecturers with a more accurate data on the student learning ability achievement as Rasch focuses on constructing the measurement instrument with accuracy rather than fitting the data to suit a measurement model with of errors (Azrilah et al. 2008).

This paper presents the students' achievement on COs in Civil Engineering Design II (KH4253) course using Rasch Measurement Model. The outputs from this study can be used as guidance for the lecturers in monitoring the performance of each COs and simultaneously can also be used to improve teaching delivery method.

2. Methodology

This study was conducted to all 64 final year students of Semester 2 session 2010/2011in the Department of Civil & Structural Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia (UKM). Basically, the Civil Engineering Design course consists of 2 parts in which the first semester students are required to register for Engineering Design I course (KH4013) and in the second semester for Engineering Design II course (KH4253). Students were given the real design project in the first semester and they have to continue with the same project in the second semester. However in this paper, the students' achie vement on COs for the course KH 4253 in the second semester are only presented and discussed here.

The students' achievement on related COs was measured using four type of assessment tools that is; design project, BQ report, presentation and peer assessment. There was no written exam in the course as the whole course only focused in the group project. Each assessment tool has been outlined with specific course outcomes (CO) and program outcomes (PO) that need to be measured. These Cos were linked to each POs using course mapping as shown in Table 1. From the course mapping, there are eight COs that have been identified and need to be assessed. At the end of semester, students are required to submit the group project report with drawing and BQ report. The group presentation will be carried out and will be evaluated by the panel consisting of lecturers

and professional engineers who have been specially invited to evaluate the students. Students were assessed individually and in group. Besides that, students were also need to evaluate their team members (peer assessment) based on their contribution in completing the project.

Table 1. Mapping of POs and COs for KH 4253

IS"o Co urse Outcome {CO}

XIe a sur em eut & As s e s s in e n t Methods

Able to describe project site, identify problems;, constraints and propose concepts and solutions

Able to identify and apply appropriate parameters, assumptions and design criteria in consideration of health, and safety {example: the use of codes of practice}.. ethics, economics,

environment, sustainability Able to carry out manual design calculations based an the required criteria

Able to carry out design and prepare drawings using relevant computer software (Excel spread sheet, AutoCad and other design software) Able to produce presentable report containing executive summarv, introduction, tasks distribution, concepts, design calculations, drawings for tender do cumentation. conclusions, etc.

Able to perform ta.sk.s individuallv and be an effective group member-Able to prepare bill of quantities and cost estimation

Able to execute and deliver tasJc "with integrity and responsibly_

Rep ort &

Presentation

Hep ort &

Presentation

Hep ort &

Presentation

Hep ort &

Presentation

Hep ort &

Presentation

Assessment BQ Report Peer

Assessment

PO1 - Math, science & engineering knowledge

PO2 - Problem solving

PO3 - Project design

PO4 - Ethics

PO5 - Experiment's skill

PO6 - Engineering techniques & tools PO7 - Communication PO8 - Teamwork PO9 - life long learnind

PO10 - project management & entrepreneurship

In this study, the raw marks for each CO from all four assessment tool were compiled and tabulated as shown in Table 2. Students were sorted according to their gender and coded as Xnn where X referring to student's gender; M for Male and F for Female followed by the number of students. Then the raw marks were transformed into logit to attain uni-dimensionality measurement using linear interval scale of rating scale which similar to the typical order rank A-E known as Grade Rating (info). The rated raw mark were then tabulated in Excel*prn format for further evaluation using Rasch software, Winstep. In this format, grade rating was used as the input instead of student mark percentage. Then, the analysis outputs obtained from the Winstep were analyzed to determine the achievement for each given CO.

Table 2.

Siti Aminah Osman et al. / Procedia - Social and Behavioral Sciences 56 (2012) 59 - 66 Students mark entries for each COs

STUDENTS CO1 CO2 CO3 AVERAGE PERCENTAGE (%) CO4 CO5 CO6 CO7 CO8

M01 85 92 93 92 93 100 95 100

F02 72 78 80 77 80 94 80 100

M03 75 77 75 80 80 100 0 100

M04 82 87 87 87 92 100 75 100

F05 75 72 67 75 87 100 75 100

F06 83 75 83 75 82 98 70 100

M07 83 77 72 85 83 100 75 100

M08 73 77 70 77 73 100 90 100

M09 . (M10 - F61) 70 73 73 77 60 100 70 100

M62 77 77 78 83 85 100 75 100

M63 78 77 83 80 80 100 80 100

F64 77 68 68 70 80 100 90 100

3. Results and Discussion

From the analysis of Rasch Model in Winsteps, the Person-Item Distribution Map (PIDM) is established where the output from the analysis is presented as shown in Figure 1. The map details out the exact position of each student in relation to the respective COs. Rasch Model tabulates the persons; i.e. student on the right side and the item; the course outcomes (CO) are plotted on the left side of the map in the same logit scale in line with the Latent Trait Theory which gives a precise overview on the student's achievement of each COs (Siti Aminah et al. 2011c). This will give a clearer view of students' ability towards items difficulty.

In PIDM, item means, Meanitem serves as a threshold and it is set to zero on the logit scale. The higher the location of item from the Meanitem the more difficult the item compared to an item on a lower location. Same goes to person distribution where the excellent students were located at top of the map while the poor students were located at the bottom of the map. Therefore, the level of a person's ability can be identified from PIDM by looking at the separation between the person and item on the map. The bigger the separation, the more able a person is likely to achieve the item (Rozeha et al. 2007).

The PIDM reveals that CO3, CO4 and CO7 are the most difficult items for student to achieve while the easiest items are CO6 and CO8. There is a large separation between easy and difficult item which is shown by a huge gap between CO5 and the lowest for CO6 and CO8. This shows the level of difficulty of the item that students had to deal with during project completion. The end of very difficult item also has a hollow area that needs to be patched up to close the gap so that students' performance level can be divided equally. The findings on the students' achievement of all COs and the level of difficulty of the COs are exactly matched with the results reported by (Siti Aminah et al. 2011a) through the conventional method. Comparative analysis on the percentage of students' achievement in each CO also has similar pattern when compared with that of conventional method where the highest percentage was for CO6 and CO8 whilst the lowest percentage was for CO7.

Fig. 1. Person-item distribution map

Figure 1 shows that the Person mean value, Meanperson for this analysis is 3.02 which is highly above the threshold value, Meanitem = 0. Besides that, totally 53 students (82.8%) were found to be above the Meanitem and the highest person managed to score 9.09 logit. These indicate that the overall student's performance is over the expected performance and students have a good knowledge on the expected COs. These students were able to achieve all the measured COs and this shows that the students have exceeded the level of difficulties for each CO. The achievement of the students shows that they have obtained and managed to solve the given project and almost all of the students have performed well in this course. In contrast, only 11 students (17.2%) were located below the Meanitem and have some difficulty in achieving all COs except for CO5, CO6 and CO8 which are among the easiest item. There are five students with the lowest ability since they only capable on the easiest item (CO6 and CO8) and have the lowest score (-1.89 logit). These students clearly have difficulties in completing the project and understanding this course. Specific corrective action must be carried out to these students in order to improve the achievement of this course. From PIDM also shows that MO1 is the best student in the course since he is located at the top of the figure with the highest scores (9.09 logit) and has the highest ability compared to other students. On the other hand, M46 and M58 are the poorest student for this course since they are located at the bottom of the map and have the lowest score (-1.89 logit).

Table 3 shows the summary statistic for person and item category for the course. According to the table, the value of Cronbach-a = 0.66 which is slightly higher than the acceptable level 0.6. This validate that the model is acceptable. From the analysis also it is found that Person Reliability is 0.57 and Item Reliability is 0.00 which is

rather low. Thus, both person and item category will need further inspection. The value for students' separation also is rather low which is 1.16 and this is not enough to separate them into different performance level.

Table 3. Course outcomes students' assessment: person item statistic

SUMMARY OF 64 MEASURED (EXTREME AND NON-EXTREME) Persons

RAW SCORE COUNT MEASURE MODEL ERROR INFIT OUTFIT MNSQ ZSTD MNSQ ZSTD

MEAN 24. 5 6.0 3.02 1.43

S. D. 2.1 .0 2.46 .68

MAX. 30.0 6.0 9.09 2. 33

MIN. 17.0 6.0 -1.69 . 34

REAL MODEL 1 S.E. RMSE 1.61 ADJ.SD RMSE 1.59 ADJ.SD OF Person MEAN = .31 1 1 .66 SEPARATION .68 SEPARATION 1.16 Person RELIABILITY (Ts?) 1.18 Person RELIABILITY 75E

person raw score-to-measure correlation = .91 (approximate due to missing data) cronbach alpha (kr-20) person raw score reliability = (approximate due to missing data)

summary of 6 measured (non-extreme) Items

RAW SCORE COUNT MEASURE MODEL ERROR INFIT MNSQ ZSTD OUTFIT MNSQ ZSTD 1

MEAN 260.6 64.0 .00 . 30 .94 -.6 .73 -.9

S. D. 3.0 .0 .2:6 .02 .90 2. 5 .65 1. 5

MAX. 267.0 64.0 . 2:5 . 33 2.63 3.9 1.99 1.8

MIN. 256.0 64.0 - . 59 .26 .22 -3. 3 .16 -2.6

REAL MODEL 1 S.E. RMSE . 34 RMSE .30 OF Item MEAN ADJ. SD ADJ. SD = .12 .00 .00 SEPARATION SEPARATION .00 Item .00 Item RELIABILITY RELIABILITY CççD TTO 1 1

minimum extreme score: 2 items

umean=.000 uscale=1.000

Item spread of logit scale from Table 3 shows that the maximum item value, CO7 is at +0.25 logit and the minimum item value, CO6 and CO8 is at -10.86 logit. The difference between logitmax and logitmm is 5 =11.11. This indicates that the Item difficulty of the items spread over 11.11 logit unit (evaluate).

Further analysis on the COs (content) validity can also be carried out through Point Measure Correlation (PMC) as shown in Table 4. In Rasch analysis, item value is considered as misfit only when all three controls (Point Measure, MNSQ and ZSTD) for respective COs were not in the range. From the table, the Point Measure value for CO6 and CO8 = 0.0<0.4. Therefore, it needs further checking since the acceptable value for the Point Measure shall be in between 0.4<x<0.8. This must probably means that the respondent is behaving the opposite way and furthermore the measurement made in these COs was from peer assessment. Usually the trend in giving marks for peer assessment always results with the highest score and these sometimes cannot represent the actual data. Next, is to verify the respondent by looking at the Outfit Mean Square (MNSQ) = y-value where the value must be in the range of 0.5<y<1.5 or else, it will be difficult to obtain accurate result. From the table, it shows that CO7, CO3, CO4, CO1 and CO2 are having MNSQ not in the fit range. The final check would be on the Outfit z-standard (ZSTD) and the value must also be within the range of -2<z<2 or further check were needed. From the analysis, only CO3 and CO2 having ZSTD values outside the fit range. Since, none of the COs have the values out of fit range for all three controls item mentioned above, then all COs are considered in fit range thus the review is not required.

Table 4. Point measure correlation: item validity

TABLE 13.1 KH4253 ZOU522WS.TXT Aug 23 12:56 2011

INPUT: 64 Persons В Items MEASURED: 64 Persons 6 Items 5 CATS 3.66.2

person: real sep.: 1.06 rel. : .54 ... Item: real sep.: .00 rel. : .00 Item statistics: measure order

ENTRY TOTAL MODEL| INFIT | OUTFIT |PT-MEASURE |EXACT MATCH|

NUMBER SCORE COUNT MEASURE S.E. MNSQ ZSTDMNSQ ZSTD|CORR. EXP. | OBS% EXP%| Item

256 259 259 261 261 267 320 320

64 64 64 64 64 64 64 64

. 25 . 17 . 17 . 00 . 00 -. 59 -10. 66 -10. 66

. 26 . 29 . 29 . 30 . 30 . 33 1. 65 1. 65

1. <?q| 1. 6

.41 . 36 -1.5 -1. 6

.iE -2.61

MEAN 275.6 S.D. 25.7

64.0 -2.71 .0 4.71

. 66 I .94

.671 .90

.63 3.9

.33 -2.7

.46 -1.9

.61 -1.2

.22 -3.3

.19 .7

MINIMUM MEASURE

MINIMUM MEASURE ---------+

-.6| .73 -.9| 2. 5 I .65 1. 5 I

4. Conclusion

This study proved that the students' course outcome in Engineering Design II course (KH4253) can be measured using Rasch Measurement Model and the findings also have similar patterns with the conventional method. From the Rasch analysis, students were classified according to their achievement on each CO in which reflects their learning ability in this course. Rasch Model has the ability to produce the association pattern between students and the performance level for each CO which cannot be produced using standard measurement method (Saifudin et al. 2010). This makes Rasch Model as a better assessment model for measuring COs performance. The outputs from Rasch analysis can be used as guidance for the lecturer in monitoring students' performance for each CO in which COs reflect the effectiveness of the teaching and learning plan for any course (Shahrir et al. 2008). From this results it shows that why Rasch Model is used as an instrument for the implementation of quality-focused performance measurement system in IPTA due to its abilities to analyze students achievement more accurately thus making evaluation clearer to read and easier to understand (Saifudin et al. 2010).

Acknowledgements

The authors wish to acknowledge the financial support received from the Centre for Engineering Education Research, Universiti Kebangsaan Malaysia for the grant PTS-2011-001 and PTS-2011-017 in the effort of improving the quality of teaching and learning in engineering education.

References

Azrilah, A.A., Azlinah, M., Azami, Z., Sohaimi, Z., Hamzah, A. G., & Mohd Saidfudin, M. (2008). Evaluation of Information Professionals Competency Face Validity Test Using Rasch Model. 5th WSEAS/IASMEInternational Conference on Engineering Education (EE'08). July 22-24, 2008

Bradley,K.D., Cunningham, J., Haines, T., Mueller, C.E., Royal, K.D., Sampson, S.O., Singletary, G., & Weber, J.A. (2010). Constructing and Evaluating Measures: Applications of the Rasch Measurement Model.

Kamsuriah, A., Nazlena, M. A. & Suhaila, Z. (2011). An Improved Course Assessment Measurement for Analyzing Learning Outcomes Performance Using Rasch Model. Procedia Social and Behavioral Sciences 18 442-449. Elsevier Ltd.

Rozeha, A. R., Azami, Z. & Mohd Saidfudin, M. (2007). Application of Rasch Measurement in Evaluation of Learning Outcomes: A Case Study in Electrical Engineering. Regional Conference on Engineering Mathematics, Mechanics, Manufacturing & Architecture 2007 (EM3ARC).

Saifudin, M., Ghulman, H. A., Razimah, A. & Rozeha, A. (2008). Application of Rasch-based ESPEGS Model in Measuring Generic Skills of Engingeering Students: A New Paradigm. WSEAS Transactions on Advances in Engineering Education, Issue 8 Vol 5, WSEAS Press. Pp. 591-602, August 2008

Saidfudin, M., Azrilah, A.A., Rodzo'an, N. A., Omar, M.Z., Zaharim, A. & Basri, H. (2010). Easier Learning Outcomes Analysis using Rasch Model in Engineering Education Research. EDUCATION' 10 Proceedings of the 7th WSEAS International Conference on Engineering Education. Pp 442-447.

Shahrir, A., Riza Atiq Abdullah, O.K. R, Azami, Z., Norhamidi, M., Baba, M.D., Noorhisham, T.K., Mardina, A., Mazlan, M.T., Andanastuti, M. & Che Husna, A. (2008). Implementing Continual Review Of Programme Educational Objectives and Outcomes For Obe Curiculum Based on Stakeholders' Input. Proceedings Of The 7h WSEAS International Conference On Education And Educational Technology (Edu '08), Venice Italy 21-23 Nov 2008.

Siti Aminah, O., Othman, J., Wan Hamidon, W.B. & Riza Atiq Abdullah, O.K.R. (2011a). Course Outcomes (COs) Evaluation for Civil Engineering Design II Course. Prosiding Kongres Pengajaran & Pembelajaran UKM, 18 - 20 Disember 2011.

Siti Aminah, O., Wan Hamidon, W.B., Kamarudin, A.T., Abdul Khalim, R. & Roszilah, H. (2011b). The Program Outcomes Achievement for Basic Reinforced Concrete Design Course between Two Different Sessions. Procedia Social and Behavioral Sciences 18 (2011) 260-265. Elsevier Ltd. 2011

Siti Aminah, O., Wan Hamidon, W.B., Roszilah, H., Kamarudin, A.T., Noraini, H. & Othman, J. (2011c). Assessment On Students Performance Using Rasch Model In Reinforced Concrete Design Course Examination. Proceedings of 10th WSEAS Internation Conference on Education and Educational Technology (EDU '11). October 3-5, 2011