Scholarly article on topic 'Measurement and Evaluation of Program Outcomes in the Civil Engineering Courses'

Measurement and Evaluation of Program Outcomes in the Civil Engineering Courses Academic research paper on "Civil engineering"

CC BY-NC-ND
0
0
Share paper
OECD Field of science
Keywords
{Measurement / Evaluation / "Outcome Based Learning" / "Programme Outcomes" / "Final Year Students"}

Abstract of research paper on Civil engineering, author of scientific article — Azrul A. Mutalib, Riza A.A. Rahmat, Abd. Khalim Abd. Rashid, Fatihah Suja, Suraya Sahril

Abstract Outcome Based Learning System (OBL) contains six programme objectives and 12 programme outcome (PO) that can be measured in some certain courses. Measurement and evaluation are important aspects in determining whether or not a programme outcome is achieved. Nevertheless, measurement is one of the hardest aspects to be done because there is no proper method or range that can be used. This paperwork contains the measurement of programme outcome as an implementation in Civil Engineering Programmes’ courses. Those courses are Computer Programming (KF1243), Engineering Measure (KH2243), Liquid Mechanic (KH2133), Design Project II (KH4253) and Academic Project II (KH4203). Aside from that, there is also a comparison made between the measurements of the programme outcome done by lecturers with ones from the perspective of final year students of 2005/2006 session. It is hoped that this implementation of measurement method will be used as reference by lecturers at the Engineering Faculty, Universiti Kebangsaan Malaysia (UKM) particularly, and in Malaysia generally.

Academic research paper on topic "Measurement and Evaluation of Program Outcomes in the Civil Engineering Courses"

Available online at www.sciencedirect.com

SciVerse ScienceDirect

Procedía - Social and Behavioral Sciences 60 (2012) 333 - 342

UKM Teaching and Learning Congress 2011

Measurement and Evaluation of Program Outcomes in the Civil

Engineering Courses

Azrul A. Mutaliba'b'*, Riza A. A. Rahmata,b, Abd. Khalim Abd. Rashidab, Fatihah Sujaab,

Suraya Sahrila b

aCentre for Engineering Education Research, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia bDepartment of Civil and Structural Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia

Abstract

Outcome Based Learning System (OBL) contains six programme objectives and 12 programme outcome (PO) that can be measured in some certain courses. Measurement and evaluation are important aspects in determining whether or not a programme outcome is achieved. Nevertheless, measurement is one of the hardest aspects to be done because there is no proper method or range that can be used. This paperwork contains the measurement of programme outcome as an implementation in Civil Engineering Programmes' courses. Those courses are Computer Programming (KF1243), Engineering Measure (KH2243), Liquid Mechanic (KH2133), Design Project II (KH4253) and Academic Project II (KH4203). Aside from that, there is also a comparison made between the measurements of the programme outcome done by lecturers with ones from the perspective of final year students of 2005/2006 session. It is hoped that this implementation of measurement method will be used as reference by lecturers at the Engineering Faculty, Universiti Kebangsaan Malaysia (UKM) particularly, and in Malaysia generally.

© 2011PublishedbyElsevier Ltd. Selection and/orpeer reviewedunderresponsibilityofthe UKMTeachingand LearningCongress 2011

Keywords: Measurement; Evaluation; Outcome Based Learning; Programme Outcomes; Final Year Students

1. Introduction

Outcome based learning system (OBL) had been introduced in Engineering Faculty, Universiti Kebangsaan Malaysia (UKM) since 2003 after Malaysia was admitted to the Washington Accord as a provisional signatory nation alongside Singapore and Germany (Basri et al., 2004). In order to implement this system, a program must formulate the program educational objectives and identify the program outcomes (PO) that directly address the educational objectives and incorporate certain specified outcomes (Felder & Brent, 2003). Various approach and steps had been introduced as well as series of courses, workshops and seminar had been carried out to ensure that this system is a success. Measurement and evaluation of PO as shown in Figure 1 is one on the most important aspect to determine the OBL system is successful. This is to ensure the OBL system is improved constantly. Besides

* Corresponding author. Tel.: +6-03-8921-6212; fax: +6-03-8921-6417. E-mail address: rul@eng.ukm.my.

1877-0428 © 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and Learning Congress 2011 doi:10.1016/j.sbspro.2012.09.388

that, data and evidence collecting system of measurement and evaluation carried out in the courses should be introduced so that records can be kept and compiled safely and effectively.

Since measurement and evaluation are important in accreditation programme at Civil and Structural Engineering Department (CSED), this paperwork will discuss a few important matters to fulfil the stated objectives:

(a) to discuss methods to perform measurement and evaluation of programme outcome at CSED;

(b) to analyse direct measurement of programme outcome of some courses in Civil Engineering Programme; and

(c) to analyse indirect measurement of programme outcome of some courses in Civil Engineering Programme.

Innovative teaching method

Innovative measurement and assesement method

Data and evidence collecting system

Figure 1. Steps to perform outcome based learning system

2. Definition of measurement and evaluation

According to Wan Hamidon, 2005;

1) Measurement is a process of identifying, collecting, using and preparing data to asses the achievement of programme objective and outcome, while

2) Evaluation is a process to interpret data and evidence from the measurement and to identify whether or not the programme outcome had been achieved and steps that need to be taken to improve the programme.

In order to improve and increase the courses offered, the measurement and evaluation process as shown in Figure 2 is carried out. Suitable assessment method needs to be chosen, depending on the expected course outcome and the delivery method (Rashid et al 2008, Jaafar et al 2008).

Various methods of measurement

Displays the achievement

Programme objectives and outcome

Results from assesement is used to

Programme improvement

Figure 2. Process of measuring and assessing programme objectives and outcome

3. Implementation approaches at CSED

A few series of workshop were held to ensure the OBL approach at CSED is a success. A workshop held in 16 February 2006 venue at the Danau (Annual Report 2006), UKM had agreed that programme outcome for Civil and Structural Engineering and Civil and Environmental Engineering Programme are followed as in Table 1 and Table 2. Contributions by each compulsory course to the program's learning outcomes were also revised in this workshop to make sure that the programme outcome had been stressed reasonably from each compulsory course. List of contribution is shown in Figure 3 (Annual Report, 2006).

Table 1. Programme outcomes for Civil and Structural Engineering Programme

PO # Program Outcomes

PO1 Ability to gain and apply basic engineering knowledge

PO2 Ability to communicate effectively not only among engineers, also with non-technical society

PO3 Possessed high technical skills in the field of civil and structural engineering

PO4 Ability to identify civil engineering problems also able to obtain solution using modern engineering tools

PO5 Ability to adapt systemic approaches in designing civil engineering infrastructure and performing economical evaluation.

PO6 Ability to function effectively individually or in group with capabilities to become a leader or manager, as well as an active group member.

PO7 Understanding the responsibility and ethics as a professional engineer from social, cultural, global and environmental aspect as well as the need for an eternal development.

PO8 Posses a prediction of needs to undergo lifelong learning, and the capacity to do so.

PO9 Ability to design and execute tests, and also able to analyse and interpret data.

PO10 Ability to function in a multi-discipline group.

PO11 Possessed knowledge of current issues related to civil and structural engineering

PO12 Understanding the elements of managing construction projects, wealth management, civil regulations, administration, business and entrepreneurship.

Table 2. Programme outcomes for Civil And Environmental Engineering Programme

PO# Program Outcomes

PO1 Similar to Table 1

PO2 Similar to Table 1

PO3 Possessed high technical skills in the field of civil and environmental engineering

PO4 Similar to Table 1

PO5 Similar to Table 1

PO6 Similar to Table 1

PO7 Similar to Table 1

PO8 Similar to Table 1

PO9 Similar to Table 1

PO10 Similar to Table 1

PO11 Possessed knowledge of current issues related to civil and environmental

engineering.

PO12 Similar to Table 1

Figure 3. Contribution of course outcomes to programme outcomes (Source: Annual Report, (2006))

The CSED had also appointed a Programme Outcome's Mark Coordinator to generate a data collecting system and assessing the PO from the PO marks measured in offered courses (CO). The approaches used in the system are shown in Figure 4.

Figure 4. Collection of marks from measurement and evaluation of PO process

An information gathering system was established as shown in Figure 5. This enables the department to record and asses the PO performance of students for each session until their graduation. Figure 6 shows how the program can be improved. Measurement and evaluation on programme outcome not only showed the lowest PO but also the list of weak students for a particular PO. This enables the department to take more proactive measures to aid the students identified as weak.

Figure 5. Information gathering systems

camp, etc

Figure 6. Continuous quality improvement processes

4. Measurement

4.1. Direct Measurement

Measurement of programme outcome had been done in a few courses for Civil Engineering Programme such as KH1243 Computer Programming, KH2243 Engineering Measure, KH2133 Liquid Mechanics, KH4203 Academic Project, KH4213 Design Project and KH4543 Town's Mobility Planning.

A formative and summative measurement is obtained by giving out test and exam questions to the level of its application, synthesis and evaluation in Bloom taxonomy. This process went on during the evaluation of the questions by the board of evaluation. Measurement of learning outcome related to exams and tests such as steep knowledge and analytical skills, was measured by collecting the marks of courses that contribute to the direction of achieving the particular learning outcome. Personal skills measurement such as ability to work in groups, communication and lifelong learning is carried out by observing the students' behaviour while conducting assignments and projects.

Starting on the first semester of 2006, measurements of learning outcome listed in department's courses had been started. More attention should be given to academic project and final year design project (Capstone project). Figure 7-10 shows the average overall point for courses that had undergo PO measurement. Only the 4th year overall PO mark such as in Diagram 10, shows the total mark for 12 PO because it involves two identical courses and all 12 PO were measured.

Figure 7. Average PO measurement mark for KH1243

Figure 8. Average PO measurement mark for KH2243

Figure 9. Average PO measurement mark for KH2133

80 -70 -£60 -I-

¡50 -H <u

°"40 -j-l

30 -20 -10 0

Lowest 82%

Highest 89%

Lowest 82%

PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 P010 PO11 PO12

Figure 10. Average PO measurement mark for KH4213 and KH4203

4.2. Indirect Measurement

Indirect measurement had been used to conduct an exit survey and peer assessment. Survey forms had been distributed to all final year students of CSED. Results form the survey is shown in Table 3. From this, it is known that achievement aspect for all programme outcomes (PO) is good except for PO5 where its achievement is moderate.

Table 3. Ranking of PO achievements from the student's perspective

EXIT SURVEY

PO10 4.08 Good

PO7 4.00 Good

PO8 3.90 Good

PO1 3.88 Good

PO2 3.88 Good

PO6 3.88 Good

PO9 3.85 Good

PO3 3.83 Good

PO11 3.73 Good

PO12 3.64 Good

PO4 3.61 Good

PO5 3.39 Moderate

Perception of PO achievement is categorized as Good, Neutral or Weak achievement. This is shown in Figure 11. There are 5 POs that is weak in achievements, which are PO5, PO8, PO9, PO11 and PO12. PO5 tops the list of weak achievement result with 8% and modest achievement for PO5 leads the good achievement at 47% and 44% respectively. PO12 is the second lowest good achievement with 64% compared to 34% and 2% for modest and weak achievement respectively. Most of the PO achievement perception shows percentage of good achievement is above 70%, where PO1Q shows the highest good achievement with 88%.

o t HP7

CS HP9

Q. PHP11

78 22 0

8 0 20 0 20 0

76 76 24 0 24 0

44 1 47 8 8

75 75 25 0

81 81 1 90

73 73 25 2

76 76 22 2 22 2

88 88 12 0 12 0

73 73 25 2 25 2

64 1 34 2

10%_20%

□ Good ONeutra! OWeak

30% 40% 50% 60% Percentage

70% 80% 90% 100%

Figure 11. Aspects of 12 PO achievements throughout the study

Direct measurement was also conducted by the lecturers in Designing Project and Academic Project courses. Comparison between direct and indirect measurement (exit survey) is shown in Figure 12. The results of PO achievement is rather different or contra, where, through direct measurement, PO9 is the highest while PO6 and PO10 is the lowest. Contrary to the student's perception, they thought that their PO10 is the highest and PO5 is the lowest.

<u ra c 0) o cu Q.

100 90 80 70 60 50 40 30 20

PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 P010 PO11 PO12

□ Exit Survey

Lecturer's evaluation

Figure 12. Achievement aspect of 12 PO throughout the study

5. Conclusion

A committee should be established to develop an integrated 'tool of assessment and evaluation' that can be used by all lecturers to measure and evaluate programme outcome. In the department level, the standard operating system was introduced which includes collection of marks from the POs assessment, information gathering system and process of continuous quality improvement. Two types of measurements have been conducted in CSED to measure the achievement of the POs. Direct measurement was carried out in few courses while the indirect measurement had been used in the exit survey of final year students. In direct measurement PO9 is the highest score while PO6 and POlO are the lowest. This ic contrary to the student's perception where they thought that their POlO achievement is the highest and PO5 is the lowest. Hopefully with that this start of implementing courses measurement can be used as references to all the lecturers at Faculty of Engineering in UKM and all around the world.

Acknowledgement

We would like to thank UKM for providing the research grant (PTS-2011-141). References

Basri H., A.B. Che Man, W.H. Wan Badaruzzaman and M J.M. Nor, (2004). Malaysia and the Washington Accord - What it takes for full

membership. International Journal of Engineering and Technology, 1(1), 64-73. Jawatankuasa Semakan Kurikulum dan OBE. (2006). Annual Report 2006, Department of Civil and Structural Engineering, UKM. Jaafar M.S., N. K. Nordin, R. Wagiran, A. Aziz, M.J.M.M. Noor, M.R. Osman, J. Noorzaei and F.N.A. Abdulaziz. (2008). Assessment Strategy

for an Outcome Based Education. International Conference on Engineering Education. Rashid R.A., R. Abdullah, A. Zaharim, H.A Ghulman and M.S. Masodi. (2008). Engineering Students Performance Evaluation of Generric Skills

Measurement: ESPEGS Model. 5th WSEAS / IASME International Conference on Engineering Education (EE'08). Felder R.M. and R. Brent, (2003). Designing and teaching courses to satisfy the ABET engineering criteria. Journal of Engineering Education, 92(1), 7-25.

Badaruzzaman W.H.W. (2005). Outcome-Based Approach to Education — An Overview. Port Dickson, 21hb Disimber 2005.