Scholarly article on topic 'Program Outcomes Measurement and Assessment Processes'

Program Outcomes Measurement and Assessment Processes Academic research paper on "Educational sciences"

CC BY-NC-ND
0
0
Share paper
OECD Field of science
Keywords
{"Direct Assessment" / accreditation / "program outcomes" / "course outcomes" / "program outcome assessment" / "course outcomes assessment ;"}

Abstract of research paper on Educational sciences, author of scientific article — Hamimi Fadziati A. Wahab, Afida Ayob, Wan Mimi Diyana W. Zaki, Hafizah Hussain, Aini Hussain, et al.

Abstract The current engineering program accreditation criteria requires an effective program outcome (PO) assessment procedure along with well documented results, a complete description of the evaluation process that involves extensive participation of the faculty members. To achieve all these, an effective method to estimate students’ performance as well as the effectiveness of the program outcomes (PO) is required. As such, the direct assessment method has been identified and deemed suitable to directly assessed specific students’ knowledge during examination or through skills observation. Specifically, this work reports the effort made by the Department of Electrical, Electronics and System, (EESE), Universiti Kebangsaan Malaysia (UKM) to implement the direct assessment method to measure the PO achievement of its students. This is in line with the EESE Dept.’s continuous quality improvement (CQI) practice to improve its PO assessment method before the next visit by the Accreditation Council in 2011. To do so, the assessment results from each semester of the academic sessions 2008/2009 and 2009/2010 are analysed to produce the annual PO. Measurement of PO achievement is carried out on selected sampled courses. In selecting the courses for the assessment, at least two courses from each academic year that are offered in the two different semesters are selected. Since the academic program is a 4 year program, 8 courses need to be selected for the purpose. Accordingly, this paper is written to describe the assessment method used to measure the PO via direct assessment of the course outcomes (Cos) as an improvement to previous practice where the COs were assessed indirectly using survey and examination results. The direct assessment results showed that the average PO attainment for both academic session 2008/2009 and 2009/2010 are greater than 3.0 and fulfils the benchmark value set by the department. Nevertheless, the Dept. is still putting up an effort to improve the assessment method in order to find the best method to directly assess both COs and POs simultaneously.

Academic research paper on topic "Program Outcomes Measurement and Assessment Processes"

Available online at www.sciencedirect.com

\ ScienceDirect Procedia

Social and Behavioral Sciences

Procedia Social and Behavioral S ciences l!S(S()ll) 4^55

Kongres Cengajaran San Cembelajaran UKM, 2515

Cragram Outcomes Measurement anS Assessment Processes

Hamimi FaSziati A. Wahab, AfiSa Ayob, Wan Mimi Diyana W. Zaki, Hafizah Hussain,

Aini Hussain, hiti halasiah Makri*

Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, 43600, Bangi, Selangor, Malaysia

Abstract

The current engineering program accreSitatian criteria requires an effective program outcome (CO) assessment praceSure along with well SacumenteS results, a complete Sescriptian of the evaluation process that involves extensive participation of the faculty members. To achieve all these, an effective methaS to estimate stuSents' performance as well as the effectiveness of the program outcomes (CO) is requireS. As such, the Sirect assessment methaS has been iSentifieS anS SeemeS suitable to Sirectly assesseS specific stuSents' knawleSge Suring examination or through skills observation. Specifically, this work reports the effort maSe by the Department of Electrical, Electronics anS System, (EEhE), Universiti Kebangsaan Malaysia (UKM) to implement the Sirect assessment methaS to measure the CO achievement of its stuSents. This is in line with the EEhE Dept.'s continuous quality improvement (CQI) practice to improve its CO assessment methaS before the next visit by the AccreSitatian Council in 2511. To So so, the assessment results from each semester of the acaSemic sessions 2558/2559 anS 2559/2515 are analyseS to praSuce the annual CO. Measurement of CO achievement is carrieS out an selecteS sampleS courses. In selecting the courses far the assessment, at least two courses from each acaSemic year that are affereS in the two Sifferent semesters are selecteS. hince the acaSemic program is a 4 year program, 8 courses neeS to be selecteS far the purpose. AccarSingly, this paper is written to Sescribe the assessment methaS useS to measure the CO via Sirect assessment of the course outcomes (Cos) as an improvement to previous practice where the COs were assesseS inSirectly using survey anS examination results. The Sirect assessment results shaweS that the average CO attainment far bath acaSemic session 2558/2559 anS 2559/2515 are greater than 3.5 anS fulfils the benchmark value set by the Separtment. Nevertheless, the Dept. is still putting up an effort to improve the assessment methaS in arSer to finS the best methaS to Sirectly assess bath COs anS COs simultaneously.

© 2011 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of Kongres Pengajaran & Pembelajaran UKM, 2010

Keywords: Direct Assessment; accreSitation; program outcomes; course outcomes; program outcome assessment; course outcomes assessment;

1. Introduction

NawaSays, accreSitatian of engineering program has shifteS its focus an well Sacumentatian of stuSents learning outcomes' achievement. The canSitian SictateS by the Engineering AccreSitatian Council (EAC) far any program to implement assessment baseS an learning anS program outcomes has spurreS the engineering eSucatian community to engage in formal assessment methaS anS substantiate its achievement of learning anS program outcomes in arSer to earn accreSitatian. Apparently, many engineering programs are still striving to iSentify the best outcomes assessment tool of their program as requireS by Criterion 3 Engineering Criteria stanSarS (EC2555) (hhaeiwitz anS

*CorresponSing author. Tel.: +653-89216359; fax: +653-89216146. E-mail address: siti1955@eng.ukm.my

1877-0428 © 2011 Published by Elsevier Ltd. doi:10.1016/j.sbspro.2011.05.008

Briedis, 2007). This is in accordance to the authors' experience as program assessors and chairmans of more than 30 accreditation visits. Most of them still depend on surveys and indirect assessment approaches which do not accurately reflect the true students' achievement as these methods are perceptions oriented.

Essentially, outcomes measurement based on direct assessment technique is more effective as it directly assess students' specific knowledge or skills. Direct assessment technique could be implemented in so many ways; for instances; in final examination, assignments, capstone projects and portfolios; which constitute the mapping of course outcomes (CO) to program outcomes (PO) (Shaeiwitz and Briedis, 2007). Plenty of reports on direct assessment method are available in the literature. (O.Biney and R.Bryant, 2010) assess students' specific knowledge and skills in coursework via direct mapping of CO to PO for every semester. At the end of each semester, the average PO is measured for each PO and these results are used to produce annual program outcomes assessment report.

(Memon and A Harb, 2009) report a method that uses two complex mathematical equations for CO and PO measurements. They develop a software application that accomplishes CO-PO measurement with iteration process is done automatically. This is so in order to reduce the computational burden of calculating each CO-PO for different semester. The direct assessment method as reported by (N. Turkmen et al., 2010) is also based on a complex process that includes determination of weightage and use of several mathematical equations for every assessment tool used. An Excel template is utilized to transform students' grade to each outcome's achievement in percentage.

(Gurocak, 2008) uses an approach where knowledge and skills that depict each CO are tested on specific problem during assignments or examinations. At the end of semester, students are given their final grade as well as CO achievement in 1-5 scale. Scores at course level are then used to assess program outcomes. Such method involves the development of CO-PO mapping via Criteria Performance Set. (Houshangi, 2003) mentions that 35 performance indexes are developed for 12 PO in his department. These performance indexes of PO provide a meaningful mapping of CO - PO and indicate clear actions that the course instructor need to carry out for outcomes assessment. The author comments that most of the available PO statements are too general and not specific. Henceforth, they are not measurable unless clear performance indexes are developed for each PO so that every CO-PO mapping is concisely defined.

2. Implementation

During the last engineering program accreditation visit in August 2008, the accreditation panel questioned the approach of using students' self assessment marks and their final grades for each course to demonstrate CO-PO attainment. They argued that the self assessment test is an indirect assessment method while the students' final grades do not reflect their specific knowledge and skills. Accordingly, to address the issue raised by these panels, the department has devised a new direct PO assessment method that only measures selected sampled courses. To ensure all POs are addressed, at least one course is selected for each academic year of the 4 years engineering program. There are 10 courses selected respectively for academic session 2008/2009 and 2009/2010. Figure 1 shows the list of the selected courses, mapping of these courses to PO and the avenues/assessment tools where each PO is measured.

For every selected course, related POs to be measured have been identified and linked to each CO. The course instructor then will measure students' knowledge or skills based on these specified CO-PO relations via assessment tools such as quiz, oral presentation, written report, project, lab experiments, tutorial, assignment and examinations. In general, these assessment tools are categorized into three categories; course work assessment, expert (lecturer/employer) assessment and peer review assessment. Then, the marks in percentage from each assessment tool of each course are mapped to related PO to obtain quantitative PO attainment for this course. Collectively, data from all selected courses are gathered and mapped to related PO (in percentage) by the Department's Learning, Teaching and Assessment committee. We use excel template as shown in Figure 2 to convert students marks (in percentage) into score based on Likert scale 1-5. Score 5 signifies a very good PO achievement. Table 1 details the transformation of raw marks (in percentage) to Likert scale 1-5. The average PO attainment for each class is compared to a benchmark of score 3.0; that is equivalent to grade B- (medium). Score in range of 3-5 means that the PO is achieved and vice versa. If the score is less than 3, the responsible course instructor is informed by the department to identify possible roots to this lacking and to find means to improve students' performance in terms of desired knowledge and skills in the next academic year for improvement.

Course Semester POl P02 РОЗ PCM P05 P06 P07 POS P09 РОЮ POll PG12

KF1243 Computer Programming Sem 1 Final Exam Part A Presentation in individual project X X X X X X X X X Group Project

KF1063 Introduction to Electrical Engineering Sem 2 Final ExamPart A X X Final Exam Part Bl, B2,B3 X X X X Laboratory (average all) X X X

KL2091 Electrical & Electronic Laboratory 1 Sem 1 Final Exam X Laborator Y {average all) Final exam Project X X X Laboratory (average all) X X X

KL208J Digital Design Sem 2 Final Exam + Quiz -s- Mid Exam X Laborator V + Project final exam f mid exam project X X X Laboratory X Sackgrou nd study + Literatur X

KL3193 Microprocessor &. Microcomputer Sem i Final Exam + Quiz + Mid Exam presentation report +project report Laborator y + Project final exam fmid exam X PRK + MOM X X Laboratory X X X

KF3283 Engineering Ethics and Technology Development Sem 1 X X X X X X Mid Exam X X И X X

KL3083 System Design Sem 2 Final Exam X Project-presentat ion Final exam Raw_ Prob. Sd Teamwork X X Project Achievement X X X

K1.3065 Industrial Training Sem 3 Assessment of Student's performs nc e by employer Assessment of Student's performanc e by employer X X Assessment of Student's performance by employer Assess me nt of Student's performan ce by Assessme nt of Student's performan ce by X X X X

KCKTKL4933 Project 1 Sem 1 Proposal report Presentation X Proposal report X X X SAR X X X X

KCKTKL4943 Projek 2 Sem 2 Thesis -Final draft Presentation X Thesis -Final draft X X X Managern ent X X X X

Figure 1 Courses with related PO for academic session 2008/2009

COURSE FILE:

KL3193 (Microprocessor & Microcomputer) SEM 12008/2009

NO OF STUDENTS:

Program Assess From

5.GG 4.GG 3.GG 2.GG l.GG G.GG

KL319312008/2009

PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10PO11PO12 ProgramBDutcome

Year®

Benchmark

РОИ I P012 I

Score Likert Score Likert Score Likert (10096)1 Scorel (10096)1 Scorel (loo%)l Scorel

- : 72.0 1

Program Outcomes POl P02 РОЗ P04 POS P06 P07 P08 P09 РОЮ POll P012

Summation 77 75 75 70 0 80 0 1 0 1 68 0 0 0

Total Average 84.06 4.81 86.63 4.69 86.12 ua 78.78 4.38 92.69 5.00 7Я.73

Standard Deviation 0.40 0.40 0.48 0.48 0.48 0.48 0.62 0.62 0.00 0.00 0.45 1 0.45

Figure 2 Example of Excel template to analyze PO for KL3193 course (2008/2009)

Hamimi Fadziati A. Wahab et al. /Procedia Social and Behavioral Sciences 18 (2011) 49-55 Table 1. Coursework mark transformed to students' performance score

Marks Percentage (%) Score Mastery Level

81-100 5 Very Good

61-80 4 Good

41-60 3 Medium

21-40 2 Poor

1-20 1 Very Poor

3. Results and Discussion

After direct assessment is carrieS out for all selecteS courses, next is to summarize average CO score in every semester. Thus, the average CO score for each course out of 10 selecteS courses are analyzeS using the following mathematical equation:

0l _ NCt

Zj=lcjnj

Oi = Total average i obtained for all courses in a semester O,j = Average total achievemen t i measured by course j Cj = Credit hour for course j nj = No of students attend course j NCi = No of courses measured outcome i

The score of every CO is measureS baseS on average weightage. The weightage is derived from the total creSit hours of every selecteS course. Table 2 anS Figure 3 show the average CO score baseS on acaSemic session 2008/2009 anS 2009/2010. The results show that the average CO score is more than 3.5; the benchmark value. Most of the measureS COs measureS achieve the target of more than 3.5 except for CO4 anS CO10 at hemester 2, 2009/2010. All CO measureS in hemester 3 2009/2010 has increaseS compareS to the previous year.

Table 2. PO Average score results based on semester

Program Outcome Academic Year 2008/2009 Academic Year 2009/2010

Sem 1 Sem 2 Sem3 Sem 1 Sem 2 Sem 3

PO1 4.29 3.75 3.97 3.92 3.68 4.24

PO2 4.28 4.69 4.16 4.30 4.72 4.40

PO3 4.78 4.70 4.55 4.75

PO4 4.11 3.61 3.90 3.39

PO5 4.50 4.00 4.00 3.60

PO6 4.83 4.06 4.28 4.04 3.60 4.52

PO7 4.00 4.42 3.66 4.51

PO8 4.42 3.91 4.42 4.34 4.89 4.69

PO9 4.42 4.43 4.22 4.54

PO10 4.00 3.37

PO11 5.00 4.08

PO12 4.65 4.41

PO Average Score for Semester of 2008/2009

PO1 PO3 PO5 PO7 PO9 PO11 Program Outcome

Sem 1 2008/2009 Sem 3 2008/2009

Sem 2 2008/2009 Benchmark

PO Average Score for Semester of 2009/2010

<r <r <r <P <r <r <r <r <i° jy 9<y <¡0*

Program Outcome

Sem 1 2009/2010 Sem 3 2009/2010

hem 2 2559/2515 Benchmark

Figure 3 CO Average scare results baseS an semester graph

Data from average score of CO in every semester as in Table 3 are useS to generate the annual program outcome achievement report. The previous mentioneS mathematical equations are useS to measure CO average score. Figure 4 Sepicts the annual average score of each CO compareS to the benchmark value.

Table 3. Average scare of annual CO

Program Outcome (PO) Academic Session 2008/2009 Academic Session 2009/2010

PO1 3.99 3.84

PO2 4.35 4.46

PO3 4.73 4.67

PO4 3.81 3.58

PO5 4.17 3.70

PO6 4.40 4.27

PO7 4.01 4.08

PO8 4.24 4.64

PO9 4.43 4.44

PO10 3.59 3.37

PO11 4.69 4.08

PO12 4.65 4.41

Average Program Outcome Annual Score for 2008/2009 and 2009/2010

5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00

PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 P010 PO11 PO12

Program Outcome

2008/2009

2009/2010

Benchmark

Figure 4 Average program outcome score 2008/2009 and 2009/2010

This study shows that all POs achieve more than the targeted score at scale 3.0 for both academic sessions. In overall, PO achievement of 2009/2010 shows a mere decline as compared to 2008/2009. It is also found that all POs exceed 3.50 for 2009/2010 except for P010. In both academic sessions 2008/2009 and 2009/2010, PO3 attains the highest score of 4.73 while PO10 scores the least, 3.59 and 3.37 respectively. This analysis also shows that 8 P0s; PO2, PO3, PO6, PO7, PO8, PO9, PO11 and PO12 score more than 4.0 while three POs; PO1, PO4 and PO10 score less than 4.0. This analysis is used to evaluate the effectiveness of the program in achieving program outcomes and objectives.

4. Conclusion and Proposals

In this paper, the main component of PO direct assessment tool based on students' knowledge and skills as used by the Department of Electrical, Electronics and System, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia is portrayed. In this method, average PO data obtained from every selected course are used to evaluate overall achievement of program outcomes. Results have shown that program outcomes are successfully achieved exceeding 3.0 benchmark value set by the department. This success advocates department continuous effort to provide first rate engineering curriculum delivery in parallel to the faculty requirement and university as a whole. This new strategy emerges in response to the concern raised by engineering accreditation panel during 2008 visit as well as department preparation for accreditation revisit in 2011 based on EC 2000 criteria.

Although this direct assessment tool seems viable to be used and maintained for a long term, it does not provide true CO achievement data for every student and every course. As per current, the department is still extensively research on the best method to measure CO - PO simultaneously. Besides, the PO statements provide general information, thus they are difficult to measure. Some performance indexes that explicitly defined each PO are required so that objective measurement can be implemented. The development of performance indexes is currently in progress with expectation that this approach is able to standardize CO statements among courses in the department to refine the outcomes evaluation and measurement processes. The other improvement is to strengthen the process of closing the loop at course and department level. At course level, a new mechanism of course report shall be considered that reports CO-PO achievement and any proposal recommended by the course instructor to improve on lacking CO/PO/knowledge/skills for reference of the next course instructor. At department level, the department needs to review annual PO achievement for each PO and identify unfulfilled POs. Courses relate to these

POs also need to be reviewed, students' low performance needs to be investigated and propose changes that could be done at program level to improve their performance in future.

References

Gurocak, Hakan. (2008). Direct Measures For Course Outcomes Assessment For ABET Accreditation. Paper read at American Society Engineering Education, 22 - 25 June 2008, at Pittsburgh, PA.

Houshangi, Nasser. (2003). Curiculum Assessment and Enhancement at Purdue University Calumet Based on ABET 2000. In American Society for Engineering Education. Illinois, Indiana.

N.Turkmen, A.R.A.Khaled, O.M.al-Raghbi, G.Zaki, M.Siddique, A.Gari, and A.Bokhary. A Procedure to Assess Engineering Courses based on Program Outcomes. Retrieved October 15, 2010 from http://ipac.kacst.edu.sa/eDoc/2007/168457_1.pdf.

O.Biney, Paul, and Milton R.Bryant. A Novel Strategy for the Direct Assessment and Improvement ofEngineering Programs Developed and Implemented by Prairie View A&M University. Retrieved October 15, 2010 from http://engg.kau.edu.sa/AAU/AssExample.pdf.

QA Memon, and A Harb. (2009). Developing Electrical Engineering Education Program Assessment Process at UAE University. Australian Journal of Engineering Education Vol.15 No 3:10.

Shaeiwitz, Joseph, and Daina Briedis. (2007). Direct Assessment Measures. Paper read at American Society of Engineering Education, 24 - 27 June, Honolulu, Hawaii.