Assessment of the Quality of Educational Climate During Undergraduate Clinical Teaching Years in the College of Medicine, Taibah University Academic research paper on "Educational sciences"

EDUCATIONAL STUDY

Assessment of the Quality of Educational Climate During Undergraduate Clinical Teaching Years in the College of Medicine, Taibah University

Naeem M. Makhdoom MD, FACHARZT

Department of ENT, College of Medicine Taibah University Al Madinah Al Munawarah, Saudi Arabia

Abstract Background

The undergraduate curricula of medical schools in Taibah University Saudi Arabia, is traditional, like most of the medical schools in the Middle East region. A measure of the educational environment in the college of medicine as perceived by students would assist educators and college administration personnel in gauging the quality of the learning occurring within this important venue. Objectives

1)To assess the quality of educational climate during undergraduate clinical teaching years in College of Medicine, Taibah University, AL-Madinah AL-Munawarah, Kingdom of Saudi Arabia. 2)To detect problem areas that should be remediated. 3)To compare the gender difference in the perceptions of the educational environment in the clinical teaching stage. Methods

During the academic year 2007/2008, the DREEM questionnaire was distributed and collected by the undergraduate student leader of the same year to all 280 females and males' clinical stage medical students. 4th year, 5th year and 6th year clinical stage students are located at different hospital sites for their clinical teaching. Each hospital site has an undergraduate administrator who distributed and collected the questionnaire. Comparisons between, students' responses according to their studying years in the college and their gender were taken in consideration. Results

One hundred and ninety-six female and male students completed the questionnaire from the total students sample (280) representing a response rate of 70%. There were 109 male out of (150) representing72.6 % and 95 female out of (130) representing 73%: the female students composite 48.5 % of the total responding students, while the male students represented 51.5%. There were no individual areas of excellence (that is no item scored > 3.5). Although the environment was perceived as more positive than negative in both females and males the mean total score was highly statistical significant difference where the score was 113/200 in the females when compared to the males score 107/200. In four out of the five subscales females scored higher than males indicating that the females students appear happier in the clinical stage and the vice versa for the males students.

Conclusion

The DREEM provides useful diagnostic information about medical schools, whether it is in developing or western developed countries. The DREEM gives a clear indication of the priorities for reform of the curriculum. These data can also serve as a baseline for a longitudinal quality assessment of student's perceptions of the changes planned for the college of medicine Taibah University, Saudi Arabia.

Key words: Educational climate, Educational environment, Medical culture, Medical education

Journal of Taibah University Medical Sciences 2009; 4(1): 42-52

Correspondence to:

Dr. Naeem M. Makhdoom

Department of ENT, Head of Department of Medical Education

College of Medicine, Taibah University

M Al Madinah Al Munawarah

Kingdom of Saudi Arabia

8 +966 4 8460008

Ä, +966 4 8461407

^ garee22@yahoo.com

Introduction

There is an increasing interest and concern regarding the role of learning environment in undergraduate medical education in the recent years. Educational environment is one of the most important factors determining the success of an effective curriculum1. The quality of educational environment has been identified to be crucial for effective learning. Curriculum's most significant manifestation and conceptualization is the environment, educational and organizational which embraces everything that is happening in the medical school.

In addition, the learning environment is an important determinant of behavior; elements of the education environment are related to academic achievement, course satisfaction and aspirations. Perceptions of the climate may be affected by the increasing diversity of the student population. Critically, we can both measure and change the educational climate. It has been used diagnostically to identify areas of strength and weakness in a current educational environment. It has also been used to compare different medical education institutions, students at different stages of the course, and gender has been used to measure the existing educational environment as a precursor to curriculum change; to identify priority areas for change and to act as a baseline for comparison after curriculum change; to compare old and new curricula, and to investigate the impact of a new curriculum on perceptions of the educational environment. The General Medical Council of the UK

recommendations for 'Tomorrow's Doctors' have stimulated educational innovations and new curricula in British medical schools2. The UK Standing Committee on Postgraduate Education (1991) highlighted the importance of educational environment in their statement that 'A working environment that is conducive to learning is critically important to successful training. In adult learning theories, teaching is as much about setting the context or climate for learning as it is about imparting knowledge or sharing expertise3. The learning environment has been defined as everything that is happening in the classroom or department or faculty or university4, 5. Measurement of the educational environment comprehensively assesses what is happening, or how things are in the medical school4. It is a way of assessing the nature of the educational practice of the medical school. It also provides a holistic, comprehensive, systematic, and detailed picture of the overall state of affairs in the education process6. The World Federation for Medical Education (WFME) singled out the "learning environment" as one of the "targets" for what it terms "the conduction

of the evaluation of medical education

programme. 7.

To measure such an environment Roff et al8 developed the Dundee Ready Education Environment Measure (DREEM). The DREEM has been used in many countries9-11 producing validated global reading and diagnostic analyses of educational environments.

Aims of the study

1. To assess the quality of educational climate during undergraduate clinical teaching years in College of Medicine, Taibah University, AL-Madinah AL-Munawarah, Kingdom of Saudi Arabia.

2. To detect problem areas that should be remediated.

3. To compare the gender difference in the perceptions of the educational environment in the clinical teaching stage.

Materials and Methods

During the academic year 2007/2008, the DREEM questionnaire was distributed and collected by the undergraduate student leader of the same year to all 280 females and males' clinical stage medical students. 4th year, 5th year and 6th year clinical stage students are located at different hospital sites for their clinical teaching. Each hospital site has an undergraduate administrator who distributed and collected the questionnaire.

Genn (2001) in AMEE Guide No.23 clearly established the importance of educational climate. This is exemplified by the following quote, "Considerations of climate in the medical school, along the lines of continuous quality improvement and innovation, are likely to further the medical school as a learning organization with the attendant benefits".

To measure such an environment, the Dundee Ready Education Environment Measure (DREEM) was developed8. It can produce global readings and diagnostic analyses of undergraduate educational environments in medical schools and other health professions institutes. It is non-culturally specific and allows quality assurance comparisons between courses as well as within components of a course. The DREEM contains 50 statements relating to a range of topics directly relevant to the education climate. The inventory can be administered by postal survey or face to face

in the teaching session's room. Students are asked to read each statement carefully and to respond using a 5 point Likert-type scale ranging from strongly agree to strongly disagree. It is important that each student applies the items to their own current learning situation and response to all 50.

Scoring the DREEM

Items scored are categorized as: 4 for Strongly Agree (SA), 3 for Agree (A), 2 for Uncertain (U), 1 for Disagree (D) and 0 for Strongly Disagree (SD) The 50-item DREEM has a maximum score of 200 indicating the ideal educational environment as perceived by the registrar. A score of 0 is the minimum and would be a very worrying result for any medical educator. As well as the total DREEM score divided into five subscales: students' perceptions of learning, students' perceptions of course organisers, students' academic self perceptions, students' perceptions of atmosphere, students' social self perception.

The DREEM can also be used to pinpoint more specific strengths and weaknesses within the educational climate. To do this one needs to look at the responses to individual items. Items that have a mean score of 3.5 or over are real positive points. Any item with a mean of 2 or less should be examined more closely as it indicates problem areas. Items with a mean between 2 and 3 are aspects of the climate that could be enhanced.

Results

Two hundreds and four female and male students completed the questionnaire from the total students sample (280) representing a response rate of 72.8%. There were 109 male out of (150) representing 72.6 % and 95 female out of (130) representing 73%: the female students comprised 46.6 % of the total responding students, while the male students represented 53.4% (Table1).

Table 1: Students' gender distribution among different studying grades

Students grades Fen No nale 0/ % Mai No e 0/ %

4th year 41 43.2 43 39.4

5th year 22 23.1 45 41.3

6th year 32 66.7 21 19.3

Total 95 48.5 109 51.5

Table 2: Students' mean age and standard deviations (±SD) distribution according to their grade and gender

Students Female age Mali age

P value

Mean ±SD Mean ±SD

4th year 21.1 0.54 21.63 0.77

5th year 22.59 0.85 22.57 0.8

6th year 23.41 0.66 23.37 0.59

Total 22.33 1.22 22.52 1.02

Figure 1: Students' mean DREEM components scores of the students according to their gender.

There are no significant differences between the female and male students' mean age and standard deviations values according to their grade, where the mean age was 22.42±1.12 years (Table 2).

The DREEM score

The mean total score of the students was 109.92 ±18.76 (guide to interpreting the

overall score; 101-150 More Positive than Negative) and components of DREEM were; academic self perception 19.02 ±4.18, perception of atmosphere 25.24 ±5.36, social self perception 16.97 ±2.22, perception of learning 25.21 ±5.26 and finally perception of course organizers 23.50 ±5.47 (Figure 1 ). Statistical analysis using Pearson's test shows that there are significant positive

correlations between the gender of the students with Total DREEM Score interpretation, perception of learning interpretation, perception of course organizers interpretation, and perception of atmosphere interpretation where the female students recorded high scores in all of the above items (Table 3 and Figures 2, 3 and 4).

Although the environment was perceived as more positive than negative in both females

and males the mean total score was highly statistical significant difference where the score was 113/200 in the females when compared to the males score 107/200. In four out of the five subscales females scored higher than males indicating that the females students appear happier in the clinical stage and the vice versa for the males students (Table 4 and graph 5).

Table 3: Pearson Correlations between total DREEM score and its components, also with the students gender.

Characteristic Total score Perception of learning score Perception of course organisers score Academic self perception score Perception of atmosphere score Social self perception score

Gender Pearson Correlation .161(*) .209(**) .194(**) .158(*) -.103 .057

Sig. (2-tailed) .021 .003 .005 .024 .142 .419

** Correlation is significant at the 0.01 level (2-tailed). * Correlation is significant at the 0.05 level (2-tailed).

perception of learning score interpretation

teaching is a more teaching

viewed positive highly

very poor negatively perception thought of

O O tj

;;7]~| M.H'- Stiiilents Uf] I ^ihttlt Students

Figure 2: Students' mean of perception of learning interpretation according to their gender.

interpreting the overall score

MORE POSITIVE PLENTY OF THAN VERY POOR PROBLEMS NEGATIVE

EXCELLENT

interpreting the overall score

MORE POSITIVE PLENTY OF THAN VERY POOR PROBLEMS NEGATIVE EXCELLENT

Table 4: The total DREEM score and its components among females and males students

Male (mean; SD) Female (mean; SD)

Components of assessment P value

DREEM overall score 107±17.285 113±17.285 0.021

Perception of learning score 24.07±5.889 26.51±5.50 0.003

Perception of course organisers score 22.50±5.757 24.63±4.91 0.005

Academic self perception score 18.80±4.436 19.27±3.88 0.419

Perception of atmosphere score 24.45±5.957 26.14±4.43 0.024

Social self perception score 17.28±3.19 16.61±3.233 0.142

Discussion

Continuous quality improvement and innovation are essential in a medical school. The DREEM is a reliable and validated instrument which identifies specific problem areas within an institution, for different components of the same student body. More importance should be given to the students' perception of the learning environment. Their perception can be used to initiate change and improvement. Medical education is very expensive and academic failure is wasteful both to society and to the individual. Consequently we need to ensure that the environment is as conducive as possible to learning, thus reducing the risk of academic underachievement. Use of the DREEM as a monitoring tool would permit timely interventions to remediate problematic educational environments. We have used the Dundee Ready Education Environment Measure (DREEM) in 'diagnosing' the educational environment of the College of Medicine, Taibah University and making comparative analysis within the students scores according to their gender. There is no accepted agreement on what is an acceptable DREEM inventory score from published literature. Nevertheless, our DREEM overall score of the environment was perceived as more positive than negative in both females and males the mean total score was highly statistical significant difference where the score was 113/200 in the females when compared to the males score 107/200. In four out of the

five subscales females scored higher than males indicating that the females students appear happier in the clinical stage. The overall mean DREEM score for female and male was 110/200, or expressed as a percentage 55%, whereas, 56.5% for the females and 53.5% for males. The educational learning environment did not vary between centers. The two highest scoring contributory domains, academic self-perception (59%) and social self-perceptions (60%), were statistically significantly different from the other three lowest DREEM domains perception of learning score (53%), perception of course organisers score (53%), and perception of atmosphere score (50%). The total mean score of 109 compares less favorably with total mean scores achieved in A large UK medical school (125), Scottish medical school (136), Irish Medical School (130), and some medical schools in the Middle East for example, Nepalese Medical School (130), Arab Gulf University (127), and Arab United Emirates University (125), Nigerian Undergraduate Medical School (118), , but compares more favorably with other medical schools in the Middle East, King Faisal Medical School Saudi Arabia, Umm Al-Qura University (107), King Abdul Aziz University Saudi Arabia (102), Al-Yemen University (99). Varma et al12 study DREEM score of 139/200 was higher than other reports. An Obstetrics and Gynecology undergraduate

teaching module allocates 40-50 final year medical students to eight teaching hospital sites in the West Midlands region, the overall mean DREEM score was 139/200, a study of final year medical students in Trinidad reported an overall mean DREEM of 109.9/200 and the total mean scores for the subgroups -male students, male interns, female students and female interns- were 103.39, 111.82, 111.33 and 113.15, respectively. The lowest scores were assigned to students' social self-perceptions and students' perceptions of the atmosphere. All of the participants except the male interns recorded the highest scores for the subscale academic self-perceptions14. A larger scale study, involving students from both final and earlier undergraduate training years, showed a DREEM score of 118/200 in a Nigerian medical school and 130/200 in a Nepalese medical school 6. The author return this as an indicator of better hospital teaching environment, the positive value of using a comprehensive course handbook, and the encouragement of formative self-assessment as guided by the course handbook and web-based package. A recent Cross-Sectional study using the already validated Dundee Ready Educational Environment (DREEM) questionnaire was used to assess the Undergraduate Medical Educational Environment in an Irish Medical School during the first semester 2004/2005. 389 of 476 students (82%) completed the questionnaire. The mean total score was 130 out of a maximum of 200 (65%) indicating relative satisfaction with the environment but with room for improvement. There were no individual areas of excellence identified. The following two areas were identified as being problematic (a) lack of a support system for stressed students (b) over emphasis on memorization of facts. These areas were perceived by clinical students to be greater when compared to pre-clinical students. Female students appeared happier in their environment but male students were more confident about passing exams. Non-Irish students had a lower overall score when compared to Irish students. The DREEM is a useful tool to assess the overall

teaching environment and highlight areas of weakness. Use of DREEM as a monitoring tool would be useful to re-evaluate the environment following appropriate intervention15.

An Indian study was motivated to identify those students who are vulnerable to academic failure at this level of training and to identify the features of the educational environment that they perceive differently from students who are succeeding academically in order to design intervention strategies. Gender differences in perceptions of the educational environment might well emerge in particular academic or cultural contexts, with particular curricula. Concerned and focused on comparisons between academic achievers and under-achievers and male and female students of Kasturba Medical College, India. Overall sample of the study rated educational environment in this institution was as average. The overall mean DREEM score was significantly higher for academic achievers. Compared to under-achievers, academic achievers scored significantly higher on perceptions regarding teachers, academic atmosphere and social self-perceptions. In addition to this, the overall rating (total DREEM score) of female students was significantly less compared to males in the academically vulnerable group16.

A Sri Lanka study to ascertain the overall DREEM score for the newly established Faculty of Medical Sciences of University of Sri Jayewardenepura, Sri Lanka and to compare the DREEM score of the students in the pre-, para- and clinical phases of the traditional curriculum practiced in this school. A total of 339 students belonging to the pre- (n = 147), para- (n = 116) and clinical (n = 76) phases of the medical course participated in the study. The DREEM questionnaire was administered face-to-face after one of their routine lectures to each group of students. The age of the students ranged from 20 to 28 years and the gender distribution was almost equal. The overall DREEM score was 108 (54%) for the pooled data for all three phases. There was no significant difference on the overall DREEM

score obtained by each phase of students. The overall DREEM scores of pooled data and also for each domain indicated that the position of the Faculty of Medical Sciences of University of Sri Jayewardenepura qualifies to be placed just on the third grade (one below the best) within the overall DREEM scale. However, on analysis of the responses for each domain, Students' Perception of Teachers (SPT), Students' Academic Self-Perceptions (SAP) and Students' Social Self-Perceptions (SSP) showed significant difference between the pre-, para- and clinical phases. Similarly, 22 out of the 50 items showed significant differences between the pre- and clinical phases. Overall, the DREEM demonstrated compatibility of its scores along with the gradual development that took place at this medical school over the last 10 years. Thus, the DREEM could be utilized in a variety of situations17.

Till18 used the DREEM to compare the actual educational environment experienced by students with their ideal or preferred environment. Information on the discrepancies between the actual and ideal environments can be used to improve the quality of the environment. In the reported study the expectations of new medical students about the educational environment -they were about to enter- were compared with the actual climate experienced. The first cohort of students on the 5-year MBBS program at the new Medical School at the University of East Anglia started in September 2002. Unusually for the UK, each class comprises school leavers (~50%), graduates (bachelor, master or PhD) (~30%) and students from access-to-medicine courses (~20%). The latter groups are older students whose academic qualifications were unsuitable for entry to medical school but who have taken a 1 year pre-med course. All students are asked to complete a comprehensive course evaluation at the end of each year, which includes the DREEM. Previous course evaluations at the end of academic years 2002-03 and 2003-04 had been generally positive, with many of the individual DREEM items scoring > 3.0, and few areas of weakness. Although this is very

encouraging, we wanted to ascertain if there were any areas where the actual experience was less positive than had been expected. It was hypothesized that student expectations of their educational environment would be high and that in some areas the actual experience would fall short of expectations. Whittle et al19 using qualitative analysis, in association with the DREEM questionnaire, to evaluate the educational environment across all five years of a large undergraduate medical school, and identify areas for change to enhance student experience. Their method was administered the DREEM questionnaire to 968 undergraduate students, together with an open question asking for suggested changes to current medical school practices. Items of concern highlighted by this study were further defined through qualitative analysis, using focus groups, email questionnaires and introduction of Stressful Incident reporting. They reported that the responses to the open question, two items with low scores on the DREEM questionnaire were identified as requiring remediation. Focus groups and email questionnaires were used to define the underlying causes of poor scores, which varied by student year group. Stress resulting from experiences on clinical placement was highlighted by some students, but on closer investigation found to be rare. Remedial steps to improve student support are described. Although the environment was perceived as more positive than negative in both groups the mean total score was significantly greater in female students 113 compared to the male students 107. In three out of the five subscales DREEM items the students in the final clinical year (6th year) scored higher than their colleagues in the 4th and 5th clinical years indicating that the older students appear happier. The change to a clinical hospital environment may also contribute to a happier student. No item had a score greater than 3.5 in either group. On individual questions the older clinical students report higher satisfaction with the learning environment than the younger preclinical students.

It is not possible to say definitively whether

this effect is due to educational background, as there is a confounding effect of age. All of the graduate and access-to-medicine students were aged 19 years or more. A gender difference has been noted in other studies. In Middle Eastern countries the perception of the environment by males is more positive than by females. The opposite has been seen in Dundee medical school where female students' perception is more positive20. The gender differences between the results identified in UK medical schools compared to Middle Eastern medical schools can probably be accounted for by the different types of curricula 16. In the Middle East a more traditional didactic course is still taught and male and female students are separated, the latter often being taught via video-link. In the UK the curricula are generally systems based with integration, and divided into core subjects and special study modules. In our study although the environment was perceived as more positive than negative in both females and males. Also, shows that, in four of the five DREEM subscales, the students in the females scored higher than their colleagues' males in the clinical years indicating that the females' students appear happier than the males. The scores for the individual DREEM items give a clear indication of where the priorities for reform should take place at the study site. One of the main areas for concern, and highlighted by all clinical years, was the support system for students who get stressed. This could be addressed by a number of avenues including:

a) a more structured and available personal tutoring system

b) peer tutoring

c) approachable chaplaincy service

d) accessibility of school office staff and

e) senior to junior student mentoring Students are also stressed by memorization of too many facts. The curriculum is focusing on the retention of too many facts rather than the acquisition of practical skills. Another area of concern was that of timetabling. This is likely to be due to under-resourcing of the school office. Alternatively it may be due to a lack of coordination

between staff within the school office and administrative staff at distant clinical teaching sites, or lack of direction from more senior academic staff within the medical school. Addressing this issue may do a lot to alleviate the anxiety expressed by students.

References

1. Bassaw B, Roff S, McAleer S, Roopnarinesing S, Lisle JD, Teelucksing S, Gopaul S. Students' perspectives on the educational environment, Faculty of Medical Sciences, Trinidad. Med Teach 2003; 25: 522-526

2. General Medical Council. Tomorrow's Doctors: Recommendations on Undergraduate Medical Education. London: GMC(1993)

3. Hutchinson L . ABC of learning and teaching, educational

environment. BMJ 2003; 326: 810-812

4. Genn, JM. AMEE Medical Education Guide No. 23 (Part 2): Curriculum, environment, climate, quality and change in medical education - a unifying perspective. Medical Teacher 2001; 23(4): 445-454

5. Roff S, McAleer. What is educational climate? Medical Teacher 2001; l. 23 (4): 333-334

6. Genn, JM, Harden RM. What is Medical Education Here Really Like? Suggestions for action research studies of climates of medical environments. Medical Teacher 1986; 8(2): 111-121

7. Genn JM. AMEE Medical Education Guide No. 23 (Part 1): Curriculum, environment, climate, quality and change in medical education - a unifying perspective. Medical Teacher 2001; 23 (4): 445-454

8. Roff S, McAleer S, Harden RM, Al-Qahtani M, Uhmed AU, Deza H, Groenen G, Primparyon P. Development and validation of the Dundee Ready Evaluation Environment Measure (DREEM). Medical Teacher 1997; 19: 295-299

9. Pimparyon P, Roff S, McAleer S, Poonchai, B and Pemba S. Educational environment: Student approaches to learning and academic achievement in a Thai nursing school. Medical Teacher 2000; 22: 359-364

10. Bassaw B, Roff S, McAleer S, Roopnarinesingh S, De Lisle J, Teelucksingh S, Gopaul S. Students' perspectives on the educational environment, Faculty of Medical Sciences, Trinidad. Medical Teacher 2003; 25: 522-526

11. Roff S, McAleer S, Ifere OS, Bhattacharya S. A global diagnostic tool for measuring educational environment: comparing Nigeria and Nepal. Med Teach 2001; 23: 378-382

12. Jamieson S. Likert scales: how to (ab)use them. Med Educ 2004; 38: 1217-1218

13. Varma R, Tiyagi, E, Gupta JK. Determining the quality of educational climate across multiple undergraduate teaching sites using the DREEM inventory. BMC Med Educ 2005; 21(5): 1-8

14. Roff S, and McAleer S. What is educational climate? Med Teach 2001;

23: 333-334

15. Bassaw B, Roff S, McAleer S, Roopnarinesingh S, De Lisle S, Teelucksingh S, Gopaul S. Students'

perspectives on the educational environment, Faculty of Medical Sciences, Trinidad. Med Teach 2003; 25(5): 522-526

16. Avalos G, Freeman C, Dunne F. Determining the quality of the medical educational environment at an Irish medical school using the DREEM inventory. Ir Med J 2007; 100(7): 522-525

17. Mayya S, Roff S. Students' perceptions of educational environment: a comparison of academic achievers and under-achievers at Kasturba Medical College, India. Education Health 2004; 17(3): 280-291

18. Till H. Identifying the perceived weaknesses of a new curriculum by means of the Dundee Ready Education Environment Measure (DREEM) Inventory. Medical Teacher 2004; 26: 39-45

19. Whittle SR, Whelan B, Murdoch-Eaton DG. DREEM and beyond; studies of the educational environment as a means for its enhancement. Educ Health (Abingdon) 2007; 20: 1-7

20. AL-Hazimi A, AL-Hayani A, Roff, S. Perceptions of the educational environment of the medical school in King Abdul Aziz University, Saudi Arabia. Med Teach 2004; 26(6): 570-573