Scholarly article on topic 'The hybrid returns-to-scale model and its extension by production trade-offs: an application to the efficiency assessment of public universities in Malaysia'

The hybrid returns-to-scale model and its extension by production trade-offs: an application to the efficiency assessment of public universities in Malaysia Academic research paper on "Educational sciences"

Share paper
Academic journal
Annals of Operations Research
OECD Field of science

Academic research paper on topic "The hybrid returns-to-scale model and its extension by production trade-offs: an application to the efficiency assessment of public universities in Malaysia"

Ann Oper Res

DOI 10.1007/s10479-015-1854-0

The hybrid returns-to-scale model and its extension by production trade-offs: an application to the efficiency assessment of public universities in Malaysia

Victor V. Podinovski1 • Wan Rohaida Wan Husain2

© Springer Science+Business Media New York 2015

Abstract Most applications of data envelopment analysis (DEA) employ standard constant or variable returns-to-scale models. In this paper we suggest that these models may sometimes underutilize our knowledge of the underlying production process. For example, in the context of higher education considered in the reported application, individual universities often maintain a certain student-to-staff ratio which points that there should be an approximately proportional relationship between students and staff, at least in the medium to long run. A different example is an observation that the teaching of postgraduate students generally requires more resources than the teaching of the same number of undergraduate students. In order to incorporate such information in a DEA model, we propose a novel approach that combines the recently developed hybrid returns-to-scale DEA model with the use of production trade-offs. The suggested approach leads to a better-informed model of production technology than the conventional DEA models. We illustrate this methodology by an application to Malaysian public universities. This approach results in a tangibly better efficiency discrimination than would be possible with the standard DEA models.

Keywords Data envelopment analysis • Hybrid returns to scale • Production trade-offs • Weight restrictions • Higher education • Malaysia

B Victor V. Podinovski

Wan Rohaida Wan Husain

1 Warwick Business School,University of Warwick, Coventry CV4 7AL, UK

Business Administration Department, KENMS, International Islamic University Malaysia, Jalan Gombak, 53100 Kuala Lumpur, Malaysia

Published online: 31 March 2015

1 Springer

1 Introduction

Data envelopment analysis (DEA) has been widely used for the assessment of efficiency and performance of the education sector at different levels (Liu et al. 2013). Some recent applications of DEA to the assessment of school education are discussed by Portela et al. (2012), Blackburn et al. (2014), Brennan et al. (2014) and Johnson and Ruggiero (2014). In the context of higher education, DEA has recently been used for the efficiency assessment of universities and their departments (Johnes 2006, 2008; Johnes and Yu 2008; Worthington and Lee 2008; Thanassoulis et al. 2011).

Most of the above applications employ the conventional variable and constant returns-to-scale (VRS and CRS) DEA models (Cooper et al. 2007; Thanassoulis et al. 2008). Some of the reported applications use extended DEA models obtained by the incorporation of additional weight restrictions representing value judgements with respect to the selected inputs and outputs (Athanassopoulos and Shale 1997; Sarrico et al. 1997; Sarrico and Dyson 2004; Khalili et al. 2010).1

Recently, Podinovski et al. (2014) argued that, in the context of school education, a viable alternative to the standard VRS and CRS models is a hybrid approach based on a combination of variable and constant returns-to-scale assumption in one model. The idea of this approach is based on the observation that the number of pupils and teaching staff at schools may be assumed proportional, i.e., an increase of pupils by, for example, 10% is technologically possible provided the teaching staff is also increased by 10 %. At the same time other factors, e.g., those representing the academic quality of pupils on entry and exit, may not be a part of this proportion. The specification of such non-proportional inputs and outputs makes the use of CRS DEA model unsubstantiated. At the same time, it is clear that the VRS model is too restrictive and is based on the production technology that does not take into account the proportional relationship between staff and pupils. Podinovski et al. (2014) resolve this modelling problem by using the hybrid approach that combines VRS and CRS assumptions in one technology. This is based on the notion of selective proportionality leading to the hybrid returns-to-scale (HRS) model of Podinovski (2004a).2

In this paper we consider the use of the HRS model in an application to the higher education sector, namely, to the efficiency assessment of public universities in Malaysia. The input and output specification of our model relies on the data made available to us by the Ministry of Education Malaysia. Similar to the application to schools, we argue that the number of students of three different types (undergraduate, master and doctoral) may be assumed proportional to academic staff, which is consistent with the established student-to-staff ratios at different universities. However, the research funding available to universities treated as an input and the number of research publications treated as an output may be left out of this proportion. This naturally leads to the use of the HRS model in the efficiency assessment of universities. We further show that the HRS model results in tangibly better discrimination on efficiency than the standard VRS model.

As a further development, we suggest an expansion to the HRS model by the incorporation of production trade-offs, first introduced for the VRS and CRS models in Podinovski (2004b). An example of such trade-offs is the judgement that one undergraduate student

1 The use of weight restrictions as a means to incorporate value judgements in DEA models and related theoretical issues are discussed in Allen et al. (1997), Pedraja-Chaparro et al. (1997), Podinovski (1999, 2001) and Thanassoulis et al. (2004).

2 The treatment of selective proportionality in the HRS model is somewhat similar to the treatment of good and bad outputs in the weakly disposable VRS technology of Kuosmanen (2005). The latter is further explored and extended in Kuosmanen and Podinovski (2009) and Podinovski and Kuosmanen (2011).

does not require more teaching resources than one master student. Stating this as a trade-off, we conclude that it is technologically possible to teach more undergraduate students if the master students are reduced by the same number. This change does not require extra teaching resources and may be implemented while keeping research funding and publications constant. In our paper we show how production trade-offs can be incorporated in the HRS model, leading to even better discrimination on efficiency.3

Several contributions of our paper to the methodology of DEA should be highlighted. First, we point out that in the context of efficiency assessment in the higher education sector, the HRS model is a viable alternative to the standard VRS and CRS models. The underlying HRS technology may represent the university production process better than the existing conventional approaches because it allows for the specification of proportional relationship between a subset of inputs (academic staff) and a subset of outputs (students). Second, we demonstrate that the HRS model can be further enhanced by the incorporation of production trade-offs—simple judgements that specify technologically feasible substitutions between different inputs and outputs. The resulting HRS model with production trade-offs represents a better-informed model compared to conventional models. Third, we apply the suggested model to the efficiency assessment of public universities in Malaysia. Although there are only 20 universities in the sample, we show that the additional assumptions we make about the production technology (selective proportionality between staff and students, and technological feasibility of certain production trade-offs) help us overcome the limitations of the actual sample size. The resulting model provides good discrimination on efficiency and highlights possible areas of weakness of the universities important for policy makers.

2 Preliminaries

2.1 Application background

The higher education sector of Malaysia is represented by public and private providers, including colleges, polytechnics and universities. In this paper we illustrate the proposed DEA methodology by an application to all 20 public universities that operate in the country.

The existing academic literature on the performance of universities in Malaysia primarily focuses on the quality of teaching provision and student satisfaction (see, e.g., Wilkinson and Yussof 2005; Yu et al. 2009). Yet, the Ministry of Education Malaysia (MOE) places great emphasis on the international research reputation of public universities. To highlight the research excellence of international standards, in 2006 MOE designated the best four public universities (UKM, UM, UPM and USM) as research universities. In 2010, UTM became the fifth university awarded this highly-praised status.4

Further four public universities (UIAM, UMS, UniMAS and UiTM) are designated as comprehensive universities. These universities offer a wide range of subject areas and are primarily expected to demonstrate high quality of the teaching provision. The remaining 11 public universities are designated as focused universities. These are expected to offer high quality of teaching and also have specialist research-oriented subject areas.

3 Podinovski (2004b) shows that production trade-offs are mathematically equivalent to weight restrictions in the dual multiplier models. The assessment of trade-offs in DEA applications to different sectors, other than higher education, using conventional DEA models was illustrated, for example, by Amado and Dyson (2009), Amado and Santos (2009), Santos and Amado (2014).

4 Throughout our paper we use the abbreviated names of universities as shown on the official webpage (MOE 2014).

2.2 Inputs and outputs

In the reported application we use official data collected by MOE for the year 2012 and made available for our analysis.5 The specification of outputs and inputs in our model and the level of detail captured by them are primarily determined by these data and are in line with many applications of DEA to the higher education sector.

The four outputs measures are the teaching and research activities of universities. The first three are the three different categories of student: undergraduate, master and doctoral (Ph.D.) students. The fourth output is research publications in reputable (indexed) journals in the current year.

The two inputs are academic staff and research funding. The former is the full-time academic staff of all levels, from lecturer to professor.6 The latter represents the total amount of research funding obtained from different sources expressed in million of Malaysian Ringgit (MYR). It is worth noting that the DEA literature is not unanimous on the question whether research funding should be regarded as an input or output (Beasley 1990, 1995). For example, the former approach is adopted in Johnes and Johnes (1995), Athanassopoulos and Shale (1997), Ng and Li (2000) and Johnes and Yu (2008). The latter is implemented in Flegg et al. (2004) and Thanassoulis et al. (2011).

One argument for treating research funding as an output is that research grants are often competitive and may be used as a measure of research quality. In our application we take a view that the quality of academic research is ultimately and primarily measured by research publications in reputable outlets, which is considered an output. Bidding for research funding is somewhat similar to bidding for extra academic staff in order to strengthen particular research areas, and the latter is universally regarded as an input. It is therefore our view in the reported application that the efficiency of a university should be judged by its ability to use the two types of resource to produce teaching and research outputs, and not so much by its ability to secure the resources.

The above argument is also consistent with the treatment of inputs and outputs in DEA models. Note that the only difference between inputs and outputs in the VRS and CRS DEA models is in the way they satisfy the assumption of free disposability (see, e.g., Axiom 2 below). To use this assumption in order to decide whether research funding is input or output, consider the following scenario. Let the research funding of a university be increased while the academic staff is kept unchanged. Then the university should at least be able to produce the same teaching and research outputs as without the extra funding. This assumption is consistent with the treatment of research funding as an input. However, it is generally unreasonable to assume that the outputs would be unaffected if the funding is reduced, pointing that the latter should not be viewed as an output.

It is worth noting that the treatment of research funding as an input is not critical to the suggested methodology, and its incorporation as an output is also mathematically possible. In the latter case research funding is treated in the same way as research publications. We comment on this this in Remark 1 in the next section.

Table 1 shows descriptive statistics on the two inputs and four outputs used in our model.

5 We consider these data as representative of the two academic years 2011-2012 and 2012-2013.

6 Similar to the higher education systems in other countries, academic staff at Malaysian universities are expected to engage both in teaching and research.

Table 1 Descriptive statistics

Inputs Outputs

Academic Research funding Undergraduate Master Doctoral Research

staff (MYR million) students students students publications

Median 693 5.08 12,141 1278 344 204

Mean 1301 6.38 15,257 2730 1252 698

SD 1736 6.19 14,881 2830 1431 919

Min 225 0.04 2087 58 32 8

Max 8476 19.73 74,064 8513 4035 2884

l£> v> ■-o

2.3 Limitations of our study

There are several obvious limitations of the above model specification that need highlighting. Similar limitations are common for applications of DEA to the higher education sector and have been discussed by other authors as well. First, our model does not include direct measures of quality of inputs and outputs. Thus, the data on academic staff does not allow us to distinguish between different levels of staff, from lecturers at the beginning of their careers to established professors. Also, we have no data on the academic quality of the student intake or the data on the quality of degrees awarded that are important to account for the quality of the teaching process. Second, we do not have data on the overall expenditure by universities, including expenditure on centralised academic services (e.g., library and computers) that should have a positive impact on both teaching and research outcomes. However, we do directly account for the number of academic staff (instead of staff costs) and research funding—the two main contributors to overall costs. Third, we assume that there is no heterogeneity issue, i.e., we consider all 20 public universities as operating in the same production technology. This assumption may be problematic with respect to universities that specialise in the areas that require greater than average funding for research.

Because we are unable to address the above limitations by a better specification of the model, the results of our analysis presented below should be viewed with caution. If a particular university is shown as inefficient, this may be because some strengths of this university, e.g., the quality of teaching or research, or the fact that this university specialises in an area that requires more than the average amount of funding, are not taken into account. A full analysis of university performance that would take all such considerations into account goes beyond the scope of our paper.

Despite the above limitations, the input and output set available to us is sufficient to demonstrate the suggested methodology and its potential advantages over the standard DEA models. This application gives us the context in which the various assumptions of the HRS model with production trade-offs can be discussed and the results of computation interpreted.

3 The hybrid returns-to-scale model

Traditional applications of DEA are based on either the VRS or CRS production technology (Banker et al. 1984; Charnes et al. 1978). Both technologies are convex and freely disposable. If the analyst believes that scale economies (or diseconomies) cannot be ignored, the VRS model is used. Otherwise, if the effect of scale economies is assumed to be negligible, the units in the technology can be proportionally scaled up and down, and the CRS technology is a reasonable modelling choice.

In deciding whether the VRS or CRS technology should be used in our application, we make two conflicting observations. First, we note that it is a common managerial practice for universities to link their recruitment decisions to the number of registered students, thus maintaining a certain student-to-staff ratio, at least in the long run. Furthermore, in many universities the teaching load of each member of academic staff is typically calculated in proportion to the number of students attending their classes.

It is therefore reasonable to assume that, for example, a proportional increase of all three categories of student (undergraduate, master and doctoral) by 10% is possible provided the number of academic staff is also increased by the same factor. This does not change

the student-to-staff ratios and the teaching load per member of academic staff. A similar proportional relationship holds if the students and staff are reduced. Therefore, if our model included only academic staff as the single input, and the three outputs measuring students of different types, we could use the CRS model.

Note, however, that the above proportional relationship between students and staff may not extend to the research activities represented by research funding as input, and publications as output. For example, it may be argued that a larger amount of funding does not necessarily result in an increase of publications in the same proportion. Therefore the assumption of CRS may not be fully justified in the given context and the more conservative VRS technology should be used.

It is clear that the described situation presents us with a dilemma as to which DEA model should be used. While the assumption of CRS is problematic and the assumption of VRS is acceptable, the latter underutilizes our knowledge that students and staff are proportional. Below we show that this knowledge can be fully incorporated in the hybrid returns-to-scale (HRS) technology.

3.1 Underlying assumptions

The HRS technology was developed by Podinovski (2004a). It bridges the proportionality gap between the VRS and CRS technologies by assuming that only some subsets of inputs and outputs are mutually proportional (staff and students in our application), while the remaining inputs and outputs are excluded from this proportion (research funding and publications). In other words, in the HRS model the subset of inputs and outputs deemed proportional exhibits CRS, while the remaining measures exhibit VRS. Podinovski (2004a) refers to this relationship as selective proportionality.

To see the exact meaning of selective proportionality, we consider two scenarios that help us refine this notion and justify it in the context of higher education.

First, consider the expansion scenario. In this case we make the following assumption.

Assumption 1 For any university, it is technologically possible to change its inputs and outputs as follows:

• The number of academic staff and the three types of student are all increased in the same proportion p > 1.

• The amount of research funding and research publications remain unchanged.

Note that Assumption 1 is reasonably conservative and easy to justify. It may be argued that, as the number of staff increases, so should the number of research publications. However, because the amount of research funding is not increasing (because as discussed it is probably unsafe to include it in the proportion), the corresponding increase of research publications may be significantly below the proportion in which the academic staff is increased. Therefore, keeping the research publications in Assumption 1 constant is equivalent to accepting the worst-case scenario (very low or zero rate of increase) with regard to this output. This logic helps us to avoid a possibly unsubstantiated discussion as to what would be a reasonable rate of increase for publications with respect to a proportional increase of academic staff, if the funding is not increased.7

7 By Assumption 1, we do not mean that research publications do not increase with respect to academic staff. We only accept that the actual increase cannot be described by a proportion. Below we argue that a suitable relationship between academic staff and research publications can still be formalised by an appropriate production trade-off—see Assumption 7.

Now consider the contraction scenario in which the staff and students are proportionally reduced by some factor 0 < a < 1. As in the expansion scenario, we keep the research funding fixed. However, in contrast with Assumption 1, we can no longer assume that the publications would remain constant. This is because both research funding and academic staff are important for the publications, and a reduction of one of them should be detrimental to the research output. A particular difficulty here is that we do not know in what proportion the publications should be reduced. As demonstrated in Podinovski (2004a) and Podinovski et al. (2014), we can make two equivalent assumptions in this respect.

First, we can assume that the research publications should be reduced in the same proportion a < 1 as academic staff. This appears reasonable because the time available for research (measured by staff numbers) is reduced by a while the research funding is not changed at all.

Second, we may completely ignore the above reason and accept the worst-case scenario, namely, that the research publications would drop to zero, for any 0 < a < 1. This is the most conservative assumption which is difficult to argue against. Interestingly, as shown in Podinovski (2004a) and further explained in Podinovski et al. (2014), the latter worst-case assumption is equivalent to the former assumption of proportional reduction of research publications by a, in any convex technology.

Assumption 2 For any university, it is technologically possible to change its inputs and outputs as follows:

• The number of academic staff and the three types of student are all reduced in the same proportion 0 < a < 1.

• The amount of research funding remains unchanged.

• The number of research publications is reduced to zero. (Note that below we assume that the technology is convex, and an equivalent assumption is that the number of publications is also reduced in proportion a.)

Remark 1 As noted in Sect. 2.2, in many DEA applications to the higher education sector research funding is viewed as an output. Let us consider how this affects the above two assumptions. First, as in Assumption 1, let both the academic staff and the three types of student increase in some proportion / > 1. Because academic staff is now the single input, its increase by / should have a positive effect on the two outputs: research funding and publications. As above, it appears unsafe to assume that these two outputs could increase in the same proportion / . However, we can safely assume that both should not decline and could remain constant. This means that Assumption 1 does not change if we treat research income as an output.

Now consider Assumption 2. If research funding is an output and the academic staff is reduced, so is the capacity of the staff to generate the funding. Similar to research publications, and to account for the worst-case scenario, we should assume that research funding might drop to any value, for which the only safe lower bound is zero. As noted, in a convex technology this is equivalent to the research funding being reduced in the same proportion a in which the staff and students are reduced. This implies that, if we treat research funding as an output, the last two statements of Assumption 2 need changing to the following one: the amount of research funding and number of research publications are reduced to zero (or, equivalently, are reduced in proportion a, if technology is convex).

3.2 The HRS technology

The HRS technology introduced by Podinovski (2004a) is an extension to the VRS technology in which some specified subset of outputs is assumed proportional to a subset of inputs, in the sense stated by Assumptions 1 and 2.8

Consider the following notation. Let T denote a production technology (to be defined below), and let I = {1,..., m} and O = {1,..., s} be the sets of inputs and outputs, respectively. The elements of T are referred to as decision making units (DMUs) and denoted (X, Y), where X e R+ and Y e R+ are the input and output vectors, respectively. Denote J = {1,..., n} the set of observed DMUs. The latter are stated as (Xj, Yj), j e J.9

To represent selective proportionality as in Assumptions 1 and 2, let IP C I and OP C O be the subsets of inputs and outputs that are assumed to be mutually proportional. In our application, the set IP includes academic staff and the set OP includes undergraduate, master and doctoral students.

Furthermore, let INP = I\IP and ONP = O\OP be, respectively, the subsets of inputs and outputs that are excluded from the assumption of proportionality. In our application, the set INP includes research funding, the set ONP includes research publications.10 Using this notation, we represent any DMU in T in the extended form as follows:

(X, Y) = (XP, XNP YP Ynp) ,

where the subvectors XP, XNP, YP, and YNP include the inputs and outputs in the subsets IP, INP, OP, and ONP, respectively.

The HRS technology is defined by the following six axioms (Podinovski 2004a).

Axiom 1 (Feasibility of observed data) (Xj, Yj) e T, for all j e J.

Axiom 2 (Free disposability) Let (X, Y) e T. Consider any DMU (XY') e R+ x R+ such that X' > X and 0 < Y' < Y. Then (XY') e T n

Axiom 3 (Convexity) Technology T is a convex set.

Axiom 4 (Selectiveproportionality: expansion scenario) Let (X, Y) e T.ThenDMU(PXP, XNP, pYP, Ynp) e T, for any p> 1.

Axiom 5 (Selective proportionality: contraction scenario) Let (X, Y) e T. Then DMU (aXP, XNP,aYP, 0) e T, for any 0 < a < 1.

Axiom 6 (Closedness) Technology T is a closed set.

The above six axioms describe the properties of the HRS technology. The first three of these are the standard axioms of the VRS technology stated by Banker et al. (1984). Axioms 4 and 5 are the selective analogues of the ray unboundness (proportionality) axiom of Banker

8 A further extension to the HRS technology in developed in Podinovski (2009).

9 We further make a usual assumption required for the unproblematic definition of technology and efficiency measures. Namely, we assume that each observed DMU has at least one positive input and at least one positive output. We also assume that no input and no output is equal to zero for all observed DMUs.

10 If, as in Remark 1, research funding is treated as output, the set INP is empty and the set ONP includes both research funding and publications.

11 We use bold notation 0 for the vector of zeros of appropriate dimension.

et al. (1984). Note that Axioms 4 and 5 are consistent with Assumptions 1 and 2 about the technology employed by universities.

Finally, Axiom 6 is automatically satisfied in the VRS and CRS technology, and does not need stating. However, as shown by example in Podinovski (2004a), it does not generally follow in the case of selective proportionality, and needs an explicit statement.

Following the approach of Banker et al. (1984), the HRS technology Thrs is defined using the minimum extrapolation principle. Namely, Thrs is the intersection of all sets in Rm x Rs that satisfy Axioms 1-6. It is straightforward to verify that this intersection, i.e. technology Thrs itself, satisfies the above six axioms. Therefore, Thrs is the smallest technology that satisfies the stated axioms. This last characteristic is important because it means that Th rs includes only those DMUs that are needed to satisfy the stated properties (axioms) of technology and does not include any unnecessary (unsubstantiated by assumptions) DMUs.12 The following theorem establishes an equivalent constructive description of technology Th rs. In this formulation and below, bold vectors 0 and 1 denote vectors of zeros and ones of appropriate dimensions, respectively. The symbol T denotes transposition.

Theorem 1 (Podinovski 2004a) Technology Thrs is the set of all nonnegative DMUs (X, Y) e R+ x R+ for which there exist vectors k, p,v e R+ such that the following conditions are satisfied:

£(kj + nj - Vj)XP < XP, (1a)

X kjXNP < XNP, (1b)

X (kj + Vj - Vj)YP > yp, (1c)

X (kj - vj)YNP > YNP, (1d)

k - v > 0, (1e)

1Tk = 1. (1f)

The vector k in the above formulation plays the same role as in the conventional statement of the VRS technology. The variables pj are used to allow for the selective expansion of observed DMUs in accordance with Assumption 1. Similarly, variables vj allow for the selective contraction of observed DMUs as in Assumption 2. The full meaning of conditions (1) is discussed in Podinovski et al. (2014) who give an alternative statement of technology Thrs in terms of the expansion and contraction factors / and a used in Axioms 4 and 5.

3.3 The HRS model

Similar to the standard VRS and CRS technologies, the HRS technology Th rs can be used to explore different policy questions. Depending on the task, DEA models based on these technologies may be output or input-oriented, or employ other directions for the projection of inefficient DMUs (see, e.g., the discussions in Johnes 2006; Thanassoulis et al. 2011). In

12 Podinovski et al. (2014) illustrate the importance of the minimum extrapolation principle by the following example. It is well known (Banker etal. 1984) that the VRS technology is defined by Axioms 1-3, stated above. Note that there are many other technologies (e.g., CRS) that also satisfy the same three axioms. However, the VRS technology is the smallest of all such technologies—it is the intersection of all of them.

our application we use the output-oriented HRS model—this common approach corresponds to the scenario in which the policy maker wishes to identify the highest feasible levels of the teaching and research outputs for the fixed levels of allocated resources (academic staff and research funding).

Let DMU (Xo, Yo ) e Thrs—this means that (Xo, Yo ) satisfies conditions (1) with some vectors X, p, v e R+. The output radial efficiency of DMU (Xo, Yo) is found as the inverse of the optimal value n* in the following linear program:

n = max n

subject to ^ (X j + p j - vj ) xP < XP,


X (Xj + pj - vj)YP > nYP,

X (X j - v j )YNP > nYNP,

X - v > 0,

1TX = 1,

X, p,v > 0, n sign free. (2)

Alternatively, model (2) can be stated in the vector-matrix form as follows. Let X be the n x m matrix whose columns are the input vectors Xj of observed DMUs. Similarly, let Y be the n x s matrix of output vectors Yj of observed DMUs. Further, let the n x m matrix XX be obtained from matrix X by changing to zero all its elements in rows i e INP. Similarly, the n x s matrix Y is obtained from matrix Y by changing to zero all elements in rows r e ONP. In this notation model (2) is stated as follows:

n * = max n

subject to XX + Xp - Xv < Xo,

Y x + Yp - Y v > nYo, X - v > 0, 1TX = 1,

X, p, v > 0, n sign free. (3)

It is worth pointing out the difference between the terms - X v and - Y v in the input and output inequalities of program (3). In the former of these we use matrix XX and in the latter we use matrix Y. This corresponds to the different structure of inequalities (1b) and (1d) in the description of technology Thrs.

4 Production trade-offs

Below we suggest a further extension to the HRS technology employed by the universities by incorporating additional judgements about the production process. Such judgements take

on the form of production trade-offs that we believe are technologically realistic in our application.13

To illustrate this development, consider the following six assumptions.

Assumption 3 For any university,14 it is technologically possible to change its outputs simultaneously as follows, while keeping all the other inputs and outputs unchanged:

• increase the number of undergraduate students by 1,

• reduce the number of master students by 1.

The above assumption takes into account the fact that the amount of resources (staff in our model) required for the teaching of one master student is not smaller than the amount of resources needed to teach one undergraduate student (e.g., because master students are typically taught in smaller classes than undergraduates). Therefore, the reduction of master students by 1, as stated in Assumption 3, releases sufficient resources to teach one additional undergraduate student.

Assumption 3 states the trade-off between undergraduate and master students that can be expressed as follows:

P1 = (0, 0)T, Qx = (1,-1, 0, 0)T. (4)

In the above notation, vectors P1 and Q1 show the simultaneous changes to, respectively, the inputs and outputs that are assumed technologically feasible by Assumption 3. In particular, P1 is a zero vector because no inputs are changed. Vector Q1 identifies a feasible increase of undergraduate students by 1 and simultaneous reduction of master students by 1, while keeping doctoral students and publications unchanged.

Assumption 4 For any university, the following simultaneous changes are possible:15

• reduce the number of undergraduate students by 3,

• increase the number of master students by 1.

Assumption 4 deals with the case opposite to that stated in Assumption 3. It assumes that the amount of resources required to teach one master student does not exceed the amount of resources required for one undergraduate student by more than a factor of 3. Therefore, the resources released by the reduction of 3 undergraduate students are sufficient to teach one master student.

Assumption 4 can be formally stated as the following trade-off:

P2 = (0, 0)T, Q2 = (-3, 1, 0, 0)T. (5)

It is worth emphasising that Assumption 4 does not imply that master students require exactly 3 times more resources compared to undergraduate students. The exact ratio will likely be different for different universities and also depend on the methodology used to evaluate the resources. Therefore, we view the value 3 as a safe upper bound on all such

13 The exact numerical values of the trade-offs expressed by Assumptions 3-8 are specific to the described context. Similar trade-offs are likely to be valid in other applications to the higher education sector, e.g., in other countries, but the numerical values and the logic leading to them obviously need to be verified.

14 We assume that the changes stated in Assumptions 3-8 do not result in negative inputs and outputs. Thus, Assumption 3 applies to any university that has at least one master student.

15 In Assumptions 4-8 we identify only the changes to certain inputs and outputs that are assumed possible. To avoid repetitive statements, we assume without mentioning that the remaining inputs and outputs are kept constant.

ratios that may be assessed in different ways and vary across universities. In particular, a typical undergraduate class in our application is no more than three times larger than a class for master students. Furthermore, the credit to academic staff for the teaching of master students is typically higher (but not significantly) than the credit for teaching undergraduate students. The specified upper bound 3 is a safe estimate arising from any such differences.

Using the same logic as above, we can specify the two trade-offs relating undergraduate to doctoral students.

Assumption 5 For any university, the following simultaneous changes are possible:

• increase the number of undergraduate students by 2,

• reduce the number of doctoral students by 1.

The above statement is based on the assumption that the resources required for one doctoral student are sufficient for the teaching of at least two undergraduates.16 This is expressed as the following trade-off:

P3 = (0, 0)T, Q3 = (2, 0,-1, 0)T. (6)

Assumption 6 For any university, the following simultaneous changes are possible:

• reduce the number of undergraduate students by 10,

• increase the number of doctoral students by 1.

Assumption 6 is formally stated as:

P4 = (0, 0)T, Q4 = (-10, 0, 1, 0)T. (7)

The next two statements link changes of academic staff to the corresponding changes in student numbers and publications.

Assumption 7 For any university, the following simultaneous changes are possible:

• increase the number of academic staff by 1,

• increase the number of undergraduate students by 5, doctoral students by 0.1 and publications by 0.25.

The above assumption leads to the statement of a linked trade-off, as it involves simultaneous changes to both inputs and outputs:

P5 = (1, 0)T, Q5 = (5, 0, 0.1, 0.25)T. (8)

We view Assumption 7 as reasonably conservative and safe to accept. For example, as follows from the data, the staff-to-student ratio across all 20 universities is always within the range between 6 and 32 (if only undergraduate students are counted) and between 10 and 36 (if all students are counted). Similarly, the number of doctoral students per staff varies across

16 In the absence of a suitable reliable methodology, we estimate that, by supervising a doctoral student, academics may earn 10 % or more of their annual teaching credit. Consider a typical university with the ratio of undergraduate students to staff equal to 20. Then each undergraduate student equates to 5 % of the annual teaching credit. Based on this, we estimate that a doctoral student may require at least twice the amount of teaching resources compared to an undergraduate student, leading to Assumption 5. The upper bound on the ratio between the resources required for the doctoral and undergraduate students may be larger than 2 and is difficult to estimate. To be on the safe side, we assume that in no university it is greater than 10. Although this value may appear over-cautious, the resulting trade-off in Assumption 6 is certainly feasible in the technology sense and its use leads to a meaningful expansion of the technology (see Sect. 5).

the universities in the range between 0.12 and 2.6. Therefore, the assumption of trade-off (8) that an additional member of staff is a sufficient resource to increase the number of undergraduate and doctoral students by 5 and 0.1, respectively, should not be problematic for any university.17

Finally, eight universities in the sample have publication rates per member of staff lower than 0.25. However, most of these have high student-to-staff ratios. Because of this, we would argue that the publication rates of these universities are not representative of the rate that can be achieved if a new member of staff is required to teach only 5 students as assumed in trade-off (8).

Assumption 8 For any university, the following simultaneous changes are possible:

• reduce the number of academic staff by 1,

• reduce the number of undergraduate students by 35, doctoral students by 3 and publications by 2.

Assumption 8 translates to the following trade-off:

P6 = (-1, 0)T, Q6 = (-35, 0, -3, -2)T. (9)

Note that trade-off (9) assumes a larger reduction of the number of undergraduate and doctoral students, and publications if the number of academic staff is reduced than the increase of the same outputs if the staff number is increased, as expressed in trade-off (8). This reflects the fact that the teaching and publication requirements vary across universities. Because the stated trade-offs are assumed to be valid at any university, if an extra member of staff becomes available, we can safely assume that the student numbers and research publications will increase in the lowest proportion that is acceptable to all universities. However, if the number of staff is reduced, the only safe assumption acceptable to all universities is that the reduction of the student numbers and publications may be on the higher side across the observed universities.

5 The HRS model with production trade-offs

Podinovski (2004b) introduced the notion of production trade-offs for the CRS and VRS technology as the dual forms of weight restrictions in the multiplier model.18 In our application we extend the use of production trade-offs to the HRS model. Suppose we have identified K trade-offs stated in the form

( Pt, Qt ), t = 1,...,K, (10)

where the vectors Pt e Rm and Qt e Rs describe simultaneous changes to the inputs and outputs that are assumed technologically feasible at any DMU in technology T. Examples of this are trade-offs (4)-(9) discussed above.

Because the application of trade-off (Pt, Qt) to a DMU leads to another theoretically possible DMU, we can apply the same trade-off again, provided the inputs and outputs of the

17 Note that Assumption 7 stating that an additional member of academic staff should be able to publish at least 0.25 papers a year (without extra research funding) does not contradict Assumption 1 that excludes publications from the proportion with respect to staff.

18 Further theoretical results concerning the role of production trade-offs in expanding the model of technology were obtained by Podinovski and Bouzdine-Chameeva (2013, 2015). The dual relationship between trade-offs and weight restrictions of different types was demonstrated in Podinovski (2005, 2007a).

resulting DMU remains nonnegative. Repeating this process any number of times nt > 0 and allowing nt to take on non-integer values leads to the following axiom stated in Podinovski (2004b):

Axiom 7 (Feasibility of trade-offs) Let (X, Y) e T. Then, for any trade-off t in (10) and any nt > 0, the DMU (X + ntPt, Y + ntQt) e T, provided X + ntPt > 0 and Y + ntQt > 0.

The incorporation of production trade-offs in the HRS model (3) is straightforward and leads to the following program:

n* = max n (11a)

subject to XX + Xp - Xv + X ntPt < Xo, (11b)

YX + Yp - Yv + XntQt > nYo, (11c)

X - v > 0, (11d)

1TX = 1, (11e)

X,p,v,n > 0,n sign free. (11f)

The meaning of constraints of the above program is clear. The DMU (Xo, Yo) is bench-marked against the DMU stated on the left-hand side of inequalities (11b) and (11c). The first three terms of the latter define a DMU in the standard HRS model (3). This DMU is subsequently modified by production trade-offs (10) applied in arbitrary nonnegative proportions nt, t = 1,..., K. Therefore, the maximization of n projects DMU (Xo, Yo) on the boundary of technology Thrs which is further expanded by production trade-offs (10).

6 Computations and analysis

To demonstrate the advantages of the proposed methodology, we evaluate the output radial efficiency of public universities in Malaysia in three models: the standard VRS model, the HRS model (2) and the HRS model with production trade-offs (11). For reference purposes, we also evaluate the efficiency of universities in the standard CRS model. We use these results to highlight the differences between the assumptions of CRS and HRS.

We performed all computations in a common spreadsheet solver. The resulting output radial efficiency of the universities is shown in Table 2.

As expected, because of the relatively small number of universities in the sample, the efficiency discrimination of the standard VRS model is somewhat low: it identifies 11 universities whose output radial efficiency is equal to 1. To simplify terminology, we refer to such universities as efficient.19

Computing the efficiency in the HRS model (2) results in a significant improvement in the efficiency discrimination. This is consistent with the theoretical observation that the VRS technology is a subset of the HRS technology. Therefore the efficiency of each university in

19 A proper term here is "weakly efficient". In order to investigate whether a weakly efficient university is fully efficient (in the Pareto sense), we need to perform the second-stage procedure maximizing the sum of input and output slacks, or an analogous one-stage approach—see Thanassoulis et al. (2008) for a discussion. For the CRS and VRS models with production trade-offs an appropriate procedure was developed in Podinovski (2007b).

Table 2 Output radial efficiency in different models

University VRS HRS HRS with trade-offs (4-7) HRS with trade-offs (4-9) CRS

UDM 1 1 1 0.89 1

UIAM 0.78 0.66 0.62 0.62 0.65

UiTM 1 0.35 0.35 0.33 0.55

UKM 1 1 1 1 0.89

UM 1 1 1 1 1

UMK 1 0.58 0.54 0.54 0.58

UMP 0.72 0.61 0.56 0.56 0.69

UMS 0.65 0.6 0.56 0.56 0.62

UMT 0.86 0.74 0.61 0.61 0.74

UniMAP 1 1 1 1 1

UniMAS 0.66 0.65 0.6 0.6 0.65

UPM 1 1 1 1 1

UPNM 1 0.71 0.64 0.64 0.71

UPSI 1 1 1 1 1

USIM 0.82 0.69 0.64 0.64 0.69

USM 0.92 0.8 0.77 0.77 0.82

UTeM 0.6 0.51 0.45 0.45 0.59

UTHM 0.65 0.63 0.6 0.6 0.63

UTM 1 0.88 0.88 0.85 1

UUM 1 0.92 0.92 0.89 1

the HRS model cannot be higher than in the VRS model. As shown in Table 2, only 6 of the 11 efficient universities in the VRS model retain their efficient status in the HRS model. In some cases the reduction of efficiency is significant: for example, the efficiency of UiTM and UMK changes from 1 to 0.35 and 0.58, respectively.

An even better discrimination is observed when we incorporate production trade-offs (49) in the HRS model and use program (11) for computations. To observe this effect better, we first use the HRS model with trade-offs (4-7) only. We subsequently incorporate all six trade-offs (4-9) in the model.

The HRS model with trade-offs (4-7) links the "production" of master and doctoral students to undergraduate students by means of putting bounds on the relative amounts of resources required to teach these categories of student. This results in a noticeable improvement on efficiency discrimination. For example, the efficiency of UMT changes from 0.74 in the HRS model to 0.61 in the HRS model with trade-offs (4-7).

A further incorporation of the last two trade-offs (8) and (9) brings to the HRS model additional information about the "worth" of one member of academic staff in terms of students and publications, as stated in Assumptions 7 and 8. The results show that UDM that was efficient in all previous models, is no longer efficient in the HRS model with all six tradeoffs.

Overall our results are consistent with the recognition by MOE of the five designated research universities as the top public universities in Malaysia (see Sect. 2.1). Three of them

(UKM, UM and UPM) are clearly efficient in all the models we used in our analysis. The two other research universities (USM and UTM) are also among the best performers.20

A further two public universities, UDM and UUM, also perform well on the metrics used in our model, primarily because of the strength of their teaching output and despite the relative weakness of their research. In particular, the efficiency of UDM is highly sensitive to the assumption of the number of research publications per academic staff in trade-off (8). For example, if we assume that an extra member of staff can publish at least 0.4 or 0.5 papers per year and change trade-off (8) accordingly, the efficiency of UDM falls from the current 0.89 to 0.66 and 0.58 respectively, thus removing it from the list of best performers. No other university in our application shows a similar tendency.21 While it may be a matter of discussion how many papers a staff member may realistically publish in a year, it is clear that UDM shows a particular weakness in the publication output compared to the other best-performing universities in the sample.

The high efficiency score of UUM also relies on its teaching output. UUM has the largest number of undergraduate students and the second-highest student-to-staff ratio across all public universities. This ensures that UUM is efficient in the production technology even though it has the lowest rate of publications per staff across all universities in the data set.

Remark 2 It is interesting to compare the efficiency of universities in the HRS and CRS models. As discussed above, we do not consider the underlying technology as exhibiting CRS, because a change of the amount of academic staff and research funding by some factor may not necessarily lead to the change of the number of publications by the same factor. Nevertheless, the CRS model is sometimes considered as an acceptable compromise, especially if the sample of DMUs is small. The results in Table 2 show that the HRS model is not inferior in the sense of efficiency discrimination to the CRS model, and in many cases is actually superior to the latter.22 Indeed, only in the case of UKM (and marginally in the case of UIAM) the CRS model discriminates better. For seven other universities, the HRS model discriminates better, in particular, showing UTM and UUM as inefficient.

This observation shows that the HRS model (and its further extension with production trade-offs) may be considered as a viable alternative to the CRS model. In the reported application, the former not only shows a generally better discrimination than the latter, but it is also based on realistic production assumptions, while the CRS technology appears problematic. Note that it would obviously be unsubstantiated to extend this observation to an arbitrary context. The assumption of selective proportionality that underlines the HRS model needs to be carefully examined in each specific application along the lines of our discussion in Sect. 3.1.

7 Conclusion

In this paper we argue that in applications of DEA to the higher education sector, the conventional VRS model may not fully utilize our understanding of the production process. In

20 As discussed in Sect. 2.2, our model has a number of limitations arising from the available data. This model does not account for all measures and factors available to MOE in their own assessment.

21 The corresponding efficiency of UiTM is only marginally reduced from 0.33 to 0.32, while there is no reduction for the remaining 18 universities.

22 As shown in Podinovski (2004b), generally neither the HRS technology is a subset of the CRS technology, nor is the latter a subset of the former. The exception is the case INP = 0. In this case the HRS technology is a subset of the CRS technology.

particular, we point out that students and academic staff may be regarded as being mutually proportional. This is substantiated by, for example, the use of student-to-staff ratios in recruitment decisions and the common practice of crediting academic staff in proportion to the number of their students at different levels. At the same time, there may be no strong evidence that research funding and the number of published papers could be assumed a part of the above proportion.

Using the sample of 20 public universities in Malaysia, we show that a better model that addresses the above situation is the hybrid returns-to-scale (HRS) model that combines the assumptions of CRS with respect to students and staff and VRS with respect to research funding and publications. Similar to the conventional VRS and CRS models, the HRS model of technology is based on several assumptions stated as axioms. The critical assumption underlying the HRS model is the assumption of selective proportionality (between students and staff in our application). In our paper we provide a detailed discussion that justifies the applicability of this assumption in our study.

We further expand the HRS technology by the incorporation of several production tradeoffs. An example of such trade-offs is the statement that the teaching of one undergraduate student does not require more resources (academic staff time) than the teaching of a master or doctoral student. In our paper we show that the incorporation of production trade-offs in the HRS model leads to further significant improvements on efficiency discrimination.

Overall, we argue that in the reported application to the higher education sector, the suggested HRS model with additional production trade-offs is a better-informed model than the conventional DEA approaches. Its use leads to an improved efficiency discrimination, which is expected from theoretical considerations and is supported by calculations.

Acknowledgments The authors are grateful to Educational Planning and Research Division (EPRD) of the Ministry of Education Malaysia for permission to collect and use the data for this study.


Allen, R., Athanassopoulos, A., Dyson, R. G., & Thanassoulis, E. (1997). Weights restrictions and value judgements in data envelopment analysis: Evolution, development and future directions. Annals of Operations Research, 73, 13-34.

Amado, C. A. F., & Dyson, R. G. (2009). Exploring the use of DEA for formative evaluation in primary diabetes care: An application to compare English practices. Journal of the Operational Research Society, 60(11), 1469-1482.

Amado, C. A. F., & Santos, S. P. (2009). Challenges for performance assessment and improvement in primary health care: The case of the Portuguese health centres. Health Policy, 91(1), 43-56.

Athanassopoulos, A. D., & Shale, E. (1997). Assessing the comparative efficiency of higher education institutions in the UK by means of data envelopment analysis. Education Economics, 5(2), 117-134.

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078-1092.

Beasley, J. E. (1990). Comparing university departments. Omega, 18(2), 171-183.

Beasley, J. E. (1995). Comparing teaching and research efficiencies. Journal of the Operational Research Society, 46(4), 441-452.

Blackburn, V., Brennan, S., & Ruggiero, J. (2014). Nonparametric estimation of education production and costs using data envelopment analysis. New York: Springer.

Brennan, S., Haelermans, C., & Ruggiero, J. (2014). Nonparametric estimation of education productivity incorporating nondiscretionary inputs with an application to Dutch schools. European Journal of Operational Research, 234(3), 809-818.

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429-444.

Cooper, W. W., Seiford, L. M., & Tone, K. (2007). Data envelopment analysis. A comprehensive text with models, applications, references and DEA-Solver software (2nd ed.). New York: Springer.

Flegg, A. T., Allen, D. O., Field, K., & Thurlow, T. W. (2004). Measuring the efficiency of British universities: A multi-period data envelopment analysis. Education Economics, 12(3), 231-249.

Johnes, J. (2006). Measuring teaching efficiency in higher education: An application of data envelopment analysis to economics graduates from UK Universities 1993. European Journal of Operational Research, 174(1), 443-456.

Johnes, J. (2008). Efficiency and productivity change in the English higher education sector from 1996/97 to 2004/5. The Manchester School, 76(6), 653-674.

Johnes, J., & Johnes, G. (1995). Research funding and performance in U.K. university departments of economics: A frontier analysis. Economics of Education Review, 14(3), 301-314.

Johnes, J., & Yu, L. (2008). Measuring the research performance of Chinese higher education institutions using data envelopment analysis. China Economic Review, 19(4), 679-696.

Johnson, A. L., &Ruggiero, J. (2014). Nonparametric measurement of productivity and efficiency in education. Annals of Operations Research, 221, 197-210.

Khalili, M., Camanho, A. S., Portela, M. C. A. S., & Alirezaee, M. R. (2010). The measurement of relative efficiency using data envelopment analysis with assurance regions that link inputs and outputs. European Journal of Operational Research, 203(3), 761-770.

Kuosmanen, T. (2005). Weak disposability in nonparametric productivity analysis with undesirable outputs. American Journal of Agricultural Economics, 87(4), 1077-1082.

Kuosmanen, T., & Podinovski, V. V. (2009). Weak disposability in nonparametric production analysis: Reply to Fare and Grosskopf. American Journal of Agricultural Economics, 91(2), 539-545.

Liu, J. S., Lu, L. Y. Y., Lu, W.-M., & Lin, B. J. Y. (2013). A survey of DEA applications. Omega, 41(5), 893-902.

MOE. (2014). Ministry of Education Malaysia. Accessed October 2014.

Ng, Y. C., & Li, S. K. (2000). Measuring the research performance of Chinese higher education institutions: An application of data envelopment analysis. Education Economics, 8(2), 139-156.

Pedraja-Chaparro, F., Salinas-Jimenez, J., & Smith, P. (1997). On the role of weight restrictions in data envelopment analysis. Journal of Productivity Analysis, 8(2), 215-230.

Podinovski, V. V. (1999). Side effects of absolute weight bounds in DEA models. European Journal of Operational Research, 115(3), 583-595.

Podinovski, V. V. (2001). DEA models for the explicit maximisation of relative efficiency. European Journal of Operational Research, 131(3), 572-586.

Podinovski, V. V. (2004a). Bridging the gap between the constant and variable returns-to-scale models: Selective proportionality in data envelopment analysis. Journal of the Operational Research Society, 55(3), 265-276.

Podinovski, V. V. (2004b). Production trade-offs and weight restrictions in data envelopment analysis. Journal of the Operational Research Society, 55(12), 1311-1322.

Podinovski, V. V. (2005). The explicit role of weight bounds in models of data envelopment analysis. Journal of the Operational Research Society, 56(12), 1408-1418.

Podinovski, V. V. (2007a). Improving data envelopment analysis by the use of production trade-offs. Journal of the Operational Research Society, 58(10), 1261-1270.

Podinovski, V. V. (2007b). Computation of efficient targets in DEA models with production trade-offs and weight restrictions. European Journal of Operational Research, 181(2), 586-591.

Podinovski, V. V. (2009). Production technologies based on combined proportionality assumptions. Journal of Productivity Analysis, 32(1), 21-26.

Podinovski, V. V., & Bouzdine-Chameeva, T. (2013). Weight restrictions and free production in data envelopment analysis. Operations Research, 61(2), 426-437.

Podinovski, V. V., & Bouzdine-Chameeva, T. (2015). Consistent weight restrictions in data envelopment analysis. European Journal of Operational Research, 244(1), 201-209.

Podinovski, V. V., Ismail, I., Bouzdine-Chameeva, T., & Zhang, W. (2014). Combining the assumptions of variable and constant returns to scale in the efficiency evaluation of secondary schools. European Journal of Operational Research, 239(2), 504-513.

Podinovski, V. V., & Kuosmanen, T. (2011). Modelling weak disposability in data envelopment analysis under relaxed convexity assumptions. European Journal of Operational Research, 211(3), 577-585.

Portela, M. C. S., Camanho, A. S., & Borges, D. (2012). Performance assessment of secondary schools: The snapshot of a country taken by DEA. Journal of the Operational Research Society, 63(8), 1098-1115.

Santos, S. P., & Amado, C. A. F. (2014). On the need for reform of the Portuguese judicial system—Does data envelopment analysis assessment support it? Omega, 47, 1-16.

Sarrico, C. S., & Dyson, R. G. (2004). Restricting virtual weights in data envelopment analysis. European Journal of Operational Research, 759(1), 17-34.

Sarrico, C. S., Hogan, S. M., Dyson, R. G., & Athanassopoulos, A. D. (1997). Data envelopment analysis and university selection. Journal of the Operational Research Society, 48(12), 1163-1177.

Thanassoulis, E., Kortelainen, M., Johnes, G., & Johnes, J. (2011). Costs and efficiency of higher education institutions in England: A DEA analysis. Journal of the Operational Research Society, 62(7), 1282-1297.

Thanassoulis, E., Portela, M. C., & Allen, R. (2004). Incorporating value judgements in DEA. In W. W. Cooper, L. M. Seiford, & J. Zhu (Eds.), Handbook on data envelopment analysis (pp. 99-138). Boston: Kluwer.

Thanassoulis, E., Portela, M. C. S., & Despic, O. (2008). Data envelopment analysis: The mathematical programming approach to efficiency analysis. In H. O. Fried, C. A. K. Lovell, & S. S. Schmidt (Eds.), The measurement of productive efficiency and productivity growth (pp. 251-420). New York: Oxford University Press.

Wilkinson, R., & Yussof, I. (2005). Public and private provision of higher educationin Malaysia: A comparative analysis. Higher Education, 50(3), 361-386.

Worthington, A. C., & Lee, B. L. (2008). Efficiency, technology and productivity change in Australian universities, 1998-2003. Economics of Education Review, 27(3), 285-298.

Yu, M. L., Hamid, S., Ijab, M. T., & Soo, H. P. (2009). The e-balanced scorecard (e-BSC) for measuring academic staff performance excellence. Higher Education, 57(6), 813-828.