The impacts of household retrofit and domestic energy efficiency schemes: A large scale, ex post evaluation Academic research paper on "Earth and related environmental sciences"

Contents lists available at ScienceDirect

Energy Policy

journal homepage: www.elsevier.com/locate/enpol

The impacts of household retrofit and domestic energy efficiency schemes: A large scale, ex post evaluation

Phil Webber3, Andy Gouldsonbn, Niall Kerra

a ESRC Centre for Climate Change Economics and Policy, School of Earth and Environment, University of Leeds, UK b ESRC Centre for Climate Change Economics and Policy, School for Policy Studies, University of Bristol, UK

HIGHLIGHTS

• A large scale, ex post evaluation of the impacts of a household retrofit scheme.

• A new methodology to separate retrofit impacts from background trends.

• Shows impacts of retrofit have been 1.2-1.7 times higher than predicted.

• Impacts as predicted in lower income areas, higher in middle and upper income areas.

• Findings support the case for the wider and faster adoption of domestic retrofit.

ARTICLE INFO ABSTRACT

There is widespread interest in the ability of retrofit schemes to shape domestic energy use in order to tackle fuel poverty and reduce carbon emissions. Although much has been written on the topic, there have been few large-scale ex post evaluations of the actual impacts of such schemes. We address this by assessing domestic energy use before and after the Kirklees Warm Zone (KWZ) scheme, which by fitting insulation in 51,000 homes in the 2007-2010 period is one of the largest retrofit schemes completed in the UK to date. To do this, we develop and apply a new methodology that isolates the impacts of retrofit activity from broader background trends in energy use. The results suggest that the actual impacts of the KWZ scheme have been higher than predicted, and that the scale of any performance gaps or rebound effects have been lower than has often been assumed. They also suggest that impacts on energy use in lower income areas are consistent with predictions, but that impacts in middle and higher income areas are higher than predicted. These findings support the case for the wider and/or accelerated adoption of domestic retrofit schemes in other contexts.

© 2015 Published by Elsevier Ltd.

Article history: Received 12 January 2015 Received in revised form 31 March 2015 Accepted 21 April 2015

Keywords:

Energy

Efficiency

Retrofit

Households

Evaluation

Impacts

1. Introduction

Globally, over one-third of all final energy and half of electricity are consumed in buildings, and this consumption generates approximately one-third of global carbon emissions (IEA, 2013a). Energy efficiency in buildings is therefore of critical importance in areas such as fuel poverty, energy security and climate change. Although retrofit activities (i.e. the upgrading of existing buildings to improve their energy efficiency) seem to be technically viable and sometimes also economically attractive, there are multiple barriers to change that prevent the take-up of energy efficiency measures in households (see below). Many governments have

n Corresponding author. E-mail addresses: pwebber@gn.apc.org (P. Webber), a.gouldson@bristol.ac.uk (A. Gouldson), n.kerr@see.leeds.ac.uk (N. Kerr).

http://dx.doi.org/10.1016/j.enpol.2015.04.020 0301-4215/© 2015 Published by Elsevier Ltd.

introduced retrofit schemes that attempt to overcome these barriers and to influence the diverse behaviours and practices that give rise to energy use in buildings. Questions about the extent to which these schemes are actually able to help different types of households to overcome the multiple barriers to change that prevent them from taking up various energy efficiency measures are critically important in the often highly political debates on the future of government support for retrofit.

Much has been written about the design and delivery of retrofit schemes and the extent to which they might change energy behaviours in households (c.f. Dowson et al., 2012; Hoicka et al., 2014; Achtnicht and Madlener, 2014). Although there have been other evaluations of the impacts of retrofit schemes and measures (see Clinch and Healy, 2001; Chapman et al., 2008; Grimes et al., 2011; Tovar, 2012), most studies rely upon models of energy performance rather than actual, measured energy usage, and there

are few larger scale, ex post evaluations of the impacts of retrofit schemes (IEA, 2008; Wingfield et al., 2008; Rosenow and Galvin, 2013). This paper seeks to address this important knowledge gap by assessing domestic energy use (and hence energy bills and carbon emissions) before and after the Kirklees Warm Zone (KWZ) scheme, which by fitting insulation in 51,000 UK homes in Kirklees area in the north of England in the 2007-2010 period is one of the largest retrofit schemes completed in the UK to date.

With so many factors shaping domestic sector energy use, and with scarce data on many of these, it is often hard to isolate the impact of the retrofit activity from the influence of all other factors. We therefore develop and apply an approach that compares domestic sector energy use area in Kirklees before and after the KWZ scheme and that separates the impacts of the KWZ scheme from the wider background trends. The KWZ scheme is a good case study through which to conduct such a detailed analysis because of the scale and geographical concentration of its activities and the data that was collected by the local authority both on the insulation measures that were installed and the different types of buildings that they were installed in (see below). This makes it possible for this study to discern the impacts of retrofit activity on household energy use and bills, and also carbon emissions, using data on energy use in small geographical areas that is publically available at the local level. An alternative approach would have been to compare energy use in Kirklees where the KWZ scheme took place with energy use in a directly comparable area that did not have such a retrofit scheme. However, the presence of variable levels of retrofit activity in other areas, the variability of the multiple factors that shape domestic energy use in different areas and the absence of comparable local authority data in other areas, led us to conclude that evaluating the impacts of the KWZ scheme by using a control group would introduce a much wider range of causal factors and significantly higher levels of statistical uncertainty into the analysis.

The paper is structured as follows. We start with a brief literature review that considers the barriers to change and the need for retrofit schemes. We then introduce the case and its context more fully, before discussing the data that we use and the broad approach that we apply. We then explore the key features of the methodology in more detail, particularly those aspects that allow us to isolate the influence of broader background trends in domestic sector energy use from the impacts of the KWZ scheme. We then present the results of the analysis and discuss their significance for our understanding of the impacts of retrofit schemes. We also consider their implications for the accuracy of the models that have been used to predict the impacts of domestic energy savings measures and the assumptions that they contain about the scale of performance gaps and rebound effects. We conclude by discussing the significance of these results for broader debates on energy saving in the domestic sector and for policies on retrofit.

2. Literature review

From a technical and economic perspective, many assessments have highlighted the presence of cost-effective opportunities to reduce energy use in buildings (IPCC, 2014). However, the IEA (2013b) and the IPCC (2014) note the significance of multiple barriers that prevent the take-up of energy efficiency measures in buildings. These include lack of awareness and concern, limited access to reliable information from trusted sources, fears about risk, disruption and other 'transaction costs', concerns about up-front costs and inadequate access to suitably priced finance, a lack of confidence in suppliers and technologies and the presence of split incentives between landlords and tenants (IEA, 2013b; IPCC, 2014; Long et al., 2014; Owen et al., 2014). The widespread

presence of these barriers led the IEA (2013a) to predict that without a concerted push from policy, two-thirds of the economically viable potential to improve energy efficiency will remain unexploited by 2035. These barriers therefore represent a classic market failure and a basis for government intervention.

While these assessments focus on the technical, financial or economic barriers preventing the take-up of energy efficiency options in buildings, others emphasise the significance of the often deeply embedded social practices that shape energy use in buildings (c.f. Spaargaren, 2011; Judson and Maller, 2014; Viasova and Gram-Hanssen, 2014; Bartiaux et al., 2014). These analyses focus not on the preferences and rationalities that might shape individual behaviours, but on the 'entangled' cultural practices, norms, values and routines that underpin domestic energy use (Barr et al., 2011; Ozaki and Shaw, 2014). As Shove (2010) argues, focusing on the practice-related aspects of consumption generates very different conceptual framings and policy prescriptions than those that emerge from more traditional or mainstream perspectives. But the underlying case for government intervention to help to promote retrofit and the diffusion of more energy efficient practices is still apparent, even though the forms of intervention advocated are often very different to those that emerge from a more technical or economic perspective.

Based on the recognition of the multiple barriers to change, and the social, economic and environmental benefits that could be realized if they were overcome, government support for retrofit has been widespread. Retrofit programmes have been supported and adopted in diverse forms in many settings, and their ability to recruit householders and then to impact on their energy use has been discussed quite extensively (c.f. Dowson et al., 2012; Hoicka et al., 2014; Achtnicht and Madlener, 2014). Frequently, these discussions have criticised the extent to which retrofit schemes rely on incentives and the provision of new technologies to change behaviour whilst ignoring the many other factors that might limit either participation in the schemes or their impact on the behaviours and practices that shape domestic energy use. These factors are obviously central to the success of retrofit schemes, but evaluations of different schemes have found that they can still have significant impacts.

Various studies (c.f. Clinch and Healy, 2001; Chapman et al., 2008; Grimes et al., 2011; Tovar, 2012; Rosenow and Galvin, 2013) have found that retrofit schemes have significant impacts on domestic energy use and that the direct economic benefits from reduced energy bills significantly outweigh the direct economic costs of the schemes. They also frequently find that retrofit schemes can generate very substantial indirect benefits relating to reductions in fuel poverty and improvements in public health -although this raises the prospect of trade-offs between fuel-poverty related goals and energy security and carbon reduction related goals. However, various studies (c.f. Milne and Boardman, 2000; Sorrell, 2007; Sorrell et al., 2009; Chitnis et al., 2013) also find that the scale of the savings and benefits generated are reduced through performance gaps (i.e. the difference between the technical and actual savings that could be generated by a measure) and direct rebound effects (i.e. increased demand for cheaper energy services).

Sanders and Phillipson (2006) suggest that the best estimate of the gap between the technical potential and the actual in-situ performance of energy efficiency measures is 50%, with 35% coming from performance gaps and 15% coming from 'comfort taking' or direct rebound effects. Sorrell (2007) suggests that the direct rebound effect for energy efficiency measures related to household heating is likely to be less than 30% while Chitnis et al. (2013) suggest that rebound effects for various domestic energy efficiency measures vary from 5 to 15% and arise mostly from indirect rebound effects (i.e. where savings from energy efficiency

lead to increased demand for other goods and services). Other analyses also note that the gap between technical potential and actual performance is likely to vary by measure, with the range extending from 0% for measures such as solar water heating to 50% for measures such as improved heating controls (DECC, 2012). And others note that levels of comfort taking are likely to vary according to the levels of consumption and fuel poverty in the sample of homes where insulation is installed, with the range extending from 30% when considering homes across all income groups to around 60% when considering only lower income homes (Milne and Boardman, 2000). The scale of these gaps is significant because it materially affects the impacts of retrofit schemes, and expectations and perceptions of these impacts go on to influence levels of political, financial and public support for retrofit schemes.

In summary then, the background literature on retrofit highlights the presence of multiple barriers to change and the need for government support if these are to be overcome. Although much has been written on the extent to which different forms of support enable the wider take-up of domestic energy efficiency measures, behaviours and practises, various areas of contestation remain and there is still an absence of robust ex post evidence on the extent to which retrofit schemes actually do lead to the social, economic and environmental benefits that are widely claimed.

3. Context for the study

Such questions about the performance of domestic sector retrofit are significant in many settings, but they are particularly important in the UK. Most of the 26 million residential buildings in the UK were constructed before 1980 with relatively low levels of energy efficiency, and typically less than 2% of the housing stock is added each year (Sweatman and Managan, 2010). Despite the presence of background trends that from 1991 to 2011 reduced total energy use from the UK domestic sector by 13%, per household energy use by 25% and per household carbon emissions by 37% (DECC, 2013a), domestic energy use still accounts for around 25% of UK carbon emissions, with over 80% of this energy use coming from space and water heating (Palmer and Cooper, 2012; DECC, 2013a). As a result, the retrofitting of much of the existing housing stock is required if rates of reduction in domestic energy use are to be sustained and accelerated. However, recent rates of installation of loft and cavity wall insulations are unlikely to meet current targets, and a substantial increase in the rate of their adoption is required (CCC, 2014).

Kirklees is a metropolitan area with a population of 422,000 situated between Manchester and Leeds in the north of England. It incorporates Huddersfield, which with a population of 162,000 is the 11th largest town in the United Kingdom, as well as a number of other areas that are semi-rural in character. As with many such towns in the UK, Kirklees has a highly diverse housing stock with a high proportion of older and less energy efficient housing. The Kirklees Warm Zone (KWZ) scheme, that ran from 2007 to 2010 with a budget of £21 m, was initiated and coordinated by the local authority (Kirklees Council) in an attempt to improve energy efficiency in the domestic sector within the area. The scheme was managed by a not-for-profit local energy company (Yorkshire Energy Services) with insulation installed by the private sector. It offered free energy assessments and surveys and, where technically feasible, free loft and cavity wall insulation to all households in the area. Of the 176,000 households in the area, 134,000 had a preliminary (doorstep) assessment, 111,000 of which went on to have a fuller survey and 51,000 households had measures installed. A total of 64,000 measures were installed, including insulation in 43,000 lofts and 21,000 cavity walls. Levels of take-up were even across lower, middle and upper income areas. As

related research on KWZ has demonstrated (Long et al., 2014), this level of participation and take-up was only secured through sustained marketing and repeated household visits from a trusted provider that placed great emphasis on customer care and the quality of installations. It also relied on the provision of insulation measures at no cost with steps (such as assisted loft clearances) taken to limit disruption in participating households.

The KWZ scheme makes a good case study for a large-number, ex post analysis of the actual impacts of retrofit for two main reasons. The first relates to the scale and the geographical and temporal concentration of the associated retrofit activities. With 28.9% of the households within the Kirklees area receiving insulation measures between 2007 and 2010, the impacts on actual energy use should be more discernible using publically available data on household energy use than they would be for smaller schemes or for schemes with activities that were more diffuse geographically or temporally. The second reason relates to the data on KWZ activity that was collected by the local authority. As is discussed in more detail below, this anonymised data set includes information both on insulation measures that were installed, on the buildings that they were installed in and on various socioeconomic characteristics of the occupants. Although the KWZ scheme therefore presented a good case for investigation, some significant methodological challenges had to be overcome before the impacts of the KWZ scheme could be identified. We discuss these below.

4. Data and methods

Ideally, any ex post study on the actual impacts of retrofit would have access to high-resolution data on household energy use not only in the study area but also in a directly comparable area that could be used as a control group. Household level energy use data is not publically available in the UK, but it is available at the local level both for sub-areas that are known as 'medium level super output areas' (MLSOAs) that include between 2000 and 6000 households. Within these sub-areas, data is also available for even more local 'lower level super output areas' (LLSOAs) that include between 400 and 1200 households (DECC, 2013b). Whilst assessing the impacts of retrofit schemes on energy use at the more local scale would have added greater resolution to the evaluation, the publication of energy use data at this level only started in 2008, meaning that the data set was not suitable for assessing the impacts of a scheme that started in 2007. Household energy use data is therefore analysed for 58 MLSOAs - hereafter referred to as sub-areas-that include 49,000 households within the Kirklees area from 2007 (pre-scheme) to 2011 (post-scheme).1 In essence this means that the analysis considers the impact in 2011 of all influences on household energy use, including the measures installed through the KWZ scheme, in Kirklees in the period from 2007 to 2011.

We combine this energy use data with the anonymised data supplied by the local authority responsible for the KWZ scheme mentioned above. This data set includes information on the physical characteristics of the houses that participated in the scheme in any way and on the levels and forms of insulation that were installed in each of the 49,000 participating households. This data set includes information on the sizes (number of bedrooms), types (detached, semi-detached, end of terrace, bungalow, flat), approximate age and build type and location of each participating

1 One other MLSOA on the border of the Kirklees area is particularly large. Preliminary statistical tests showed it to be an outlier that had a significant influence on the wider results; we therefore removed it from the analysis.

property. It also includes data on the presence of lofts and wall cavities in each property, on the levels of insulation before and after participation in the scheme, the dates when insulation measures were installed and a range of socio-economic data covering income and ownership.

Our methodology then involves five key stages:

• First, we develop aggregated indicators of the forms and levels of retrofit scheme insulation activity in each sub-area within the broader study area. This allows us to compare levels of retrofit activity in different areas on a like-for-like basis.

• Second, we develop normalised measures of average household energy demand in each sub-area that control for the influence of factors such as population change and weather variations. This allows us to compare levels of energy use at different times on a consistent basis.

• Third, we identify the relationship between different levels of retrofit insulation activity and the normalised indicators of household energy demand in each sub-area. This allows us to identify the changes in household energy demand that are commensurate with different levels (including a zero level) of retrofit insulation activity and then to isolate the influence of retrofit activity from the influence of background trends.

• Fourth, based on the above, we compare the predicted and actual impacts of retrofit activity, enabling us to comment on the accuracy of predictive methodologies and the assumptions that they are based on, including those relating to performance gaps and rebound effects.

• Fifth, we consider any variations in the impact of retrofit in lower, middle and upper income areas.

Further details on the way we applied each stage of the methodology to the KWZ case are given below.

• Stage 1-Developing aggregated indicators of retrofit insulation activity

Using the detailed data about the insulation measures installed through the KWZ scheme, we develop aggregated indicators of the levels of insulation installed in 58 of the sub-areas that make up the Kirklees area. As different levels and forms of insulation were installed in different types of houses in each sub-area, as a common metric we use the predicted impact of the different levels of insulation installed in each sub-area on energy use. We generate this common metric using two different predictive methodologies or models.

The first method to predict energy savings from installed insulation (which we term PM1) - was developed by the Buildings Research Establishment for the UK Committee on Climate Change. It models the impact of different insulation improvements using a standard 'average' UK house. This model assumes that all homes have been upgraded to a good standard of energy efficiency before predicting the extra contribution of different levels of loft and cavity wall insulation. This model thus generates what are currently conservative predictions of energy savings reflecting the minimum savings that are likely, given that many houses are not yet at this level of efficiency. In this model, the impacts of loft insulation are calculated based on different pre-existing depths of insulation (0 mm, 50 mm, 100 mm etc.) which in the KWZ case are topped up to a depth of 300 mm. A standard saving is used for every house with added cavity wall insulation. The total estimated energy saving is then adjusted by 56.2% (41.2% to take into account of defective fitting of insulation and performance gaps and 15% 'comfort taking' (improved heating of the house by its occupants)) (BRE, 2008). In other words, this model assumes that home energy consumption will achieve 43.8% of the full technical potential for

these measures. This model was developed to generate readily defendable figures even in the context of significant uncertainties about the gaps between the technical and realistic performance of energy efficiency measures (Jaffe and Stavins, 1994). The model has been used to underpin various assessments of the economic potential to reduce the energy use and carbon footprints of different cities within the UK (see Gouldson et al., 2012).

The second predictive method (which we term PM2) was developed by the Energy Saving Trust for the UK Department of Environment, Food and Rural Affairs. This model makes assumptions for the impact of insulation for a wide range of different house types (e.g. terraced, detached) and sizes (based on the number of bedrooms). It also estimates the impact of different levels of loft and cavity wall insulation. Predictions are made for only two levels of pre-existing loft insulation (less or more than 60 mm). Based upon research by the Energy Saving Trust (2004), the model assumes that 15% of the technical potential of insulation is lost through poor installation, and 35% is lost due to comfort taking. This means that the model assumption is that insulation measures only realise 50% of the energy savings that technically they might achieve. This approach was important in UK policy as it was approved by the Office of Gas and Electricity Markets (OFGEM, the main UK energy regulator) to inform the payments made to energy companies for the energy savings that they delivered under the UK Carbon Emissions Reduction Target (CERT).

Using PM1 and PM2, we then calculate the predicted energy saving for each model for each house participating in KWZ. To preserve anonymity, and as we only have energy data at sub-area level, we then aggregate the predicted energy saving into a PM1 and PM2 predicted insulation score for each sub-area.

• Stage 2 - Developing normalised measures of household energy demand

We then develop a normalised measure of actual household energy demand in each sub-area that enables us to compare domestic energy use in each sub-area before (i.e. in 2007) and after (i. e. in 2011) the KWZ scheme on a consistent or like-for-like basis. The base data on gas consumption used in the study was already adjusted to take account of the impacts of weather variations on household energy consumption (see DECC, 2014). However, the base data on heating-related electricity consumption was not adjusted to take account of weather variations. We used degree day data to adjust the heating related electricity consumption data, in order to make it suitable to be used alongside the weather adjusted gas consumption data. Degree day data is routinely used as a reliable predictor of heating demand, with an increase in heating degree days leading to an increase in energy usage for space heating. Regularly published degree day data counts the number of hours during normal heating times (mornings and evenings) when the external temperature falls below a set heating point, normally 15.5 °C. These hours are then aggregated into degree days. We use figures for the East Pennine area that includes Kirklees for 2007 and 2011. With a base of 100 heating degree days in 2007, the colder weather in Kirklees in 2011 meant that it had an indexed number of degree days of 106.9. As this suggests that space heating demands would be 6.9% higher in 2011 than 2007, we reduce observed heating-related electricity consumption in 2011 by 6.9%. By combining this with the weather adjusted figure for gas consumption we generate a single weather adjusted figure for heating related energy usage.

We then further control for the influence of changes in the number of households in the study area that can be expected to have influenced aggregate levels of household energy demand over the study period. Overall, there was a 1.2% rise in the number of metered households in Kirklees between 2007 and 2011, with

some sub-areas seeing increases in the number of metered households and others seeing decreases. To remove the influence of these changes, we adjust 2011 level energy consumption data for each sub-area in proportion to the change in the number of households in that MLSOA in the period from 2007 to 2011.

• Stages 3 and 4 - Isolating the impacts of retrofit insulation activity and comparing predicted and actual impacts

We then attempt to separate the influence of all other factors shaping household energy demand from the influence of the insulation installed through the KWZ scheme through statistical analysis. Our aim is to identify the average background reduction in energy use that is consistent with a zero level of KWZ retrofit activity, and to use this to generate estimates of the reductions in household energy use that can be attributed to KWZ retrofit activity.

The start of our approach above is expressed in Eq. (1):

-En + Nn *b = - m*Pn

m*Pn -Nn *b

where E is the observed reduction in household energy use in each of n = 1-58 MLSOAs in the period from 2007 to 2011 (for all reasons, including KWZ, adjusted for variations in population and weather), m is the gradient of the line of best fit when the results of this are plotted graphically, P is the predicted energy reduction in each sub-area due to different levels of KWZ activity, N is the number of households in each sub-area and b is an average background (non-KWZ related) reduction in energy use per household in each sub-area. If we can determine the value of b, we can then remove all non-KWZ related energy saving and thereby isolate the impacts of KWZ activity.

To estimate the value of b, we rewrite Eq. (1) slightly to generate a measure of reduction in energy use per household (- E/N) for each sub-area, as expressed in Eq. (2):

-En/Nn = - m*Pn/Nn - b

Plotting the reduction in energy use per household (E/N) against predicted energy savings per household in each sub-area as a result of different levels of KWZ activity (P/N), we generate a line of best fit. The intercept (b) is a level of reduction in energy use that is commensurate with a zero level of KWZ activity. By identifying a value for b, we are able to estimate of the average level of background energy saving per household that is independent of KWZ. Using this value for b in Eq. (1), we can then also identify the levels of energy saving per household that can be attributed to KWZ.

If we rewrite Eq. (1), we can then reveal the influence of the KWZ scheme after taking into account the influence of all other (non-KWZ related) background trends in household energy use. This is expressed in Eq. (3):

Plotting the results of this analysis graphically should reveal a line of best fit that passes through the origin (showing that we have stripped out the impact of background trends) but with a slope that reflects the relationship between the predicted and actual impacts of KWZ. Where predicted impacts are equal to actual reductions in energy use after background trends, the slope of the line is - 1. Where actual impacts are less than predicted, the slope of the line is flatter. Where actual impacts are greater than predicted, the slope of the line is steeper.

• Stage 5 - Comparing impacts in lower, middle and upper income areas.

We then examine whether the impacts of retrofit activity vary according to the income of participating households in each sub-area. Using data on the declared income of participating households collected by the local authority, we divide the 58 sub-areas into lower, middle and upper income brackets, with each tercile containing 19 or 20 sub-areas. We then examine whether there are any statistically significant differences in the impacts of KWZ in each tercile. With a larger number of data points we could have explored this issue in finer resolution.

5. Results

Using the approach outlined above, we compare the actual reductions in energy use in each sub-area that are the result of all influences, with the predicted reductions in energy use that are the result of KWZ activity. We do this using the predictions of PM1 and PM2. The results suggest an average reduction in per household energy use - commensurate with a zero level of KWZ activity - of 2185 kWh per household using PM1 and 2175 kWh using PM2. Given their proximity, for the remainder of the analysis we assume that over the 2007 to 2011 period the average background reduction in weather adjusted, per household energy use, independent of any KWZ influence, was 2180 kWh.

We then use this average per household background trend in our correlations between actual and predicted reductions in energy use using the two models for predicting energy use PM1 and PM2. We present the results in Fig. 1 for PM1 and Fig. 2 for PM2. For PM1, the results of our correlation between the observed reductions in average household energy use in each sub-area (after adjusting for changing populations, weather variations and average non-KWZ related background trends in household energy use) suggest a line of best fit with a slope of -1.7. Similarly, for PM2, the results suggest a line of best fit with a slope of - 1.2.

The results suggest that the predictive models underpinning

S« -2 2 =

« £ -3

§ I 3

■c " .

u D -4 = -a

¡■S -6 J

Predicted impact of different levels of insulation installed by the KWZ scheme in ^ each MLSOA (M kwh)

♦ ♦

y = -1.7309x - 0.2172 R2 = 0.4094

Fig. 1. Correlation between predicted and actual impacts of different levels of insulation installed by the KWZ scheme in each MLSOA using PM1.

Predicted impact of different levels of insulation installed by the KWZ scheme in each MLSOA (M kwh)

y = -1.2366x - 0.2213 R2 = 0.5301

Fig. 2. Correlation between predicted and actual impacts of different levels of insulation installed by the KWZ scheme in each MLSOA using PM2.

PM1 and PM2 both under-estimate the actual savings from domestic insulation measures, with PM1 being more conservative than PM2. PM1 assumes that 44% of the full technical energy saving potential of insulation would be realized in practice, with the gap between technical potential and observed performance determined by performance gaps and rebound effects. However, our central estimate is that 1.73 times this level of predicted saving is actually realised, which suggests that 76% of the full technical potential as assumed by this model is actually realized in practice. PM2 by comparison assumes that 50% of the technical potential of insulation measures is actually realized, but again our central estimate is that we see 1.24 times this level of predicted saving, suggesting that 62% of the full technical potential as assumed by the model is actually realized in practice. Both predictive models therefore seem to be too conservative in their estimates.

We also analyse whether our findings vary according to the

average household income of participants in each sub-area. To do this, we compare the relationships between the actual and predicted impacts of the KWZ scheme for the lower, middle and higher income sub-area terciles. Statistical tests show that the results for the lower income sub-areas are significantly different from those of the middle and higher income sub-areas at a 95% confidence level.

We find that the average per household energy savings that can be attributed to the KWZ scheme are 1.2 times the levels predicted by PM1 in the lowest income tercile, and 1.9 and 2.1 times the predicted levels in the middle and highest income terciles. This indicates that 53% of the technical potential of insulation is realized in the lowest income areas, but that 85-93% of the technical potential is realized in the middle and highest income areas. Similarly, we find that these savings are 0.97 times the levels predicted by PM2 in the lowest income tercile, and 1.4 times the levels predicted in both the middle and highest income terciles. These findings suggest that 49% of the technical energy savings potential of the insulation is secured in the lowest income areas, but that 70-71% of the technical potential is realized in the middle and higher income sub-areas. These results indicate that PM2 (which assumes that 50% of the technical energy savings potential of insulation is actually realised) closely predicts the actual energy savings realized through insulation in poorer areas, but that it is too conservative in its predictions of savings in middle or higher income areas. These findings are consistent with other research that has suggested losses of 50% in lower income areas (Milne and Boardman, 2000), and with research that has predicted losses of up to 20% in previously well-heated homes (Sorrell, 2007).

We also calculate the financial savings and reductions in carbon emissions that can be attributed to the KWZ scheme, with results presented in Table 1 and in Fig. 3. When combined with changing energy prices in the 2007-2011 period, they suggest that for its initial investment of £21 m, the KWZ has generated reductions in energy bills totalling £6.2 m a year at 2011 energy prices. This is equivalent to an average annual saving of £125per year at 2011 energy prices for each participating household where insulation was installed, which represents a saving on the total average

Table 1

Summary of actual and predicted impacts of the KWZ scheme on household energy use, energy bills and carbon emissions in Kirklees. Sources: DECC (2011, 2013c).

Baseline figures

Total domestic en- Total domestic energy use ergy bill

M kWh/yr degree day £m/yr corrected

Total domestic carbon Average AH en-emissions ergy use

KTCO2/yr kWh/yr

Average AH energy bill

£ /yr

Average AH carbon emissions

kgC02/yr

2007 (space & water) 3102

2007 (all energy) 3709

2011 (space & water) 2575

2011 (all energy) 3166

110 178 121 208

598 913 492 756

17.637 21,084

14.638 17,999

625 1013 688 1182

3400 5192 2797 4298

Estimated Savings

Total energy savings

M kWh/yr

Total financial savings

£M/yr

Total carbon savings

KTCO2/yr

Average PH energy savings

kWh/yr

Average PH financial savings

Average PH carbon savings

kgC02/yr

Actual estimated KWZ 131.4/131.1

savings 4.2%

Predicted 75.9

Savings PM1 2.4%

Predicted 106

Savings PM2 3.4%

25.1 3.3% 14.5 1.9% 20.3 2.7%

125.0 10.6% 72.3 6.1% 101 8.5%

507.9 11.8% 293.7 6.8% 410.3 9.6%

AH - all households (n = 175,904) PH-participating households (n=49,436).

Energy mix for domestic heating: 95% gas, 4% electricity and 1% oil.

Retail price of gas: 3.33 p/kWh in 2007 and 4.42 p/kWh in 2011.

Retail price of electricity: 11.16 p/kWh in 2007,14.81 p/kWh in 2011.

Carbon intensity of gas: 0.184 kgCO2/kWh in 2007 and 2011.

Carbon intensity of electricity: 0.52 kgCO2/kWh in 2007, 0.45 kgCO2/kWh in 2011.

Lowest income tercile

Middle income tercile

High income tercile

■ Average fuel bill »Average saving

Fig. 3. Average 2011 fuel bills and energy savings for lower, middle and upper income areas.

household energy bill of 10.6%. Similarly, when combined with changes in the carbon intensity of energy supplied to households through the UK's national grid, they suggest a total carbon saving of 25.1 ktCO2/year and an average saving of 508 kg CO2/year for each participating household. This is equivalent to a 11.8% reduction against 2011 levels in the average carbon emitted as a consequence of the total energy consumed within households.

Our results also offer an insight into trends in household energy use for space heating in the 58 sub-areas studied in Kirklees from 2007 to 2011. At an aggregate level, total household gas and electricity use in Kirklees in 2007 was 3709 M kWh(GWh) and in 2011 was 3166 M kWh, after being corrected for weather variations. Of this, the space and water heating component fell from 3102 to 2575 - a drop of 527 M kWh. Using our results, we can identify a reduction of 131 M kWh (4.2% of 2007 space and water heating use) in demand for domestic energy use related to space and water heating that can be attributed to KWZ and a further 383 M kWh (12.3% of 2007 space and water heating use) that is independent of KWZ and that can therefore be attributed to background trends.2

When spread across all 176,000 households in the Kirklees area, the KWZ savings realized in the 49,000 households included in the study are equivalent to an average per household reduction of 746 kWh. However, for the households in the study area that had insulation measures installed through the KWZ scheme, this amounts to an average per household KWZ reduction in energy use of 2655 kWh over the 2007-2011 period, which is larger than the reductions in energy use attributed to the background trend of 2177kWh.

6. Discussion

The results of our evaluation indicate that the KWZ scheme has had a greater effect on domestic energy use than key predictive methodologies would have forecast. Although previous research (c.f. Sanders and Phillipson, 2006) has found and some predictive models have assumed that 50% or more of the technical energy savings potential of retrofit activities is lost due to the combined effects of performance gaps and rebound effects, our central estimates are that across all areas total losses are 38% or less. Our results also indicate that losses of energy savings potential are greater in lower income areas, where we estimate that actual losses are close to those predicted at slightly less than 50%, but

that in middle and upper income areas losses may be 30% or lower. Our results therefore indicate while predictions of the impacts of retrofit activities in lower income areas where there is likely to be greater fuel poverty and thus greater comfort taking from extra insulation are quite accurate, they may be under estimating impacts in middle and upper income areas.

In other words, our results suggest that the KWZ scheme has been at least as effective in reducing energy use (and in turn we assume in combating fuel poverty) in lower income areas as predicted, for example, by Milne and Boardman (2000). But the results also suggest that the KWZ scheme has been more effective in reducing energy use and carbon emissions in middle and upper income areas than would have been predicted based on previous research. This is because the combined impacts of performance gaps and rebound effects are lower than predicted - suggesting in turn that households in middle and upper income areas were already adequately heated and that the insulation added led to reduced energy consumption rather than improved comfort levels. This finding is reinforced by the results of a related survey of the impacts of participation in the scheme that found that thermostat settings in participating households were largely unaffected by KWZ insulation3 (see Long et al., 2014). Our results therefore offer some evidence to underpin the social case for retrofit based on its expected contribution to tackling fuel poverty. They also offer evidence which suggests that both the economic and the carbon case for retrofit are much stronger than has been predicted; if the KWZ scheme is representative, then we can say that retrofits are a more effective way of reducing energy use, saving money and cutting carbon emissions than has thus far been assumed.

The economic case for retrofit seems to be particularly compelling. With costs of £21 m and annual savings of £6.2 m, the direct benefits of retrofit can be expected to outweigh the costs in around 3.4 years. But retrofit schemes continue to generate benefits for much longer - OFGEM (2015) predicts that the measures such as loft and cavity wall insulation have a functional lifetime of over 40 years. However, allowing for the possibility of households with installed measures changing use or falling out of use we calculate the direct savings of these measures over a 25 year period as being in the range of £148-218 million.4 Aside from the direct benefits, we can also expect significant indirect benefits from reductions in fuel poverty, stimulation of the local economy and of course ultimately from reduced carbon emissions. Other econometric research has estimated that the direct and indirect economic benefits of the KWZ scheme were £39 m over 5 years and that it generated 126 jobs directly and 117 jobs indirectly (Butterworth et al., 2011). Research on the health related impacts of the scheme has estimated that it generated health benefits of £4.9 m, primarily in quality of life improvements (Liddell et al., 2011). These results broadly support the findings of previous research on the economic case for retrofit (see Clinch and Healy, 2001; Chapman et al., 2008; Tovar, 2012; Galvin and Sunikka-Blank, 2013).

Whilst these findings are positive, it is interesting to compare the impacts of a large-scale retrofit programme with those of background trends in domestic energy use. As noted, in the 20072011 period, we estimate that background trends generated a

2 The difference between our estimates of the combined impacts of KWZ activities and background trends (514 M kWh) and the actual change in the measured consumption of energy for space heating in the study area (527 M kWh) is within the margin of error for the analysis.

3 This survey found that 68% of participants in the scheme said their thermostat had stayed at the same level before and after participation, 29% said they had set it to at lower level after participation and only 4% said they had turned it up.

4 These undiscounted figures are based on DECC's forecasts of future energy prices - with the lower figure using the 'low' energy price forecast and the upper the 'high' energy price forecast (DECC, 2013c). Applying a standard UK Treasury social discount rate of 3.5% would reduce the estimated savings by a factor of 0.423 over 25 years. The estimates do not take into account any possible impacts of climate change on energy demand over the 25 year period.

« 400

12.3% drop in domestic space and water heating energy use within the study area. As stated above, these reductions can be attributed to a range of factors, including the gradual upgrading of the housing stock, the steady replacement of older and less efficient space heating technologies, the impacts of various government energy efficiency policies and behavioural responses to increases in energy prices and changing economic conditions (including those that drive increases in fuel poverty). In comparison, the KWZ scheme, that offered free insulation to householders and that led to 29% of households having insulation installed, led to a 4.2% drop in domestic energy use across all households in the area. At the area-wide level, the influence of background trends therefore seems to be much greater than the influence of even a large-scale retrofit scheme. However, at the householder level the KWZ delivered a saving of 14.8%, which is comparable to 5 years of average background energy reductions. If it were possible to achieve higher participation levels, this demonstrates that retrofit schemes have the potential to exceed current trends in reductions in domestic space heating energy use.

For the future, we clearly need to understand the causes and consequences of background trends in domestic energy use in as much detail as we do the impacts of retrofit schemes. We also need to better understand the factors that drive greater participation in retrofit schemes, as the savings for participating households are significant and could also be secured by many more households. Central to this of course is a need for a much more nuanced understanding of the influence of different ways of designing, financing and delivering retrofit activities.

7. Conclusions and implications for policy

Policy debates on domestic sector retrofit frequently emphasise the scale of the opportunity to improve domestic energy efficiency, the presence of various barriers that prevent households from doing so, and the role that different forms of policy intervention can play in overcoming these barriers. However, in some contexts prominent voices in those policy debates have become sceptical about the ability of retrofit schemes - and of the measures that they seek to promote, and the agencies that are charged with implementing them-to actually deliver real reductions in fuel poverty, energy bills or carbon emissions. This is particularly evident in the UK where a flagship policy on retrofit - the Green Deal - has thus far failed to persuade large numbers of households to participate, partly due to concerns about financing arrangements and partly because of scepticism that the scheme will actually generate the savings that it claims (c.f. Harvey, 2013; Collinson, 2014).

The evaluation of the KWZ scheme presented in this paper has shown that a large scale domestic sector retrofit scheme can deliver real ex post outcomes that exceed modelled ex ante predictions, whether judged socially, economically or environmentally. The results demonstrate that impacts in lower income areas have reduced energy use and carbon emissions whilst also improving comfort levels and reducing fuel poverty to a level that is fully consistent with previous predictions. The indirect benefits of this, particularly on public health, can be expected to be significant. The results have also demonstrated that impacts in middle and upper income households are higher than has previously been predicted and that overall losses stemming from performance gaps and rebound effects have therefore been lower than has been previously assumed.

The case for further policy support for retrofit activity - whether motivated by concerns about fuel poverty, energy use or carbon emissions - therefore seems to be fully supported and in some cases significantly extended by the results of this study. The broader economic case for retrofit activity has also been

strengthened - at least in the case considered the direct benefits significantly outweigh the direct costs of the scheme, and if indirect benefits were included in the assessment then the cost-benefit case would be stronger still.

Such evidence could certainly be used to make the case for both national and local policies to promote retrofit activity. However, there are still some significant gaps in the evidence base on retrofit. We still need to understand some of the issues highlighted in this paper in greater resolution, particularly those relating to the public's willingness to participate in such schemes (even when they are free, as in the case of the KWZ scheme) and to the varied impacts of participation in different areas and across different socio-economic groups. We would hope that a finer grained quantitative analysis of the impacts of such schemes could explore the factors driving the background trends in domestic energy use that are more significant in aggregate but are arguably less heavily researched than the retrofit schemes that have been the main focus of this paper. However, we note that to enable such finer grained research, access to the high-resolution data on household energy use and on the range of energy related interventions adopted in different households of the type used in this study would be needed. Currently in the UK, data on household energy use is only available at the very local (LLSOA) level after a delay of two years and data on energy related interventions is frequently not available at all. Clearly this limits the scope for timely, detailed analysis and for the rapid appraisals that are needed to accelerate policy learning in this area.

We also note that by its nature a large-scale, ex post quantitative analysis of the impacts of a retrofit scheme such as that presented here cannot fully consider the influence of important qualitative and contextual factors, or the extent to which and the ways in which impacts might change if the scheme had been designed, financed or delivered differently. There are some good examples of previous research that has considered the influence of some of these factors (c.f. Clinch and Healy, 2001; Chapman et al., 2008; Grimes et al., 2011; Tovar, 2012; Dowson et al., 2012; Hoicka et al., 2014; Achtnicht and Madlener, 2014). However, there is still a significant need for further mixed methods and multi-level research that combines large-scale, ex post quantitative analysis of the actual outcomes of retrofit schemes with in-depth qualitative research on the contexts within which participation is secured, measures are installed and behaviours and practises are affected, not only in the period immediately after a retrofit scheme has been completed but also over a longer period of time.

Acknowledgements

The authors gratefully acknowledge the financial support for the research that led to this paper that was provided by the Ash-den Trust through the Climate Change Collaboration and the ESRC Centre for Climate Change Economics and Policy (Grant no. ES/ K006576/1).

References

Achtnicht, M., Madlener, R., 2014. Factors influencing german house owners' preferences on energy retrofits. Energy Policy 73, 254-263.

Barr, S., Gilg, A., Shaw, G., 2011. Citizens, consumers and sustainability: (re)framing environmental practice in an age of climate change. Global Environ. Change 21 (4), 1224-1233.

Bartiaux, F., Gram-Hanssen, K., Foseca, P., Ozalina, L., Christensen, T., 2014. A practice-theory approach to homeowners' energy retrofits in four European areas. Build. Res. Inform. 42 (4), 525-538.

BRE., 2008. MAC Curves for the Domestic and Non-Domestic Building Sectors: Technical Documentation, Committee on Climate Change. Accessed via http://products.ihs.com/Ohsis-SEO/989079.html.

Butterworth, N., Southernwood, J. and Dunham, C., 2011. Kirklees Warm Zone Economic Impact Assessment-Final Report, Carbon Descent, <http://www.car bondescent.org.uk/index.php) (last viewed 4.12.14).

Chapman, R., Howden-Chapman, P., Viggers, H., O'Dea, D., Kennedy, M., 2008. Retrofitting Houses with Insulation: a cost-benefit analysis of a randomized community trial. J. Epidemiol. Commun. Health 63, 271-277.

Chitnis, M., Sorrell, S., Druckman, A., Firth, S., Jackson, T., 2013. Turning lights into flights: estimating direct and indirect rebound effects for UK households. Energy Policy 55, 234-250.

Clinch, J.P., Healy, J., 2001. Cost-benefit analysis of domestic energy efficiency. Energy Policy 29, 113-124.

Collinson, P., 2014 The energy efficiency 'savings' that are just hot air. The Guardian (last accessed 19.12.14).

CCC Meeting Carbon Budgets, 2014. Progress Report to Parliament 2014. UK Committee on Climate Change (last viewed 04.12.14).

DECC, 2011. Electricity Emissions Factors to 2100. UK Department for Energy and Climate Change.

DECC, 2012. How the Green Deal will Reflect the In-situ Performance of Energy Efficiency Measures. UK Department for Energy and Climate Change (last viewed 04.12.14).

DECC, 2013a. Energy Consumption in the UK. UK Department for Energy and Climate Change (last viewed 04.12.14).

DECC, 2013b. MLSOA and LLSOA Electricity and Gas Estimates 2013a UK Department for Energy and Climate Change (last viewed 04.12.14).

DECC, 2013c. Quarterly Energy Prices UK Department for Energy and Climate Change (last viewed 04.12.14).

DECC, 2014. Overview of Weather Correction of Gas Industry Consumption Data. UK Department for Energy and Climate Change (last viewed 04.12.14).

Dowson, M., Poole, A., Harrison, D., Susman, G., 2012. Domestic UK retrofit challenge: barriers, incentives and current performance leading into the Green Deal. Energy Policy 50, 294-305.

Galvin, R., Sunikka-Blank, M., 2013. Economic viability in thermal retrofit policies: learning from ten years of experience in Germany. Energy Policy 3, 43-351.

Gouldson, A., Kerr, N., Topi, C., Dawkins, E., Kuylenstierna, J., Pearce, R., 2012. The economics of low carbon cities: a mini-stern review for the leeds city region Centre for Low Carbon Futures. <http://media.climatesmartcities.org.ccc.cdn. faelix.net/sites/default/files/Mini-Stern%20Review.pdf) (last accessed: 19.07.13).

Grimes, A., Young, C., Arnold, R., Denne, T., Howden-Chapman, P., Preval, N., Telfar-Barnard, L., 2011. Warming Up New Zealand: Impacts of the New Zealand Insulation Fund on Metered Household Energy Use, paper prepared for the Ministry of Economic Development. Accessed at http://www.med.govt.nz/sec tors-industries/energy/pdf-docs-library/energy-and-the-environment/energy-efficiency/nzif-energy-report.docx.pdf.

Harvey, F., 2013. Green deal 'unlikely to deliver promises', say experts. The Guardian (last accessed: 19.12. 2014).

Hoicka, C., Parker, P., Andrey, J., 2014. Residential Energy Efficiency Retrofits: How program design affects participation and outcomes. Energy Policy 65, 594-607.

IEA, 2008. Promoting Energy Efficiency Investments: Case Studies in the Residential Sector. International Energy Agency, Paris.

IEA, 2013a. Transition to Sustainable Buildings: Strategies and Opportunities to 2050. International Energy Agency, Paris.

IEA, 2013b. World Energy Outlook 2012 - Executive Summary. International Energy Agency, Paris.

IPCC, 2014. 5th Assessment Report on Climate Change 2014: Mitigation of Climate Change, Intergovernmental Panel on Climate Change.

Jaffe, A., Stavins, R., 1994. The energy efficiency gap: what does it mean? Energy

Policy 22 (10), 804-810.

Judson, E., Maller, C., 2014. Housing renovations and energy efficiency: insights from homeowners practices. Build. Res. Inform. 42 (4), 501-511.

Liddell, C., Morris, C., Lagdon, S., 2011. Kirklees Warmzone: The Project and Its Health Impacts - a Cost Benefit Analysis. University of Ulster (last viewed 04.12.14).

Long, T., Young, W., Webber, P., Gouldson, A., Harwatt, H., 2014. The impact of domestic retrofits on householder attitudes and behaviours. J. Environ. Plan. Manag., 1-24.

Milne, G., Boardman, B., 2000. Making cold homes warmer: the effect of energy efficiency improvements in low-income homes. a report to the energy action grants agency charitable trust. Energy Policy 28, 411-424.

OFGEM, 2015. Energy Companies Obligation (eCo): Measures Table, Office of Gas and Electricity Markets, <https://www.ofgem.gov.uk/ofgem-publications/ 83100/energycompaniesobligation-measures.pdf).

Owen, A., Mitchell, D., Gouldson, A., 2014. Unseen influence: the role of low carbon retrofit advisers and installers in the adoption and use of domestic energy technology. Energy Policy 73, 169-179.

Ozaki, R., Shaw, I., 2014. Entangled practices: governance, sustainable technologies and energy consumption. Sociology 48 (3), 590-605.

Palmer, J., Cooper, I., 2012. United Kingdom Housing Energy Fact File, Department of Energy and Climate Change. Accessed at https://www.gov.uk/government/up loads/system/uploads/attachment_data/file/201167/uk_housing_fact_file_2012. pdf.

Rosenow, J., Galvin, J., 2013. Evaluating the evaluations: evidence from energy efficiency programmes in Germany and the UK. Energy Build. 62, pp450-458.

Sanders, C., Phillipson, M., 2006. Review of Differences between Measured and Theoretical Energy Savings for Insulation Measures, Centre for Research on Indoor Climate and Health. Glasgow Caledonian University (last viewed 04.12.14).

Shove, E., 2010. Beyond the ABC: climate change policy and theories of social change. Environ. Plan. A 42, 1273-1285.

Sorrell, S., 2007. The Rebound Effect: An Assessment of the Evidence for Economy-wide Energy Savings from Improved Energy Efficiency. UK Energy Research Centre London (last accessed 04.12.14).

Sorrell, S., Dimitropoulos, J., Sommerville, M., 2009. Empirical estimates of the rebound effect: a review. Energy Policy 37, 1356-1371.

Spaargaren, G., 2011. Exploring the relevance of practice theories for the governance of sustainable consumption practices in the new world order. Global Environ. Change 21 (3), 813-822.

Sweatman, P., Managan, K., 2010. Financing Energy Efficiency Building Retrofits: International policy and business model review and Regulatory Alternatives for Spain, Climate Strategy and Partners. Accessed at http://www.climatestrategy. es/index.php?id=19.

Tovar, M., 2012. The structure of energy efficiency investment in UK households and its average monetary and environmental savings. Energy Policy 50, 723-735.

Viasova, L., Gram-Hanssen, K., 2014. Incorporating inhabitants everyday practices into domestic retrofits. Build. Res. Inform. 42 (4), 512-524.

Wingfield, J., Bell, M., Miles-Shenton, D., South, T., Lowe, R.J., 2008. Evaluating the Impact of an Enhanced Energy Performance Standard on Load-bearing Masonry Domestic Construction-final Report of the Partners in Innovation Project CI 39/ 3/663-Lessons from Stamford Brook: Understanding the gap between designed and real performance. Leeds Metropolitan University (last viewed 09.02.14).