Scholarly article on topic 'Uncertainty and sensitivity analysis of normalization factors to methodological assumptions'

Uncertainty and sensitivity analysis of normalization factors to methodological assumptions Academic research paper on "Earth and related environmental sciences"

Share paper
Academic journal
Int J Life Cycle Assess

Academic research paper on topic "Uncertainty and sensitivity analysis of normalization factors to methodological assumptions"

Int J Life Cycle Assess

DOI 10.1007/s11367-015-1013-5

I CrossMark


Uncertainty and sensitivity analysis of normalization factors to methodological assumptions

Lorenzo Benini1 • Serenella Sala1

Received: 30 June 2015 / Accepted: 3 December 2015

# The Author(s) 2015. This article is published with open access at


Purpose Normalisation is an optional step of a life cycle assessment, supporting the interpretation of the results of the characterization in terms of relative environmental relevance of the impacts. Normalisations factors (NFs) are calculated as results of regional/global inventories of emission and resources characterized through impact assessment methods. Several methodological assumptions are needed for building the inventory, as presented in Sala et al. (Int J Life Cycle Assess 20:1568-1585, 2015). NFs for EU27 have been calculated for 2010 compliant with ILCD recommendations defining a methodological approach for sources selection and the use of proxy indicators. Qualitative and quantitative uncertainty evaluation is needed for assessing the robustness of final figures. The present work aims at quantifying the influence of key methodological choices on the variability of the normalisation factors.

Materials and methods Five sources of uncertainty have been analyzed in this work: (F1) the selection of the sources of data; (F2) the classification of data as life cycle inventory (LCI) elementary flows; (F3) the classification of substances for characterization; (F4) the specification of the emission compartments and (F5) the use of spatially differentiated

Responsible editor: Jeroen Guinée

Electronic supplementary material The online version of this article (doi: 10.1007/s11367-015-1013-5) contains supplementary material, which is available to authorized users.

* Serenella Sala

1 European Commission, Joint Research Centre, Institute for

Environment and Sustainability, Sustainability Assessment Unit, Via Enrico Fermi 2749, 21027 Ispra, VA, Italy

Published online: 14 December 2015

characterization factors. The sensitivity of the normalization factors to such uncertainties were assessed through a global sensitivity method, for the impact categories acidification (ACID), terrestrial eutrophication (ET), marine eutrophication (EM), photochemical ozone formation (POF), respiratory inorganics/particulate matter (RIPM) and water depletion (WD).

Results and discussion The results demonstrate the need of thorough uncertainty and sensitivity analysis for supporting the use of NFs. Uncertainties are high for the impact categories respiratory inorganics (RIPM) and water depletion (WD) and improvement of these categories is a priority. For RIPM this is explained by the high variability amongst the characterization factors for PM2.5 and PM10, together with the contextual lack of information about the height of the emission source in the inventory. For WD this is explained by variability of the regionalized factors available within the ILCD. For ACID, ET and EM the uncertainty is relatively low and generally completely led by factors F1 and F2. However, regionalized characterization factors were not tested for ACID and ET, therefore the results might be underestimating the overall uncertainty. For what concerns POF, the main source of uncertainty—amongst those included in the analysis—is the selection of the data source. Overall, improvements in the spatial resolution of the inventory are needed in order to confine uncertainty. This would allow the use of characterization factors specific for emission source typology and geographical location. Conclusions The uncertainty associated with the methodological choices made for calculating normalization factors (Sala et al. in Int J Life Cycle Assess 20:1568-1585, 2015) was assessed. Generally, the value calculated by Sala et al. (Int J Life Cycle Assess 20:1568-1585,2015) compare well against average and median values estimated in this analysis for ACID, ET, EM and POF. Instead, the impact categories RIPM and WD show different patterns. For RIPM, although


the average value is very similar to the normalization factor reported by Sala et al. (Int J Life Cycle Assess 20:1568-1585, 2015), the median value is far lower. For what concerns WD, the median value is much higher. Future improvements of the normalization factors should therefore prioritize the development of more detailed inventories of emissions by including higher substance resolution, height of emission as well as the use of spatially differentiated characterization factors. To support the interpretation of normalized results, we recommend that the normalization factors from Sala et al. (Int J Life Cycle Assess 20:1568-1585, 2015) are applied together with two additional sets of normalization factors i.e. the 'median values' and the set of 'average+standard deviation' values, so to better capture their uncertainty. Similarly, the interpretation of the results should build on the qualitative estimates of robustness provided by Sala et al. (Int J Life Cycle Assess 20:1568-1585,2015).

Keywords Normalization factors . ILCD . Life cycle impact assessment. Domestic inventory . Sensitivity analysis ■ System boundaries . Spatial differentiation

1 Introduction

Normalization is an optional step of a life cycle assessment (LCA), supporting the interpretation of the results of the characterization in terms ofrelative environmental relevance ofthe impacts. In fact, normalization allows the LCA practitioner expressing results after characterization using a common reference impact (Laurent at al. 2011), and it may be particularly of help if results need to be communicated to decision makers in business and policy contexts. Normalization factors (NF) are calculated as results of regional/global inventories of emission and resources characterized through impact assessment methods. Several modelling choices and assumptions are needed for building the underlying inventory and this may imply high uncertainties of the final figures.

Uncertainties evaluation and sensitivity analysis in life cycle assessment, from inventory to impact assessment, are increasingly discussed in literature (e.g. Hong et al. 2010; Clavreul et al. 2012, 2013; Imbeault-Tetreault et al. 2013; Wei et al. 2015). Moreover, review articles on uncertainty have been published by various authors (Heijungs and Huijbregts 2004; Heijungs and Lenzen 2014; Lloyd and Ries 2007).

Inaccurate normalized data have been recognized as a possible source of uncertainty (Huijbregts et al. 1998; Ciroth et al. 2004). As discussed by Heijungs et al. (2007), both normalization factors and process inventories generally suffer from incompleteness due to a lack of emission data and/or characterization factors. This leads to biases which, in turn, might lead to over or underestimation of the normalized results for

some impact categories, with severe problems in using normalized scores (Heijungs et al. 2007). Indeed, moving from inventory to characterization, up to normalization and weighting, may increase uncertainty as the number of sensitivity coefficients which have to be taken into account increases as well (Hejiungs 2010). Nevertheless, the uncertainties possibly related to normalization factors are rarely discussed and tested ((e.g. Hung and Ma 2009) which included normalization in the uncertainty assessment) not even in papers focusing on normalization only (e.g. Lautier et al. 2010). To date, a quantitative uncertainty and variability assessment due to the normalization step has not yet been performed.

Sala et al. (2015) presented a methodology for an extended inventory underpinning normalization factors for Europe in 2010 where key choices on sources and flow mapping as well as methodological assumptions for building proxy indicators have been detailed. From the results, few flows dominate the results of many impact categories (e.g. NOx) and the qualitative assessment of the robustness of some impact category is low (e.g. water depletion). The present work aims at quantifying the influence of different methodological choices on the final normalization factors by conducting uncertainty and sensitivity analysis on them.

The paper is organized as follows: after presenting an overview of the sources of uncertainties, the methodological section illustrates the sensitivity analysis conducted on the inventory used for the calculation of the normalization factors for EU27 (e.g. the total of emissions and resource extractions in the year 2010), whereas in the results and discussion section the outcome and the implications on the characterized factors (i.e. normalization factors) are presented.

2 Sources of uncertainties in the calculation of normalization factors

Uncertainties in the calculation of the normalization factors in LCA may be related to different sources; these are listed in Table 1 alongside of the LCA steps.

The specific sources ofuncertainty, i.e. factors ofuncertain-ty, which were analyzed in this study are reported in Table 1 and listed below by groups:

• F1: the selection for the sources of data amongst statistical database for NOx, SOx, NH3, CO, PM2.5/PM10 and water withdrawals (from F1.1 to F1.6);

• F2: the classification of environmental statistics as ILCD elementary flows forNOx, and SOx (F2.1 and F2.2);

• F3: the classification of PM2.5 and PM10;

• F4: the assumptions made on the typology and height of emission sources for NOx, SOx, NH3, CO and PM2.5/ PM10 (F4.1, F4.2, F4.3, F4.4, F4.5);

Table 1 Possible sources of uncertainties in the normalization factors for each step of the LCA-based calculation and factors included in this analysis

LCA step Sources of uncertainty Specific sources investigated in Factors of uncertainty tested within the paper Alternatives tested Probability distribution of the First order Sobol index for each of the uncertainty factors for each impact category, referred to the total variance observed

this paper alternatives ACID ET EM POF RIPM WD

Goal and scope 1. Definition of the system boundaries not included not included NA NA NA NA NA NA

F1.1 NOx 1% 2% 3% 31% 0% 0%

F1.2 SOx 1. Sala et al. 2015 15% 0% 0% 2% 0.2% 0%

2. Selection of the sources of data (F1) selection between available F1.3 NH3 2. EMEP modeled 3. EMEP Reported all data sources have been assumed to be 35% 32% 0% 0% 0.3% 0%

F1.4 CO 4. EEA 5. EDGAR v4.2 0°% 0% 0% 67% 0% 0%

datasets F1.5 PM equally probable 0°% 0% 0% 0% 1% 0%

F1.6 water withdrawals 1. Sala et al. 2015 2. Vandecasteele et al. 2014 0% 0% 0% 0% 0% 0%

Inventory 3. Estimation and data gap-filling of missing variables and values not included not included NA NA NA NA NA NA

4. Classification of group of F2.1: NO/NO2 1. NOx is mapped as NO2 2. NOx is mapped as NO all mapping alternatives have 43% 66% 97% 0% 0% 0%

data as LCI elementary flows (F2) substances: NOx and SOx F2.2: SO/SO2/SO3 1. SOx is mapped as SO3 2. SOx is mapped as SO2 3. SOx is mapped as SO been assumed to be equally probable 5% 0% 0% 0% 0% 0%

Total due to Inventory 100% 100% 100% 100% 1% 0%

LCA step Sources of uncertainty Specific sources investigated in Factors of uncertainty tested within the paper Alternatives tested Probability distribution of the First order Sobol index for each of the uncertainty factors for each impact category, referred to the total variance observed

this paper alternatives ACID ET EM POF RIPM WD

5. Classification of substances (F3) classification of PM2.5 and PM10 F3 1. PM2.5 is characterized, PM10 not 2. PM10 is characterized, PM2.5 not all classification alternatives have been assumed to be equally probable 0% 0% 0% 0% 23% 0%

F4.1: NOx 1. emissions to air, unspecified 2. Emissions to nonurban air or from high 0% 0% 0% 0% 0% 0%

6. Specification of the emission definition of the height of F4.2: SOx all heights have 0% 0% 0% 0% 0% 0%

emission source for: NOx, SOx NH3, CO, PM2.5/PM10; F4.3: NH3 been assumed to be equally probable 0% 0% 0% 0% 0% 0%

compartments (F4) F4.4: CO stacks 3. Emissions to urban 0% 0% 0% 0% 0% 0%

F4.5: PM air close to ground 0% 0% 0% 0% 36% 0%

Characterization 7. Spatial differentiation of characterization factors (F5) use of regionalized characterization factors (water only) F5: Water Depletion 1. OECD average scarcity values are used 2. Country-specific CFs + gaps filled with OECD average values 3. Country-specific CFs + gaps filled with proxies (see Table S9) the choice of the characterization factors has been assumed equally probable 0% 0% 0% 0% 0% 100%

8. Uncertainty associated with the impact assessment models not included not included NA NA NA NA NA NA

Total due to Characterization 0% 0% 0% 0% 59% 100%

Unspecific 9. Interaction among sources of uncertainty 0% 0% 0% 0% 40% 0%

ACID acidification, ETeutrophication terrestrial, EM eutrophication marine, POFphotochemical ozone formation, RIPMrespiratory inorganics/partic-ulate matter, WD water depletion

• F5: the use of regionalized characterization factors (CFs) The uncertainty factors above refer to specific methodolog-for water depletion only. ical choices which are performed when calculating

normalization factors, and were selected because of their ex-pectedly high effect on the calculation of the normalization factors performed by Sala et al. (2015). This is either because of their effect on the elementary flows contributing the most to the normalization factors developed by Sala et al. 2015 or because of the low robustness of the underlying impact assessment models (e.g. water depletion).

For each of the factors listed above there is at least an alternative approach which can be adopted for calculating the normalization factors, in addition to the one adopted by Sala et al. (2015), being still coherent with the ILCD recommended impact assessment models. The existence of such alternatives generates what could be defined as a methodological uncertainty.

The impact categories which are affected by the uncertainty factors introduced above are the following: acidification (ACID); eutrophication terrestrial (ET); eutrophication marine (EM); photochemical ozone formation (POF); respiratory inorganics/particulate matter (RIPM); and water depletion (WD). Not all sources of uncertainty which affect the normalization factors calculated by Sala et al. (2015) were assessed in this study. We did not assess some of the general uncertainties associated with normalization factors identified in Table 1: (i) the definition of the system boundaries; (ii) the estimation of elementary flows not included international statistics (see Sala et al. 2014) and the techniques applied for filling data gaps; and (iii) the underpinning impact assessment models and related characterization factors. In order to include the latter point, separate uncertainty and sensitivity analysis should be conducted by models' developers on the underpinning models of estimation, such as those used for toxicity-related impact categories (multimedia models for characterization, emission estimating model for heavy metals and pesticide, etc.) and made available to users. However, this is not the case for the majority of impact assessment models used, and therefore it is beyond the aim of the present study. The sources of uncertainties which are included in this analysis are further detailed in the following sections. Details of calculations and underpinning data are reported as Electronic Supplementary Material and references in the following text (e.g. Table S1, Electronic Supplementary Material).

2.1 Sources of uncertainty in the inventory

2.1.1 Selection of the data sources (F1)

Sala et al. (2015) proposed a hierarchical approach to the selection of data sources. The order of preference was the following: (i) officially reported data provided by EU and international bodies (e.g. Eurostat, FAO, OECD etc.), based on agreed models, methods and standards, with documented metadata and periodical quality checks. 2. Activity-based estimations, derived as 'activity data times emission factor'; (ii)

activity data were from officially reported data, emission factors were based on scientific literature, grey literature (e.g. sectorial reports) and available life cycle inventories (LCIs); (iii) statistical proxies (time, flows) when the correlation is statistically significant; (iv) speculative assumption(s), based on cause-effect models, not statistically tested.

For what concerns NOx, NH3, SOx and CO the priority in selecting the data sources has been set as follows: UNFCCC (2013)> EMEP_modeled (EMEP/CEIP 2013b) > EMEP_reported (EMEP/CEIP 2013a) > EDGARv4.2 (EC— JRC & PBL 2011). This is coherent to decisions by a team of experts from EC-JRC, Netherlands Environmental Assessment Agency (PBL), United Nations Framework Convention on Climate Change (UNFCCC), European Monitoring and Evaluation Programme (EMEP), as reported in EC-JRC (2011b) on the basis of ECE (2010). The data sources for those substances are then: UNFCCC for CO and NOx (reported as NO2) and the EMEP/CEIP database for NH3, and SOx (reported as SO2) (EMEP/CEIP 2013b).

In Tables S1 and S2 (Electronic Supplementary Material) are reported, the data retrieved from different sources for the elementary flows NOx, NH3, SOx, CO, PM2.5 and PM10 and their relative difference between what was selected by Sala et al. (2015). In Table S3 (Electronic Supplementary Material) the statistics on water withdrawal retrieved from different sources (Sala et al. 2015 and Vandecasteele et al. 2014) are provided. Within the uncertainty and sensitivity analysis carried out in this paper, all of the data sources reported in Tables S1, S2, S3 (Electronic Supplementary Material) were given the same likelihood.

2.1.2 Classification of data as LCI elementary flows (F2) —NOx and SOx

Both nitrogen and sulphur oxides are pollutants which contribute to several impact categories of those recommended by the ILCD; SOx contributes to acidification, photochemical ozone formation and respiratory inorganics, and NOx contributes to terrestrial and marine eutrophication in addition to the ones listed for SOx. Nevertheless, the ILCD list of recommended characterization factors does not provide factors which are specific for these two groups of substances. Therefore, the attribution of a characterization factor to these groups of substances is a methodological choice to which the impact categories listed above are expected to be sensitive to. This is due to the fact that NOx and SOx are major contributors to the total EU27 figures (Sala et al. 2015) as well as to the fact that currently available statistics do not differentiate between NO2 and NO, (or sulphur oxides) (only NOx totals expressed as NO2 equivalents in mass is available), in combination to the fact that the characterization factors might differ sensibly amongst different oxides within the same group (e.g. the CF for NO can be twice as the one for NO2).

Sala et al. (2015) mapped the group of substances SOx and NOx into NO2 and SO2 for the calculation of the normalization factors. It is important to assess how sensitive is the result to the mapping of the groups of substances NOx and SOx into a specific oxide. In order to assess the uncertainty associated to this methodological choice, we assumed that the mapping of the groups of substances into a specific elementary flow would be equally possible. An additional source of uncertainty which is not included in the current assessment, is represented by the fact that for several impact categories the CFs are generally calculated and made available by models developers only for NO2 and SO2, and that the factors for NO, SO and SO3 were approximated by the EC-JRC (2011a) on the basis of molecular weight and charge. Similarly, the breakdown of these groups of substances into individual substances and the disaggregation of reported emissions into e.g. NO and NO2 are beyond the purpose of this paper.

2.2 Sources of uncertainty in the characterization

2.2.1 Classification of substances (F3)—PM25 andPM10

The model recommended by the ILCD (EC-JRC 2011a) for assessing impacts associated to respiratory inorganic substances and particulate matter (Humbert 2009) characterizes both PM2.5 and PM10. However, the characterization factor associated to PM2.5 is estimated by the model developer by assuming that all impacts originating from the exposure to PM10 are due to PM2.5 (Humbert 2009). This implies that only one of the two inventoried group of substances should then be characterized, as PM10 contains PM2.5, being the latter a fraction of the former. The selection of which of the two elementary flows between PM10 and PM2.5 to account and characterize further is therefore a clear source of methodological uncertainty. In this assessment, we assumed that both options are equally viable.

2.2.2 Specification of the emission compartments (F4)

The lack of detail regarding the specific emission compartment into which pollutants are emitted represents another source of uncertainty which could potentially affect the normalization factors.

Some of the ILCD recommended impact assessment models provide different CFs for substances emitted into air, depending on to the height of the emission source. In fact, emissions to air can be further specified into different flows: 'emissions to lower stratosphere and upper troposphere', 'emissions to non-urban air or from high stacks', 'emissions to urban air close to ground' and 'emissions to air, unspecified'. The detail regarding the height of emission sources is generally not found in country-scale statistics; therefore, Sala et al. (2015) applied generic CFs (i.e. 'emissions to air,

unspecified') in order to characterize the inventory of EU27 total emissions. For the impact categories ACID, ET, EM, and POF analyzed in this paper, such detail is irrelevant as it does not lead to differences in the CFs for any of the pollutants included in the analysis, whereas this aspect is expected to be of relevance for RIPM (see Tables S4 to S8, Electronic Supplementary Material). In fact, PM2.5 emitted to air has different CFs ranging between 3.3, 1 and 0.35 kg PM2.5eq/ kg, respectively, for emissions close to ground in urban areas, generic emissions and emissions to non-urban air or from high stacks. With the purpose of the assessment of uncertainty of the normalization factors, all the three options listed above were considered to be equally viable, given the fact that the distribution of emissions of particulate matter is unknown.

2.2.3 Spatial differentiation of characterization factors (F5)

Increasingly, life cycle impact assessment models provide spatially resolved characterization factors. This implies that the selection of generic factors vs spatially resolved ones may lead to uncertainties associated to neglecting the spatial variability. We tested these aspects for the impact category of water depletion. According to the ILCD recommendations (EC-JRC 2011a), water depletion can be quantified by either applying a generic characterization factor or by applying spatially resolved factors at country level. In fact, the method recommended in the ILCD (Frischknecht et al. 2008) provides spatially resolved CFs for 29 OECD countries. In Sala et al. 2015 the 'OECD average scarcity' factor was used for characterizing impacts associated with water scarcity. This choice was made in order to guarantee consistency with the other impact assessment methods, for which only a limited set of regionalized flows is available.

However, as reported in Table S3 (Electronic Supplementary Material), the CFs for water stress present a wide variability across the EU countries. Characterization factors are not available for a number of countries (see Table S3, Electronic Supplementary Material) and for this reason two alternative extrapolations were adopted: (i) attributing the characterization factor 'OECD average scarcity' to the missing factors and (ii) attributing characterization factors by analogy of climatic conditions. Both choices have been estimated in this analysis in order to assess the relevance thereof on the final result.

3 Uncertainty and sensitivity analysis

Uncertainty and sensitivity analysis (SA) have been conducted testing different modelling choices applied for calculating the normalization factors developed by Sala et al. 2015.

As defined by Campolongo etal. (2011 p. 978), 'uncertainty and sensitivity analyses study how the uncertainties in the

model inputs (X1, X2,..., Xk) affect the model's response Y, Y=Y (X1, X2, . . ., Xk). Uncertainty analysis quantifies the output variability while Sensitivity Analysis describes the relative importance of each input in determining this variability'. As reported by Pannell (1997), SA can be useful for a range of purposes, such as testing the robustness of the results of a model or system, reducing uncertainty and enhancing communication from modellers to decision makers. This is possible because SA helps in increasing the understanding of the relationships between input and output variables in a system or model and has been increasingly demonstrated in scientific arenas and recognized by international institutions (Campolongo et al. 2011). SA methods can be generally classified into local and global methods (Campolongo et al. 2011; Wei et al. 2015). While local sensitivity measures, also called one-at-a-time (OAT) methods, assess how uncertainty in one factor affects the model output keeping the other factors fixed to a nominal value (Campolongo et al. 2011), global sensitivity measures (GSM) assess the effects of uncertainty factors while being changed simultaneously. The advantage of global sensitivity analysis is that it is able to take into account interactions amongst factors, whereas OAT methods cannot (Campolongo et al. 2011; Wei et al. 2015). As reported by Wei and co-workers (Wei et al. 2015), in LCA, most case studies are done with OAT methods (Bala et al. 2010; Heijungs et al. 1994), whereas global methods are rarely applied (see Padey et al. 2013; Wei et al. 2015).

In order to assess the effect of the methodological choices described in Table 1 on the normalization factors, the SA has been performed by means of the Sobol global sensitivity indices (Sobol 1993, 2001). According to Saltelli et al. (2010 p.159), variance-based methods such as the Sobol indices, 'are the computer experiment equivalents of the experimental design's analysis of the variance of an experimental outcome (Archer et al. 1997). Unlike experimental design, where the effects of factors are estimated over levels, variance based methods look at the entire factors distribution, using customarily Monte Carlo methods'.

The analysis of uncertainty and sensitivity through variance based method implies the following steps:

3.1 Step 1: Definition of the uncertainty factors and their probability distribution

All the possible alternatives configurations for the sources included in Table 1 were assumed to have equal probability, simulating a situation of 'complete ignorance' relative to the plausibility of probability distribution of the input factors (e.g. although it is unlikely that all the emissions of PM2.5 reported in statistical datasets are occurring in urban areas, this has been assumed to be equally probable to the other alternatives listed in Table 1). Therefore, the probability distribution for each of the factors is a discrete distribution, with a number of points equal to the number of alternatives for that factor (see Table 1 and Electronic Supplementary Material).

3.2 Step 2: Generation of a sample set of combinations (input)

The sample set of combinations has been generated within the software environment SimLab v2.2 (Tarantola 2005), selecting the Sobol's sampling method (Sobol 1993, 2001). The Sobol's sampling method was selected as it maximizes the computational efficiency of the exploration of the space of input factors. A sample of 32,768 combinations was generated; it was assumed to be of a sufficient size considering the number of factors included in the analysis (i.e. 15). The sample set is reported in Table S10 (Electronic Supplementary Material). Each of the factors can assume a value between 1 and the number of its alternative states. For instance, factor F11 can assume values from 1 to 5, as NOx can assume 5 alternative values according to the data source selected. This information is thus translated in the next step, 'evaluation of the model's outputs', so that the configuration of factor F11 is translated into the corresponding value for emissions of NOx (see Table S1, Electronic Supplementary Material) e.g. if F1.1 = 1 therefore the value ofNOx which is reported by the first data source listed in Table S1 (Electronic Supplementary Material) (i.e. Sala et al. 2015), whereas if F1=5 the fifth data source (i.e. EDGARv4.2) is selected. The same procedure applies to all factors.

1. Definition of the set of uncertainty factors which are in input to the model, including their probability distribution;

2. Generation of a sample set of combinations of uncertainty factors, accordingly to a specific design (e.g. random, quasi-random, etc.);

3. Evaluation of the model's outputs for the generated sample;

4. Calculation of Sobol's sensitivity indices consistently with the sample design via Monte Carlo-based estimators.

The following sections details how the steps above were

developed within the current analysis.

3.3 Step 3: Evaluation of the model's outputs

The model's outputs (i.e. the normalization factors for each of the considered impact categories) were evaluated for each of the 32,768 samples generated in the previous step. This is done by converting the value of the factors into the corresponding quantities (both inventory values and characterization factors) which can be used for the calculation of the normalization factors. The calculation of the normalization factor consists of the sum of the inventory flows multiplied by the respective characterization factors. This operation is performed for each of the combinations of factors generated in

the previous step (see Table S11, Electronic Supplementary Material).

3.4 Step 4: Calculation of the Sobol's sensitivity indices

The calculation of the Sobol's sensitivity indices was performed within the software SimLabv2.2. The software allows importing the outputs of the model's calculation and supports the estimation of the Sobol's sensitivity indices. In this paper, the first order and total order sensitivity indices were calculated (see Saltelli et al. 2010; Wei et al. 2015). The first order index (Si) for factor i-th provides the estimation of the variance explained by the i-th without considering its interactions with other factors, whereas the total order index (STi) for factor i-th provide the estimation of the variance explained by factor i-th and its interaction with other factors.

4 Results and discussion

4.1 Analysis of the model's outputs

A set of statistics (arithmetic average—AVG, median—MED, minima and maxima—MIN and MAX, absolute and relative standard deviation—STD and RSTD, mean absolute deviation from the median—MAD and its relative value— RMAD) was calculated on the probability distributions of the normalization factor of each impact category. The results were compared to the normalization factors calculated by Sala et al. (2015) (see Table 2). The model's output obtained after step 3 was plotted as box plot (Tukey 1977) in the Matlab Mathworks® environment, so to compare their spread over a central value (Fig. 1). In order to compare different impact categories together in Fig. 1, each of the model's outputs was divided by the average value of the normalization factor of the respective impact category. Therefore, the value 1 in the y-axes of Fig. 1 represents the arithmetic average for each of the impact categories reported in the chart. Additionally, the median, average and the values Sala et al. (2015) were plotted in Fig. 1 as red lines, blue crosses and green asterisks, respectively. In Fig. 1, the central mark is the median, the edges of the box are the 25th and 75th percentiles and the whiskers extend to the most extreme data points not considered outliers. Points are drawn as outliers in the box plot and marked as red crosses if they are larger than q3+w(q3-q1) or smaller than q1-w(q3-q1), where q1 and q3 are the 25th and 75th percentiles, respectively, and w is set by default as 1.5.

4.2 Analysis of the sensitivity indices

The sensitivity indices of first and total order were calculated for the uncertainty factors included in the analysis, for each of

the impact categories. The detailed results are reported in Table S12 (Electronic Supplementary Material). The value of the interaction of the i-th factor with the others has been calculated by subtracting the value of the total Sobol index for factor i-th by the value of the Sobol index of first order for the same factor.

As the Sobol's indices express the share of variance explained by each of the factor, the resulting values numbers can be read also as relative percentage of variance explained. When models are non-linear, the total order indices of the factors sum up to values higher than 1 and the difference between the variance explained by first order sensitivity indices and the total order indices quantify the interaction. In Table 1 are reported the shares of variance explained by the different factors, including the overall interaction. The SimLab v2.2 software has some limitations to the accuracy of the estimation of the Sobol's indices, which, according to the authors of this paper can reach up to the value 0.05; therefore such errors should be taken into account when making comparisons.

4.3 Analysis by impact category

4.3.1 Acidification (ACID)

The uncertainty associated with the normalization factor for the impact category ACID can be considered medium to low as the maximum and minimum values observed for the distribution are roughly +30 and -20 % of the arithmetic average, respectively (see Table 2). Moreover, the arithmetic average and the median are very close one to each other and the size of the box is small i.e. the 25th and the 75th percentiles of the distribution are not too distant from the average and median values (see Fig. 1 and Table 2). The measures of dispersion relative standard deviation (RSTD) and relative mean difference from the median (RMAD) are relatively low, being 10 and 8 %, respectively (see Table 2). The normalization factor calculated by Sala et al. (2015) reported in Table 2 is roughly 12 % lower than the arithmetic average and the median (see Table 2). As reported on Fig. 2 and Table 1, uncertainty factors F1—'selection of the sources of data' and F2—'classification of data as LCI elementary flows' explain, respectively, 52 and 48 % of the variance associated with the normalization factor of the impact category acidification. In particular, the selection of the elementary flow to which to map the group of substances NOx i.e. NO or NO2 (factor F 2.1), explains the highest share of variability (43 %), whereas the selection of the data source on NH3 (F1.3) accounts for 35 % of the variance (see Fig. 3). Sala et al. (2015) defined a hierarchy for selecting data sources according to some rules, leading to the selection of the most robust and, therefore, most likely dataset values. Instead, in this analysis all data sources are assumed to be equally probable and this has the effect of inflating the

Table 2 Normalization factors (NFs) from Salaetal. 2015, measures of central tendency (arithmetic average and median), measures ofdispersion (maximum and minimum, standard deviation, mean distance from the

median, relative standard deviation and relative mean distance from the median), ratios between arithmetic average and NFs from Sala et al. (2015) and between median and NFs from Sala et al. (2015)


[mol H+ eq.] [mol N eq.] [kg N eq.] [kg NMVOC eq.] [kg PM2.5 eq.] [m3 water eq.]

NFs as from Sala et al. (2015) 2.36E+10 8.76E+10 8.44E+09 1.58E+10 1.93E+09 4.06E+10

Central tendency measures

Arithmetic average 2.66E+10 1.03E+11 9.46E+09 1.57E+10 1.93E+09 1.55E+11

Median 2.69E+10 1.08E+11 8.85E+ 09 1.57E+10 1.29E+ 09 2.07E+ 11

Dispersion measures

Maximum 3.51E+10 1.32E+11 1.09E+10 1.67E+10 6.08E+09 2.19E+11

Minimum 2.22E+10 8.70E+10 8.39E+09 1.45E+10 7.00E+08 3.84E+10

Standard deviation 2.78E+09 1.30E+10 9.79E+08 5.64E+08 1.49E+09 8.14E+10

Mean distance from the median 2.22E+09 1.06E+10 9.63E+08 3.69E+08 9.18E+08 5.94E+10

Relative standard deviation 10 % 13% 10% 4% 77 % 52 %

Mean distance from the median divided 8 % 10 % 11 % 2 % 71 % 29 %

by the median

Ratios between the parameter and

NF from Sala et al. (2015)

Arithmetic average of model's outputs 1.13 1.18 1.12 0.99 1.00 3.82

NFs as median of model's output 1.14 1.23 1.05 0.99 0.67 5.09

Factor recommended for comparison

Arithmetic average of model's output+1 2.94E+10 1.16E+11 1.04E+10 1.62E+10 3.43E+ 09 2.37E+11

standard deviation

In italics are the set of normalization factors which are recommended for use in the LCA interpretation as alternatives to the values calculated by Sala etal. (2015)

uncertainty related to the normalization factor for acidification, as it is most likely that the real value of emissions is closer to the inventory value used by Sala et al. (2015). Instead, no information is available regarding the level of

Fig. 1 Box plot of the normalization factors calculated for impact categories: ACID acidification, ET eutrophication terrestrial, EM eutrophication marine, POF photochemical ozone formation, RIPM respiratory inorganics, WD water depletion. The following statistics are shown in the chart: arithmetic average value (blue cross), median value (red line) and confidence boundaries (black lines). The green asterisks are the normalization factors calculated in Sala et al. (2015)

detail about the speciation between NO and NO2, therefore, this can be considered a source of uncertainty not currently assessed in the calculation performed by Sala and co-workers. Because of the reasons explained above, the normalization factor calculated by Sala et al. (2015) for the impact category acidification can be considered as relatively robust.

The use of regionalized characterization factors for this impact category was not investigated; therefore, the uncertainty affecting normalization factors for acidification is likely to be underestimated. Further refinements of this analysis should look at this aspect as well.

4.3.2 Eutrophication—terrestrial (ET)

The normalization factor for the impact category terrestrial eutrophication calculated in this paper is characterized by a relatively low dispersion around a central value. This can be observed in Fig. 1 as the size of the box is relatively small, and therefore, the majority of the values is not extremely dispersed. This is confirmed by the measures of dispersion RSTD and RMAD are relatively low for this distribution, being equal to 13 % of the average and 10 % of the median, respectively. The majority of the uncertainty observed for this impact category (66 %) is explained by the classification of data on NOx as elementary flows (NO or NO2), while the

Fig. 2 Share of the variance explained by each uncertainty factor (F1: F5) and by the interactions amongst them

100% 90% 80°% 70°% 60% 50% 40% 30% 20% 10% 0%



I Interaction

I F5 Regionalization

I F4 Height of the emission

■ F3 Classification of PM

F2 Classification of data as LCI elementary flows

I F1 Data sources

remaining 32 and 2 % are due the choice ofthe data source for NH3 and NOx, respectively. The normalization factor calculated by Sala et al. (2015) for this impact category is around 18 % lower than the arithmetic average (see Table 2), meaning that Sala and co-workers might have underestimated actual impact on terrestrial ecosystems due to eutrophication. However, this holds true only if all data sources are assumed to be equally likelihood to the real value. Instead, Sala et al. (2015) documented that a proper hierarchy of data sources can be established and therefore, the main uncertainty factor which remains to be addressed in Sala et al. (2015) is the translation of NOx emissions into NO and NO2. Although the difference in absolute terms is not very high, still a better refined inventory for NOx emissions would allow for better estimation of the NF.

The use of regionalized characterization factors for this impact category was not investigated; therefore the uncertainty affecting normalization factors for terrestrial eutrophication is likely to be underestimated. Further refinements of this analysis should look at this aspect as well.

4.3.3 Eutrophication—marine (EM)

Similarly to what discussed for ACID and ET, the normalization factor for marine eutrophication is sensitive to factor F2.1. In fact, the majority ofthe uncertainty observed for this impact category (97 %) is explained by the classification of data on NOx as elementary flows (NO or NO2), while the remaining 3 % by the choice ofthe data source for NOx emissions to air. The dispersion of the distribution of normalization factor values is relatively low, as RSTD and RMAD score, respectively, 10 and 11 %. The normalization factor calculated for this impact category by Sala and co-workers is very similar the arithmetic average and the median value of the distribution, differing by 12 and 5 % (see Table 2) from the former and the latter, respectively.

4.3.4 Photochemical ozone formation (POF)

According to Fig. 1, POF presents the lowest uncertainty associated to its average normalization factor (RSTD=4 %,

Fig. 3 Share ofthe variance explained by each ofthe specific uncertainty factors and by the interaction amongst them

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

interaction F5 water F45 PM F44 CO F43 NH3 F42 SOx F41 NOx F3 PM2.5/PM10 F22 SOx F21 NOx F16 water F15 PM F14 CO F13 NH3 F12 SOx F11 NOx

RMAD=2 %, see Table 2). However, not all of the relevant uncertainty sources were analyzed for the normalization factor of POF, as the only uncertainty factors input data on CO, NOx and SOx which led to some variability of the outputs (67, 32 and 2 %, respectively—see Table 1), whereas it is well known and documented in Sala et al. (2015) that non-methane volatile organic compounds significantly contribute to this impact category as well. No uncertainty is associated with the mapping of group of substances (factors F2.1 and F2.2) when using the ILCD characterization factors as the CFs for NOx and SOx are the same within the two groups of substances for this impact category for the ILCD recommended impact assessment model. However, this might vary significantly according to other models, such as in the case of Jenkin and Hayman (1999) and Derwent et al. (1998), for which NO and NO2 have an antagonistic role.

4.3.5 Respiratory inorganics/particulate matter (RIPM)

As it is possible to note from Fig. 1 and Table 2, RIPM is the impact category showing the highest uncertainty. In fact, the values of normalization factors span from a maximum 3.15 times higher than the arithmetic average, to a minimum equal to 0.36 times the arithmetic average. The median value is 67 % of the arithmetic average and the measures of dispersion are very high, 77 % for RSTD and 71 % for RMAD. Nevertheless, the normalization factor calculated by Sala et al. (2015) for this impact category is very close to the arithmetic average calculate in this analysis (see Table 2). The variability which characterizes the normalization factor of this impact category is almost entirely explained by the classification of PM2.5 or PM10 (F3) (23 %), in combination with the height of the emission source for PM (F4.5) (36 %), as well as their interaction (40 %), whereas only 1 % of the overall uncertainty is explained by the selection of the data source (see Tables 1 and S12). Other sources of uncertainty are estimated as negligible for this impact category. As described in the respective section, PM2.5 was selected by Sala et al. (2015) for calculating the normalization factors, as this choice is consistent with the assumptions underlying the impact assessment model (Humbert 2009). Additionally, such approach was recommended by the method developer himself (Humbert 2014). But no information on the height at which the emissions of the pollutants contributing to RIPM could be accessed and the characterization factor 'emission to air, unspecified' was used by Sala and co-workers. Therefore, better inventories specifying emission sources and location should be then developed in order to improve the precision and the robustness of normalization factor for respiratory inorganics. This is of paramount importance given the significant correlation between residential exposure and health impacts (Raaschou-Nielsen et al. 2013).

4.3.6 Water depletion (WD)

Water depletion shows high variability of results, with a maximum 1.4 times higher than the arithmetic average and the minimum equal to 0.25 times the average, and RSTD and RMAD being equal to 52 and 29 %, respectively (Table 2). The use of the generic factors 'OECD average scarcity' for characterizing water withdrawals calculating normalization factors as done by Sala et al. (2015) results in a normalization factor 5.4 times lower than the one obtained by applying regionally differentiated characterization factors (Table 2). The normalization factor calculated by Sala et al. (2015) for water depletion is proven to be very sensitive to the choice of regional characterization factors instead of average ones, as almost 100 % of the observed uncertainty is due to this choice (see Table 1). Therefore, better normalization factors could be calculated by making use of regionalized CFs, no matter whether the underlying statistics on water withdrawals normalization inventory is based on the data collected by Sala et al. (2014, 2015) or by Vandecasteele et al. (2014).

4.3.7 Additional sources of uncertainty not covered in the analysis

Some of the uncertainty factors listed in Table 1 were not tested within this work, e.g. uncertainty associated with the impact assessment models, although they might play a significant role. For instance:

(i) Estimation of missing characterization factors: if the values for the specific molecules composing e.g. NOx or SOx are not provided by models' developers, the factors for NO, NO2, SO, and SO2 SO3, should be approximated as done in EC-JRC (2011a), simply through molecular weight and charge. Indeed this might entail some errors. To judge whether this approximation is good enough is outside the scope of this paper, anyhow this aspect should be taken into account, together with the limitations regarding the robustness of the impact categories.

(ii) Consistency between models and inventories' system boundaries. The models underlying the characterization factors might have system boundaries which are not necessarily consistent with the data statistics collected for the inventory.

(iii) Use of marginal or average characterization factors. ILCD recommended LCIA methods might refer to marginal characterization factors or average ones (Huijbregts et al. 2011). In case marginal factors are used for characterizing the inventory values, this might lead to cross-scale inconsistencies and to the wrong quantification of the normalization factors (see e.g. for water Pfister and Bayer 2014; Benini et al. 2015).

It is important to consider that the ranges of uncertainties which are reported above are specific for the combination of the inventory values (e.g. the total of emissions and extractions in the year 2010 for EU27) and the ILCD recommended impact assessment methods which lead to the ILCD normalization factors. If other models were tested they would have probably lead to different results. For instance, the effect of NO and NO2 is modelled to be the same according to the model recommended for the impact category photochemical ozone formation within the ILCD (Van Zelm et al. 2008), having the same characterization factor. Instead, their effect is antagonistic according to e.g. Jenkin and Hayman (1999) and Derwent et al. (1998). Therefore, the conclusions derived in this work should be correctly interpreted and not extrapolated to different contexts than the one of the ILCD recommended impact categories.

Beyond the quantitative uncertainty sources estimated in this paper, a set of quality issues (e.g. robustness and completeness of the inventory and underlying estimation techniques) can potentially undermine the robustness ofthe results and questioning its applicability in decision making. Hence, qualifying the robustness of the output is essential for the use and communication of the results, especially in the science-policy interface (Maxim and van der Sluijs 2011). In fact, as suggested by Functowicz and Ravetz (1990), both estimations of uncertainty (qualitative and quantitative) should be jointly considered when using a model output. In fact, quantitative assessment of uncertainty and sensitivity cannot substitute 'sensitivity auditing' (Saltelli and Funtowicz 2014) as the former refer to probabilistic assessments which account only for factors' uncertainties for those factors already included within a model, whereas the latter points towards systematic biases of the analysis, such as the lack ofthe inclusion of some relevant flows or mechanisms, or the limited robustness and quality of impact assessment models. Therefore, Sala et al. (2015) applied a qualitative assessment of the normalization factors to this purpose. This allows making explicit and systematically reflecting upon various dimensions of uncertainty (van der Sluijs et al. 2005).

5 Conclusions

Various methodological uncertainties may affect estimated normalization factors. The results of this paper show that for some impact categories methodological uncertainties are high, although they can be reduced by further refinements at the level of the inventory data and at the level of impact assessment models. The uncertainty associated with the impact categories RIPM and WD, is the highest and certainly represents a priority for future improvement ofthe ILCD-compliant normalization factors. The other impact categories, ACID, ET and EM, are characterized by relatively low uncertainty. The

only exception is POF, which is characterized by very low uncertainty values, most probably because ofthe fact that only a part of the uncertainty factors which characterize its calculation is taken into account.

A number of improvements in the inventory would be needed in order to reduce methodological uncertainty Improving the resolution of the inventory by adding information on the emission sources (i.e. breakdown into specific substances, specific height and geographical location of the emission or withdrawal) would allow for using more specific factors for calculating ILCD-recommended normalization factors therefore improving their robustness. From an operational perspective, it is recommendable that the uncertainty figures calculated in this paper are taken into account by practitioners using the normalization factors calculated by Sala et al. (2015), especially when interpreting the results of their LCA studies. Therefore, as a complement to the normalization factors provided by Sala et al. (2015) we recommend to test, for all impact categories assessed in this paper, at least the median value and the arithmetic average+1 standard deviation (see Table 2, in bold). As a conclusive remark, we warmly recommend the joint use of quantitative uncertainty estimates when applying the normalization factors calculated by Sala et al. (2015) and qualitative assessments ofthe robustness ofthe factors estimated by Sala et al. (2015) along the lines ofthe sensitivity auditing proposed by Saltelli and Funtowicz (2014).

Acknowledgments The project to develop normalization factors was financially supported by the Directorate General for the Environment (DG ENV) ofthe European Commission in the context ofthe Administrative Arrangement 'Environmental Footprint and Material Efficiency Support for Product Policy' (No. 70307/2012/ENV.C.1/635340).

The authors thanks the two anonymous reviewers of this and of the paper on normalization factors for ILCD (Sala et al. 2015) which suggested the need of a systematic uncertainties and sensitivity analysis.

Compliance with ethical standards

Conflict of interest The authors declare that they have no competing interests.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.


Archer G, Saltelli A, Sobol I (1997) Sensitivity measures, ANOVA-like techniques and the use of bootstrap. J Stat Comput Simul 58:99-120

Bala A, Raugei M, Benveniste G, Gazulla C, Fullana-I Palmer P (2010) Simplified tools for global warming potential evaluation: when good enough is best. Int J Life Cycle Assess 15:489-498 Benini L, Boulay AM, Sala S (2015) Cross-scale consistency in life-cycle impact assessment: the case of water scarcity. Proceedings of the SETAC Europe conference 2015, 3-7 May 2015, Barcelona Campolongo F, Saltelli A, Cariboni J (2011) From screening to quantitative sensitivity analysis. A unified approach. Comput Phys Commun 182:978-988

Ciroth A, Fleischer G, Steinbach J (2004) Uncertainty calculation in life

cycle assessments. Int J Life Cycle Assess 9(4):216-226 Clavreul J, Guyonnet D, Christensen TH (2012) Quantifying uncertainty in LCA-modelling of waste management systems. Waste Manag 32(12):2482-2495 Clavreul J, Guyonnet D, Tonini D, Christensen TH (2013) Stochastic and epistemic uncertainty propagation in LCA. Int J Life Cycle Assess 18(7):1393-1403

Derwent RG, Jenkin ME, Saunders SM, Pilling MJ (1998) Photochemical ozone creation potentials for organic compounds in northwest Europe calculated with a master chemical mechanism. Atmos Environ 32:2429-2441 ECE—Economic Commission for Europe (2010) Hemispheric transport of air pollution 2010 part A: ozone and particulate matter. Eds: Dentener F. Keating T., Akimoto H. Prepared by the Task Force on Hemispheric Transport of Air Pollution acting within the framework of the Convention on Long-range Transboundary Air Pollution. Geneve and New York, United Nations EC-JRC (2011a) Recommendations based on existing environmental impact assessment models and factors for life cycle assessment in European context. Luxembourg: Publications Office of the European Union. EUR24571EN. ISBN 978-92-79- 17451-3. Available at EC-JRC (2011b) EDGAR-HTAP: a harmonized gridded air pollution emission dataset based on national inventories. JRC Scientific and Technical reports. European Commission, Joint Research Centre, Institute for Environment and Sustainability. ISBN: 978-92-7923123-0

EC—JRC & PBL (Netherlands Environmental Assessment Agency) (2011) Emission Database for Global Atmospheric Research (EDGAR), release version 4.2. Accessed 10 Dec 2014 EMEP/CEIP (2013a) Centre on Emission Inventories and Projections (CEIP) Country- and sector-specific pollutant emission data. Available at home1/ceip_home/webdab_emepdatabase/reported_emissiondata/. Accessed 10 Dec 2014 EMEP/CEIP (2013b) Present state of emissions as used in EMEP models. Available at emepmodels/. Accessed 10 Dec 2014 Frischknecht R, Steiner R, Jungbluth N (2008) The Ecological Scarcity Method—Eco-Factors 2006. A method for impact assessment in LCA. Environmental studies no. 0906. Federal Office for the Environment (FOEN), Bern, 188 pp Functowicz SO, Ravetz J (1990) Uncertainty and quality in science for

policy. Kluwer, Dordrecht Heijungs R (1994) A generic method for the identification of options for

cleaner products. Ecol Econ 10:69-81 Heijungs R (2010) Sensitivity coefficients for matrix-based LCA. Int J

Life Cycle Assess 15(5):511-520 Heijungs R, Huijbregts MAJ (2004) A review of approaches to treat uncertainty in LCA. p. 332-339. In: Pahl-Wostl C, Schmidt S, Rizzoli AE, Jakeman AJ (eds) Complexity and integrated resources \management. Transactions of the 2nd Biennial Meeting of the International Environmental Modelling and Software Society, Volume 1. iEMSs (ISBN 88-900787-1-5), Osnabrück. 2004, 1533 pp

Heijungs R, Lenzen M (2014) Error propagation methods for LCA—a

comparison. Int J Life Cycle Assess 19(7):1445—1461 Heijungs R, Guinée J, Kleijn R, Rovers V (2007) Bias in normalization: causes, consequences, detection and remedies. Int J Life Cycle Assess 12(4):211-216 Hong J, Shaked S, Rosenbaum RK, Jolliet O (2010) Analytical uncertainty propagation in life cycle inventory and impact assessment: application to an automobile front panel. Int J Life Cycle Assess 15(5):499-510

Huijbregts MAJ (1998) Application of uncertainty and variability in

LCA. Int J Life Cycle Assess 3(5):273-280 Huijbregts MAJ, Hellweg S, Hertwich E (2011) Do we need a paradigm shift in life cycle impact assessment? Environ Sci Technol 45(9): 3833-3834

Humbert S (2009) Geographically differentiated life-cycle impact Assessment of human health. PhD Thesis. University of Berkeley. 265 p

Humbert S (2014) personal communication by Sebastien Humbert, September 2014

Hung ML, Ma HW (2009) Quantifying system uncertainty of life cycle assessment based on Monte Carlo simulation. Int J Life Cycle Assess 14(1):19-27 Imbeault-Tétreault H, Jolliet O, Deschênes L, Rosenbaum RK (2013) Analytical propagation of uncertainty in life cycle assessment using matrix formulation. J Ind Ecol 17(4):485-492 Jenkin ME, Hayman GD (1999) Photochemical ozone creation potentials for oxygenated volatile organic compounds: sensitivity to variations in kinetic and mechanistic parameters. Atmos Environ 33:1775-1293 Laurent A, Lautier A, Rosenbaum RK, Olsen SI, Hauschild MZ (2011) Normalization references for Europe and North America for application with USEtox™ characterization factors. Int J Life Cycle Assess 16(8):728-738 Lautier A, Rosenbaum RK, Margni M, Bare J, Roy P, Deschênes L (2010) Development of normalization factors for Canada and the United States and comparison with European factors. Sci Total Environ 409(1):33-4 Lloyd SM, Ries R (2007) Characterizing, propagating, and analyzing

uncertainty in life-cycle assessment. J Ind Ecol 11(1):161 -179 Maxim L, van der Sluijs JP (2011) Quality in environmental science for policy: assessing uncertainty as a component of policy analysis. Environ Sci Policy 14:482-492 Padey P, Girard R, le Boulch D, Blanc I (2013) From LCAs to simplified models: a generic methodology applied to wind power electricity. Environ Sci Technol 47:1231-1238 Pannell DJ (1997) Sensitivity analysis of normative economic models: theoretical framework and practical strategies. Agric Econ 16:139-152 Pfister S, Bayer P (2014) Monthly water stress: spatially and temporally explicit consumptive water footprint of global crop production. J Clean Prod 73:52-62 Raaschou-Nielsen O, Andersen ZJ, Beelen R, Samoli E, Stafoggia M, Weinmayr G, Cesaroni G (2013) Air pollution and lung cancer incidence in 17 European cohorts: prospective analyses from the European Study of Cohorts for Air Pollution Effects (ESCAPE). Lancet Oncol 14(9):813-822 Sala S, Benini L, Mancini L, Ponsioen T, Laurent A, van Zelm R, Stam G, Goralczyk M, Pant R (2014) Methodology for building LCA-compliant national inventories of emissions and resource extraction. Background methodology for supporting calculation of Product Environmental Footprint normalisation factors. JRC scientific and policy report. EUR 26871. Luxembourg (Luxembourg): Publications Office of the European Union; 2014. JRC92036 Sala S, Benini L, Mancini L, Pant R (2015) Integrated assessment of environmental impact of Europe in 2010: data sources and

extrapolation strategies for calculating normalisation factors. Int J Life Cycle Assess 20:1568-1585 Saltelli A, Funtowicz SO (2014) When all models are wrong. Issue in

science and technology, pp 79-85 Saltelli A, Annoni P, Azzini I, Campolongo F, Ratto M, Tarantola S (2010) Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Comput Phys Commun 181:259-270

Sobol IM (1993) Sensitivity estimates for nonlinear mathematical

models. Math Model Comput Exp 1:407-414 Sobol IM (2001) Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math Comput Simul 55: 271-280

Tarantola S (2005) SimLab 2.2 Reference Manual. Institute for Systems, Informatics and Safety, European Commission, Joint Research Centre, Ispra, Italy Available at simlab. Accessed Oct 2015 Tukey JW (1977) Exploratory Data Analysis. Addison-Wesley. ISBN-13: 978-0201076165

UNFCCC (2013) United Nations Convention on Climate Change http:// 3814.php. Accessed Jun 2014 Van der Sluijs JP, Craye M, Funtowicz S, Kloprogge P, Ravetz J, Risbey J (2005) Combining quantitative and qualitative measures of uncertainty in model-based environmental assessment: the NUSAP system. Risk Anal 25(2):481-492 Van Zelm R, Huijbregts MAJ, Den Hollander HA, Van Jaarsveld HA, Sauter FJ, Struijs J, Van Wijnen HJ, Van de Meent D (2008) European characterization factors for human health damage of PM10 and ozone in life cycle impact assessment. Atmos Environ 42:441-453

Vandecasteele I, Bianchi A, Batista e Silva F, Lavalle C, Batelaan O

(2014) Mapping current and future European public water withdrawals and consumption. Hydrol Earth Syst Sci 18:407-416

Wei W, Larrey-Lassalle P, Faure T, Dumoulin N, Roux P, Mathias JD

(2015) How to conduct a proper sensitivity analysis in life cycle assessment: taking into account correlations within LCI data and interactions within the LCA calculation model. Environ Sci Technol 49(1):377-385