Guideline for ex-post evaluation of measures and instruments in flood risk management

(1)

Integrated Flood Risk Analysis

and Management Methodologies

Guideline for ex-post evaluation of measures and

instruments in flood risk management

Date April

2008

Report Number

T12-07-03

Revision Number 1_3_01

Deliverable Number: D12.1

Due date for deliverable: May 2007 Actual submission date: November 2007

Task Leader IOER

FLOODsite is co-funded by the European Community

Sixth Framework Programme for European Research and Technological Development (2002-2006) FLOODsite is an Integrated Project in the Global Change and Eco-systems Sub-Priority

Start date March 2004, duration 5 Years Document Dissemination level

PU Public PU

PP Restricted to other programme participants (including the Commission Services) RE Restricted to a group specified by the consortium (including the Commission Services) CO Confidential, only for members of the consortium (including the Commission Services)

Co-ordinator: HR Wallingford, UK Project Contract No: GOCE-CT-2004-505420

(2)

D

OCUMENT

I

NFORMATION

Title Guideline for ex-post evaluation of measures and instruments in flood risk

management

Lead Author Alfred Olfert Contributors

Distribution Public Document Reference T12-07-03

D

OCUMENT

H

ISTORY

Date Revision Prepared by Organisation Approved by Notes

19.12.07 1.0 A. Olfert IOER

10/04/08 1.3 Paul Samuels HR Wallingford Formatting for Year 4 reporting Added Summary

A

CKNOWLEDGEMENT

The work described in this publication was supported by the European Community’s Sixth Framework Programme through the grant to the budget of the Integrated Project FLOODsite, Contract GOCE-CT-2004-505420.

D

ISCLAIMER

This document reflects only the authors’ views and not those of the European Community. This work may rely on data from sources external to the FLOODsite project Consortium. Members of the Consortium do not accept liability for loss or damage suffered by any third party as a result of errors or inaccuracies in such data. The information in this document is provided “as is” and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and neither the European Community nor any member of the FLOODsite Consortium is liable for any use that may be made of the information.

(3)

S

UMMARY

Recent years have seen a rapid development of a modern understanding of flood risk management. This entails the advancement of traditional flood defence approaches towards a more holistic understanding of the sources, processes and consequences of flooding as well as of stakeholders, strategies and options for risk reductions. Risk reduction and the resulting reduction of losses is an essential task of flood risk management.

Effective risk reduction requires the optimal understanding of the causes of risk, options of risks reduction and ways of their implementation. Especially with regard to risk reduction options much has changed in recent decades. Many measures and instruments, which are the actual tools of risk reduction, have developed to reduce risk in various settings. This includes the advancement of existing as well as the development of new options and approaches. While measures are direct physically tangible interventions (e.g. dikes), instruments mainly form the scope of action and often are prerequisite for the implementation measures (e.g. flood zones, flood warning). Often, several measures and/or instruments are applied in a portfolio. Many of these options are hardly being taken into consideration due to a lack of knowledge how they perform under conditions of flooding. Even with regard to traditional measures, the knowledge regarding their actual performance appears scarce. As a result, planning for flood risk reduction often takes recourse to an only limited record of past experience. On the one hand, this limitation applies to the variability of options. Mainly decentral measures are not sufficiently known and understood. Also many instruments are usually not taken into consideration as option. On the other hand, the limitation applies to the information regarded when deciding on the options. Often, mainly the expected effectiveness is considered. Aspects outside the range of primary intentions, sustainability or long term performance are often not sufficiently considered.

One important reason for this is the sheer lack of required information and of methods to derive it. The methodology presented with the guideline provides a tool box for the ex-post evaluation of risk reduction implemented in the past. Ex-post evaluation can close the knowledge gap between past / current practice and future decisions in flood risk reduction.

The methodology provides a framework for the ex-post evaluation of measures and instruments applied for flood risk reduction. This framework is laid out to be generically applicable with all conceivable measures and instruments. In contrast to the evaluation which is applied for the selection of the most appropriate option before implementation (ex-ante evaluation), ex-post evaluation has a retrospective view on how the intervention worked out in reality. However, both types of evaluation are closely related. The relation is closest in terms of types of effects regarded and the different evaluation criteria. In the end, this relation is even compulsory if the assessment of potential solutions shell take recourse to experiences from the past.

With the application of the methodology, stakeholders of flood risk reduction will be enabled to derive the maximum advantage from their previous action for the improvement of future activities. The methodology described by the guideline will enable the interested parties to learn about the intended and unintended effects, effectiveness, efficiency and other aspects of risk reduction.

(4)

(5)

Figure 1: Functions and addressees of ex-post evaluation (Olfert in prep.) 2 Figure 2: The concept of ex-post evaluation of measures and instruments (Olfert in prep.) 4 Figure 3: Relation of measures and instruments (Olfert in prep.) 5 Figure 4: Action in ex-post evaluation and support provided by the methodology 11 Figure 5: Procedure for the case specific selection of effect indicators (Olfert in prep.) 13 Figure 6: Types of information in ex-post evaluation (Olfert in prep.) 16

(7)

A INTRODUCTION

1. Background

Recent years have seen a rapid development of a modern understanding of flood risk management. This entails the advancement of traditional flood defence approaches towards a more holistic understanding of the sources, processes and consequences of flooding as well as of stakeholders, strategies and options for risk reduction. Risk reduction and the resulting reduction of losses is an essential task of flood risk management.

Effective risk reduction requires the optimal understanding of the causes of risk, possibilities for risks reduction and ways of their implementation. Especially with regard to risk reduction options much has changed in recent decades. Many measures and instruments, which are the actual tools of risk reduction, have developed to reduce risk in various settings. This includes the advancement of existing as well as the development of new options and approaches. While measures are direct and physically tangible interventions (e.g. dikes), instruments mainly form the scope of action and often work through the implementation of measures (e.g. flood zones, flood warning). Often, several measures and/or instruments are combined in a portfolio. Many of these potentially available options are hardly being taken into consideration due to a lack of knowledge about their performance under conditions of flooding. Even with regard to traditional measures, the depth of knowledge regarding the actual performance appears scarce.

As a result, planning for flood risk reduction often takes recourse to an only limited record of past experience. On the one hand, this limitation applies to the variability of options. Many measures are not sufficiently known, understood and considered. Also many instruments are usually not taken into consideration as option. On the other hand, the limitation applies to the comprehensiveness of information regarded when deciding on the options. Often, expected effectiveness is the only criterion considered. Aspects outside the range of primary intentions, sustainability or long term performance are often not sufficiently covered.

One important reason for this is the sheer lack of required information and of methods to derive it. The methodology presented with this guideline provides a tool box for the ex-post evaluation of implemented measures and instruments. Ex-post evaluation can close the knowledge gap between past and current practice and future decisions in flood risk reduction.

With the application of the methodology, involved stakeholders shall be enabled to derive the maximum advantage from their previous action for the improvement of future activities. The methodology described by the guideline shall also enable the interested parties to learn about the intended and unintended effects, effectiveness, efficiency and other aspects of the overall performance of applied options.

2. Approach

and

merits of ex-post evaluation

2.1 What for conducting ex-post evaluation?

Ex-post evaluation derives added value from past and current practice for the improvement of future flood risk management. To achieve this, the ex-post perspective addresses measures and instruments already in place and installed mainly with the aim to reduce flood risk. Based on observable effects, ex-post evaluation generates information on criteria such as the achieved effectiveness, efficiency, robustness and other criteria.

Through the investigation of observed effects, post evaluation is different form the so called ex-ante evaluation conducted before the realisation of risk reduction projects – e.g. while selecting an alternative. In ex-ante evaluation, assumptions regarding future effects and costs are used in order to construct the potential performance of alternatives. In order to support this planning task, the

(8)

observation of actual performance is highly important. Thus the by analysing past and present performance, ex-post evaluation passes lessons on to future flood risk management.

Questions addressed by ex-post evaluation with regard to different criteria can be: 1) Which intended or unintended impacts were caused by a measure/instrument? 2) To which extent did a measure or instrument achieve the objectives?

3) At which benefit/cost ratio did the measure or instrument achieve its effects? 4) How did the measure or instrument perform under different conditions?

The methodology described by this guideline provides the approaches and methods to answer these questions.

2.2 Whom for conducting ex-post evaluation?

Flood risk management includes various stakeholders. All of those fulfil different tasks at different professional and administrative levels which finally altogether contribute to the goal of reducing risks and losses. This includes engineers, planners, implementing bodies and companies as well as decision makers. On the one hand, they require feedback for their informed action and are thus potential addressees of ex-post evaluation. On the other hand, ex-post evaluation is important to make achievements transparent. Thus, also the wider public can be the addressee for results or ex-post evaluation.

As a result, findings need to be specific for the different spectra of interest of addressees. It is part of the task of ex-post evaluation to deliver information and conclusions relevant and understandable by the addressees in various fields of action (Figure 1). On the practical level these can be project managers in charge of the implementation of measures and instruments. They require detailed and up to date information for operational decisions and adaptations. On the strategic level these can be planners and specialised authorities involved in the development of visions and strategic alternatives for certain areas and locations. Here, experience from previous action is imporant to develop most appropriate solutions for the future. On the political level more generalised information supporting the legitimation of actions and expenses is needed. Neither of these levels should be neglected to ensure the continuous process, advancement and transparency of flood risk management.

Functions and addressees of ex-post evaluation Ex-post Evaluation

=

Generation of information

Learning

Knowledge generation in support of future strategy development

and intervention planning Controlling

Feed back of information in support of operational

project management

Legitimacy Accounting for public

resources spent and advances achieved Functions of ex-post evaluation Addressees of ex-post evaluation Policy makers and planners Project managers

High level civil servants and politicians Output of

ex-post evaluation

(9)

B FRAMEWORK OF EX-POST EVALUATION

3. Methodology for ex-post evaluation of measures and instruments

3.1 Concept of ex-post evaluation

The methodology provides a framework for the ex-post evaluation of measures and instruments applied for flood risk reduction (based on Olfert & Schanze 2007). This framework is laid out to be generically applicable with all conceivable measures and instruments. In contrast to the evaluation which is applied for the selection of the most appropriate option before implementation (ex-ante evaluation), ex-post evaluation has a retrospective view on how the intervention worked out in reality. However, both types of evaluation are closely related. The relation is closest in terms of types of effects regarded and the evaluation criteria. In the end, this relation is even compulsory if the assessment of potential solutions shell take recourse to experiences from the past.

Thus, ex-post evaluation is the evaluation of the actual performance of an implemented measure or instrument with the aim to derive experiences which can be used to ensure best possible flood risk management in future. To gain these experiences, ex-post evaluation looks at a number of criteria which describe different aspects of the overall performance. The latter is important to ensure the comprehensive view on an intervention in the light of sustainability which is the accepted guiding principle of flood risk management (FLOODsite Consortium 2005).

The concept of ex-post evaluation is composed of several elements (Figure 2). Two elements mainly support the definition of the evaluation case by considering the type of an intervention and the conditions under which it performs:

• Classification of measures and instruments based on functional characteristics • Conditions of performance and perspective of evaluation

The methodology itself is made up of two other elements for the identification of effects and their evaluation in criteria:

• Indicators of effects and method for case specific selection of indicators • Criteria of evaluation and methods for their determination

Measures and instruments are interventions at project level. Interventions because they intervene into existing settings in order to change or adapt those to a more convenient situation. Project level means they are concrete in terms of aims, costs and spatial reference. Large scale programmes such as state-wide investment programmes are not primarily addressed by the methodology. Measures and instruments either directly impose changes (measures) or trigger mechanisms (instruments) which can lead to intended or unintended impacts. Different types of interventions are distinguished by their ways of functioning. This understanding is important for the evaluation as it allows the pre-determination of possible effect-pathways as basis for the selection of indicators. The terms measures and instruments are closely related to the traditional differentiation of structural and non-structural measures. However, they are not congruent. The latter have developed from the perspective of traditional flood protection. As a result, most traditional engineering works were called structural measures. Any other or new approaches were put under non-structural measures. As a result, many different understandings of structural and non-structural measures can be identified still lacking a clear definition. With introducing the terms measures and instruments, the fundamentally different functioning of the two groups of interventions is referred to which also pays respect to a substantially extended repertoire of options in flood risk management (Annex 1).

Conditions describe the setting in which measures and instruments perform. These form a particularly important background for the evaluation as they can be decisive for the performance of an intervention and hence essential for the correct understanding of the observed performance. Interventions and conditions only together make up the actual case that is being evaluated. Combining the type of

(10)

intervention with further conditions, the methodology allows the selection of a case specific indicator set supported by a web-based selection tool (www.floodsite.net).

Indicators are yardsticks to measure intended and unintended effects of measures and instruments. Effects are changes brought about through the realisation and operation of an intervention. The indicator set provided within the methodology is a source base containing indicators for all considered measures and instruments. The set includes social, economic and ecological indicators and allows thus the consideration of the dimensions of sustainability. Indicators are made operational by providing methods for the acquisition of data and the effect analysis (Annex 2-5). Effect indicators are the basis for the evaluation in the criteria.

Criteria are the actual viewpoints of an evaluation. Criteria relate effects to values and allow thus a more comprehensive understanding of the overall performance of a measure or instrument. For criteria considered by the methodology, methods for their determination are provided. As for effect analysis, methods proposed for the determination of effectiveness and efficiency are quantitative. With respect to robustness, a qualitative approach is proposed that takes recourse to quantitative information and the analysis of contexts conditions. This is complimented by a quantitative approach aiming at facilitating the comparison of cases.

Chapters 3.2 to 3.5 introduce the elements of the methodology more in detail. Section C of the guideline describes the procedure for the identification of the evaluation case before the actual steps of ex-post evaluation are described in section D.

METHODOLOGY

INDICATORS of effects and METHODS for data acquisition and effect analysis

Effectiveness Efficiency Robustness Case1 T y pe of w a ter bod y T y pe of _floo d T y pe o f la nd u s e T y pe of in te rv e n tio n COMPA R ISO N of ca s e s M E A S URE S and IN S T RU M E NTS CRITERIA of evaluation and METHODS for their determination Cross-case comparison

CONCEPT OF EX-POST EVALUATION

M e a s ur e/ In s tr u m e n t 1 CONDITIONS M_a Mb Mn I_a+ Ib+ In+ E ffe ct ana ly s is Effectiveness Efficiency Robustness Case₂ Meas ur e /In s tru m e n t 2 M_c Md Mn I_c+ Id+ In+ Effect a n a ly sis Effectiveness Efficiency Robustness Case_n M e a s ur e/ In s tr u m e n t n M_e M_f Mn I_e+ I_f+ In+ Effect a n a ly s is

(11)

Generally, each evaluation case is regarded stand alone. This is consistent with the main objective of ex-post evaluation which is the feed back of information primarily addressing stakeholders related to this case. However, by providing a structured approach based on a comprehensive set of indicators, the consideration of conditions and by reflecting various criteria the methodology supports the generation of potentially comparable evaluation cases (Figure 3). In future this can provide surplus information gaining importance beyond that of single-case evaluation.

3.2 Single measures and instruments and portfolios

Measures and instruments are the objects of evaluation. These are the actual tools to reduce flood risk. The differentiation between measures and instruments describes the fundamentally different functioning of these different kinds of tools (Annex 1).

Measures are physically tangible interventions which cause effects directly through their existence. These include all kinds of flood control and defence works, traditionally called structural measures such as dams, dikes or river training. But, here also belong the more recent approaches such as land management techniques, river rehabilitation, mobile defences, different types of flood proofing or evacuation measures.

Instruments are interventions which cause effects indirectly by shaping the scope for action or by improving risk perception and preparedness of stakeholders incl. land users. Examples are land use regulations, financial incentives, flood warning or hazard maps.

In other words, measures, if once implemented, appear as physically tangible components of the flood risk system. Instruments again are important to support the realisation of possible measures or to facilitate the understanding of different stakeholders for existing risks and the preparedness to take responsibility for risk reduction. As a result, instruments mainly shape the scope for action by applying more or less restrictive tools. Nevertheless, instruments can also be indispensible triggers for the implementation of certain measures. For example, this is true for all contingent measures such as the erection of mobile defences, sand bagging or evacuation which are highly reliant of the timely and correct flood warning (instrument).

Risk reduction policy

Measures Instruments

Shaping the scope for action

Reduction of risks and losses

Figure 3: Relation of measures and instruments (Olfert in prep.)

Not always the object of evaluation is restricted to single measures and instruments. In many cases, combinations of interventions, so called portfolios, are applied to achieve certain objectives such as the reduction of flood levels or the protection of developed land. For example, this can occur in urban situations where a defence line may be composed of permanent dike and (semi-) mobile wall sections. In case of flooding they act as a compound and, as a result, can only be evaluated as such. Another typical example of portfolios are private small scale measures. Here, the flood proofing of single buildings is usually realised by combinations of sealing, sandbagging, shielding, pumping, controlled flooding etc.. Additionally, usually also evacuation of inventory and technical infrastructures is implemented. In such cases the possibility to isolate the contributions of single interventions within the compound is limited and may thus need to be considered as a portfolio.

(12)

Also instruments can act in combination. For example, the regulation instrument flood zone may need support of financial stimulation like allowances or subsidised loans in order to ensure implementation of objectives. In such cases, observed risk reduction cannot be fully attributed to the regulation instrument alone and the combination of both instruments becomes the object of evaluation. Often however, instruments act in combination with measures. This is naturally the case, as most instruments shape the scope for the implementation of measures (Figure 3). For example, land use regulations may require the installation of provisions to reduce the risk of flooding in a certain area. Here, the land use regulation itself does not provide any risk reduction until required measures (e.g. check valves, relocation of the heating central or fuse box) are implemented in reaction to the regulation. In such cases, ex-post evaluation needs to identify the effect path of instruments and if necessary to consider measures which finally are applied. Also flood warning, one of the most important instruments in flood risk management would remain meaningless without the action it ought to trigger.

Therefore, the early identification whether a single intervention or a portfolio is object of evaluation is crucial from the very beginning. This is decisive for the development of an appropriate evaluation design and, as a result, for the generation of meaningful evaluation results.

3.3 Conditions for performance and evaluation

The implementation, operation and performance of measures and instruments is influenced by pre-existing (and changing) external natural and socio-economic conditions. Above this, measures and instruments have own, internal conditions which are also important for their performance. While internal conditions are descriptive for the state of a measure or instrument at a certain point of time, external conditions describe the scope under which the intervention functions.

Internal conditions influence the functioning of measures and instruments from within. This may include factors such as the structural quality of a measure, but also the change of its constitution during its life cycle. Internal conditions are often influenced by external conditions.

External conditions represent the physical (e.g. flood event, physiography, land uses), political (e.g. extent and duration of political support), socio-cultural (e.g. planning culture, societal expectations) or economic (e.g. availability of resources for maintenance and operation, real estate market development) factors which form the overall scope for interventions. External conditions are often more or less static boundary conditions under which a measure or instrument is introduced and operated. But, they also can be dynamic, as emphasised by the discussion of climate change. Especially the magnitude and dynamics of flood events is a highly variable external factor which can be decisive for the performance and which makes the cases of ex-post evaluation very unique and often little comparable.

Conditions influence the appearance and the functioning of each intervention. Appearance describes the substance in terms of its actual shape (e.g. technical design of a dike), its information content (e.g. of a land use regulation) or its organisation and capacity (e.g. evacuation, communication). The functioning refers to the conversion of the substance of an intervention into outcomes. Therefore, the knowledge of conditions is important for constructing the hypotheses of the cause-effect relationship upon which evaluation is based.

In evaluation, conditions are used in conjunction with the measure or instrument of interest. All conditions which are needed to sufficiently describe the case of evaluation should be considered in order to understand the case and to enable its potential comparability with other cases. Accordingly, selected conditions are considered by the methodology and used for the selection of indicators. Four main external conditions are formalised and should be applied in all cases:

• Type of flood event • Probability of event • Type of land use • Type of water body

(13)

However, also case specific conditions (chapter 3.3.2) can be highly important. Beyond these, the consideration of internal conditions can be important in the evaluation process. These explain, whether the evaluated interventions were in a proper state to provide the intended services. This is based on the assumption, that best performance of measures and instruments is achieved when they are in best internal conditions. As a result, information provided by ex-post evaluation shall consider whether the intervention were in a state permitting the intended performance. This state is influenced by a number of factors which can make a large difference in the performance on measures and instruments. These factors can include:

• Age and physical constitution of measures

• Quality/completeness of implementation of measures • Legal backing and financing of instruments

• Timely warning and penetration of warning

For example, contingent measures require timely and sufficiently precise warning. If warning does not penetrate in time or does not convey the needed information, taken provisions (e.g. mobile flood defences) may not be activated in time. While an evaluation could e.g. show insufficient effectiveness of risk reduction, the knowing of internal conditions can be decisive for the generalisation of results.

3.3.1 Description of external conditions

Type of flood

The flood type describes the general source and dynamics of flooding. Five types are distinguished: • Flash floods

• Plain floods • Estuarine floods • Coastal floods • Urban floods

Flash floods are induced by heavy, often local, precipitation or sudden snow melt or combination of both. It is characterised by fast onset (minutes to hours after a causative event), a heavy multiplication of discharge and high flow velocities and is often accompanied by large sediment and debris flow. Plain floods or slow rise floods are induced by large scale precipitation or snow melt. In contrast to flash floods, slow rise floods or riverine floods are characterised by relatively slow onset (up to several days), a less severe increase of discharge, increased but still moderate flow velocities and moderate sediment and debris transport.

Estuarine floods are induced either by a costal flood whereby flood water is driven into the estuary or the incident of coastal and slow rise flooding meeting in the estuary.

Coastal floods or storm surges are induced by the combination of temporary sea level rise resulting from extreme meteorological conditions and especially severe in coincidence with high-tide and is characterised by pronounced wave swell (Smith & Ward 1998, Evans et al. 2004).

Urban floods or intra-urban floods are composed of storm water runoff induced by heavy precipitation on urban surfaces, and includes surface flood waves, internal flooding of areas and properties through overloaded sewer systems as well as through backing up of culverted and other water courses (cf. Evans et al. 2004). Urban floods are characterised by fast onset, high dynamics of discharge and often uncertain pathways of the flood water.

(14)

Probability of flooding

The probability primarily describes potential frequency of flooding. At the same time it refers to the magnitude or severity of the event. The probability as condition is important to compare the flood event with the design standard (if available) of the evaluated intervention in order to understand whether it was meant to cope with a flood of the observed magnitude. Based on this information, evaluation results can be valued differently. Flood frequency is expressed as the probability ratio. For example, a 1:10 event is a flood event that statistically is expected to occur every 10 years. Alternatively, the event of that probability can be called a 10-years flood.

Type of land use

Land uses affected by flooding or flood risk reduction are important for the generation of types and amount of losses. Two categories describe the type of assets which are exposed to the hazard respectively to the risk reduction intervention:

• Developed (urban/industrial/rural buildings or urban land uses affected) • Undeveloped (no buildings, only rural land uses)

Compared to developed areas, in non-built up areas generally different types of effects can be expected and potential for losses is lower.

Type of water body

The type of water body describes ecological properties of a water classified by the European Water Framework Directive (WFD). Flood risk reduction is often closely related to water bodies. On the one hand, the ecological aspects of risk reduction requires due consideration of differences between water bodies. This allows the a-priory exclusion of certain effects for certain types of intervention. The inclusion of these relations enables the targeted consideration of potential side effects. The following types of water bodies are differentiated according to the WFD:

• Rivers • Lakes

• Transitional waters • Coastal waters

Rivers are bodies of inland water flowing for the most part on the surface of the land but which may flow underground for part of its course (Article 2(4)).

Lakes are bodies of standing inland surface water (Article 2(5)).

Transitional waters are bodies of surface water in the vicinity of river mouths which are partly saline in character as a result of their proximity to coastal waters but which are substantially influenced by freshwater flows (Article 2(6)).

Coastal waters are waters on the landward side of a line, every point of which is at a distance of one nautical mile on the seaward side from the nearest point of the baseline from which the breadth of territorial waters is measured, extending where appropriate up to the outer limit of transitional waters (Article 2(7)).

(15)

3.3.2 Specific conditions

Beyond the conditions already discussed, each case may have its specific conditions which are important for understanding the effect spectrum as well as the intended performance of an intervention. A non-exhaustive list is given below. The relevance of specific conditions should be considered in all cases in order to avoid misinterpretation. Examples of specific conditions are:

• Exposure of hazardous substances • Presence of natural heritage sites • Presence of cultural heritage sites • Political support

• Quality of maintenance

• Internal conditions of measure or instrument

3.4 Indicators of effects

Indicators describe changes (effects) generated by measures and instruments. They are the practical units of evaluation applied for the measurement of obtained effects, mobilised resources or accomplished outputs (cf. EVALSED 2003). Observable effects described by indicators are the basis to derive further evaluation in criteria (chapter 3.5).

Effects caused by measures and instruments can relate to physical and societal elements of the flood risk system. Effects can be both intended or unintended. Unintended effects are referred to as side effects. In order to enable comprehensiveness of evaluation in terms of sustainability, four main categories of effects are considered.

• Socio-cultural effects • Economic effects

• Hydrological/hydraulic effects • Ecological effects

Socio-cultural effects describe outcomes in the field of social and cultural aspects. These refer to usually intangible values which can be affected by the implementation, existence or operation of risk reduction. Proposed categories address issues of public health, social stability and cultural/natural heritage (for related indicators see Annex 2a).

Economic effects describe possible outcomes of risk reduction with regard to tangible values with the emphasis on financial aspects (for related indicators see Annex 2b).

Hydrological and hydraulic effects describe common primary services of many flood risk reduction measures. These do not constitute effects in terms of risk reduction. However, these effects are important milestones in the evaluation related measures with regard to actual risk reduction effects. For example, it can be valuable to know the effect of a measure on the water level to value its contribution for loss reduction. Hydrological and hydraulic effects are closely related to limnological effects, but address a different aspect and are therefore dealt with separately (for related indicators see Annex 2c).

Ecological effects describe impacts of measures and instruments on elements of the ecosystem in areas of implementation and downstream. This effect-category mainly refers to unintended effects. Ecological effects constitute an important part of the impact spectrum of interventions. The related spectra are different for the source areas, water bodies and flood plains. Therefore, effects are distinguished in these three groups. Potential effects in source areas, due to the character of measures and interventions applied there, refer to qualities of soil and vegetation. Indicators of limnological effects reflect the quality elements of the European Water Framework Directive (CEC 2000).

(16)

Ecological effects in flood plains mainly refer to natural qualities and functions of inundation areas (for related indicators see Annex 2d).

The procedure for the determination of effects is described in chapter 8 of the guideline.

3.5 Evaluation criteria

While indicators describe changes of selected properties in the affected systems (social, economic, ecological), criteria represent the actual viewpoints of evaluation. In these viewpoints, achieved (observed) effects are related to certain values such as objectives, costs or seen against different conditions of performance and consider the robustness of interventions. As a result, evaluation of these criteria is based on observed effects. The guideline addresses the evaluation with three criteria:

• Effectiveness (relation to objectives) • Efficiency (relation to costs)

• Robustness (in light of different conditions)

Effectiveness describes the extent to which the objectives of an intervention are achieved (OECD 2002, p. 20f, CEC 2003, p. 45). It is thus dependent on the availability of objectives for an intervention (chapter 9). Being also the traditional focus of evaluation, effectiveness analysis constitutes a core element of ex-post evaluation.

In the light of changing societal priorities (Plate 2003) and considering financial limitations, the question of economic efficiency is raised. In simple terms, efficiency can be determined by relating economic (financial) costs and benefits to obtain the benefit-cost ratio. The benefit-cost ratio is a measure for how economically resources are converted into results (cf. OECD 2002, p. 21). Benefits are represented by observed positive economic (financial) effects. Costs are expressed by intended and unintended as well as direct and indirect realisation, operation and maintenance costs of the intervention (chapter 10).

Robustness describes the ability of a measure or instrument to sustain the intended serviceability over different known and unknown (changing) conditions (chapter 11). Interventions are supposed to deliver their intended effects under different magnitudes or dynamics of flooding or climatic and societal changes. For example, a scheme introduced to protect against floods up to the 100 years event due to different reasons in future may be hit more frequently by a design event. Or, encroachment into the floodplain may substantially increase vulnerability and thus the risk in a flood prone area. An intervention applied to reduce risks should be able to ensure the tolerability of risk in different situations unless the investment it becomes obsolete.

4. Actions in ex-post evaluation

The proposed evaluation methodology supports the implementation of ex-post evaluation by providing the framework and methods for the realisation of the evaluation process. After presenting the elements of the methodology, main steps are addressed to promote the application of the methodology. Figure 4 shows the steps as consecutive actions within an ex-post evaluation and describes how each step is addressed by the presented methodology. The elements, methods and tools, developed to support these steps are described following this chapter.

(17)

Action 1

Identification of the case

Action 2

Selection of indicators

Action 3

Acquisition of data

Action 4

Effect analysis and evaluation of criteria

Action 5

Learning from results

The identification of the object of evaluation (single or portfolio) is supported by a systematisation of measures and instruments (chapter 5, Annex 1). A set of standard and case specific conditions completes the description of the case (chapter 5).

A two-step method and a web based tool support the case specific selection of effect indicators (chapter 6).

For each indicator potential methods for data acquisition are proposed (chapter 7, Annexes 2 to 5).

Quantitative methods are provided for effects analysis (chapter 8) and the determination of effectiveness (chapter 9) and efficiency (chapter 10). Two approaches (qualitative and quantitative) are proposed for the consideration of robustness (chapter 11).

Learning from results remains an indispensible, but individual task for each evaluator and the addressees of ex-post evaluation.

Steps of evaluation Contribution of the methodology

Figure 4: Action in ex-post evaluation and support provided by the methodology

C PREPARING THE EVALUATION

5. Identifying the case (Action 1)

The identification of the case is the first step of evaluation. It is important for the determination of questions the evaluation is going to answer. Two main elements are used to describe the case:

1) The actual intervention of interest, which can be either a a. Single intervention or

b. Portfolio of different measures or/and instruments

2) The conditions, which influence the performance

In the following, the determination of these elements is described.

5.1 Object of evaluation

5.1.1 Single intervention or combination of measures and instruments

The first step is used to determine whether the object of evaluation is a single intervention or a combination of measures and instruments (portfolio). This clarification is important for the development of an appropriate evaluation design and here especially for the decision whether effect indicators are to be selected for one single intervention or whether different effect spectra need to be taken into consideration including those of different (types of) measures or instruments. The question behind this issue is:

Does the intervention generate outcomes by itself or do further interventions contribute to the effects? The answers to this question differ slightly for measures and instruments.

(18)

For measures

If outcomes are generated by one single measure, then no further risk reduction intervention needs to be included in the evaluation. If more than one measure is accountable for the outcomes, the full portfolio of contributing interventions should be identified including all interventions involved. For example, a protection line can consist of different sections including a dike, flood walls and mobile defences which jointly protect the same area. The impact of neither of the measures can be evaluated without considering the other. As a result, the object of evaluation is a portfolio which needs to be further substantiated by identifying all contributing measures.

For instruments

The same principle can be true for instruments. One single instrument may not be sufficient to trigger the intended outcomes. For example, the implementation of a land use regulation may need support by some stimulating financial instrument. Also here, the effects may be attributed to the combination of instruments instead of only one of both.

For measures and instruments

Not always an instrument is associated with other instruments in a portfolio. Often, the instrument triggers the implementation of measure which are actually accountable for intended or unintended effects. If, e.g., risk reduction is actually a result of measures triggered by the instrument(s), then the applied measures should be included into the portfolio. In fact, while instruments like “flood warning” do not directly affect losses, contingent measures will not be possible without the timely release and penetration of the warning. Thus, on the one hand, to describe the risk reduction effect of flood warning, the subsequent measures must be regarded. On the other hand, evaluation of contingent action should consider that warning and possibly other factors can be indispensable conditions for the implementation of measures. But, there are also instruments, which act directly on perception and preparedness of addressees. In these cases the direct path from instrument towards action may not be educible.

5.1.2 Types of measures and instruments

As important is the clarification, which measures and instrument are the actual object of evaluation. A classification of measures and instruments supports the clear identification of certain types and interventions, which are further substantiated by known and concretely named measures and instruments (Annex 1). A web-based information tool on measures and instruments of flood risk management supports the identification of certain forms of interventions (www.floodsite.net).

5.2 Conditions of performance

Conditions describe the case-specific setting of the evaluated intervention(s). This is important for understanding the actual case. These are important for a more precise selection of effect indicators (chapter 6). A sensible selection of indicators can only be accomplished knowing the potential effects. This requires the understanding of the case which includes the relevant conditions. The disregard of conditions may lead to an inappropriate indicator set which might miss the goals of the evaluation. A too narrow indicator set would face the risk of being not representative. An overburdened indicator set again could thwart the evaluation as a whole.

At the same time, the relation between intervention and case-specific conditions is prerequisite for the understanding of the case by external parties interested in or requiring the evaluation results. The correct understanding of evaluation results will only be useful if sufficiently background information is available. This also applies for the transfer of information to other potential cases. Also, the comparison of cases will be only possible, if the differentiating properties, which arte mainly described by conditions, are known. External and internal conditions are discussed in chapter 3.3.

(19)

6. Case-specific selection of effect indicators (Action 2)

The case-specific selection of indicators is a most sensitive step in the evaluation process. The correct selection of indicators is particularly important in the elaboration of a meaningful and focussed evaluation design. The selection is supported by a two step selection method. The fist step makes use of known potential effect-spectra for each single type of measures and instruments. It allows the reduction of the overall indicator set to those indicators which are potentially conceivable if considering the functional characteristics of an intervention. This step is supported by a web-based indicator selection tool (www.flodsite.net). The second step involves the evaluator’s knowledge of the case to insure the final selection from the reduced set. However, the selection of appropriate indicators can be more or less difficult depending on whether a single measure or instrument, a portfolio consisting of measures or instruments or even a combination of instruments and measures is given. Figure 5 schematically summarises the procedure of indicator selection. Chapter 6.1 presents a procedure which shall facilitate the case specific selection of effect indicators for single interventions. Specialities of indicator selection for cases evaluating portfolios are addressed in chapter 6.2.

Overall indicator set

Type of intervention

Type of

flood, water body, land use

Perspective of evaluation

Step 1 Pre-selection

Question 1

Is it conceivable, that the object of evaluation bears consequences for the aspect described by the indicator?

Step 2

manual selection

yes

no

Question 2

Can the effect described by the indicator be fully or to a known extent attributed to the object of evaluation?

no

Indicator is selected and should be integrated in the evaluation design

Indicator is not further regarded

yes

1 2 3 4 5 6 7 8 9 …

Reduced indicator set

1 - 3 4 - - 7 - 9 … All indicators

Remaining indicators (example)

Procedure of indicator selection

Figure 5: Procedure for the case specific selection of effect indicators (Olfert in prep.)

6.1 Indicator selection for single interventions

6.1.1 Step 1: Pre-selection of indicators with the help of conditions

The goal of this first step is to reduce the overall set of indicators. This reduction is achieved by excluding those indicators, which are not conceivable on the grounds of functional characteristics of the intervention or founding combination with identified conditions. This step is based on certain information about the case which are processed with support of the provided web-based selection tool (www.floodsite.net). Different categories of information are used for the formalised pre-selection of indicators based on the type of intervention defined within the classification and the selected conditions:

(20)

• Functional character of measure or instrument • Type of flood

• Type of water body • Type of land use

• Perspective of evaluation

These selection categories are considered in conjunction. The tool checks each indicator for its potential relevance for the object of evaluation and these conditions. Indicators remaining after pre-selection are understood as ‘potentially relevant’. In order to be classified ‘potentially relevant’, an indicator must show at least the potential of relevance with regard to an intervention and conditions. However, only in few cases the provided tool will deliver the final indicator set. The indicators need further specification through considering additional individual characteristics of each specific case (chapter 6.1.2).

Description of applied conditions

The functional character of an intervention is described within the classification of measures and instruments (Annex 1). Here, the wide array of existing approaches of flood risk reduction is considered. Due to the partially fundamental differences in the functioning of interventions, many impacts (described by indicators) are specific to certain interventions. And vice versa, regarding one certain intervention, many indicators can be disregarded a priory, considering that there is no conceivable effect path leading to the related effect (see below). The same is true for conditions (e.g. ‘type of flood’ below, see below).

The application of the functional character is particularly difficult in case of instruments as the content of the same instrument can differ substantially and so does the conceivable spectrum of effects. As a result, indicator selection for instruments should be supported by considering the functional character of triggered measures. This includes that ex-post evaluation of instruments often will require the consideration of portfolios.

The type of flood differentiates effects which are specific to different flood types such as slow swell floods, flash floods, coastal and estuarine floods and urban floods. For example, an effect occurring only in coastal waters or during urban flooding can be excluded if dealing with inland waters and vice versa.

The type of water bodies differentiates effects of interventions and flooding for riverine, coastal, estuarine and transitional waters. It is mainly relevant for the selection of ecological indicators and therefore follows the framework proposed by the Water Framework Directive. Therefore, the relation to involved water bodies is primarily applicable for limnological indicators. For example, the evaluation of an intervention in conditions of river flooding will automatically exclude ecological indicators applying to coastal waters.

The type of land use differentiates effects which occur in developed (urban/industrial) and undeveloped (rural) areas. For example, qualities of agricultural soils are not important in the urban context. However, it becomes more complicated when an intervention is located in a rural area but has clear, and often intended, outcomes for urban areas. Then the selection by this category is usually postponed to the “manual” step of case specific selection (chapter 6.1.2).

Finally, the perspective of evaluation differentiates effects of an intervention which occur through its simple being in place (e.g. costs, landscape qualities, ecological aspects) and those effects which occur only in coincidence with flooding (e.g. losses avoided etc.). Two different perspectives are considered. The first perspective refers to the typical case of ex-post evaluation in light of single or multiple flood event(s). The second perspective considers the evaluation of outcomes, which materialise independently from flood events. The latter is relevant for evaluations of interventions after

(21)

implementation but before their first exposure to flooding. Thus, the second perspective primarily applies to economic costs and side effect of the intervention.

6.1.2 Step 2: Case-specific selection of indicators

Indicators remaining from pre-selection in most cases do not yet constitute the final indicator set. They usually need further individual selection using the specific knowledge about the case, the objectives of the intervention or considering specific interests of involved parties. As a result, case-specific and flexible arrangements of the indicator set are possible. This step relies largely on the specific knowledge of the evaluator and can be a challenge to the evaluator’s commitment to a comprehensive evaluation. In the following, a procedure for the manual selection of effect indicators is proposed. The procedure is based on two main questions. It is applied to each single indicator of the reduced set.

Question 1

Is it possible/likely, that the object of evaluation bears (positive / negative) consequences for the aspect described by the indicator?

If the answer is “no”, the indicator is not further regarded. If the answer is “yes”, continue with Question 2.

The question initiates the detailed debate about each single indicator left after the pre-selection from the overall set. The answer to the question requires the evaluator’s detailed knowledge about the intervention logic considering expected performance as well as to the potential for economic, ecological and social effects. The answer to the question describes whether an impact on the indicator is potentially conceivable with respect to functional character, and further internal and external conditions of the intervention. However, the answer leaves open the consideration whether or to which extent the impact really occurred or whether the evaluation is feasible if seen against the background of available resources and availability of data.

Question 2: Can the effect described by the indicator be fully or to a known extent attributed to the object of evaluation?

If the answer is “no”, the indicator is not further regarded.

If the answer is “yes”, the indicator is selected and should be integrated in the evaluation design.

The question pays respect to the fact, that a system is not only impacted by a single intervention or portfolio. Many more activities and conditions can have impact on its performance. The evaluation will only be plausible, if investigated effects can really be attributed fully or to a known extent to the object of evaluation. If parallel actions in other fields do not allow this attribution, evaluation of the indicator may become obsolete. As a result, only those effects can be described by an ex-post evaluation, that can be clearly related to the interventions of interest.

6.2 Indicator selection for portfolios

The approach to indicator selection in cases with portfolios is generally the same as described for single interventions (chapter 6.1). However, here the speciality of different effect spectra of included interventions must be considered. For example, a portfolio consisting of measures to avoid the ingress of flood water into buildings and the evacuation of mobile goods allows the expectation of more effects than for only one of these. Indicator selection for portfolios is accomplished by selecting indicators for each single intervention and collating these in the evaluation design. While many indicators are representative for many interventions, single interventions may contribute more specific aspects to the indicator set of the case. With this procedure, the loss of information for a comprehensive evaluation is avoided.

(22)

7. Data acquisition (Action 3)

Data acquisition is the first direct step towards evaluation results. The raw data provided by this step are the basis for the analysis of effects and the following evaluation of the criteria (Action 4). As a result, data acquisition should deliver different types of information to enable further evaluation. Three main types of information are important for each indicator. Figure 6 describes the relationship of the types of information.

• Baseline state - describing the state of the condition of an indicator prior to the implementation of an intervention

• Target state - describing of how an intolerable baseline condition shall be developed by the intervention (refers to intended effects only)

• Observed state – describing the observed development of the indicator (can be identical or even beyond the target level)

Target state

Observed state

Objective

Effect

Baseline state

Levels of information in ex-post evaluation

Figure 6: Types of information in ex-post evaluation (Olfert in prep.)

The baseline state describes the state of an indicator before the intervention. This can be a certain intolerable state of an element of the flood risk system, often the actual impetus for an intervention. But, this can also relate to properties not primarily addressed by the intervention. The intolerability of a state can be more or less static – this means already in place when discovered. In such a case, identification of the baseline state is comparably certain. Or, the state can be changing due to other conditions. With other words it might be in transition from a still tolerable current state to an intolerable state in future or from an already intolerable to an even more intolerable expected state. The baseline condition should describe the most representative state which is observed or expected at a certain point of time if no intervention is implemented. The application of prognoses can be valuable where a forecast is sensible and achievable.

The observed state describes the actually measured state of an indicator at a certain point of time after implementation of an intervention. Here, it is important to be aware whether the found state is static, or whether it is still in state of development. This can have considerable impact on the interpretation of evaluation results. Figure 6 shows this level schematically somewhere between baseline state and target state. In reality, the observed state can be at any other point beyond, below or identical with the target or baseline state. Particularly with regard to unexpected and undesired (side-)effects the observed development can also worsen in comparison with the baseline condition.

The target state describes the state of an indicator as intended by the intervention. This is mainly important to derive the objective of an intervention which is indispensable for the evaluation of effectiveness. The objective is described by the margin between the identified baseline and target states. Ideally, the target state is stated prior to implementation or it can be interpreted from project documentation. If both is not the case, the target level can be determined by choosing a representative target level or by (ex-post) agreement of involved stakeholders. The target state has only limited

(23)

applicability for not intended effects. However, for some indicators binding stipulations exist that might be applicable (e.g. limnological conditions defined by the WFD), even if not intended.

These three types of information are essential for the evaluation of effects and effectiveness. The effect is represented by the margin of states between baseline and observed state. The margin between baseline and target states indicate the development intended by an intervention and thus expresses its objective. Finally, effectiveness is determined by relating the measured effect to the objective. Effectiveness is expressed by the proportion to which the objective is fulfilled.

(24)

D THE EVALUATION – FROM DATA TO FINDINGS

8. Analysing effects (Action 4a)

Effects are outcomes of measures and instruments which can be intended and unintended, direct and indirect as well as short term and long term. The description of effects consists mainly of two elements: the identification of the type and the description of the qualitative or quantitative impact of the intervention. On the one hand, effects provide valuable information about the extent of impact the intervention has on the aspects of interest. On the other hand, effects are indispensable for the valuation of these outcomes against certain values in order to determine effectiveness, efficiency and robustness.

Effects can be observed at different scales for different measures and instruments. Many central flood protection measures such as retention dams or dike systems impact large areas onsite (retention basin) and along considerable sections of the river downstream. Other measures, such as shorter flood walls or ring dikes around smaller areas, can have rather local effects e.g. affecting only parts of a flood plain. Again other interventions, such as flood proofing, can be restricted to single objects. Instruments apply to defined areas for which they convey certain regulations, financial incentives or other information. In all these cases, effect is the change of state of an indicator, but with a different special reference.

In general, each single effect (E) is described by the margin between the baseline state (state without intervention, e.g. loss potential) and the observed state (actually incurred losses) after implementing the intervention (cf. Figure 6).

bs os

E= −

with os: observed state, e.g. incurred losses (losses actually incurred/observed including all relevant loss categories in all considered flood events)

bs: baseline state, e.g. loss potential (expected losses without intervention including all relevant loss categories in all considered flood events)

Depending on the type of indicator, the information available on the different states of the related element can be treated in two different ways to describe the effect:

Effects of states can be cumulated, e.g. to describe the impact on economic flood losses:

∑

= = n i i a E E 1

Effects can be taken as average, e.g. to describe the impact on the flood water level:

n

E

n i i b

∑

=

1

with Ei =

observed effect (E) on the single affected element (i)

n = total number of affected elements

However, the allocation of the outcomes to the evaluated intervention requires the well-founded assumption of a cause-effect relationship between intervention and the change of state of an indicator (intervention logic).

(25)

Example for calculating effects

Object of evaluation

Portfolio of flood proofing and evacuation measures (incl. shielding/sealing of openings, pumping of ingress water, relocation of mobile goods and installations (example adapted from Olfert 2007)

Indicators for the intended effect (examples)

Direct economic losses avoided (incl. structural and inventory losses) Baseline state

Potential for losses at structure and inventory up to € 35.000 Determination of effects

Indicator: Direct economic losses avoided (incl. structure, installations and inventory)

The effect is described by the difference of potential losses and actually incurred losses, based on estimations of affects land users.

bs loss potential for the flood event: € 35.000 (15.000 structure and installations, 20.000 inventory). os incurred losses: € 8.000 (€ 7.000 structural, € 1.000 inventory)

E Effect (direct economic losses avoided): € 27.000

Text box 1: Example for calculating effects

9. Evaluating effectiveness (Action 4b)

9.1 Effectiveness - the degree of goal achievement

Effectiveness describes, to which extent an intervention achieved objectives. The determination of effectiveness is based mainly on intended effects. These are confronted with formulated or assumed objectives.

According to most usual objectives of measures and instruments, primarily hydrological/hydraulic, socio-cultural and economic aspects are of interest for the investigation of effectiveness. However, in the light of the raising formalisation in the ecological sector (e.g. by the Water Framework Directive) effectiveness also becomes evaluable with regard to ecological (particularly limnological) indicators. For example, goals formulated in the Water Framework Directive can be applied for all activities related to water bodies. In this case, goals of the WFD are considered implicitly.

Table 1 to Table 3 illustrate the principle of transformation of effect indicators (Annex 6). The examples are given taking recourse to indicators of effects listed above. However, the determination of effectiveness also requires the consideration of the objective connected to a certain effect. This intention cannot be reflected in the formulation of an indicator but is individual to each single evaluation.

Beyond these, also other aspects can be expressed in terms of effectiveness which are neither based on effects nor which constitute actual risk reduction in a narrow sense. However, these can be important to properly consider the contribution of an intervention. For example, warning is essential as trigger for contingent measures, but does not provide risk reduction by itself. As a result, its performance hardly can be measured by outcomes in terms of loss reduction. But, the contribution can be verified through specific indicators such as the accuracy or penetration of warning (cf. Parker et al. 1994). Both can clearly be related to certain objectives such as the targeted lead time and the addressed population. Such indicators of effectiveness are considered in an extra category – the “intervention specific effectiveness”. Table 4 (Annex 6) gives a non exhaustive outline of possible indicators.

Guideline for ex-post evaluation of measures and instruments in flood risk management

Integrated Flood Risk Analysis

and Management Methodologies