Exploratory modeling
and analysis
A promising method to deal with
deep uncertainty
A promising method to deal with deep uncertainty
Proefschrift
ter verkrijging van de graad van doctor aan de Technische Universiteit Delft,
op gezag van de Rector Magnificus prof. dr. ir. J.T. Fokkema voorzitter van het College voor Promoties,
in het openbaar te verdedigen op maandag 28 april 2008 om 10.00 uur door
Buyung AGUSDINATA
Ingenieur Luchtvaart en Ruimtevaart geboren te Mataram Indonesië
Prof. dr. W.E. Walker
Samenstelling promotiecommissie: Rector Magnificus, voorzitter
Prof. dr. ir. W.A.H. Thissen, Technische Universiteit Delft, promotor Prof. dr. W.E. Walker, Technische Universiteit Delft, promotor Prof. dr. J.J.C Bruggink, Vrije Universiteit Amsterdam
Prof. dr. R.J. Lempert, Pardee RAND Graduate School, USA Prof. dr. R.M. Cooke, Technische Universiteit Delft
Prof. dr. ir. J. Rotmans, Erasmus Universiteit Rotterdam Prof. dr. ir. M.P.C. Weijnen, Technische Universiteit Delft
ISBN 978-90-71382-26-0
Cover photo: Girih patterns (private collection)
(Reference on Girih patterns: Lu et al., 2007, Decagonal and Quasi-Crystalline Tilings in Medieval Islamic Architecture, Science 315, p.1106)
This research has been supported by the NGI Foundation and the Delft Research Centre for next generation infrastructure
© 2008. D.B. Agusdinata. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the pr ior permission in writing from the copyright owner.
“Your task is not to seek for Love, but merely to seek and find all the
barriers within yourself that you have built against It.”
-Rumi
for my parents Ibu and Ayah,
my wife Iki, my daughter Khairunisa, and my son Gibran
I would like to express my gratitude to the many individuals who have directly or indirectly contributed to this work. Without their support, this dissertation would not be possible. I, for sure, will miss to mention some of them.
First of all, to my supervisors Wil Thissen and Warren Walker. I would like to thank them for giving me a lot of freedom in pursuing my research and for pushing me to the limit. They are the ones whose quality on critical judgment and attention to both the broad picture and details I really admire. Their influence will follow me into my future career.
To the individuals with whom I collaborated in writing papers during my research: Vincent Marchau, Dan DeLaurentis, Jan-Willem van der Pas, Lars Dittmar, and Ivo Wenzler. I thank them for the time, valuable ideas, challenge, and encouragement they have given me. With them I also shared the moments of the pressure from deadlines and deadlocks, the nervous waiting for editor’s decision, and some of the the yes! moments of paper acceptance.
To Steve Bankes and Rob Lempert of Evolving Logic and RAND who not only generously provided the CARs software but also the training and support.
To Pieter Bots who was involved in the early stage of the research and whose enthusiasm are contagious.
To Scott Cunningham who often pointed to some interesting ideas and useful tools.
To the members of my PhD peer group: Telli van der Lei, Geertje Bekebrede, Igor Nikolic, and Anish Patil. I benefited from their constructive feedback.
To colleagues in the Policy Analysis Section of the Faculty of Technology, Policy and Analysis: Els van Dalen, Bert Enserink, Jill Slinger, Leon Hermans, Alexander de Haan, Monique and Gonny, Niki, Heleen, PJ and others, whose company and support helped me going through the research. I also appreciate their visits to share the joy of the birth of my two children.
To the Dutch and Indonesian community in the Netherlands: Nina and Richard Zontjes, Siti Wurian, Sajida Abdusattar, Onder, Rini Zanur, Edy Santosa, Danny Soetanto (who also pointed to the rough set approach), Hans van der Meulen,
To Klaas and Corry Smit who show me and my family one of the best examples of Dutch hospitality.
To my parents, Ibu and Ayah, whose prayer provides inner strength to me.
Finally, to my wife Iki whose care, prayer, and affection to me and to our children Khairunisa and Gibran I value most. These three special individuals are the constant reminder for me that besides the exciting world of research, there is another colorful part of life.
Salam (peace).
Table of Contents
Acknowledgement
viiChapter 1 Introduction and problem statement
1.1 Background 1
1.1.1 High stake policy problems with large uncertainties 1 1.1.2 Successes and failures in dealing with uncertainty 2 1.1.3 Increased realization for the importance of dealing
with uncertainty 4
1.2 Classes of policy problems 4
1.2.1 The changing nature of policy problems 4 1.2.2 The development of policy approaches and analytical tools 5 1.3 Overview of some prevailing policy approaches to deal with uncertainty 7 1.3.1 Do-nothing policy approach 8
1.3.2 Delay policy approach 8
1.3.3 ‘Optimal’ policy approach 9
1.3.4 Static robust policy approach 9
1.3.5 Adaptive policy approach 9
1.4 Quantitative analytical methods for dealing with uncertainty 9 1.4.1 Analytical methods for dealing with uncertainty 10 1.4.2 Information gained from methods for dealing with uncertainty 10 1.4.3 The progress of the development of analytical methods for
dealing with uncertainty 11
1.5 Exploratory modeling and analysis 12
1.5.1 A brief account of exploratory modeling and analysis 12
1.5.2 The status of EMA 12
1.5.3 EMA is worth further developing 13
1.6 Research Questions 13
1.7 Outline of the dissertation 14
References 15
Chapter 2 Research methodology and selection of application cases
2.1 Introduction 19
2.2 Defining the scope and specifying a research methodology 19
2.3 Research approach 20
2.3.1 Research philosophy 20
2.3.2 Research strategy 21
2.3.3 Choice of research instruments 22 2.3.4 Consequences of research instrument choice 23
2.4 Selection of application cases 23
2.4.1 Typology of policy problem to guide selection 24 2.4.2 Selection criteria and positioning of the application case 26
2.4.3 Selected application cases 26 2.4.4 The choice of application cases satisfy criteria
for methodological development 28
2.5 Evaluation framework for EMA 28
2.5.1 Specification of added values 28 2.5.2 Approaches to assess the added values 29 2.6 The structure of the application cases 29
References 30
Chapter 3 A conceptual basis for dealing with uncertainty
3.1 Introduction 33
3.2 Philosophical views underlying the notion of uncertainty 33 3.2.1 Positivism vs. social constructivism 34 3.2.3 On the validity of scientific knowledge 34 3.3 The notion of uncertainty from a system perspective 35
3.3.1 The nature of uncertainty 35
3.3.2 The location and level of uncertainty 36 3.3.3 Additional characterizations of uncertainty 38
3.3.4 Deep uncertainty 39
3.4 Conceptual basis for dealing with uncertainty 39 3.4.1 Probability concept and theory 40 3.4.2 Ways of reasoning under uncertainty 41 3.4.3 Normative theories of decisionmaking 44 3.4.4 Descriptive theories of decisionmaking 46
3.5 Conclusions 48
References 48
Chapter 4 State of the art of exploratory modeling and analysis
4.1 Introduction 53
4.2 The fundamental elements of exploratory modeling 54 4.2.1 The core ideas of exploratory modeling 54 4.2.2 Exploratory modeling challenges prevailing
concepts and practices 54
4.2.3 Methodological challenges to exploratory modeling 56
4.3 The policy analysis framework 59
4.4 Major procedural elements of exploratory modeling and analysis 61 4.4.1 Step 1: conceptualize the policy problem 61 4.4.2 Step 2: specify the uncertainties relevant for policy analysis 62 4.4.3 Step 3: develop a computer model 67 4.4.4 Step 4: perform computational experiments 67 4.4.5 Step 5: specify a criterion for choosing a policy 70 4.4.6 Step 6: explore and display the outcomes of computational
experiments to reveal useful patterns of system behavior 71 4.4.7 Step 7: making policy recommendations 77
4.5 Summary 78
Chapter 5 Electricity Power Plant Investment
5.1 Introduction 81
5.2 Real options analysis 81
5.2.1 Real options 82
5.2.2 Real options and uncertainty 83
5.3 A simple electricity power plant investment model 84
5.3.1 The real option investment 84
5.3.2 The system model 85
5.3.3 Investment alternatives and characteristics 87
5.3.4 Uncertainty category and treatments 87
5.4 Applying exploratory modeling to support real options analysis 89
5.4.1 Plausible future scenarios 89
5.4.2 Computational experiments 90
5.4.3 Decision criteria 90
5.5. Insights from EMA 91
5.5.1 Determination of option value 93
5.5.2 Determination of regret value of the real option 94
5.5.3 The robustness of investment performance 95
5.5.4 Seeking further future conditions that might turn a robust decision into a failure 99
5.5.5 Summary on the insights obtained from EMA 100
5.5.6 Final decision and corrective responses to improve decision performance 100
5.6 Insights from existing methods (Monte Carlo simulation) 101
5.6.1 Specify probability density function 102
5.6.2 Specify correlations between input variables 103
5.6.3 Sampling the joint probability density functions 104
5.6.4 Perform sensitivity analysis 104
5.6.5 Perform robustness analysis 105
5.7 Comparing added value of EMA and traditional methods 106
5.7.1 Limitation of the real option set up 106
5.7.2 EMA comparison with existing methods 107
5.7.3 Added value measures 108
References 111
Chapter 6 The Implementation of Intelligent Speed Adaptation
6.1 Introduction 1136.2 Problem specification 114
6.2.1 Problem statement 114
6.2.2 A simple model of the road safety system 115
6.2.3 Specifying the system model and the uncertainty ranges 118
6.2.4 Definition of success under uncertainty 125
6.2.5 Complete policy system space for computational experiments 125
6.2.6 Basic results 126
6.3 Application of EMA to support an adaptive policy design 128
6.3.1 The design of an adaptive policy 129 6.3.2 Step 1: Specifying the policy problem 130
6.3.3 Step 2: Assembling a basic policy 131 6.3.4 Step 3: Specifying the rest of the policy 132 6.3.5 Step 4: Learning from real world experience 134
6.3.6 Step 5: Adapting the policy 136
6.3.7 Summary 137
6.4 Multi-criteria analysis for ISA implementation 138 6.4.1 Multi-criteria analysis for ISA implementation 138
6.4.2 Step 1: Conceptualization 139
6.4.3 Step 2: Specification of the uncertainties 140
6.4.4 Step 3: Integration of Analytic Hierarchy Process (AHP) 144 6.4.5 Step 4: Performing computational experiment 147
6.4.6 Step 5: Specification of a robustness criterion
for choosing ISA policy 147
6.4.7 Step 6: display and analysis of insights from EMA 148
6.5 Evaluation of EMA added value 154
6.5.1 Prior work on dealing with uncertainty in representing
the road safety system 154
6.5.2 Existing approaches in dealing with uncertainty in MCA 155
6.5.3 Assessment of EMA added values 157
References 159
Chapter 7 Policy design to reduce carbon emissions in the Dutch
household sector
7.1 Introduction 163
7.2 The policy issue in the Dutch household sector 164 7.3 A conceptual framework to deal with uncertainty and complexity 165
7.3.1 System-of systems perspective 166
7.3.2 System-of-Systems lexicon 167
7.3.3 Conventional representation of the energy sector 168 7.3.4 Representation of the energy sector from a SoS perspective 169 7.3.5 Transformation from a system-of-systems (SoS) into
a system of policy systems (SoPS) 171 7.3.6 Specification of the system of policy systems 173
7.3.7 Summary 182
7.4 EMA to support adaptive policy design 184
7.4.1 Computer model 184
7.4.2 Computational experiments 187
7.4.3 Trajectories of carbon emission reduction 188 7.4.4 Circumstances required to achieve the 2025 target 190 7.4.5 Conditions and guidance for policy adaptation 191
7.4.6 Analysis of Case1 191
7.4.7 Analysis of Case2 194
7.4.8 Implications for policy design 195
7.5 Evaluation of EMA added values 197
7.5.1 Comparison with safe landing approach 197
7.5.2 Value added assessment 199
Chapter 8 Sampling and visualization within exploratory modeling
and analysis
8.1 Introduction 205
8.2 Establishing an appropriate number of samples 206
8.2.1 First Analysis 206
8.2.2 Second analysis 211
8.3 Generating and presenting EMA results 213
8.3.1 Explorative scorecard approach 215
8.3.2 ‘If-then’ rules by the rough set approach 221
8.3.3 ‘If-then’ rules using CART 224
8.3.4 ‘If-then’ rules using CART’s box plot 225 8.4 Comparison of EMA visualizations with those of existing methods 227
8.4.1 Analysis of variance (ANOVA) 227
8.4.2 Factor interaction analysis 228
8.4.3 Value added assessment on visualization 229
References 231
Chapter 9 Added values, reflections, and further study
9.1 Introduction 233
9.2 Insights from EMA 234
9.3 EMA insights to support policy design 236
9.4 Comparison with traditional uncertainty analysis methods 238 9.4.1 Added value in generating insights 239 9.4.2 Added value in using insights to support policy design 243 9.4.3 EMA added values and additional cost of resources 244 9.5 Contributions to and lessons from EMA development 245
9.5.1 Model building 246
9.5.2 Performing computational experiments 247
9.5.3 Making inferences 249
9.5.4 Policy design support 251
9.5.5 Summary of Contributions in furthering EMA development 252
9.6 Reflections 253
9.6.1 On the risk attitude in decisionmaking 253 9.6.2 On the scenario versus the probabilistic approach 254
9.6.3 On the concept of plausibility 256
9.6.4 On overall versus relevant insight 257 9.6.5 On the system of system perspective 259
9.7 Some personal accounts on EMA 260
9.7.1 Application cases as vehicles for learning 260 9.7.2 Human process aspects of EMA development and application 261
9.7.3 Acceptance of EMA 262
9.8 Further study 262
References 264
Appendices
Appendix 1: Formulation of the investment model in
Appendix 2: An interpretation of translational and transformational
relationships for ISA 267
Appendix 3: Results from the application of rough set approach to
the household heating case 272
Executive summary 275
Samenvatting 279
Chapter 1
Introduction
and problem statement
“If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties”
- Francis Bacon
1.1 Background
In our life, the consideration of uncertainty does not often play a big role in most decisions we make. The reason is that either the choices and consequences of the action are so obvious or the issue is not significant enough to require a substantial consideration. Most of the time, as psychologists have confirmed, our decisions are carried out heuristically limited by our capacity to process information (e.g. Simon, 1957; Tversky, 1972). However, there are occasions, when the stakes are so high and consequences of alternative actions are so uncertain that a systematic consideration of uncertainty is warranted or even required (Raiffa, 1968, p. ix). 1.1.1 High stake policy problems with large uncertainties
Public decisionmakers are often faced with such high stake issues. These decisionmakers, who are accountable to their constituents, share the goals of achieving and improving public goods (Walker, 2000, p. 11). Thus, recognizing the stakes at hand, policymaking involves considering many issues, often requiring a considerable amount of effort and resources. One of the issues is dealing with uncertainty.
There are several areas in which dealing with uncertainty is crucial. One important area is the design and planning of infrastructure. This area has a high stake because “our infrastructures constitute the physical framework within which our economy and society operate” (Hansman, Magee, deNeufville et al., 2006, p. 147). Disruptions in infrastructure, therefore, pose considerable threats to some of the very pillars of modern life.
Despite this fact, until recently, the provision of infrastructure has more or less been taken for granted. However, recent structural changes, such as deregulation in the energy sector, have revealed infrastructure’s vulnerabilities. Public dissatisfactions began to surface as the lack of capacity, reliability and vulnerability of such provisions become increasingly prevalent causing politicians to take serious considerations. At the same time, many infrastructures have appeared to be inert to change because these infrastructures have long technical life time and are deeply interwoven in our social, economic, and political structure. Lack of systematic knowledge in addressing uncertainties may contribute to ad-hoc political decisions, which may block timely adaptation and discourage further investment in the infrastructure system (Weijnen, ten Heuvelhof, Herder et al., 2003).
The issue of climate change also epitomizes such a high-stake and large- uncertainty decision problem. The doubts on whether climate change is taking place have partly (if not largely) subsided due to overwhelming scientific evidence (e.g. Stern, 2007). The stakes for not taking actions are so high that it is no longer a viable option. Yet, given this converging agreement, uncertainties remain surrounding the issues, such as to what extent and when measures to mitigate the effects of climate change should be implemented. Many debates are still going on in the policy analysis arena concerning the appropriate ways to handle the uncertainties that still surround climate change (e.g. Morgan, Kandlikar, Risbey et al., 1999; Lempert, Nakicenovic, Sarewitz et al., 2004).
1.1.2 Successes and failures in dealing with uncertainty
Successes and failures of handling uncertainty have been recorded. Below, we provide some examples from the domain of private and public sector. Within the public sector, we focus on the examples from the field of infrastructure design and planning.
Private sector
In the private sector, decision analytic methods have been very useful tools for strategic business planning. It has been argued that decision analysis can aid decisionmakers in formally formulating and guiding ill-structured, complex organizational decision problems (e.g. Bunn and Thomas, 1977). Another example of successful tool application is Shell’s (an oil company) pioneering use of scenario analysis that helped it weather the energy crisis in the mid 1970’s (Wack, 1985; van der Heijden, 1996). Scenario analysis (and scenario thinking) not only can support testing different alternative decisions under various future scenarios but also can facilitate change in decisionmakers’ mind set on thinking and communicating about how the future may unfold (Schwartz, 1997). Furthermore, the capacity to adapt amidst a changing and uncertain business environment has been one of the key factors that makes companies such as DuPont and Procter & Gamble able to sustain their identities for more than 100 years (see. e.g. de Geus, 1997; Collins and Porras, 2000).
It is worth noting, however, that there seems to be not so much innovation taking place regarding the scenario method. For example, the Hart-Rudman Commission
found that around 70% of 50 scenario study projects are still based on or an adaptation of Shell’s scenario process (Hart and Rudman, 2000). This process puts a strong emphasis on internal consistency on the logical chain of cause-effect relationships of key variables, a practice that is prone to inherent biases (MacKay and McKiernan, 2004). These biases include, for example, overconfidence bias (‘I knew it all along’) and creeping determinism (‘it could not happen in any other way’). These biases can produce scenarios that lack a robust and balanced view of affairs by frequently missing ‘soft signals’ and small patterns and failing to generate ‘broader possibilities’ (ibid, p. 71). Another study suggests that the scenario approach may fail to deliver its promise because in a considerable number of cases (i.e. 11 out of 22 scenario studies considered) the exploration of potential future discontinuity/ surprise has been excluded from the scenario analysis (van Notten, Sleegers, and van Asselt, 2005).
Public sector
Surprisingly enough, unlike the private sector, it is not that easy to find documented success stories (though there are enough failure cases) in the public sector. The Mont Fleur scenario process involving various stakeholders has been considered instrumental in aiding the peaceful transition of South Africa from apartheid to democracy (Roux, Maphai, and Mohr, 1992).
In the field of global climate change, the failure to deal with disagreement in value judgment, especially on the value of human life has led to controversy (Masood and Ochert, 1995). One study of the Intergovernmental Panel on Climate Change (IPCC) applied different criteria to assess losses in rich and poor countries. For example, a loss of an individual in a poor country was valued at US$100,000, one-fifteenth of the value in a rich country.
Infrastructure design and planning
In the area of infrastructure design and planning, failing to deal with uncertainty has often led to erroneous findings which mislead policy recommendations. Typically, the benefits of an infrastructure project are overestimated, while the costs are underestimated. In the area of transport planning, for instance, Flyvbjerg et al. carried out a survey on 210 big infrastructure projects to find the discrepancies between the forecast and actual travel demand (Flyvbjerg, Holm, and Buhl, 2005). One major finding is that the travel demand forecasts had been significantly overestimated; that is by 106% in 9 out of 10 rail projects and by more than 20% for half of all road projects. Furthermore, the accuracy of the forecasts has also not been improving over a 30-year period. Some of these failures involved systematic biases (e.g. Kahneman and Tversky, 1979) and in some cases these systematic biases may be intentional (e.g. to secure funding). Similarly, Walker highlights a forecasting failure of underestimating air transport demand for Schiphol Airport in the Netherlands (Walker, 2000). This discrepancy in the demand forecast can be ascribed mainly to failing to foresee structural changes such as the impacts of industry liberalization and airline alliances.
The implications of such failures in dealing with uncertainty have been significant. Specific solutions to infrastructure (e.g. Boston Logan’s Terminal E) that fail to
factor in future changing conditions are often rendered obsolete by the time they are implemented (de Neufville, 2004). Such failures may result in substantial financial losses like the one that was incurred in the Denver International airport project (Szyliowicz and Goetz, 1995).
In economic terms, failure to adequately consider uncertainty may lead also to lost opportunities, as in the case of Schiphol Airport, which may have a far reaching impact in affecting the country’s overall competitiveness. In the area of climate change, such failures may potentially cause a great deal of human suffering (Morgan, Kandlikar, Risbey et al., 1999). And amidst these failures, the credibility of analysts (including forecasters) and the practice of quantitative analysis itself, might be at stake (Wildavsky, 1993; Piermartini and Teh, 2005).
1.1.3 Increased realization for the importance of dealing with uncertainty Faced with such consequences, the importance of properly handling uncertainties has been increasingly recognized. Many policy institutions have responded by encouraging and facilitating good practices for handling and managing uncertainty. The Intergovernmental Panel on Climate Change (IPCC), for instance, have set requirements for explicit and consistent treatment of uncertainty, which aims to provide common approaches and languages to develop expert judgments evaluation of uncertainties, and communication of uncertainties and confidence in assessment findings (IPCC, 2005). The Netherlands Environmental Assessment Agency, triggered by public revelation of its poor practice, has produced a check list of structured process for reviewing uncertainty (Janssen, Petersen, van der Sluijs et al., 2004). It contains questions such as “Can you imagine a scenario by which it turned out that the main results were substantially incorrect or not valid?”, which are aimed to make sure that uncertainties are assessed and communicated within the context of policy advice.
Furthermore, decisionmakers such as the European Commission have adopted a precautionary principle. The principle contains guiding rules that are established for measures or actions taken under inadequate scientific evidence, especially when those measures put human and environment under considerable risk (European Commission, 2000).
In the area of infrastructure design and planning, a paradigm change has been suggested. Uncertainty should no longer be seen as something to be avoided but something to benefit from. The notion of real options allows a proactive approach to deal with uncertainty (e.g. designing flexibility in the system) rather than the previous norm of a passive approach (e.g. creating multiple redundancy) (de Neufville, 2003).
1.2
Classes of policy problems
1.2.1 The changing nature of policy problems
The nature of some of the policy problems is changing. These policy problems increasingly involve broader scales in terms of complexity, spatial, temporal, and
socio-political aspects than the scope of conventional policy analysis. For example, global climate change has become an issue of both regional and global concern (Morgan, Kandlikar, Risbey et al., 1999). This issue requires resolutions that cut across significantly more cultural and political boundaries than those of a nation or an organization. In addition, the time span considered for the policy problem covers the lifetime of several generations into the future.
This type of policy problems has been characterized as ‘wicked’ (Rittel and Webber, 1973). For wicked problems, there is no definitive and objective problem description and solution. One major characteristic of such problems is that it is almost impossible to define the boundaries of the problem. Consequently, alternative explanations or models to describe the same phenomenon, which in turn determine the nature of possible solutions, are justifiable. Moreover, the solution itself becomes contentious because in a pluralistic society with diverse value judgments, what is considered as a ‘public good’ is often disputable.
Besides the increased complexity and ‘wickedness’ of policy problems, of particular interest are policy problems in which decisionmakers need to deal with large degrees of uncertainty. For example, over a long time span, the realizations and the time-varying relationships of relevant factors in a policy problem are difficult to predict. A large degree of uncertainty may also be present for a system that has not yet existed, because of the non-existence of empirical data. These kinds of situation have led to the notion of deep uncertainty, which is defined as a condition in which analysts do not know or the parties to a decision cannot agree upon (1) the appropriate conceptual models to describe interactions among a system’s variables, (2) the probability distributions to represent uncertainty about key parameters in the models, and/or (3) how to value the desirability of alternative outcomes (Lempert, Popper, and Bankes, 2003, p. xii).
1.2.2 The development of policy approaches and analytical tools
It appears that each class of policy problem (as described above) can be addressed by different policy approaches (or combination of them), which in turn can be supported by different combinations of analytical methods/tools (see Figure 1-1). More precisely, an approach can be analyzed as a general framework together with a set of problems, a collection of methods and a set of goals (Bunge, 1997; p. 87).
This argument implies some dynamics in the development of policy approaches to address uncertainty in policy problems and the supporting analytical methods. It can also explain the debates surrounding the issue whether a certain approach or method is appropriate to handle a specific policy problem. A claim on a new or special class of policy problem may entail the development of a new approach, and in turn new or improved analytical methods. Such a development may be triggered by the perceived inadequacy or unfitness of the underpinning conventional methods/tools to address the emerging class of policy problem. For example, some policy problems, such as climate change, pose challenges to the assumptions on which most conventional methods/tools, such as economic cost-benefit analysis and expected utility theory, are based. Such assumptions include
among others: (i) uncertainty is modest and manageable and (ii) values are known, static, and exogenously determined (Morgan, Kandlikar, Risbey et al., 1999).
Classes of policy problem
Approaches for addressing policy problem
Analytical methods/tools supporting approaches
Class 1 Class 2 …. Class m
Approach 1 Approach 2 …. Approach n
Method 1 Method 2 …. Method p
Figure 1-1. Logical relationships among classes of policy problem, approaches, and analytical methods
In some cases, however, the policy problems themselves are not necessarily new. Nor are analysts and decisionmakers ignorant of the need to address these classes of policy problem. For example, the importance of addressing uncertainties that stem from, for instance broader time scales, has been widely acknowledged. Lempert et al. argue that quantitative analysis covering time horizons long into the future has been rarely attempted in recent years, not because analysts and decisionmakers do not deem it important, but more because there has been a lack of credible tools/ methods to perform the analysis (Lempert, Popper, and Bankes, 2003). Attempts in the past to simulate system behavior over a long-term period such as the World3 model (Meadows, Meadows, Randers et al., 1972), had been criticized for among others using too limited (and pessimistic) key assumptions, and therefore failing to account for the sensitivities of simulation results across a larger set of assumptions (e.g. Cole, Freeman, Jahoda et al., 1973). Another study that challenged the standard assumptions of the World3 model, for instance, suggests a much lower maximum sustainable size of population (Thissen, 1978). In the remainder of this chapter, we first briefly review some prevailing approaches for dealing with uncertainty and discuss their appropriateness for specific classes of problem. We then summarize some quantitative analytical methods that support the approaches. We particularly focus on an analytical method called exploratory modeling and analysis (EMA) as a promising method for supporting approaches to deal with deep uncertainty. Finally, we identify some opportunities for further development of EMA and formulate the research questions of the dissertation.
1.3
Overview of prevailing policy approaches to deal with
uncertainty
In the presence of uncertainty, policymakers have been responding in different ways. At this point, it is useful to first clarify our research position and focus regarding concept of uncertainty. Uncertainty is a broad concept that includes both ‘measurable’ and ‘unmeasurable’ uncertainty (cf. Knight, 1921). The measurable uncertainty can be represented by probability distributions, whereas for the unmeasurable uncertainty, the probability distributions are unknownable. In this research, we focus on the class of uncertainty called deep uncertainty (see the definition in Section 1.2.1).
To begin with, in the literature, Klinke and Renn, 2002), for example, suggest three strategies for managing uncertainty, which include: so-called precaution-based, risk-precaution-based, and discourse-based strategies. First, the risk-based strategies are appropriate when both the probability of occurrence and extent of damage are relatively well known. The risk-based strategies require concentrating the efforts on reducing the disaster potential (e.g. reduce the probability of a core meltdown in a nuclear power plant). When there is a good knowledge on the damage potential, but the probability distributions are unknown, the risk-based strategies include among others more research to improve the specification of probability distributions. Second, the precaution-based strategies are appropriate when the risk potentials are characterized by a relatively high degree of uncertainty (the greenhouse effect falls into this category). In this particular situation, the traditional risk-based approach (e.g. more research to improve the specification of probability distributions) becomes counterproductive. The precaution-based strategies include among others the application of precautionary measures and the introduction of strict liability and compulsory insurance for the parties that generate the potential damage. The third category, the discourse-based strategies, is beyond the scope of our research.
From a policy design perspective, we can characterize policy approaches under conditions of uncertainty along two dimensions: (1) the nature of the decisions made under uncertainty and (2) the type of actions taken to deal with uncertainty. For the nature of the decision dimension, there are two types of policy: static and dynamic. Static policies are one-shot policies; that is the policies that are implemented once in full fashion for the whole policy time horizon. In contrast, dynamic policies are designed to change over time as the situation changes. For the action dimension, at one extreme are actions that require more knowledge to be gained before policies are implemented (i.e. reduce uncertainty), resulting in no immediate change or intervention in the system (policy delays). On the other are actions taken to change the system (i.e. the system relevant to the policy problem). This characterization results in five policy approach categories (see Figure 1-2). The five categories are the following:
• Do-nothing policy approach (static-no change in the system category) • Delay policy approach (dynamic- no change in the system category) • ‘Optimal’ policy approach (static-change the system category) • Static robust policy approach (static-change the system category)
• Adaptive policy approach (dynamic-change the system category)
For each category, some examples of the approach are given. The following five subsections describe each of the categories.
1.3.1 Do-nothing policy approach
The do-nothing policy approach implies that decisionmakers implement no policy to change the system, in hope that the negative impacts to system outcome are not too bad. Change in the system (‘optimal’ policy approach) • Predict the future and implement ‘optimal’ policy for that future
(static robust policy approach) • Identify plausible
futures and find policy that works acceptably well across most of them • Hedge against vulnerabilities/ contingencies (adaptive policy approach) • Adapt policy over
time as conditions change and learning takes place
No change in the system
(do-nothing policy approach) • No policy until the uncertainty is
resolved
(delay policy approach) • Do more research • Negotiate with other
parties for a consensus or compromise Static policy Dynamic policy
Figure 1-2. Categories of policy approaches under conditions of uncertainty
1.3.2 Delay policy approach
In the delay policy approach, the current ‘business as usual’ policy is maintained while efforts are made to reduce or better characterize uncertainty by gaining more knowledge. Doing more research is a typical example of these efforts. When the uncertainties involve behavior of other parties involved in the decision, uncertainty can be reduced by way of negotiation.
1.3.3 ‘Optimal’ policy approach
In the ‘optimal’ policy approach, decisionmakers base their policies on ‘best estimate’ models and as a result on ‘best’ (most likely) predictions about system behavior. Using this information, decisionmakers choose an ‘optimal’ policy. The extreme of this policy approach is to adopt an ‘optimal’ policy based on deterministic assumptions (i.e. completely disregarding uncertainty).
1.3.4 Static robust policy approach
In the static robust policy approach, a policy is chosen that performs acceptably well across most plausible future scenarios. To achieve a robust policy in a static way, decisionmakers can introduce multiple redundancies to make sure that the system as a whole will still function in case of major disruptions or discontinuities (e.g. Hastings and McManus, 2004). The idea of hedging against possible contingencies also belongs to this approach (Dewar, Builder, Hix et al., 1993). 1.3.5 Adaptive policy approach
The adaptive policy approach is based on the view that decisionmakers can start to implement a policy that performs fairly well across a wide range of uncertainty and then adapt it as some of the uncertainties are resolved. Uncertainties can be resolved by having provisions for learning about the real world, including a system that monitors relevant future developments (Walker, Rahman, and Cave, 2001). It should be noted though that new information may also increase uncertainty. In the same spirit, Morgan, for example, endorses the idea of devising adaptive strategies for dealing with uncertainty (Morgan, 2003). One main motivation behind this way of responding to uncertainties can be to avoid the consequences of making wrong decisions/ unexpected circumstances.
Lempert et al. has demonstrated that adaptive policies perform better than static policies because they allow mid-course adjustments and therefore can avoid serious consequences when the estimates turn out to be wrong (Lempert, Schlesinger, and Bankes, 1996). In an attempt to make the adaptive approach operational, Walker et al. have proposed a structured conceptual framework for implementing the adaptive approach that includes monitoring signpost variables and establishing threshold values that trigger responses to correct or capitalize on opportunities (Walker, Rahman, and Cave, 2001). Adaptive policies can be enabled, for example, by creating flexibilities in the decision (e.g. to scale up or down) and in the system (e.g. overbuilt) using ‘real options’ (e.g. de Neufville, 2004). In this way one can actually ‘benefit’ from uncertainty in the sense that the larger the uncertainties are, the more value the flexibility measures would deliver.
1.4
Quantitative analytical methods for dealing with uncertainty
In making decisions under uncertainty, decisionmakers can make use of information from different sources. One source can be a system model that is used to analyze the consequences of alternative policies under uncertainty (Quade, 1989; Walker, 2000). Due to the complexity of the policy problem and the limited capacity of humans to process information, the use of a system model is often
indispensable. This section describes some analytical methods that are used together with a system model.
1.4.1 Analytical methods for dealing with uncertainty
In addition to the use of a system model, a variety of quantitative methods have been particularly developed to address uncertainty in the analysis of policy problems. Some of them and their simplified definitions are listed as follows: • Statistical/ Stochastic analysis – interprets quantitative data and applies
probability theory to estimate population parameters from samples.
• Uncertainty propagation analysis – Given a vector of system model output y = (y1,y2,…ym) and associated input represented by x = (x1,x2,…xn), uncertainty
propagation analysis answers the question “ what is the uncertainty in y(x) given the uncertainty in x ?” (Helton and Davis, 2000).
• Sensitivity analysis – Given a vector of system model output y = (y1,y2,…ym)
and associated input represented by x = ( x1,x2,…xn), sensitivity analysis
answers the question “ How important are the individual elements of x with respect to the uncertainty in y(x)” (Saltelli, Chan, and Scott, 2000).
• Scenario analysis – analyzes system model outcomes for a variety of plausible futures and model parameters.
• Decision analysis – a systematic representation of a decision problem in which three elements of the decision problem are considered: the decision to be made, the probabilities or likelihoods of future events that may emerge from the decision and can affect the decision outcome, and the decision outcome itself that can be guided by measures such as optimal expected utility (Raiffa, 1968).
1.4.2 Information gained from methods for dealing with uncertainty
Associated with each of the above analytical methods, there are various ways with which analysts try to convey the information about uncertainty aspects of a system model. Though incomplete the following list summarizes some of the ways: • Statistical/ Stochastic, uncertainty propagation, and sensitivity analysis:
Statistical and stochastic measures such as range of result values (e.g. Alcamo, 1994), confidence interval, cumulative probability distribution, mean and variance, and box plot.
• Uncertainty propagation analysis: Response surface ( a 3-D diagram) of mean values of the independent variables together with their associated probability distributions (e.g. Dowlatabadi and Morgan, 1993) and t-statistics (e.g. Zhao and Kockelman, 2002).
• Sensitivity analysis: Tornado diagram – depicts the ranking of input variables in terms of their effect on the spread of model outputs (e.g. Eschenbach, 1992). Such ranking can be also be represented using F-values derived from an analysis of variance (ANOVA) (e.g. Frey, Mokhtari, and Danish, 2003).
• Scenario analysis: Scenario plots – paints what might plausibly occur in the future as manifestations of various variables (e.g. Schwartz, 1997). In addition, scenario analysis may also be presented in time series – show different paths of variable realizations over time. In some fields the emphasis is on predicting the path by extrapolating the past trend using different sets of assumptions about
the system variables (see e.g. Bollen, Manders, and Mulder, 2004 ). In the system dynamics field, the focus is more on identifying a recognizable pattern of the future path (e.g. an exponential increase or oscillatory change) (see e.g. Meadows, Behrens, Meadows et al., 1974)
• Decision analysis: Decision tree/ decision flow diagram – provides information about alternative decisions, future states of the system with their probability of occurrence, and the expected utility of each alternative decision . The ways of presenting and visualizing the uncertainty information about the system model highlight different aspects of the information. For instance, scenarios emphasize the multiplicity of future states of the world and possibilities of surprises or trend breaks without suggesting their likelihood. The rest of the presentations, however, tend to emphasize the likelihood of outcomes (i.e. estimation) in the face of the uncertainty of model inputs.
1.4.3 The progress of the development of analytical methods for dealing with uncertainty
Scenario analysis has been used to test how policies will perform in multiple plausible circumstances. The number of scenarios considered in most scenario analyses is still limited, typically to around five scenarios.
One obstacle limiting the number of scenarios may partly lie in the perceived lack of computing power to perform the analysis as well as the inability to adequately present the model outcomes from a larger number of scenarios. Another explanation may be that analysts consciously limit the number of scenarios to a subset of several extremes and leave them in qualitative terms in order just to provide a global picture of the future and with the hope to challenge conventional world views (e.g. van Asselt, Rotmans, and Rothman, 2005). Finally, the limited number of scenarios considered may have to do partly with the difficulties in quantifying some relevant ‘soft’ factors (Schwartz, 1997).
Some progress, however, can already be found in the literature. There have been a few examples of assessments of the impact of structural changes on policy performance, mostly in some integrated assessment models on climate change carried out at Carnegie Mellon University. For example, comparisons of policy performance across more scenario variables as well as alternative model structures have been carried out (e.g. on whether for oil and gas ‘reserves will be exhausted by 2050’ or ‘new reserves will be discovered’) (Morgan and Dowlatabadi, 1996). This attempt to explore different system representations has helped the identification of conditions in which a certain policy will cause an unexpected outcome (e.g. carbon taxes may actually increase carbon emission). But, in these cases, the assessment of the robustness remains heavily dependent on the definition of the probability distributions of the input variables.
Among these methodological developments, one alternative method that is of particular interest is exploratory modeling and analysis. The next section describes this method.
1.5 Exploratory modeling and analysis
1.5.1 A brief account of exploratory modeling and analysis
Exploratory modeling and analysis (EMA) is based on the pioneering work of Bankes (the original name of the method is exploratory modeling) (Bankes, 1993). One major driver behind EMA development is to challenge the idea of using a model as a surrogate for the real world system in order to predict the behavior of the system. Though successful to represent closed systems like engineering artifacts, the view of substituting the real world system with a model, however, is problematic when applied to policy problems. Since open systems are involved, defining a policy problem is subject to among others ambiguity and uncertainty (as the ‘wicked’ term, for example, implies). This is why, although sufficiently verified and validated, a model can still often be characterized as ‘bad’ (Hodges, 1991) or even ‘wrong’ (Sterman, 2002). Having said this, however, such a model can still be useful. One role that will be argued for and demonstrated in this dissertation is to use a model as a hypothesis generator to reason about the behavior of the system under uncertainty.
EMA is meant to specifically cope with deep uncertainty. In analyzing system behavior, it relaxes model assumptions by performing computational experiments: calculations of model outcomes across a large ensemble of plausible system representations. Enabled by the availability of relatively vast and cheap computing power, the computational experiments cover, among others, uncertainty in the external scenarios, model parameters, and structural uncertainty.
EMA helps to explore system behavior by taking as broad assumptions into account as are useful and resources allow. EMA treats the outcome of one plausible system representation as one hypothesis about system behavior and asks what if a given hypothesis were correct. By exploring the implications of a large set of such hypotheses (thousands to hundreds of thousands), one can explore which statements about system behavior are generally true.
1.5.2 The status of EMA
The subject of EMA is relatively new and has been developed and used within a small community (mostly within the RAND Corporation). To give an indication about how Bankes’ idea has been developed, we conducted a content analysis (see Porter and Cunningham, 2004) on the Web of Science database by tracking all the articles that quote Bankes’ article (1st tier) and all articles that quote the 1st tier articles (2nd tier). As of 2004, there were 29 1st tier and 94 2nd tier articles.
Counting the keywords used in the articles, we found that, in the area of applications, the majority of the articles are dedicated to the subject of climate change/global warming (40 counts). The next areas are on AIDS and other diseases (8 counts), military/war related issues (6 counts), production planning (6 counts), and economic policy (1 count). EMA is associated with a variety of types of analysis, such as computer simulation (27 counts), probabilistic risk assessment (10 counts), and what-if analysis (1 count). EMA can be linked to the use of models for policymaking (15 counts), devising strategies (9 counts), prediction (4 counts), and adaptive planning (2 counts).
To give some specifics, most of the research using exploratory modeling has focused on the issues of climate change (e.g. Lempert, Schlesinger, and Bankes, 1996; Lempert and Schlesinger, 2000), economic policy (Brooks and Lewis, 2002), and sustainable development (Lempert, Popper, and Bankes, 2003). In these areas, achievement can be summarized as follows. EMA enables the search for among others (i) robust policies across multiple scenarios, (ii) assumptions that drive a certain policy, and (iii) key trade-offs among policies. The authors of these papers claim that though these achievements are not new, EMA enables them to systematically obtain those achievements. EMA is still a method in progress, which leaves room for further development to solidify it.
1.5.3 EMA is worth further developing
In summary, we can establish several major drivers that lead to a strong case for further developing EMA. To begin with, it has been apparent that there is an increased realization from both decisionmakers and scientific community on the importance of appropriate handling of uncertainty, especially deep uncertainty. There has been sufficient evidence to expose the stakes at hand when failing to deal with deep uncertainty.
At the same time, the limitations of existing approaches for dealing with uncertainty have also been recognized when the policy problem involves conditions of deep uncertainty. The past two decades have witnessed a development of ideas and tools that lead to better handling of uncertainty. In particular, there has been a movement from methods that support policy based on best estimate system models to approaches that use system models to more extensively explore uncertain future scenarios as well as include structural change. With vast and cheaper computing power, more extensive analyses can be carried out at a relatively low cost.
EMA is one of the promising methods that can support such approaches. But as has been argued, the applications are still limited (e.g. in the area of infrastructure) and therefore leave some room for methodological improvements. Furthermore, there is a need to provide a more systematic procedure to perform EMA in order to enable a more widespread use of the method. These are the aims of this research.
1.6 Research
Questions
The research questions of this research revolve around the issues of a further development and application of EMA for policy analysis and to support policy design. The first three questions are intended to verify the previous claims about the perceived added values of the method and attempt to add new elements to these claims.
1. What insights can be obtained from the applications of EMA? 2. How could such insights be used to support policy design? 3. What are the added values from the application of EMA?
The fourth research question addresses methodological and technical problems encountered during the application of the method.
4. What contributions can be made to the methodological development of EMA?
1.7 Outline of the dissertation
The dissertation consists of nine chapters (see Figure 1-3). Chapter 2 sets the research methodology to answer the research questions defined in Chapter 1. It also defines criteria for selecting and briefly introducing the application cases to advance the development.
Chapter 1. Introduction Chapter 2. Research methodology and selection of application cases Chapter 3. The notion of uncertainty from a broad perspective Chapter 4.
State of the art of exploratory modeling and analysis
Chapter 5.
Analysis for power plant investment
Chapter 6.
Adaptive policy design for implementing Intelligent Speed Adaptation
Chapter 7.
Carbon emission reduction in the Dutch household sector
Chapter 8.
Visualization and sampling aspect of exploratory modeling and analysis
Chapter 9.
Conclusions, reflections, and further study
Chapter 3 sets out to discuss the notion of uncertainty from a broad perspective. Here the very notion of uncertainty is examined together with related themes such as knowledge acquisition, reasoning and decisionmaking under uncertainty, and how uncertainty is perceived and treated in some application areas.
Chapter 4 presents the state of the art of EMA. It includes analysis of previous conceptual ideas and applications. It continues with presenting some major elements of EMA as well as new techniques which will be used in the application cases.
Our approach to answering the research questions employs three application cases. Chapter 5 applies EMA to support an analysis for investment in a power plant. Chapter 6 demonstrates how EMA can support the design of adaptive policies for implementing Intelligent Speed Adaptation, a technology meant to improve road safety. Chapter 7 presents an EMA application to a case of policy design to reduce carbon emissions in the Dutch household sector. Furthermore, based on the case in Chapter 7, Chapter 8 discusses two important aspects of EMA: visualization and sampling.
Finally, Chapter 9 summarizes the answers to the four research questions. It also reflects on the different aspects of EMA development and application and makes recommendations for further study.
References
Alcamo, J. (1994). Image2.0 Integrated Modeling of Global Climate Change, Dordrecht, Kluwer Academic Publishers.
Bankes, S. (1993). "Exploratory modeling for policy analysis," Operations Research, 41(3): 435 - 449.
Bollen, J., T. Manders and M. Mulder (2004). Four Futures for Energy Markets and Climate Change, CPB Netherlands Bureau for Economic Policy Analysis. Brooks, A. C. and G. B. Lewis (2002). "Enhancing policy models with exploratory
analysis," Journal of Public Administration Research and Theory, 12(1): 129-136.
Bunge, M. (1997). "General systems and holism," General Systems, 22: 87-90.
Bunn, D. and H. Thomas (1977). "Decision-Analysis and Strategic Policy Formulation," Long Range Planning, 10(6): 23-30.
Cole, H. S. D., C. Freeman, M. Jahoda, et al. (1973). Models of Doom: A Critique of the Limits to Growth New York, Universe Books.
Collins, J. C. and J. I. Porras (2000). Built to Last: successful Habits of Visionary Companies (3rd edition), London Random House.
de Geus, A. (1997). "The living company," Harvard Business Review, 75(2): 51-&. de Neufville, R. (2003). "Real Options: Dealing With Uncertainty In Systems
Planning And Design," Integrated Assessment Journal, 4(Special Issue, No.1): 26-34.
de Neufville, R. (2004). Uncertainty Management for Engineering Systems Planning and Design, Engineering Systems Symposium.
Dewar, J. A., C. H. Builder, W. M. Hix, et al. (1993). Assumption-based planning, A planning tool for very uncertain times, Santa Monica, RAND (Doc. No.: MR-114-A).
Dowlatabadi, H. and M. G. Morgan (1993). "A Model Framework for Integrated Studies of the Climate Problem," Energy Policy, 21(3): 209-221.
Eschenbach, T. G. (1992). "Spiderplots Versus Tornado Diagrams for Sensitivity Analysis," Interfaces, 22(6): 40-46.
European Commission (2000). Communication from the commission on the precautionary principle. COM(2000) 1. Brussels.
Flyvbjerg, B., M. K. S. Holm and S. L. Buhl (2005). "How (in) accurate are demand forecasts in public works projects? The case of transportation," Journal of the American Planning Association, 71(2): 131-146.
Frey, H. C., A. Mokhtari and T. Danish (2003). Evaluation of Selected Sensitivity Analysis Methods Based Upon Applications to Food Safety Process Risk Models. Washington, DC, Office of Risk Assessment and Cost-Benefit Analysis, U.S. Department of Agriculture.
Hansman, R. J., C. Magee, R. deNeufville, et al. (2006). "Research agenda for an integrated approach to infrastructure planning, design and management," Int. J. Critical Infrastructures, 2(2/3): 146-159.
Hart, G. and W. B. Rudman (2000). 21st Century Terrorism and National Security. Washington, US Commission on National Security.
Hastings, D. and H. McManus (2004). A Framework for Understanding Uncertainty and its Mitigation and Exploitation in Complex System, Engineering Systems Symposium.
Helton, J. C. and F. J. Davis (2000). "Sampling based Methods". in "Sensitivity Analysis," A. Saltelli, K. Chan and E. M. Scott (eds). Chichester, Willey. Hodges, J. S. (1991). "Six (or so) things you can do with a bad model," Operations
Research, 39(3): 355-365.
IPCC (2005). "Guidance Notes for Lead Authors of the IPCC Fourth Assessment Report on Addressing Uncertainties."
Janssen, P. H. M., A. C. Petersen, J. P. van der Sluijs, et al. (2004). RIVM/MNP Guidance for Uncertainty Assessment and Communication.
Kahneman, D. and A. Tversky (1979). "Prospect theory: an analysis of decisionmaking under risk," Econometrica, 47: 263-291.
Klinke, A. and O. Renn (2002). "A new approach to risk evaluation and management: Risk-based, precaution-based, and discourse-based strategies," Risk Analysis, 22(6): 1071-1094.
Knight, F. H. (1921). Risk, Uncertainty, and Profit, Bradford, Lund Humphries. Lempert, R., N. Nakicenovic, D. Sarewitz, et al. (2004). "Characterizing
Climate-Change Uncertainties For Decision-Makers- An Editorial Essay," Climatic Change, 65: 1-9.
Lempert, R., S. Popper and S. Bankes (2003). Shaping the next one hundred years, New methods for quantitative Long-term policy analysis, MR-1626-RPC, Santa Monica, The RAND Pardee Centre.
Lempert, R. J. and M. E. Schlesinger (2000). "Robust strategies for abating climate change," Climatic Change, 45(1): 387-401.
Lempert, R. J., M. E. Schlesinger and S. C. Bankes (1996). "When We Don't Know the Costs or the Benefits: Adaptive Strategies for Abating Climate Change," Climatic Change, 33: 235-274.
MacKay, R. B. and P. McKiernan (2004). "Exploring strategy context with foresight," European Management Review, 1: 69-77.
Masood, E. and A. Ochert (1995). "UN climate change report turns up the Heat," Nature, 378(6553): 119-119.
Meadows, D. H., D. L. Meadows, J. Randers, et al. (1972). The Limits to Growth: A Report for the Club of Rome's Project on the Predicament of Mankind, New York, Universe Books.
Meadows, D. L., W. W. Behrens, D. H. Meadows, et al. (1974). Dynamics of growth in a finite world, Cambridge, Wright-Allen Press.
Morgan, M. G. (2003). "Characterizing and Dealing With Uncertainty: Insights from the Integrated Assessment of Climate Change," Integrated Assessment, 4(1): 46-55.
Morgan, M. G. and H. Dowlatabadi (1996). "Learning from integrated assessment of climate change," Climatic Change, 34(3-4): 337-368.
Morgan, M. G., M. Kandlikar, J. Risbey, et al. (1999). "Why conventional tools for policy analysis are often inadequate for problems of global change - An editorial essay," Climatic Change, 41(3-4): 271-281.
Piermartini, R. and R. Teh (2005). Demystifying Modeling Methods for Trade Policy, World Trade Organization.
Porter, A. L. and S. W. Cunningham (2004). Tech Mining: Exploiting New Technologies for Competitive Advantage, New Jersey, Wiley-Interscience. Quade, E. S. (1989). Analysis for public decisions, New York, Elsevier Science
Publishing.
Raiffa, H. (1968). Decision Analysis; introductory lectures on choices under uncertainty, Reading, Addison-Wesley.
Rittel, H. W. J. and M. M. Webber (1973). "Dilemmas in a General Theory of Planning," Policy Sciences, 4(2): 155-169.
Roux, P. l., V. Maphai and P. Mohr (1992). The Mont Fleur scenarios. Deeper news. 7.
Saltelli, A., K. Chan and E. M. Scott (2000). Sensitivity Analysis, Chichester, John Wiley & Sons.
Schwartz, P. (1997). The art of the long view; planning for the future in an uncertain world, Chichester, Wiley.
Simon, H. A. (1957). " A Behavioral Model of Rational Choice". in "Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting," (eds). New York, Wiley.
Sterman, J. D. (2002). "All models are wrong: reflections on becoming a systems scientist," System Dynamics Review, 18(4): 501-531.
Stern, N. (2007). The Economics of Climate Change: The Stern Review, Cambridge, Cambridge University Press.
Szyliowicz, J. S. and A. R. Goetz (1995). "Getting Realistic About Megaproject Planning - the Case of the New Denver-International-Airport," Policy Sciences, 28(4): 347-367.
Thissen, W. (1978). "Investigations into the World3 Model: Overall Model Behavior and Policy Conclusions," IEEE Transactions on Systems, Man, and Cybernetics, 8(3): 172-182.
Tversky, A. (1972). "Elimination by aspects: a theory of choice," Psychological Review, 79: 281-299.
van Asselt, M. B. A., J. Rotmans and D. S. Rothman (2005). Scenario Innovation: Experiences from a European Experimental Garden, Leiden, Taylor & Francis.
van der Heijden, K. (1996). Scenarios: The Art of Strategic Conversation Chichester, Wiley.
van Notten, P. W. F., A. M. Sleegers and M. B. A. van Asselt (2005). "The future shocks: On discontinuity and scenario development," Technological Forecasting and Social Change, 72(2): 175-194.
Wack, P. (1985). "Scenarios - Shooting the Rapids," Harvard Business Review, 63(6): 139-150.
Walker, W. E. (2000). "Policy Analysis: A Systematic Approach to Supporting Policymaking in the Public Sector," Journal of Multi-Criteria Decision Analysis, 9: 11-27.
Walker, W. E. (2000). Uncertainty: The Challenge for Policy Analysis in the 21st Century (Full professorship inaugural speech), Delft University of Technology.
Walker, W. E., S. A. Rahman and J. Cave (2001). "Adaptive Policies, Policy Analysis, and Policy-making," European Journal of Operational Research, 128(2): 282-289.
Weijnen, M. P. C., E. F. ten Heuvelhof, P. M. Herder, et al. (2003). Next Generation Infrastructures: Main proposal and subprogram descriptions, Delft University of Technology Faculty of Technology, Policy and Management.
Wildavsky, A. (1993). Speaking truth to power, New Jersey, Transaction Publisher. Zhao, Y. and K. M. Kockelman (2002). "The propagation of uncertainty through
travel demand models: An exploratory analysis," Annals of Regional Science, 36(1): 145-163.
Chapter 2
Research methodology
and selection of application cases
“…, you are yourself a part of the analytical process.”
- Howard Raiffa
“Reason is wholly instrumental. It cannot tell us
where to go; at best it can tell us how to get there” - Herbert Simon
2.1
Introduction
This research is about the methodological development and application of exploratory modeling and analysis (EMA). The first main issue we address in this chapter is how we are going to answer the research questions posed at the end of Chapter 1. This requires a definition and specification of a research methodology (Section 2.2.). We describe the research approach that guides how the research is carried out (Section 2.3). We then describe the case selection, which includes the description of the application cases and justification for the choice of the cases (Section 2.4). To address specifically the third research question, we outline an evaluation framework in which measures of EMA’s added values are specified and approaches to assess the added value are proposed (Section 2.5). Lastly, we present an analytic structure that integrates the four research questions (Section 2.6).
2.2
Defining the scope and specifying a research methodology
The research questions defined in Chapter 1.6 indicate the research scope. The questions 1 till 3 are concerned with the product of the application of EMA, whereas the question 4 is concerned with EMA itself as a method. In other words, the first three cover the application and the fourth the methodological development of EMA.
In the application of EMA, we limit the research to the insights that are produced by analysts based on computational outcomes. This implies that we do not consider how such insights will be interpreted and used by policymakers and policy actors in the policymaking process.
In this sense, the research scope can be put into a hierarchy of method impacts on the decisionmaker (see Figure 2-1) (Bots, 2003). In this hierarchy, first a method has to produce knowledge/information that is scientifically sound and valid. Second, the information has to be well understood by analysts, before it can be understood and accepted by policymakers. The next level of impact will be that the recommendations based on the knowledge produced are acted upon and finally, the utmost impact occurs if policy actions bring about desired results. So, within this view, the scope of this research is limited to insights that can be understood by analysts. Scientifically sound Und ersto od Effe ctiv e Acted upo n Acc epte d
Figure 2-1. Hierarchy of method impact
2.3
Research approach
A research approach for a scientific inquiry can be defined as following a research strategy in which a set of research instruments are employed in order to study the research subject, guided by a certain research philosophy (adapted from deVreede, 1995).
2.3.1 Research philosophy
The success of a research about a (further) development of a new method will rest on the credibility of the claim on the added value the method provides. In our case, the added value is assessed by comparing the insights obtained from EMA with those obtained from the existing methods. Some questions may be raised regarding both the claims of added value and the process of assessing the added value. Are the claims of knowledge objective? who are the assessors? how impartial are they? These questions are associated with the philosophical issues of how knowledge is produced and claimed (epistemology). They are also related to the issues of knowledge objectivity and to the relationship between observers (or assessors) and
