Delft University of Technology
Update (1.2) to ANDURIL and ANDURYL: Performance improvements and a graphical
user interface
Rongen, Guus; ’t Hart, Cornelis Marcel Pieter ; Leontaris, Georgios; Morales-Nápoles, Oswaldo
DOI
10.1016/j.softx.2020.100497
Publication date
2020
Document Version
Final published version
Published in
SoftwareX
Citation (APA)
Rongen, G., ’t Hart, C. M. P., Leontaris, G., & Morales-Nápoles, O. (2020). Update (1.2) to ANDURIL and
ANDURYL: Performance improvements and a graphical user interface. SoftwareX, 12, [100497].
https://doi.org/10.1016/j.softx.2020.100497
Important note
To cite this publication, please use the final published version (if applicable).
Please check the document version above.
Copyright
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons. Takedown policy
Please contact us and provide details if you believe this document breaches copyrights. We will remove access to the work immediately and investigate your claim.
This work is downloaded from Delft University of Technology.
Contents lists available atScienceDirect
SoftwareX
journal homepage:www.elsevier.com/locate/softx
Software update
Update (1.2) to ANDURIL and ANDURYL: Performance improvements
and a graphical user interface
Guus Rongen
a,b,∗, Cornelis Marcel Pieter ’t Hart
a,c, Georgios Leontaris
a,d,
Oswaldo Morales-Nápoles
aaCivil Engineering and Geosciences, Delft University of Technology, The Netherlands bHKV consultants, The Netherlands
cTunnel Engineering Consultants (TEC), Amersfoort, The Netherlands dVattenfall, The Netherlands
a r t i c l e i n f o Article history:
Received 28 April 2020
Received in revised form 30 April 2020 Accepted 1 May 2020
Keywords:
Structured expert judgment Cooke’s classical model Expert opinion Python module EXCALIBUR software ANDURIL GUI a b s t r a c t
This is an update to PII: S2352711018300608and S2352711019302419In this paper, we present three main improvements of ANDURIL and its python version ANDURYL. First the MATLAB version ANDURIL is brought to the Python version standard by implementing (i) user defined quantiles and (ii) the possibility to deal with missing values. Second, the computational engines of both ANDURIL and ANDURYL were significantly improved making calculation time lower and improving further accuracy. Finally a standalone Graphical User Interface is presented which we believe will make the software more accessible to practitioners of Cooke’s method.
© 2020 Published by Elsevier B.V.
Software metadata
Current code version ANDURYL v1.2
Permanent link to code/repository used for this code version GitHub(peer review version)
Legal Code License GNU General Public License
Code versioning system used GitHub
Software code languages, tools, and services used Python, PyQt5, Numpy, Matplotlib Compilation requirements, operating environments & dependencies Python version 3.6+
If available Link to developer documentation/manual Available from GUI andGithub
Support email for questions g.w.f.rongen@tudelft.nl
Code metadata
Current code version Code: ANDURYL v1.2, Paper v1.2
Permanent link to code/repository used for this code version GitHub(peer review version)
Legal Code License GNU General Public License
Code versioning system used None
Software code languages, tools, and services used Python, PyQt5, Numpy, Matplotlib Compilation requirements, operating environments & dependencies Python version 3.6+
If available Link to developer documentation/manual Available from GUI andGithub
Support email for questions g.w.f.rongen@tudelft.nl
DOIs of original articles:https://doi.org/10.1016/j.softx.2018.07.001,
https://doi.org/10.1016/j.softx.2019.100295. ∗
Corresponding author at: Civil Engineering and Geosciences, Delft University of Technology, The Netherlands.
E-mail address: g.w.f.rongen@tudelft.nl(G. Rongen).
1. Motivation and significance
Software implementing Cooke’s classical model [1] for struc-tured expert judgment was presented in [2] and [3]. The earlier https://doi.org/10.1016/j.softx.2020.100497
2 G. Rongen, C.M.P.’t. Hart, G. Leontaris et al. / SoftwareX 12 (2020) 100497
Fig. 1. Illustration of decision maker interpolation.
Fig. 2. Overview of the ANDURYL GUI, with on the background the main window and on the foreground the CDF of each expert and the DM for a specific question. Table 1
Computational times of different version of AI and AY in robustness analysis. Up to four items left out at a time, global weights, no optimization.
AI v1.0 AY v1.1 AI v1.2 AY v1.2
15 min 60 s 30 s 4 s
MATLAB version is named ANDURIL (AI) while the Python version is ANDURYL (AY).
In this update:
1. ANDURIL is brought to the Python version standard by implementing: (i) user defined quantiles and (ii) the pos-sibility to deal with missing values. These features will not be discussed further. The reader is referred to [3] for an
explanation of the main features now also available in AI v1.2 (ANDURIL version 1.2).
2. The code of both ANDURIL and ANDURYL was significantly improved, reducing the calculation time. The calculation times on a PC with an Intel Core I5-5300U CPU of 2.3 GHz for robustness analysis (global weights without optimiza-tion) for the study presented in [5] are shown inTable 1. Up to 4 of the 13 calibration questions at a time were ex-cluded, resulting in 1092 combinations of excluded items. The MATLAB version AI v1.2 is 30 times faster than AI v1.0 for the study under consideration. Similarly AY v1.2 is a factor 15 faster than AY v1.1 and approximately 220 times faster than AI v1.0.
Table 2
Comparison of results presented in Table 1 of [4] (CC) and calculations with AI (AI) and AY (AY). Note that only the 7 studies that had or still have differences are shown. The other 26 studies have no differences in the outcomes.
Study #E #S Equal Weight Global No Op. PW Global PW Item Best Expert
Sa In Co Sa In Co Sa In Co Sa In Co Sa In Co CDC ROI (CC) 20 10 0.23 1.23 0.29 0.39 1.35 0.52 0.72 2.31 1.66 0.72 2.31 1.66 0.72 2.31 1.66 CDC ROI* (AI 1.0) – – – – – – – – – – – – – – – – – CDC ROI (AY 1.1) 20 10 0.23 1.23 0.29 0.39 1.35 0.52 0.72 2.30 1.66 0.72 2.30 1.66 0.72 2.30 1.66 CDC ROI (AI 1.2) 20 10 0.23 1.23 0.29 0.39 1.35 0.52 0.72 2.30 1.66 0.72 2.30 1.66 0.72 2.30 1.66 CDC ROI (AY 1.2) 20 10 0.23 1.23 0.29 0.39 1.35 0.52 0.72 2.30 1.66 0.72 2.30 1.66 0.72 2.30 1.66 CWD (CC) 14 10 0.47 0.93 0.44 0.47 0.94 0.45 0.49 1.22 0.60 0.68 1.33 0.90 0.31 2.19 0.69 CWD (AI 1.0) 14 10 0.47 0.93 0.44 0.47 0.94 0.45 0.49 1.21 0.60 0.68 1.33 0.90 0.31 2.19 0.69 CWD (AY 1.1) 14 10 0.47 0.93 0.44 0.47 0.94 0.45 0.49 1.21 0.60 0.68 1.33 0.90 0.31 2.19 0.69 CWD (AI 1.2) 14 10 0.47 0.93 0.44 0.47 0.94 0.45 0.49 1.21 0.60 0.68 1.33 0.90 0.31 2.19 0.69 CWD (AY 1.2) 14 10 0.47 0.93 0.44 0.47 0.94 0.45 0.49 1.21 0.60 0.68 1.33 0.90 0.31 2.19 0.69 Gerstenberger (CC) 12 14 0.64 0.48 0.31 0.35 0.61 0.21 0.93 1.10 1.02 0.76 1.20 0.91 0.54 1.74 0.93 Gerstenberger* (AI 1.0) – – – – – – – – – – – – – – – – – Gerstenberger (AY 1.1) 12 14 0.64 0.48 0.31 0.35 0.61 0.21 0.93 1.09 1.02 0.76 1.09 0.82 0.54 1.74 0.93 Gerstenberger (AI 1.2) 12 14 0.64 0.48 0.31 0.35 0.61 0.21 0.93 1.09 1.02 0.76 1.20 0.91 0.54 1.74 0.93 Gerstenberger (AY 1.2) 12 14 0.64 0.48 0.31 0.35 0.61 0.21 0.93 1.09 1.02 0.76 1.20 0.91 0.54 1.74 0.93 Goodheart (CC) 6 10 0.55 0.28 0.15 0.47 0.35 0.16 0.71 0.96 0.68 0.71 0.96 0.68 0.71 0.96 0.68 Goodheart (AI 1.0) 6 10 0.55 0.28 0.15 0.47 0.35 0.16 0.47 0.35 0.17 0.68 0.64 0.43 0.71 0.96 0.68 Goodheart (AY 1.1) 6 10 0.55 0.28 0.15 0.47 0.35 0.16 0.47 0.35 0.17 0.68 0.64 0.43 0.71 0.96 0.68 Goodheart (AI 1.2) 6 10 0.55 0.28 0.15 0.47 0.35 0.16 0.71 0.96 0.68 0.71 0.96 0.68 0.71 0.96 0.68 Goodheart (AY 1.2) 6 10 0.55 0.28 0.15 0.47 0.35 0.16 0.71 0.96 0.68 0.71 0.96 0.68 0.71 0.96 0.68 Hemopilia (CC) 18 8 0.25 0.20 0.05 0.31 0.27 0.08 0.31 0.49 0.15 0.31 0.46 0.14 0.85 1.07 0.91 Hemopilia* (AI 1.0) – – – – – – – – – – – – – – – – – Hemopilia (AY 1.1) 18 8 0.25 0.20 0.05 0.31 0.27 0.08 0.31 0.30 0.09 0.31 0.15 0.05 0.85 1.07 0.91 Hemopilia (AI 1.2) 18 8 0.25 0.20 0.05 0.31 0.27 0.08 0.31 0.29 0.09 0.31 0.39 0.12 0.85 1.07 0.91 Hemopilia (AY 1.2) 18 8 0.25 0.20 0.05 0.31 0.27 0.08 0.31 0.30 0.09 0.31 0.41 0.13 0.85 1.07 0.91 IceSheets (CC) 10 11 0.49 0.52 0.25 0.62 0.70 0.43 0.40 1.55 0.62 0.62 1.04 0.64 0.40 1.55 0.62 IceSheets (AI 1.0) 10 11 0.49 0.52 0.25 0.37 0.66 0.25 0.40 1.55 0.62 0.62 1.04 0.64 0.40 1.55 0.62 IceSheets (AY 1.1) 10 11 0.49 0.52 0.25 0.37 0.66 0.25 0.40 1.55 0.62 0.62 1.04 0.64 0.40 1.55 0.62 IceSheets (AI 1.2) 10 11 0.49 0.52 0.25 0.37 0.66 0.25 0.40 1.55 0.62 0.62 1.04 0.64 0.40 1.55 0.62 IceSheets (AY 1.2) 10 11 0.49 0.52 0.25 0.37 0.66 0.25 0.40 1.55 0.62 0.62 1.04 0.64 0.40 1.55 0.62 Topaz (CC) 21 16 0.63 0.92 0.58 0.31 1.12 0.34 0.41 1.46 0.60 0.41 1.46 0.60 0.41 1.46 0.60 Topaz (AI 1.0) 21 16 0.63 0.92 0.58 0.31 1.12 0.34 0.41 1.45 0.60 0.41 1.45 0.60 0.41 1.45 0.60 Topaz (AY 1.1) 21 16 0.63 0.92 0.58 0.31 1.12 0.34 0.41 1.45 0.60 0.41 1.45 0.60 0.41 1.45 0.60 Topaz (AI 1.2) 21 16 0.63 0.92 0.58 0.31 1.12 0.34 0.41 1.45 0.60 0.41 1.45 0.60 0.41 1.45 0.60 Topaz (AY 1.2) 21 16 0.63 0.92 0.58 0.31 1.12 0.34 0.41 1.45 0.60 0.41 1.45 0.60 0.41 1.45 0.60
The new code led also to improved accuracy of both AI and AY. That is, both solutions are closer to EXCALIBUR (CC). The differences between CC and AI and AY for the 7 studies where differences were observed, are shown in Table 2. This will be elaborated further below.
3. A standalone Graphical User Interface of ANDURYL is pre-sented. A screen shot of the GUI is presented inFig. 2 2. ANDURYL and ANDURIL code improvement
The main improvement in speed and accuracy is the result of a different implementation for calculating the Decision Maker’s (DM) cumulative distribution function (CDF). In version 1.0 and 1.1, the DM’s CDF was calculated by integrating the probabil-ity densprobabil-ity function (PDF) of the weighted DM’s numerically (quadrature method) through an anonymous function. Solving this integral is numerically expensive and when the probability density of one or more expert are very concentrated in a range in relation to that of other experts, parts of the PDF were skipped in the discretization used in the numerical integration.
In the new (AY v1.2 and AI v1.2), the old implementation of the integral is replaced by an interpolation of the CDF. As long as the PDF between the given quantiles is uniform (or log-uniform), this gives the same results as solving the integral, but much quicker and without inaccuracies due to the discretization of the integral.Fig. 1illustrates the process of interpolation for the decision maker.
Note that the DM quantiles (‘‘DM full’’ in the figure) are determined by interpolating each of the (two in this case) experts’
answers (following the dashed lines). This results in the full detailed CDF of the decision maker. This can subsequently be interpolated at the percentiles of interest (which is EXCALIBUR’s output). Note that the interpolation is not carried out over the quantile direction.
3. ANDURYL GUI
The main improvement for the Python version is the graphical user interface. This interface, programmed with the Python mod-ule PyQt5, is compiled with PyInstaller (for Windows), such that it is a stand-alone executable. This makes ANDURYL accessible to non-Python users. The layout of the user interface consists of 4 overviews, for the experts, items, assessments and results, as shown inFig. 2.
The following list gives an overview of the functionalities that the stand-alone GUI offers:
•
Assessments per expert or item can be plotted as a PDF, CDF, survival function or range. The CDF option is shown inFig. 2on the foreground.
•
Because of the improvements in computational performance, it is now less demanding to do a robustness analysis for excluding multiple experts or items. The results of the robustness analysis can be shown in box plots.•
The program has options for saving the project inEXCAL-IBUR format or a more common JSON format.
•
Separate DM’s results, such as the full CDFs, can be exported or copied to clipboard.4 G. Rongen, C.M.P.’t. Hart, G. Leontaris et al. / SoftwareX 12 (2020) 100497
•
The AY code is separated between calculation and userinterface functionalities so that the Python-module can also be used from a script or Jupyter notebook. For research purposes this is a useful functionality.
•
The fact that AY is still significantly faster than AI, as shown in Table 1, is due to differences in implementation. In AI several expensive operations are re-calculated for different iterations. In AY the amount of data that is re-calculated is minimized.4. Comparing with previous studies
In [4], 33 post-2006 studies using Cooke’s classical method are presented using CC. We use these data to compare output from AY and AI to both CC, the MATLAB implementation AI of the v1.0 paper [2] and the Python implementation of the paper [3].
The differences are smaller compared to the results from the last code version. For two studies, ‘‘Hemophilia’’ and ‘‘Ice sheets’’ the differences are still significant. For four other studies the results seem to be due to rounding errors. Of the remaining 26 studies, the majority have equal results.Table 2 shows the differences for the studies where differences are still observed. 5. Conclusions
The Python module named ANDURYL (AY) has been extended with a graphical user interface and is available as stand-alone ex-ecutable. The MATLAB toolbox named ANDURIL (AI) for combin-ing expert judgments applycombin-ing Cooke’s method has been further extended by adding functionalities for user defined quantiles and handling missing values. The stand-alone GUI enables practition-ers and researchpractition-ers that have no Python or MATLAB experience to apply Cooke’s method with ANDURYL. For users that are more familiar with programming, the MATLAB toolbox and Python GUI are a means to perform or analyze expert elicitations in a reproducible way. The improved speed and accuracy contribute to this cause. Both codes are open source to encourage usage and further development.
CRediT authorship contribution statement
Guus Rongen: Methodology, Software, Writing - original draft. Cornelis Marcel Pieter ’t Hart: Methodology, Validation. Georgios Leontaris: Methodology. Oswaldo Morales-Nápoles: Conceptualization, Writing - review & editing, Project adminis-tration, Funding acquisition.
Declaration of competing interest
The authors declare that they have no known competing finan-cial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This research was funded by the TKI project EMU-FD. This research project is funded by Rijkswaterstaat, The Netherlands, Deltares, The Netherlands and HKV consultants, The Netherlands. References
[1] Cooke R. Experts in uncertainty: Opinion and subjective probability in science. Environmental ethics and science policy, Oxford University Press; 1991.
[2] Leontaris G, Morales-Nápoles O. Anduril: A matlab toolbox for analysis and decisions with uncertainty: learning from expert judgments. SoftwareX 2018;7:313–7.http://dx.doi.org/10.1016/j.softx.2018.07.001.
[3] ’t Hart CMP, Leontaris G, Morales-Nápoles O. Update (1.1) to ANDURIL — A MATLAB toolbox for analysis and decisions with uncertainty: Learn-ing from expert judgments: ANDURYL. SoftwareX 2019;10:100295.http:// dx.doi.org/10.1016/j.softx.2019.100295, URL http://www.sciencedirect.com/ science/article/pii/S2352711019302419.
[4] Colson AR, Cooke RM. Cross validation for the classical model of structured expert judgment. Reliab Eng Syst Saf 2017;163:109–20.
[5] Puig D, Morales-Nápoles O, Bakhtiari F, Landa G. The accountability imper-ative for quantifying the uncertainty of emission forecasts: evidence from mexico. Clim Policy 2018;18(6):742–51.
