Introduction New accuracy measures Test cases Conclusion & outlook
Accuracy of Pedestrian and Traffic
Flow Models
Meaningful Quantifications
Femke van Wageningen-Kessels
Serge Hoogendoorn, Winnie Daamen
TFTC Summer Meeting and Conference
Celebrating 50 Years of Traffic Flow Theory — Portland, Oregon, 2014
Introduction New accuracy measures Test cases Conclusion & outlook
Background
How good is a traffic/pedestrian flow model?
Observations
▸
reality
▸
experiment
Model
▸
simulations
Predictions
Compare
▸
qualitative
▸
quantitative
calibration and validation
parameter estimation
Introduction New accuracy measures Test cases Conclusion & outlook
Background
How good is a traffic/pedestrian flow model?
Observations
▸
reality
▸
experiment
Model
▸
simulations
Predictions
Compare
▸
qualitative
▸
quantitative
calibration and validation
parameter estimation
Introduction New accuracy measures Test cases Conclusion & outlook
Background
How good is a traffic/pedestrian flow model?
Observations
▸
reality
▸
experiment
Model
▸
simulations
Predictions
Compare
▸
qualitative
▸
quantitative
calibration and validation
parameter estimation
Introduction New accuracy measures Test cases Conclusion & outlook
Accuracy measures review
▸
Goodness of Fit of speed, spacing, density, flow
▸
(Root Mean) Squared (Normalized) Error, Mean
(Absolute) (Normalized) Error,
▸
GEH statistic
▸
Correlation Coefficient
▸
Theil’s Bias/Variance/Covariance Proportion,
Theil’s Inequality Coefficient
▸
Likelihood
▸
Total flux, time spent, evacuation time
Usually do not take into account specific features
Numerical simulations
Introduction New accuracy measures Test cases Conclusion & outlook
Contribution: new accuracy measures
▸
Allow focus on certain feature of flow instead
of averaging
▸
Gives insight into type of error
▸
⇒
Insight into how to improve accuracy
Introduction New accuracy measures Test cases Conclusion & outlook
Outline
Introduction
New accuracy measures
Test cases
Traffic congestion simulation
Bi-directional pedestrian flow modelling
Conclusion & outlook
Introduction New accuracy measures Test cases Conclusion & outlook
New accuracy measures
x
y
Exact
x
y
Phase error
Is location of high/low
density/speed area correct?
x
y
Diffusion error
Do sharp transitions between
high/low density/velocity
areas stay sharp?
Introduction New accuracy measures Test cases Conclusion & outlook
New accuracy measures
x
y
Exact
x
y
Phase error
Is location of high/low
density/speed area correct?
x
y
Diffusion error
Do sharp transitions between
high/low density/velocity
areas stay sharp?
Introduction New accuracy measures Test cases Conclusion & outlook
New accuracy measures
x
y
Exact
x
y
Phase error
Is location of high/low
density/speed area correct?
x
y
Diffusion error
Do sharp transitions between
high/low density/velocity
areas stay sharp?
Introduction New accuracy measures Test cases Conclusion & outlook
From concept to quantification
Centre of mass for phase error
▸
In x - and y -direction
▸
Large difference ⇒ large phase error
x
y
x
y
x
ρ
x
ρ
Introduction New accuracy measures Test cases Conclusion & outlook
From concept to quantification
Centre of mass for diffusion error
▸
In density- (or speed-) direction
▸
Large difference ⇒ large diffusion error
x
y
x
y
x
ρ
x
ρ
Introduction New accuracy measures Test cases Conclusion & outlook
Test case 1: Traffic congestion simulation
Exact solution of LWR model ⇔ simulation results
2km jam
time
0
critical density
jam density
free flow
congestion
Solve with different numerical methods
Introduction New accuracy measures Test cases Conclusion & outlook
Test case 1: Traffic congestion simulation
Exact solution of LWR model ⇔ simulation results
2km jam
time
congestion spills back
congestion solves
0
critical density
jam density
free flow
congestion
Solve with different numerical methods
Introduction New accuracy measures Test cases Conclusion & outlook
Numerical solutions: space time density
time
space
time
space
time
space
Min supply demand
Upwind explicit
Upwind implicit
0
critical density
jam density
free flow
congestion
Introduction New accuracy measures Test cases Conclusion & outlook
Numerical solutions: density cross section t = 600 s
Min supply demand
Upwind explicit
Upwind implicit
Centre of mass ⇒ phase & diffusion error
Introduction New accuracy measures Test cases Conclusion & outlook
Numerical solutions: density cross section t = 600 s
Min supply demand
Upwind explicit
Upwind implicit
Centre of mass ⇒ phase & diffusion error
Introduction New accuracy measures Test cases Conclusion & outlook
Phase error
Diffusion error
Upwind implicit
Min supply
demand
Upwind explicit
Results help selecting appropriate numerical method
▸
Small time steps: upwind is best
▸
Big time steps: use implicit, but at cost of
phase error
Introduction New accuracy measures Test cases Conclusion & outlook
Phase error
Diffusion error
Upwind implicit
Min supply
demand
Upwind explicit
Results help selecting appropriate numerical method
▸
Small time steps: upwind is best
▸
Big time steps: use implicit, but at cost of
phase error
Introduction New accuracy measures Test cases Conclusion & outlook
Test case 2:
Bi-directional pedestrian flow modelling
Experimental data ⇔ model
−5 0 5 −2 0 2 Densities class 1 (−>), t=350 (s) location (m) location (m) 0 0.5 1 −5 0 5 −2 0 2 Densities class 2 (<−), t=350 (s) location (m) location (m) 0 0.5 1
class 1 →
class 2 ←
model
data
model
data
Continuum flow model
2 parameters for avoidance
β
u
=
0.8, β
o
=
2.3 (set 1)
Test with other parameter settings
Introduction New accuracy measures Test cases Conclusion & outlook
Test case 2:
Bi-directional pedestrian flow modelling
Experimental data ⇔ model
−5 0 5 −2 0 2 Densities class 1 (−>), t=350 (s) location (m) location (m) 0 0.5 1 −5 0 5 −2 0 2 Densities class 2 (<−), t=350 (s) location (m) location (m) 0 0.5 1
class 1 →
class 2 ←
model
data
model
data
model
data
model
data
Continuum flow model
2 parameters for avoidance
β
u
=
0.8, β
o
=
2.3 (set 1)
Test with other parameter settings
Introduction New accuracy measures Test cases Conclusion & outlook
Test case 2:
Bi-directional pedestrian flow modelling
Experimental data ⇔ model
−5 0 5 −2 0 2 Densities class 1 (−>), t=350 (s) location (m) location (m) 0 0.5 1 −5 0 5 −2 0 2 Densities class 2 (<−), t=350 (s) location (m) location (m) 0 0.5 1
class 1 →
class 2 ←
model
data
model
data
model
data
model
data
Continuum flow model
2 parameters for avoidance
β
u
=
0.8, β
o
=
2.3 (set 1)
Test with other parameter settings
Introduction New accuracy measures Test cases Conclusion & outlook
Experimental data ⇔
model with different
parameter settings
model
data
model
data
β
u
=
0.8, β
o
=
2.3 (set 1)
almost perfect
model
data
model
data
β
u
=
0.7, β
o
=
1.36 (set 2)
lanes swapped
model
model
model
model
model
model
model
model
model
model
model
model
model
model
model
data
data
β
u
=
0.63, β
o
=
0.63 (set 3)
no lanes
Parameters are calibrated for total flux
Introduction New accuracy measures Test cases Conclusion & outlook
Experimental data ⇔
model with different
parameter settings
model
data
model
data
β
u
=
0.8, β
o
=
2.3 (set 1)
almost perfect
model
data
model
data
β
u
=
0.7, β
o
=
1.36 (set 2)
lanes swapped
model
model
model
model
model
model
model
model
model
model
model
model
model
model
model
data
data
β
u
=
0.63, β
o
=
0.63 (set 3)
no lanes
Parameters are calibrated for total flux
Introduction New accuracy measures Test cases Conclusion & outlook
Results
No difference for parameter settings according to:
▸
MAE & RMSE of class specific speed
▸
Diffusion error
▸
Total flux
But: ME & RMSE of v
x
and phase error show set 1
is best
Introduction New accuracy measures Test cases Conclusion & outlook
Results
No difference for parameter settings according to:
▸
MAE & RMSE of class specific speed
▸
Diffusion error
▸
Total flux
But: ME & RMSE of v
x
and phase error show set 1
is best
Introduction New accuracy measures Test cases Conclusion & outlook
Results
ME & RMSE of v
x
0
200
400
0
0.1
0.2
0.3
0.4
0.5
t (s)
rmse of v (m/s)
Beta0u=0.8, Beta0o=2.3
0
200
400
−0.4
−0.2
0
0.2
0.4
t (s)
me of v (m/s)
0
200
400
0
0.5
1
1.5
2
2.5
3
t (s)
rmse (black) and mae (green) of vx (m/s)
0
200
400
−6
−4
−2
0
2
4
6
t (s)
phase error x (m)
0
200
400
−2
−1
0
1
2
t (s)
phase error y (m)
0
200
400
−0.4
−0.2
0
0.2
0.4
t (s)
difference v CM (m/s)
set 1
model
data
model
data
0
200
400
0
0.1
0.2
0.3
0.4
0.5
t (s)
rmse of v (m/s)
Beta
u=0.7, Beta
o=1.36
0
200
400
−0.4
−0.2
0
0.2
0.4
t (s)
me of v (m/s)
0
200
400
0
0.5
1
1.5
2
2.5
3
t (s)
rmse (black) and mae (green) of vx (m/s)
0
200
400
−6
−4
−2
0
2
4
6
t (s)
difference x CM (m)
0
200
400
−2
−1
0
1
2
t (s)
difference y CM (m)
0
200
400
−0.4
−0.2
0
0.2
0.4
t (s)
difference v CM (m/s)
ME
RMSE
set 2
model
data
model
data
0
200
400
0
0.1
0.2
0.3
0.4
0.5
t (s)
rmse of v (m/s)
Beta
u=0.63, Beta
o=0.63
0
200
400
−0.4
−0.2
0
0.2
0.4
t (s)
me of v (m/s)
0
200
400
0
0.5
1
1.5
2
2.5
3
t (s)
rmse (black) and mae (green) of vx (m/s)
0
200
400
−6
−4
−2
0
2
4
6
t (s)
difference x CM (m)
0
200
400
−2
−1
0
1
2
t (s)
difference y CM (m)
0
200
400
−0.4
−0.2
0
0.2
0.4
t (s)
difference v CM (m/s)
set 3
model
model
model
model
model
model
model
model
model
model
model
model
model
model
model
data
data
▸
Set 1 best
▸
Large phase error for set 2
Introduction New accuracy measures Test cases Conclusion & outlook
Results
Phase error y -direction
0 200 400 0 0.1 0.2 0.3 0.4 0.5 t (s) rmse of v (m/s) Beta0u=0.8, Beta0o=2.3 0 200 400 −0.4 −0.2 0 0.2 0.4 t (s) me of v (m/s) 0 200 400 0 0.5 1 1.5 2 2.5 3 t (s)
rmse (black) and mae (green) of vx (m/s)
0 200 400 −6 −4 −2 0 2 4 6 t (s) phase error x (m) 0 200 400 −2 −1 0 1 2 t (s) phase error y (m) 0 200 400 −0.4 −0.2 0 0.2 0.4 t (s) difference v CM (m/s)
set 1
model
data
model
data
0 200 400 0 0.1 0.2 0.3 0.4 0.5 t (s) rmse of v (m/s) Beta u=0.7, Betao=1.36 0 200 400 −0.4 −0.2 0 0.2 0.4 t (s) me of v (m/s) 0 200 400 0 0.5 1 1.5 2 2.5 3 t (s)rmse (black) and mae (green) of vx (m/s)
0 200 400 −6 −4 −2 0 2 4 6 t (s) difference x CM (m) 0 200 400 −2 −1 0 1 2 t (s) difference y CM (m) 0 200 400 −0.4 −0.2 0 0.2 0.4 t (s) difference v CM (m/s)
class 1 →
class 2 ←
set 2
model
data
model
data
0 200 400 0 0.1 0.2 0.3 0.4 0.5 t (s) rmse of v (m/s) Beta u=0.63, Betao=0.63 0 200 400 −0.4 −0.2 0 0.2 0.4 t (s) me of v (m/s) 0 200 400 0 0.5 1 1.5 2 2.5 3 t (s)rmse (black) and mae (green) of vx (m/s)
0 200 400 −6 −4 −2 0 2 4 6 t (s) difference x CM (m) 0 200 400 −2 −1 0 1 2 t (s) difference y CM (m) 0 200 400 −0.4 −0.2 0 0.2 0.4 t (s) difference v CM (m/s)
set 3
model
model
model
model
model
model
model
model
model
model
model
model
model
model
model
data
data
▸
Set 1 best
▸
Large phase error for set 2
Introduction New accuracy measures Test cases Conclusion & outlook
Conclusion & outlook
▸
Phase error and diffusion error
▸
Applications
▸
Road traffic & pedestrian flow. Future: NFD?
▸
Comparing data vs model, model vs simulation, ...
▸
Parameter estimation or assessment of
model/simulation method
▸
Distinguish between different outcomes →
interpretation needed:
▸
Phase error sometimes ok, sometimes not
▸
Insight into possible improvements
▸
Future research:
▸
Larger networks with many features?
▸
Include time
Introduction New accuracy measures Test cases Conclusion & outlook
Conclusion & outlook
▸
Phase error and diffusion error
▸
Applications
▸
Road traffic & pedestrian flow. Future: NFD?
▸
Comparing data vs model, model vs simulation, ...
▸
Parameter estimation or assessment of
model/simulation method
▸
Distinguish between different outcomes →
interpretation needed:
▸
Phase error sometimes ok, sometimes not
▸
Insight into possible improvements
▸
Future research:
▸
Larger networks with many features?
▸
Include time
Introduction New accuracy measures Test cases Conclusion & outlook
