Michał Rubaszek

(1)

Advanced Econometrics Topic 8: Forecasting

Michał Rubaszek

SGH Warsaw School of Economics

1

(2)

Introduction

(3)

Michał Rubaszek, Advanced Econometrics

Introduction

The ultimate goal of a positive science is to develop a theory or hypothesis that yields valid and meaningful predictions about phenomena not yet

observed. Theory is judged by it's predictive power.

A hypothesis can't be tested by its assumptions. What is important is

specifying the conditions under which the hypothesis works. What matters is it's predictive power, not it's conformity to reality.

Milton Friedman, 1953. The Methodology of Positive Economics.

in Essays in Positive Economics: University of Chicago Press.

3

(4)

Introduction

 Predicting future economic outcomes is helpful in making appropriate plans, making investment decisions, conducting economic policies.

 We make inference about future outcome using available data (for time series: current and past data) and statistical models.

We call this process econometric forecasting

 Point forecast from model , horizon and information set Ω

_!

:

"

_!,#^$

% "

_!&#|!

% (

_!

"

_!&#

| % (("

_!&#

| , Ω

_!

)

 Density forecast provides information on all quantiles of the distribution. We

focus on the entire distribution (pdf):

(5)

Point forecast

5

(6)

Density forecast

(7)

Fan chart

7

Source: Bank of England, February 2013 Inflation Report

(8)

Economic forecasting - introduction

Types of time series forecasts

 Qualitative / model-based (e.g. from VAR/DSGE model)

 Quantitative / expert based (e.g. survey forecast, SPF)

General characteristics of time series forecasts:

 Forecasting is based on the assumption that the past predicts the future

Think carefully if the past is related to what you expect about the future

 Forecasts are always wrong

However, some models/methods might work better or worse than the other

 Forecasts are usually more accurate for shorter time periods

But, economic theories are more informative for longer horizon

(9)

Ex-ante forecast

9

(10)

Ex-ante vs ex-post forecast

 Ex-ante forecast is a true inference about the future It is for periods in which we don't know the realization

 Ex-post forecast is to check model reliability

It is for periods in which we know the realization

(11)

Ex-ante forecast:

error in known ARMA/VAR model



Assume we know DGP, i.e. the parameters and the specification of ARMA/VAR.

We therefore know the parameters of infinite moving average representation

"

_-

% ? @ A

_B

C

_-

@ A

_D

C

_-ED

+ A

_F

C

_-EF

+ A

_G

C

_-EG

… C

_-

∼ I(0, K)



Forecast from known DGP is called optimum forecast.

We cannot obtain more accurate forecast from another model.



Forecast error of optimal forecast is solely due to futures shocks (random error):

"

_!&#

L "

_!&#|!

% A

_B

C

_!&#

@ A

_D

C

_!&#ED

+ ⋯ + A

_#ED

C

_!&D



The resulting variance of forecast is:

( "

_!&#

L "

_!&#|! ^F

% A

_B

A

_B^N

@ A

_D

A

_D^N

+ ⋯ + A

_#ED

A

_#ED^N

11

(12)

Ex-ante forecast

error in estimated ARMA/VAR model

 Assume that we don't know the true DGP but use a model instead

 The variance of our forecast is:

Component A: error of "optimum forecast" (see previous slide) Component B: estimation / misspecification error

we want to minimize this value

Component C: equals to 0 if we cannot forecast future shock

(13)

Ex-ante forecast

estimation / misspecification error

Let us focus on the estimation / misspecification error and model complexity (S "

_!&#

L "

_!&#|! ^F

T

I. Large / complex models

 many parameters % large estimation error (high variance)

 many explanatory variables % good specification (low bias)

II. Small / simple models

 few parameters % small estimation error (low variance)

 few explanatory variables % potential misspecification (high bias)

Which effect dominates? We don't know and need to check it

13

(14)

Ex-ante forecast

Illustration of the variance / bias trade-off

(15)

Ex-ante forecast

Illustration of the variance / bias trade-off

 Let as assume that the true DGP is AR(1):

"

_-

% ? @ W "

_-

L ? @ X

_-

 We have a sample of 180 monthly observations (15 years) for "

_-

and would like to decide on one of the three competing models:

RW, Random walk: ?

^[\

% 0 and W

^[\

% 1

HL, 5-year half life model: ?

^{^_}

% "` and W

^{^_}

% 0.5

^D/aB

AR, estimated AR model: ?

^b[

and W

^b[

are estimated

 Which model performs best?

It depends on the value of W

15

(16)

Ex-ante forecast

Illustration of the variance / bias trade-off

(17)

Ex-post forecast

17

(18)

About ex-post forecast

 We usually work with models that performed well in the past

 In ex-post forecast we ask a question :

how accurate forecasts the model would deliver if it was used in the past

 We evaluate ex-postforecasts to be sure about model reliability

 An important issue in the use of "real time data, RTD"

(19)

About ex-post forecast

 We compare forecast "

_-,#^c

from model

_d

to realization "

_-&#

to assess:

 the absolute quality of forecasts from model

_d

MFE, effciency/unbiasedness tests, sequential forecasts, PIT

 the relative quality of forecasts from models

_d

and

_e

RMSFE/MSFE/MAFE, log predictive scores

 Various forecasting schemes

 rolling scheme

 recursive schemes

 fixed schemes

 A very important choice relates to the split of the sample into estimation and evaluation subsamples

19

(20)

Recursive forecasting scheme - illustration

(21)

Recursive forecasting scheme with RTD illustration

21 Source: Rubaszek M. & Skrzypczynski P., 2008. On the forecasting performance of a small-scale DSGE model, International Journal of Forecasting

(22)

Point forecasts accuracy measures

MFE - Mean Forecasts Error

f(_# % 1

g_# h "_-&# L "_-,#^$

!E#

-i!_j&D

(23)

Point forecasts accuracy measures

RMSFE - Root Mean Squared Forecast Error:

k lf(_# % 1

g_# h "_-&# L "_-,#^{$ F}

!E#

-i!_j&D

where T_# % g L g_D L @ 1

MSFE - Mean Squared Forecast Error:

lf(_# % 1

g_# h "_-&# L "_-,#^{$ F}

!E#

-i!_j&D

MAFE - Mean Absolute Forecast Error:

mf(_# % 1

g_# h |"_-&# L "_-,#^$ |

!E#

-i!j&D

23

(24)

Point forecasts accuracy measures

Diebold-Mariano test for equal forecast accuracy

Forecast errors from two competing models n_D-,# % "_-&# L "_D-,#^$ and n_F-,# % "_-&# L "_F-,#^$ The quadratic loss function o_-,# % n_D-,#^F L n_F-,#^F

The null of equal forecast accuracy (RMSFE) p_B: ((o_-,#) % 0

Test statistic:

q % o̅_-,#

l/g_# ∼ I 0,1

where l % ∑^#ED_{diE #ED} tu(v)is the estimate of ``long-term’’ variance Important: loss function does not necessary need to be quadratic!

(25)

Point forecasts accuracy measures

25 Source: M. Kolasa & M. Rubaszek & P. Skrzypczyński, 2012. Putting the New Keynesian DSGE Model to the Real-Time Forecasting Test,

Journal of Money, Credit and Banking

RMSFE / DM test example

(26)

Point forecasts accuracy measures

RMSFE – graphical illustration

Notes: Each line represents the ratio of RMSE from a given method to RMSE from the random walk, where values below unity indicate better accuracy of point forecasts. The straight and dotted lines stand for VAR1 and VAR2, respectively. The forecast horizon is expressed in months.

(27)

Point forecasts accuracy measures

Efficiency / unbiasedness test – graphical illustration

(28)

Point forecasts accuracy measures

Efficiency / unbiasedness test

A relatively good forecast accuracy does not imply that they are satisfactory in the absolute sense! Absolute performance include ME and efficiency/unbiasedness test. For regression:

"_-&# % {_B @ {_D"_-,#^$ @ X_-,#

we test whether {_B % 0 and {_D % 1.

[ the alternative specification is n-,# % {B@ {D"_-,#^$ @ X-,# in which we test {B% 0 and {D% 0 ]

(29)

Point forecasts accuracy measures

29 Source: Ca’ Zorzi M. & Kolasa M. & Rubaszek M., 2017. Exchange rate forecasting with DSGE models, Journal of International Economics

(30)

Density forecasts accuracy measures

PIT – Probability Integral Transform

|Kg

_-,#

% }

^~^•€•

,

_-,#^$

. o.

E‚

∈ „0,1…

where ,_-,#^$ (.) is the forecast for density distribution.

 In other words, PIT is the value forecasted cdf at realization

(31)

Density forecasts accuracy measures

PIT – Probability Integral Transform

 For a well calibrated model the series |Kg_-,# should be drawn from KKq † 0,1

 We can check it through QQ plot or histogram

(32)

Density forecasts accuracy measures

LPS – Log Predictive Score

|l_-,# % log(,_-,#^$ "_-&# ) where ,_-,#^$ () is the forecast for density distribution.

(33)

Density forecasts accuracy measures

Amisano-Giacomini (2007) test

 AG test allows to compare LPS from two competing models The loss differential ‰_-,# % ‰|l_D-,# L ‰|l_F-,#

The null of equal forecast accuracy p_B: ((‰_-,#) % 0 Test statistic: Šm % ^_`_‹/!^•,•

• → I 0,1

where l is the HAC estimate of the ``long-term’’ variance for ‰_-,#

 Interpretation of average LPS difference between models:

‰|l_D L ‰|l_F

average percentage difference in data fit to predictive density

* Amisano, G., Giacomini, R., 2007. Comparing density forecasts via weighted likelihood ratio tests. Journal of Business & Economic Statistics 25, 177-190.

33

(34)

Density forecasts accuracy measures

LPS / AG test: example

(35)

Exercises

Exercise 1.

Select a country among US, EA, UK, CAD, AUS and try to replicate selected results from Table 2 from Ca'Zorzi, Kolasa & Rubaszek (2017).

A. Calculate recursive point forecasts for real exchange rate (level) for % 1: 24 from:

RW, ARIMA(1,1,1), ARMA(1,0), VAR(2) on levels, VAR(2) on growt rates

•_- % „"_- ,_- v_- –_- —˜_- "_-^∗ ,_-^∗ v_-^∗…′

B. Calculate MFE and RMSFE for the 5 methods

C. Compare the accuracy of forecasts from 4 models to RW with DM test

D. Conduct efficiency test / draw a scatter-plot for forecasts from two VAR models E. Make a plot for sequential forecasts from two VAR models

F. Add the ''AR fixed'' model to the competition

G. Compare the results to those from Ca'Zorzi, Kolasa and Rubaszek (2017)?

Notes: perform model evaluation using data from the period 1995:1-2013:4, as in Ca' Zorzi Michele, Marcin Kolasa and Michał Rubaszek, 2017. Exchange rate forecasting with DSGE models, Journal of International Economics

35

(36)

Exercises

Exercise 2.

Select a country among UK, CAD, AUS and a variable (output, prices or interest rates) and try to replicate selected results from Tables 4 and 5 from Kolasa & Rubaszek (2018).

A. Calculate recursive point and density forecasts for output level from BVAR(4) models:

 NK model: •_- % „"_- ,_- v_-…′

 LS model: •_- % „"_- ,_- v_- –_- —˜_-…′

B. Compare RMSFE from both models with DM test

C. Calculate PIT from both models for 4-quarter ahead horizon. Make a graph D. Compare LPS from both models with GA test

E. Are the results the same as in Kolasa and Rubaszek?

Notes: perform model evaluation using data from the period 1995:1-2013:4, as in Kolasa M.

& Rubaszek M., 2018. Does the foreign sector help forecast domestic variables in DSGE