Forecasting GDP in the Netherlands and Belgium : a comparison of AR, bridge, Mixed Frequency VAR and AR-MIDAS models

(1)

University of Amsterdam

Forecasting GDP in the Netherlands

and Belgium: A comparison of AR,

bridge, Mixed Frequency VAR and

AR-MIDAS models

Author: Lucia Leijen

10533524

Master’s programme: MSc in Econometrics Track: Financial Econometrics

Date: July 14, 2017

Supervisor: dr. N.P.A. van Giersbergen Second reader: dr. K.J. van Garderen

(2)

Abstract

Gross Domestic Product (GDP) is a quarterly published variable. Other macroeconomic variables are often observed at higher frequency, such as every month. Auto Regressive Mixed Data Sampling (AR-MIDAS) models and Mixed Frequency VAR (MF-VAR) models can deal with mixed frequency data directly, while bridge models do this in an indirect manner. The research in this thesis compares these three models and an AR model for their usage in forecasting GDP in the Netherlands and Belgium. Various monthly economic indicators are included as explanatory variables and the forecast horizon ranges from one month to one year ahead. The main results found in this thesis are as follows. A simple AR model performs quite well for forecast horizons of six months or shorter, whereas the MF-VAR model then performs very poorly. The forecast performance of the AR-MIDAS model is more constant over different forecast horizons. Most of the time, either the AR-MIDAS or the MF-VAR model yields better forecasting results than the bridge model. For the Netherlands, the AR-MIDAS model generally performs better than the MF-VAR model for a forecast horizon of eight months or shorter. For Belgium, this is five months or shorter. Taking the average of the forecasts based on different monthly macroeconomic variables within a model class, which is called the ensemble model, provides solid forecasts. Because the estimation and forecasting of the MF-VAR model is time consuming, this model might be less attractive for use in practical applications. An AR-MIDAS model might be more suitable, or an AR model for short forecast horizons.

(3)

Statement of originality

This document is written by Lucia Leijen who declares to take full responsibility for the contents of this document.

I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it. The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.

(4)

Introduction

Gross Domestic Product (GDP) is a commonly used economic indicator that measures the total value of final goods and services in a country, minus the total value of all import, in a specific period of time. It does not include more subjective aspects like living standards or quality of life. GDP can be used for the comparison of countries globally (OECD, 2017h). Forecasting GDP is a relevant issue for, among others, banks, as GDP contains information on the economy as a whole. Forecasts of GDP can indicate in which direction the economy might be going in the future, which can help banks to take appropriate measures (De Winter, 2011). Besides banks, policy makers have an interest in the state of the economy, therefore this subject can be relevant to policy makers as well (Foroni and Marcellino, 2014).

GDP is often released quarterly, while other macroeconomic variables that are possibly helpful in explaining the future course of GDP, like unemployment or government bond yield, are released every month. This raises the question whether it is possible to incorporate the monthly structure of macroeconomic indicators to help to forecast quarterly observed GDP. In fairly recent literature, two modelling approaches have been analysed quite often for the purpose of dealing with mixed frequency data directly, namely Mixed Frequency VAR models and Autoregressive Mixed Data Sampling models. These can be abbreviated to MF-VAR and AR-MIDAS respectively. In the case of quarterly observed GDP and monthly observed macroeconomic variables, these models can be employed to use monthly indica-tors directly as input of the models to forecast quarterly GDP. Both also incorporate an autoregressive structure of GDP. As its name suggests, the MF-VAR model has a multivari-ate structure, while the regression framework of an AR-MIDAS model consists of a single equation. MF-VAR models introduce a particular latent time series and are estimated in state-space form. On the other hand, AR-MIDAS models are simply said AR models for quarterly observed GDP, which are extended by including monthly indicators as explanatory variables. A transformation is applied to these monthly macroeconomic variables by mul-tiplying them with a certain lagged polynomial. All things considered, AR-MIDAS models are quite straight forward in their estimation in comparison with the MF-VAR approach.

An earlier econometric method to deal with data sampled at mixed frequencies, is the use of bridge equations. Bridge models do not deal with mixed frequency data directly like the MF-VAR and AR-MIDAS model. Instead, the high frequency data are first aggregated to low frequency data. In this research, this means that the monthly data are aggregated to quarterly data. In essence, bridge models are Autoregressive Distributed Lag models

(7)

(ADL), where the aggregated monthly indicator data and lagged values of quarterly GDP are used as explanatory variables of GDP.

The main topic of interest in this thesis is to investigate if MF-VAR and AR-MIDAS models, which are able to deal directly with mixed frequency data, are to be preferred over models that do not have that ability, like regular AR or bridge models. The models are compared based on their forecast accuracy. The analysis is carried out for two countries, the Netherlands and Belgium. This research contributes to existing literature, since to my knowledge, the AR-MIDAS or MF-VAR analysis has never been carried out for the GDP of Belgium. The main objective of this thesis is not necessarily to find forecast predictions of GDP. Instead, the main focus lays on comparing the different models, based on their forecast accuracy. Furthermore, it is analysed if the models have a different forecast accuracy for forecasts horizons ranging from one month ahead to one year ahead. That is, it is tried to answer the question if a particular model performs better for a shorter or longer forecast horizon. The influence of the use of different monthly indicators on the forecast accuracy is also investigated.

The structure of the remaining of this thesis is as follows. First, Chapter 1 gives a short overview of the outcomes of research in previously published literature. Chapter 2 gives a theoretical background about the models and discusses the methodology used in the estimation and forecasting process. In Chapter 3 the data is described and Chapter 4 reports the results of the analysis. Finally, Chapter 5 concludes.

(8)

Chapter 1

Literature review

GDP is released quarterly. This means that models to forecast GDP often have a speci-fication based on quarterly data, such as AR or ADL at a quarterly frequency (Clements and Galvão, 2008). In contrary, potential explanatory macroeconomic variables such as unemployment and industrial production are observed every month or even at a higher frequency.

To exploit the higher frequency of explanatory variables, different methods are proposed. One can use a MIDAS model, which is a univariate specification that is based on the use of lagged polynomials for the high frequency variable coefficients. A way of extending the MIDAS model is by introducing autoregressive terms, which can then be called AR-MIDAS (Kuzin et al., 2011). Another option is to use MF-VAR models, which are related to regular VAR models. This modelling approach is able to jointly explain both the high frequency and low frequency variable (Foroni and Marcellino, 2014). An advantage of MF-VAR compared to MIDAS is that this estimation method does not require to impose any restrictions on the functional dynamics a priori, which might not be flexible enough. However, this also causes that the MF-VAR approach can experience the curse of dimensionality (Kuzin et al., 2011). Where MIDAS can directly provide multi-step forecasts, MF-VAR gives iterative forecasts (Kuzin et al., 2011). In the literature, various extensions of MIDAS and MF-VAR models are considered, for example by introducing Markov switching dynamics in both methods (Foroni et al., 2015), but these are not further discussed in this thesis.

A way of forecasting without directly taking advantage of the high frequency structure of explanatory variables, is the use of bridge models. This technique is based on aggregating the high frequency variables to low frequency variables. Foroni and Marcellino (2014) find that these relatively easy bridge models perform well overall, compared to the AR-MIDAS and MF-VAR model. Moreover, they find that bridge equations are the best technique to use for some components. Baffigi et al. (2004) find that bridge models perform better than AR models.

Clements and Galvão (2008) explore whether MIDAS models perform better than quar-terly AR or ADL models that do not incorporate the monthly structure of possible explana-tory variables. Their application is based on quarterly sampled U.S. output growth and

(9)

higher frequency indicators. They indeed find that the root mean squared error is reduced by using MIDAS models for short forecast horizons in their practical exercise. AR-MIDAS models are labelled by them as a useful addition to regular MIDAS models. In theory, there is not a clearly superior method when comparing MF-VAR and MIDAS models (Kuzin et al., 2011).

Foroni and Marcellino (2014) apply bridge equations, MIDAS, AR-MIDAS and MF-VAR models to nowcast quarterly GDP growth in the Euro area, based on a large set of monthly explanatory indicators. Their conclusion is that MIDAS models performs better than MF-VAR models for short horizons.

Kuzin et al. (2011) analyse the monthly nowcast and forecast performance of MIDAS and MF-VAR models. Similarly to Foroni and Marcellino (2014), Kuzin et al. (2011) find that MIDAS models have a higher relative forecast accuracy than MF-VAR models for short horizons, up to four or five months. However, their findings suggest that the MF-VAR model provides better forecasts for horizons up to nine months. They conclude that the two methods are more to be treated as complements than substitutes. In addition, AR-MIDAS outperforms both MIDAS and MF-VAR models for forecasts up to three months.

Ghysels et al. (2016) note that MF-VAR models are more sensitive to specification errors than AR-MIDAS models, since MF-VAR models are estimated in state-space form and make use of the Kalman filter. This substantially increases the number of required parameters, which may raise complexities in the computation of the model (Ghysels et al., 2016). In other words: the MF-VAR model suffers more from the curse of dimensionality than the AR-MIDAS model (Kuzin et al., 2011). AR-MIDAS models on the other hand can be seen as a reduced form of a state-space representation. Computationally, they are easier to implement than MF-VAR models and therefore suffer less from specification errors (Ghysels et al., 2016). MF-VAR models jointly explain the high frequency and low frequency variable. If one of the two equations in the model is misspecified, this can have an effect on the other variables as well (Kuzin et al., 2011).

(10)

Chapter 2

Methodology

The objective of the chapter is to provide a theoretical background of the different models. In addition, the estimation and forecasting methods are discussed, as well as the ways of evaluating the forecast accuracy.

2.1 AR benchmark model

The goal of this thesis is to compare the forecast accuracy of the AR-MIDAS and MF-VAR model with a bridge model. A benchmark model is defined to measure the relative performance of these models against this benchmark. Both Kuzin et al. (2011) and Foroni and Marcellino (2014) use a standard AR model specification for this purpose, which is therefore followed in this thesis. This univariate model for GDP growth has a quarterly structure. A general AR(p) model for p ≥ 1 is defined as

yt= φ0+ φ1yt−1+ ... + φpyt−p+ εt, (2.1.1)

where ε_t is white noise (Tsay, 2005, p.33). In this formula y_t represents quarterly GDP, possibly transformed. The model can be consistently estimated by ordinary least squares (OLS). Parameter p is determined by Akaike’s Information Criterion (AIC), based on the data in the estimation sample. Time index t indicates a quarter of a year.

GDP forecasts for every quarter in the evaluation sample are made for different forecast horizons, ranging from one quarter ahead to one year ahead. The desired outcome is achieved by expanding the estimation sample one quarter at a time.

2.2 Bridge model

2.2.1 Bridge model specification

As noted by Foroni and Marcellino (2014), bridge models are an early technique to incorpo-rate high frequency data to predict low frequency data. In short, the high frequency data in a bridge model is aggregated to have the same frequency as the low frequency data. The

(11)

aggregated data is then used to forecast the data that is observed at the low frequency. Subsequently, the monthly structure of the explanatory variable is not directly utilised.

In this research, quarterly GDP is forecasted based on quarterly aggregates of monthly indicator data. The bridge model has the following structure:

yt= α + I X i=1 βiLiyt+ J X j=0 γjLjx(q)t + εt. (2.2.1)

Here t is measured in quarters and L is a quarterly lag operator (Foroni and Marcellino, 2014). The aggregated data is denoted by x(q)_t . The aggregation of a monthly indicator x_t to a quarterly variable is done analogously to Clements and Galvão (2008), namely by

x(q)_t = 1

3(xt+ xt−1/3+ xt−2/3). (2.2.2) Variables x(q)_t are constructed at t = 1, 2, 3, ..., which here means at the end of every quarter of a year.

2.2.2 Bridge model estimation and forecasting

The amount of lags included in the bridge model is chosen by AIC, based on the data in the estimation sample. Note that in Equation (2.2.1) y_t is modelled as if it depends on the current value of x(q)_t . This term is included, with the goal of providing h-month ahead forecasts using the bridge model.

Considering that ytis to be forecasted at least one month ahead, xt is definitely not yet

observed at time t − 1/3 or earlier. Hence x(q)_t is also unknown at those points in time. The way of constructing monthly forecasts of GDP using a bridge model is as follows. First, the monthly indicator is forecasted over the remainder of the quarter. After that, the bridge model is used to forecast quarterly GDP. Take for example a forecast horizon of two months ahead, which means that ˆy_t|t−2/3 has to be estimated. In this case, x_t−2/3 is observed, but xt and xt−1/3 are not yet known. The values ˆxt and ˆxt−1/3 are estimated based on

the univariate series of x_t−2/3 and previous observations of the monthly indicator. These monthly forecasts are obtained from an AR(p) model as described in the previous section. The resulting estimated value of the aggregated variable is ˆx(q)_t = 1₃(ˆxt+ ˆxt−1/3+ xt−2/3).

This variable is then used in the bridge model to find, in this case, a 2-month ahead forecast of y_t. To illustrate this, the 2-month ahead forecast of y_t is

ˆ yt|t−2/3= ˆα + I X i=1 ˆ βiLiyt+ ˆγ0xˆ (q) t|t−2/3+ J X j=1 ˆ γjLjx (q) t = ˆα + I X i=1 ˆ βiLiyt+ ˆγ0 1 3(ˆxt|t−2/3+ ˆxt−1/3|t−2/3+ xt−2/3) + J X j=1 ˆ γjLjx(q)t . (2.2.3)

(12)

2.3 MF-VAR model

2.3.1 MF-VAR model approach

One way to deal with mixed frequency data is the so called Mixed Frequency VAR model. This approach is based on the regular VAR model for a multivariate vector Yt,

Yt= φ0+ Φ1Yt−1+ ... + ΦpYt−p+ εt. (2.3.1)

Here Y_t, ε_t and φ₀ are k × 1 vectors and Φ_i are k × k matrices for i = 1, ..., p. Vector ε_t is defined to have expectation zero and covariance matrix Σ. Tsay (2005) notes that in the existing literature it is frequently assumed that ε_t ∼ N (0, Σ), which is a convenient but more restrictive assumption. Another way of writing down the VAR model is

Φ(L)Yt= φ0+ εt, (2.3.2)

with L being the lag operator and Φ(L) = I_k− Φ₁L − ... − ΦpLp (Tsay, 2005, p.353).

The MF-VAR model can be explained as follows. Let GDPtm denote the quarterly

observed GDP. It is treated as a monthly time series, though it is only observed every third month. Note that the time index used, t_m, now denotes months instead of quarters like in the two previous models. Subsequently, GDPtm is observed at time tm = 3, 6, 9, ....

The observations are considered missing in the other periods. Let indtm be a univariate

macroeconomic variable that is observed every month, so at t_m = 1, 2, 3, .... This monthly indicator is transformed such that it is stationary and it is called {xtm}. For example,

{x_t_m} = {∆ log(ind_t_m)}, if {log(indtm)} is an integrated process of order 1. Define {gtm} =

{log(GDP_t_m)}, which also is an integrated process of order 1. Now latent variable ∆g_t∗_m = g_t∗_m − g∗_t

m−1 is introduced. It is the monthly equivalent of the quarterly variable ∆3gtm,

where ∆₃gtm = gtm − gtm−3. So if ∆3gtm is the GDP growth in one quarter, ∆g

∗ tm is

the month-on-month growth of GDP. This variable is purely theoretical, as it is not truly observed.

Assume that ∆₃gtmis the average of the quarterly GDP growths in the last three months,

of which one is observed and two are unobserved; it is the average of ∆3gtm, ∆3gtm−1 and

∆3gtm−2. In addition, three consecutive month-on-month growths of GDP are equal to the

GDP growth in one quarter: ∆₃gtm = ∆g

∗

tm + ∆g

∗

tm−1 + ∆g

∗

(13)

results in an expression of ∆₃gtm in the latent variables, namely ∆3gtm = 1 3∆3gtm+ 1 3∆3gtm−1+ 1 3∆3gtm−2 = 1 3(∆g ∗ tm+ ∆g ∗ tm−1+ ∆g ∗ tm−2) + 1 3(∆g ∗ tm−1+ ∆g ∗ tm−2+ ∆g ∗ tm−3) +1 3(∆g ∗ tm−2+ ∆g ∗ tm−3+ ∆g ∗ tm−4) = 1 3∆g ∗ tm+ 2 3∆g ∗ tm−1+ ∆g ∗ tm−2+ 2 3∆g ∗ tm−3+ 1 3∆g ∗ tm−4 = (1 3 + 2 3L + L 2₊2 3L 3₊1 3L 4_)∆g∗ tm. (2.3.3)

This can be called the aggregation equation (Foroni and Marcellino, 2014). Here, L is a monthly lag operator. Renaming y_t_m = ∆3gtm as the variable that eventually is to be

forecasted and naming y_t∗_m = ∆g_t∗_m, leads to an expression of (ytm, xtm)

0_: ytm xtm ! =h 1 3 0 0 1 ! + 2 3 0 0 0 ! L + 1 0 0 0 ! L2+ 2 3 0 0 0 ! L3+ 1 3 0 0 0 ! L4i y ∗ tm xtm ! = [H0+ H1L + H2L2+ H3L3+ H4L4] y∗_t_m xtm ! = H(L) y ∗ tm xtm ! , (2.3.4) where simply the result of the derivation in Equation (2.3.3) and the fact that x_t_m = xtm

are used. Taking the expectations, defined as

E ytm xtm ! = µy µx ! = µ and E y ∗ tm xtm ! = µ ∗ y µx ! = µ∗, (2.3.5)

the following relationship is found: ytm xtm ! − µ = H(L)h y ∗ tm xtm ! − µ∗i Ytm = H(L)Y ∗ tm. (2.3.6)

The model employs demeaned series, since Kuzin et al. (2011) find that it is often rather difficult to estimate the means, since this would increase the amount of parameters that have to be estimated. To define the MF-VAR model, it is assumed that the monthly variables

(14)

minus their expectations theoretically follow a bivariate VAR(p) process, Φ(L) h y∗ tm xtm ! − µ∗i= εtm Φ(L)Y_t∗_m = εtm. (2.3.7)

The errors are presumed to follow a bivariate normal distribution, εtm ∼ N (0, Σ) (Mariano

and Murasawa, 2010).

2.3.2 State-space representation MF-VAR model

The MF-VAR model is estimated in state-space form. Mariano and Murasawa (2010) and Foroni and Marcellino (2014) both show how to define a state-space representation of the MF-VAR model for p < 5:

stm = Astm−1+ Bztm−1, (2.3.8) Ytm = ytm xtm ! − µ = Cs_t_m. (2.3.9)

Here (2.3.8) is the transition equation and (2.3.9) is the measurement equation. The transi-tion equatransi-tion is a direct result of Equatransi-tion (2.3.7). The measurement equatransi-tion comes from Equation (2.3.4), where the variables are demeaned. The 10 × 1 state vector is

stm =          y∗_t_m− µ∗_y xtm− µx ! .. . y∗_t_m₋₄− µ∗_y xtm−4− µx !          =     Y_t∗_m .. . Y_t∗_m₋₄     . (2.3.10)

This represents the ’hidden states’, since the values of y∗_t

m are not observed. The other

matrices, A (10 × 10), B (10 × 2) and C (2 × 10) are

A = Φ1. . . Φp 02×2(5−p) I8 08×2 ! , (2.3.11) B = Σ 1/2 08×2 ! , (2.3.12) C = H0 H1 H2 H3 H4 (2.3.13) and it is assumed that z_t_m ∼ N (0, I₂). Vector Ytm contains the data that is used as input

of the model. All four parameters in matrices Φ1, ..., Φp are free. Matrix Σ1/2 has three

(15)

Σ1/2 = s11 s12 s12 s22

!

, (2.3.14)

since Σ is defined as a variance-covariance matrix.

For p ≥ 5 the transition equation and measurement equation stay the same, but the state vector and matrices A, B and C are slightly changed. The (2p) × 1 state vector becomes

stm =          y_t∗_m− µ∗_y xtm− µx ! .. . y_t∗_m_−p+1− µ∗ y xtm−p+1− µx !          =     Y_t∗_m .. . Y_t∗_m_−p+1     (2.3.15)

for p ≥ 5 and the A, B and C matrices become

A = Φ1. . . Φp−1 Φp I_2(p−1) 0_2(p−1)×2 ! , (2.3.16) B = Σ 1/2 02(p−1)×2 ! , (2.3.17) C = H0 H1 H2 H3 H4 02×2(p−5) . (2.3.18)

2.3.3 Kalman filter and smoother

An underlying principle that is used for the estimation of the MF-VAR model is the Kalman filter or Kalman smoother. They are used to estimate the expected values of the latent variables y∗_t_m (the hidden states) (Holmes et al., 2014, p.19). The aim of the Kalman filter is to update the available knowledge over time whenever new observations are available (Tsay, 2005, p.495). This means that at time t_m, E(y_t∗

m|Ftm) is determined. Here Ftm is

the information set with all known information at time tm. The Kalman smoother can

be applied when one wants to estimate E(y_t∗

m|FT). Information set FT includes all data,

including all future observations (Holmes et al., 2014, p.19). To specify the Kalman filter, the following is defined:

ztm−1 = E[stm−1|Y0, ..., Ytm−1], (2.3.19)

Ptm−1 = E[(stm−1− ztm−1)(stm−1− ztm−1)

0_], _(2.3.20)

where Ptm−1 is the mean squared error of stm−1 in matrix form. The Kalman filter uses two

types of equations. The time update equations are

z_t_m|tm−1 = E[ztm|Ftm−1] = Aztm−1, (2.3.21)

P_t_m|tm−1 = E[Ptm|Ftm−1] = APtm−1A

0_{+ BI}

(16)

Let F_t_m= CP_t_m|tm−1C

0_{, then the measurement update equations are}

If the time update equations are plugged into the measurement update equations, a recursive algorithm is found (Tusell et al., 2011).

The Kalman filter that is used for estimating the MF-VAR model is modified, as needed for the use of the Expectation-Maximisation algorithm (EM algorithm). This includes the lag one covariance smoother algorithm (Holmes et al., 2015).

2.3.4 MF-VAR model estimation and forecasting

The MF-VAR model in Equations (2.3.8) and (2.3.9) can be estimated by maximum likeli-hood. For this the MARSS−package in R is used. Not all elements of vector ytm are known,

only every third month. In theory this causes no problems, since the Kalman filter deals with these missing observations. Estimation is done by maximum likelihood via the EM algorithm. First, the optimal amount of lags in the model, p ≤ 6, is determined by AIC based on the data in the estimation sample. Using the chosen model specification, monthly forecasts are made.

The results are sensitive to the chosen starting values to start up the Kalman filter, therefore the analysis is carried out for different combinations of starting values in the neighbourhood of zero, ranging from −0.5 to 0.5. The initial values that result in the lowest sum of mean squared errors over the forecast horizons, one to twelve months ahead, are selected. Because of the demeaning of the variables in the state vector s_t_m, the initial values should be close to zero. For this reason, starting values around zero are tested.

It is possible to perform monthly forecasts using MF-VAR, since the model is able to work with missing values of quarterly GDP. For example, one, four and seven month ahead forecasts of quarterly GDP are performed at the end of February, May, August and November. Similarly, forecasting two, five and eight months ahead is done at the end of January, April, July and October. Finally, forecasting three, six or nine months ahead is done at the end of the months December, March, June and September.

2.4 AR-MIDAS model

A second way to deal directly with mixed frequency data is the AR-MIDAS approach. It requires less parameters to be estimated than the MF-VAR model. In AR-MIDAS models, the effect of the high frequency variable on the low frequency variable is directly modelled using lagged polynomials.

(17)

2.4.1 AR-MIDAS model specification

A basic MIDAS model specification without autoregressive structure can be used to forecast quarterly GDP solely based on lags of a single monthly macroeconomic variable. More generally, the high frequency variable xt is observed m times more often than the low

frequency variable yt. That means that m = 3 in the case of quarterly observed GDP growth

and monthly observed indicators. A simple MIDAS model for h-step ahead forecasting is yt= β0+ β1B(L1/m; θ)x(m)_t−h+ εt. (2.4.1)

Time index t represents quarters of a year. Here h is also quarters, so h = 1₃ for a forecast of one month ahead. If L is a quarterly lag operator, L1/3 is a monthly lag operator. For the lag operator, it holds that L(k−1)/mx(m)_t−h = x(m)_{t−h−(k−1)/m} (Clements and Galvão, 2008). B(L1/m; θ) = PK

k=1b(k; θ)L(k−1)/m is a particular lagged polynomial. K indicates

the amount of lags of the monthly indicator that are included in the model. Following both Kuzin et al. (2011) and Clements and Galvão (2008), b(k; θ) is parametrised using the exponential Almon lag function,

b(k; θ) = exp(θ1k + θ2k

2₎

PK

k=1exp(θ1k + θ2k2)

, (2.4.2)

though there are other parametrisation schemes available (Clements and Galvão, 2008). It is reasonable to assume that yt also depends on autoregressive terms, like in

pre-viously discussed models. Therefore the AR-MIDAS model is introduced, which includes autoregressive dynamics in its specification. Again, a single monthly explanatory variable is included. A seemingly logical way of including an AR(1) term of the low frequency dependent variable in the simple MIDAS model (2.4.1) is

yt= β0+ β1B(L1/m; θ)x(m)_t−h+ λyt−1+ εt. (2.4.3)

However, this would cause seasonal effects of x(m) on y, even when x(m) does not display any seasonal pattern (see Clements and Galvão (2008)).

Following the suggested solution for this problem by Clements and Galvão (2008), the AR-MIDAS model to perform h-period forecasts is defined as

yt= λyt−d+ β0+ β1B(L1/m; θ)(1 − λLd)x(m)t−h+ εt, (2.4.4)

which is also known as an AR-MIDAS model with a common factor restriction (Ghysels et al., 2016). Here y_tis forecasted h quarters ahead. The corresponding amount of quarters is d. For example, if h = 1₃,2₃, 1, then d = 1, but if h = 4₃,5₃, 2, then d = 2, etc.

(18)

patterns into account, results in yt= D X i=0 λiyt−d−i+ β0+ β1B(L1/m; θ)(1 − D X i=0 λiLd+i)x(m)_t−h+ εt. (2.4.5)

2.4.2 AR-MIDAS model estimation and forecasting

A consistent way of estimating regular MIDAS models is by nonlinear least squares. The es-timations of the AR-MIDAS model are performed in R using the midasr -package (Clements and Galvão, 2008). Nonlinear least squares estimation of the MIDAS models is very sen-sitive to the used starting values. Hence, the optimal amount of lags is chosen based on models estimated by the Nelder-Mead algorithm, since models with a wrong lag structure do not fit well. In AR-MIDAS models, the amount of lags of the monthly macroeconomic indicator and quarterly GDP is determined by AIC.

2.5 Ensemble model

The ensemble model is not a new modelling method like the models that are discussed before. Instead, it is a way of combining the forecasts of the bridge, MF-VAR or AR-MIDAS models, based on the individual monthly macroeconomic variables within a model class. For every forecast horizon h, the equal-weighted average of the model forecasts for all monthly indicators is calculated. That is

ˆ y_t|t−h= 1 N N X n=1 ˆ y_n,t|t−h, (2.5.1)

where ˆy_n,t|t−h is the h-month ahead GDP forecast based on the bridge, MF-VAR or AR-MIDAS model, where monthly indicator n is included in the model.

In the case of AR-MIDAS models, Clements and Galvão (2008) find that the AR-MIDAS ensemble model performs better than the so-called Multiple AR-MIDAS model. This M-AR-MIDAS model simultaneously includes multiple monthly indicators as explanatory variables in one model.

2.6 Forecast performance

The main goal of this thesis is to compare the forecast accuracy of the models that are defined earlier in this chapter. The models are based on the estimation sample and their performance is analysed based on their forecast accuracy in the evaluation sample.

2.6.1 Mean Squared Error

The Mean Squared Error (MSE) is a method to evaluate prediction errors. It does not take into account whether there is a positive or negative deviation with respect to the true value,

(19)

since the errors are squared.

First, the mean squared errors for the bridge, MF-VAR and AR-MIDAS model are determined separately for h-month ahead forecasts, where h = 1, ..., 12. The mean squared errors for the quarterly forecasts based on the benchmark AR(p) model for 3-, 6-, 9- and 12-month ahead forecasts are calculated as well. Next, the mean squared errors for the h-month ahead forecasts of the bridge model, MF-VAR model and AR-MIDAS are divided by the MSE of the corresponding h_AR-month ahead forecast of the benchmark AR(p) model. hAR = 3 if h = 1, 2, 3 and hAR = 6 if h = 4, 5, 6 etc, as the AR(p) model gives quarterly

forecasts of yt, whereas the other models give monthly forecasts. In formula form the MSE

is M SE(ˆyt+h|t) = 1 M M X m=1 (ˆyt+h|t;m− yt+h|t;m)2, (2.6.1)

where ˆyt+h is the h-step ahead forecast at time t and M is the amount of observations in

the evaluation sample (Tsay, 2005, p.194).

2.6.2 Mean Absolute Error

A different way of analysing the forecast accuracy of the models is the Mean Absolute Error (MAE). Instead of the sum of squared errors, the MAE takes the sum of the absolute errors. Consequently, this measure also is indifferent to positive or negative deviations of the true value.

The MSE squares the errors, therefore large errors cause a relatively high MSE compared to the MAE. A low MSE or MAE is desired, so the MSE gives a higher penalty to large errors than the MAE. The MAE is calculated as follows:

An explicit way to compare if two forecasts are equally good predictions, is by employing the Diebold-Mariano test. Say there are two h-period ahead forecasts, ˆy1_t+h|t and ˆy_t+h|t2 , and their corresponding forecast errors are ei_t+h|t= yt+h− ˆyi_t+h|t, for i = 1, 2 (Diebold and

Mariano, 1995). The loss functions are denoted by g(ei_t+h|t) and dt= g(e1_t+h|t) − g(e2_t+h|t) is

defined as the difference between the loss functions of two forecasts. The null hypothesis of the test is that two forecasts have an equal accuracy, namely H0: E(dt) = 0 (Diebold and

Mariano, 1995). The alternative hypothesis can take three different forms, namely that the forecast accuracy differs, that forecast 1 has a higher accuracy than forecast 2 or the reverse that forecast 2 is more accurate than forecast 1. Take for example g(ei_t+h|t) = (ei

t+h|t)2,

which is the squared error. The null hypothesis in words then is that the expectation of the squared errors of the two model predictions is equal. The test statistic for h-step ahead

(20)

forecasts is DM = d¯ ˆ Ω1/2, ˆ Ω ≈ 1 n[ˆγ0+ 2 h−1 X k=1 ˆ γk], (2.6.3) where ¯d = _T1 PT

t=1dt and ˆγk are kth order autocovariances of dt, that is ˆγk=cov(dc t, dt−k) (Harvey et al., 1997). For large sample sizes, the distribution of DM is approximately standard normal.

A modified test is proposed by Harvey et al. (1997), which is also implemented in the forecast package in R (Hyndman et al., 2017). The adjusted test statistic is

DM∗=

hn + 1 − 2h + n−1h(h − 1) n

i1/2

DM, (2.6.4)

which has a t-distribution with n − 1 degrees of freedom. The original test has the disad-vantage that it can become oversized for h = 2 or larger. The modified test reduces this problem and is therefore used in place of the original test (Harvey et al., 1997).

(21)

Chapter 3

Data

In this thesis, models with quarterly observed GDP as the dependent variable are considered. The GDP data for the Netherlands and Belgium is retrieved from the Organisation for Economic Co-operation and Development (OECD). The OECD provides a data set that contains the percentage change of GDP in one quarter. The percentage change is used to construct a new variable, GDPt, which is done as follows. The GDP in the first quarter of

2010 is set to 100. Based on the percentage changes from the original data set, GDP at all other observation times is determined, compared to the 100 in the first quarter of 2010.

The data of the monthly indicator ’Harmonised unemployment rate’ is also retrieved from the OECD. The harmonised unemployment rate is defined as the percentage of unemployed people in the total labour force in a country. People are considered to be unemployed if they are of working age, are available for work and have taken steps to find work, but are nonetheless without work (OECD, 2017a).

The monthly variable oil price is measured by ’Cushing, OK Crude Oil Future Contracts’ in dollars per barrel (U.S. Energy Information Administration, 2017). The rest of the macroeconomic indicators are collected from the Federal Reserve Bank of St. Louis (FRED), who credit the OECD as their source of the data. This involves ’Long-Term Government Bond Yield: 10-year’, ’Production of Total Industry’ and ’Passenger Car Registrations’. These last two variables are measured against the reference index 2010 = 100. Government bond yield is measured in percentages.

All variables are chosen based on previous literature. Foroni and Marcellino (2014) use unemployment indices, industrial production, car registrations and interest rates as macroeconomic monthly indicators in their analysis. All country dependent data is collected for the Netherlands and Belgium. GDP, the harmonised unemployment, production of total industry and passenger car registrations are seasonally adjusted. The indicators for oil price and government bond yield are not seasonally adjusted.

The data set contains GDP observations between the first quarter of 1985 and the third quarter of 2016. The observation span of the monthly indicators ranges from January 1985 up to September 2016. The estimation sample contains the observations of all variables between 1985 and 2010. To determine the forecast accuracy of the models, an evaluation

(22)

sample is used. The evaluation sample contains 23 observations of quarterly GDP, starting in quarter one of 2011 and ending in quarter three of 2016.

Tables 3.1 and 3.2 show some descriptive statistics for GDP and the monthly indicators that are used as explanatory variables, for respectively the Netherlands and Belgium. The tables also show which transformation is applied to the data.

To investigate the stationarity of the time series, three unit root tests are carried out. The first one is the Augmented Dickey-Fuller test (ADF test). The null hypothesis of this test is that the series has a unit root versus the alternative hypothesis that the series is stationary. Therefore, the null hypothesis that the series is not stationary is rejected for a p-value smaller than the significance level of 0.05. A drift (a constant) and a trend are included. Lag order k used in the test is determined by k = b(length of time series)1/3c.

The second test is called the Phillips-Perron test (PP test). This test has the same null and alternative hypothesis as the ADF test. A difference with the ADF test is that the PP test uses nonparametrics to examine the correlation in the residuals (Perron, 1988).

Finally, the third stationarity test is the Kwiatkowski-Phillips-Schmidt-Shin test (KPSS test). This test differs from the two previous ones in the sense that the null hypothesis of this test is stationarity versus the alternative of a unit root (Kwiatkowski et al., 1992).

Tables A.1 and A.2 in Appendix A show the resulting test statistics for the three dis-cussed tests for all time series. When only the natural logarithm is applied to the data, the ADF and KPSS test do not indicate stationarity for any series. Taking first differences for all time series and multiplying this by 100 for GDP, the three tests suggest stationarity in most cases. Based on the test results in these two tables, the transformations as shown in Table 3.1 and Table 3.2 are applied.

(23)

Table 3.1: Descriptive statistics, the Netherlands

Variable Frequency Mean SE Min Max Transformation

GDP (2010=100) Quarterly 64.77 26.13 23.36 108.21 ∆ log ·100 Harmonised Unemployment Rate (%) Monthly 6.08 1.64 3.10 9.50 ∆ log Crude Oil Future Contract ($/barrel) Monthly 41.85 29.05 11.31 134.02 ∆ log Government Bond Yield, 10 year (%) Monthly 5.08 2.32 0.03 9.22 ∆ log Production of Total Industry (2010=100) Monthly 84.33 12.22 59.30 107.80 ∆ log Passenger Car Registrations (2010=100) Monthly 100.27 15.71 27.10 181.90 ∆ log

Table 3.2: Descriptive statistics, Belgium

Variable Frequency Mean SE Min Max Transformation

GDP (2010=100) Quarterly 66.85 24.76 24.94 108.17 ∆ log ·100 Harmonised Unemployment Rate (%) Monthly 8.39 1.22 6.00 11.00 ∆ log Crude Oil Future Contract ($/barrel) Monthly 41.85 29.05 11.31 134.02 ∆ log Government Bond Yield, 10 year (%) Monthly 5.90 3.00 0.15 12.55 ∆ log Production of Total Industry (2010=100) Monthly 75.98 18.25 50.80 109.80 ∆ log Passenger Car Registrations (2010=100) Monthly 83.86 13.21 46.60 180.70 ∆ log

In Figures 3.1 and 3.2, the graphs show all series over time from 1985 up to 2016. The figures include the transformations of the series as well. There are no missing variables.

For the harmonised unemployment rate in Figure 3.1, the graph for the Netherlands looks smoother than the graph for Belgium. This seems odd, since the data for both countries is downloaded from the same source: the OECD. The OECD also provides harmonised unemployment rate data for other countries. Inconsistencies can be seen there as well. Similar to the Netherlands, the graphs for France and Germany follow a fairly smooth course. On the other hand, for most other countries, the graphs are bumpier like for Belgium. The graphs might indicate that there are differences in the methods used to determine the harmonised unemployment rate per country.

(24)

Figure 3.1: Graphs of quarterly GDP, harmonised unemployment rate and oil price and their transformations

(25)

Figure 3.2: Graphs of government bond rate, production and car registrations and their transformations

(26)

Chapter 4

Results

This chapter examines the estimation and forecasting results. First, remarks about the esti-mation approach and some complications are discussed. Then, in Section 4.2, the forecasting results are analysed.

4.1 Estimation results

To forecast quarterly GDP based on autoregressive lags and monthly macroeconomic indi-cators, the first step is to determine the lag structure of all models. This model structure is based on the estimation sample, containing all data from 1985 up to 2010. Afterwards, the AR, bridge, MF-VAR and AR-MIDAS models are estimated and are used to forecast GDP in the evaluation sample. The evaluation sample starts in 2011 and ends in the third quarter of 2016.

During the estimation and forecasting of the AR-MIDAS and MF-VAR models some complications arose. Both the MF-VAR and AR-MIDAS model are dependent on starting values. In the case of the AR-MIDAS models, one has to provide starting values for the parameters in the Almon lag. Nonlinear least squares is very sensitive to the selected starting values and model estimation using nonlinear least squares does not succeed in all cases. In those instances, the AR-MIDAS model is estimated and forecasted by applying the Nelder-Mead algorithm. The estimation outcomes of the MF-VAR models heavily depend on the chosen starting values, which have to be provided in order to start up the Kalman filter. For this reason, multiple combinations of initial values are tried. This is the main cause of the very time consuming estimation and forecasting of the MF-VAR models. These are clear disadvantages of the MF-VAR approach in comparison to the other approaches.

Ghysels et al. (2016) note that computational complexities are more inclined to occur during estimation and forecasting using MF-VAR models, since MF-VAR models make use of the Kalman filter, which requires more parameters than the AR-MIDAS model. The complications arising with respect to the MF-VAR model in this research are therefore in line with these findings.

(27)

hypothesis of the Breusch-Godfrey LM test that there is no serial correlation in the error terms up to order 3 is not rejected for both the Netherlands and Belgium. The Jarque-Bera test statistic indicates that the errors are not normally distributed, since the null hypothesis that the skewness and excess kurtosis are zero, is rejected at a five percent significance level. These same results also hold for all AR-MIDAS models for Belgium. When oil price is used as an explanatory variable in an AR-MIDAS model for the Netherlands, the LM test reveals evidence of the presence of serial correlation in the error terms. There is no evidence that the error terms of all other AR-MIDAS models for the Netherlands contain serial correlation.

4.2 Results of AR, bridge, MF-VAR and AR-MIDAS models

The forecast performance of all models is evaluated by comparing the mean squared errors and mean absolute errors in the evaluation sample. In Appendix A, the tables with MSE and MAE ratios can be found. Appendix B gives graphs of the forecasts of ∆ log(GDP_t)·100 for the different countries, models and monthly macroeconomic variables.

4.2.1 Comparison of the models and forecast horizons

Tables A.3 and A.5 in the appendix contain the relative mean squared errors of the bridge, AR-MIDAS and MF-VAR models against the benchmark AR model for respectively the Netherlands and Belgium. The mean squared errors are determined based on the perfor-mance of the models in the evaluation sample, where the model structure is based on the data in the estimation sample. For every monthly indicator, the MSE of the bridge, AR-MIDAS and MF-VAR model is divided by the corresponding MSE of the AR model. The tables also contain the mean squared error ratios of the average of the forecasts over the monthly indicators, within a model class. These are called ensemble models.

The forecast horizon h ranges from one month ahead to one year ahead. The lower the mean absolute error or mean squared error, the better the forecast accuracy is. A ratio less than one indicates that the AR benchmark model gives less accurate forecasts than the bridge, AR-MIDAS or MF-VAR model for a particular horizon and monthly macroeconomic variable. However, the AR model can only provide quarterly, but no monthly forecasts. To illustrate this, for h = 1, 2, 3, the one quarter ahead forecasts of the AR model are used, and for forecast horizons of h = 4, 5, 6 months ahead, the two quarters ahead AR forecasts are used, and so on.

Tables A.3 and A.5 have their layout for comparability to results found in previous research, for example the results in Table 1 of (Kuzin et al., 2011). In addition, Tables A.4 and A.6 in the appendix show the ratios based on the mean absolute errors. The mean absolute error is less affected by large deviations from the true value of GDP than the mean squared error.

A mean squared error ratio of 1.775, the most top left ratio in Table A.3, means that when forecasts are performed one month ahead, using a bridge model, with the harmonised

(28)

unemployment rate as a monthly explanatory variable, the bridge model performs 1.775 times worse than the AR model, based on the mean squared error. Analogously, the corre-sponding mean absolute error ratio is 1.201. This suggests that, based on the mean absolute error, the bridge model performs 1.201 times worse than the AR model.

The results for Belgium in Table A.5 show that for forecast horizons of h = 6 or shorter, the mean squared error ratios are almost always greater than one. This implies that the AR model provides relatively good forecasts for short forecast horizons. It also gives the impression that the inclusion of macroeconomic variables does not necessarily improve the forecasts for low values of h. For larger values of h, the ratios often drop below one, at least for the AR-MIDAS and MF-VAR models for Belgium. For the Netherlands, this is quite similar for the bridge and MF-VAR models. However, such a clear distinction if ratios are greater or less than one can not be seen for the AR-MIDAS model.

Another difference between the two countries is that the MSE ratios of the bridge, MF-VAR and AR-MIDAS ensemble models for Belgium are greater than those for the Netherlands in all but four cases. The reason why the mean squared error ratios in the tables for Belgium are higher than the ratios for the Netherlands, is that the AR model gives more accurate forecasts for Belgium than for the Netherlands most of the time. Consequently, this increases the ratios for Belgium, since more accurate forecasts correspond to lower mean squared errors. On average, the mean squared errors of the benchmark AR model for the Netherlands are 3.1 times as high as for Belgium.

In previous literature, evidence is found that MIDAS models provide better forecasts for shorter forecast horizons up to five months ahead and that MF-VAR models perform better for longer horizons (Kuzin et al., 2011). The outcomes in this research do not contradict with these findings. To illustrate this more clearly, Table 4.1 and Table 4.2 show the mean squared errors of the AR-MIDAS models divided by the mean squared errors of the MF-VAR models. Thus, a ratio less than one means that the AR-MIDAS model provides more accurate forecasts than the MF-VAR model.

Focusing on the results of the ensemble models, the MSE ratios for Belgium are exactly in line with the results found by Kuzin et al. (2011). Namely, after h = 5, the ratios become greater than one, which would indicate that the MF-VAR model starts performing better than the AR-MIDAS model. For the Netherlands, there also is a switching point, but it occurs after h = 8 instead of h = 5. No clear-cut turning point is visible for the models including single monthly macroeconomic variables. The tables in Appendix A.2 and A.3 show that in nearly all cases, either the MF-VAR or the AR-MIDAS model provides more accurate forecasts than the bridge model, which would possibly make the bridge model redundant.

(29)

Table 4.1: Relative mean squared error of AR-MIDAS model against MF-VAR model, the Netherlands

Monthly indicator h=1 h=2 h=3 h=4 h=5 h=6 h=7 h=8 h=9 h=10 h=11 h=12

Harm. unempl. rate 0.354m _0.162m _0.589m _1.079 _0.525 _0.787 _0.489 _1.192 _2.097 _1.697 _1.101 _2.233 Oil price 0.518m 0.342m 0.696 0.757 0.635 0.831 1.060 1.211 2.085a 1.867a 1.359 2.104a Gov. bond 10 year yield 1.435 0.225m _1.241 _2.430 _0.290m _1.046 _1.670 _1.055 _1.449a _2.589 _1.639 _1.611 Production 0.149m _0.122m _0.410m _0.766 _0.589m _0.652 _0.627m _0.916 _0.911 _1.115 _0.871 _0.833 Car registrations 0.458m _0.204m _0.689 _0.527 _0.591m _0.605 _0.837m _0.584 _1.026 _0.943 _1.188 _1.021 Ensemble 0.615m 0.275m 0.572 0.862 0.508m 0.807 0.867 0.965 1.344 1.502 1.414 1.419

a_{The Diebold-Mariano test reveals evidence that the MF-VAR model is more accurate than the AR-MIDAS}

model at a five percent significance level (for the same monthly indicator and forecast horizon).

m

The Diebold-Mariano test reveals evidence that the AR-MIDAS model is more accurate than the MF-VAR model at a five percent significance level (for the same monthly indicator and forecast horizon).

Table 4.2: Relative mean squared error of AR-MIDAS model against MF-VAR model, Belgium

Monthly indicator h=1 h=2 h=3 h=4 h=5 h=6 h=7 h=8 h=9 h=10 h=11 h=12

Harm. unempl. rate 0.181m _0.094m _0.366m _0.683 _0.470 _0.752 _1.110 _0.940 _0.793m _1.160 _1.183 _0.586m Oil price 0.540m _0.169m _0.590m _0.882 _1.268 _1.562 _1.226 _1.072 _1.466a _1.615a _1.396a _1.080 Gov. bond 10 year yield 2.537 1.124 2.609 0.622 0.316m _2.406 _0.847 _1.297a _0.855 _1.390a _1.710a _2.060a Production 0.455m 0.333m 0.259m 0.582m 0.370 0.973 0.889 0.980 0.735 0.724 1.054 1.225 Car registrations 0.669 0.188m _0.820 _0.640m _0.575 _0.979 _1.039 _1.126 _1.755a _1.177a _0.982 _1.276 Ensemble 0.614 0.230m _0.593 _0.759 _0.566 _1.382 _1.197 _1.128 _1.039 _1.185 _1.157 _1.108

a

The Diebold-Mariano test reveals evidence that the MF-VAR model is more accurate than the AR-MIDAS model at a five percent significance level (for the same monthly indicator and forecast horizon).

m

The Diebold-Mariano test reveals evidence that the AR-MIDAS model is more accurate than the MF-VAR model at a five percent significance level (for the same monthly indicator and forecast horizon).

The figures in Appendix B show the forecasts of transformed GDP in the evaluation sample. The graphs are shown for forecast horizons up to one year ahead and for different models and monthly macroeconomic variables. For the MF-VAR models, Figures B.3 and B.7 show that the forecasts are not very stable and fluctuate a lot, especially for shorter forecast horizons. This corresponds to the finding based on the MSE and MAE ratios that MF-VAR models do not perform well for short forecast horizons. The graphs for the other models seem to be following the true course of the transformed GDP data better than the MF-VAR model.

Kuzin et al. (2011) find mean squared error ratios ranging from 0.72 to 1.47 for AR-MIDAS and MF-VAR models. Table A.3 and Table A.5 show that the range of the mean squared error ratios for the AR-MIDAS and MF-VAR models found in this research is wider: approximately 0.45 to 22.23. High ratios are especially found for short forecast horizons. A possible cause for this is the big shock in ∆ log(GDPt) · 100 that occurs around 2009, which

can be seen in the top right graph of Figure 3.1. This shock might affect the forecast errors in 2011 (and possibly 2012 and further). The European Central Bank observed this trough in economic activity in 2009 as well. They note variation in GDP growth following this trough (European Central Bank, 2011). The mean absolute errors for both the Netherlands

(30)

and Belgium for short forecast horizons are considerably lower than their mean squared error equivalent. Several large forecasting errors will have a bigger effect on the mean squared error than on the mean absolute error.

4.2.2 Diebold-Mariano test results

In addition to the mean squared error ratios, Tables A.3 and A.5 show whether or not the Diebold-Mariano test indicates that a model provides significantly more accurate forecasts than a different model, based on the mean squared errors. For comparing a bridge, AR-MIDAS or MF-VAR model with a bridge, AR-AR-MIDAS or MF-VAR model, the test is carried out for an equal forecast horizon and the same monthly variable. When the forecast accuracy of one of these three models is tested against an AR model, the one quarter ahead forecast of the AR model is used for h = 1, 2, 3, the two quarters ahead forecast of the AR model is used for h = 4, 5, 6, etc.

For short forecast horizons, the Diebold-Mariano test regularly reveals evidence that the bridge or especially the AR-MIDAS model gives more accurate forecasts than the MF-VAR model. This is hardly the case for longer forecast horizons. Naturally, this is in line with the previous observation that the MF-VAR model does not perform well on short forecast horizons.

On the other hand, for longer forecast horizons it can be seen that the Diebold-Mariano test sometimes reveals evidence that the bridge, AR-MIDAS or MF-VAR model performs better than the AR model. Again, this is in line with the observation that the AR model performs quite well on short forecast horizons, but that on longer forecast horizons the AR-MIDAS or MF-VAR model often performs better. In many cases for the Belgian ensemble models, either the AR-MIDAS or MF-VAR model significantly has a higher forecast accuracy than the bridge model.

The test provides some outcomes that might seem odd at first sight. For example, for the ensemble models in Table A.3, for a forecast horizon of nine months ahead, the Diebold-Mariano test reveals evidence that the ensemble bridge model is significantly more accurate than the AR model. However, this is not the case for the ensemble MF-VAR model, even though the mean squared error ratio for the ensemble bridge model is 0.859, compared to a much lower ratio of 0.642 for the ensemble MF-VAR model. The natural expectation based on these ratios would be that if there is proof that the bridge model provides more accurate forecasts than the AR model, the MF-VAR model should certainly do this as well. The Diebold-Mariano test statistic depends on the difference in squared errors, but also on the autocovariances of the differences between the two squared errors. This means that apparently the effect of the autocovariances outweighs the fact that the mean squared error for the MF-VAR model is lower than the mean squared error for the bridge model.

(31)

4.2.3 Comparison of the monthly indicators

Five different macroeconomic variables are included in the bridge, AR-MIDAS and MF-VAR models to help forecast quarterly GDP. The ensemble models of the forecasts based on these five variables are also determined.

For the Netherlands, total production in a country and the amount of passenger car registrations seem to provide accurate forecasts based on the mean squared error ratios. For Belgium, oil price and total production generally give good forecasts. For short forecast horizons, the ten year government bond yield performs very poorly for Belgium. However, the overall differences between the variables are not considerably large that there are clearly favourable or unfavourable monthly indicators. The results of the mean absolute error ratios substantiate this. Since for both the Netherlands and Belgium the ensemble model performs well and provides steady forecasts for longer and shorter forecast horizons, this may be an appropriate choice.

(32)

Chapter 5

Conclusion

GDP is an economic indicator that is published every quarter, while other macroeconomic variables are observed at a higher frequency. This thesis focuses on forecasting GDP, based on autoregressive lags of GDP itself and other, more frequently observed macroeconomic variables. The analysed models are the AR, bridge, MF-VAR and AR-MIDAS models. The AR-MIDAS and MF-VAR directly incorporate mixed frequency data, whereas the bridge model is not able to do that. The AR model only uses GDP data. The accuracy of the forecasts found by these models is analysed by comparing the mean squared errors and mean absolute errors. In addition, the Diebold-Mariano test is used to test for statistical evidence if the forecast accuracies of the models significantly differ from each other.

The results found in this thesis indicate that there is not a model that clearly outper-forms all other models. However, the different models do have different forecast accuracies depending on the forecast horizon. For a forecast horizon of h = 6 or shorter for Belgium, the mean squared error and mean absolute error ratios are mostly greater than one. This would imply that the AR model performs best for short forecast horizons. In turn, this gives the impression that the macroeconomic variables do not or do hardly provide any extra relevant information to forecast GDP. For the Netherlands, the ratios for the bridge and MF-VAR model against the AR model behave quite similarly.

For a forecast horizon of seven months or more, the AR-MIDAS and MF-VAR model, and to a lesser extent the bridge model, seem to perform better than the AR model. The MF-VAR model performs very poorly for short forecast horizons. The AR-MIDAS model is more constant in its forecast performance for different forecast horizons. Based on the ensemble models for the Netherlands, the MF-VAR model starts to perform better than the AR-MIDAS model for a forecast horizon longer than eight months ahead. For Belgium, this point lies after a forecast horizon of five months ahead. Often, either the AR-MIDAS or the MF-VAR model is superior to the bridge model. Overall, the Diebold-Mariano test results substantiate these conclusions.

In terms of the monthly indicators, the ensemble model performs consistently and typi-cally well for different forecast horizons.

(33)

straightforward. The MF-VAR model, however, encounters computational difficulties and the estimation and forecasting of the MF-VAR model take up a lot of time. Compared to the other models, the resulting forecast accuracies are not remarkably impressive, therefore the MF-VAR model does not seem to be a very good choice for practical applications. Better options for use in practice may be the AR model for short forecast horizons, or the AR-MIDAS model.

In this thesis, one way of parametrising the lagged polynomial in the AR-MIDAS model is used, namely by employing the Almon lag. A direction for future research might be to explore the influence of a wider variation of lagged polynomial structures on the forecast accuracy of AR-MIDAS models. Also, the used mixed frequency data is always observed on the same two frequencies. The low frequency data, GDP, is observed quarterly and the high frequency data, the macroeconomic variables, are observed every month. In this area, further research can focus on the effect of weekly data instead of monthly data. Examples could be weekly oil prices or weekly effective federal fund rates.

Finally, this thesis has not taken into account any possible publication delays in the variables. In the practical application of forecasting GDP, one should keep this in mind.

(34)

Bibliography

Baffigi, A., Golinelli, R., and Parigi, G. (2004). Bridge models to forecast the euro area GDP. International Journal of Forecasting, 20(3):447–460.

Clements, M. P. and Galvão, A. B. (2008). Macroeconomic forecasting with mixed-frequency data: Forecasting output growth in the United States. Journal of Business & Economic Statistics, 26(4):546–554.

De Winter, J. (2011). Forecasting GDP growth in times of crisis: private sector forecasts versus statistical models (working paper November 2011).

Diebold, F. X. and Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3):253–263.

European Central Bank (2011). Output volatility in the euro area. Monthly Bulletin, 12:75– 78.

Foroni, C., Guérin, P., and Marcellino, M. (2015). Markov-switching mixed-frequency VAR models. International Journal of Forecasting, 31(3):692–711.

Foroni, C. and Marcellino, M. (2014). A comparison of mixed frequency approaches for nowcasting euro area macroeconomic aggregates. International Journal of Forecasting, 30(3):554–568.

Ghysels, E., Kvedaras, V., and Zemlys, V. (2016). Mixed frequency data sampling regression models: The R package midasr. Journal of Statistical Software, 72(4):1–35.

Harvey, D., Leybourne, S., and Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of forecasting, 13(2):281–291.

Holmes, E., Ward, E., and Scheuerell, M. (2014). Analysis of multivariate time-series using the MARSS package. NOAA Fisheries, Northwest Fisheries Science Center, 2725. Holmes, E., Ward, E., and Wills, K. (2015). Package MARSS: Multivariate autoregressive

state-space modeling, version 3.9.

Hyndman, R. J., O’Hara-Wild, M., Bergmeir, C., Razbash, S., Wang, E., and Hyndman, M. R. (2017). Package ’forecast’. https://cran. r-project. org/web/packages/forecast/fore-cast.pdf.

(35)

Kuzin, V., Marcellino, M., and Schumacher, C. (2011). MIDAS vs. mixed-frequency VAR: Nowcasting GDP in the euro area. International Journal of Forecasting, 27(2):529–542. Kwiatkowski, D., Phillips, P. C., Schmidt, P., and Shin, Y. (1992). Testing the null

hypoth-esis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root? Journal of econometrics, 54(1-3):159–178.

Mariano, R. S. and Murasawa, Y. (2010). A coincident index, common factors, and monthly real GDP. Oxford Bulletin of Economics and Statistics, 72(1):27–46.

OECD (2017a). Harmonised unemployment rate (hur) (indicator). doi: 10.1787/52570002-en. Accessed: April 19, 2017.

OECD (2017b). Long-term government bond yields: 10-year: Main (including bench-mark) for Belgium. Retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/IRLTLT01BEM156N, April 19, 2017.

OECD (2017c). Long-term government bond yields: 10-year: Main (including bench-mark) for the Netherlands. Retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/IRLTLT01NLM156N, April 19, 2017.

OECD (2017d). Passenger car registrations in Belgium. Retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/BELSACRMISMEI, April 19, 2017.

OECD (2017e). Passenger car registrations in the Netherlands. Retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/NLDSACRMISMEI, April 19, 2017.

OECD (2017f). Production of total industry in Belgium. Retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/BELPROINDMISMEI, April 19, 2017.

OECD (2017g). Production of total industry in the Netherlands. Retrieved from FRED, Fed-eral Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/NLDPROINDMISMEI, April 19, 2017.

OECD (2017h). Quarterly GDP (indicator). doi: 10.1787/b86d1fc8-en. Accessed: April 19, 2017.

Perron, P. (1988). Trends and random walks in macroeconomic time series: Further evidence from a new approach. Journal of economic dynamics and control, 12(2):297–332.

Tsay, R. S. (2005). Analysis of financial time series. John Wiley & Sons, 2 edition. Tusell, F. et al. (2011). Kalman filtering in R. Journal of Statistical Software, 39(2):1–27.

(36)

U.S. Energy Information Administration (2017). Cush-ing, OK crude oil future contract 1. Retrieved from https://www.eia.gov/dnav/pet/hist/LeafHandler.ashx?n=PETs=RCLC1f=M, April 19, 2017.

(37)

Appendix A

Tables

The first section of Appendix A shows the results of the stationarity tests for the used economic variables. After that, tables with mean squared error and mean absolute error ratios are given, for both the Netherlands and Belgium.

(38)

A.1 Stationarity test results

Table A.1: Stationarity test statistics, the Netherlands

No first differences First differences1

Variable ADF PP KPSS ADF PP KPSS

GDP -1.096 -0.654 4.184 -3.524∗ -9.156∗ 0.818 Harm. Unemployment rate -3.368 -1.408 1.882 -2.936 -13.279∗ 0.348∗ Oil price -2.683 -2.828 5.984 -7.591∗ -13.875∗ 0.066∗ Gov. bond 10 year yield 2.089 2.439 5.003 -6.193∗ -11.329∗ 0.624 Production -1.512 -5.075∗ 7.231 -9.188∗ -35.252∗ 0.050∗ Car registrations -3.343 -12.976∗ 0.995 -10.022∗ -46.266∗ 0.026∗

1

Multiplied by 100 for GDP

∗

The unit root test proposes stationarity of the time series at a five percent significance level. For the ADF test and PP test this is the case when the null hypothesis is not rejected and for the KPSS test when the null hypothesis is rejected.

Table A.2: Stationarity test statistics, Belgium

No first differences First differences1

Variable ADF PP KPSS ADF PP KPSS

GDP -1.247 -1.165 4.274 -5.269∗ -8.354∗ 0.417∗ Harm. Unemployment rate -2.666 -2.268 0.656 -4.893∗ -9.491∗ 0.110∗ Oil price -2.683 -2.828 5.984 -7.591∗ -13.875∗ 0.066∗ Gov. bond 10 year yield 1.492 2.079 5.543 -5.328∗ -12.478∗ 0.559 Production -2.687 -3.446∗ 7.541 -7.024∗ -31.553∗ 0.043∗ Car registrations -2.766 -11.710∗ 4.133 -9.434∗ -46.920∗ 0.019∗

1

Multiplied by 100 for GDP

∗

The unit root test proposes stationarity of the time series at a five percent significance level. For the ADF test and PP test this is the case when the null hypothesis is not rejected and for the KPSS test when the null hypothesis is rejected.

(39)

A.2 Results the Netherlands

A.2.1 MSE against AR the Netherlands

T able A.3 : Relativ e mean sq uared error of differen t mo dels against b enc h mark AR m o del, the Netherlands AR 1 quar te r ahead AR 2 quarters ahead AR 3 quarters ahead AR 4 quarters ahead Mon thly ind icator Mo del h = 1 h = 2 h = 3 h = 4 h = 5 h = 6 h = 7 h = 8 h = 9 h = 10 h = 11 h = 12 Bridge 1.77 5 m 1.907 m 1.600 m 1.308 1.329 1.275 1.175 1.158 1.083 1.013 1.036 0.997 Harm. unemplo ymen t rate AR-MID AS 1.2 95 m 1.189 a,b 1.644 m 1.146 1.219 1.374 1.025 1.136 1.408 1.128 1.119 1.317 MF-V AR 3.654 7.335 2.792 1.062 2.320 1.746 2.097 0.953 0.671 AR 0.665 1.016 0.590 Bridge 1.322 1.175 m 1.126 1 .258 1.225 1.153 1.326 1.332 1.207 1.242 1.263 1.152 Oil price AR-MID AS 1.154 m 1.055 m 1.240 0 .7 39 0.853 0.932 0.662 b 0.780 b 1.312 0.953 b 0.701 b 1.021 b MF-V AR 2.227 3.089 1.783 0.975 1.343 1.121 0.624 0.644 b 0.629 AR ,a,b 0.510 AR ,a,b 0.516 b 0.485 AR ,a,b Bridge 4.421 3.704 1.610 3.381 3.133 1.542 1.714 1.779 1.300 1.487 1.536 1.142 Go v. b ond 10 y ear yield AR-MID AS 6.603 3.191 m 4.064 5 .084 1.031 m,b 1.602 1.733 1.128 b 1.214 1.496 1.273 1.123 MF-V AR 4.603 14.165 3.276 2.092 3.553 1.532 1.037 b 1.070 b 0.838 a 0.578 b 0.777 b 0.697 Bridge 1.242 m 1.716 m 1.441 1 .249 1.904 1.461 0.842 1.162 0.971 0.669 0.885 0.753 Pro duction AR-MID AS 0.851 m 0.781 b,m 1.105 m 0.714 0.721 m 0.770 b 0.567 b,m 0.487 b 0.519 b 0.521 0.456 b 0.455 MF-V AR 5.697 6.412 2.695 0.931 1.224 1.182 0.903 0.531 b 0.569 0.467 AR 0.523 0.546 Bridge 1.063 m 1.069 m 1.045 1 .0 35 1.053 1.043 1 .067 1.082 1.075 1.119 1.135 1.110 Car registrations AR-MID AS 1.024 m 0.766 b,m 2.085 0 .7 63 0.665 m 0.872 0.510 m 0.526 0.716 0.503 b 0.604 b 0.528 MF-V AR 2.238 3.765 3.028 1.449 1.126 1.441 0.609 b 0.902 0.698 0.534 AR ,b 0.509 b 0.517 b Bridge 1.13 3 m 1.118 m 0.968 0 .965 0.965 0.897 AR 0.904 0.898 0.85 9 AR 0.825 0.821 AR 0.787 AR Ensem ble AR -M ID AS 1.011 m 0.920 m 1.218 0 .964 0.763 m 0.947 0.717 0.721 0.863 0.772 0.707 0.775 MF-V AR 1.643 3.349 2.129 1.118 1.500 1.175 0.827 0.748 0.642 0.514 AR ,b 0.500 AR 0.546 AR AR The Dieb old-Mariano test rev eals evidence that the bridge, AR-MID AS or MF-V AR mo del is more ac cu ra te than the AR m o del at a fiv e p ercen t sig n ificance lev el. F or h = 1 , 2 , 3 , th e 1 qu a rter ahead AR forecast is used, for h = 4 , 5 , 6 , the 2 quarters ahead AR forecast is used, and so on . a The D ie b old-Mariano test rev e als evidence th a t the m o del is more accurate than the AR-MID AS mo del at a fiv e p e rcen t signifi ca n c e lev el (for the same mon thly indicator and forecast horizon). b The Dieb old-Mariano test rev eals evidence that the mo d e l is m ore accurate than the bridge mo del at a fiv e p ercen t significance lev el (for the same mon thly indicator and fo recast horizon). m The Dieb old-Mariano test rev eals evidence that the mo del is m ore accurate than the MF-V AR mo del at a fiv e p ercen t significance lev e l (for the same mon thly indicator and forecast horizon).

Forecasting GDP in the Netherlands and Belgium : a comparison of AR, bridge, Mixed Frequency VAR and AR-MIDAS models

University of Amsterdam