Average relative accuracy indices

Daniele Girolimetto

2021-05-21

Let \[ \hat{e}_{i,j,t}^{[k],h} = y_{i,t+h}^{[k]} - \hat{y}_{i,j,t}^{[k],h}, \quad \begin{array}{l} i=1,\ldots,n, \\ j=0,\ldots,J, \end{array} \; t=1,\ldots, q , \; \begin{array}{l} k \in {\cal K}, \\ h=1,\ldots,h_k , \end{array} \] be the forecast error, where \(y\) and \(\hat{y}\) are the actual and the forecasted values, respectively, suffix \(i\) denotes the variable of interest, \(j\) is the forecasting technique, where \(j=0\) is the benchmark forecasting procedure, \(t\) is the forecast origin, \({\cal K}\) is the set of the time frequencies at which the series is observed, and \(h\) is the forecast horizon, whose lead time depends on the time frequency \(k\).

Denote by A\(_{i,j}^{[k],h}\) the forecasting accuracy of the technique \(j\), computed across \(q\) forecast origins, for the \(h\)-step-ahead forecasts of the variable \(i\) at the temporal aggregation level \(k\). For example, A\(_{i,j}^{[k],h} = MSE_{i,j}^{[k],h}\) (type=“mse” in score_index() function, default), otherwise we might have A\(_{i,j}^{[k],h} = MAE_{i,j}^{[k],h}\) (type=“mae” in score_index() function) or A\(_{i,j}^{[k],h} = RMSE_{i,j}^{[k],h}\) (type=“rmse” in score_index() function), where \[ \begin{array}{rcl} MSE_{i,j}^{[k],h} & = & \displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q} \left(\hat{e}_{i,j}^{[k],h}\right)^2 \\ MAE_{i,j}^{[k],h} & = & \displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q} \left|\hat{e}_{i,j}^{[k],h}\right| \\ RMSE_{i,j}^{[k],h} & = & \sqrt{\displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q} \left(\hat{e}_{i,j}^{[k],h}\right)^2} \end{array} \]

In any case, we consider the relative version of the accuracy index \(A_{i,j}^{[k],h}\), given by: \[ r_{i,j}^{[k],h} = \displaystyle\frac{A_{i,j}^{[k],h}}{A_{i,0}^{[k],h}} , \quad i=1,\ldots,n, \quad j=0,\ldots, J, \quad k \in {\cal K}, \quad h=1, \ldots, h_k, \] and use it to compute the Average relative accuracy index of the forecasting procedure \(j\), for given \(k\) and \(h\), through the geometric mean: \[ \text{AvgRelA}_{j}^{[k],h} = \left(\displaystyle\prod_{i=1}^{n} r_{i,j}^{[k],h} \right)^{\frac{1}{n}} , \quad j=0,\ldots,J . \]

We may consider the following average relative accuracy indices for selected groups of variables/time frequencies and forecast horizons:

Average relative accuracy indices for a single variable at a given time frequency, for multiple forecast horizons \[ \label{AvgRelAikh1h2} \text{AvgRelA}_{i,j}^{[k],q_1:q_2} = \left(\prod_{h=q_1}^{q_2} r_{i,j}^{[k],h}\right)^{\frac{1}{q_2 - q_1 + 1}}, \; \begin{array}{l} i=1,\ldots,n, \\ j=0,\ldots,J, \end{array} \; \begin{array}{l} k \in {\cal K}, \\ 1 \le q_1 \le q_2 \le h_k \end{array} . %i=1,\ldots,n , \; j=0,\ldots, J, \; k \in {\cal K}, \; 1 \le q_1 \le q_2 \le h_k. \] Average relative accuracy indices for a group of variables (either all, or selected groups, e.g. a: uts, b: bts) at a given time frequency, either for a single forecast horizon or across them \[ \begin{array}{rcll} \text{AvgRelA}^{[k],h}_j & = & \left(\displaystyle\prod_{i=1}^{n} r_{i,j}^{[k],h}\right)^{\frac{1}{n}}, & j=0,\ldots,J , \; k \in {\cal K}, \; h=1,\ldots,h_k \\[.5cm] \text{AvgRelA}^{[k],h}_{a,j} & = & \left(\displaystyle\prod_{i=1}^{n_a} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k],h}_{b,j} & = & \left(\displaystyle\prod_{i=n_a+1}^{n} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_j & = & \left(\displaystyle\prod_{i=1}^{n} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n h_k}}, & \; j=0,\ldots,J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_{a,j} & = &\left(\displaystyle\prod_{i=1}^{n_a} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a h_k}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_{b,j} & = &\left(\displaystyle\prod_{i=n_a+1}^{n} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b h_k}}, & j=0,\ldots, J, \; k \in {\cal K} \end{array} \]

Average relative accuracy indices for a single variable or for a group of variables (all, a: uts, b: bts), across all time frequencies and forecast horizons \[ \begin{array}{rcll} \text{AvgRelA}_{i,j} & = & \left(\displaystyle\prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{k^*+m}}, & \begin{array}{l} i=1,\ldots,n \\ j=0,\ldots,J \end{array} \\[.15cm] \text{AvgRelA}_j & = & \left(\displaystyle\prod_{i=1}^{n} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n(k^*+m)}} , & j=0,\ldots,J \\[.15cm] \text{AvgRelA}_{a,j} & = & \left(\displaystyle\prod_{i=1}^{n_a} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a(k^*+m)}} , & j=0,\ldots,J \\[.15cm] \text{AvgRelA}_{b,j} & = & \left(\displaystyle\prod_{i=n_a+1}^{n} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b(k^*+m)}} , & j=0,\ldots,J \end{array} \]

The score_index() function outputs

The score_index() function returns a summary table called Avg\(\_\)mat (if compact option is TRUE, default), otherwise it returns a list of four tables. Fixed method \(j\), we have:

\(\textbf{all}\) \(\textbf{uts}\) \(\textbf{bts}\)
\(\textbf{m}\) \(\text{AvgRelA}^{[m]}_j\) \(\text{AvgRelA}^{[m]}_{a,j}\) \(\text{AvgRelA}^{[m]}_{b,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{k}\) \(\text{AvgRelA}^{[k]}_j\) \(\text{AvgRelA}^{[k]}_{a,j}\) \(\text{AvgRelA}^{[k]}_{b,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{1}\) \(\text{AvgRelA}^{[1]}_j\) \(\text{AvgRelA}^{[1]}_{a,j}\) \(\text{AvgRelA}^{[1]}_{b,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{all}\) \(\text{AvgRelA}_j\) \(\text{AvgRelA}_{a,j}\) \(\text{AvgRelA}_{b,j}\)
\({\cal K}\) \(\textbf{m}\) \(\dots\) \(\textbf{k}\) \(\dots\) \(\textbf{1}\)
\(\textbf{h}\) \(\textbf{1}\) \(\dots\) \(\textbf{1}\) \(\dots\) \(\mathbf{h_k}\) \(\dots\) \(\textbf{1}\) \(\dots\) \(\mathbf{m}\)
\(\textbf{1}\) \(\text{RelA}^{[m],1}_{1,j}\) \(\dots\) \(\text{RelA}^{[k],1}_{1,j}\) \(\dots\) \(\text{RelA}^{[k],h_k}_{1,j}\) \(\dots\) \(\text{RelA}^{[1],1}_{1,j}\) \(\dots\) \(\text{RelA}^{[1],m}_{1,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{i}\) \(\text{RelA}^{[m],1}_{i,j}\) \(\dots\) \(\text{RelA}^{[k],1}_{i,j}\) \(\dots\) \(\text{RelA}^{[k],h_k}_{i,j}\) \(\dots\) \(\text{RelA}^{[1],1}_{i,j}\) \(\dots\) \(\text{RelA}^{[1],m}_{i,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{n}\) \(\text{RelA}^{[m],1}_{n,j}\) \(\dots\) \(\text{RelA}^{[k],1}_{n,j}\) \(\dots\) \(\text{RelA}^{[k],h_k}_{n,j}\) \(\dots\) \(\text{RelA}^{[1],1}_{n,j}\) \(\dots\) \(\text{RelA}^{[1],m}_{n,j}\)
\({\cal K}\) \(\textbf{m}\) \(\dots\) \(\textbf{k}\) \(\dots\) \(\textbf{1}\)
\(\textbf{h}\) \(\textbf{1:1}\) \(\dots\) \(\textbf{1:1}\) \(\dots\) \(\mathbf{1:h_k}\) \(\dots\) \(\textbf{1:1}\) \(\dots\) \(\mathbf{1:m}\)
\(\textbf{1}\) \(\text{RelA}^{[m],1:1}_{1,j}\) \(\dots\) \(\text{RelA}^{[k],1:1}_{1,j}\) \(\dots\) \(\text{RelA}^{[k],1:h_k}_{1,j}\) \(\dots\) \(\text{RelA}^{[1],1:1}_{1,j}\) \(\dots\) \(\text{RelA}^{[1],1:m}_{1,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{i}\) \(\text{RelA}^{[m],1:1}_{i,j}\) \(\dots\) \(\text{RelA}^{[k],1:1}_{i,j}\) \(\dots\) \(\text{RelA}^{[k],1:h_k}_{i,j}\) \(\dots\) \(\text{RelA}^{[1],1:1}_{i,j}\) \(\dots\) \(\text{RelA}^{[1],1:m}_{i,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{n}\) \(\text{RelA}^{[m],1:1}_{n,j}\) \(\dots\) \(\text{RelA}^{[k],1:1}_{n,j}\) \(\dots\) \(\text{RelA}^{[k],1:h_k}_{n,j}\) \(\dots\) \(\text{RelA}^{[1],1:1}_{n,j}\) \(\dots\) \(\text{RelA}^{[1],1:m}_{n,j}\)
\(\textbf{m}\) \(\dots\) \(\textbf{k}\) \(\dots\) \(\textbf{1}\) \(\dots\) \(\mathbf{all}\)
\(\textbf{1}\) \(\text{AvgRelA}^{[m]}_{1,j}\) \(\dots\) \(\text{AvgRelA}^{[k]}_{1,j}\) \(\dots\) \(\text{AvgRelA}^{[1]}_{1,j}\) \(\dots\) \(\text{AvgRelA}_{1,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{i}\) \(\text{AvgRelA}^{[m]}_{i,j}\) \(\dots\) \(\text{AvgRelA}^{[k]}_{i,j}\) \(\dots\) \(\text{AvgRelA}^{[1]}_{i,j}\) \(\dots\) \(\text{AvgRelA}_{i,j}\)
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
\(\textbf{n}\) \(\text{AvgRelA}^{[m]}_{n,j}\) \(\dots\) \(\text{AvgRelA}^{[k]}_{n,j}\) \(\dots\) \(\text{AvgRelA}^{[1]}_{n,j}\) \(\dots\) \(\text{AvgRelA}_{n,j}\)
\({\cal K}\) \(\textbf{m}\) \(\dots\) \(\textbf{k}\) \(\dots\) \(\textbf{1}\)
\(\textbf{h}\) \(\textbf{1}\) \(\dots\) \(\textbf{1}\) \(\dots\) \(\mathbf{h_k}\) \(\dots\) \(\textbf{1}\) \(\dots\) \(\mathbf{m}\)
\(\textbf{all}\) \(\text{AvgRelA}^{[m],1}_{j}\) \(\dots\) \(\text{AvgRelA}^{[k],1}_{j}\) \(\dots\) \(\text{AvgRelA}^{[k],h_k}_{j}\) \(\dots\) \(\text{AvgRelA}^{[1],1}_{j}\) \(\dots\) \(\text{AvgRelA}^{[1],m}_{j}\)
\(\textbf{a}\) \(\text{AvgRelA}^{[m],1}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[k],1}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[k],h_k}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[1],1}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[1],m}_{a,j}\)
\(\textbf{b}\) \(\text{AvgRelA}^{[m],1}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[k],1}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[k],h_k}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[1],1}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[1],m}_{b,j}\)
\({\cal K}\) \(\textbf{m}\) \(\dots\) \(\textbf{k}\) \(\dots\) \(\textbf{1}\)
\(\textbf{h}\) \(\textbf{1:1}\) \(\dots\) \(\textbf{1:1}\) \(\dots\) \(\mathbf{1:h_k}\) \(\dots\) \(\textbf{1:1}\) \(\dots\) \(\mathbf{1:m}\)
\(\textbf{all}\) \(\text{AvgRelA}^{[m],1:1}_{j}\) \(\dots\) \(\text{AvgRelA}^{[k],1:1}_{j}\) \(\dots\) \(\text{AvgRelA}^{[k],1:h_k}_{j}\) \(\dots\) \(\text{AvgRelA}^{[1],1:1}_{j}\) \(\dots\) \(\text{AvgRelA}^{[1],1:m}_{j}\)
\(\textbf{a}\) \(\text{AvgRelA}^{[m],1:1}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[k],1:1}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[k],1:h_k}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[1],1:1}_{a,j}\) \(\dots\) \(\text{AvgRelA}^{[1],1:m}_{a,j}\)
\(\textbf{b}\) \(\text{AvgRelA}^{[m],1:1}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[k],1:1}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[k],1:h_k}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[1],1:1}_{b,j}\) \(\dots\) \(\text{AvgRelA}^{[1],1:m}_{b,j}\)

Examples

library(FoReco)
data(FoReco_data)

# Cross-temporal framework
oct_recf <- octrec(FoReco_data$base, m = 12, C = FoReco_data$C,
                   comb = "bdshr", res = FoReco_data$res)$recf
oct_score <- score_index(recf = oct_recf,
                         base = FoReco_data$base,
                         test = FoReco_data$test, m = 12, nb = 5)

# Cross-sectional framework#'
# monthly base forecasts
id <- which(simplify2array(strsplit(colnames(FoReco_data$base), split = "_"))[1, ] == "k1")
mbase <- t(FoReco_data$base[, id])
# monthly test set
mtest <- t(FoReco_data$test[, id])
# monthly residuals
id <- which(simplify2array(strsplit(colnames(FoReco_data$res), split = "_"))[1, ] == "k1")
mres <- t(FoReco_data$res[, id])
# monthly reconciled forecasts
mrecf <- htsrec(mbase, C = FoReco_data$C, comb = "shr", res = mres)$recf
# score
hts_score <- score_index(recf = mrecf, base = mbase, test = mtest, nb = 5)

# Temporal framework
data(FoReco_data)
# top ts base forecasts ([lowest_freq' ...  highest_freq']')
topbase <- FoReco_data$base[1, ]
# top ts residuals ([lowest_freq' ...  highest_freq']')
topres <- FoReco_data$res[1, ]
# top ts test ([lowest_freq' ...  highest_freq']')
toptest <- FoReco_data$test[1, ]
# top ts recf ([lowest_freq' ...  highest_freq']')
toprecf <- thfrec(topbase, m = 12, comb = "acov", res = topres)$recf
# score
thf_score <- score_index(recf = toprecf, base = topbase, test = toptest, m = 12)

References

Di Fonzo, T., Girolimetto, D. (2020), Cross-Temporal Forecast Reconciliation: Optimal Combination Method and Heuristic Alternatives, Department of Statistical Sciences, University of Padua, arXiv:2006.085701.


  1. Appendix 7: Average relative accuracy indices for selected groups of variables/time frequencies/forecast horizons, in a rolling forecast experiment↩︎