# Code for "ProbHardE2E" 


This is a fork of [GluonTS](https://github.com/awslabs/gluon-ts/tree/master) accompanying the paper 
"End-to-End Learning of Coherent Probabilistic Forecasts for Hierarchical Time Series" presented at ICML 2021.

## Setup

The code is written in [GluonTS](https://github.com/awslabs/gluon-ts/tree/master), 
we recommend installing it the following way 

```
pip install --upgrade mxnet
pip install -e .
```

The model will also be released in GluonTS mainline, this fork is created to keep a version with results as close as possible to 
the one published in the paper. We do encourage you, however, to try out GluonTS mainline as well; due to code changes on mainline, results may 
change over time there.

(**Skip this step if you want to run only our method.**) We also provide a python wrapper for running existing hierarchical methods that were implemented in R.
To run them, `rpy2` must be installed along with R and `hts` package: 

```
pip install rpy2==2.9
pip install jinja2
R -e 'install.packages(c("hts"), repos="https://cloud.r-project.org")'
``` 
For running the competing method `PERMBU_MINT`, more packages need to be installed; in case of any issues, check R-specific [README.md](https://github.com/rshyamsundar/gluonts-hierarchical-ICML-2021/tree/master/src/gluonts/model/r_forecast/R), which provides help for both ubuntu and mac environments.
``` 
R -e 'install.packages(c("here", "SGL", "matrixcalc", "igraph", "gsl", "copula", "sn", "scoringRules", "fBasics", "msm", "gtools", "lubridate", "forecast", "abind", "glmnet", "propagate", "SuppDists"))'
```

## Running

All the methods compared in the paper can be run as follows. Our method is denoted as "ProbHardE2E".

```
python experiments/run_experiment_with_best_hps.py --dataset dataset --method method
```
where dataset is one of `{labour, traffic, tourism, tourismlarge, wiki}` and method is one of `{ProbHardE2E, HierE2E, DeepVAR, DeepVARPlus, ETS_NaiveBU, ARIMA_NaiveBU, ETS_MINT_shr, ETS_MINT_ols, ARIMA_MINT_shr, ARIMA_MINT_ols, ETS_ERM, ARIMA_ERM, PERMBU_MINT}`.                        


This will run the selected method 5 times on the selected dataset with the hyperparameters used in the paper. This script also saves the results (level-wise as well as overall scores) in `experiments/results`.

One can also limit the number of repetitions of the same method using the command line argument `num-runs`:

```
python experiments/run_experiment_with_best_hps.py --dataset dataset --method method --num-runs 1
```
This allows doing the multiple runs of the same method in parallel.

The following script fetches the saved results of previous runs and prints the mean and standard deviation over multiple runs (controlled by `num-runs`):

```
python experiments/show_results.py --dataset dataset --method method --num-runs 5 
```
If results are available for fewer number of runs, then mean/std is calculated over only those results available in `experiments/results` folder.
