TIM forecasting output and how to read it
A typical forecasting output might look like the following table:
When you look at the table, you can notice that there are multiple different "yhat" values for a single timestamp indicating that different forecasts were made for this point in time. We will try to explain why that happens.
Why multiple forecasts for the same timestamp?¶
When TIM is deployed in production, it looks at the current situation and selects the most appropriate model from the Model Zoo.
For example, consider a Model Zoo prepared to forecast at both 07:00 and 10:00, for both one day ahead and two days ahead. If a user wants to make a forecast at "2019-03-02 07:05:35" for "2019-03-03 15:00:00", TIM will automatically recognize that the data availability corresponds to the 07:00 scenario and that the desired forecast corresponds to the one day ahead scenario.
TIM's ability to recognize the current situation is important; the desired forecast could also have been made in other situations, for example using the data availability scenario at 10:00 the previous day with the two days ahead scenario. This would result in less accurate forecasts, as this mode would ignore the most recent available data. TIM's ability to recognize situations thus allows to seamlessly select the best possible model to ensure the best possible forecasts are made. When using TIM to regularly make forecasts based on deployed models, TIM is able to automate all of this.
However, the situation gets more complicated when doing so-called backtesting. A user might be interested in a model's performance, before actually deploying it in production. Therefore, models are often tested on historical data before they are deployed. Every historical timestamp could be "forecasted" from different perspectives because the data is available to support these multiple views. That is why each timestamp might have multiple forecasts in multiple qualities.
What information can I find in the output table?¶
- datetime Timestamp of the record for which the forecast was done. In cases where user wants forecasts for records that are not part of his forecasting routine (e.g. forecast for 3 PM with a model trained to do 1 sample ahead at midnight), the timestamp won't be listed.
- date from A specific date from which the forecast would have been done (backtesting) or was done (production).
- time from A specific time from which the forecast would have been done (backtesting) or was done (production).
- target Actual value. In case this information is not available to TIM, value missing is used.
- forecast Forecast for the given record.
- forecast type There are 3 possible values. 0 indicates that the forecast was made for a record that was used for model building (in-sample backtest). 1 indicates that the forecast was made for a record that was not used for model building but is still a known historical value (out-of-sample backtest). 2 indicates that the forecast belongs to the desired forecasting horizon (production forecast). Indicating whether this is the best forecast possible for the given timestamp. Can only be true for out-of-sample records (see below).
- relative distance
Relative distance of the record from the date and time it was forecast from. In cases where data is sampled with a sampling period from interval
, it is given in days (e.g. D+7). In cases where it is not, it is given in samples (e.g. S+5).
- model index Index of a model from the Model Zoo that was used to create the forecast.
- samples ahead Distance of the forecast record from the last available target sample measured in samples. "Available" stands for "could be used in the simulated/current forecasting situation".
- lower bound Lower bound of the prediction interval for the given record forecast.
- upper bound Upper bound of the prediction interval for the given record forecast.
- bin Serves to split forecasts into continuously plottable signals.
How do I extract information from the output table?¶
To meaningfully measure accuracy or plot the output structure, one can use many filters. Plots can be created only for filters where there are no duplicit "datetime" values. In general, it is preferable to distinguish between "out-of-sample" and "in-sample" values.
- All forecasts with "forecast type" "Production" This filters forecasts and records that should be looked at when doing a real time production forecast.
- All forecasts This filter can not be plotted, however by averaging its accuracy fields we can get a good idea about how our model performs in general.
- Forecasts for a specific "bin" Easiest filter to plot a continuous signal. Concatenated from different raw forecasts (see below) and split to different parts of the forecasting horizon so that the signals do not overlap.
- Forecasts with a specific "samples ahead" Can be plotted. Provides a basic idea about how the accuracy deteriorates with the rising forecasting horizon. Good for backtesting.
- Forecasts with a specific "time from" and "relative distance" Can be plotted. Especially useful in the daily repeating scenarios where one forecasts multiple days ahead. Gives answers to questions like "What would my day-ahead accuracy be if I forecast every day at 7AM?". Good for backtesting.
- Forecasts from a specific "date from" and "time from" Can be plotted. Serves as a way of knowing how one specific forecast in the past would look like.
5. Forecasts for a specific "model index" Can be plotted. Gives an insight about the model creation and why the forecast looks like it does. Nice for discovering origins of faulty forecasts.