Experiments

The experiments screen allows users to evaluate the quality of a model generated by TIM. This is done in a process called backtesting. Backtesting involves building a model on historical data and then assessing its quality by applying it to ‘unseen’ data. This gives an impression of how the model would behave in production, i.e. on new data. When a certain model produces satisfying results in backtesting, the model can be moved into production. This is done by building a new model, with the same settings, on all of the available data.

To backtest, follow these steps in the experiments screen:

1. Choose a model building definition.
2. Select an in-sample period. By default, the in-sample period corresponds to the first 2/3 of the dataset, but this can be changed. The model will be built with all of the MBD's properties, except the training region, which will be replaced by this in-sample period. Click the "Confirm"-button to build the model.
3. Aggregated predictions for the in-sample part and a treemap get displayed.
4. Click on "Validate model" to select an out-of-sample period. This period should correspond to data that the model hasn't seen yet. By default, the out-of-sample region corresponds to the last 1/3 of the dataset, butthis can be changed. Click the "Confirm"-button to validate the model.
5. Another row of aggregated predictions will be returned, this time calculated on the selected out-of-sample period.

Anytime you click "Build Model", the current experiment will be discarded and a new one will start.