Skip to main content

What is an Experiment

In Data Science, "back-testing" is a term that refers to testing a predictive models on historical data. Experimenting in context of TIM Studio means nothing else than back-testing with your data and settings to TIM Engine to influence how models are built.

TIM Studio makes experimenting as intuitive as possible allowing you to rely on Auto (default) value in most of the parameters, however, if you are interested in fine tuning mathematical, or other settings that is equally possible.

MLDevOpsIconML.png

Experiments belong to ML in MLDevOps. TIM automates this process by automated model building.

Part of bigger picture​

An Experiment always belongs to (business) Use Case, it can hold multiple Experiments.

UseCaseExperimentIterationDataset.png

Use Case can hold multiple experiments

Iterations​

Every time you change tiny detail (e.g. prediction horizon, mathematical parameter etc.) you basically initiate creation of the new experiment Iteration which will be built by TIM engine. When built is complete it will contain:

  • settings to TIM Engine,
  • input data (based on selected version of your Dataset),
  • Model Zoo built by TIM Engine,
  • accuracy results.

Iteration cannot be changed, it is immutable.

Each Experiment can hold multiple Iterations.

Dataset and Experiment​

An Experiment is tightly coupled with Dataset, thus if you want to back-test with different Dataset, you need to create separate Experiment.

Each Experiment Iteration can use different version of Dataset (if it is available), this offers great flexibility, and at the same time protects your research work after you have updated existing Dataset with new values.

Getting to Production​

Once you are satisfied with the settings of certain Iteration, activate it as Production setup.