TIM Edge - example notebook

This notebook illustrates how to use the TIM Edge REST API with a simple example. A model is uploaded, a forecast is run with the uploaded model, its status and logs are checked and results are retrieved. Finally, the notebook shows how to visualize the dataset and evaluate the calculated forecast.

In [1]:
import os, requests, json
import pandas as pd
import plotly as plt
import plotly.express as px
import plotly.graph_objects as go
from io import StringIO

Resources and definitions

In [2]:
api_url = 'http://localhost:8079'
headers = {}

def pretty_print(json_object):
    print(json.dumps(json_object, indent=2))

dataset = {
    'name': 'data',
    'path': os.path.join(os.getcwd(), 'data.csv'),
    'timestamp_format': 'yyyy-mm-dd HH:MM:SS',
    'decimal_separator': '.',
    'csv_separator': ';'
}

model = {
    'name': 'model',
    'path': os.path.join(os.getcwd(), 'model.json'),
}

Dataset visualization

The following dataset contains several predictors as well as electricity load (Load) as target variable that should be forecasted.

In [3]:
pd_data = pd.read_csv(dataset['path'], delimiter=dataset['csv_separator'])
nrow, ncol = pd_data.shape
cols = pd_data.columns

fig = plt.subplots.make_subplots(rows=2, cols=1, shared_xaxes=True, vertical_spacing=0.02)
fig.add_trace(go.Scatter(x=pd_data.loc[:, cols[0]], y=pd_data.loc[:, cols[1]], name=cols[1]), row=1, col=1)

for col in cols[2:ncol]:
    fig.add_trace(go.Scatter(x=pd_data.loc[:, cols[0]], y=pd_data.loc[:, col], name=col), row=2, col=1)

fig.update_layout(height=700, width=1000, title_text="Data visualization")

fig.show()

Upload a new model

POST /forecast-jobs/model

We first need to upload a model that was built on historical data.

In [4]:
files = [('model', (model['name'], open(model['path'], 'rb'), 'application/json'))]

post_model_response = requests.request("POST", f'{api_url}/forecast-jobs/model', headers=headers, files=files)

pretty_print(post_model_response.json())

{
  "message": "Model successfully uploaded.",
  "code": "WR41711"
}

Get the latest uploaded model

GET /forecast-jobs/model

Now, the latest uploaded model is retrieved. In this case, it is the model that was uploaded in the previous API call.

In [5]:
get_model_response = requests.get(f'{api_url}/forecast-jobs/model', headers=headers)

# pretty_print(get_model_response.json())

Execute a prediction using the latest uploaded model

POST /forecast-jobs/predict/execute

After successfully uploading the model, it's time to calculate a forecast with it. To do this, the POST /forecast-jobs/predict/execute endpoint should be called, with a dataset included in the request payload. Optionally, the payload can also contain a JSON configuration. In this example, the dataset visualized in the beginning of this notebook will be used. In the configuration we set the prediction horizon (predictionFrom, predictionTo) to next 24 samples and we validate the model on last 20 days (outOfSampleRows).

In [6]:
p = {
  'data': {
    'timestampFormat': dataset['timestamp_format'],
    'decimalSeparator': dataset['decimal_separator'],
    'csvSeparator': dataset['csv_separator'],
    'outOfSampleRows': {
      'baseUnit': 'Day',
      'value': 20
    }
  },
  'configuration': {
    'predictionFrom': {
      'baseUnit': 'Sample',
      'value': 1
    },
    'predictionTo': {
      'baseUnit': 'Sample',
      'value': 24
    }
  }
}

payload = {
    'configuration': json.dumps(p)
}

files = [('dataset', (dataset['name'], open(dataset['path'], 'rb'), 'text/csv'))]

execute_response = requests.post(f'{api_url}/forecast-jobs/predict/execute', headers=headers, data=payload, files=files)

pretty_print(execute_response.json())

{
  "message": "Forecast job was successfully executed.",
  "code": "WR42710"
}

Get log and status of the latest executed job

GET /forecast-jobs/log

GET /forecast-jobs/status

GET /forecast-jobs/status/collect

While the prediction is running, its log can be checked, as well as the current status and historical status records.

In [7]:
log_response = requests.get(f'{api_url}/forecast-jobs/log', headers=headers)

pretty_print(log_response.json())

[
  {
    "message": "Execution finished.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.854"
  },
  {
    "message": "Saving results.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.784"
  },
  {
    "message": "Starting prediction.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.740"
  },
  {
    "message": "Difficulty: 33.02459208778084",
    "messageType": "Debug",
    "createdAt": "2021-09-14 13:59:46.735"
  },
  {
    "message": "TIM has detected no new situations and will use the attached Model Zoo.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.731"
  },
  {
    "message": "Validation successful.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.720"
  },
  {
    "message": "Used sampling period 3600 seconds.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.706"
  },
  {
    "message": "TIM has detected and will consider this dataset to be daily-cycle.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.705"
  },
  {
    "message": "Execution started.",
    "messageType": "Info",
    "createdAt": "2021-09-14 13:59:46.703"
  }
]
In [8]:
status_response = requests.get(f'{api_url}/forecast-jobs/status', headers=headers)

pretty_print(status_response.json())

{
  "status": "Finished",
  "progress": "100",
  "CPU": "0",
  "memory": "0",
  "createdAt": "2021-09-14 13:59:46.821"
}
In [9]:
status_collect_response = requests.get(f'{api_url}/forecast-jobs/status/collect', headers=headers)

pretty_print(status_collect_response.json())

[
  {
    "status": "Finished",
    "progress": "100",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.821"
  },
  {
    "status": "Running",
    "progress": "99",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.821"
  },
  {
    "status": "Running",
    "progress": "98",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.820"
  },
  {
    "status": "Running",
    "progress": "97",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.819"
  },
  {
    "status": "Running",
    "progress": "96",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.818"
  },
  {
    "status": "Running",
    "progress": "95",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.817"
  },
  {
    "status": "Running",
    "progress": "94",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.817"
  },
  {
    "status": "Running",
    "progress": "93",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.816"
  },
  {
    "status": "Running",
    "progress": "92",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.815"
  },
  {
    "status": "Running",
    "progress": "90",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.814"
  },
  {
    "status": "Running",
    "progress": "90",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.789"
  },
  {
    "status": "Running",
    "progress": "89",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.788"
  },
  {
    "status": "Running",
    "progress": "88",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.787"
  },
  {
    "status": "Running",
    "progress": "87",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.787"
  },
  {
    "status": "Running",
    "progress": "86",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.786"
  },
  {
    "status": "Running",
    "progress": "85",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.785"
  },
  {
    "status": "Running",
    "progress": "83",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.784"
  },
  {
    "status": "Running",
    "progress": "80",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.735"
  },
  {
    "status": "Running",
    "progress": "20",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.722"
  },
  {
    "status": "Running",
    "progress": "0",
    "CPU": "0",
    "memory": "0",
    "createdAt": "2021-09-14 13:59:46.704"
  }
]

Get result of the latest executed job

GET /forecast-jobs/results/table

Once the calculation of the forecast is finished, the result can be retrieved as a table containing the predictions and other useful information.

In [10]:
table_response = requests.get(f'{api_url}/forecast-jobs/results/table', headers=headers)

df = pd.read_csv(StringIO(table_response.text), sep =",")
print(df)

                   timestamp   date_from     time_from   target      forecast  \
0    2014-12-12 00:00:00.000  2014-12-11  23:00:00.000  17538.7           NaN   
1    2014-12-12 01:00:00.000  2014-12-11  23:00:00.000  16944.8           NaN   
2    2014-12-12 02:00:00.000  2014-12-11  23:00:00.000  16572.5           NaN   
3    2014-12-12 03:00:00.000  2014-12-11  23:00:00.000  16646.7           NaN   
4    2014-12-12 04:00:00.000  2014-12-11  23:00:00.000  16736.3           NaN   
..                       ...         ...           ...      ...           ...   
499  2015-01-01 19:00:00.000  2014-12-31  23:00:00.000      NaN  20002.502561   
500  2015-01-01 20:00:00.000  2014-12-31  23:00:00.000      NaN  18503.197455   
501  2015-01-01 21:00:00.000  2014-12-31  23:00:00.000      NaN  17779.569112   
502  2015-01-01 22:00:00.000  2014-12-31  23:00:00.000      NaN  17393.586962   
503  2015-01-01 23:00:00.000  2014-12-31  23:00:00.000      NaN  16639.688308   

    forecast_type relative_distance  model_index  samples_ahead   lower_bound  \
0     OutOfSample               D+1          NaN              1           NaN   
1     OutOfSample               D+1          NaN              2           NaN   
2     OutOfSample               D+1          NaN              3           NaN   
3     OutOfSample               D+1          NaN              4           NaN   
4     OutOfSample               D+1          NaN              5           NaN   
..            ...               ...          ...            ...           ...   
499    Production               D+1         20.0             20  19550.407964   
500    Production               D+1         21.0             21  18058.340947   
501    Production               D+1         22.0             22  17374.920289   
502    Production               D+1         23.0             23  17032.656489   
503    Production               D+1         24.0             24  16294.861996   

      upper_bound       bin  
0             NaN  S+1:S+24  
1             NaN  S+1:S+24  
2             NaN  S+1:S+24  
3             NaN  S+1:S+24  
4             NaN  S+1:S+24  
..            ...       ...  
499  20461.325510  S+1:S+24  
500  18911.184223  S+1:S+24  
501  18153.922031  S+1:S+24  
502  17750.351517  S+1:S+24  
503  16971.766316  S+1:S+24  

[504 rows x 12 columns]

Add predictions to the visualization

In [11]:
fig.add_trace(go.Scatter(x=df.loc[df["forecast_type"]=="OutOfSample", "timestamp"], \
  y=df.loc[df["forecast_type"]=="OutOfSample", "forecast"], name="Forecast - out of sample", \
  line=dict( color='Orange', width=1)), row=1, col=1)

fig.add_trace(go.Scatter(x=df.loc[df["forecast_type"]=="Production", "timestamp"], \
  y=df.loc[df["forecast_type"]=="Production", "forecast"], name="Forecast - production", \
  line=dict( color='Coral')), row=1, col=1)

fig.show()