Skip to main content

Performance

All benchmarks in this section were performend on hardware described in Infrastructure Requirements section.

TIM DB performance measurements

To measure performance of TIM DB, we upload 1000 datasets. POST /datasets calls are executed in sequence, persistence of data is running in parallel. When all the data is successfully uploaded, we update all the datasets - we add 24 rows to each dataset. PUT /datasets/{id} calls are executed in sequence, persistence of data is running in parallel. When all updates are finished, we delete all the datasets in sequence.

ActionConfigurationExecution time
Upload 1000 datasetsEach dataset: 12 variables, 96384 rows, 7.5MB CSV size1h 51m 15s
Update 1000 datasets with new rowsEach dataset: 12 variables, 24 rows, 2KB CSV size1h 28m 10s
Delete 1000 datasets with 2 versionsEach dataset: 12 variables, 96408 rows10m 40s

Total time: 3h 30m 06s

The measurement was performed by Python 3.11 script with use of TIM Python client v5.2.0. The TIM Python client was used in the way that corresponds to a documentation.

Following tables show different aspects of TIM DB with respect to different number of rows and columns in a dataset. In Each table, rows represent number of rows in a dataset, columns represent number of columns in a dataset.

Size of CSV file1510501001505001000
1002.6 kB5.0 kB8.0 kB32.0 kB62.0 kB92.0 kB302.4 kB602.9 kB
100025.9 kB49.6 kB79.1 kB315.6 kB611.0 kB906.6 kB3.0 MB5.9 MB
10000259.0 kB495.0 kB790.0 kB3.1 MB6.1 MB9.1 MB29.7 MB59.2 MB
1000002.6 MB4.9 MB7.9 MB31.5 MB61.0 MB90.5 MB297.0 MB592.0 MB
100000025.9 MB49.5 MB79.0 MB315.0 MB610.0 MB905.0 MB3.0 GB5.9 GB
Dataset size in DB1510501001505001000
1008.2 kB41.0 kB41.0 kB73.7 kB122.9 kB163.8 kB434.2 kB843.8 kB
100090.1 kB122.9 kB163.8 kB483.3 kB942.1 kB1.4 MB4.1 MB8.2 MB
10000630.8 kB958.5 kB1.4 MB4.6 MB9.1 MB13.7 MB41.0 MB82.0 MB
1000006.1 MB9.3 MB13.5 MB45.6 MB91.1 MB136.6 MBN/AN/A
100000060.3 MB93.1 MB134.4 MBN/AN/AN/AN/AN/A
Duration of POST /datasets API call1510501001505001000
1001s1s1s1s1s1s1s1s
10001s1s1s1s1s1s4s5s
100001s1s1s2s3s8s22s39s
1000002s3s4s24s30s58sN/AN/A
100000013s26s40sN/AN/AN/AN/AN/A
Duration of dataset upload1510501001505001000
1004s4s4s4s6s9s25s1m 14s
10004s5s5s5s8s10s33s1m 11s
100005s5s5s14s23s41s3m 59s6m 10s
1000009s14s18s1m 58s3m 8s5m 24sN/AN/A
10000001m 7s2m 24s3m 35sN/AN/AN/AN/AN/A
Duration of dataset delete1510501001505001000
10014s14s14s14s16s19s36s1m 24s
100015s15s15s15s17m20m41m1m 17s
1000015s15s15s22s31s45s3m 49s5m 45s
10000018s21s24s1m 45s2m 49s4m 36sN/AN/A
10000001m 5s2m 8s3m 5sN/AN/AN/AN/AN/A

TIM Forecasting performance measurements

We measure the performance of TIM Forecasting capabilities by running 3 scenarios (described below) with different dataset sizes.

  1. scenario: Register and execute forecast build-model jobs with the following registration payload:
{
"useCase": {
"id": "<use-case-id>"
}
}
  1. scenario: Register and execute forecast predict jobs with the model built in scenario 1 and with following registration payload:
{
"data": {
"outOfSampleRows": [
{
"from": "0001-01-01 00:00:00",
"to": "3000-01-01 00:00:00"
}
]
}
}
  1. scenario: Register and execute forecast predict jobs with the model built in scenario 1 and with empty JSON in the registration payload:
{}

Scenario 1

Max CPU usage in %1510501001505001000
1000.060.060.190.330.260.19N/AN/A
1000N/A0.260.470.370.150.14N/AN/A
10000N/A0.210.820.191.080.4N/AN/A
1000000.720.791.450.991.31N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Max RAM usage in MiB1510501001505001000
1003250.03271.03263.03269.03274.03304.0N/AN/A
1000N/A3325.03336.03382.03400.03424.0N/AN/A
10000N/A3506.03716.04523.04623.04600.0N/AN/A
1000002511.04961.05700.03562.04558.0N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Model building and prediction time in seconds1510501001505001000
1001.131.220.91.141.221.28N/AN/A
1000N/A1.091.111.551.932.35N/AN/A
10000N/A3.234.822.0726.9830.57N/AN/A
10000015.6228.5238.1952.6485.1N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Total execution time in seconds1510501001505001000
1007.537.499.6821.3141.9950.48312.4311.64
1000N/A8.269.5324.2436.6949.87312.33313.21
10000N/A9.914.1349.0264.6483.31309.52312.62
10000028.9343.7753.090.17151.38N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Forecasting result table size in DB1510501001505001000
10016.4 kB16.4 kB16.4 kB16.4 kB16.4 kB16.4 kBN/AN/A
1000N/A147.5 kB147.5 kB147.5 kB147.5 kB147.5 kBN/AN/A
10000N/A1.4 MB1.4 MB1.4 MB1.4 MB1.4 MBN/AN/A
10000014.1 MB14.1 MB14.1 MB14.1 MB14.1 MBN/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A

Scenario 2

Max CPU usage in %1510501001505001000
1000.120.050.680.330.110.2N/AN/A
1000N/A0.70.480.290.520.21N/AN/A
10000N/A1.10.030.250.290.5N/AN/A
1000000.210.420.350.661.1N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Max RAM usage in MiB1510501001505001000
1003270.03265.03260.03279.03277.03293.0N/AN/A
1000N/A3317.03344.03378.03412.03407.0N/AN/A
10000N/A2003.01869.02005.01962.02219.0N/AN/A
1000002161.02159.02255.03136.04198.0N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Model building and prediction time in seconds1510501001505001000
1001.770.180.170.30.290.29N/AN/A
1000N/A0.180.20.360.460.53N/AN/A
10000N/A0.860.955.216.2811.42N/AN/A
10000011.4314.0718.1345.6680.33N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Total execution time in seconds1510501001505001000
10010.177.497.4719.1238.6247.79N/AN/A
1000N/A7.29.5323.0135.5349.84N/AN/A
10000N/A9.629.6433.544.8674.72N/AN/A
10000023.1728.4134.9680.79150.37N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Forecasting result table size in DB1510501001505001000
10016.4 kB16.4 kB16.4 kB16.4 kB16.4 kB16.4 kBN/AN/A
1000N/A147.5 kB147.5 kB147.5 kB147.5 kB147.5 kBN/AN/A
10000N/A1.4 MB1.4 MB1.4 MB1.4 MB1.4 MBN/AN/A
10000014.1 MB14.1 MB14.1 MB14.1 MB14.1 MBN/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A

Scenario 3

Max CPU usage in %1510501001505001000
1000.110.040.630.410.110.49N/AN/A
1000N/A0.640.40.170.451.12N/AN/A
10000N/A0.520.740.20.440.03N/AN/A
1000000.020.180.090.120.12N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Max RAM usage in MiB1510501001505001000
1003241.03276.03255.03278.03281.03285.0N/AN/A
1000N/A3315.03332.03354.03349.03379.0N/AN/A
10000N/A3387.03369.03381.03444.03598.0N/AN/A
1000003570.03610.03601.03601.03594.0N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Model building and prediction time in seconds1510501001505001000
1000.150.730.140.140.150.16N/AN/A
1000N/A0.140.140.140.150.16N/AN/A
10000N/A0.150.190.150.150.23N/AN/A
1000000.140.180.160.150.2N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Total execution time in seconds1510501001505001000
1005.087.187.414.5128.2536.28N/AN/A
1000N/A5.037.319.226.2338.16N/AN/A
10000N/A4.977.321.6127.8941.58N/AN/A
1000005.177.217.5216.7531.38N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Forecasting result table size in DB1510501001505001000
1008.2 kB8.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/A
1000N/A8.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/A
10000N/A8.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/A
1000008.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A

TIM Detection performance measurements

We measure the performance of TIM Detection capabilities by running 3 scenarios (described below) with different dataset sizes.

  1. scenario: Register and execute detection build-model jobs with the following registration payload:
{
"useCase": {
"id": "<use-case-id>"
}
}
  1. scenario: Register and execute detection detect jobs with the model built in scenario 1 and with empty JSON in the registration payload:
{}
  1. scenario: Register and execute detection detect jobs with the model built in scenario 1 and with the following registration payload:
{
"data": {
"rows": {
"baseUnit": "Sample",
"value": 1
}
}
}

Scenario 1

Max CPU usage in %1510501001505001000
1000.780.210.070.260.021.83N/AN/A
10000.240.10.750.320.110.33N/AN/A
100001.351.141.421.60.890.55N/AN/A
1000003.334.155.217.633.39N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Max RAM usage in MiB1510501001505001000
1003601.03620.03610.03619.03640.03659.0N/AN/A
10003667.03688.03703.03733.03744.03754.0N/AN/A
100003847.03879.04012.04938.05081.04837.0N/AN/A
1000002401.04530.05741.03372.03058.0N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Model building and detection time in seconds1510501001505001000
1001.32.681.41.381.541.56N/AN/A
10002.322.182.222.412.682.46N/AN/A
1000013.516.8419.932.2333.335.31N/AN/A
10000079.88127.28102.11123.86104.35N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Total execution time in seconds1510501001505001000
1005.057.27.9717.1326.3238.32311.15311.3
10009.77.599.716.8231.4638.95312.07311.62
1000020.0524.7426.6747.8261.3575.59320.2334.63
10000098.02145.21117.01148.01153.97N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Detection result table size in DB1510501001505001000
10016.4 kB16.4 kB16.4 kB16.4 kB16.4 kB16.4 kBN/AN/A
1000114.7 kB114.7 kB114.7 kB114.7 kB114.7 kB114.7 kBN/AN/A
100001.1 MB1.1 MB1.1 MB1.1 MB1.1 MB1.1 MBN/AN/A
10000010.9 MB10.9 MB10.9 MB10.9 MB10.9 MBN/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A

Scenario 2

Max CPU usage in %1510501001505001000
1000.710.180.130.180.161.95N/AN/A
10000.190.080.630.250.140.34N/AN/A
100001.311.031.181.250.621.13N/AN/A
1000000.90.240.470.60.35N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Max RAM usage in MiB1510501001505001000
1003605.03615.03642.03611.03638.03656.0N/AN/A
10003675.03683.03757.03729.03735.03753.0N/AN/A
100003844.03816.03857.03872.03901.03910.0N/AN/A
1000003893.03955.03986.04588.05119.0N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Model building and detection time in seconds1510501001505001000
1000.740.860.830.760.790.76N/AN/A
10000.990.880.920.820.810.86N/AN/A
100001.131.061.282.271.941.87N/AN/A
1000004.694.585.5311.3113.88N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Total execution time in seconds1510501001505001000
1005.17.1210.0916.6728.8338.41N/AN/A
10009.178.017.2216.7835.7238.23N/AN/A
100007.567.397.3919.2229.5941.52N/AN/A
10000014.2214.317.1536.6351.86N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Detection result table size in DB1510501001505001000
10016.4 kB16.4 kB16.4 kB16.4 kB16.4 kB16.4 kBN/AN/A
1000114.7 kB114.7 kB114.7 kB114.7 kB114.7 kB114.7 kBN/AN/A
100001.1 MB1.1 MB1.1 MB1.1 MB1.1 MB1.1 MBN/AN/A
10000010.9 MB10.9 MB10.9 MB10.9 MB10.9 MBN/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A

Scenario 3

Max CPU usage in %1510501001505001000
1000.660.170.110.220.12.09N/AN/A
10000.170.160.530.180.080.22N/AN/A
100001.10.870.991.520.370.93N/AN/A
1000000.830.220.390.490.28N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Max RAM usage in MiB1510501001505001000
1003606.03612.03623.03622.03632.03647.0N/AN/A
10003657.03715.03698.03723.03701.03764.0N/AN/A
100003797.03798.03817.03838.03855.03819.0N/AN/A
1000003835.03850.03825.03857.04045.0N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Model building and detection time in seconds1510501001505001000
1000.660.270.30.290.290.27N/AN/A
10000.380.30.280.270.330.28N/AN/A
100000.330.280.270.260.30.26N/AN/A
1000000.290.250.250.290.27N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Total execution time in seconds1510501001505001000
1004.854.937.3814.429.0638.84N/AN/A
10004.967.267.5716.7634.6836.21N/AN/A
100007.845.037.4717.1526.737.73N/AN/A
1000007.777.237.216.8827.07N/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A
Detection result table size in DB1510501001505001000
1008.2 kB8.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/A
10008.2 kB8.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/A
100008.2 kB8.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/A
1000008.2 kB8.2 kB8.2 kB8.2 kB8.2 kBN/AN/AN/A
1000000N/AN/AN/AN/AN/AN/AN/AN/A

Performance measurements of end-to-end scenarios

Multiple datasets are uploaded to TIM from the perspective of a certain date in the past. To simulate a production scenario, the following data points are patched to the datasets, creating multiple versions. Multiple jobs are then created, combining each dataset version with the same configuration. These jobs are then executed in parallel to give the user production grade results.

DatasetsDataset VersionsConfigurationsModelsExecution time
16 (11 variables, 953 rows each)32 (11 variables, 43 rows each) for each dataset21024 in parallel2h 17m 57s