Anomaly Detection Layer¶
TIM's anomaly detection layer is built on top of the proven technology for automatic model generation for time-series forecasting. By selecting the relevant features and creating the most appropriate model, TIM automates the anomaly detection process.
This process can be characterized as follows. First, data is gathered. This data is typically unlabelled and contains mostly normal observations, thus anomalies are rare. Then, the notion of normality is extracted from this data. In the final stage TIM looks into the distribution of the data from various perspectives.
This process results in an anomaly indicator, whose value contains information about each data point, indicating to what extent the data point goes beyond what could be considered ‘normal’.
These features improve anomalous behavior (AB) learner by revealing hidden patterns in the output of normal behavior (NB) learner from different perspectives. Each of these features is suited for different situation and looks on the output either globally or locally.
Let's assume that the output from NB learner looks like the one in the image below.
AB learner with Residual feature applied to this output tries to detect outliers in the NB from a global perspective. This is depicted in the image below. The AB learner with Residual Change feature treats as anomaly every outlier that exhibits "extreme" behavior comparing to what it can see on in-sample period.
AB learner with Residual Change feature seeks for local outliers in the NB output. It does so by detecting the most extreme changes in the NB, i.e. relatively stable periods followed by extreme residual. If the change is too "extreme" compared to what the AB learner can see on in-sample period, it is considered as anomaly.
Fluctuation feature describes the fluctuation of output from NB learner from a global perspective. Different values of fluctuation in the output reflects into different values of the feature. Thus, using the Fluctuation feature, AB learner can detect different fluctuations than those which were present during the in-sample period (see image below).
Fluctuation Change feature focuses on fluctuation of NB output similarly as Fluctuation feature. The difference is, however, that Fluctuation Change seeks only for the change in fluctuation. The image below shows in-sample period without change and then anomalies on out of sample period caused by changes in fluctuation.
Let's assume for this and the following subsection that the output from NB learner looks like the one in the image below.
Imbalance feature helps to detect anomalies accompanied by deviation of NB output from zero. Significance and direction of the deviation reflect into magnitude and direction of the feature respectively. The image below shows an example of AB learner detecting two periods with high imbalance, each one in the opposite direction.
AB learner with Imbalance Change feature is suited for detecting anomalies that occur when the change of imbalance in NB output is present. The image below shows in-sample period without change in imbalance and four out of sample periods where the change occurred.
Hour, Day of Week, Month¶
These three features help AB learner to better utilize the context of time. It can help for example when the underlying anomaly detection problem has different working regimes which are then reflected also in the output from NB learner.
- Hour feature - separates hours of day, values ranging from 1 to 24
- Day of Week feature - separates days of week, values ranging from 1 to 7
- Month feature - separates months of year, values ranging from 1 to 12