An important setting in the anomaly detection is the sensitivity to anomalies. It goes hand in hand with the anomaly indicator which conveys the information about the extent to which the given observations are anomalous. Sensitivity comes into play to define the decision boundary, allowing the model to distinguish between anomalies and normal points. The real world examples show that this decision is often ambiguous and depends on the use case, user, business risk profile etc.
Sensitivity is defined as the percentage of the (training) data that is expected to be anomalous. It indicates how often an anomaly alert is triggered in the training data. For example, a sensitivity of 3% considers the 3% most anomalous observations in the training data to be anomalies; the user would have been alerted in these 3% of the training data.
It is recommended to choose the training data with as few anomalies as possible and with sufficient length allowing TIM to find the best possible model. In case you have labelled data, this can be accomplished by omitting anomalous observations from the training data and setting the sensitivity to 0%. In this case the model is not affected by anomalous points at all. Of course, usually you do not know which points are anomalous, however, in general, the percentage of anomalies should not exceed 5%. If it exceeds, supervised learning could be a more appropriate approach than anomaly detection.
For better understanding the link between data, sensitivity, anomaly indicator and threshold, consider the following example. Let's assume we have a univariate problem - a KPI/target variable without any explanatory variables. The training period with the KPI is depicted on the image below.
Running the anomaly detection model building with the sensitivity parameter set to 0% results in an anomaly indicator which is below the threshold on the entire training period.
Running the model building with 5% sensitivity causes the anomaly indicator to exceed the threshold exactly on 5% of training observations. Note that by changing the sensitivity the threshold remains the same (equal to 1) but the anomaly indicator is rescaled.
Remember that by setting the sensitivity parameter you are basically estimating what percentage of anomalies you expect on a training period. It is also true that the higher the sensitivity the more anomalies on the out of sample period (or when using the model in production). However, it does not mean that the percentage of anomalies on the out of sample period will be similar to the percentage of anomalies on the in-sample period.
Automatic sensitivity estimation¶
By default, the sensitivity parameters are estimated automatically.
Two major reasons for automatizing the sensitivities are:
- increase large scale potential
- in most of the cases, the number of anomalies is unknown
As we already know, sensitivity is a very important parameter in the anomaly detection process, that has to be adjusted correctly as it affects how many anomalies are found. Finding the right sensitivity can be tedious work in an unlabeled data, which is often the case. Obviously, if there would not be the possibility to find this parameter automatically, it would not be very realistic to serve many time series in an automatic way.
Of course, in case you would like to be more conservative (do not want to be alerted so often, only in most anomalous cases; false positives cost you significantly more than false negatives), or in contrary, your domain requires to be alerted more often (false negatives cost you significantly more than false positives) you can manually set the sensitivity to lower/ higher percentage.