Anomaly Detection

I want to go through the Wikipedia series on Machine Learning and Data mining. Data mining is the process of extracting and discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems.

Date Created:
0 72

References


Notes

In data analysis, anomaly detection (also referred to as outlier detection and sometimes as novelty detection) is generally understood to be the identification of rare items, events or observations which deviate significantly from the majority of the data and do not conform to a well defined notion of normal behavior. Such examples may arouse suspicions of being generated by a different mechanism, or appear inconsistent with the remainder of that set of data.

Three board categories of anomaly detection techniques exist:

  1. Supervised anomaly detection technique require a data set that has been labeled as normal and abnormal and involves training a classifier. However, this approach is rarely used in anomaly detection due to the general unavailability of labelled data and inherent nature of the classes
  2. Semi-supervised anomaly detection techniques assume that some portion of the data is labelled. The techniques construct a model representing normal behavior from a given normal training data set, and then test the likelihood of a test instance being generated by the model
  3. Unsupervised anomaly detection techniques assume the data is unlabeled and are by far the most common used due to their wider and relevant application.

There is no agreed upon definition for what qualifies as an anomaly, but here are some example definitions:

  • An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism
  • Anomalies are instances of collections of data that occur very rarely in the data set and whose features differ significantly form most of the data
  • An outlier is an observation which appears to be inconsistent with the remainder of that set of data

Anomaly detection is applicable in a very large number and variety of domains, and is an important subarea of unsupervised machine learning.

Comments

You can read more about how comments are sorted in this blog post.

User Comments