6 Anomaly Detection

This chapter describes anomaly detection, an unsupervised mining function for detecting rare cases in the data.

About Anomaly Detection

The goal of anomaly detection is to identify cases that are unusual within data that is seemingly homogeneous. Anomaly detection is an important tool for detecting fraud, network intrusion, and other rare events that may have great significance but are hard to find.

Anomaly detection can be used to solve problems like the following:

A law enforcement agency compiles data about illegal activities, but nothing about legitimate activities. How can suspicious activity be flagged?

The law enforcement data is all of one class. There are no counter-examples.

An insurance agency processes millions of insurance claims, knowing that a very small number are fraudulent. How can the fraudulent claims be identified?

The claims data contains very few counter-examples. They are outliers.

One-Class Classification

Anomaly detection is a form of classification. See "About Classification" on page 5-1 for an overview of the classification mining function.

Anomaly detection is implemented as one-class classification, because only one class is represented in the training data. An anomaly detection model predicts whether a data point is typical for a given distribution or not. An atypical data point can be either an outlier or an example of a previously unseen class.

Normally, a classification model must be trained on data that includes both examples and counter-examples for each class so that the model can learn to distinguish between them. For example, a model that predicts side effects of a medication should be trained on data that includes a wide range of responses to the medication.

A one-class classifier develops a profile that generally describes a typical case in the training data. Deviation from the profile is identified as an anomaly. One-class classifiers are sometimes referred to as positive security models, because they seek to identify "good" behaviors and assume that all other behaviors are bad.

Note:

Solving a one-class classification problem can be difficult. The accuracy of one-class classifiers cannot usually match the accuracy of standard classifiers built with meaningful counterexamples.

The goal of anomaly detection is to provide some useful information where no information was previously attainable. However, if there are enough of the "rare" cases so that stratified sampling could produce a training set with enough counterexamples for a standard classification model, then that would generally be a better solution.

Anomaly Detection for Single-Class Data

In single-class data, all the cases have the same classification. Counter-examples, instances of another class, may be hard to specify or expensive to collect. For instance, in text document classification, it may be easy to classify a document under a given topic. However, the universe of documents outside of this topic may be very large and diverse. Thus it may not be feasible to specify other types of documents as counter-examples.

Anomaly detection could be used to find unusual instances of a particular type of document.

Anomaly Detection for Finding Outliers

Outliers are cases that are unusual because they fall outside the distribution that is considered normal for the data. For example, census data might show a median household income of $70,000 and a mean household income of $80,000, but one or two households might have an income of $200,000. These cases would probably be identified as outliers.

The distance from the center of a normal distribution indicates how typical a given point is with respect to the distribution of the data. Each case can be ranked according to the probability that it is either typical or atypical.

The presence of outliers can have a deleterious effect on many forms of data mining. Anomaly detection can be used to identify outliers before mining the data.

Sample Anomaly Detection Problems

These examples show how anomaly detection might be used to find outliers in the training data or to score new, single-class data.

Figure 6-1 shows six columns and ten rows from the case table used to build the model. Note that no column is designated as a target, because the data is all of one class.

Figure 6-1 Sample Build Data for Anomaly Detection

Example: Find Outliers

Suppose you want to create a data set consisting of demographic data for typical customers. You might start by identifying the most atypical customers and removing them from the data.

To find the outliers, you can use the anomaly detection model to score the build data. Figure 6-2 shows that customer 101,505 is anomalous and should be removed.

Note:

A prediction of 0 is considered anomalous. A prediction of 1 is considered typical.

Figure 6-2 Outliers in the Build Data

Description of "Figure 6-2 Outliers in the Build Data"

Note:

Oracle Data Miner displays the generalized case ID in the DMR$CASE_ID column of the apply output table. A "1" is appended to the column name of each predictor that you choose to include in the output. The predictions (1 for typical or 0 for anomalous in Figure 6-2) are displayed in the PREDICTION column. The probability of each prediction is displayed in the PROBABILITY column.

Example: Score New Data

Suppose that you have a new customer, and you want to evaluate how closely he resembles a typical customer in your current customer database.

You can use the anomaly detection model to score the new customer data. The new customer is a 40-year-old male executive who has a bachelors degree and uses an affinity card. This example uses the SQL function PREDICTION_PROBABILITY to apply the model svmo_sh_clas_sample (the sample anomaly detection model provided with the Oracle Data Mining sample programs). The function returns .05, indicating a 5% probability that the new customer is typical. This means that 95%, of your customers are more like your average customer than he is. The new customer is somewhat of an anomaly.

COLUMN prob_typical FORMAT 9.99
SELECT PREDICTION_PROBABILITY (svmo_sh_clas_sample, 1 USING
    'M' AS cust_gender,
    'Bach.' AS education,
    'Exec.' AS occupation,
    40 AS age,
    '1' AS affinity_card) prob_typical
    FROM DUAL;
PROB_TYPICAL------------         .05

Algorithm for Anomaly Detection

Oracle Data Mining supports One-Class Support Vector Machine (SVM) for anomaly detection. When used for anomaly detection, SVM classification does not use a target.