Finding Data Anomalies You Didn't Know to Look For
Anomaly detection is the detective work of machine learning: finding the unusual, catching the fraud, discovering strange activity in large and complex datasets. But, unlike Sherlock Holmes, you may not know what the puzzle is, much less what “suspects” you’re looking for. This O’Reilly report uses practical examples to explain how the underlying concepts of anomaly detection work.
From banking security to natural sciences, medicine, and marketing, anomaly detection has many useful applications in this age of big data. And the search for anomalies will intensify once the Internet of Things spawns even more new types of data. The concepts described in this report will help you tackle anomaly detection in your own project.
- Use probabilistic models to predict what’s normal and contrast that to what you observe
- Set an adaptive threshold to determine which data falls outside of the normal range, using the t-digest algorithm
- Establish normal fluctuations in complex systems and signals (such as an EKG) with a more adaptive probablistic model
- Use historical data to discover anomalies in sporadic event streams, such as web traffic
- Learn how to use deviations in expected behavior to trigger fraud alerts
|
1 Looking Toward the Future |
|
|
1 | (6) |
|
2 The Shape of Anomaly Detection |
|
|
7 | (8) |
|
|
|
8 | (3) |
|
If you enjoy math, read this description of a probabilistic model of "normal"... |
|
|
10 | (1) |
|
|
|
11 | (1) |
|
|
|
12 | (2) |
|
Once again, if you like math, this description of anomalies is for you... |
|
|
13 | (1) |
|
Take-Home Lesson: Key Steps in Anomaly Detection |
|
|
14 | (1) |
|
A Simple Approach: Threshold Models |
|
|
14 | (1) |
|
3 Using t-Digest for Threshold Automation |
|
|
15 | (8) |
|
The Philosophy Behind Setting the Threshold |
|
|
17 | (2) |
|
Using t-Digest for Accurate Calculation of Extreme Quantiles |
|
|
19 | (1) |
|
Issues with Simple Thresholds |
|
|
20 | (3) |
|
4 More Complex, Adaptive Models |
|
|
23 | (12) |
|
|
|
25 | (3) |
|
Matches with the Windowed Reconstruction: Normal Function |
|
|
28 | (2) |
|
Mismatches with the Windowed Reconstruction: Anomalous Function |
|
|
30 | (2) |
|
A Powerful But Simple Technique |
|
|
32 | (2) |
|
Looking Toward Modeling More Problematic Inputs |
|
|
34 | (1) |
|
5 Anomalies in Sporadic Events |
|
|
35 | (12) |
|
|
|
36 | (2) |
|
Arrival Times Are the Key |
|
|
38 | (3) |
|
|
|
40 | (1) |
|
Event Rate in a Worked Example: Website Traffic Prediction |
|
|
41 | (2) |
|
Extreme Seasonality Effects |
|
|
43 | (4) |
|
|
|
47 | (6) |
|
|
|
47 | (2) |
|
The No-Phishing-Allowed Anomaly Detector |
|
|
49 | (1) |
|
|
|
50 | (1) |
|
|
|
51 | (2) |
|
7 Anomaly Detection for the Future |
|
|
53 | (4) |
| A Additional Resources |
|
57 | |
Ted Dunning is Chief Applications Architect at MapR Technologies and committer and PMC member of the Apache Mahout, Apache ZooKeeper, and Apache Drill projects and mentor for these Apache projects: Spark, Storm, Stratosphere, and Datafu. He contributed to Mahout clustering, classification, and matrix decomposition algorithms and helped expand the new version of Mahout Math library. Ted was the chief architect behind the MusicMatch (now Yahoo Music) and Veoh recommendation systems, built fraud-detection systems for ID Analytics (LifeLock), and has issued 24 patents to date. Ted has a PhD in computing science from University of Sheffield. When he's not doing data science, he plays guitar and mandolin. Ellen Friedman is a consultant and commentator, currently writing mainly about big data topics. She is a committer for the Apache Mahout project and a contributor to the Apache Drill project. With a PhD in Biochemistry, she has years of experience as a research scientist and has written about a variety of technical topics including molecular biology, nontraditional inheritance, and oceanography. Ellen is also co-author of a book of magic-themed cartoons, A Rabbit Under the Hat. Ellen is on Twitter at @Ellen_Friedman.
