Ganglia Monitoring System. Overview — Graphite 0.9.10 documentation. Etsy/statsd. Mining Time-series with Trillions of Points: Dynamic Time Warping at scale. Take a similarity measure that's already well-known to researchers who work with time-series, and devise an algorithm to compute it efficiently at scale. Suddenly intractable problems become tractable, and Big Data mining applications that use the metric are within reach. The classification, clustering, and searching through time series have important applications in many domains. In medicine EEG and ECG readings translate to time-series data collections with billions (even trillions) of points. In fact many research hospitals have trillions of points of EEG data. Other domains where large time series data collections are routine include gesture recognition & user interface design, astronomy, robotics, geology, wildlife monitoring, security, and biometrics.
The problem is that existing algorithms don't scale1 to sequences with hundreds of billions or trillions of points. What is Dynamic Time Warping? SQRT[ Σ (xi - yi)2 ] The previous result is representative of what the UCR team found. Etsy/oculus. Etsy/skyline. Bring the Noise: Continuously Deploying Under a Hailstorm of Metrics.
Introducing Kale. Posted by Abe Stanway | Filed under data, monitoring, operations In the world of Ops, monitoring is a tough problem. It gets harder when you have lots and lots of critical moving parts, each requiring constant monitoring. At Etsy, we’ve got a bunch of tools that we use to help us monitor our systems. You might be familiar with some of them: Nagios, StatsD, Graphite, and Ganglia. Today, we’d like to introduce you to a new tool that we’ve been working on for the past few months. This tool is designed to solve the problem of metrics overload. Of course, if a graph isn’t being watched, it might misbehave and no one would know about it.
We’d like to introduce you to the Kale stack, which is our attempt to fix both of these problems. Skyline Skyline is an anomaly detection system. You can hover over all the metric names and view the graphs directly. Once you’ve found a metric that looks suspect, you can click through to Oculus and analyze it for correlations with other metrics! Oculus.