background preloader

StatsD & Graphite

Facebook Twitter

Graphite. Data Types. The statsd server supports a number of different data types, and performs different aggregation on each of them.

Data Types

The three main types are counters, timers, and gauges. The statsd server collects and aggregates in 30 second intervals before flushing to Graphite. Graphite usually stores the most recent data in 1-minute averaged buckets, so when you’re looking at a graph, for each stat you are typically seing the average value over that minute. Counters¶ Counters are the most basic and default type. The statsd server collects counters under the stats prefix. Counters are managed with the incr and decr methods of StatsClient: from statsd import StatsClient statsd = StatsClient() statsd.incr('some.event') You can increment a counter by more than one by passing a second parameter: statsd.incr('some.other.event', 10) You can also use the rate parameter to produce sampled data. Rate is a float between 0 and 1: # Increment this counter 10% of the time.statsd.incr('some.third.event', rate=0.1) Timers¶ Note. URL API. The graphite webapp provides a /render endpoint for generating graphs (and retreiving raw data).

URL API

This endpoint accepts various arguments via query string parameters. These parameters are separated by an ampersand (&) and are supplied in the format: To verify that the api is running and able to generate images, open in a browser. The api should return a simple 330x250 image with the text “No Data”. Once the api is running and you’ve begun feeding data into carbon, use the parameters below to customize your graphs and pull out raw data. Graphing Metrics To begin graphing specific metrics, pass one or more target parameters and specify a time window for the graph via from / until. target.

Alerts. Accurate Counting. We have been using statsd and Graphite at HootSuite to get visibility into the frequency of events within our technology stack for over a year now.

Accurate Counting

And to great effect. In real time we are able to see the health of the HootSuite dashboard, supporting services, the myriad APIs our product uses and get to better understand things like our payments systems. Aggregation. I recently wanted to start recording various business level metrics for our Rails app.

Aggregation

We already use New Relic for performance monitoring and while you can inject custom metrics you can only surface them if you have the Pro version (unless they happen to surface naturally as one of the top n slowest/highest throughput/highest total time metrics). I’d been hearing a lot of good things about statsd and graphite so decided to try my hand at setting those up. Graphite is written in python, so this meant venturing out of my cosy little ruby world. Installation. Dashboards for Graphite. It’s no secret I like Graphite - it’s a great example of Open Source software which is just as good (actually, it’s even better) than similar closed-source, enterprise-grade (and much more expensive) software.

Dashboards for Graphite

While Graphite is fantastic, the front-end that comes with it leaves a little bit to be desired. You can use the built-in rendering and dashboard for viewing your system metrics, just like you can use Internet Explorer 6 to browse the Internet - it’ll work, but you’re gonna have a bad time. Thankfully, there are a lot of alternatives out there, but the Graphite Tools page is a little out of date (with abandoned/incomplete projects), so here’s a list or dashboards I have or would consider using… Client-side Dashboards. Rickshaw. Graphing Toolkit Rickshaw provides the elements you need to create interactive graphs: renderers, legends, hovers, range selectors, etc.

Rickshaw

You put the pieces together. See Demo → Built on d3.js It's all based on d3 underneath, so graphs are drawn with standard SVG and styled with CSS. Open Source Rickshaw is free and open source, available under the MIT license. Getting Started. TimeseriesWidget. Giraffe. Giraffe is using the Rickshaw d3 charting library.

Giraffe

You can therefore configure almost any visual aspect supported by rickshaw/d3 inside giraffe. This includes: color scheme, interpolation, renderer and more. Each graph can span over 1,2 or 3 columns using the colspan metric parameter. Top panel The top panel allows toggling between time ranges (not visible on the demo, but should work fine with graphite). Legend Clicking on the legend will show the legend under each chart. Summary In addition to the legend, each chart can have one summary value displayed next to its title.

Whisper. StatsD. Bash client. Etsy’s great post about tracking every release has some good tips about tracking releases with statsd and graphite (including some essential graphite config tweaks).

Bash client

I was wondering how to do this from within a shell script, and I had to dig through lots of StatsD code and examples to find this snippet. I forget where I eventually found it, and thought it’d make it easier to find. Deploy scripts are just one place where a concise and safe way to record a metric/event in important. From Etsy’s blog, using vertical lines to represent distinct events (code deployments) to give more context to the login trends: Sending a metric from the command line, with netcat or curl, is just one bit of ‘glue’ that is essential for pulling together a complete metrics solution. While short-and-sweet, this snippit contains two measures of protection. And the bash one-liner: PHP client. Java client. Measure Everything. Admin TCP interface. Carbon-cache cpu usage 100% when sent metric with too big name.

How to cleanup the graphite whisper's data? Statsd namespacing. 10 Things I Learned Deploying Graphite.