Tuesday, December 17, 2013

Logstash Metrics Filter and Graphite Output

Not many people have published more advanced metrics filter configurations. After spending a day with the examples and source code I have a more advanced configuration to share. NOTE: I saw weird behavior with logstash 1.2.2 and I'm not sure if it was my in-progress configuration at the time, but after upgrading to 1.3.1 everything worked as expected.

The problem: We are trying to get metrics on our API usage by user. We were already logging the operation and the user to disk and picking it up in Logstash. Now we want metrics on how frequently each user makes each call.

After quite a bit of googling, I couldn't find an example where the metric name has more than one field name in it. The last message in this thread had some pieces of the puzzle as we needed to translate some special characters to be more graphite-friendly. But I got confused as to what was contained in the metric event. The metrics filter creates a new event, but until I threw the event to a file output and saw it I didn't realize the new metric event didn't have any knowledge of any field of the message that generated it (Actually, the "meter" option can use any name from the original event, but no other option can. I tried to use add_field inside the metrics filter and it didn't work). So instead of mutating the metric event, I have to mutate the original event. Also, you can't gsub a newly added field in the same mutate block, so I had to break the gsub to a second mutate.


    if [type] == "apicalls" {
    mutate {
      add_field => [ "modhost", "%{host}" ]
      add_field => [ "modorgname", "%{org_name}" ]
    }
    mutate {
      gsub => [ "modhost", "\.", "_", "modorgname", "[\.\,\ ]", "_" ]
    }
    metrics {
      meter => [ "apioperations.%{pod}.%{modhost}.%{operation}.byOrg.%{modorgname}" ]
      add_tag => [ "apiopmetric" ]
    }

  }

Now when I send those metrics events to a file output I see my resulting event is

{
"@timestamp":"2013-12-17T19:08:42.968Z",
"@version":"1",
"message":"server.name",
"apioperations.qa1.host_domain.Login.byOrg.my_org_name.count":11,
"apioperations.qa1.host_domain.Login.byOrg.my_org_name.rate_1m":0.0,
"apioperations.qa1.host_domain.Login.byOrg.my_org_name.rate_5m":0.0,
"apioperations.qa1.host_domain.Login.byOrg.v.rate_15m":0.0,
"apioperations.qa1.host_domain.GetModifiedRecipients.byOrg.my_org_name.count":2,
"apioperations.qa1.host_domain.GetModifiedRecipients.byOrg.my_org_name.rate_1m":0.0,
"apioperations.qa1.host_domain.GetModifiedRecipients.byOrg.my_org_name.rate_5m":0.0,
"apioperations.qa1.host_domain.GetModifiedRecipients.byOrg.my_org_name.rate_15m":0.0,
"tags":["apiopmetric"]
}

Next is to figure out how to get those to graphite. Since there is an unknown number of operations and users, I had to go to the source code to really figure out how all the options to the graphite output work. It turns out you can't use the "metrics" option as that would need to enumerate each name. The magical "fields_are_metrics" option sends all the fields in the event to graphite. All you need to do is use "include_metrics" or "exclude_metrics" to  get just what you want to graphite. Our graphite output looks like this (the file output was for debug purposes only and is turned off now that it works):

  if "apiopmetric" in [tags] {
    graphite {
      host => [ "10.1.1.1" ]
      include_metrics => [ "apioperations.*" ]
      fields_are_metrics => true
    }
    file {
      path => "/var/log/logstash/apidebug.log"
    }
  }

And Shazam! All your metrics start flowing into Graphite!

Tuesday, April 9, 2013

Marching Off the Map

The title is not a new one, but it is a great image of what I feel like the Devops community is doing. The Map is the way businesses have been run for the last 100 years and which the IT industry adopted in the 80's and was mostly adopted even during the dotcom days. In the 80's and 90's Enterprise is what everone wanted. In the 00's as the large web operations started growing "Enterprise" became a dirty word among the cool kids. Now, Enterprise does describe some very large companies, but many of the Enterprise ways are in many smaller companies (generally older ones). Damon Edwards used the term "Classic Organization" and I think that is a much more inclusive and less emotionally charged term than Enterprise, so I will use that term to mean "Orgainzations operating with the culture and processes akin to Enterprise". Classic Organizations are the epitome of the "before" picture in the Devops transformation. Devops (building on Lean and Agile and others) is marching off the map of business models and, I think, incoporating much of the best of the past into new models to lead us into the future.

Recently, I realized my personal life has been paralleling my professional life in many ways. I'm seeing the core principals of Devops echoed throughout my life. Many people are discovering that things happening in the tech industry will work in running a household or other community too. Not sure if this is behind a paywall but the Wall Street Journal ran a story by Bill Gates in January where he describes what sounds a lot like Lean thinking as a solution to fixing global problems. http://online.wsj.com/article/SB10001424127887323539804578261780648285770.html and some interesting replies that generally uphold Lean principles and illustrate the challenges of applying Lean in a "Classic" culture http://online.wsj.com/article/SB10001424127887324156204578275993802414124.html. There are also many articles around the internet on running a household on Agile principles.

Then I heard a few podcasts from Growing Leaders describing the need to look for new ways of communicating with and educating young people today.http://growingleaders.com/blog/podcast-7-an-interview-with-dan-pink/, http://growingleaders.com/blog/podcast-8-the-benefits-of-a-gap-year/. One theme they share is that in school you are measured on (roughly) 75% IQ and 25% EQ, but in the workforce the proportions are reversed. This tweet illustrates that shift.
The conclusion is that school is not teaching people how to be productive workers. For Devops and Lean to work there needs to be more focus on EQ development in people. It is said that your IQ is relatively fixed from birth, but that EQ can be trained and developed. When you have your technical people thinking more with their "Right Brain" (Big Picture, Context, Synthesis) you should see the culture fall into place much easier. The "Left Brain" logical, analytical stuff is so easy, probably too easy, that we use it as a crutch to not work on peope, culture, empathy, systems thinking, and such. Just read some of the stories about the Etsy Hacker Grant program and its effects. "Right Brain" thinking can be developed and learned.

This post is rambling a bit through multiple topics but my main point is that I feel Devops is on the right path because its driving principles are echoed throughout life and so many cultures. I put a lot of weight on "uncommon, common sense" where you re-discover eternal truths that are built into human nature (respect, empathy, purpose, quality) and build on top of those. The name "Devops" doesn't really matter and will pass away, but the principles behind it should always be the foundation of all we do. I'm marching off the map at work and marching my kids off the map in their education at home. It's a little scary, but exciting to be doing something new and discovering a vibrant community around you to let you know you are not alone.