Automating Application Configuration

A lot of what I will discuss below is useful for anyone beginning any configuration automation project. I will detail out where I feel application configuration has specific challenges that differ from system configuration

Before You Begin

The first and most important thing anyone must do before embarking on an automation project is to create the standards for the environment. You must have a consistent naming scheme, packaging scheme, and directory layout. There has to be a simple way to derive "any file of this type shall be named like this and stored in a place like this". Have you ever written up a presentation to your boss after an outage where one slide is titled "Everything is Different" and go on to tell about how much room for human error exists because you have no standards?

But consistent naming is not enough.

Model Your Environment

What caught us off guard when implementing Chef was the amount of effort it took to model our environment. At first we didn't even know we needed to model our environment. We had a great set of standards for naming conventions, but the thing Chef offered us was now a hierarchy of attributes. We could define something once and have it used repeatedly. That always sounds great on paper, but when you get down to implementing it, it opens up cans of worms you never knew you had. We had to invent an appropriate hierarchy of configuration data. Even more, it needed to exist in a way compatible with a tool we knew little about.

I think modeling is more involved for application configuration automation than for system automation. (Please correct me in comments if you think otherwise). For system config automation you focus on the standard dev/qa/prod hierarchy of systems. For application config automation you turn that on its head and your "system" is now an instance of your application suite. The suite is the unit of configuration where you stamp out a fully running suite in dev or qa, or prod. Servers are secondary because the suite resides on a set of servers working as a unit and the suite needs to be aware of all the servers participating in that instance , but your primary focus is on configuring the application suite.

We built a cookbook called "derivations" that has a bunch of recipes that derive other sets of attributes. Yes, we could have done all the searches on the fly, but our thinking right now is that we like seeing our configuration persisted as node attributes. So our derivations cookbook makes a bunch of node attributes that describe various collections of servers needed to configure the application suite. Here, each node has attributes describing what all the other nodes in the suite are called and some things about what they do. Examples: "All app tier servers", "all memcached servers", etc... Various recipes can use those node attributes to write configuration files. Know that all this is point-in-time and if you add a new server to the environment, you need to run chef-client on all other nodes to have them discover their new neighbor.

How much is Just Enough

First you come up with an elaborate hierarchy that's perfect until you realize that as soon as an actual person six months from now needs to add an attribute they are going to spend half a day debating where it belongs. Or someone needs to figure out where something is defined and has to look through dozens of files and hours tracing complex paths of inheritance. DBA's have wrestled with this issue: just google "normalize too much" for lively discussion. As Adam Jacob said (paraphrasing) "If you can get to a 98% solution you will find that you likely can change something external so that the 2% edge-cases go away."

Add to this challenge that you barely know how this new tool works. How does the tool handle hierarchy and inheritance? How do I keep from going down a design path that isn't supported by the tool? At some point you have to work from both ends and check your thinking against the tool(s) you are going to use.

We honestly spent several months coming up with a model of our environment with what we believed was "Just Enough" hierarchy that could meet our design goals of reducing the sources of configuration data to as few locations as possible, while keeping it "human" where we believe someone 1 year later can figure out where things are. I encourage you to spend a lot of time on this. Getting your hierarchy close to right the first time will pay back dividends later. If you skimp here, you may be spending a lot of time reworking code to a new convention. (More below on the "Third Time" rule)

For our model we chose our nodes to have a run list that is all roles. Most roles have only attributes in them to give us our "just enough" hierarchy of configuration. Only one type of role has a run list with actual recipes in it. It looks like:

• datacenter - a few attributes defining the datacenter - useful for searches like "Find everything in datacenter X". Exclusively attributes, no run list.
• logicalsite - the term we came up with sort-of equivalent to environment. We group our servers inside a private DNS Top Level Domain to differentiate environments (dev, qa, load test, staging, production, etc..) so our logicalsite name is the DNS TLD of the environment. A datacenter may have many logicalsites in it. Almost exclusively attributes and the place we get the most bonus of high-level attributes.
• pod - our name for one complete, running instance of our entire product suite. A logicalsite may have multiple pods running in it. Exclusively attributes, no run list.
• tier - based on standard tiers like "app" and "data" but used to break apart the suite into deployed code. In our tier role we set the run list to satisfy all the dependencies to build a node of that tier. A node can have multiple tiers and we specifically designed them that one server can be all tiers or all tiers can be spread among individual servers (here is where your naming convention gets tested). A tier role is mostly run list and only a few attributes.
• constants - Not a role, but a Chef data bag. Some things you need to be universal constants across every possible environment. Here are things like IP Ports, mbean names and such. Now you know that every environment will have each service listening on the same port without variation.

The Product Manifest - The Secret Sauce

All of the above configuration is great, and in some ways is not really specific to application or system configuration. We have one additional set of attributes that comes into the Chef run from the outside and ties everything together. When our product suite is built one artifact that comes with it is what we call the "product manifest" (props to our awesome CM team who built this). It is a JSON file that describes every piece of code that needs to run and lots of metadata about it. In one blow I know everything about every piece of code that needs to run (build stamp so we get the right version, tier to deploy it on the right servers, dependencies like java or tomcat version). Now I have the ability to say "Deploy manifest version X to this environment" and the right code goes on the right server types with the right configuration data. There is no "dev manifest" and "prod manifest". It is one manifest used for all environments with no variation. Your variation comes only in your Chef roles named above, and that variation is as little as possible (URL names, memory settings and such).

Third Time Rule

The first time you are so excited simply that it works.
The second time it works again, but you may have some misgivings about imperfections.
The third time the flaws in the design become clear. You realize "We should have called this something else" or "We should have grouped this way" or ... By the third time, you figure out what is really important, and it's often not what you thought the first time.

Be patient and diligent. Refactor fast and often. Don't let bad code languish. Stamp out technical debt while it's fresh on your mind. Bad automation is REALLY BAD! The tool can implement something destructive really fast across all your server. (See below about testing). You have a limited window to get buy-in and the more you have to stammer "ummm, well, it really should be ..." the harder it is.

What Next?

First, determine where you are on the spectrum of standardization. You may not have one naming standard, but instead have a dozen evolutionary standards set by multiple admins, datacenter moves and company acquisitions. This is the hardest case because you will be implementing one new standard along with new automation software. The new standard has implications on monitoring and daily administration and can be very invasive. The "Third Time" rule will likely bite you frequently as well, because you don't know what you don't know. You may have a good file, package and directory structure, but lack a coherent model of hierarchy (this is where we were). Or you may have great standards and were just waiting to plug them into a tool.

Spend a lot of time working out your standards on paper. I would expect it to take over 3 months to come up with standards and a model. In this time you aren't writing a single line of code. Resist the urge to code. Learn the tool inasmuch as it will help you make standards, models, and hierarchy. You're asking questions like "What should I call this?" "How does this fit with feature X?"

Iterate

Yes, I just told you to spend a lot of time modeling your environment, but I assume some of that time is learning a new tool, or comparing multiple tools. Once you have your tool and your model I think the Agile philosophy is great for development. Start with a small problem and solve it. Every release should be production ready. When you have one thing done go to the next. Early on, "Production" is your test system, but before long you will have real value and want to start using your shiny new automation everywhere.

As Ops people we wanted to plan for every possible future scenario. Here is another use of "Just Enough". We decided to iterate and make just enough tuneable to match our current world. If we need more, we will add it later. We didn't want to spend all our time coding for situations where we don't have an immediate need. Be diligent about writing "Just Enough" code. It's easy to fall into the trap of "but we might need this feature..."

Try to minimize backward compatibility by fixing as many standards first, but you will surely find yourself needing to support some one-off situation by having some compatibility logic in code. The good thing about automation is that you can turn it all into code. "If OSVERSION=X then install pkgX, else install pkgY". Get a plan to fix the on-off in your environment and remove the code as soon as possible.

Follow Your Development Lifecycle

Caveat: We are not a continuous deployment shop. If you are continuous deployment, your lifecycle will probably be very different.

The other thing that caught us off guard was the strong need to follow our product's development lifecycle. It helped immensely that the group implementing Chef were Ops guys inside Dev and we sat really close to the software CM team. If you are in Ops, make friends with the team managing your source code. I can't say it strongly enough: If you are automating application configuration, the automation will follow to a high degree the lifecycle of the product. You need to know the release cycle and branching strategy.

With Chef, our Chef Environments are primarily Product Release Versions, not dev/qa/prod. We branch many of our cookbooks with the product, and we release some new Chef features with product releases. With a focus on application configuration we have found we use Chef in ways not common in the community.

Automation needs its own QA

Again it helped that we were inside Dev and managing the dev and QA systems, so we had a clear sense of building and testing the automation before releasing to production. I know how fallible I am when it comes to  making changes in production. I, personally, have been responsible for building tools for production only and never implementing them in Dev and QA. Picture in your head what it would be like if your lowliest dev server and your biggest production server were all configured by the same templates. Birds sing and unicorns dance. It is possible. Don't settle for less. You have one shot at this. (re-read above about the product manifest)

Practice, practice, practice. Test your automation on a bare system and on built, running systems.

Your "development" Chef server does not manage your development systems, it is a Chef server that manages your sandbox environments where you can write new code and break stuff. Your product development systems are still "production" to someone, so you need to develop and test your Chef code in a way that won't break any environment that the business depends on.

The Devops Problem - Application vs. System

What if you want to keep system management and application management separate events? Out of the box you think about Chef in a one node, one run list way where system and application configuration converge in one run. There is a way to separate the events where you can let Chef be used by the systems team for system work, and the application team use it for application work and have as little overlap as possible. It turns out to be quite useful (not perfect, but quite workable) to maintain that separation where you can have the ability to implement a "system" change event or a "product" change event without having to coordinate between them.

In Chef, one server can have multiple nodes defined on it. One node is used for "system" management and the other for "product" management. They have separate run lists so you can tell them exactly what needs to be done. Given this, the app support team can now update application configuration without impacting any system configuration. Some system events may need to have a "not_if" tied to the running application, because some system configuration can have negative impact on running code.

Here you find ways to tie into your orchestration tool. We write out application config files to a directory named "predeploy" and the orchestration tools are responsible for copying the files to the running location at the right time. Chef makes the application config files but does not implement the changes into the applications.

There is a gray area here. In the perfect world your product run list will have all dependencies needed for the product (including system dependencies), but then you find yourself in the one run list model. We settled on a tight-loose coupling between system and product. When a server is bootstrapped the system node bootstrap goes first then the product node bootstrap, then there is a running system. After that they are loosely coupled and can iterate at their own pace. There may be some cases where one may depend on another (e.g. big datacenter-wide structure change), but by taking the 98% lesson, try to make those go away instead of coding for them or ensure they are so seldom they can be a one-off event.

Lessons Learned

It was a huge victory the day someone said "server X isn't working the way it should" and I immediately, without effort, threw away the notion that it was configured wrong, and started looking elsewhere. Think of the mental energy you save when you can eliminate one whole set of variables with a wave of a hand.

The Toss Test. When we needed to upgrade the OS, we re-kicked the boxes and bootstrapped from scratch. Throw the old box away and build it fresh.

If you automate something you have to be 100%. If automation manages a file put a banner at the top saying "Managed by Chef. All manual changes will be overwritten!!!!"

As scary as it sounds you do want chef-client to run often (you define often). The less change between runs the better. Once all change is converged, subsequent runs will do nothing and are safe.

Automation Getting Started Guide Chapter Q

I don't know if I'm actually writing an entire Getting Started Guide to automation tools, but here is a chapter in the book that is getting written all through the community.

After you have installed your tool Configuration Management tool (e.g. Chef, Puppet, etc...) and have done your proof of concept and think it's really cool and is going to be really useful and have your first set of Kanban stickies on the board to write some code, read this.

But before you read this, read the chapter called Infrastructure as Code in "Test-Driven Infrastructure with Chef" by Stephen Nelson-Smith. Specifically the section titled "The Risks of Infrastructure as Code" It will justify you for being where you are, and also set the appropriate tone of sobriety to your automation endeavors,  and get you off to the right start.

One thing that hit me full force was the need to emphasize the CODE part of Infrastructure as Code. If you are a sysadmin like me, strong coding practice is not in your DNA. Reach out to your CM Team or whoever manages your source code repository and get some lessons in code management from them. Then reach out to a developer or two and get some lessons in (very) basic design patterns and a few "What Not To Do" tips.

Now, when you write some automation code (cookbook, manifest, etc...) that is destined for production, after you get the first draft written so that it compiles and runs without error your first job of refactoring is to answer the following questions. Repeat this exercise for every block of code you write (generically described below as a "function")

1. What will happen if this function runs on a brand-new system? (What are the prerequisites?)
2. What will happen if this function runs on an existing system but has never been run before? (different prerequisites?)
3. What will happen if this function's behavior is changed from the last run?
4. What will happen if this function is re-run on the same system with no other changes? (idempotence)
5. What will happen if the prior run failed? How will the function recover from a failed or partial run?

We found from experience that until all of the above questions have been considered your code is not ready for production, because you are at risk of unexpected behavior. The key to automation is predictable behavior. You will be amazed at how many ways automation can be unpredictable because you coded it poorly.

Not every question needs to be answered every time, and they are most important when new code is written.

There is some great discussion on this topic in the devops-toolchain Google Group, notably advice on using as many features of your tool to help make the above questions non-issues and to avoid code repetition. Also some interesting discussion on Greenfield systems (question 1) and Brownfield systems (question 2).

What are some of your getting started lessons learned? Care to add a chapter to the guide?

Done Means Deployed

John Willis of DTO Solutions was shaking out their "Devops Workshop" here in Atlanta. Trying to cover 2 days of material in 1 day was a herculean effort. I took a bunch of notes, but one sentence resonated strongly with me.

Done Means Deployed.

If you read Agile books and blogs, or probably if you work in a software shop you have read or heard discussions about "Done-Done". We have them fairly regularly. For us, Done-Done means QA accepted all available tests. I don't think regression is even a requirement. Done-Done means the code is ready to be deployed to production (not all code gets regression tests).

When I heard John say "Done means Deployed" it all clicked. If your Development Department believes "Done-Done" is "QA Accepted" then your developers have the classice "Throw it over the wall" mentality discussed by the dev2ops blog.

If your developers adopt a "Done means Deployed (to production)" mentality, then they are invested in the code all the way until it is in front of the customer. They can't disengage from it until it passes the ultimate QA (the end-user). We haven't achieved this here, but I would bet anyone who has has seen a dramatic increase in software qualtiy as a result.

In order to do this you need, at a minimum, frequent deployments, and ideally approaching continuous deployment. If your deployment cycles are on the order of months, it is impossible because too much happens between check-in and deployment. You can't wait 3 months to call something done, and you can't expect a software developer to stay fresh on that much code.

We're working on speeding up our deployment cycle. I'm keeping this idea in front of us as a stretch goal.

What makes a good Generalist?

I am glad that Devops is bringing generalists out of the closet and showing how valuable they are and how much companies need them. Also, I would bet money that the majority of successful "specialists" have a fairly broad set of  knowledge outside of their specific job function. So, your best generalist is a competent specialist, and your best specialist is a competent generalist.

So, "What makes a good generalist?"

Since we can only learn one thing at a time, I think most generalists started out as specialists (Solaris System Administrator, got CCNA certified, passed the MCSE test, Oracle certified, etc...). But, over time they gained knowledge outside of their special area in order to solve a problem. Over time that accumulation of knowledge became patterns of understanding they can use to synthesize information and make decisions. Finally they gain wisdom to have insight into things and project into the future.

I think one of the greatest contributions Devops makes is in shining a strong light on the fact that there is only one problem: The Business Problem. There is not a system problem and a software problem and a network problem and a security problem, there is only the business problem. The more everyone in the business knows about the rest of the business, the more they can understand how the parts relate, and ultimately make wise decisions to help the business succeed.

You can develop knowledge, understanding, and wisdom from others. One way to foster generalist growth is to rotate new employees through various departments when they are hired in order to give them a full picture of their peers. It's not quite cross-training because you don't need them to be competent in the job. It's more like cross-exposure so they can feel some of the pain and absorb some of the knowledge, understanding, and wisdom of the other departments. Every department has wisdom worth knowing. The level 1 CS person knows a lot about the software after the 10th customer calls up about a feature that doesn't work like it should. Or, the Ops person gains knowledge as they sift through all the "normal" errors in the log file to find the one, little "important" error buried in the noise. And the developer knows how to work with a team sharing a software repository; and knows what is the right version of the application and how a lifecycle is important for any code.

Our company rotates new developers into a support role which is a good start. I think it would be better if every Dev or Ops employee spent a week in some rotation like: Customer Support -> QA -> Security -> Development -> Operations to see the full context of their work and how it affects all departments and how all departments affect what they do.

When we have a broad knowledge of our business and feel some of the pain of our peers, I think we will be more successful at whatever specific role we have.

Two Approaches to Devops

I know I'm a little late in posting my follow up from the April Devops Meetup in Atlanta, but it was a great morning and I wanted to share some of the things discussed. Not everything in this post was explicitly discussed in the meetup, but some are thoughts I had related to the topics.

Artisan Server Crafting
First, I put out a challenge/request for John Christian to record his routine on "Artisan Server Crafting". He talks about how the traditional system administrator "crafts" each server, gives them personal names, and treats them like family. "Oh Look! Gandalf has been up for 365 days, let's throw a party!" The cloud makes the family too big that you can't give each of your children the attention they "deserve". I guess we just have to treat our machines like, well, machines. Automation, not art.

Devops, bottom-up
Next, John talked about how Devops got started inside his company. They started with one person from Dev and one from Ops working together on some automation to improve their lives. This is a common vector for Devops in companies. Start small as a skunkworks project, produce some results that show business value, then get management buy-in to continue the work and hopefully dedicate more time to the project, show more business value from your incremental success, then the business is hooked and you can't go back. I'll ask John to write a guest post to go into more detail.

The challenge of the bottom up process is that it is hard to get past the 1.0. You start out with a few energetic people that get things going, but scaling up can only happen with management's support. How do you cope with the original team moving on to something else? Are the new people going to be able to sustain progress on energy and enthusiasm alone? To move beyond 1.0 you have to show business value and show how the goals of Devops are aligned with the goals of the company. Also, you need to maintain the balance of Dev and Ops. A dominant personality can sway projects one way or another and alienate one side of the team.

Bottoms-up does work and has the potential to create a great deal of cohesion between Dev and Ops. Just be aware that at some point someone from the team is going to have to sell the story to management and get the business bought in. Devops is not complete unless the alignment goes all the way to the top of the management chain.

The Devops Team

A second way of introducing Devops occurred at my company. We unintentionally, through a series of reorgs, created a "Devops Department" without really knowing it. We created an Ops team inside Dev to take care of the non-production (Dev, QA, Load Test, CM, etc..) systems. Since this team reported up to the Dev executive and was chartered to take care of Dev's needs, there was a natural alignment of goals. This team and the Configuration Management Team got together and started automating deployments. After about a year building up Control Tier to deploy the code and succeeding in automating all deployments from Dev through Production the focus went to configuration. We have a suite of Java apps that are "overconfigured". Our current project is automating configurations of applications.

Automation and "Devops" got started with a standalone team inside Dev, but through a reorg that team merged with Production Operations. This was actually the best thing that could possibly happen because the biggest risk of a dedicated Devops Team in an organization with separate executives for Dev and Ops is that the team must naturally report up one silo and not have an affinity to the other silo. Also there is a more subtle factor that comes to play. The Devops team is not a part of any one Dev team, and not a part of any one Ops team, so Dev and Ops both think the team is an outsider. The whole point of Devops is lost. Dev doesn't have any ownership and Ops doesn't have any ownership. The team spends a lot of time trying to sell to the bottom and to the top.

Now, we were fortunate that the original Devops Team was populated with some of the senior people in the company that had deep relationships inside Dev and Ops so the selling wasn't too hard, but if you are considering a Devops team, the team will have to have strong support from both Dev and Ops executives with the ability to roam freely within both organizations. (This assumes Dev and Ops have different executives.)

So now our Devops 1.0 was a standalone team inside of Dev, but after the reorg the members are in Ops. But we have the benefit that the first project was to automate deployments which helps both teams, the second was to automate configuration which simplifies life in Ops and gives more ownership to Dev (that's just how things have been over here). Our third phase now fully branches with Ops embracing automation for system configuration and Dev thinking of the code in terms of operational impact and how it can be run and maintained easier and with automation.

But, you argue, Devops is about People first, then process, then tools. I think if we analyze the stories from companies we will find that the tools are the gateway drug to bring in Devops. You can stop at tools and just have some automation, or you can show the business value your tools bring and start a revolution that will ultimately encompass people and process. We have our tools now, we are in the long stage of battle to align the people and process.

I'll conclude with my wholehearted agreement that I believe that Devops will be most successful if Dev and Ops report to the same executive before the CEO. If you have two silos and they don't share goals, Devops will remain a bottoms-up battle.

Chef Is My Documentation

We have an ongoing project to automate the management of our custom software's configuration files. There is a hierarchy and some groupings of configuration data so we wanted to define configuration at the highest level possible and have its use be inherited at lower levels and with groups. We looked at all the "configuration management" software in the open source and decided that Chef had the right flexibility for our need. It wasn't a perfect match, but it was the closest thing available.

When we started the project to automate configuration I was in dev, but have since moved back into ops. Since one of Chef's main purposes is "system configuration management", the work we have done for configuration files is directly applicable to production operations' system management. So we're training and "selling" the tool to the Ops and Infrastructure teams as more than an application configuration tool. As I was putting the finishing touches on a presentation giving an overview of the effort and the rationale behind a new configuration management system, I came across a blog post by Jez Humble in his post in Agile Web Operations where he said:

"Effective configuration management – including automation of the build, deploy, test and release process, provisioning and maintenance of infrastructure, and data management – make the whole delivery process completely transparent. As any good auditor will verify, there is no better documentation than a fully automated system that is responsible for making all changes to your systems, and a version control repository that contains everything required to take you from bare metal to a working system."

That quote summed up the presentation I wrote on why we needed to automate our configuration file management. I didn't have as cogent a thought when I wrote my presentation, but am thankful for Jez to frame the problem so well. The words "there is no better documentation" jumped out at me and I used that to shift my thinking and reframe the rationale behind why we are automating configurations.

Chef is our documentation

Everyone makes an attempt at documenting their configuration. Between wiki pages and emails you can probably piece together 80% of your documentation. The problem comes when you make changes you have to keep your documentation up to date and in the heat of the battle documentation almost always gets left behind. Then you are tasked with building another system and you spend weeks of trial and error, copy and paste, search and replace to build a system. Your documentation never seems to be complete or up-to-date enough.

I've lived there in all my ops jobs and we're now fixing that problem. Our documentation is runnable configuration. The emphasis is on the runnable. If your documentation is a copy of your configuration (a wiki) you will never be able to keep it up to date. If your documentation is runnable, it will always be up to date. Now if that "good auditor" comes by and asks for documentation of how our systems and apps are configured we have complete, accurate documentation at all times. We don't have to scramble at the last minute to update a bunch of wiki pages. As soon as a new release of software goes into production the act of configuring the software for the deployment is also the act of updating the documentation.

The way we are doing it, the Chef database is the presentation of the documentation but not the source of the documentation. All configurations are saved in version controlled JSON files (either roles with attributes or databags) so all configuration is versioned and even if the Chef database gets destroyed we can re-create the database from source JSON. The files are named, scoped, versioned, and updated in a way that requires the fewest places to make changes while maintaing adequate clarity and re-use.

I'll follow up with a post on some of the technical details of what we are doing and the decisions we made along the way. We just discovered a few behaviors of Chef that somewhat complicate this plan, but nothing severe enough to be a showstopper.

Let me know what you are doing or what you think about the plan.

First Atlanta Devops Meetup Thursday 7PM

Register and details at http://www.eventbrite.com/event/764112481

Event Details

We'll be talking DevOps, Agile Infrascturcture, Agile Operations, or whatever you want to call it (Damon Edwards, from DTO Solutions, explains it best).
Come hang out and talk with folks who deal with the same issues of Systems Administration, Development, Deployment, and Operations that you do.
Talk withyour peers that are actively working on building tools, providing services and architecting frameworks to assist and build a DevOps community; and companies like Maxmedia, Turner, and t_sys, who run their organizations using Agile and DevOps concepts.
This is our 1st Meetup and it'll be casual - no presentations, no speakers, no slideshow, just food, beer and a bunch of people who want to talk about the challenges of making the world a better place for IT.

Where

Taco Mac Perimeter
1211 Ashford Crossing
Atlanta, GA 30338