We have an ongoing project to automate the management of our custom software's configuration files. There is a hierarchy and some groupings of configuration data so we wanted to define configuration at the highest level possible and have its use be inherited at lower levels and with groups. We looked at all the "configuration management" software in the open source and decided that Chef had the right flexibility for our need. It wasn't a perfect match, but it was the closest thing available.
When we started the project to automate configuration I was in dev, but have since moved back into ops. Since one of Chef's main purposes is "system configuration management", the work we have done for configuration files is directly applicable to production operations' system management. So we're training and "selling" the tool to the Ops and Infrastructure teams as more than an application configuration tool. As I was putting the finishing touches on a presentation giving an overview of the effort and the rationale behind a new configuration management system, I came across a blog post by Jez Humble in his post in Agile Web Operations where he said:
"Effective configuration management – including automation of the build, deploy, test and release process, provisioning and maintenance of infrastructure, and data management – make the whole delivery process completely transparent. As any good auditor will verify, there is no better documentation than a fully automated system that is responsible for making all changes to your systems, and a version control repository that contains everything required to take you from bare metal to a working system."
That quote summed up the presentation I wrote on why we needed to automate our configuration file management. I didn't have as cogent a thought when I wrote my presentation, but am thankful for Jez to frame the problem so well. The words "there is no better documentation" jumped out at me and I used that to shift my thinking and reframe the rationale behind why we are automating configurations.
Chef is our documentation
Everyone makes an attempt at documenting their configuration. Between wiki pages and emails you can probably piece together 80% of your documentation. The problem comes when you make changes you have to keep your documentation up to date and in the heat of the battle documentation almost always gets left behind. Then you are tasked with building another system and you spend weeks of trial and error, copy and paste, search and replace to build a system. Your documentation never seems to be complete or up-to-date enough.
I've lived there in all my ops jobs and we're now fixing that problem. Our documentation is runnable configuration. The emphasis is on the runnable. If your documentation is a copy of your configuration (a wiki) you will never be able to keep it up to date. If your documentation is runnable, it will always be up to date. Now if that "good auditor" comes by and asks for documentation of how our systems and apps are configured we have complete, accurate documentation at all times. We don't have to scramble at the last minute to update a bunch of wiki pages. As soon as a new release of software goes into production the act of configuring the software for the deployment is also the act of updating the documentation.
The way we are doing it, the Chef database is the presentation of the documentation but not the source of the documentation. All configurations are saved in version controlled JSON files (either roles with attributes or databags) so all configuration is versioned and even if the Chef database gets destroyed we can re-create the database from source JSON. The files are named, scoped, versioned, and updated in a way that requires the fewest places to make changes while maintaing adequate clarity and re-use.
I'll follow up with a post on some of the technical details of what we are doing and the decisions we made along the way. We just discovered a few behaviors of Chef that somewhat complicate this plan, but nothing severe enough to be a showstopper.
Let me know what you are doing or what you think about the plan.
