One of the big themes that emerged during #ChefConf was that we should be testing our infrastructure code. Software engineers have been practicing test-driven development, behavior-driven development, continuous integration, and many other testing-related practices for a long time. It’s becoming more important for the infrastructure engineers to learn from and apply these practices to our day-to-day workflow. When it comes to testing Chef-driven infrastructure automation, there are a number of tools and practices that are starting to emerge. In this article I’ll look at a “minimum viable testing” (MVT) approach to this problem using Foodcritic and Travis CI. Follow the steps in this article to get your public cookbooks tested after every git push.

Testing with Chef

The idea of building automated tests for your infrastructure code has been getting a lot of traction lately. When it comes to Chef, many tools are starting to emerge.

The first tool in this area to get any significant traction, that I know of, was cucumber-chef. I first learned of this tool when I saw a pre-release copy of Test-Driven Infrastructure with Chef at the O’Reilly booth at Velocity Conf 2011. Stephen Nelson-Smith, the book’s author and framework’s lead developer, proposes an outside-in approach to testing where your tests can also act as monitors that look after the health of your infrastructure. I like the idea of this approach and feel it makes a lot of sense in a greenfield environment. One benefit of this approach is that it blurs the line between testing and monitoring. You can easily hook-up your monitoring system to your cucumber tests.

ChefSpec is another tool for testing your Chef code. It is a gem that makes it easy to write RSpec examples for Chef cookbooks. This style of testing allows you to execute your tests without needing to converge the node that your tests are running on. In other words, you can execute your tests without needing to provision a server. One huge appeal to this style of testing is that the feedback loop is very small. You’ll get feedback about your cookbook changes within seconds or a very few minutes of saving your changes.

Minitest Chef Handler is yet another tool for testing with Chef. This runs a suite of minitest tests as a report handler in your Chef-managed nodes. As you may know, report handlers are run at the end of each chef run, or convergence.

Testing at ChefConf

At the inaugural #ChefConf there were many sessions that included information about many companies’ approach to testing. Here’s a quick list of some of the sessions:

• Food Fight Show Episode #10 - TESTALLTHETHINGS – This wasn’t actually part of #ChefConf but is ‘required listening’ for anyone interested in learning more about this space.

• #ChefConf Pre-event Hackday: TEST ALL THE THINGS!!!

• NTP Cookbook with tests - tests were added to this cookbook as part of the hackday event.

• Test-driven Development for Chef Practitioners (video)

• Test Driven Development Roundtable (video)

When I first started working with Chef, there were a couple of areas that I knew were going to be really awesome and helpful but I wasn’t sure how to get started with them. In this presentation, I’ll provide a quick introduction to five things you’ve always wanted to know about Chef but were afraid to ask.

I gave this presentation at #ChefConf 2012.

Level-up your Chef skills by learning about these areas of Chef:

• Attribute Precedence - Role, environment, cookbook, data bag? Which attribute value will be used in my chef run?
• Encrypted Databags - Chef 0.10 brought us encrypted databags. We’ll look at how to create and use databags and how to keep them up-to-date in your repository.
• LWRP - What is a LWRP? How and why do you create one? We’ll look at a couple of sample LWRPs and learn how to build a simple one.
• Error Handlers - Demystify exception and report handlers by writing a simple one and seeing examples of how they work in the wild.
• Capistrano and Chef - Take a quick look at why and how to integrate Chef search into your Capistrano configuration to make deploying your Rails apps even easier.

One thing I didn’t mention in the presentation was how to use the data from the encrypted data bag. I’ve updated the slides to include this info but it doesn’t appear in the video. In any case, here’s a quick demo of how you might use it:

1
2
3
4
5
6
7
8
9
10
11

creds = Chef::EncryptedDataBagItem.load("db", "creds")
env_db_creds = db_creds[node["rails_env"]]

template "#{app_dir}/shared/config/database.yml" do
  source "database.yml.erb"
  variables(
    :rails_env => node["rails_env"],
    :username => env_db_creds["username"],
    :password => env_db_creds["password"]
  )
end

Video

Slides

_{Reposted from the CustomInk Technology blog.}

You’ve heard of Chef, Puppet, and other frameworks that can help you build out your infrastructure. You’ve been meaning to play around with one or more of them for some time now. Now’s your chance; Start cooking up on your own servers!

In this presentation, I provide an introduction to Chef with a focus on what you’ll need to know to get a Rails application up and running.

Topics include: * Introduction to Chef * Nodes, roles, environments, and other terminology * Introduction to cookbooks * Provisioning an environment for a Rails application * Deploying with Capistrano

You won’t be ready to compete in Iron Chef, but you will be ready to serve up your own Rails environment in no time.

I gave slightly different versions of this presentation at RubyNation 2012 and #ChefConf 2012.

I’d really appreciate any comments, questions, or feedback in the comments section below.

Video from ChefConf

Slides

_{Reposted from the CustomInk Technology blog.}

At RailsConf 2012, I gave a presentation on how the CustomInk web operations team enables developer productivity.

There’s always a bit of tension when getting features from idea to production. In this talk, I describe some of the changes CustomInk has made to reduce this friction and keep the new features coming. Gone are the days of bi-monthly deploys, office pools dedicated to guessing when this deploy will be rolled back, and the ceremony surrounding the deploy-rollback-fix-deploy cycle. Today, ideas flow from product managers to developers to production with ease thanks to a number of changes that we’ve made to our teams, processes and tools.

Presenting at RailsConf was a really enjoyable experience and the presentation was well received. There were lots of questions from the audience after the presentation. Unfortunately, the Q & A section was not captured in the video. I’d really appreciate any questions or feedback you have, just drop a comment below.

Video

Slides

_{Reposted from the CustomInk Technology blog.}

This week at CustomInk, the Web Operations team was asked by the development teams to make some configuration changes on a couple of different servers.

They were simple changes, adding a line or two to the services.yml file for each application. The details really aren’t important but let’s look at how we worked together to implement the changes.

Before Chef

In the past, here’s how the changes likely would have been implemented.

1. Developer realizes a change is required.
2. Developer asks the ops team to make the change
3. Ops makes the update using the appropriate tools

OR

1. Developer realizes a change is required.
2. Developer considers asking the ops team to make the change but thinks better of it
3. Developer makes the change manually, Ops doesn’t know, Ops can no longer provision a new server properly.

With Chef

We’ve been using Chef for some time and have just started asking our developers to help maintain their own apps. This week, two applications needed to know about some additional external services. This required a simple update to a YAML file in each application. In both cases, I asked the developers to clone our chef repo, make the changes they needed, and submit a pull request.

In one instance, the simple services.yml turned into a pull request with updates to a number of nagios nrpe checks that we’re running. Something that the developer didn’t ask for originally but took the initiative to add while in the code.

Thanks to @chmurph2 and @jmorton for taking their first steps into Chef.

Is this a huge accomplishment? No. But it is a great first step.

”.@nathenharvey Working together == every engineer is on the same team and you stop celebrating (or thinking about) cross-team collaboration.”

We’ve always worked as one team but continue to have some clear areas of responsibility. While I understand what Brian’s saying, I’m not sure everyone doing everything makes sense. We’re one team but we each have our strengths. Agree that we should stop celebrating about this a cross-team collaboration; it should be the norm. But, we have to start somewhere and these were the first steps into the world of infrastructure as code for the developers. In my mind, that’s a WIN!

In my previous post, I introduced Green Screen, a build monitoring tool that is designed to be used as a dynamic Big Visible Chart (BVC) in your work area. It lets you add links to your build servers and displays the largest possible information on a monitor so that the team can see the build status from anywhere in the room.

It is easy enough to get Green Screen up and running on your own server or VM. The project’s README includes all the information you’ll need for doing so. In this post, I’ll describe the steps necessary to run Green Screen on Heroku or on your own server using Chef.

Deploying to Heroku

Deploying to Heroku is probably the easiest way to get up and running with Green Screen. You’ll need a Heroku account but a free one should be sufficient. Check the quick start guide if you don’t yet have an account.

Once you’ve got your Heroku account set-up, simply follow these steps to get your Green Screen app deployed:

1. git clone git@github.com:customink/greenscreen.git
2. cd greenscreen
3. gem install heroku
4. heroku create
5. git push heroku master
6. heroku open

If your build servers are running on the Internet, Heroku may be all that you need.

Warning this default Green Screen looks at all of the builds currently running on http://ci.jenkins-ci.org. This is fine for demo purposes but you may find it to be a bit overwhelming since it’s over 300 builds at the time of this writing.

You can see a sample of this app running at http://greenscreenapp.com

Green Screen is a build monitoring tool that is designed to be used as a dynamic Big Visible Chart (BVC) in your work area. It lets you add links to your build servers and displays the largest possible information on a monitor so that the team can see the build status from anywhere in the room.

We use Green Screen at CustomInk to look after our continuous integration servers, currently 3 Hudson servers and one Jenkins cluster. We have a monitor mounted in the engineering office that makes it easy for everyone to quickly assess the build status.

Green Screen is a simple Sinatra application that is easy to configure and deploy. It works well with any continuous integration server that conforms to the multiple project summary reporting standard.

You can see a sample Green Screen app running at http://greenscreenapp.com. Be forewarned, this sample Green Screen looks at all of the builds currently running on http://ci.jenkins-ci.org. This is fine for demo purposes but you may find it to be a bit overwhelming since it’s over 300 builds at the time of this writing.

I’ve recently switched my blog from Blogger to Octopress. After working with Blogger, WordPress, and TypePad I’ve found that the Octopress works best for my personal blog.

Creating and editing posts in a text editor instead of a browser is a step-saver for me. Previously I always worked with a local copy of each article and would cut-n-paste between my text editor and the browser.

The rake- and git-based workflow feels very natural. After all, this is a “blogging framework for hackers.”

The standard layout and plugins are working well for me with little customization.

Deploying is a snap. I’m currently using Heroku to host the blog but could just as easily be using github:pages.

The simplicity and familiar workflow really make for an excellent blogging platform! Thanks @imathis for giving us Octopress!

During this year’s MongoSV, 10gen presented awards to peer-nominated community members for their contributions to the MongoDB project.

There were 3 different categories:

1. Innovative Application Award - recognizes a company or individual who has built an innovative application using MongoDB.
2. Core Contributor Award - recognizes a community member for significant contribution to the codebase of the MongoDB core server, language drivers, or tools.
3. Community Champion Award - recognizes an individual for their efforts evangelizing and growing the MongoDB community.

This is the first year that 10gen has organized these awards. I think they’re a great way to recognize the contributions made by a few members of the community. I hope a significant portion of people in the community took the time to nominate or vote for another individual. I suspect more community members will participate and 10gen will do a more to advertise the program in years to come.

I was selected to share some of my experience with MongoDB at MongoSV.

“MongoSV is an annual one-day conference in Silicon Valley dedicated to the open source, non-relational database MongoDB. The comprehensive agenda includes 50+ sessions covering topics for both the novice and experienced user, with presentations from 10gen engineers as well as MongoDB users.”

Impressions and Observations

I’ve been to a few other similar MongoDB events in NYC and DC but this was the largest by far. There were over 1,000 attendees and 5 tracks plus whiteboard and birds of a feather sessions.

Schema free != Design free

As freeing as a document store is, there’s still work to be done in designing how you’ll store and retrieve your data. It’s super-easy to get up and running without giving this much thought but you will need to design your documents eventually. There were at least 3 sessions about schema design during MongoSV.

Aggregation Framework

The new aggregation framework is great, highly anticipated, and much needed. During the Keynote, Eliot Horowitz demonstrated an application built on MonogDB that used the Twitter API to capture tweets related to the day’s events. The demo site ran throughout the day and was used in a contest to see who tweeted the most about the conference and who was mentioned most. The source code for the app was subsequently made available on github. Check out Chris Westin’s presentation on MongoDB’s New Aggregation Framework.