Nathen Harvey

a blog

MVT: Foodcritic and Travis CI

One of the big themes that emerged during #ChefConf was that we should be testing our infrastructure code. Software engineers have been practicing test-driven development, behavior-driven development, continuous integration, and many other testing-related practices for a long time. It’s becoming more important for the infrastructure engineers to learn from and apply these practices to our day-to-day workflow. When it comes to testing Chef-driven infrastructure automation, there are a number of tools and practices that are starting to emerge. In this article I’ll look at a “minimum viable testing” (MVT) approach to this problem using Foodcritic and Travis CI. Follow the steps in this article to get your public cookbooks tested after every git push.

Testing with Chef

The idea of building automated tests for your infrastructure code has been getting a lot of traction lately. When it comes to Chef, many tools are starting to emerge.

The first tool in this area to get any significant traction, that I know of, was cucumber-chef. I first learned of this tool when I saw a pre-release copy of Test-Driven Infrastructure with Chef at the O’Reilly booth at Velocity Conf 2011. Stephen Nelson-Smith, the book’s author and framework’s lead developer, proposes an outside-in approach to testing where your tests can also act as monitors that look after the health of your infrastructure. I like the idea of this approach and feel it makes a lot of sense in a greenfield environment. One benefit of this approach is that it blurs the line between testing and monitoring. You can easily hook-up your monitoring system to your cucumber tests.

ChefSpec is another tool for testing your Chef code. It is a gem that makes it easy to write RSpec examples for Chef cookbooks. This style of testing allows you to execute your tests without needing to converge the node that your tests are running on. In other words, you can execute your tests without needing to provision a server. One huge appeal to this style of testing is that the feedback loop is very small. You’ll get feedback about your cookbook changes within seconds or a very few minutes of saving your changes.

Minitest Chef Handler is yet another tool for testing with Chef. This runs a suite of minitest tests as a report handler in your Chef-managed nodes. As you may know, report handlers are run at the end of each chef run, or convergence.

Testing at ChefConf

At the inaugural #ChefConf there were many sessions that included information about many companies’ approach to testing. Here’s a quick list of some of the sessions:

Foodcritic

Foodcritic is a lint tool for your Chef cookbooks.

“Foodcritic has two goals:

  • To make it easier to flag problems in your Chef cookbooks that will cause Chef to blow up when you attempt to converge. This is about faster feedback. If you automate checks for common problems you can save a lot of time.

  • To encourage discussion within the Chef community on the more subjective stuff - what does a good cookbook look like? Opscode have avoided being overly prescriptive which by and large I think is a good thing. Having a set of rules to base discussion on helps drive out what we as a community think is good style.”

Why start with Foodcritic?

Given the plethora of options available, why should you start with Foodcritic? Well, you have to start somewhere. We felt Foodcritic was a good choice because it was easy to get started with, the tests ran quickly, and we are working under the assumption that once we started some automated testing, we’ll start layering on more and more pieces as we go. After some initial experiments, we found that we could get Foodcritic looking after our each cookbook in a matter of minutes and local tests running in seconds.

The pseudo-converge approaches (like ChefSpec) initially feel like we’ll need to do a lot of mocking that will take some time to get correct. The post-converge approaches (like cucumber-chef and minitest) will take longer to run and are a bit more complex.

One benefit of the post-converge approach is the ability to use your tests as health monitors. We already have monitoring in place and use it as an indicator that a node is fully provisioned. We call this “monitor-driven development.” Given that, it was better for us to get started with something that runs without requiring a full converge. Foodcritic fit the bill quite nicely.

Travis CI

Travis CI is:

“A hosted continuous integration service for the open source community.”

Using Travis CI in conjunction with Foodcritic, we’d have a basic automated test foundation to build on.

Automated Foodcritic tests with Travis CI

Using Foodcritic and Travis CI, you can quickly set-up a “minimum viable testing” (MVT) environment. The idea is that once you have some sort of tests running against your cookbooks, you’ll want to add more and doing so will be easy. Let’s look at how to add Foodcritic and Travis CI to your cookbook workflow.

Initial set-up

Follow these steps to get everything set-up and ready for your first tests:

  1. gem install foodcritic
  2. Go to Travis CI and follow the Sign In link at the top.
  3. Activate the GitHub Service Hook for your cookbook’s repository from your TravisCI profile page. Each of your cookbooks has its own repository, right?!

Configure your project

The next step is to add a .travis.yml file to your project.

.travis.yml
1
2
3
4
5
6
7

language: ruby
gemfile:
   - test/support/Gemfile
rvm:
  - 1.9.2
  - 1.9.3
script: bundle exec rake foodcritic

This file tells Travis CI how to build your project. We’ve specified the language (ruby) and the versions of ruby to use when testing this cookbook (1.9.2 and 1.9.3). We’ve also specified a Gemfile and script to execute when testing this project. Let’s add a Gemfile to a new directory in our cookbook, test/support.

1
2

mkdir -p test/support
touch test/support/Gemfile

Our Gemfile is pretty simple, just include rake and foodcritic.

Gemfile
1
2
3
4

source "https://rubygems.org"

gem 'rake'
gem 'foodcritic'

Finally, we’ll need to add a Rake file that will be run each time Travis builds our project.

Rakefile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/usr/bin/env rake

desc "Runs foodcritic linter"
task :foodcritic do
  if Gem::Version.new("1.9.2") <= Gem::Version.new(RUBY_VERSION.dup)
    sandbox = File.join(File.dirname(__FILE__), %w{tmp foodcritic cookbook})
    prepare_foodcritic_sandbox(sandbox)

    sh "foodcritic --epic-fail any #{File.dirname(sandbox)}"
  else
    puts "WARN: foodcritic run is skipped as Ruby #{RUBY_VERSION} is < 1.9.2."
  end
end

task :default => 'foodcritic'

private

def prepare_foodcritic_sandbox(sandbox)
  files = %w{*.md *.rb attributes definitions files libraries providers
recipes resources templates}

  rm_rf sandbox
  mkdir_p sandbox
  cp_r Dir.glob("{#{files.join(',')}}"), sandbox
  puts "\n\n"
end

This Rakefile will copy the contents of our cookbook to a temporary directory and run the foodcritic tests on the temporary directory. Note the --epic-fail tag is used to fail the build (return a non-zero exit code) on any rule that does not pass.

That’s it! When you push your commit to github, you should see Travis CI pick-up the changes, run your build, and report on status.

Share Your Build Status

One final step that you may consider is adding a build status indicator to your README. This simple line in your README will let others know what the current build status is for your cookbook.

1

[![Build Status](https://secure.travis-ci.org/[YOUR_GITHUB_USERNAME]/[YOUR_PROJECT_NAME].png)](http://travis-ci.org/[YOUR_GITHUB_USERNAME]/[YOUR_PROJECT_NAME])

Thanks & Additional Resources

A big “Thank You!” shout-out to Fletcher Nichol and Eric G. Wolfe from whom I ‘borrowed’ the Rakefile and .travis.yml used in this post.

More information on Foodcritic and Travis CI can be found here:

Next Post

Be sure to read the next post on this topic: MVT: knife test and Travis CI

Comments