TECH.insight

Testing infrastructure with Test Kitchen

Monday 16 March 2015

Test Kitchen should be thought of as TDD for infrastructure

Infrastructure always has a habit of being the last piece of the puzzle. It is always the part that ends up taking much longer than expected, no matter how many times you’ve done it before.

We have automated testing for Software Development so why not Infrastrucutre?

With the emergence of ‘infrastructure as code’, the responsibility for provisioning infrastructure is no longer the domain of a system administrator alone. This makes the need to test changes in isolation even more important, since bringing down an infrastructure stack will often have more drastic consequences than a single bug in a piece of software.

This is where Test Kitchen comes in – as the glue between provisioning tools e.g. Chef, Puppet or Ansible, the infrastructure being provisioned, e.g. AWS, Docker or VirtualBox and the tests to validate the setup is correct. Historically, Test Kitchen is used to test single cookbooks or packages, but is easily adapted to test the group of cookbooks that make up your environment.

The concept behind Test Kitchen is that it allows you to provision (convergence testing) an environment on a given platform (or platforms) and then execute a suite of tests to verify the environment has been set up as expected. But, as with software, you need to write effective tests – not just the assert(true) tests I see all too often.

This can be particularly useful if you want to verify a setup against different operating systems (OS) and/or OS package versions. This can even be set up as part of your Continuous Integration (CI) and/or delivery pipelines, and also feeds nicely into the concept of ‘immutable infrastructure’, which I’m a big fan of.

With Test Kitchen, we break the configuration into Driver, Provisioner, Platform and Test Suites. In this simple example, running Test Kitchen would launch an Ubuntu 14 and using the apt cookbook run an apt update.

---
driver:
  name: vagrant
provisioner:
  name: chef_solo
platforms:
  - name: ubuntu-14.04
suites:
  - name: default
    run_list:
    - "recipe[apt]"
    attributes:

The Driver section allows you define which underlying virtualisation platform you are going to use to launch your environment and tests. The defaults are Vagrant and VirtualBox, but others such as AWS EC2, Docker and Digital Ocean are also supported.

The Platforms section allows you define the OS versions you want to run on. When using drivers such as Docker and Vagrant, Test Kitchen is pretty good at figuring out which containers these map to. Others like AWS require AMIs to know what to launch.

The Provisioner section defines what you want to use to converge the environment. Chef Solo/Zero and shell provisioners are the easiest to get started with, but there are provisioners also available for Puppet and Ansible.

Finally the Test Suites section is where the actual value comes into play. This is where you define the tests to run against each platform when convergence execution has completed.

Putting it all together

While setting up a platform, I would usually wrap all the recipes into a single-wrapper cookbook, known as the Environment Cookbook Pattern.

While you can use Chef roles and environments to manage recipes, I find it much cleaner to use a single-wrapper cookbook to define a stack used by an app. This approach also makes it far easier to version and feed into different tools (e.g. Test Kitchen, Packer or Chef Server) instead of maintaining different configuration files for each setup. For this example, the recipe run list and Chef configuration are defined in the Test Kitchen configuration to make it easier to follow.

I tend to use Docker to test with as it is much quicker to spin up instances compared with default Vagrant and VirtualBox instances. In more complex setups, it is possible to set the driver on a per-test or platform basis to allow you to run against different virtualisation platforms.

Sample Test Kitchen config

---
driver:
  name: docker
provisioner:
  name: chef_solo
platforms:
  - name: ubuntu-12.04
    driver_config:
      provision_command:
        - apt-get update -y
  - name: ubuntu-14.04
  - name: centos-6.4
    driver_config:
      provision_command:
      - yum install -y tar curl
suites:
  - name: default
    run_list:
    - "recipe[mongodb]"
    - "recipe[tomcat]"
    - "recipe[java]"
    - "recipe[apache]"
    attributes:
      java:
        install_flavor: openjdk
        jdk_version: 7

Now we are able to run Test Kitchen and verify that each of the Chef recipes used to provision our environment will execute successfully. But do we actually know that everything is in place? Have the correct Tomcat and Java versions been installed that we expected, for example?

This is where we need to write some tests to verify that everything is in place. We should test things like versions installed, services running, and files in the correct place. Tests don’t lie.

We have all been caught out by a gem or cookbook being updated – and then spent hours trying to figure out why things just stopped working… The update to the Apache cookbook from the default Apache version 2.2 to 2.4 is a prime example. Even while putting together a sample for this article, I found a couple of bugs and unexpected behaviour in some of the cookbooks I was using. (I also notice that very few cookbooks seem to show CentOS any love.)

With infrastructure tests in place, we can pick up these changes sooner in the deployment lifecycle rather than getting caught with a broken environment that others rely on.

The easiest to get running is to write simple Bash script tests using Bats. If you’ve ever written a Bash script, Bats is easy to get the hang of.

A simple Bats testing to verify Java is installed:

@test "that java is installed" {
  run test -f /usr/bin/java
  [ "$status" -eq 0 ]
}

@test "that java libs is installed" {
  run test -d /usr/lib/jvm
  [ "$status" -eq 0 ]
}

If you require more complex tests, then Serverspec might be a better fit since it has a far richer domain for writing platform-agnostic tests. Serverspec provides resources to check such issues as services installation, file contents and OS settings.

An example Serverspec test to verify Java version and installed services is shown below:

require 'serverspec'

set :backend, :exec

describe command('java -version'), :if => os[:family] == 'redhat' do
  its(:stdout) { should match /1.7.0_75/ }
end

describe command('java -version'), :if => os[:family] == 'ubuntu' do
  its(:stdout) { should match /1.7.0_65/ }
end

describe service('tomcat7')
  it { should be_installed }
  it { should be_running }
end

describe service('httpd'), :if => os[:family] == 'redhat' do
  it { should be_installed }
  it { should be_running }
end

describe service('apache2'), :if => os[:family] == 'ubuntu' do
  it { should be_installed }
  it { should be_running }
end

If applied to immutable infrastructure, you would start to add app tests to verify the entire app setup has been deployed and can run.

Running Test Kitchen

There are two key phases to Test Kitchen: converge and verify. Converge will provision the infrastructure and verify will run test suites against the infrastructure.

To start with, you can see which tests are available by running kitchen list, which will list all the combinations of platforms and tests configured.

$ kitchen list
Instance             Driver  Provisioner  Last Action
default-ubuntu-1204  Docker  ChefSolo     Created
default-centos-64    Docker  ChefSolo     <Not Created>

You can run each phase of Test Kitchen separately, but running kitchen test will create a container, run the convergence, run the tests and tear everything down.

$ kitchen test ubuntu
///... Lots of output while the tests runs
///... Sample output: https://travis-ci.org/peterabbott/testing.infrastructure/jobs/53912697
$ kitchen list
Instance             Driver  Provisioner  Last Action
default-ubuntu-1204  Docker  ChefSolo     Converged
default-centos-64    Docker  ChefSolo     <Not Created>

As the example above shows, only the Ubuntu instances have been converged. Test Kitchen can take multiple formats of inputs: you could run all tests, tests only against Ubuntu or specific test suites:

kitchen test  # run all test suites against all platforms
kitchen test ubuntu  # run all test suites only against Ubuntu
kitchen test default  # run default test suite against all platforms
kitchen test default-ubuntu-1204  # run default test suite against Ubuntu 12.04

During testing, you might encounter times when tests fail and you can’t figure out why. The staged approach of Test Kitchen allows you to run converge and verify separately without tearing down the infrastructure stack.

This means you can connect to an environment and its (partially) provisioned setups to have a poke around, which can be particularly useful during initial setup of a new OS or packages. It allows you to look around and see how things are set up without breaking any ‘real’ environments. And if you do break something, you can just throw it away and start again.

To make connecting to a provisioned environment easier, Test Kitchen provides a login command so you don’t have to worry about figuring out ports and SSH keys – it remembers all of that for you.

So as we see, Test Kitchen can provide an easy mechanism by which to test your infrastructure, particularly after it has been through your provisioning tool of choice. You spin up or update environments, safe in the knowledge that you won’t bring them down and have angry users blaming you – or an annoyed support team whose servers you accidentally broke…

At the very least, testing the convergence of any setup you create should always come with a basic Test Kitchen config to verify the environment convergence. The next step is to start adding tests to verify your expectations.

Using Test Kitchen should be thought of as TDD (Test Driven Development) for infrastructure.

What’s next?

After you have Test Kitchen in place, you could set up a tool such as Packer to generate your infrastructure VMs with confidence it will work. All this together would put you in a good position to implement a successful CD pipeline.

The example in this article uses Chef Solo as the provisioning tool and Docker as the VM platform – both available on GitHub if you would like to try it out for yourself.

About The Author

Peter Abbott is a Technical Architect based in Wellington, New Zealand. He's a Cycling and Gelato fanatic.

@piemanpete