From Code Review to OpsWorks

May 30, 2014

You pride yourself on code quality. You do code reviews. You have discrete environments for staging code changes. You have a continuous integration server. Hey, maybe you even do continuous deployment.

Maintaining all of these processes can still take a toll on productivity. All of these moving parts are automated on their own, but how can we integrate all of them into one seamless review, stage, and deploy process? We'd like to share what we have come up with.

Let's map out exactly how we are going to string all of these disparate systems together. We'll start in chronological order:

Step 1: Code Review

Here at Vistar, we do code reviews. This may seem like an arduous task. It's not. To help us along, we use Gerrit. This post will not contain instructions on how to set up Gerrit, but here is a brief summary:

Gerrit integrates with your git repositories, and allows code to be pushed to ephemeral branches. There, it can be reviewed and checked out without a permanent commit to the destination branch. Until it's been tested and reviewed, it will only exist in The World of Gerrit (a very mystical place).

Step 2: Continuous Integration

There are countless CI servers out there. We use Jenkins which has a very handy plugin for integration with Gerrit.

Let's assume that your git repository is named your-project, and that you've already created the corresponding project in Gerrit.

The goal here is to have Gerrit trigger a build in Jenkins on a new change. Then, Jenkins can test the change and mark the patch Verified in Gerrit. This is to ensure that only code that has been reviewed and tested will be merged. Let's create a job in Jenkins and give it a descriptive name, maybe your-project.gerrit? Then, assuming the Gerrit Trigger plugin has been installed in Jenkins, we'll fill in the "Build Trigger" section as show below:

Jenkins example

After this configuration is saved, this job will run any time a new patch is sent to Gerrit for review. Make sure that the "Poll SCM" checkbox in not checked. Assuming you have configured the job to build your code and run your unit tests, when the build passes, you will see this lovely sight on your patch in Gerrit:

Gerrit verified Oh joy

Now all that is left to do is get Mark to give me that much-coveted +2, and I can submit this code for a merge. Gerrit will very generously take care of this for me once I submit my patch. Once submitted in Gerrit, the other Jenkins job we have secretly (too lazy to mention it) set up, perhaps named your-project, is building because this change has been merged to the actual master branch (instead of the ephemeral one created by Gerrit). And now that the build is complete, we are ready to deploy. You can sleep soundly because every line of code has been both reviewed and tested.

Yesssssssssss!

But to where?

Step 3: Continuous Deployment

Having a fresh build ready to go on every checkin is great and all, but it still needs to be released into the wild. Deployment methods vary from platform to platform. For many of our deployments, we use Amazon OpsWorks.

OpsWorks allows you to divide your instances into different stacks, which could represent different environments (development, staging, production). Within those stacks, you can create layers which represent the types of instances you wish to deploy (REST server, scheduler, etc.). In this case, we have a production stack which contains a your-project layer. It might look something like this in the AWS Console:

OpsWorks layer

Another big plus for us, is that OpsWorks gives you the ability to use a set of custom Chef cookbooks and execute the recipes within them programatically. This has allowed us to abstract out a flexible and re-usable deployment process into a set of Chef providers. We then use these providers in recipes specific to each of our many applications.

Magic

Without getting into the details of the providers right now, let's just say that when the your_project cookbook is used, your instance is configured properly and your code is deployed. You can link a stack with your cookbooks by editing the stack and completing the Configuration Management form in the AWS Console:

Configuration Management

Then when you want to do a deployment, you can either do so manually through the console, or utilize the API and automate that jawn. Clearly we want to automate this, but debugging tools are always important. To run a deployment manually in the console, go to the Deployments section of your OpsWorks stack. Click the Run Command button, and fill out this form:

Deployment form

Now for the automation. We are big fans of both Python and AWS, and as such are very familiar with the swiss-army-knife that is the boto package. The latest versions of boto have OpsWorks integration. We will also be using the clize package, which will turn our script into an easy to use command line utility.

#!/usr/bin/python

import boto.opsworks.layer1 as opsworks

from clize import clize
from clize import run


STACK_NAME = 'production'

def get_stack(conn, stack_name):
  for stack in conn.describe_stacks()['Stacks']:
    if stack.get('Name') == STACK_NAME:
      return stack

@clize
def app(app_name):
  conn = opsworks.OpsWorksConnection()
  stack = get_stack()

  args = {
    'stack_id': stack.get('StackId'),
    'comment':  'Automated Deployment',
    'command':  {
      'Name': 'execute_recipes',
      'Args': {'recipes': [app_name]}
    }
  }

  conn.create_deployment(**args)

Then, after giving the script execute permissions, you should be able to run:

$ ./deploy app your-project

A deployment will be created in OpsWorks that will run the your-project recipe from your custom cookbooks. Now, all that is left to do is run this script in the your-project job in Jenkins (after the tests pass, of course).

What about multiple environments?

If you have, say, a discrete development environment for testing, you will probably want to deploy to that environment before production. We'll only have to make a few minor adjustments to get that working.

First, let's assume you have a develop branch which represents the code you expect to be deployed to the development environment. We will also assume that you have a stack set up in OpsWorks named development with a corresponding your-project layer. Next, we'll setup a new job in Jenkins named your-project.develop, and set it to watch the develop branch. Now we'll just need a little modification to our deploy script:

#!/usr/bin/python

import boto.opsworks.layer1 as opsworks

from clize import clize
from clize import run


def get_stack(conn, stack_name):
  for stack in conn.describe_stacks()['Stacks']:
    # Use the passed stack name instead of a static one
    if stack.get('Name') == stack_name:
      return stack

@clize
def app(app_name, environment):
  # Take the environment name as an argument which
  # matches the stack name.
  conn = opsworks.OpsWorksConnection()
  stack = get_stack(environment)

  args = {
    'stack_id': stack.get('StackId'),
    'comment':  'Automated Deployment',
    'command':  {
      'Name': 'execute_recipes',
      'Args': {'recipes': [app_name]}
    }
  }

  conn.create_deployment(**args)

$ ./deploy app your-project development

There you go. Now all of your environments are happy and deployed. Look at those little guys. All grown up.