Last week Engine Yard announced the winners of their inaugural Engine Yard Innovators Awards.

I was both honored and surprised when I learned that I had been chosen as the Engine Yard Innovator in the DevOps category.

Our DevOps category winner is Nathen Harvey. A devops guru who has been travelling from conference to conference evangelizing the use of Chef, Nathen is known for his “Rails With Chef” proficiency. He works to help others understand the importance of backend compatibility.

I was asked to write a post about some my work. This seemed like a good time to reflect on some of the work I’ve been doing for the past few years.

In late 2009, I joined CustomInk to head up the Web Operations team. While there, I was able to help drive the adoption of many DevOps practices including automation, continuous delivery, collaboration across teams, expanded responsibility, and participation in open source communities.

Learning Chef Series

• Part 1 - Introduce the project, configure workstation, and register a node with hosted Chef
• Part 2 - Download cookbooks from the community site, add MongoDB, Apache, and Passenger to our node
• Part 3 - Start writing a cookbook to deploy our application
• Part 4 - Finish the application deploy, introduce roles

Part 4 of our Learning Chef tutorial was run as a Google+ Hangout that was streamed to YouTube.

In Part 4, we completed the application deploy and then looked at roles.

I’ll update this post soon with a breakdown of each step we took during this session. In the meantime, you can watch the entire video below.

Also, you can grab the code from the following repositories on github:

• Chef Repository
• Rails Application

What’s Next

In Learning Chef - Part 5 we will move the MongoDB to it’s own VM.

In the meantime, please let us know what you think of this post and these videos! Drop a note in the comments or reach out to @nathenharvey or @mulpat on twitter.

Learning Chef Series

• Part 1 - Introduce the project, configure workstation, and register a node with hosted Chef
• Part 2 - Download cookbooks from the community site, add MongoDB, Apache, and Passenger to our node
• Part 3 - Start writing a cookbook to deploy our application
• Part 4 - Finish the application deploy, introduce roles

Part 3 of our Learning Chef tutorial was run as a Google+ Hangout that was streamed to YouTube.

In Part 3, we added a bunch of cookbooks from the community site including git, application, and application_ruby. After adding these cookbooks, we created a cookbook of our own to deploy a sample Rails application.

The application wasn’t fully deployed by the end of the tutorial but we’ll pick-up from there next time.

I’ll update this post soon with a breakdown of each step we took during this session. In the meantime, you can watch the entire video below.

Also, you can grab the code from the following repositories on github:

• Chef Repository
• Rails Application

What’s Next

In Learning Chef - Part 4 we finish the deployment of the sample application and then explore roles.

In the meantime, please let us know what you think of this post and these videos! Drop a note in the comments or reach out to @nathenharvey or @mulpat on twitter.

Learning Chef Series

• Part 1 - Introduce the project, configure workstation, and register a node with hosted Chef
• Part 2 - Download cookbooks from the community site, add MongoDB, Apache, and Passenger to our node
• Part 3 - Start writing a cookbook to deploy our application
• Part 4 - Finish the application deploy, introduce roles

Part 2 of our Learning Chef tutorial was run as a Google+ Hangout that was streamed to YouTube.

Review of Part 1

• Read the blog post for Part 1

Discuss Chef Solo vs. Chef Server

Chef Solo allows you to run Chef Cookbooks without a Chef Server. There are a number of things that you don’t get when using Chef Solo. Check the Chef Solo page on the wiki for more information.

Learning Chef Series

• Part 1 - Introduce the project, configure workstation, and register a node with hosted Chef
• Part 2 - Download cookbooks from the community site, add MongoDB, Apache, and Passenger to our node
• Part 3 - Start writing a cookbook to deploy our application
• Part 4 - Finish the application deploy, introduce roles

In November of 2012, Patrick Mulder posted a request on the Chef mailing list. He was

looking for some 1-1 teaching via skype to help me get going in setting up a basic DB server from scratch, as well as a basic dev server as intermediary step.

I thought this would be an excellent opportunity to feed my recent addiction to Google+ Hangouts. I would provide Patrick some one-on-one tutoring if he would agree to having the sessions broadcast live on YouTube. We had some technical issues getting our first session going in a Google+ Hangout but we were able to meet via Skype and I captured video of the session.

Our goal is to help you get up and running on Chef by following our progress. The intent is to have additional sessions run via Google+ Hangouts that are steamed live to YouTube. This post includes our first session which has been broken into nine short videos. I hope you enjoy these videos and are able to learn something about Chef, too. Both Patrick and I are looking forward to your feedback on this experiment.

You can watch all of the videos in the YouTube playlist or keep reading and watch each video in turn.

Introduction

In this video we introduce ourselves and the project.

In my last post, MVT: Foodcritic and Travis CI I described the process for having Travis CI look after your cookbooks and run Foodcritic, the cookbook lint tool, on your cookbook after each git push. In this post, we’ll iterate on the “Minimum Viable Test” idea by adding in support for knife’s cookbook testing.

Wait, I’m already running foodcritic, do I really need to run knife cookbook test, too?

I’ll use a very simple example to demonstrate that you do.

Let’s create a very basic cookbook:

1
2
3
4

knife cookbook create very_basic
** Creating cookbook very_basic
** Creating README for cookbook: very_basic
** Creating metadata for cookbook: very_basic

Next, we’ll write a flawed recipe:

1
2
3
4

package "flawed" do
  action :nothing
end
end

Now, run foodcritic on this cookbook:

1

foodcritic cookbooks/very_basic

Foodcritic doesn’t throw any errors or find any problem with the cookbook.

Let’s try testing it with knife:

1
2
3
4
5
6

knife cookbook test very_basic
checking very_basic
Running syntax check on very_basic
Validating ruby files
FATAL: Cookbook file recipes/default.rb has a ruby syntax error:
FATAL: /Users/nharvey/projects/chef-hosted/.chef/../cookbooks/very_basic/recipes/default.rb:22: syntax error, unexpected keyword_end, expecting $end

OK, it should now be obvious that knife cookbook test should be included as part of our MVT.

One of the big themes that emerged during #ChefConf was that we should be testing our infrastructure code. Software engineers have been practicing test-driven development, behavior-driven development, continuous integration, and many other testing-related practices for a long time. It’s becoming more important for the infrastructure engineers to learn from and apply these practices to our day-to-day workflow. When it comes to testing Chef-driven infrastructure automation, there are a number of tools and practices that are starting to emerge. In this article I’ll look at a “minimum viable testing” (MVT) approach to this problem using Foodcritic and Travis CI. Follow the steps in this article to get your public cookbooks tested after every git push.

Testing with Chef

The idea of building automated tests for your infrastructure code has been getting a lot of traction lately. When it comes to Chef, many tools are starting to emerge.

The first tool in this area to get any significant traction, that I know of, was cucumber-chef. I first learned of this tool when I saw a pre-release copy of Test-Driven Infrastructure with Chef at the O’Reilly booth at Velocity Conf 2011. Stephen Nelson-Smith, the book’s author and framework’s lead developer, proposes an outside-in approach to testing where your tests can also act as monitors that look after the health of your infrastructure. I like the idea of this approach and feel it makes a lot of sense in a greenfield environment. One benefit of this approach is that it blurs the line between testing and monitoring. You can easily hook-up your monitoring system to your cucumber tests.

ChefSpec is another tool for testing your Chef code. It is a gem that makes it easy to write RSpec examples for Chef cookbooks. This style of testing allows you to execute your tests without needing to converge the node that your tests are running on. In other words, you can execute your tests without needing to provision a server. One huge appeal to this style of testing is that the feedback loop is very small. You’ll get feedback about your cookbook changes within seconds or a very few minutes of saving your changes.

Minitest Chef Handler is yet another tool for testing with Chef. This runs a suite of minitest tests as a report handler in your Chef-managed nodes. As you may know, report handlers are run at the end of each chef run, or convergence.

Testing at ChefConf

At the inaugural #ChefConf there were many sessions that included information about many companies’ approach to testing. Here’s a quick list of some of the sessions:

• Food Fight Show Episode #10 - TESTALLTHETHINGS – This wasn’t actually part of #ChefConf but is ‘required listening’ for anyone interested in learning more about this space.

• #ChefConf Pre-event Hackday: TEST ALL THE THINGS!!!

• NTP Cookbook with tests - tests were added to this cookbook as part of the hackday event.

• Test-driven Development for Chef Practitioners (video)

• Test Driven Development Roundtable (video)

When I first started working with Chef, there were a couple of areas that I knew were going to be really awesome and helpful but I wasn’t sure how to get started with them. In this presentation, I’ll provide a quick introduction to five things you’ve always wanted to know about Chef but were afraid to ask.

I gave this presentation at #ChefConf 2012.

Level-up your Chef skills by learning about these areas of Chef:

• Attribute Precedence - Role, environment, cookbook, data bag? Which attribute value will be used in my chef run?
• Encrypted Databags - Chef 0.10 brought us encrypted databags. We’ll look at how to create and use databags and how to keep them up-to-date in your repository.
• LWRP - What is a LWRP? How and why do you create one? We’ll look at a couple of sample LWRPs and learn how to build a simple one.
• Error Handlers - Demystify exception and report handlers by writing a simple one and seeing examples of how they work in the wild.
• Capistrano and Chef - Take a quick look at why and how to integrate Chef search into your Capistrano configuration to make deploying your Rails apps even easier.

One thing I didn’t mention in the presentation was how to use the data from the encrypted data bag. I’ve updated the slides to include this info but it doesn’t appear in the video. In any case, here’s a quick demo of how you might use it:

1
2
3
4
5
6
7
8
9
10
11

creds = Chef::EncryptedDataBagItem.load("db", "creds")
env_db_creds = db_creds[node["rails_env"]]

template "#{app_dir}/shared/config/database.yml" do
  source "database.yml.erb"
  variables(
    :rails_env => node["rails_env"],
    :username => env_db_creds["username"],
    :password => env_db_creds["password"]
  )
end

Video

Slides

_{Reposted from the CustomInk Technology blog.}

You’ve heard of Chef, Puppet, and other frameworks that can help you build out your infrastructure. You’ve been meaning to play around with one or more of them for some time now. Now’s your chance; Start cooking up on your own servers!

In this presentation, I provide an introduction to Chef with a focus on what you’ll need to know to get a Rails application up and running.

Topics include: * Introduction to Chef * Nodes, roles, environments, and other terminology * Introduction to cookbooks * Provisioning an environment for a Rails application * Deploying with Capistrano

You won’t be ready to compete in Iron Chef, but you will be ready to serve up your own Rails environment in no time.

I gave slightly different versions of this presentation at RubyNation 2012 and #ChefConf 2012.

I’d really appreciate any comments, questions, or feedback in the comments section below.

Video from ChefConf

Slides

_{Reposted from the CustomInk Technology blog.}

At RailsConf 2012, I gave a presentation on how the CustomInk web operations team enables developer productivity.

There’s always a bit of tension when getting features from idea to production. In this talk, I describe some of the changes CustomInk has made to reduce this friction and keep the new features coming. Gone are the days of bi-monthly deploys, office pools dedicated to guessing when this deploy will be rolled back, and the ceremony surrounding the deploy-rollback-fix-deploy cycle. Today, ideas flow from product managers to developers to production with ease thanks to a number of changes that we’ve made to our teams, processes and tools.

Presenting at RailsConf was a really enjoyable experience and the presentation was well received. There were lots of questions from the audience after the presentation. Unfortunately, the Q & A section was not captured in the video. I’d really appreciate any questions or feedback you have, just drop a comment below.

Video

Slides

_{Reposted from the CustomInk Technology blog.}