Reliably Deploying Rails Applications: Hassle free provisioning, reliable deployment (2014)

3.0 - Chef Definitions

Introduction

The ‘simplest’ way to provision a new server is to create a new VPS on something like Linode or Digital Ocean, login via SSH and start apt-get’ing the packages you need, dropping into vim to tweak config files and adding a few custom package sources where newer versions are needed.

When something new is needed you ssh back in, install or upgrade a package, building the server up in layers. Somewhere there’s a text file or wiki page with “all the commands you need” written down should you ever need to do it again.

This approach has a few key pitfalls;

1) It’s very hard to keep track of what you’ve done. With the best will in the world the text file doesn’t quite get all of the commands. Nobody realises until the time comes to provision a new server (often under adverse conditions) and running all the commands in the file doesn’t quite yield a working server.

2) It’s slow (and therefore expensive). Even if the process is perfectly documented, you still need someone to sit and run the commands. This is repetitive, boring work, but will often need to be done by an engineer whose time could be better spent working on the product itself.

3) It doesn’t scale. Having an engineer type in a list of commands might, with a lot of discipline, hold together when there’s only one server involved. Expand that to five or six servers and the cracks will soon start to show.

Automation

The goal of this section and of this book as a whole, is to take the manual processes and automate them. As a general rule, any long process which I’d expect to repeat more than once or twice a year in the life-cycle of deploying and managing I try and automate.

Automation relates to an approach more than it relates to any particular tool. If you’re doing anything which involves running more than one command or sshing into a remote server more than once in a month, it’s probably worth stopping and thinking, “how can I automate this?”

The benefits are often visible after a remarkably short period of time. Automating server deployments not only makes disaster recovery easier, it makes the creation of accurate test and staging environments easier, making the testing of new deployments easier and more efficient and so decreasing downtime.

Automating the copying of production databases from production to development and staging environments makes it more likely developers will use current data in their tests. This makes tests in development more meaningful and so increases productivity and decreases costs.

Finally automation is usually easier than you think. Once you’ve got your head around chef for automating provisioning tasks and Capistrano for automating your deployment process and subsequent interactions with production and staging servers, it becomes clear that the time taken to automate additional tasks is rarely significantly more than the time taken to perform them once.

The Tools for automating provisioning

Chef & Chef Solo

Chef is an automation platform made by Opscode which uses a ruby DSL (Domain Specific Language) to represent the commands required to provision a server in a reusable format.

With chef you can define the steps required to configure a server to fulfill a “role,” for example a rails application server or a database server and then apply combinations of these roles to a particular remote machine.

Chef is often run in a hub and spoke style arrangement. A central chef server “knows” the roles that a large number of other servers should have applied to them. If you update the role, the changes are applied to all of those servers automatically.

While this is an excellent and very powerful configuration when managing 10’s or 100’s of servers, it’s overly complicated if we’re just looking at managing at managing 1-10 servers.

Luckily it can also be run in a “solo” configuration (chef-solo). Here we use our local development workstation to define server roles and configurations and then manually apply these configurations to servers as and when we need to.

This is perfect for small projects running all parts of the stack on a single box but I’ve also used it with great success on setups consisting of up to 10 related servers.

If you’re project does grow beyond that, almost all of the work done now to automate with chef solo, will be re-usable with chef server.

Knife & Knife Solo

Knife is the command line tool that provides the interface between a local (on our development machine) chef repository and a remote server.

Traditionally this remote server would be the master “chef server” but an additional tool “knife solo” allows us to use chef in solo mode and interact directly with the server we’re trying to provision.

More information about knife solo is available on its Github page

https://github.com/matschaffer/knife-solo

and full usage instructions are included in the next chapter.

Berkshelf

The commands to install an individual component on a system are called a “recipe” in chef terminology. For example you may have a recipe for installing ruby and another recipe for installing common rails gem dependencies. Several recipes relating to a particular piece of functionality (for example MySQL server and MySQL client) will often be bundled together in a Cookbook.

Berkshelf is like Bundler for these recipes. You define the recipes your configuration is dependent on, for example the “rbenv” recipe and the “rails_gem_dependencies” recipe. In the same way “bundle install” goes out and gets your gem dependencies (including specific versions), Berkshelf will retrieve the chef recipes (including specific versions) that your server configuration is dependent on.