Ansible for DevOps: Server and configuration management for humans (2015)

Chapter 6 - Playbook Organization - Roles and Includes

So far, we’ve used fairly straightforward examples in this book. Most examples are ad-hoc for a particular server, and listing all tasks in one long listing makes for a fairly long playbook.

Ansible is very flexible when it comes to organizing your tasks in more efficient ways so you can make your playbooks more maintainable, reusable, and powerful. We’ll look at two ways to split up tasks more efficiently: using includes and roles. Finally, we’ll explore Ansible Galaxy, a repository of some community-maintained roles that help configure common packages and applications.

Includes

We’ve already seen one of the most basic ways of including other files in Chapter 4, when vars_files was used to place variables into a separate vars.yml file instead of inline with the playbook:

- hosts: all

vars_files:

- vars.yml

Tasks can easily be included in a similar way. In the tasks: section of your playbook, you can add include directives like so:

tasks:

- include: included-playbook.yml

Just like with variable include files, tasks are formatted in a flat list in the included file. As an example, the included-playbook.yml could look like:

---

- name: Add profile info for user.

copy:

src: example_profile

dest: "/home/{{ username }}/.profile"

owner: "{{ username }}"

group: "{{ username }}"

mode: 0744

- name: Add private keys for user.

copy:

src: "{{ item.src }}"

dest: "/home/.ssh/{{ item.dest }}"

owner: "{{ username }}"

group: "{{ username }}"

mode: 0600

with_items: ssh_private_keys

- name: Restart example service.

service: name=example state=restarted

In this case, you’d probably want to name the file user-config.yml, since it’s used to configure a user account and restart some service. Now, in this and any other playbook that provisions or configures a server, if you want to configure a particular user’s account, add the following in your playbook’s tasks section:

- include: example-app-config.yml

We used {{ username }} and {{ ssh_private_keys }} variables in this include file instead of hard-coded values so we could make this include file reusable. You could define the variables in your playbook’s inline variables or an included variables file, but Ansible also lets you pass variables directly into includes using normal YAML syntax. For example:

- { include: user-config.yml, username: johndoe, ssh_private_keys: [] }

- { include: user-config.yml, username: janedoe, ssh_private_keys: [] }

To make the syntax more readable, you can use structured variables, like so:

- include: user-config.yml

vars:

username: johndoe

ssh_private_keys:

- { src: /path/to/johndoe/key1, dest: id_rsa }

- { src: /path/to/johndoe/key2, dest: id_rsa_2 }

- include: user-config.yml

vars:

username: janedoe

ssh_private_keys:

- { src: /path/to/janedoe/key1, dest: id_rsa }

- { src: /path/to/janedoe/key2, dest: id_rsa_2 }

Include files can even include other files, so you could have something like the following:

tasks:

- include: user-config.yml

inside user-config.yml

- include: ssh-setup.yml

Handler includes

Handlers can be included just like tasks, within a playbook’s handlers section. For example:

handlers:

- include: included-handlers.yml

This can be helpful in limiting the noise in your main playbook, since handlers are usually used for things like restarting services or loading a configuration, and can distract from the playbook’s primary purpose.

Playbook includes

Playbooks can even be included in other playbooks, using the same include syntax in the top level of your playbook. For example, if you have two playbooks—one to set up your webservers (web.yml), and one to set up your database servers (db.yml), you could use the following playbook to run both at the same time:

- hosts: all

remote_user: root

tasks:

...

- include: web.yml

- include: db.yml

This way, you can create playbooks to configure all the servers in your entire infrastructure, then create a master playbook that includes each of the individual playbooks. When you want to initialize your entire infrastructure, make changes across your entire fleet of servers, or just check to make sure their configuration matches your playbook definitions, you can run one ansible-playbook command!

Complete includes example

What if I told you we could remake the 137-line Drupal LAMP server playbook from Chapter 4 in just 21 lines? With includes, it’s easy; just break out each of the sets of tasks into their own include files, and you’ll end up with a main playbook like this:

1 ---

2 - hosts: all

3

4 vars_files:

5 - vars.yml

6

7 pre_tasks:

8 - name: Update apt cache if needed.

9 apt: update_cache=yes cache_valid_time=3600

10

11 handlers:

12 - include: handlers/handlers.yml

13

14 tasks:

15 - include: tasks/common.yml

16 - include: tasks/apache.yml

17 - include: tasks/php.yml

18 - include: tasks/mysql.yml

19 - include: tasks/composer.yml

20 - include: tasks/drush.yml

21 - include: tasks/drupal.yml

All you need to do is create two new folders in the same folder where you saved the Drupal playbook.yml file, handlers and tasks, then create files inside for each section of the playbook.

For example, inside handlers/handlers.yml, you’d have:

1 ---

2 - name: restart apache

3 service: name=apache2 state=restarted

And inside tasks/drush.yml:

1 ---

2 - name: Check out drush master branch.

3 git:

4 repo: https://github.com/drush-ops/drush.git

5 dest: /opt/drush

6

7 - name: Install Drush dependencies with Composer."

8 shell: >

9 /usr/local/bin/composer install

10 chdir=/opt/drush

11 creates=/opt/drush/vendor/autoload.php

12

13 - name: Create drush bin symlink.

14 file:

15 src: /opt/drush/drush

16 dest: /usr/local/bin/drush

17 state: link

Separating all the tasks into separate includes files means you’ll have more files to manage for your playbook, but it helps keep the main playbook more compact (meaning it’s easier to see all the installation and configuration steps the playbook contains), and also separates tasks into individual, easily-maintainable groupings. Instead of having to browse one playbook with twenty-three separate tasks, you now maintain eight included files with two to five tasks, each.

It’s much easier to maintain a more granular set of tasks than one very long playbook. However, there’s no reason to try to start writing a playbook with lots of individual includes. Most of the time, it’s best to start with a monolithic playbook while you’re working on the setup and configuration details, then move sets of tasks out to included files after you start seeing logical groupings.

You can also use tags (demonstrated in the previous chapter) to limit the playbook run to a certain include file. Using the above example, if you wanted to add a ‘drush’ tag to the included drush file (so you could run ansible-playbook playbook.yml --tags=drush and only run the drush tasks), you can change line 20 to the following:

20 - include: tasks/drush.yml tags=drush

Roles

Including playbooks inside other playbooks makes your playbook organization a little more sane, but once you start wrapping up your entire infrastructure’s configuration in playbooks, you might end up with something resembling Russian nesting dolls.

Wouldn’t it be nice if there were a way to take bits of related configuration, and package them together nicely? Additionally, what if we could take these packages (often configuring the same thing on many different servers) and make them flexible so we can use the same package throughout our infrastructure, with slightly different settings on individual servers or groups of servers?

Ansible Roles can do all that, and more!

Let’s dive into what makes an Ansible role by taking one of the playbook examples from Chapter 4 and splitting it into a more flexible structure using roles.

Role scaffolding

Instead of requiring you to explicitly include certain files and playbooks in a role, Ansible automatically includes any main.yml files inside specific directories that make up the role.

There are only two directories required to make a working Ansible role:

role_name/

meta/

tasks/

If you create a directory structure like the one shown above, with a main.yml file in each directory, Ansible will run all the tasks defined in tasks/main.yml if you call the role from your playbook using the following syntax:

1 ---

2 - hosts: all

3 roles:

4 - role_name

Your roles (each one as its own directory, where the directory name is used by Ansible as the name of the role) can live in a couple different places—in the default global Ansible role path (configurable in /etc/ansible/ansible.cfg), or in a ‘roles’ folder directly within the same directory as your main playbook file.

Building your first role

Let’s clean up the Node.js server example from Chapter four, and break out one of the main parts of the configuration—installing Node.js and any required npm modules.

Create a roles folder in the same directory as the main playbook.yml file like we created in Chapter 4’s first example, and inside that folder, create a new folder nodejs (which will be our role’s name). Create two folders inside the nodejs role directory, meta and tasks.

Inside the meta folder, add a simple main.yml file with the following contents:

1 ---

2 dependencies: []

The meta information for your role is defined in this file. In basic examples and simple roles, you just need to list any role dependencies (other roles that are required to be run before the current role can do its work), but you can add much more to this file to describe your role to Ansible and to Ansible Galaxy. We’ll dive deeper into the meta information later. For now, save the file and head over to the tasks foler.

Create a main.yml file in this folder, and add the following contents (basically copying and pasting the configuration from the Chapter 4 example):

1 ---

2 - name: Install Node.js (npm plus all its dependencies).

3 yum: name=npm state=present enablerepo=epel

4

5 - name: Install forever module (to run our Node.js app).

6 npm: name=forever global=yes state=latest

The Node.js directory structure should now look like the following:

1 nodejs-app/

2 app/

3 app.js

4 package.json

5 playbook.yml

6 roles/

7 nodejs/

8 meta/

9 main.yml

10 tasks/

11 main.yml

You now have a complete Ansible role that you can use in your node.js server configuration playbook. Delete the Node.js app installation lines from playbook.yml, and reformat the playbook so the other tasks run first (in a pre_tasks: section instead of tasks:), then the role is included, then the rest of the tasks (in the main tasks: section). Something like:

pre_tasks:

# EPEL/GPG setup, firewall configuration...

roles:

- nodejs

tasks:

# Node.js app deployment tasks...

Once you finish reformatting the main playbook, everything would run exactly the same during an ansible-playbook, with the exception of the tasks inside the nodejs role being prefixed with nodejs | [Task name here].

This little bit of extra data shown during playbook runs is useful because it automatically prefixes tasks with the role that provides them, without you having to add in descriptions as part of the name values of the tasks.

Our role isn’t all that helpful at this point, though, because it still does only one thing, and it’s not really flexible enough to be used on other servers that might need different Node.js modules to be installed.

More flexibility with role vars and defaults

To make our role more flexible, we can make it use a list of npm modules instead of a hardcoded value, then allow playbooks using the role to provide their own module list variable to override our role’s default list.

When running a role’s tasks, Ansible picks up variables defined in a role’s vars/main.yml file and defaults/main.yml (I’ll get to the differences between the two later), but will allow your playbooks to override the defaults or other role-provided variables if you want.

Modify the tasks/main.yml file to use a list variable and iterate through the list to install as many packages as your playbook wants:

1 ---

2 - name: Install Node.js (npm plus all its dependencies).

3 yum: name=npm state=present enablerepo=epel

4

5 - name: Install npm modules required by our app.

6 npm: name={{ item }} global=yes state=latest

7 with_items: node_npm_modules

Let’s provide a sane default for the new node_npm_modules variable in defaults/main.yml:

1 ---

2 node_npm_modules:

3 - forever

Now, if you run the playbook as-is, it will still do the exact same thing—install the forever module. But since the role is more flexible, we could create a new playbook like our first, but add a variable (either in a vars section or in an included file via vars_files) to override the default, like so:

1 node_npm_modules:

2 - forever

3 - async

4 - request

When you run the playbook with this custom variable (we didn’t change anything with our nodejs role), all three of the above npm modules will be installed.

Hopefully you’re beginning to see how this can be powerful!

Imagine if you had a playbook structure like:

1 ---

2 - hosts: appservers

3 roles:

4 - yum-repo-setup

5 - firewall

6 - nodejs

7 - app-deploy

Each one of the roles would live in its own isolated world, and could be shared with other servers and groups of servers in your infrastructure.

· A yum-repo-setup role could enable certain repositories and import their GPG keys.

· A firewall role could have per-server or per-inventory-group options for ports and services to allow or deny.

· An app-deploy role could deploy your app to a directory (configurable per-server) and set certain app options per-server or per-group.

All these things become very easy to manage when you have small bits of functionality separated into different roles. Instead of managing 100+ lines of playbook tasks, and manually prefixing every name: with something like “Common |” or “App Deploy |”, you now manage a few roles with 10-20 lines of YAML each.

On top of that, when you’re building your main playbooks, they can be extremely simple (like the above example), enabling you to see everything being configured and deployed on a particular server without scrolling through dozens of included playbook files and hundreds of tasks.

Other role parts: handlers, files, and templates

Handlers

In one of the prior examples, we introduced handlers—tasks that could be called via the notify option after any playbook task resulted in a change—and an example handler for restarting Apache was given:

1 handlers:

2 - name: restart apache

3 service: name=apache2 state=restarted

In Ansible roles, handlers are first-class citizens, alongside tasks, variables, and other configuration. You can store handlers directly inside a main.yml file inside a role’s handlers directory. So if we had a role for Apache configuration, our handlers/main.yml file could look like the following:

1 ---

2 - name: restart apache

3 command: service apache2 restart

You can call handlers defined in a role’s handlers folder just like you would handlers included directly in your playbooks (e.g. notify: restart apache).

Files and Templates

For the following examples, let’s assume our role is structured with files and templates inside files and templates directories, respectively:

1 roles/

2 example/

3 files/

4 example.conf

5 meta/

6 main.yml

7 templates/

8 example.xml.j2

9 tasks/

10 main.yml

when copying a file directly to the server, add the filename or the full path from within a role’s files directory, like so:

- name: Copy configuration file to server directly.

copy: >

src=example.conf

dest=/etc/myapp/example.conf

mode=644

Similarly, when specifying a template, add the filename or the full path from within a role’s templates directory, like so:

- name: Copy configuration file to server using a template.

template: >

src=example.xml.j2

dest=/etc/myapp/example.xml

mode=644

The copy module copies files from within the module’s files folder, and the template module runs given template files through the Jinja2 templating engine, merging in any variables available during your playbook run before copying the file to the server.

Organizing more complex and cross-platform roles

For simple package installation and configuration roles, you can get by with placing all tasks, variables, and handlers directly in the respective main.yml file Ansible automatically loads. But you can also include other files from within a role’s main.yml files if needed.

As a rule of thumb, I like to keep my playbook and role task files under 100 lines of YAML if at all possible (that way it’s easier for me to keep the entire set of tasks in my head while working through any issues). If I start nearing that limit, I usually split the tasks into logical groupings, and include files from the main.yml file.

Let’s take a look at the way my geerlingguy.apache role is set up (it’s available on Ansible Galaxy and can be downloaded to your roles directory with the command ansible-galaxy install geerlingguy.apache; we’ll discuss Ansible Galaxy itself later).

Initially, the role’s main tasks/main.yml file looked something like the following (generally speaking):

1 - name: Ensure Apache is installed (via apt).

2

3 - name: Configure Apache with lineinfile.

4

5 - name: Enable Apache modules.

Soon after creating the role, though, I wanted to make the role work with both Debian and RedHat hosts. I could’ve added two sets of tasks in the main.yml file, resulting in twice the number of tasks and a bunch of extra whenstatements:

1 - name: Ensure Apache is installed (via apt).

2 when: ansible_os_family == 'Debian'

3

4 - name: Ensure Apache is installed (via yum).

5 when: ansible_os_family == 'RedHat'

6

7 - name: Configure Apache with lineinfile (Debian).

8 when: ansible_os_family == 'Debian'

9

10 - name: Configure Apache with lineinfile (Redhat).

11 when: ansible_os_family == 'RedHat'

12

13 - name: Enable Apache modules (Debian).

14 when: ansible_os_family == 'Debian'

15

16 - name: Other OS-agnostic tasks...

If I had gone this route, and continued with the rest of the playbook tasks in one file, I would’ve quickly surpassed my informal 100-line limit. So I chose to use includes in my main tasks file:

1 - name: Include OS-specific variables.

2 include_vars: "{{ ansible_os_family }}.yml"

3

4 - include: setup-RedHat.yml

5 when: ansible_os_family == 'RedHat'

6

7 - include: setup-Debian.yml

8 when: ansible_os_family == 'Debian'

9

10 - name: Other OS-agnostic tasks...

Two important things to notice about this style of distribution-specific inclusion:

1. When including vars files (with include_vars), you can actually use variables in the name of the file. This is very handy for a variety of use cases, and here we’re including a vars file in the format distribution_name.yml. For our purposes, since the role will be used on Debian and RedHat-based hosts, we can create Debian.yml and RedHat.yml files in our role’s defaults and vars folders, and put distribution-specific variables there.

2. When including playbook files (with include), you can’t use variables in the name of the file, but you can do the next best thing: include the files by name explicitly, and use a condition to tell Ansible whether to run the tasks inside (the when condition will be applied to every task inside the included playbook).

After setting things up this way, I put RedHat and CentOS-specific tasks (like yum tasks) into tasks/setup-RedHat.yml, and Debian and Ubuntu-specific tasks (like apt tasks) into tasks/setup-Debian.yml. There are other ways of making roles work cross-platform, but using distribution-specific variables files and included playbooks is one of the simplest.

Now this Apache role can be used across different distributions, and with clever usage of variables in tasks and in configuration templates, it can be used in a very wide variety of infrastructure that needs Apache installed.

Ansible Galaxy

Ansible roles are powerful and flexible; they allow you to encapsulate sets of configuration and deployable units of playbooks, variables, templates, and other files, so you can easily reuse them across different servers.

It’s annoying to have to start from scratch every time, though; wouldn’t it be better if people could share roles for commonly-installed applications and services? Enter Ansible Galaxy.

Ansible Galaxy, or just ‘Galaxy’, is a repository of community-contributed roles for common Ansible content. There are already hundreds of roles available which can configure and deploy common applications, and they’re all available through the ansible-galaxy command, introduced in Ansible 1.4.2.

Galaxy offers the ability to add, download, and rate roles, and you can register either using a social account or a normal account on the site (though you don’t need an account to install and use roles from Galaxy).

Getting roles from Galaxy

One of the primary functions of the ansible-galaxy command is retrieving roles from Galaxy. Roles must be downloaded before they can be used in playbooks.

Remember the basic LAMP (Linux, Apache, MySQL and PHP) server we installed earlier in the book? Let’s create it again, but this time, using a few roles from Galaxy:

$ ansible-galaxy install geerlingguy.apache geerlingguy.mysql geerlingguy.php

Using role requirements files to manage dependencies

If your infrastructure configuration requires have five, ten, fifteen or more Ansible roles, installing them all via ansible-galaxy install commands can get very tiring. Additionally, if you host roles internally (e.g. via an internal Git repository), you can’t install the roles through Ansible Galaxy. The ansible-galaxy command, however, can be passed a ‘requirements’ file that it will parse and use to automatically download all the included dependencies—whether on Ansible Galaxy or in some other role repository.

There are two simple syntaxes, the first which follows Python’s pip requirements convention. Generally, you can create a requirements.txt file (I usually add one in the root of my project directory), and include a line for each role, using the same syntax we used in the install command earlier:

1 geerlingguy.apache,1.3.1

2 geerlingguy.mysql

3 geerlingguy.php,1.4.1

Specify one Ansible Galaxy role per line either by itself, or with a particular version number, and then run ansible-galaxy install -r requirements.txt. Ansible will download all the roles from Galaxy into your local roles path.

If you want to be able to install roles from other sources, like GitHub, an HTTP download, BitBucket, etc., or if you’d like to specify the path into which the roles should be downloaded, you can use a requirements.yml file with the following syntax:

1 # From Ansible Galaxy, like the earlier requirements.txt example.

2 - src: geerlingguy.firewall

3

4 # From GitHub, into a particular path, with a custom name and version.

5 - src: https://github.com/geerlingguy/ansible-role-passenger

6 path: /etc/ansible/roles/

7 name: passenger

8 version: 1.0.2

9

10 # From a web server, with a custom name.

11 - src: https://www.example.com/ansible/roles/my-role-name.tar.gz

12 name: my-role

Installing roles defined in this file is similar to the .txt example: ansible-galaxy install -r requirements.yml. For more documentation on Ansible requirements files, read the official documentation: Advanced Control over Role Requirements Files.

A LAMP server in six lines of YAML

With the Apache, MySQL, and PHP roles installed, we can quickly create a LAMP server. This example assumes you already have a CentOS-based linux VM or server booted and can connect to it or run Ansible as a provisioner via Vagrant on it, and that you’ve run the ansible-galaxy install command above to download the required roles.

First, create an Ansible playbook named lamp.yml with the following contents:

1 ---

2 - hosts: all

3 roles:

4 - geerlingguy.mysql

5 - geerlingguy.apache

6 - geerlingguy.php

Now, run the playbook against a host:

$ ansible-playbook -i path/to/custom-inventory lamp.yml

After a few minutes, an entire LAMP server should be set up and running. If you add in a few variables, you can configure virtualhosts, PHP configuration options, MySQL server settings, etc.

We’ve effectively reduced about thirty lines of YAML (from previous examples dealing with LAMP or LAMP-like servers) down to three. Obviously, the roles have extra code in them, but the power here is in abstraction. Since most companies have many servers using similar software, but with slightly different configurations, having centralized, flexible roles saves a lot of repetition.

You could think of Galaxy roles (which typically install and configure common software like Apache or MySQL, or set up security rules or frameworks) as glorified packages; they not only install software, but they configure it exactly how you want it, every time, with minimal manual labor. Additionally, many of these roles work across different flavors of Linux and UNIX, so you have better configuration portability!

A Solr server in six lines of YAML

Let’s grab a few more roles and build an Apache Solr search server, which requires Java and Apache Tomcat to be installed and configured.

$ ansible-galaxy install geerlingguy.java geerlingguy.tomcat6 geerlingguy.solr

Then create a playbook named solr.yml with the following contents:

1 ---

2 - hosts: all

3 roles:

4 - geerlingguy.java

5 - geerlingguy.tomcat6

6 - geerlingguy.solr

Now we have a fully-functional Solr server, and we could add some variables to configure it exactly how we want, by using a non-default port, or changing the memory allocation for Tomcat6.

I think you might get the point. Now, I could’ve also left out the java and tomcat6 roles, since they’ll be automatically picked up during installation of the geerlingguy.solr role (they’re listed in the solr role’s dependencies).

A role’s page on the Ansible Galaxy website highlights available variables for setting things like what version of Solr to install, where to install it, etc. (as an example, view the geerlingguy.solr Galaxy page).

Using community-maintained roles, you can build a wide variety of servers with minimal effort. Instead of having to maintain lengthy playbooks and roles unique to each server, Galaxy lets you build a list of the required roles, and a few variables that set up the servers with the proper versions and paths. Configuration management with Ansible Galaxy becomes true configuration management—you get to spend more time managing your server’s configuration, and less time on packaging and building individual services!

Helpful Galaxy commands

Some other helpful ansible-galaxy commands you might use from time to time:

· ansible-galaxy list displays a list of installed roles, with version numbers

· ansible-galaxy remove [role] removes an installed role

· ansible-galaxy init can be used to create a role template suitable for submission to Ansible Galaxy

You can configure the default path where Ansible roles will be downloaded by editing your ansible.cfg configuration file (normally located in /etc/ansible/ansible.cfg), and setting a roles_path in the [defaults] section.

Contributing to Ansible Galaxy

If you’ve been working on some useful Ansible roles, and you’d like to share them with others, all you need to do is make sure they follow Ansible Galaxy’s basic template (especially within the meta/main.yml and README.md files). To get started, use ansible-galaxy init to generate a basic Galaxy template, and make your own role match the Galaxy template’s structure.

Then push your role up to a new project on GitHub (I usually name my Galaxy roles like ansible-role-[rolename], so I can easily see them when browsing my repos on GitHub), and add a new role while logged into galaxy.ansible.com.

Summary

Using includes and Ansible roles organizes Playbooks and makes them maintainable. This chapter introduced different ways of using include, the power and flexible structure of roles, and how you can utilize Ansible Galaxy, the community repository of configurable Ansible roles that do just about anything.

_________________________________________

/ When the only tool you own is a hammer, \

| every problem begins to resemble a |

\ nail. (Abraham Maslow) /

-----------------------------------------

\ ^__^

\ (oo)\_______

(__)\ )\/\

||----w |

|| ||