Ansible: Up and Running (2015)
Chapter 6. Deploying Mezzanine with Ansible
It’s time to write an Ansible playbook to deploy Mezzanine to a server. We’ll go through it step by step, but if you’re the type of person that starts off by reading the last page of a book to see how it ends,1 you can find the full playbook at the end of this chapter as Example 6-27. It’s also available on GitHub. Check out the README before trying to run it directly.
I’ve tried to hew as closely as possible to the original Fabric scripts that Mezzanine author Stephen McDonald wrote.2
Listing Tasks in a Playbook
Before we dive into the guts of our playbook, let’s get a high-level view. The ansible-playbook command-line tool supports a flag called --list-tasks. This flag will print out the names of all of the tasks in a playbook. It’s a handy way to summarize what a playbook is going to do. Here’s how you use it:
$ ansible-playbook --list-tasks mezzanine.yml
Example 6-1 shows the output for the mezzanine.yml playbook in Example 6-27.
Example 6-1. List of tasks in Mezzanine playbook
playbook: mezzanine.yml
play #1 (Deploy mezzanine on vagrant):
install apt packages
check out the repository on the host
install required python packages
install requirements.txt
create a user
create the database
generate the settings file
sync the database, apply migrations, collect static content
set the site id
set the admin password
set the gunicorn config file
set the supervisor config file
set the nginx config file
enable the nginx config file
remove the default nginx config file
ensure config path exists
create tls certificates
install poll twitter cron job
Organization of Deployed Files
As we discussed earlier, Mezzanine is built atop Django. In Django, a web app is called a project. We get to choose what to name our project, and I’ve chosen to name it mezzanine-example.
Our playbook deploys into a Vagrant machine, and will deploy the files into the home directory of the Vagrant user’s account.
/home/vagrant/mezzanine-example is the top-level directory we’ll deploy into. It also serves as the virtualenv directory, which means that we’re going to install all of the Python packages into that directory.
home/vagrant/mezzanine-example/project will contain the source code that will be cloned from a source code repository on GitHub.
Variables and Secret Variables
As you can see from Example 6-2, this playbook defines quite a few variables.
Example 6-2. Defining the variables
vars:
user: "{{ ansible_ssh_user }}"
proj_name: mezzanine-example
venv_home: "{{ ansible_env.HOME }}"
venv_path: "{{ venv_home }}/{{ proj_name }}"
proj_dirname: project
proj_path: "{{ venv_path }}/{{ proj_dirname }}"
reqs_path: requirements.txt
manage: "{{ python }} {{ proj_path }}/manage.py"
live_hostname: 192.168.33.10.xip.io
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.io
repo_url: git@github.com:lorin/mezzanine-example.git
gunicorn_port: 8000
locale: en_US.UTF-8
# Variables below don't appear in Mezzanine's fabfile.py
# but I've added them for convenience
conf_path: /etc/nginx/conf
tls_enabled: True
python: "{{ venv_path }}/bin/python"
database_name: "{{ proj_name }}"
database_user: "{{ proj_name }}"
database_host: localhost
database_port: 5432
gunicorn_proc_name: mezzanine
vars_files:
- secrets.yml
I’ve tried for the most part to use the same variable names that the Mezzanine Fabric script uses. I’ve also added some extra variables to make things a little clearer. For example, the Fabric scripts directly use proj_name as the database name and database username. I prefer to define intermediate variables named database_name and database_user and define these in terms of proj_name.
A couple of things to note here. First of all, note how we can define one variable in terms of another. For example, we define venv_path in terms of venv_home and proj_name.
Also, note how we can reference Ansible facts in these variables. For example, venv_home is defined in terms of the ansible_env fact that is collected from each host.
Finally, note how we have specified some of our variables in a separate file, called secrets.yml, by doing this:
vars_files:
- secrets.yml
This file contains credentials such as passwords and tokens that need to remain private. Note that my repository on GitHub does not actually contain this file. Instead, it contains a file called secrets.yml.example that looks like this:
db_pass: e79c9761d0b54698a83ff3f93769e309
admin_pass: 46041386be534591ad24902bf72071B
secret_key: b495a05c396843b6b47ac944a72c92ed
nevercache_key: b5d87bb4e17c483093296fa321056bdc
# You need to create a Twitter application at https://dev.twitter.com
# in order to get the credentials required for Mezzanine's
# twitter integration.
#
# See http://mezzanine.jupo.org/docs/twitter-integration.html
# for details on Twitter integration
twitter_access_token_key: 80b557a3a8d14cb7a2b91d60398fb8ce
twitter_access_token_secret: 1974cf8419114bdd9d4ea3db7a210d90
twitter_consumer_key: 1f1c627530b34bb58701ac81ac3fad51
twitter_consumer_secret: 36515c2b60ee4ffb9d33d972a7ec350a
To use this repo, you need to copy secrets.yml.example to secrets.yml and edit it so that it contains the credentials specific to your site. Also note that secrets.yml is included in the .gitignore file in the Git repository to prevent someone from accidentally committing these credentials.
It’s best to avoid committing unencrypted credentials into your version control repository because of the security risks involved. This is just one possible strategy for maintaining secret credentials. We also could have passed them as environment variables. Another option, which we will describe in Chapter 7, is to commit an encrypted version of the secrets.yml file using Ansible’s vault functionality.
Using Iteration (with_items) to Install Multiple Packages
We’re going to need to install two different types of packages for our Mezzanine deployment. We need to install some system-level packages, and because we’re going to deploy on Ubuntu, we’re going to use apt as our package manager for the system packages. We also need to install some Python packages, and we’ll use pip for the Python packages.
System-level packages are generally easier to deal with than Python packages, because system-level packages are designed specifically to work with the operating system. However, the system package repositories often don’t have the newest versions of the Python libraries we need, so we turn to the Python packages to install those. It’s a trade-off between stability versus running the latest and greatest.
Example 6-3 shows the task we’ll use to install the system packages.
Example 6-3. Installing system packages
- name: install apt packages
apt: pkg={{ item }} update_cache=yes cache_valid_time=3600
sudo: True
with_items:
- git
- libjpeg-dev
- libpq-dev
- memcached
- nginx
- postgresql
- python-dev
- python-pip
- python-psycopg2
- python-setuptools
- python-virtualenv
- supervisor
There’s a lot to unpack here. Because we’re installing multiple packages, we use Ansible’s iteration functionality, the with_items clause. We could have installed the packages one at a time, like this:
- name: install git
apt: pkg=git
- name: install libjpeg-dev
apt: pkg=libjpeg-dev
...
However, it’s much simpler to write and read if we group all of the packages together in a list. When we invoke the apt module, we pass it {{ item }}. This is a placeholder variable that will be populated by each of the elements in the list of the with_items clause.
NOTE
Ansible always uses item as the name of the loop iteration variable.
In addition, in the case of the apt module, it’s more efficient to install multiple packages using the with_items clause. That’s because Ansible will pass the entire list of packages to the apt module, and the module will invoke the apt program only once, passing it the entire list of packages to be installed. Some modules, like apt, have been designed to handle lists intelligently like this. If a module doesn’t have native support for lists, then Ansible will simply invoke the module multiple times, once for each element of the list.
You can tell that the apt module is intelligent enough to handle multiple packages at once, because the output looks like this:
TASK: [install apt packages] **************************************************
ok: [web] => (item=git,libjpeg-dev,libpq-dev,memcached,nginx,postgresql,
python-dev,python-pip,python-psycopg2,python-setuptools,python-virtualenv,
supervisor)
On the other hand, the pip module does not handle lists intelligently, so Ansible must invoke it once for each element of the list, and the output looks like this:
TASK: [install other python packages] *****************************************
ok: [web] => (item=gunicorn)
ok: [web] => (item=setproctitle)
ok: [web] => (item=south)
ok: [web] => (item=psycopg2)
ok: [web] => (item=django-compressor)
ok: [web] => (item=python-memcached)
Adding the Sudo Clause to a Task
In the playbook examples of Chapter 2, we wanted the whole playbook to run as root, so we added the sudo: True clause to the play.
When we deploy Mezzanine, most of the tasks will be run as the user who is SSHing to the host, rather than root. Therefore, we don’t want to run as sudo the entire play, just select tasks.
We can accomplish this by adding sudo: True to the tasks that do need to run as root, such as Example 6-3.
Updating the Apt Cache
NOTE
All of the example commands in this subsection are run on the (Ubuntu) remote host, not the control machine.
Ubuntu maintains a cache with the names of all of the Apt packages that are available in the Ubuntu package archive. Let’s say you try to install the package named libssl-dev. We can use the apt-cache program to query the local cache to see what version it knows about:
$ apt-cache policy libssl-dev
The output is shown in Example 6-4.
Example 6-4. apt cache output
libssl-dev:
Installed: (none)
Candidate: 1.0.1f-1ubuntu2.5
Version table:
1.0.1f-1ubuntu2.5 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1.0.1f-1ubuntu2 0
500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
As we can see, this package is not installed locally. According to the local cache, the latest version is 1.0.1f-ubuntu2.5. We also see some information about the location of the package archive.
In some cases, when the Ubuntu project releases a new version of a package, it removes the old version from the package archive. If the local apt cache of an Ubuntu server hasn’t been updated, then it will attempt to install a package that doesn’t exist in the package archive.
To continue with our example, let’s say we were to attempt to install the libssl-dev package:
$ apt-get install libssl-dev
If version 1.0.1f-ubuntu2.5 were no longer available in the package archive, we’d see the following error:
Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main libssl-dev amd64
1.0.1f-1ubuntu2.5
404 Not Found [IP: 91.189.88.153 80]
Err http://security.ubuntu.com/ubuntu/ trusty-security/main libssl-dev amd64
1.0.1f-1ubuntu2.5
404 Not Found [IP: 91.189.88.149 80]
Err http://security.ubuntu.com/ubuntu/ trusty-security/main libssl-doc all
1.0.1f-1ubuntu2.5
404 Not Found [IP: 91.189.88.149 80]
E: Failed to fetch
http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl-dev_1.0.1f-1ubuntu2.
5_amd64.deb
404 Not Found [IP: 91.189.88.149 80]
E: Failed to fetch
http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl-doc_1.0.1f-1ubuntu2.
5_all.deb
404 Not Found [IP: 91.189.88.149 80]
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
On the command line, the way to bring the local apt cache up-to-date is to run apt-get update. When using the apt Ansible module, the way to bring the local apt cache up-to-date is to pass the update_cache=yes argument when invoking the module, as shown in Example 6-3.
Because updating the cache takes some additional time, and because we might be running a playbook multiple times in quick succession in order to debug it, we can avoid paying the cache update penalty by using the cache_valid_time argument to the module. This instructs to update the cache only if it’s older than some threshold. The example in Example 6-3 uses cache_valid_time=3600, which updates the cache only if it’s older than 3,600 seconds (1 hour).
Checking Out the Project Using Git
Although Mezzanine can be used without writing any custom code, one of its strengths is that it is written on top of the Django platform, and Django’s a great web application platform if you know Python. If you just wanted a CMS, you’d likely just use something like WordPress. But if you’re writing a custom application that incorporates CMS functionality, Mezzanine is a good way to go.
As part of the deployment, you need to check out the Git repository that contains your Django applications. In Django terminology, this repository must contain a project. I’ve created a repository on GitHub that contains a Django project with the expected files. That’s the project that gets deployed in this playbook.
I created these files using the mezzanine-project program that ships with Mezzanine, like this:
$ mezzanine-project mezzanine-example
Note that I don’t have any custom Django applications in my repository, just the files that are required for the project. In a real Django deployment, this repository would contain subdirectories that contain additional Django applications.
Example 6-5 shows how we use the git module to check out a Git repository onto a remote host.
Example 6-5. Checking out the Git repository
- name: check out the repository on the host
git: repo={{ repo_url }} dest={{ proj_path }} accept_hostkey=yes
I’ve made the project repository public so that readers can access it, but in general, you’ll be checking out private Git repositories over SSH. For this reason, I’ve set the repo_url variable to use the scheme that will clone the repository over SSH:
repo_url: git@github.com:lorin/mezzanine-example.git
If you’re following along at home, to run this playbook you must:
1. Have a GitHub account.
2. Have a public SSH key associated with your GitHub account.
3. Have an SSH agent running on your control machine with agent forwarding enabled.
To enable agent forwarding, add the following to your ansible.cfg:
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes
In addition to specifying the repository URL with the repo parameter and the destination path of the repository as the dest parameter, we also pass an additional parameter, accept_hostkey, which is related to host key checking. We discuss SSH agent forwarding and host key checking in more detail in Appendix A.
Installing Mezzanine and Other Packages into a virtualenv
As mentioned earlier in this chapter, we’re going to install some of the packages as Python packages because we can get more recent versions of those than if we installed the equivalent apt package.
We can install Python packages systemwide as the root user, but it’s better practice to install these packages in an isolated environment to avoid polluting the system-level Python packages. In Python, these types of isolated package environments are called virtualenvs. A user can create multiple virtualenvs, and can install Python packages into a virtualenv without needing root access.
Ansible’s pip module has support for installing packages into a virtualenv and for creating the virtualenv if it is not available. Example 6-6 shows the two tasks that we use to install Python packages into the virtualenv, both of which use the pip module, although in different ways.
Example 6-6. Install Python packages
- name: install required python packages
pip: name={{ item }} virtualenv={{ venv_path }}
with_items:
- gunicorn
- setproctitle
- south
- psycopg2
- django-compressor
- python-memcached
- name: install requirements.txt
pip: requirements={{ proj_path }}/{{ reqs_path }} virtualenv={{ venv_path }}
A common pattern in Python projects is to specify the package dependencies in a file called requirements.txt. And, indeed, the repository in our Mezzanine example contains a requirements.txt file. It looks like Example 6-7.
Example 6-7. requirements.txt
Mezzanine==3.1.10
The requirements.txt file is missing several other Python packages that we need for the deployment, so we explicitly specify these as a separate task.
Note that the Mezzanine Python package in requirements.txt is pinned to a specific version (3.1.10), where the other packages aren’t pinned; we just grab the latest versions of those. If we did not want to pin Mezzanine, we simply could have added Mezzanine to the list of packages, like this:
- name: install python packages
pip: name={{ item }} virtualenv={{ venv_path }}
with_items:
- mezzanine
- gunicorn
- setproctitle
- south
- psycopg2
- django-compressor
- python-memcached
Alternately, if we wanted to pin all of the packages, we have several options. We could have created a requirements.txt file. This file contains information about the packages and the dependencies. An example file looks like Example 6-8.
Example 6-8. Example requirements.txt
Django==1.6.8
Mezzanine==3.1.10
Pillow==2.6.1
South==1.0.1
argparse==1.2.1
beautifulsoup4==4.1.3
bleach==1.4
django-appconf==0.6
django-compressor==1.4
filebrowser-safe==0.3.6
future==0.9.0
grappelli-safe==0.3.13
gunicorn==19.1.1
html5lib==0.999
oauthlib==0.7.2
psycopg2==2.5.4
python-memcached==1.53
pytz==2014.10
requests==2.4.3
requests-oauthlib==0.4.2
setproctitle==1.1.8
six==1.8.0
tzlocal==1.0
wsgiref==0.1.2
If you have an existing virtualenv with the packages installed, you can use the pip freeze command to print out a list of installed packages. For example, if your virtualenv is in ~/mezzanine_example, you can activate your virtualenv and print out the packages in the virtualenv like this:
$ source ~/mezzanine_example/bin/activate
$ pip freeze > requirements.txt
Example 6-9 shows how we could have installed the packages using a requirements.txt file if we had one.
Example 6-9. Installing from requirements.txt
- name: copy requirements.txt file
copy: src=files/requirements.txt dest=~/requirements.txt
- name: install packages
pip: requirements=~/requirements.txt virtualenv={{ venv_path }}
Alternatively, we could have specified both the package names and their versions in the list, as shown in Example 6-10. We pass a list of dictionaries, and dereference the elements with item.name and item.version.
Example 6-10. Specifying package names and version
- name: python packages
pip: name={{ item.name }} version={{ item.version }} virtualenv={{ venv_path }}
with_items:
- {name: mezzanine, version: 3.1.10 }
- {name: gunicorn, version: 19.1.1 }
- {name: setproctitle, version: 1.1.8 }
- {name: south, version: 1.0.1 }
- {name: psycopg2, version: 2.5.4 }
- {name: django-compressor, version: 1.4 }
- {name: python-memcached, version: 1.53 }
Complex Arguments in Tasks: A Brief Digression
Up until this point in the book, every time we have invoked a module, we have passed the argument as a string. Taking the pip example from Example 6-10, we passed the pip module a string as an argument:
- name: install package with pip
pip: name={{ item.name }} version={{ item.version }} virtualenv={{ venv_path }}
If we don’t like long lines in our files, we could have broken up the argument string across multiple lines using YAML’s line folding, which we originally wrote about in “Line Folding”:
- name: install package with pip
pip: >
name={{ item.name }}
version={{ item.version }}
virtualenv={{ venv_path }}
Ansible also provides us with another option for breaking up a module invocation across multiple lines. Instead of passing a string, we can pass a dictionary where the keys are the variable names. This means we could have invoked Example 6-10 like this instead:
- name: install package with pip
pip:
name: "{{ item.name }}"
version: "{{ item.version }}"
virtualenv: "{{ venv_path }}"
The dictionary-based approach to passing arguments is also useful when invoking modules that take complex arguments. A complex argument is an argument to a module that is a list or a dictionary. The ec2 module, which creates new servers on Amazon EC2 cloud, is a good example of a module that takes complex arguments. Example 6-11 shows how to call a module that takes a list as an argument for the group parameter, and a dictionary as an argument to the instance_tags parameter. We’ll cover this module in more detail in Chapter 12.
Example 6-11. Calling a module with complex arguments
- name: create an ec2 instance
ec2:
image: ami-8caa1ce4
instance_type: m3.medium
key_name: mykey
group:
- web
- ssh
instance_tags:
type: web
env: production
You can even mix it up by passing some arguments as a string and others as a dictionary, by using the args clause to specify some of the variables as a dictionary. We could rewrite our preceding example as:
- name: create an ec2 instance
ec2: image=ami-8caa1ce4 instance_type=m3.medium key_name=mykey
args:
group:
- web
- ssh
instance_tags:
type: web
env: production
If you’re using the local_action clause (we’ll cover this in more detail in Chapter 7), then the syntax for complex args changes slightly. You need to add module: <modulename> as shown below:
- name: create an ec2 instance
local_action:
module: ec2
image: ami-8caa1ce4
instance_type: m3.medium
key_name: mykey
group:
- web
- ssh
instance_tags:
type: web
env: production
You can also mix simple arguments and complex arguments when using local_action:
- name: create an ec2 instance
local_action: ec2 image=ami-8caa1ce4 instance_type=m3.medium key_name=mykey
args:
image: ami-8caa1ce4
instance_type: m3.medium
key_name: mykey
group:
- web
- ssh
instance_tags:
type: web
env: production
WARNING
Ansible allows you to specify file permissions, which are used by several modules, including file, copy, and template. If you are specifying an octal value as a complex argument, it must either start the value with a 0 or quote it as a string.
For example, note how the mode argument starts with a +0:
- name: copy index.html
copy:
src: files/index.html
dest: /usr/share/nginx/html/index.html
mode: "0644"
If you do not start the mode argument with a 0 or quote it as a string, Ansible will interpret the value as a decimal number instead of an octal, and will not set the file permissions the way you expect. For details, see GitHub.
If you want to break your arguments across multiple lines, and you aren’t passing complex arguments, which form you choose is a matter of taste. I generally prefer dictionaries to multiline strings, but in this book I use both forms.
Creating the Database and Database User
When Django runs in development mode, it uses the SQLite backend. This backend will create the database file if the file does not exist.
When using a database management system like Postgres, we need to first create the database inside of Postgres and then create the user account that owns the database. Later on, we will configure Mezzanine with the credentials of this user.
Ansible ships with the postgresql_user and postgresql_db modules for creating users and databases inside of Postgres. Example 6-12 shows how we invoke these modules in our playbook.
Example 6-12. Creating the database and database user
- name: create a user
postgresql_user:
name: "{{ database_user }}"
password: "{{ db_pass }}"
sudo: True
sudo_user: postgres
- name: create the database
postgresql_db:
name: "{{ database_name }}"
owner: "{{ database_user }}"
encoding: UTF8
lc_ctype: "{{ locale }}"
lc_collate: "{{ locale }}"
template: template0
sudo: True
sudo_user: postgres
Note the use of sudo: True and sudo_user: postgres on each of these tasks. When you install Postgres on Ubuntu, the installation process creates a user named postgres that has administrative privileges for the Postgres installation. Note that the root account does not have administrative privileges in Postgres by default, so in the playbook, we need to sudo to the Postgres user in order to perform administrative tasks, such as creating users and databases.
When we create the database, we set the encoding (UTF8) and locale categories (LC_CTYPE, LC_COLLATE) associated with the database. Because we are setting locale information, we use template0 as the template.3
Generating the local_settings.py File from a Template
Django expects to find project-specific settings in a file called settings.py. Mezzanine follows the common Django idiom of breaking up these settings into two groups:
§ Settings that are the same for all deployments (settings.py)
§ Settings that vary by deployment (local_settings.py)
We define the settings that are the same for all deployments in the settings.py file in our project repository. You can find that file on GitHub.
As shown in Example 6-13, the settings.py file contains a Python snippet that loads a local_settings.py file. Django will raise an exception if the local_settings.py file does not exist.
Example 6-13. Loading the local settings
try:
from local_settings import *
except ImportError as e:
if "local_settings" not instr(e):
raise e
In addition, the .gitignore file is configured to ignore the local_settings.py file, since developers will commonly create this file and configure it for local development.
As part of our deployment, we need to create a local_settings.py file and upload it to the remote host. Example 6-14 shows the Jinja2 template that we use.
Example 6-14. local_settings.py.j2
from __future__ import unicode_literals
SECRET_KEY = "{{ secret_key }}"
NEVERCACHE_KEY = "{{ nevercache_key }}"
ALLOWED_HOSTS = [{% for domain indomains %}"{{ domain }}",{% endfor %}]
DATABASES = {
"default": {
# Ends with "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
"ENGINE": "django.db.backends.postgresql_psycopg2",
# DB name or path to database file if using sqlite3.
"NAME": "{{ proj_name }}",
# Not used with sqlite3.
"USER": "{{ proj_name }}",
# Not used with sqlite3.
"PASSWORD": "{{ db_pass }}",
# Set to empty string for localhost. Not used with sqlite3.
"HOST": "127.0.0.1",
# Set to empty string for default. Not used with sqlite3.
"PORT": "",
}
}
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTOCOL", "https")
CACHE_MIDDLEWARE_SECONDS = 60
CACHE_MIDDLEWARE_KEY_PREFIX = "{{ proj_name }}"
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
"LOCATION": "127.0.0.1:11211",
}
}
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
Most of this template is straightforward; it uses the {{ variable }} syntax to insert the values of variables such as secret_key, nevercache_key, proj_name, and db_pass. The only non-trivial bit of logic is the line shown in Example 6-15:
Example 6-15. Using a for loop in a Jinja2 template
ALLOWED_HOSTS = [{% for domain in domains %}"{{ domain }}",{% endfor %}]
If we look back at our variable definition, we have a variable called domains that’s defined like this:
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.io
Our Mezzanine app is going to respond only to requests that are for one of the hostnames listed in the domains variable: http://192.168.33.10.xip.io or http://www.192.168.33.10.xip.io in our case. If a request reaches Mezzanine but the host header is something other than those two domains, the site will return “Bad Request (400).”
We want this line in the generated file to look like this:
ALLOWED_HOSTS = ["192.168.33.10.xip.io", "www.192.168.33.10.xip.io"]
We can achieve this by using a for loop, as shown in Example 6-15. Note that it doesn’t do exactly what we want. Instead, it will have a trailing comma, like this:
ALLOWED_HOSTS = ["192.168.33.10.xip.io", "www.192.168.33.10.xip.io",]
However, Python is perfectly happy with trailing commas in lists, so we can leave it like this.
WHAT’S XIP.IO?
You might have noticed that the domains we are using look a little strange: 192.168.33.10.xip.io and www.192.168.33.10.xip.io. They are domain names, but they have the IP address embedded within them.
When you access a website, you pretty much always point your browser to a domain name such as http://www.ansiblebook.com, instead of an IP address like http://54.225.155.135.
When we write our playbook to deploy Mezzanine to Vagrant, we want to configure the application with the domain name or names that it should be accessible by.
The problem is that we don’t have a DNS record that maps to the IP address of our Vagrant box. In this case, that’s 192.168.33.10. There’s nothing stopping us from setting up a DNS entry for this. For example, I could create a DNS entry from mezzanine-internal.ansiblebook.com that points to 192.168.33.10.
However, if we want to create a DNS name that resolves to a particular IP address, there’s a convenient service called xip.io, provided free of charge by Basecamp, that we can use so that we don’t have to avoid creating our own DNS records. If AAA.BBB.CCC.DDD is an IP address, then the DNS entry AAA.BBB.CCC.DDD.xip.io will resolve to AAA.BBB.CCC.DDD. For example, 192.168.33.10.xip.io resolves to 192.168.33.10. In addition, www.192.168.33.10.xip.io also resolves to 192.168.33.10.
I find xip.io to be a great tool when I’m deploying web applications to private IP addresses for testing purposes.
Alternatively, you can simply add entries to the /etc/hosts file on your local machine, which also works when you’re offline.
Let’s examine the Jinja2 for loop syntax. To make things a little easier to read, we’ll break it up across multiple lines, like this:
ALLOWED_HOSTS = [
{% for domain in domains %}
"{{ domain }}",
{% endfor %}
]
The generated config file would look like this, which is still valid Python.
ALLOWED_HOSTS = [
"192.168.33.10.xip.io",
"www.192.168.33.10.xip.io",
]
Note that the for loop has to be terminated by an {% endfor %} statement. Also note that the for statement and the endfor statement are surrounded by {% %} delimiters, which are different from the {{ }} delimiters that we use for variable substitution.
All variables and facts that have been defined in a playbook are available inside of Jinja2 templates, so we never need to explicitly pass variables to templates.
Running django-manage Commands
Django applications use a special script called manage.py that performs administrative actions for Django applications such as:
§ creating database tables
§ applying database migrations
§ loading fixtures from files into the database
§ dumping fixtures from the database to files
§ copying static assets to the appropriate directory
In addition to the built-in commands that manage.py supports, Django applications can add custom commands, and Mezzanine adds a custom command called createdb that is used to initialize the database and copy the static assets to the appropriate place. The official Fabric scripts do the equivalent of:
$ manage.py createdb --noinput --nodata
Ansible ships with a django_manage module that invokes manage.py commands. We could invoke it like this:
- name: initialize the database
django_manage:
command: createdb --noinput --nodata
app_path: "{{ proj_path }}"
virtualenv: "{{ venv_path }}"
Unfortunately, the custom createdb command that Mezzanine adds isn’t idempotent. If invoked a second time, it will fail like this:
TASK: [initialize the database] ***********************************************
failed: [web] => {"cmd": "python manage.py createdb --noinput --nodata", "failed"
: true, "path": "/home/vagrant/mezzanine_example/bin:/usr/local/sbin:/usr/local/b
in:/usr/sbin: /usr/bin:/sbin:/bin:/usr/games:/usr/local/games", "state": "absent"
, "syspath": ["", "/usr/lib/python2.7", "/usr/lib/python2.7/plat-x86_64-linux-gnu
", "/usr/lib/python2.7/lib-tk", "/usr/lib/python2.7/lib-old", "/usr/lib/python2.7
/lib-dynload", "/usr/local/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist
-packages"]}
msg:
:stderr: CommandError: Database already created, you probably want the syncdb or
migrate command
Fortunately, the custom createdb command is effectively equivalent to three idempotent built-in manage.py commands:
syncdb
Create database tables for Django models that are not versioned with South, a library that implements database migrations for Django.
migrate
Create and update database tables for Django models that are versioned with South.
collectstatic
Copy the static assets to the appropriate directories.
By invoking these commands, we get an idempotent task:
- name: sync the database, apply migrations, collect static content
django_manage:
command: "{{ item }}"
app_path: "{{ proj_path }}"
virtualenv: "{{ venv_path }}"
with_items:
- syncdb
- migrate
- collectstatic
Running Custom Python Scripts in the Context of the Application
To initialize our application, we need to make two changes to our database.
1. We need to create a Site model object that contains the domain name of our site (in our case, that’s 192.168.33.10.xip.io).
2. We need to set the administrator username and password.
While we could make these changes with raw SQL commands, typically you’d do this by writing Python code, and that’s how the Mezzanine Fabric scripts do it, so that’s how we’re going to do it.
There are two tricky parts here. The Python scripts need to run in the context of the virtualenv that we’ve created, and the Python environment needs to be set up properly so that the script will import the settings.py file that’s in ~/mezzanine_example/project.
In most cases, if I needed some custom Python code, I’d write a custom Ansible module. However, as far as I know, Ansible doesn’t let you execute a module in the context of a virtualenv, so that’s out.
I used the script module instead. This will copy over a custom script and execute it. I wrote two scripts, one to set the Site record, and the other to set the admin username and password.
You can pass command-line arguments to script modules and parse them out, but I decided to pass the arguments as environment variables instead. I didn’t want to pass passwords via command-line argument (those show up in the process list when you run the ps command), and it’s easier to parse out environment variables in the scripts than it is to parse command-line arguments.
Passing Environment Variables to Ansible Tasks
Ansible allows you to set environment variables by adding an environment clause to a task, passing it a dictionary that contains the environment variable names and values. You can add an environment clause to any task; it doesn’t have to be a script.
In order to run these scripts in the context of the virtualenv, I also needed to set the path variable so that the first Python executable in the path would be the one inside of the virtualenv. Example 6-16 shows how I invoked the two scripts.
Example 6-16. Using the script module to invoke custom Python code
- name: set the site id
script: scripts/setsite.py
environment:
PATH: "{{ venv_path }}/bin"
PROJECT_DIR: "{{ proj_path }}"
WEBSITE_DOMAIN: "{{ live_hostname }}"
- name: set the admin password
script: scripts/setadmin.py
environment:
PATH: "{{ venv_path }}/bin"
PROJECT_DIR: "{{ proj_path }}"
ADMIN_PASSWORD: "{{ admin_pass }}"
The scripts themselves are shown in Examples 6-17 and 6-18. I put these in a scripts subdirectory.
Example 6-17. scripts/setsite.py
#!/usr/bin/env python
# A script to set the site domain
# Assumes two environment variables
#
# PROJECT_DIR: the project directory (e.g., ~/projname)
# WEBSITE_DOMAIN: the domain of the site (e.g., www.example.com)
import os
import sys
# Add the project directory to system path
proj_dir = os.path.expanduser(os.environ['PROJECT_DIR'])
sys.path.append(proj_dir)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from django.conf import settings
from django.contrib.sites.models import Site
domain = os.environ['WEBSITE_DOMAIN']
Site.objects.filter(id=settings.SITE_ID).update(domain=domain)
Site.objects.get_or_create(domain=domain)
Example 6-18. scripts/setadmin.py
#!/usr/bin/env python
# A script to set the admin credentials
# Assumes two environment variables
#
# PROJECT_DIR: the project directory (e.g., ~/projname)
# ADMIN_PASSWORD: admin user's password
import os
import sys
# Add the project directory to system path
proj_dir = os.path.expanduser(os.environ['PROJECT_DIR'])
sys.path.append(proj_dir)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from mezzanine.utils.models import get_user_model
User = get_user_model()
u, _ = User.objects.get_or_create(username='admin')
u.is_staff = u.is_superuser = True
u.set_password(os.environ['ADMIN_PASSWORD'])
u.save()
Setting Service Configuration Files
Next, we set the configuration file for Gunicorn (our application server), nginx (our web server), and Supervisor (our process manager), as shown in Example 6-19. The template for the Gunicorn configuration file is shown in Example 6-21.
Example 6-19. Setting configuration files
- name: set the gunicorn config file
template: src=templates/gunicorn.conf.py.j2 dest={{ proj_path }}/gunicorn.conf.py
- name: set the supervisor config file
template: src=templates/supervisor.conf.j2
dest=/etc/supervisor/conf.d/mezzanine.conf
sudo: True
notify: restart supervisor
- name: set the nginx config file
template: src=templates/nginx.conf.j2
dest=/etc/nginx/sites-available/mezzanine.conf
notify: restart nginx
sudo: True
In all three cases, we generate the config files using templates. The Supervisor and nginx processes are started by root (although they drop down to non-root users when running), so we need to sudo so that we have the appropriate permissions to write their configuration files.
If the supervisor configuration file changes, then Ansible will fire the restart supervisor handler. If the nginx configuration file changes, then Ansible will fire the restart nginx handler, as shown in Example 6-20.
Example 6-20. Handlers
handlers:
- name: restart supervisor
supervisorctl: name=gunicorn_mezzanine state=restarted
sudo: True
- name: restart nginx
service: name=nginx state=restarted
sudo: True
Example 6-21. templates/gunicorn.conf.py.j2
from __future__ import unicode_literals
import multiprocessing
bind = "127.0.0.1:{{ gunicorn_port }}"
workers = multiprocessing.cpu_count() * 2 + 1
loglevel = "error"
proc_name = "{{ proj_name }}"
Example 6-22. templates/supervisor.conf.j2
[group: {{ proj_name }}]
programs=gunicorn_{{ proj_name }}
[program:gunicorn_{{ proj_name }}]
command={{ venv_path }}/bin/gunicorn_django -c gunicorn.conf.py -p gunicorn.pid
directory={{ proj_path }}
user={{ user }}
autostart=true
autorestart=true
redirect_stderr=true
environment=LANG="{{ locale }}",LC_ALL="{{ locale }}",LC_LANG="{{ locale }}"
The only template that has any template logic (other than variable substitution) is Example 6-23. It has conditional logic to enable TLS if the tls_enabled variable is set to true. You’ll see some if statements scattered about the templates that look like this:
{% if tls_enabled %}
...
{% endif %}
It also uses the join Jinja2 filter here:
server_name {{ domains|join(", ") }};
This code snippet expects the variable domains to be a list. It will generate a string with the elements of domains connected together, separated by commas. Recall that in our case, the domains list is defined as:
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.io
When the template renders, the line looks like this:
server_name 192.168.33.10.xip.io, www.192.168.33.10.xip.io;
Example 6-23. templates/nginx.conf.j2
upstream {{ proj_name }} {
server 127.0.0.1:{{ gunicorn_port }};
}
server {
listen 80;
{% if tls_enabled %}
listen 443 ssl;
{% endif %}
server_name {{ domains|join(", ") }};
client_max_body_size 10M;
keepalive_timeout 15;
{% if tls_enabled %}
ssl_certificate conf/{{ proj_name }}.crt;
ssl_certificate_key conf/{{ proj_name }}.key;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# ssl_ciphers entry is too long to show in this book
# See https://github.com/lorin/ansiblebook
# ch06/playbooks/templates/nginx.conf.j2
ssl_prefer_server_ciphers on;
{% endif %}
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_pass http://{{ proj_name }};
}
location /static/ {
root {{ proj_path }};
access_log off;
log_not_found off;
}
location /robots.txt {
root {{ proj_path }}/static;
access_log off;
log_not_found off;
}
location /favicon.ico {
root {{ proj_path }}/static/img;
access_log off;
log_not_found off;
}
}
Enabling the Nginx Configuration
The convention with nginx configuration files is to put your configuration files in /etc/nginx/sites-available and enable them by symlinking them into /etc/nginx/sites-enabled.
The Mezzanine Fabric scripts just copy the configuration file directly into sites-enabled, but I’m going to deviate from how Mezzanine does it because it gives me an excuse to use the file module to create a symlink. We also need to remove the default configuration file that the nginx package sets up in /etc/nginx/sites-enabled/default.
Example 6-24. Enabling nginx configuration
- name: enable the nginx config file
file:
src: /etc/nginx/sites-available/mezzanine.conf
dest: /etc/nginx/sites-enabled/mezzanine.conf
state: link
notify: restart nginx
sudo: True
- name: remove the default nginx config file
file: path=/etc/nginx/sites-enabled/default state=absent
notify: restart nginx
sudo: True
As shown in Example 6-24, we use the file module to create the symlink and to remove the default config file. This module is useful for creating directories, symlinks, and empty files; deleting files, directories, and symlinks; and setting properties such as permissions and ownership.
Installing TLS Certificates
Our playbook defines a variable named tls_enabled. If this variable is set to true, then the playbook will install TLS certificates. In our example, we use self-signed certificates, so the playbook will create the certificate if it doesn’t exist.
In a production deployment, you would copy an existing TLS certificate that you obtained from a certificate authority.
Example 6-25. Installing TLS certificates
- name: ensure config path exists
file: path={{ conf_path }} state=directory
sudo: True
when: tls_enabled
- name: create tls certificates
command: >
openssl req -new -x509 -nodes -out {{ proj_name }}.crt
-keyout {{ proj_name }}.key -subj '/CN={{ domains[0] }}' -days 3650
chdir={{ conf_path }}
creates={{ conf_path }}/{{ proj_name }}.crt
sudo: True
when: tls_enabled
notify: restart nginx
Example 6-25 shows the two tasks involved in configuring for TLS certificates. We use the file module to ensure that the directory that will house the TLS certificates exists.
Note how both tasks contain the clause:
when: tls_enabled
If tls_enabled evaluates to false, then Ansible will skip the task.
Ansible doesn’t ship with modules for creating TLS certificates, so we need to use the command module to invoke the openssl command in order to create the self-signed certificate. Since the command is very long, we use YAML line folding syntax (see “Line Folding”) so that we can break the command across multiple lines.
These two lines at the end of the command are additional parameters that are passed to the module; they are not passed to the command line. The chdir parameter changes directory before running the command. The creates parameter implements idempotence: Ansible will first check to see if the file {{ conf_path }}/{{ proj_name }}.crt exists on the host. If it already exists, then Ansible will skip this task.
chdir={{ conf_path }}
creates={{ conf_path }}/{{ proj_name }}.crt
Installing Twitter Cron Job
If you run manage.py poll_twitter, then Mezzanine will retrieve tweets associated with the configured accounts and show them on the home page.
The Fabric scripts that ship with Mezzanine keep these tweets up-to-date by installing a cron job that runs every five minutes.
If we followed the Fabric scripts exactly, we’d copy a cron script into the /etc/cron.d directory that had the cron job. We could use the template module to do this. However, Ansible ships with a cron module that allows us to create or delete cron jobs, which I find more elegant.Example 6-26 shows the task that installs the cron job.
Example 6-26. Installing cron job for polling twitter
- name: install poll twitter cron job
cron: name="poll twitter" minute="*/5" user={{ user }} job="{{ manage }} poll_twitter"
If you manually SSH to the box, you can see the cron job that gets installed by doing crontab -l to list the jobs. Here’s what it looks like for me when I deploy as the Vagrant user:
#Ansible: poll twitter
*/5 * * * * /home/vagrant/mezzanine-example/bin/python /home/vagrant/mezzanine-
example/project/manage.py poll_twitter
Notice the comment at the first line. That’s how the Ansible module supports deleting cron jobs by name. If you were to do:
- name: remove cron job
cron: name="poll twitter" state=absent
The cron module would look for the comment line that matches the name and delete the job associated with that comment.
The Full Playbook
Example 6-27 shows the complete playbook in all its glory.
Example 6-27. mezzanine.yml: the complete playbook
---
- name: Deploy mezzanine
hosts: web
vars:
user: "{{ ansible_ssh_user }}"
proj_name: mezzanine-example
venv_home: "{{ ansible_env.HOME }}"
venv_path: "{{ venv_home }}/{{ proj_name }}"
proj_dirname: project
proj_path: "{{ venv_path }}/{{ proj_dirname }}"
reqs_path: requirements.txt
manage: "{{ python }} {{ proj_path }}/manage.py"
live_hostname: 192.168.33.10.xip.io
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.io
repo_url: git@github.com:lorin/mezzanine-example.git
gunicorn_port: 8000
locale: en_US.UTF-8
# Variables below don't appear in Mezannine's fabfile.py
# but I've added them for convenience
conf_path: /etc/nginx/conf
tls_enabled: True
python: "{{ venv_path }}/bin/python"
database_name: "{{ proj_name }}"
database_user: "{{ proj_name }}"
database_host: localhost
database_port: 5432
gunicorn_proc_name: mezzanine
vars_files:
- secrets.yml
tasks:
- name: install apt packages
apt: pkg={{ item }} update_cache=yes cache_valid_time=3600
sudo: True
with_items:
- git
- libjpeg-dev
- libpq-dev
- memcached
- nginx
- postgresql
- python-dev
- python-pip
- python-psycopg2
- python-setuptools
- python-virtualenv
- supervisor
- name: check out the repository on the host
git: repo={{ repo_url }} dest={{ proj_path }} accept_hostkey=yes
- name: install required python packages
pip: name={{ item }} virtualenv={{ venv_path }}
with_items:
- gunicorn
- setproctitle
- south
- psycopg2
- django-compressor
- python-memcached
- name: install requirements.txt
pip: requirements={{ proj_path }}/{{ reqs_path }} virtualenv={{ venv_path }}
- name: create a user
postgresql_user:
name: "{{ database_user }}"
password: "{{ db_pass }}"
sudo: True
sudo_user: postgres
- name: create the database
postgresql_db:
name: "{{ database_name }}"
owner: "{{ database_user }}"
encoding: UTF8
lc_ctype: "{{ locale }}"
lc_collate: "{{ locale }}"
template: template0
sudo: True
sudo_user: postgres
- name: generate the settings file
template:
src: templates/local_settings.py.j2
dest: "{{ proj_path }}/local_settings.py"
- name: sync the database, apply migrations, collect static content
django_manage:
command: "{{ item }}"
app_path: "{{ proj_path }}"
virtualenv: "{{ venv_path }}"
with_items:
- syncdb
- migrate
- collectstatic
- name: set the site id
script: scripts/setsite.py
environment:
PATH: "{{ venv_path }}/bin"
PROJECT_DIR: "{{ proj_path }}"
WEBSITE_DOMAIN: "{{ live_hostname }}"
- name: set the admin password
script: scripts/setadmin.py
environment:
PATH: "{{ venv_path }}/bin"
PROJECT_DIR: "{{ proj_path }}"
ADMIN_PASSWORD: "{{ admin_pass }}"
- name: set the gunicorn config file
template:
src: templates/gunicorn.conf.py.j2
dest: "{{ proj_path }}/gunicorn.conf.py"
- name: set the supervisor config file
template:
src: templates/supervisor.conf.j2
dest: /etc/supervisor/conf.d/mezzanine.conf
sudo: True
notify: restart supervisor
- name: set the nginx config file
template:
src: templates/nginx.conf.j2
dest: /etc/nginx/sites-available/mezzanine.conf
notify: restart nginx
sudo: True
- name: enable the nginx config file
file:
src: /etc/nginx/sites-available/mezzanine.conf
dest: /etc/nginx/sites-enabled/mezzanine.conf
state: link
notify: restart nginx
sudo: True
- name: remove the default nginx config file
file: path=/etc/nginx/sites-enabled/default state=absent
notify: restart nginx
sudo: True
- name: ensure config path exists
file: path={{ conf_path }} state=directory
sudo: True
when: tls_enabled
- name: create tls certificates
command: >
openssl req -new -x509 -nodes -out {{ proj_name }}.crt
-keyout {{ proj_name }}.key -subj '/CN={{ domains[0] }}' -days 3650
chdir={{ conf_path }}
creates={{ conf_path }}/{{ proj_name }}.crt
sudo: True
when: tls_enabled
notify: restart nginx
- name: install poll twitter cron job
cron: name="poll twitter" minute="*/5" user={{ user }}
job="{{ manage }} poll_twitter"
handlers:
- name: restart supervisor
supervisorctl: name=gunicorn_mezzanine state=restarted
sudo: True
- name: restart nginx
service: name=nginx state=restarted
sudo: True
Running the Playbook Against a Vagrant Machine
The live_hostname and domains variables in our playbook assume that the host we are going to deploy to is accessible at 192.168.33.10. The Vagrantfile shown in Example 6-28 will configure a Vagrant machine with that IP address.
Example 6-28. Vagrantfile
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.network "private_network", ip: "192.168.33.10"
end
Deploy Mezzanine into the Vagrant machine:
$ ansible-playbook mezzanine.yml
You can then reach your newly deployed Mezzanine site at any of the following URLs:
§ http://192.168.33.10.xip.io
§ https://192.168.33.10.xip.io
§ http://www.192.168.33.10.xip.io
§ https://www.192.168.33.10.xip.io
Deploying Mezzanine on Multiple Machines
In this scenario, we’ve deployed Mezzanine entirely on a single machine. However, it’s common to deploy the database service on a separate host from the web service. In Chapter 8, we’ll show a playbook that deploys across the database and web services on separate hosts.
You’ve now seen what it’s like to deploy a real application with Mezzanine. In the next chapter, we’ll cover some more advanced features of Ansible that didn’t come up in our example.
1 My wife, Stacy, is notorious for doing this.
2 You can find the Fabric scripts that ship with Mezzanine on GitHub.
3 See the Postgres documentation for more details about template databases.