Debugging Ansible Playbooks - Ansible: Up and Running (2015)

Ansible: Up and Running (2015)

Chapter 14. Debugging Ansible Playbooks

Let’s face it: mistakes happen. Whether it’s a bug in a playbook, or a config file on your control machine with the wrong configuration value, eventually something’s going to go wrong. In this last chapter, I’ll review some techniques you can use to help track down those errors.

Debugging SSH Issues

Sometimes, Ansible fails to make a successful SSH connection with the host. When this happens, it’s helpful to see exactly what arguments Ansible is passing to the underlying SSH client so that you can reproduce the problem manually on the command line.

If you invoke ansible-playbook with the -vvv argument, you can see the exact SSH commands that Ansible invokes. This can be handy for debugging.

Example 14-1 shows some sample Ansible output for executing a module that copies a file.

Example 14-1. Example output when verbose flags are enabled

TASK: [copy TLS key] **********************************************************

<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant

<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant

<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o',

'ControlPersist=60s', '-o', 'ControlPath=/Users/lorinhochstein/.ansible/cp/

ansible-ssh-%h-%p-%r', '-o', 'Port=2222', '-o', u'IdentityFile="/Users/

lorinhochstein/.vagrant.d/insecure_private_key"', '-o', 'KbdInteractive

Authentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,

hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant',

'-o', 'ConnectTimeout=10', u'127.0.0.1', u'/bin/sh -c \'sudo -k && sudo -H -S -p

"[sudo via ansible, key=ypkyixkznvqmrbmlhezlnlujtdhrcoam] password: " -u root

/bin/sh -c \'"\'"\'echo SUDO-SUCCESS-ypkyixkznvqmrbmlhezlnlujtdhrcoam; rc=0;

[ -r "/etc/nginx/ssl/nginx.key" ] || rc=2; [ -f "/etc/nginx/ssl/nginx.key" ] ||

rc=1; [ -d "/etc/nginx/ssl/nginx.key" ] && echo 3 && exit 0; (/usr/bin/md5sum

/etc/nginx/ssl/nginx.key 2>/dev/null) || (/sbin/md5sum -q /etc/nginx/ssl/nginx.key

2>/dev/null) || (/usr/bin/digest -a md5 /etc/nginx/ssl/nginx.key 2>/dev/null) ||

(/sbin/md5 -q /etc/nginx/ssl/nginx.key 2>/dev/null) || (/usr/bin/md5 -n /etc/

nginx/ssl/nginx.key 2>/dev/null) || (/bin/md5 -q /etc/nginx/ssl/nginx.key

2>/dev/null) || (/usr/bin/csum -h MD5 /etc/nginx/ssl/nginx.key 2>/dev/null) ||

(/bin/csum -h MD5 /etc/nginx/ssl/nginx.key 2>/dev/null) || (echo "${rc}

/etc/nginx/ssl/nginx.key")\'"\'"\'\'']

Sometimes you might need to use -vvvv when debugging a connection issue, in order to see an error message that the SSH client is throwing.

For example, if the host doesn’t have SSH running, you’ll see an error that looks like this:

testserver | FAILED => SSH encountered an unknown error. The output was:

OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011

debug1: Reading configuration data /etc/ssh_config

debug1: /etc/ssh_config line 20: Applying options for *

debug1: /etc/ssh_config line 102: Applying options for *

debug1: auto-mux: Trying existing master

debug1: Control socket "/Users/lorinhochstein/.ansible/cp/ansible-ssh-127.0.0.1-

2222-vagrant" does not exist

debug2: ssh_connect: needpriv 0

debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.

debug2: fd 3 setting O_NONBLOCK

debug1: connect to address 127.0.0.1 port 2222: Connection refused

ssh: connect to host 127.0.0.1 port 2222: Connection refused

If you have host key verification enabled, and the host key in ~/.ssh/known_hosts doesn’t match the host key of the server, then using -vvvv will output an error that looks like this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that a host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

c3:99:c2:8f:18:ef:68:fe:ca:86:a9:f5:95:9e:a7:23.

Please contact your system administrator.

Add correct host key in /Users/lorinhochstein/.ssh/known_hosts to get rid of this

message.

Offending RSA key in /Users/lorinhochstein/.ssh/known_hosts:1

RSA host key for [127.0.0.1]:2222 has changed and you have requested strict

checking.

Host key verification failed.

If that’s the case, you should delete the offending entry from your ~/.ssh/known_hosts file.

The Debug Module

We’ve used the debug module several times in this book. It’s Ansible’s version of a print statement. As shown in Example 14-2, you can use it to print out either the value of a variable or an arbitrary string.

Example 14-2. The debug module in action

- debug: var=myvariable

- debug: msg="The value of myvariable is {{ var }}"

As we discussed in Chapter 4, you can print out the values of all the variables associated with the current host by invoking:

- debug: var=hostvars[inventory_hostname]

The Assert Module

The assert module will fail with an error if a specified condition is not met. For example, to fail the playbook if there’s no eth1 interface:

- name: assert that eth1 interface exists

assert:

that: ansible_eth1 is defined

When debugging a playbook, it can be helpful to insert assertions so that a failure happens as soon as some assumption you’ve made has been violated.

If you want to check on the status of some file on the host’s file system, then it’s useful to call the stat module first and make some assertion based on the return value of that module:

- name: stat /opt/foo

stat: path=/opt/foo

register: st

- name: assert that /opt/foo is a directory

assert:

that: st.stat.isdir

The stat module collects information about the state of a file path. It returns a dictionary that contains a stat field with the values shown in Table 14-1.

Field

Description

atime

Last access time of path, in Unix timestamp format

ctime

Creation time of path, in Unix timestamp format

dev

Numerical ID of the device that the inode resides on

exists

True if path exists

gid

Numerical group ID of path owner

inode

Inode number

isblk

True if path is block special device file

ischr

True if path is character special device file

isdir

True if path is a directory

isfifo

True if path is a FIFO (named pipe)

isgid

True if set-group-ID bit is set on file

islnk

True if path is a symbolic link

isreg

True if path is a regular file

issock

True if path is a Unix domain socket

isuid

True if set-user-ID bit is set on file

mode

File mode as a string, in octal (e.g. “1777”)

mtime

Last modification time of path, in Unix timestamp format

nlink

Number of hard links to the file

pw_name

Login name of file owner

rgrp

True if group read permission enabled

roth

True if other read permission enabled

rusr

True if user read permission enabled

size

File size in bytes, if regular file

uid

Numerical user ID of path owner

wgrp

True if group write permission enabled

woth

True if other write permission enabled

wusr

True if user write permission enabled

xgrp

True if group execute permission enabled

xoth

True if other execute permission enabled

xusr

True if user execute permission enabled

Table 14-1. stat module return values

Checking Your Playbook Before Execution

The ansible-playbook command supports several flags that allow you to sanity check your playbook before you execute it.

Syntax Check

The --syntax-check flag, as shown in Example 14-3, will check that your playbook’s syntax is valid, but it will not execute it.

Example 14-3. syntax check

$ ansible-playbook --syntax-check playbook.yml

List Hosts

The --list-hosts flag, as shown in Example 14-4, will output the hosts that the playbook will run against, but it will not execute the playbook.

Example 14-4. list hosts

$ ansible-playbook --list-hosts playbook.yml

NOTE

Sometimes you get the dreaded error:

ERROR: provided hosts list is empty

There must be one host explicitly specified in your inventory, or you’ll get this error, even if your playbook only runs against the localhost. If your inventory is initially empty (perhaps because you’re using a dynamic inventory script and haven’t launched any hosts yet), you can work around this by explicitly adding the following line to your inventory:

localhost ansible_connection=local

List Tasks

The --list-tasks flag, shown in Example 14-5, will output the tasks that the playbook will run against. It will not execute the playbook.

Example 14-5. list tasks

$ ansible-playbook --list-tasks playbook.yml

Recall that we used this flag in Example 6-1 to list the tasks in our first Mezzanine playbook.

Check Mode

The -C and --check flags will run Ansible in check mode (sometimes known as dry-run), which tells you whether each task in the playbook would modify the host, but does not make any actual changes to the server.

$ ansible-playbook -C playbook.yml

$ ansible-playbook --check playbook.yml

One of the challenges with using check mode is that later parts of a playbook might only succeed if earlier parts of the playbook were actually executed. Running check mode on Example 6-27 yields the error shown in Example 14-6 because the task depended on an earlier task installing the Git program on the host.

Example 14-6. Check mode failing on a correct playbook

PLAY [Deploy mezzanine] *******************************************************

GATHERING FACTS ***************************************************************

ok: [web]

TASK: [install apt packages] **************************************************

changed: [web] => (item=git,libjpeg-dev,libpq-dev,memcached,nginx,postgresql,py

thon-dev,python-pip,python-psycopg2,python-setuptools,python-virtualenv,supervi

sor)

TASK: [check out the repository on the host] **********************************

failed: [web] => {"failed": true}

msg: Failed to find required executable git

FATAL: all hosts have already failed -- aborting

See Chapter 10 for more details on how modules implement check mode.

Diff (Show File Changes)

The -D and -diff flags will output differences for any files that will be changed on the remote machine. It’s a helpful option to use in conjunction with --check to show how Ansible would change the file if it were run normally.

$ ansible-playbook -D --check playbook.yml

$ ansible-playbook --diff --check playbook.yml

If Ansible would modify any files (e.g., using modules such as copy, template, and lineinfile), then it will show the changes in .diff format, like this:

TASK: [set the gunicorn config file] ******************************************

--- before: /home/vagrant/mezzanine-example/project/gunicorn.conf.py

+++ after: /Users/lorinhochstein/dev/ansiblebook/ch06/playbooks/templates/gunicor

n.conf.py.j2

@@ -1,7 +1,7 @@

from __future__ import unicode_literals

import multiprocessing

bind = "127.0.0.1:8000"

workers = multiprocessing.cpu_count() * 2 + 1

-loglevel = "error"

+loglevel = "warning"

proc_name = "mezzanine-example"

Limiting Which Tasks Run

Sometimes you don’t want Ansible to run every single task in your playbook, particularly when you’re first writing and debugging the playbook. Ansible provides several command-line options that let you control which tasks run.

Step

The --step flag, shown in Example 14-7, will have Ansible prompt you before running each task, like this:

Perform task: install packages (y/n/c):

You can choose to execute the task (y), skip it (n), or tell Ansible to continue running the rest of the playbook without prompting you (c).

Example 14-7. step

$ ansible-playbook --step playbook.yml

Start-at-Task

The --start-at-task taskname flag, shown in Example 14-8, tells Ansible to start running the playbook at the specified task, instead of at the beginning. This can be very handy if one of your tasks failed because there was a bug in one of your tasks, and you want to re-run your playbook starting at the task you just fixed.

Example 14-8. start-at-task

$ ansible-playbook --start-at-task="install packages" playbook.yml

Tags

Ansible allows you to add one or more tags to a task or a play. For example, here’s a play that’s tagged with foo and a task that’s tagged with bar and quux:

- hosts: myservers

tags:

- foo

tasks:

- name: install editors

apt: name={{ item }}

with_items:

- vim

- emacs

- nano

- name: run arbitrary command

command: /opt/myprog

tags:

- bar

- quux

Use the -t tagnames or --tags tagnames flag to tell Ansible to only run plays and tasks that have certain tags. Use the --skip-tags tagnames flag to tell Ansible to skip plays and tasks that have certain tags. See Example 14-9.

Example 14-9. Running or skipping tags

$ ansible-playbook -t foo,bar playbook.yml

$ ansible-playbook --tags=foo,bar playbook.yml

$ ansible-playbook --skip-tags=baz,quux playbook.yml

Onward

As this chapter ends, so does our journey together. And yet, your journey with Ansible is just beginning. I hope that you’ll come to enjoy working with it as much as I do, and that the next time you encounter colleagues who is in clear need of an automation tool, you’ll show them how Ansible can make their lives easier.