OpenStack Operations Guide (2014)

Part II. Operations

Chapter 18. Upgrades

With the exception of Object Storage, upgrading from one version of OpenStack to another can take a great deal of effort. Until the situation improves, this chapter provides some guidance on the operational aspects that you should consider for performing an upgrade based on detailed steps for a basic architecture.

Pre-Upgrade Testing Environment

Probably the most important step of all is the pre-upgrade testing. Especially if you are upgrading immediately after release of a new version, undiscovered bugs might hinder your progress. Some deployers prefer to wait until the first point release is announced. However, if you have a significant deployment, you might follow the development and testing of the release, thereby ensuring that bugs for your use cases are fixed.

Each OpenStack cloud is different, and as a result, even with what may seem a near-identical architecture to this guide, you must still test upgrades between versions in your environment. For this, you need an approximate clone of your environment.

However, that is not to say that it needs to be the same size or use identical hardware as the production environment—few of us have that luxury. It is important to consider the hardware and scale of the cloud you are upgrading, but here are some tips to avoid that incredible cost:

Use your own cloud

The simplest place to start testing the next version of OpenStack is by setting up a new environment inside your own cloud. This may seem odd—especially the double virtualization used in running compute nodes—but it’s a sure way to very quickly test your configuration.

Use a public cloud

Especially because your own cloud is unlikely to have sufficient space to scale test to the level of the entire cloud, consider using a public cloud to test the scalability limits of your cloud controller configuration. Most public clouds bill by the hour, which means it can be inexpensive to perform even a test with many nodes.

Make another storage endpoint on the same system

If you use an external storage plug-in or shared file system with your cloud, in many cases, it’s possible to test that it works by creating a second share or endpoint. This will enable you to test the system before entrusting the new version onto your storage.

Watch the network

Even at smaller-scale testing, it should be possible to determine whether something is going horribly wrong in intercomponent communication if you look at the network packets and see too many.

To actually set up the test environment, there are several methods. Some prefer to do a full manual install using the OpenStack Installation Guides and then see what the final configuration files look like and which packages were installed. Others prefer to create a clone of their automated configuration infrastructure with changed package repository URLs and then alter the configuration until it starts working. Either approach is valid, and which you use depends on experience.

An upgrade pre-testing system is excellent for getting the configuration to work; however, it is important to note that the historical use of the system and differences in user interaction can affect the success of upgrades, too. We’ve seen experiences where database migrations encountered a bug (later fixed!) because of slight table differences between fresh Grizzly installs and those that migrated from Folsom to Grizzly.

Artificial scale testing can go only so far. Once your cloud is upgraded, you’ll also need to pay careful attention to the performance aspects of your cloud.

Preparing for a Rollback

Like all major system upgrades, your upgrade could fail for one or more difficult-to-determine reasons. You should prepare for this situation by leaving the ability to roll back your environment to the previous release, including databases, configuration files, and packages. We provide an example process for rolling back your environment in “Rolling Back a Failed Upgrade”.

Upgrades

The upgrade process generally follows these steps:

1. Perform some “cleaning” of the environment prior to starting the upgrade process to ensure a consistent state. For example, instances not fully purged from the system after deletion may cause indeterminate behavior.

2. Read the release notes and documentation.

3. Find incompatibilities between your versions.

4. Develop an upgrade procedure and assess it thoroughly using a test environment similar to your production environment.

5. Run the upgrade procedure on the production environment.

You can perform an upgrade with operational instances, but this strategy can be dangerous. You might consider using live migration to temporarily relocate instances to other compute nodes while performing upgrades. However, you must ensure database consistency throughout the process; otherwise your environment may become unstable. Also, don’t forget to provide sufficient notice to your users, including giving them plenty of time to perform their own backups.

The following order for service upgrades seems the most successful:

1. Upgrade the OpenStack Identity Service (keystone).

2. Upgrade the OpenStack Image Service (glance).

3. Upgrade OpenStack Compute (nova), including networking components.

4. Upgrade OpenStack Block Storage (cinder).

5. Upgrade the OpenStack dashboard.

The general upgrade process includes the following steps:

1. Create a backup of configuration files and databases.

2. Update the configuration files according to the release notes.

3. Upgrade the packages using your distribution’s package manager.

4. Stop services, update database schemas, and restart services.

5. Verify proper operation of your environment.

How to Perform an Upgrade from Grizzly to Havana—Ubuntu

For this section, we assume that you are starting with the architecture provided in the OpenStack Installation Guide and upgrading to the same architecture for Havana. All nodes should run Ubuntu 12.04 LTS. This section primarily addresses upgrading core OpenStack services, such as the Identity Service (keystone); Image Service (glance); Compute (nova), including networking; Block Storage (cinder); and the dashboard.

Impact on Users

The upgrade process will interrupt management of your environment, including the dashboard. If you properly prepare for this upgrade, tenant instances will continue to operate normally.

Upgrade Considerations

Always review the release notes before performing an upgrade to learn about newly available features that you may want to enable and deprecated features that you should disable.

Perform a Backup

Save the configuration files on all nodes, as shown here:

# for i in keystone glance nova cinder openstack-dashboard

> do mkdir $i-grizzly

> done

# for i in keystone glance nova cinder openstack-dashboard

> do cp -r /etc/$i/* $i-grizzly/

> done

NOTE

You can modify this example script on each node to handle different services.

Back up all databases on the controller:

# mysqldump -u root -p --opt --add-drop-database \

--all-databases > grizzly-db-backup.sql

Manage Repositories

On all nodes, remove the repository for Grizzly packages and add the repository for Havana packages:

# apt-add-repository -r cloud-archive:grizzly

# apt-add-repository cloud-archive:havana

WARNING

Make sure any automatic updates are disabled.

Update Configuration Files

Update the glance configuration on the controller node for compatibility with Havana.

If not currently present and configured as follows, add or modify the following keys in /etc/glance/glance-api.conf and /etc/glance/glance-registry.conf:

[keystone_authtoken]

auth_uri = http://controller:5000

auth_host = controller

admin_tenant_name = service

admin_user = glance

admin_password = GLANCE_PASS

[paste_deploy]

flavor = keystone

If currently present, remove the following key from the [filter:authtoken] section in /etc/glance/glance-api-paste.ini and /etc/glance/glance-registry-paste.ini:

[filter:authtoken]

flavor = keystone

Update the nova configuration on all nodes for compatibility with Havana.

Add the new [database] section and associated key to /etc/nova/nova.conf:

[database]

connection = mysql://nova:NOVA_DBPASS@controller/nova

Remove defunct configuration from the [DEFAULT] section in /etc/nova/nova.conf:

[DEFAULT]

sql_connection = mysql://nova:NOVA_DBPASS@controller/nova

If not already present and configured as follows, add or modify the following keys in /etc/nova/nova.conf:

[keystone_authtoken]

auth_uri = http://controller:5000/v2.0

auth_host = controller

auth_port = 35357

auth_protocol = http

admin_tenant_name = service

admin_user = nova

admin_password = NOVA_PASS

On all compute nodes, increase the DHCP lease time (measured in seconds) in /etc/nova/nova.conf to enable currently active instances to continue leasing their IP addresses during the upgrade process:

[DEFAULT]

dhcp_lease_time = 86400

WARNING

Setting this value too high may cause more dynamic environments to run out of available IP addresses. Use an appropriate value for your environment.

You must restart dnsmasq and the networking component of Compute to enable the new DHCP lease time:

# pkill -9 dnsmasq

# service nova-network restart

Update the Cinder configuration on the controller and storage nodes for compatibility with Havana.

Add the new [database] section and associated key to /etc/cinder/cinder.conf:

[database]

connection = mysql://cinder:CINDER_DBPASS@controller/cinder

Remove defunct configuration from the [DEFAULT] section in /etc/cinder/cinder.conf:

[DEFAULT]

sql_connection = mysql://cinder:CINDER_DBPASS@controller/cinder

If not currently present and configured as follows, add or modify the following key in /etc/cinder/cinder.conf:

[keystone_authtoken]

auth_uri = http://controller:5000

Update the dashboard configuration on the controller node for compatibility with Havana.

The dashboard installation procedure and configuration file changed substantially between Grizzly and Havana. Particularly, if you are running Django 1.5 or later, you must ensure that /etc/openstack-dashboard/local_settings contains a correctly configured ALLOWED_HOSTSkey that contains a list of hostnames recognized by the dashboard.

If users will access your dashboard using http://dashboard.example.com, you would set:

ALLOWED_HOSTS=['dashboard.example.com']

If users will access your dashboard on the local system, you would set:

ALLOWED_HOSTS=['localhost']

If users will access your dashboard using an IP address in addition to a hostname, you would set:

ALLOWED_HOSTS=['dashboard.example.com', '192.168.122.200']

Upgrade Packages on the Controller Node

Upgrade packages on the controller node to Havana, as shown below:

# apt-get update

# apt-get dist-upgrade

NOTE

Depending on your specific configuration, performing a dist-upgrade may restart services supplemental to your OpenStack environment. For example, if you use Open-iSCSI for Block Storage volumes and the upgrade includes a new open-scsi package, the package manager will restart Open-iSCSI services, which may cause the volumes for your users to be disconnected.

The package manager will ask you about updating various configuration files. We recommend denying these changes. The package manager will append .dpkg-dist to the end of newer versions of existing configuration files. You should consider adopting conventions associated with the newer configuration files and merging them with your existing configuration files after completing the upgrade process.

Stop Services, Update Database Schemas, and Restart Services on the Controller Node

Stop each service, run the database synchronization command if necessary to update the associated database schema, and restart each service to apply the new configuration. Some services require additional commands:

OpenStack Identity

# service keystone stop

# keystone-manage token_flush

# keystone-manage db_sync

# service keystone start

OpenStack Image Service

# service glance-api stop

# service glance-registry stop

# glance-manage db_sync

# service glance-api start

# service glance-registry start

OpenStack Compute

# service nova-api stop

# service nova-scheduler stop

# service nova-conductor stop

# service nova-cert stop

# service nova-consoleauth stop

# service nova-novncproxy stop

# nova-manage db sync

# service nova-api start

# service nova-scheduler start

# service nova-conductor start

# service nova-cert start

# service nova-consoleauth start

# service nova-novncproxy start

OpenStack Block Storage

# service cinder-api stop

# service cinder-scheduler stop

# cinder-manage db sync

# service cinder-api start

# service cinder-scheduler start

The controller node update is complete. Now you can upgrade the compute nodes.

Upgrade Packages and Restart Services on the Compute Nodes

Upgrade packages on the compute nodes to Havana:

# apt-get update

# apt-get dist-upgrade

NOTE

Make sure you have removed the repository for Grizzly packages and added the repository for Havana packages.

WARNING

Due to a packaging issue, this command may fail with the following error:

Errors were encountered while processing:

/var/cache/apt/archives/

qemu-utils_1.5.0+dfsg-3ubuntu5~cloud0_amd64.deb

/var/cache/apt/archives/

qemu-system-common_1.5.0+dfsg-3ubuntu5~cloud0_

amd64.deb

E: Sub-process /usr/bin/dpkg

returned an error code (1)

You can fix this issue by using the following command:

# apt-get -f install

The packaging system will ask about updating the /etc/nova/api-paste.ini file. As with the controller upgrade, we recommend denying these changes and reviewing the .dpkg-dist file after completing the upgrade process.

To restart compute services:

# service nova-compute restart

# service nova-network restart

# service nova-api-metadata restart

Upgrade Packages and Restart Services on the Block Storage Nodes

Upgrade packages on the storage nodes to Havana:

# apt-get update

# apt-get dist-upgrade

NOTE

Make sure you have removed the repository for Grizzly packages and added the repository for Havana packages.

The packaging system will ask about updating the /etc/cinder/api-paste.ini file. Like the controller upgrade, we recommend denying these changes and reviewing the .dpkg-dist file after completing the upgrade process.

To restart Block Storage services:

# service cinder-volume restart

How to Perform an Upgrade from Grizzly to Havana—Red Hat Enterprise Linux and Derivatives

For this section, we assume that you are starting with the architecture provided in the OpenStack Installation Guide and upgrading to the same architecture for Havana. All nodes should run Red Hat Enterprise Linux 6.4 or compatible derivatives. Newer minor releases should also work. This section primarily addresses upgrading core OpenStack services, such as the Identity Service (keystone); Image Service (glance); Compute (nova), including networking; Block Storage (cinder); and the dashboard.

Impact on Users

The upgrade process will interrupt management of your environment, including the dashboard. If you properly prepare for this upgrade, tenant instances will continue to operate normally.

Upgrade Considerations

Always review the release notes before performing an upgrade to learn about newly available features that you may want to enable and deprecated features that you should disable.

Perform a Backup

First, save the configuration files on all nodes:

# for i in keystone glance nova cinder openstack-dashboard

> do mkdir $i-grizzly

> done

# for i in keystone glance nova cinder openstack-dashboard

> do cp -r /etc/$i/* $i-grizzly/

> done

NOTE

You can modify this example script on each node to handle different services.

Next, back up all databases on the controller:

# mysqldump -u root -p --opt --add-drop-database \

--all-databases > grizzly-db-backup.sql

Manage Repositories

On all nodes, remove the repository for Grizzly packages and add the repository for Havana packages:

# yum erase rdo-release-grizzly

# yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/ \

rdo-release-havana-7.noarch.rpm

WARNING

Make sure any automatic updates are disabled.

NOTE

Consider checking for newer versions of the Havana repository.

Update Configuration Files

Update the glance configuration on the controller node for compatibility with Havana.

If not currently present and configured as follows, add or modify the following keys in /etc/glance/glance-api.conf and /etc/glance/glance-registry.conf: