OpenStack Operations Guide (2014)

Preface

OpenStack is an open source platform that lets you build an Infrastructure as a Service (IaaS) cloud that runs on commodity hardware.

Introduction to OpenStack

OpenStack believes in open source, open design, open development, all in an open community that encourages participation by anyone. The long-term vision for OpenStack is to produce a ubiquitous open source cloud computing platform that meets the needs of public and private cloud providers regardless of size. OpenStack services control large pools of compute, storage, and networking resources throughout a data center.

The technology behind OpenStack consists of a series of interrelated projects delivering various components for a cloud infrastructure solution. Each service provides an open API so that all of these resources can be managed through a dashboard that gives administrators control while empowering users to provision resources through a web interface, a command-line client, or software development kits that support the API. Many OpenStack APIs are extensible, meaning you can keep compatibility with a core set of calls while providing access to more resources and innovating through API extensions. The OpenStack project is a global collaboration of developers and cloud computing technologists. The project produces an open standard cloud computing platform for both public and private clouds. By focusing on ease of implementation, massive scalability, a variety of rich features, and tremendous extensibility, the project aims to deliver a practical and reliable cloud solution for all types of organizations.

Getting Started with OpenStack

As an open source project, one of the unique aspects of OpenStack is that it has many different levels at which you can begin to engage with it—you don’t have to do everything yourself.

Using OpenStack

You could ask, “Do I even need to build a cloud?” If you want to start using a compute or storage service by just swiping your credit card, you can go to eNovance, HP, Rackspace, or other organizations to start using their public OpenStack clouds. Using their OpenStack cloud resources is similar to accessing the publically available Amazon Web Services Elastic Compute Cloud (EC2) or Simple Storage Solution (S3).

Plug and Play OpenStack

However, the enticing part of OpenStack might be to build your own private cloud, and there are several ways to accomplish this goal. Perhaps the simplest of all is an appliance-style solution. You purchase an appliance, unpack it, plug in the power and the network, and watch it transform into an OpenStack cloud with minimal additional configuration. Few, if any, other open source cloud products have such turnkey options. If a turnkey solution is interesting to you, take a look at Nebula One.

However, hardware choice is important for many applications, so if that applies to you, consider that there are several software distributions available that you can run on servers, storage, and network products of your choosing. Canonical (where OpenStack replaced Eucalyptus as the default cloud option in 2011), Red Hat, and SUSE offer enterprise OpenStack solutions and support. You may also want to take a look at some of the specialized distributions, such as those from Rackspace, Piston, SwiftStack, or Cloudscaling. Also, a hat tip to Apache CloudStack, which Citrix donated to the Apache Foundation after its $200 million purchase of Cloud.com. While not currently packaged in any distributions, like Eucalyptus, it is an example of an alternative private cloud software developed in an open source–like manner.

Alternatively, if you want someone to help guide you through the decisions about the underlying hardware or your applications, perhaps adding in a few features or integrating components along the way, consider contacting one of the system integrators with OpenStack experience, such as Mirantis or Metacloud.

If your preference is to build your own OpenStack expertise internally, a good way to kick-start that might be to attend or arrange a training session. The OpenStack Foundation recently launched a Training Marketplace where you can look for nearby events. Also, the OpenStack community is working to produce open source training materials.

Roll Your Own OpenStack

However, this guide has a different audience—those seeking to derive the most flexibility from the OpenStack framework by conducting do-it-yourself solutions.

OpenStack is designed for scalability, so you can easily add new compute, network, and storage resources to grow your cloud over time. In addition to several massive OpenStack public clouds, a considerable number of organizations (such as Paypal, Intel, and Comcast) have built large-scale private clouds. OpenStack offers much more than a typical software package because it lets you integrate a number of different technologies to construct a cloud. This approach provides great flexibility, but the number of options might be bewildering at first.

Who This Book Is For

This book is for those of you starting to run OpenStack clouds as well as those of you who were handed an operational one and want to keep it running well. Perhaps you’re on a DevOps team, perhaps you are a system administrator starting to dabble in the cloud, or maybe you want to get on the OpenStack cloud team at your company. This book is for all of you.

This guide assumes that you are familiar with a Linux distribution that supports OpenStack, SQL databases, and virtualization. You must be comfortable administering and configuring multiple Linux machines for networking. You must install and maintain a MySQL database and occasionally run SQL queries against it.

One of the most complex aspects of an OpenStack cloud is the networking configuration. You should be familiar with concepts such as DHCP, Linux bridges, VLANs, and iptables. You must also have access to a network hardware expert who can configure the switches and routers required in your OpenStack cloud.

TIP

Cloud computing is a quite advanced topic, and this book requires a lot of background knowledge. However, if you are fairly new to cloud computing, we recommend that you make use of the Glossary at the back of the book, as well as the online documentation for OpenStack and additional resources mentioned in this book in Appendix E.

Further Reading

There are other books on the OpenStack documentation website that can help you get the job done.

OpenStack Installation Guides

Describes a manual installation process, as in, by hand, without automation, for multiple distributions based on a packaging system:

§ Installation Guide for Debian 7.0

§ Installation Guide for openSUSE and SUSE Linux Enterprise Server

§ Installation Guide for Red Hat Enterprise Linux, CentOS, and Fedora

§ Installation Guide for Ubuntu 12.04 (LTS) Server

OpenStack Configuration Reference

Contains a reference listing of all configuration options for core and integrated OpenStack services by release version

OpenStack Cloud Administrator Guide

Contains how-to information for managing an OpenStack cloud as needed for your use cases, such as storage, computing, or software-defined-networking

OpenStack High Availability Guide

Describes potential strategies for making your OpenStack services and related controllers and data stores highly available

OpenStack Security Guide

Provides best practices and conceptual information about securing an OpenStack cloud

Virtual Machine Image Guide

Shows you how to obtain, create, and modify virtual machine images that are compatible with OpenStack

OpenStack End User Guide

Shows OpenStack end users how to create and manage resources in an OpenStack cloud with the OpenStack dashboard and OpenStack client commands

OpenStack Admin User Guide

Shows OpenStack administrators how to create and manage resources in an OpenStack cloud with the OpenStack dashboard and OpenStack client commands

OpenStack API Quick Start

A brief overview of how to send REST API requests to endpoints for OpenStack services

How This Book Is Organized

This book is organized in two parts: the architecture decisions for designing OpenStack clouds and the repeated operations for running OpenStack clouds.

Part I:

Chapter 1

Because of all the decisions the other chapters discuss, this chapter describes the decisions made for this particular book and much of the justification for the example architecture.

Chapter 2

While this book doesn’t describe installation, we do recommend automation for deployment and configuration, discussed in this chapter.

Chapter 3

The cloud controller is an invention for the sake of consolidating and describing which services run on which nodes. This chapter discusses hardware and network considerations as well as how to design the cloud controller for performance and separation of services.

Chapter 4

This chapter describes the compute nodes, which are dedicated to running virtual machines. Some hardware choices come into play here, as well as logging and networking descriptions.

Chapter 5

This chapter discusses the growth of your cloud resources through scaling and segregation considerations.

Chapter 6

As with other architecture decisions, storage concepts within OpenStack take a lot of consideration, and this chapter lays out the choices for you.

Chapter 7

Your OpenStack cloud networking needs to fit into your existing networks while also enabling the best design for your users and administrators, and this chapter gives you in-depth information about networking decisions.

Part II:

Chapter 8

This chapter is written to let you get your hands wrapped around your OpenStack cloud through command-line tools and understanding what is already set up in your cloud.

Chapter 9

This chapter walks through user-enabling processes that all admins must face to manage users, give them quotas to parcel out resources, and so on.

Chapter 10

This chapter shows you how to use OpenStack cloud resources and train your users as well.

Chapter 11

This chapter goes into the common failures that the authors have seen while running clouds in production, including troubleshooting.

Chapter 12

Because network troubleshooting is especially difficult with virtual resources, this chapter is chock-full of helpful tips and tricks for tracing network traffic, finding the root cause of networking failures, and debugging related services, such as DHCP and DNS.

Chapter 13

This chapter shows you where OpenStack places logs and how to best read and manage logs for monitoring purposes.

Chapter 14

This chapter describes what you need to back up within OpenStack as well as best practices for recovering backups.

Chapter 15

For readers who need to get a specialized feature into OpenStack, this chapter describes how to use DevStack to write custom middleware or a custom scheduler to rebalance your resources.

Chapter 16

Because OpenStack is so, well, open, this chapter is dedicated to helping you navigate the community and find out where you can help and where you can get help.

Chapter 17

Much of OpenStack is driver-oriented, so you can plug in different solutions to the base set of services. This chapter describes some advanced configuration topics.

Chapter 18

This chapter provides upgrade information based on the architectures used in this book.

Backmatter:

Appendix A

You can read a small selection of use cases from the OpenStack community with some technical details and further resources.

Appendix B

These are shared legendary tales of image disappearances, VM massacres, and crazy troubleshooting techniques to share those hard-learned lessons and wisdom.

Appendix C

Read about how to track the OpenStack roadmap through the open and transparent development processes.

Appendix D

A preview of the new features in the Icehouse release of OpenStack.

Appendix E

So many OpenStack resources are available online because of the fast-moving nature of the project, but there are also resources listed here that the authors found helpful while learning themselves.

Glossary

A list of terms used in this book is included, which is a subset of the larger OpenStack glossary available online.

Why and How We Wrote This Book

We wrote this book because we have deployed and maintained OpenStack clouds for at least a year, and wanted to be able to distribute this knowledge to others. After months of being the point people for an OpenStack cloud, we also wanted to have a document to hand to our system administrators so that they’d know how to operate the cloud on a daily basis—both reactively and proactively. We wanted to provide more detailed technical information about the decisions that deployers make along the way.

We wrote this book to help you:

§ Design and create an architecture for your first nontrivial OpenStack cloud. After you read this guide, you’ll know which questions to ask and how to organize your compute, networking, and storage resources and the associated software packages.

§ Perform the day-to-day tasks required to administer a cloud.

We wrote this book in a book sprint, which is a facilitated, rapid development production method for books. For more information, see the BookSprints site. Your authors cobbled this book together in five days during February 2013, fueled by caffeine and the best takeout food that Austin, Texas, could offer.

On the first day, we filled white boards with colorful sticky notes to start to shape this nebulous book about how to architect and operate clouds:

We wrote furiously from our own experiences and bounced ideas between each other. At regular intervals we reviewed the shape and organization of the book and further molded it, leading to what you see today.

The team includes:

Tom Fifield

After learning about scalability in computing from particle physics experiments, such as ATLAS at the Large Hadron Collider (LHC) at CERN, Tom worked on OpenStack clouds in production to support the Australian public research sector. Tom currently serves as an OpenStack community manager and works on OpenStack documentation in his spare time.

Diane Fleming

Diane works on the OpenStack API documentation tirelessly. She helped out wherever she could on this project.

Anne Gentle

Anne is the documentation coordinator for OpenStack and also served as an individual contributor to the Google Documentation Summit in 2011, working with the Open Street Maps team. She has worked on book sprints in the past, with FLOSS Manuals’ Adam Hyde facilitating. Anne lives in Austin, Texas.

Lorin Hochstein

An academic turned software-developer-slash-operator, Lorin worked as the lead architect for Cloud Services at Nimbis Services, where he deploys OpenStack for technical computing applications. He has been working with OpenStack since the Cactus release. Previously, he worked on high-performance computing extensions for OpenStack at University of Southern California’s Information Sciences Institute (USC-ISI).

Adam Hyde

Adam facilitated this book sprint. He also founded the books sprint methodology and is the most experienced book-sprint facilitator around. See http://www.booksprints.net for more information. Adam founded FLOSS Manuals—a community of some 3,000 individuals developing Free Manuals about Free Software. He is also the founder and project manager for Booktype, an open source project for writing, editing, and publishing books online and in print.

Jonathan Proulx

Jon has been piloting an OpenStack cloud as a senior technical architect at the MIT Computer Science and Artificial Intelligence Lab for his researchers to have as much computing power as they need. He started contributing to OpenStack documentation and reviewing the documentation so that he could accelerate his learning.

Everett Toews

Everett is a developer advocate at Rackspace making OpenStack and the Rackspace Cloud easy to use. Sometimes developer, sometimes advocate, and sometimes operator, he’s built web applications, taught workshops, given presentations around the world, and deployed OpenStack for production use by academia and business.

Joe Topjian

Joe has designed and deployed several clouds at Cybera, a nonprofit where they are building e-infrastructure to support entrepreneurs and local researchers in Alberta, Canada. He also actively maintains and operates these clouds as a systems architect, and his experiences have generated a wealth of troubleshooting skills for cloud environments.

OpenStack community members

Many individual efforts keep a community book alive. Our community members updated content for this book year-round. Also, a year after the first sprint, Jon Proulx hosted a second two-day mini-sprint at MIT with the goal of updating the book for the latest release. Since the book’s inception, more than 30 contributors have supported this book. We have a tool chain for reviews, continuous builds, and translations. Writers and developers continuously review patches, enter doc bugs, edit content, and fix doc bugs. We want to recognize their efforts!

The following people have contributed to this book: Akihiro Motoki, Alejandro Avella, Alexandra Settle, Andreas Jaeger, Andy McCallum, Benjamin Stassart, Chandan Kumar, Chris Ricker, David Cramer, David Wittman, Denny Zhang, Emilien Macchi, Gauvain Pocentek, Ignacio Barrio, James E. Blair, Jay Clark, Jeff White, Jeremy Stanley, K Jonathan Harker, KATO Tomoyuki, Lana Brindley, Laura Alves, Lee Li, Lukasz Jernas, Mario B. Codeniera, Matthew Kassawara, Michael Still, Monty Taylor, Nermina Miller, Nigel Williams, Phil Hopkins, Russell Bryant, Sahid Orentino Ferdjaoui, Sandy Walsh, Sascha Peilicke, Sean M. Collins, Sergey Lukjanov, Shilla Saebi, Stephen Gordon, Summer Long, Uwe Stuehler, Vaibhav Bhatkar, Veronica Musso, Ying Chun “Daisy” Guo, Zhengguang Ou, and ZhiQiang Fan.

How to Contribute to This Book

The genesis of this book was an in-person event, but now that the book is in your hands, we want you to contribute to it. OpenStack documentation follows the coding principles of iterative work, with bug logging, investigating, and fixing. We also store the source content on GitHub and invite collaborators through the OpenStack Gerrit installation, which offers reviews. For the O’Reilly edition of this book, we are using the company’s Atlas system, which also stores source content on GitHub and enables collaboration among contributors.

Learn more about how to contribute to the OpenStack docs at Documentation How To.

If you find a bug and can’t fix it or aren’t sure it’s really a doc bug, log a bug at OpenStack Manuals. Tag the bug under Extra options with the ops-guide tag to indicate that the bug is in this guide. You can assign the bug to yourself if you know how to fix it. Also, a member of the OpenStack doc-core team can triage the doc bug.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Command prompts

Commands prefixed with the # prompt should be executed by the root user. These examples can also be executed using the sudo command, if available.

Commands prefixed with the $ prompt can be executed by any user, including root.

TIP

This element signifies a tip or suggestion.

NOTE

This element signifies a general note.

WARNING

This element indicates a warning or caution.