Ubuntu Unleashed 2017 Edition (2017)

Why a Cloud?

Businesses and enterprises have built computer networks for years. There are many reasons, but usually networks are built because specific computation or data processing tasks are made easier and faster using more than one computer. The size of the network generally depends on the tasks that need to be done. Building a network usually entails taking a detailed survey of needs, analyzing those requirements, and gathering together the necessary hardware and software to fulfill those needs now, perhaps with a little room for growth if money permits.

Cloud computing is designed to make that easier by providing resources such as computing power and storage as services on the Internet in a way that is easy to access remotely, available on demand, simple to provision and scale, and highly dynamic. In the ideal case, this saves both time and money.

Some of the greatest benefits are the ease with which new resources may be added to a cloud, the fault tolerance inherent in the built-in redundancy of a large pool of servers, and the payment schedules that charge for resources only when they are used. There is also a great benefit in abstracting the complexity out of the process; clients perform the tasks they want to perform, and the cloud computing platform takes care of the details of adding resources as needed without the end user being aware of the process. Virtual machines (VMs) are created, configured, and used when needed and destroyed immediately after they are no longer needed, freeing up system resources for other purposes. These VMs can be created to suit a wide range of needs.

Hardware, storage, networks, and software are abstracted as services instead of being manually built and configured. They are then accessed locally on demand when the additional resources are required. Sometimes these service model abstractions are referred to as software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS).

Software as a Service (SaaS)

SaaS is sometimes referred to as on-demand software. In this service model, it is the software application and its related data that are moved to the cloud. Access is generally through a web browser, although a thin client and server style configuration are not uncommon. Someone else takes care of everything else. This is kind of like renting a hotel room—everything is provided and set up for you and you just enjoy and use it for a specific need. Some examples of this include email hosts like Yahoo! Mail, services like Google Docs, web games, and customer relationship management (CRM) software.

Platform as a Service (PaaS)

PaaS takes things a step further. In this service model, an entire computing platform is provided in the cloud. This typically includes the operating system, programming language interpreters or execution environments, databases, web servers, and so on. They are accessed directly for computing platform maintenance using provider portals, application programming interfaces (APIs), software development kits (SDKs), or services like SSH. Then, what is built on the platform is accessed by the end user the same way it would be accessed if it were running on a locally owned and operated piece of hardware or hardware that’s running in a large datacenter. Someone else takes care of everything else, but they take care of less than they do with SaaS, which means that you take care of more. This scenario is more like an apartment—you rent the space and decorate and configure it as you like within structured guidelines. Some examples of this include the Google App Engine, raw compute nodes used to scale services, and social application platforms like Facebook.

Infrastructure as a Service (IaaS)

IaaS goes even further. In this service model, you are transitioning your entire server to the cloud. Your provider offers computers, most like virtual ones, on which you can install any operating system (perhaps within a set menu they allow) and you can configure it as you like. Someone else takes care of the physical machines and networks, and you take care of all the rest. This is like buying a condominium—you own it and can do whatever you want inside it, but someone else takes care of the grounds and landscaping.

Metal as a Service (MaaS)

Generally, the only other step available from here is traditional server building, where you are responsible for the physical machine and everything on it. However, Ubuntu has added another service to the list, Metal as a Service (MaaS), which is designed to bring the language of the cloud to physical servers. Their goal is to make it as easy to set up the physical hardware, to deploy your app or service, and to scale up or down dynamically as it is in the cloud. It is installed on the physical hardware and then managed using one web interface to manage all the various machines. There is a section later in this chapter dedicated to this topic.

Before You Do Anything

You don’t have to create your own cloud infrastructure, but you can. You can also deploy Ubuntu Cloud to providers like Rackspace or HP. Before you do anything, you need to carefully consider what your needs are and decide what sort of service(s) you need. Do you just want to run a web application on someone else’s already-set-up server, or do you want to set up a system for scalable computing where additional Hadoop nodes can be added and removed at will when big jobs start and end? Only you know the answer. When you have it figured out, you can seek your solution and can think about how you can use Ubuntu to set it up. This chapter describes many options, but you are the one who is in control. That is a powerful, and sometimes overwhelming, position. Thought and planning prevent painful mistakes and repeated engineering.

Deploy/Install Basics: Public, Private, or Hybrid?

There are two ways to deploy Ubuntu in the cloud: on a private cloud or on a public cloud. Both have benefits and drawbacks. This section presents the things you need to consider when choosing. We also look at a way to mix the two, which is called a hybrid cloud.

A public cloud is built on a cloud provider’s systems. This means your local hardware requirements are minimal, your startup costs are low, deployment is quick, and growth is easy. This can be incredibly useful for testing, and has gained the stability and reputation for also being a great idea for production. The drawback to working this way is that you do not physically control the hardware on which your cloud is running. For many this is a benefit, but this might not be suitable for high security needs. Although you alone control the software and processes on your public cloud, there might be some worry about who has access to the machines. Although a cloud provider would not last long in business if its data centers and machines were not secure, some applications and data are so sensitive you cannot afford to allow any outside risk. Legal constraints, such as from the Sarbanes-Oxley Act, sometimes force IT policy decisions in an organization and make the public option impossible.

A private cloud is created on hardware you own and control. This requires a large upfront commitment, but you have the security of running everything behind a company firewall and with complete knowledge of who is able to physically access your machines and who is listening on the network.

One thing to consider is the possibility of starting your Ubuntu Cloud as a private cloud and then creating interfaces from there to public services, creating a hybrid cloud. Perhaps you prefer to keep some of your data and services stored on the private cloud, but you have other data that is less sensitive and want to use some services and applications on a public cloud. This is an avenue worth exploring if your company has a mixture of “must be secured and held in-house” and “we still want to keep it away from prying eyes, but if something happens it won’t be catastrophic” needs. The big issue with this method is moving data between public and private servers; if you have large amounts of data that may move between the two, this can be prohibitive. As always, do your due diligence.

Ubuntu Cloud and OpenStack

OpenStack is an Apache-licensed cloud computing platform. It was founded as a collaboration between NASA and Rackspace. After less than a year, it boasted a worldwide community of developers. Adoption has been swift, and already many large corporations, universities, and institutions are using OpenStack for cloud computing. Ubuntu and OpenStack have worked closely together for a long time, have similar release schedules, and Ubuntu is the reference operating system for OpenStack.

OpenStack is not a service provider. They don’t operate systems or data centers. OpenStack is open-source software for building public and private and hybrid clouds. There are many companies who have implemented and use OpenStack, which is a good thing. This means that if you develop your cloud deployment and it works on one company’s servers, if they are using OpenStack, you can move that deployment to another company’s servers with little or even no changes, if the second company is also running OpenStack. In fact, it is easy enough to create your deployment across several different providers, using cloud servers from multiple companies concurrently according to your needs.

For a current list of cloud providers offering OpenStack to customers for cloud deployments, see http://www.openstack.org/marketplace/public-clouds/. You will find big names like HP and Rackspace along with many you have not yet heard of who may be just a suitable or perhaps even better for your needs.

Many cloud providers offer free trials that give users a certain amount of usage for a limited amount of time. This can be handy for testing out the capabilities of different providers while testing out deployments of components of your system.

One interesting option while you are exploring and learning about cloud computing is DevStack, from http://devstack.org, which is a shell script with documentation that builds complete OpenStack development environments. This is not an option designed for production environments, but it can be very useful for getting started and testing the capabilities of cloud computing. The program and the documentation are maintained by a community of developers rather than one company, again emphasizing the portability aspect.

If what you read in this chapter interests you, the combination of a free trial period and DevStack may provide you with an easy way to try out what you are learning.

OpenStack uses a set of APIs for its services that are compatible with the Amazon EC2/S3 APIs. Client tools written for those can also be used with OpenStack. OpenStack has several main service families, each described in the following subsections.

OpenStack is in active development. Newer releases may have details that differ from what is recorded in this chapter. Check the official OpenStack website, listed in the “Resources,” section of this chapter, for current details.

Compute Infrastructure (Nova)

Nova manages the compute resources, networking, and scaling for the OpenStack cloud. By itself, it does not perform any virtualization tasks, but rather it uses libvirt APIs to interact with supported hypervisors and is the management component of the system. Nova has an Amazon EC2-compatible RESTful API.

Nova consists of several components:

nova-api—The API server provides an interface to enable outside systems to interact with the cloud infrastructure.

rabbit-mq—The message queue server performs asynchronous calls to communicate with other Nova components, such as the scheduler or the network controller.

Qpid—Like rabbit-mq, this is a message queue server and has similar functions. Research both to see which is the best fit for your situation.

nova-compute—The compute nodes’ host instances. They carry out operations based on requests received by the message queue. Instances are deployed on available compute nodes based on a scheduling algorithm managed by the scheduler.

nova-network—The network controller allocates IP addresses, configures VLANs, configures networks, and implements security groups for compute nodes. It is expected that this will eventually be replaced by Neutron, when Neutron is ready.

nova-volume—The volume worker manage Logical Volume Manager (LVM)-based storage volumes. They create and delete volumes, attach and detach volumes from instances, and provide persistent storage for use by instances.

nova-scheduler—The scheduler uses an adjustable algorithm to determine which compute, network, or storage volume servers should be used from an available pool of resources. Schedules can be configured based on server loads, availability zones, or random chance.

Storage Infrastructure (Swift)

Swift is an object store. It is scalable up to multiple petabytes and billions of objects. It is elastic. It has built-in redundancy and failover. Swift is designed to store a very large number of objects distributed across a commodity hardware.

Networking Service (Neutron)

Neutron provides “networking as a service” between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).

Identity Service (Keystone)

Keystone is the identity service used for authentication (authN) and high-level authorization (authZ). It supports token-based authN and user-service authorization.

Imaging Service (Glance)

Glance is a lookup and retrieval system for VM images. It can use one of three back ends: OpenStack Object Store, S3 storage, or S3 Storage using the OpenStack Object Store as an intermediary.

Dashboard (Horizon)

Horizon is the standard implementation of OpenStack’s Dashboard, which provides a web-based user interface to OpenStack services including Nova, Swift, Neutron, Keystone, etc.

Learning More

OpenStack has many more features not discussed here. Some include OpenStack Heat, from https://wiki.openstack.org/wiki/Heat, for orchestration and management of infrastructure and applications within an OpenStack cloud and OpenStack Ironic, from https://wiki.openstack.org/wiki/Ironic, which is a bare metal provisioning program. See https://wiki.openstack.org/wiki/Programs for a current list of official Open Stack programs.

Juju

Juju has been described as APT for the cloud. As you learned from Chapter 9, “Managing Software,” APT does an amazing job of installing, configuring, and starting complicated software stacks and services, but only as long as all of that happens on only one system. Juju extends this ability across multiple machines. Often, Linux servers are set up for similar tasks. Multiple physical machines may be deployed with similar configurations to work with one another in a network, perhaps for load distribution or redundancy to prevent downtime in the event of one failing or being overloaded. Systems administrators are masters at creating and orchestrating these networks. However, doing so traditionally requires setting up each machine individually, configuring its software settings and so on.

Tools have appeared over the years to help with this great task, such as Chef and Puppet; see Chapter 36, “Managing Sets of Servers,” for a little more about these. Juju works to do for servers what package managers do for individual systems. It enables you to deploy services quickly and easily across multiple servers, simplifying the configuration process, and is particularly designed with cloud servers in mind. As with Chef’s recipes, those services are deployed using formulas that standardize communication, for example, and which may have been written by different people.

What makes Juju different from Chef and Puppet is that the Juju formulas, called charms, encapsulate services, defining all the ways that services need to expose or consume configuration data to or from other services. This can be done many ways in the Juju charm, including via shell scripts or using Chef itself in solo mode. Also, Juju orchestrates provisioning by tracking its available resources (such as EC2, Eucalyptus, or OpenStack machines) and adding or removing them as appropriate.

Juju is pretty cool, but it hasn’t seen much serious adoption outside of Canonical, especially now that the OpenStack tools are growing in number and scope. However, it does have some unique features and it definitely is worth your consideration.

Getting Started

Start by installing Juju on a server:

Click here to view code image

matthew@wolfram~$: sudo apt-get install juju