Using Swift - Implementing Cloud Storage with OpenStack Swift (2014)

Implementing Cloud Storage with OpenStack Swift (2014)

Chapter 4. Using Swift

This chapter explains the various mechanisms that are available to access Swift. Using these mechanisms, we will be able to authenticate accounts, list containers, create containers, create objects, delete objects, and so on. Tools and libraries such as Swift Client CLI, cURL client, HTTP REST API, JAVA libraries, Ruby OpenStack libraries, and Python libraries use Swift APIs internally to provide access to the Swift cluster. In particular, we will be using the Swift Client CLI, cURL, and HTTP REST API to access Swift and perform various operations on containers and objects. Also, we will be using EVault's Long-Term Storage (LTS2) cloud storage to demonstrate the use of Swift.

Installing the clients

This section talks about installing cURL and Swift's client CLI command line tools. In this section we describe how to install these tools on a Ubuntu 12.04 Linux operating system. Please refer to the other Linux distribution command sets for installing the clients in those operating systems. Windows and Mac version of these tools are also available. The following commands are used to install the cURL and the Swift Client CLI:

· cURL: This is a command-line tool that can be used to transfer data using various protocols. The following command is used to install cURL:

· # apt-get install curl

· Swift Client CLI: This is a tool to access and perform operations on a Swift cluster. This tool is installed using the following command:

· # apt-get install python-swiftclient

· REST API Client: To access Swift services via the REST API, we can use third-party tools such as Fiddler web debugger that supports REST's architecture.

Creating a token using authentication

The first step in order to access containers or objects is to authenticate the user by sending a request to the authentication service and get a valid token that can then be used in subsequent commands to perform various operations. We are using Keystone authentication in our configuration and the examples shown in this chapter. There is another method of authentication called Swauth that can be used. It works in a slightly different way, but we don't deal with the details of Swauth here. The following command is used to get the valid keystone authentication token:

# curl -X POST -i https://auth.lts2.evault.com/v2.0/Tokens -H 'Content-type: application/json' -d '{"auth":{"passwordCredentials":{"username":"user","password":"password"},"tenantName":"tenant1"}}'

In the preceding command, https://auth.lts2.evault.com/v2.0 is EVault's authentication endpoint. Along with this the username, password, and the tenant name are also provided.

The token that is generated is shown as follows (it has been truncated for better readability):

token = MIIGIwYJKoZIhvcNAQcCoIIGFDCCBhACAQExCTAHBgUrDgMCGjCCBHkGCSqGSIb3DQEHAaCCBGoEggRme…yJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNlQwNjoxODo0Mi4zNTA0NTciLCU+KNYN20G7KJO05bXbbpSAWw+5Vfl8zl6JqAKKWENTrlKBvsFzO-peLBwcKZXTpfJkJxqK7Vpzc-NIygSwPWjODs--0WTes+CyoRD

This token is then used as a parameter in the commands accessing Swift, for example, in the following command:

curl -X HEAD -i https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b

-H 'X-Auth-Token: token' -H 'Content-type: application/json'

More details on the commands are provided in the upcoming sections.

Displaying metadata information for an account, container, or object

This section describes how we can obtain information about the account, container, or object.

Using the Swift Client CLI

The Swift Client CLI stat command is used to get information about the account, container, or object. The name of the container should be provided after the stat command to get container information. The name of the container and object should be provided after the stat command to get object information.

Execute the following request to display the account status:

# swift --os-auth-token=token --os-storage-url= https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b stat

In the preceding commands, token is the generated token as described in the previous section and 26cef4782cca4e5aabbb9497b8c1ee1b is the account name.

The response shows the information about the account, which is as follows:

Account: 26cef4782cca4e5aabbb9497b8c1ee1b

Containers: 2

Objects: 6

Bytes: 17

Accept-Ranges: bytes

Server: nginx/1.4.1

Using cURL

The following command shows how to obtain the same account information using cURL. It shows that the account contains two containers and six objects.

Execute the following request:

# curl -X HEAD -i https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b

-H 'X-Auth-Token: token' -H 'Content-type: application/json'

The response to the preceding command is as follows:

HTTP/1.1 204 No Content

Server: nginx/1.4.1

Date: Wed, 04 Dec 2013 06:53:13 GMT

Content-Type: text/html; charset=UTF-8

Content-Length: 0

X-Account-Bytes-Used: 3439364822

X-Account-Container-Count: 2

X-Account-Object-Count: 6

Using the REST API

Fiddler web debugger, which supports REST, was used to send the request and receive the HTTP response. Execute the following request:

Method : HEAD

URL : https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b Header : X-Auth-Token: token

Data : No data

The response is as follows:HTTP/1.1 204 No Content

Server: nginx/1.4.1

Date: Wed, 04 Dec 2013 06:47:17 GMT

Content-Type: text/html; charset=UTF-8

Content-Length: 0

X-Account-Bytes-Used: 3439364822

X-Account-Container-Count: 2

X-Account-Object-Count: 6

As you can see, this is a different mechanism of issuing the command, but is very similar to accessing the Swift cluster using cURL.

Listing containers

This section describes how to obtain information about the containers present in an account.

Using the Swift Client CLI

Execute the following request:

swift --os-auth-token=token --os-storage-url= https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b list

The response to the preceding request is as follows:

cities

countries

Using cURL

The following command shows how to obtain the same containers information using cURL. It shows that the account comprises of two containers and six objects.

Execute the following request:

curl -X GET –i https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b -H 'X-Auth_token: token'

The response to the request is as follows:

HTTP/1.1 200 OK

X-Account-Container-Count: 2

X-Account-Object-Count: 6

cities

countries

Here we see that the output has header and body, whereas in the previous example, we only had header and no body in the output.

Listing objects in a container

This section describes how to list the objects that are present in a container.

Using the Swift Client CLI

The following command shows how to list objects using the Swift Client CLI (in this example we are listing out the objects in the cities container):

Execute the following request:

swift --os-auth-token=token --os-storage-url= https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b list cities

The response to the request is as follows:

London.txt

Mumbai.txt

NewYork.txt

Using cURL

The following command shows how to list objects using cURL. In this example, we list the objects in the Cities container.

Execute the following request:

curl -X GET -i https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b/cities

-H 'X-Auth-Token: token '

The response of the request is as follows:

HTTP/1.1 200 OK

Content-Type: text/plain; charset=utf-8

Content-Length: 34

X-Container-Object-Count: 3

London.txt

Mumbai.txt

NewYork.txt

Using the REST API

In this example, we list the objects in the countries container.

Execute the following request:

Method : GET

URL : URL : https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b/countries

Header : X-Auth-Token: token

Data : No content

The response to the request is as follows:

HTTP/1.1 200 OK

Content-Type: text/plain; charset=utf-8

Content-Length: 38

X-Container-Object-Count: 3

France.txt

India.txt

UnitedStates.txt

Updating the metadata for a container

This section describes how to add or update metadata for a container.

Using the Swift Client CLI

In this example, we are adding metadata for countries that we have visited.

Execute the following request:

swift --os-auth-token=token --os-storage-url= https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b post countries

-H "X-Container-Meta-Countries: visited"

Using the REST API

Here we are adding metadata using the REST API.

Execute the following request:

Method : POST

URL : https://storage.lts2.evault.com/v1/26cef4782cca4e5aabbb9497b8c1ee1b/countries

Header : X-Auth-Token: token

X-Container-Meta-Countries: visited

Data : No content

Environment variables

The following environment variables can be used to simplify the CLI commands:

· OS_USERNAME: This contains the username to access the account

· OS_PASSWORD: This contains the password associated with the username

· OS_TENANT_NAME: This contains the name of the tenant

· OS_AUTH_URL: This contains the authentication URL

Once these environment variables are exported, we no longer have to pass these values as input parameters when running the Swift CLI tools.

Pseudo-hierarchical directories

OpenStack Swift object storage can simulate a hierarchical directory structure in containers by including a / (forward slash character) in the object's name.

Let's upload a file (AMERICA/USA/Newyork.txt) into the Continent container using the following command:

# swift upload Continent AMERICA/USA/Newyork.txt

Let's list the Continent container that has a few pseudo-hierarchical folders by using the following commands:

# swift list Continent

AMERICA/USA/Newyork.txt

ASIA/ASIA.txt

ASIA/China/China.txt

ASIA/INDIA/India.txt

Australia/Australia.txt

continent.txt

We can use / as the delimiter parameter to limit the displayed results. We can also use the prefix parameter along with the delimiter parameter to view the objects in the pseudo directory along with pseudo directories within that. The following are a couple of examples showing the use of these parameters:

# swift list Continent --delimiter /

AMERICA/

ASIA/

Australia/

continent.txt

# swift list Continent --delimiter / --prefix ASIA/

ASIA/ASIA.txt

ASIA/China/

ASIA/INDIA/

# swift list Continent --delimiter / --prefix ASIA/INDIA/

ASIA/INDIA/India.txt

Container ACLs

As we saw in the previous sections, in order to access containers and objects, a valid auth token has to be sent in the X-Auth-Token header with each request. Otherwise, an authorization failure code will be returned. In certain cases, access needs to be provided to other clients and applications for certain containers and objects. Access can be provided by setting a metadata element for the container called X-Container-Read. The following example sets this Access Control Lists (ACL) to the cities container:

First, let us list the container status that shows the lack of ACL. Run the following command with admin privileges (the admin user will have the permissions to run this command):

swift stat cities

The values for Read ACL and Write ACL in the following response indicates the lack of ACL:

Account: 26cef4782cca4e5aabbb9497b8c1ee1b

Container: cities

Objects: 3

Read ACL:

Write ACL:

Sync To:

When the tenant1:user1 user, who does not have access to this container, tries to access this container, a forbidden error message is returned.

Execute the following request:

swift -V 2.0 -A https://auth.lts2.evault.com/v2.0 -U tenant1:user1 -K t1 list cities

A forbidden error message is returned as the response. This error is as follows:

Container GET failed: 403 Forbidden

Access was denied to this resource

In the preceding example, the username is provided using the –U option and the key to access the account is provided using the –K option.

Now, let's set the X-Container-Read metadata element and enable READ access for tenant1:user1. This operation can only be done by the admin user by using the following command:

swift post -r tenant1:user1 cities

To check the ACL permissions, we execute the following command:

swift stat cities

The response to the preceding command is as follows:

Account: 26cef4782cca4e5aabbb9497b8c1ee1b

Container: cities

Objects: 3

Read ACL: tenant1:user1

Write ACL:

Sync To:

Now, when the tenant1:user1 user tries to access this container, access is granted and the command is successfully executed.

Execute the following request:

swift -V 2.0 -A https://auth.lts2.evault.com/v2.0 -U tenant1:user1 -K t1 list cities

The response to the request is as follows:

London.txt

Mumbai.txt

NewYork.txt

Since the X-Container-Write ACL is not set for the tenant1:user1 user for the cities container, this user cannot write to the cities container. In order to allow write access, let's set the X-Container-Write ACL as follows:

swift post -w tenant1:user1 cities

To check the ACL permissions, we execute the following command:

swift stat cities

The response to the preceding command is as follows:

Account: 26cef4782cca4e5aabbb9497b8c1ee1b

Container: cities

Objects: 3

Read ACL: tenant1:user1

Write ACL: tenant1:user1

Sync To:

Now the tenant1:user1 user will be able to write objects into the cities container.

If we want to give access to a large number of users, ACLs such as .r:*, .rlistings can be used. The .r:* prefix allows any user to retrieve objects from the container and .rlistings turns on listing for the container.

Transferring large objects

As discussed in Chapter 2, OpenStack Swift Architecture, Swift limits a single object upload to 5 GB. Larger objects can be split into 5 GB or smaller segments by specifying the segment-size option in the swift CLI tool command-line argument and uploaded to a special container (created within the container where the object is being uploaded to).

Once the upload has been completed, a manifest object has to be created that contains information about the segments. The manifest file is of zero size with headers such as X-Object-Manifest identifying the special container in which the segments are stored and the name with which all the segments will start. For example, if we have to upload France.txt, which is of size 8 GB, to the countries container, then the France.txt object has to be split into two chunks (5 GB and 3 GB). The chunk object's name will start withFrance.txt (France.txt/../00000000 and France.txt/../00000001).

A special container called countries_segments will be created and the chunks will be uploaded to this container. A manifest object called France.txt will be created in the countries container. The manifest file will have zero size and will contain the following header. (It is not mandatory to have the segments placed in a special container and they can as well exist in the same container):

X-Object-Manifest: countries_segments/France.txt

When a download request is made for the large-sized object, Swift will automatically concatenate all the segments and download the entire large-sized object.

The Swift Client CLI has the –S flag, to specify the segment size, which can be used to split a large object into segments and upload. The following command is used to upload a file with a segment size of 5368709120 bytes:

Make the following request:

swift upload countries -S 5368709120 France.txt

The response to the preceding commands is as follows:

France.txt segment 0

France.txt segment 1

France.txt segment 2

France.txt

The following command can be used to list out the containers present:

Swift list

The response to the preceding command is as follows:

Countries

Countries_segments

cities

The following command lists the objects in the countries_segments container:

Swift list countries_segments

The response to the preceding command is as follows:

France.txt/1385989364.105938/5368709120/00000000

France.txt/1385989364.105938/5368709120/00000001

Amazon S3 API compatibility

Users familiar with the Amazon S3 API and accessing S3 buckets and objects can access Swift using S3 compatible APIs with the help of Swift3 middleware.

Here, we will show the steps required for one method that uses S3 APIs to access Swift's object store. These steps explain how to install the necessary tools and packages, create credentials, and update the configuration files.

The following steps are performed on the proxy-server node that is running the Ubuntu 12.04 Linux distribution:

1. First, the user requires EC2 credentials (access key and secret key). The keystone user-list and keystone tenant-list commands can be used to obtain the user ID and tenant ID of the user. The following command can be used to create these keys (these need to be run from the proxy server):

2. keystone ec2-credentials-create --user-id 916673a90b8749e18f0ee3ec5bf17ab9 --tenant-id 6530edfe037242d1ac8bb07b7fd76046

The response is as follows:

+-----------+----------------------------------+

| Property | Value |

+-----------+----------------------------------+

| access | 1178d235dbd84d48b417170ec9aed72c |

| secret | c4ea0a8fbf7d4a469f6d0fb5cdb47d5b |

| tenant_id | 6530edfe037242d1ac8bb07b7fd76046 |

| user_id | 916673a90b8749e18f0ee3ec5bf17ab9 |

3. Install the Swift3 package by running the following commands (these commands require Git to be installed on your system):

4. # sudo git clone https://github.com/fujita/swift3.git

5. # cd swift3

6. # sudo python setup.py install

7. Install the libdigest-hmac-perl package by running the following command (this package is used for integrity checking between two entities that share a secret key):

8. apt-get install libdigest-hmac-perl

9. Edit the proxy-server.conf file and make the following changes if you want to use the keystone authentication:

· Change the pipeline line in the proxy-server.conf file to:

· [pipeline:main]

· pipeline = catch_errors cache swift3 s3token authtoken keystone proxy-server

· Add a Swift3 WSGI filter to the proxy-server.conf file using the following command:

· [filter:swift3]

· use = egg:swift3#swift3

· Add the s3token filter as in the following commands:

· [filter:s3token]

· paste.filter_factory = keystone.middleware.s3_token:filter_factory

· auth_port = 35357

· auth_host = 127.0.0.1

· auth_protocol = http

· Restart the proxy service using the following command:

· Service swift-proxy restart

10. The following steps should be performed on the client that will access Swift Object Store:

· Since we will use s3curl to execute the S3 commands, download s3-curl.zip from the following link:

· http://s3.amazonaws.com/doc/s3-example-code/s3-curl.zip

· Install the wget utility prior to running the following command:

· wget http://s3.amazonaws.com/doc/s3-example-code/s3-curl.zip

· Unzip s3-curl.zip and provide executable access to the s3curl.pl file.

· Create a .s3curl file and change the ID and key of personal account with the EC2 credentials (access and secret keys) that were given to the user. We are using vi editor to create the file as shown in the following:

· #vi ~/.s3curl

· %awsSecretAccessKeys = (

· # personal account

· personal => {

· id => '1178d235dbd84d48b417170ec9aed72c',

· key => 'c4ea0a8fbf7d4a469f6d0fb5cdb47d5b',

· },

· # corporate account

· work => {

· id => '1ATXQ3HHA59CYF1CVS02',

· key => 'WQY4SrSS95pJUT95V6zWea01gBKBCL6PI0cdxeH8',

· },

· );

Accessing Swift using S3 commands

In this section, we will give examples of S3 commands to perform various operations.

· List buckets: This command lists all the buckets for this user. Buckets in S3 are similar to containers in Swift.

· # ./s3curl.pl --id=personal -- https://auth.lts2.evault.com –v

The response is as follows:

<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://doc.s3.amazonaws.com/2006-03-01"><Buckets>

<Bucket><Name>cities</Name><CreationDate>2009-02-03T16:45:09.000Z</CreationDate></Bucket>

<Bucket><Name>countries</Name><CreationDate>2009-02-03T16:45:09.000Z</CreationDate></Bucket>

</Buckets></ListAllMyBucketsResult>

· List objects in a bucket: This command lists all the objects present in the specified bucket. Let us list all the objects in the cities bucket by using the following command:

· # ./s3curl.pl --id=personal -- https://auth.lts2.evault.com/cities -v

· Create a Bucket: The following command creates a bucket called continents:

· # ./s3curl.pl --id=personal --createBucket -- -v https://auth.lts2.evault.com/continents

· Delete a Bucket: The following command deletes a bucket called continents:

· # ./s3curl.pl --id=personal --delete -- -v https://auth.lts2.evault.com/continents

Accessing Swift using client libraries

There are several libraries available in Java, Python, Ruby, PHP, and other programming languages to access the Swift cluster. Applications can be simplified using these libraries. Let us explore a few libraries.

Java

The Apache jclouds library ( http://jclouds.apache.org/documentation/quickstart/rackspace/), particularly the org.jclouds.openstack.swift.CommonSwiftClient API can be used to write applications in Java to connect to Swift and perform various operations on accounts, containers, and objects.

A sample code is shown as follows:

import org.jclouds.ContextBuilder;

import org.jclouds.blobstore.BlobStore;

import org.jclouds.blobstore.BlobStoreContext;

import org.jclouds.openstack.swift.CommonSwiftAsyncClient;

import org.jclouds.openstack.swift.CommonSwiftClient;

BlobStoreContext context = ContextBuilder.newBuilder(provider)

.endpoint("http://auth.lts2.evault.com/")

.credentials(user, password)

.modules(modules)

.buildView(BlobStoreContext.class);

storage = context.getBlobStore();

swift = context.unwrap();

containers = swift.getApi().listContainers();

objects = swift.getApi().listObjects(myContainer);

Python

The python-swiftclient library provides Python language bindings for OpenStack Swift. After authentication, the following sample code shows how to list containers:

#!/usr/bin/env python

http_connection = http_connection(url)

cont = get_container(url, token, container, marker, limit, prefix, delimiter, end_marker, path, http_conn)

More information about the library is provided at https://github.com/openstack/python-swiftclient/.

Ruby

The ruby-openstack library (https://github.com/ruby-openstack/ruby-openstack) provides ruby bindings for the OpenStack cloud. The following sample code shows how to list containers and objects:

Lts2 = OpenStack::Connection.create(:username => USER, :api_key => API_KEY, :authtenant => TENANT, :auth_url => API_URL, :service_type => "object-store")

Lts2.containers

=>["cities" , "countries"]

Cont = Lts2.container("cities")

Cont.objects

=>[" London.txt"," Mumbai.txt"," NewYork.txt"]

Summary

In this chapter, you learned how to use various Swift clients to interact with Swift clusters and get information on accounts, containers, and objects. You were introduced to ACLs, large object transfers, and also to various Swift client libraries that can be used to write applications in your desired language such as Java, Ruby, and Python.

The next chapter talks about managing Swift and things to consider while replacing or expanding disks, nodes, and zones. It also provides information on various tools that can be used to gather information on the object storage behavior.