Implementing Cloud Storage with OpenStack Swift (2014)

Foreword

I have worked with Amar in the OpenStack San Francisco Bay Area user group and the Entertainment Technology Council cloud effort over the past year. Amar is part of the larger Seagate and Evault effort to transform a manufacturer and product commodity vendor. He has been working with Swift for around 3 years and has deep understanding of what makes it tick.

The authors, like myself, have been lured into the great experiment that is OpenStack and it has changed our careers for the better. Seagate, EVault, and Vedams are working to provide higher-level services like key value store disks and API implementations that provide novel solutions for software defined infrastructure problems. The authors have produced an excellent operational guide that will benefit anyone interested in understanding Swift.

Object storage predates the implementations of Swift and S3. It originated in the universities and spread to Internet based companies such as Yahoo and Google. Internet companies require vast amounts of eventually consistent data. As the business of search changed the way the technology industry thought about services, more uses for object stores were found. Swift was publicly released about a year after Rackspace started working on the CloudFiles replacement in August 2009. The development was born out of a tight group that blended development and operations expertise. Rackspace needed massively scalable storage that they had control over the implementation and the code base.

We are very fortunate that at the time Swift was being released to the world as a new open source project in the summer of 2010, NASA engineers were finishing up their rewrite of the virtual server software Eucalyptus. Nova, as the NASA project became known, had an engineering effort that was so similar to Swift, that both teams were stunned. NASA engineer, Joshua McKenty, noted, "We were using the same tools. We had made the same language decisions. We got the two development teams together — none of whom had ever met each other — and we both said: 'Wow, you just wrote the code that we were going to write.'" - http://www.wired.com/2012/04/openstack-histor/.

It was more than just luck that the two teams were developing similar code in a similar fashion. Similar minds came to similar conclusions. I first met Joshua McKenty, Jesse Andrews, and Vishvananda Ishaya, in May 2010. We were all at the MSST storage conference in Incline Village, NV. They were debating over the few nights available to us of what storage to use for their project. I provided some backdrop for Yahoo's storage options. Many drinks later and a few days, it seemed that they were no closer to deciding between the choices available at the time. Just a month later, Rackspace and NASA were to begin down the road of making history.

Swift is an open source private object store for companies seeking to be part of the open source software defined infrastructure movement. Storage APIs breed innovative new ways to develop and operate. Lifting the restrictions of POSIX interfaces has been cathartic. This remote storage model breaks down, however, when you factor in latency and the network cost of repatriating your data. As John Dickenson states, "Storage is key. It always grows. It is incredibly sticky. It is very hard to move around." -https://www.youtube.com/watch?v=Dd7wmJCDh4w.

Swift fills this gap of local, simple object storage. It is open source, eventually consistent, supports ACLs, large objects, failure domains, and both Swift and S3 APIs. Using simple, inexpensive servers it drives the cost down below many other vendor backed solutions. While listing off features and direct benefits is a fun exercise, the hidden benefits of using Swift are the most important. Once you start down the path of using Swift and other OpenStack projects, you are on your way to automating your infrastructure.

To properly operate distributed computing software like Swift; you will need to embrace automating your infrastructure using DevOps techniques. DevOps simply means your operations engineers must have development abilities. This is not a new idea, but making it a requirement for operations is. Additionally, when using open source software, your engineers must understand and participate in the open source community that builds and maintains Swift. I have personally built storage systems. The planning, implementation, and operations are always more complicated than expected. This is generally due to the fact of integration. Even if Swift is the first storage solution your company is implementing, you will need to expand, upgrade, and support many generations of Swift. This one facet of your evolving engineering team means your most valuable resources are your engineers, not your vendor relationships. Now even more than in the past, we are moving away from the logic and intelligence buried in the vendor's hardware.

The accomplishment of unshackling customers from the whims of vendors is grand, but it requires a renewed understanding of the value of key personnel and your partnership with the open source community. The CAPEX that would be plowed into the next generation of vendor X hardware now needs to be redirected into keeping your engineers close and committed. The commitment to DevOps engineering means focusing on OPEX to reap the innovation and cost savings from using open source software. In-house software development practices will need be adopted and curated. Consistent code releases to follow the pace of the open source community will work to encourage lasting positive DevOps behaviors. Your infrastructure workplace will be practicing some form of agile development methods. Continuous Integration pipelines and Kanban boards will be your weapons to tame the new business model.

This book gives you a powerful taste of what your DevOps software defined infrastructure will need to thrive and survive. Swift will be your inexpensive, easily expanded distributed storage system that is the backbone of your operations.

Sean Roberts

Board Director at the OpenStack Foundation,

Infrastructure Strategy at Yahoo