Ubuntu Unleashed 2017 Edition (2017)

An interesting development in the NoSQL world is that specifications are being created for a new database query language called UnQL (pronounced “uncle”), which stands for Unstructured Query Language. This is being developed as a joint project by two developers: Richard Hipp, the creator of SQLite; and Damien Katz, the creator of CouchDB. They expect more to join them soon. In a nutshell, the language contains some familiar commands, such as SELECT, INSERT, UPDATE, and DELETE. However, it is different from SQL because these commands do not work on tables, but rather on collections of unordered sets of objects that are described using JavaScript Object Notation (JSON). You can learn more about UnQL from the first product we have seen that uses it at www.unqlite.org.

The sections that follow group databases by similarities between them, choosing one standout feature as the name of the section. Because NoSQL databases are still fairly new, it is unclear which, if any, will become a long-term standard. For that reason, this chapter gives high-level coverage of a larger number of options rather than deep coverage of a couple of primary options.

Key/Value Stores

Key/value stores are listed first because they are the simplest of the NoSQL databases, at least in the sense of interactions. You have a piece of data of any type; this is your value. You give it a name of some sort; this is your key. Any time you need that specific piece of data, you ask for it using the key. Values might be bits of text, binaries, pretty much anything, and the data type does not need to be defined in advance, or often at all. The database never needs to know what the value object is, just that it is stored using the given key. These databases have no schema. The contents might be vastly different from one another in type, size, domain, and so on. It is the client, the application that uses the database, that is required to know about the value (what it is and the context in which it can be used). The database merely stores it using a key, knows the key/value pair, and serves the value when requested using its key.

Key/value stores are great for things like contents of a website shopping cart, user preference lists, a post in a social media site. Think of things that are not vital, things that might be useful but that will not cause problems if lost. You would not want to use this for credit card information, personal identification, health records, and such. You would want it for high-traffic sites that need to make sure that a local user has quick and accurate access to the information, but where the information can take time to replicate to other database nodes or which might not require replication across nodes at all, where there is heavy access to the database itself, but where users are not necessarily using the same data concurrently.

Berkeley DB

Berkeley DB was originally created at the University of California, Berkeley, to create a disk hash table that worked better than an existing solution while also helping the university clean up its free UNIX version called BSD by removing code inherited from AT&T. Several years later, Netscape asked the developers to add some desired features to make Berkeley DB more useful to them. This resulted in spinning off Berkeley DB from the university to a company founded for this purpose called Sleepycat Software, which headed development for many years. As of the purchase of Sleepycat Software in 2006, Berkeley DB is now owned by Oracle.

Although it is listed under key/value stores, this is not the only way to interact with a Berkeley DB database. Support also exists for using SQL and Java. Interaction is accomplished using an application programming interface (API). Berkeley DB is very fast and very small. As a result, it can be found running on large-scale systems and embedded within applications and even running on mobile devices.

Berkeley DB is easily the most mature database mentioned in this chapter and is most notable for its use within many well-known software projects including Subversion, Postfix, and OpenLDAP. It was even included as a data storage backend for MySQL prior to MySQL 5.1.

Cassandra

Cassandra was developed by Facebook for their Inbox searching feature. It was released open source when Facebook turned it over to Apache in 2008. Cassandra is a key/value store that runs on a flexible cluster of nodes and is also a wide column store, like HBase discussed later in the “Wide Column Store” section. Nodes may be added and removed from the cluster. Data is replicated across multiple nodes of the cluster. There is no central node, access to data exists from any node; if the node receiving the request does not house the specific data requested, it still services the request by retrieving and sending the data. The main goal of Cassandra is fast retrieval of data with fault tolerance being handled through replication across nodes and speed adjustments via adding additional nodes to create more access points.

One interesting feature is that Cassandra may be tuned to adjust the trade-off between speed of transactions and consistency of data. When data is stored, it is initially stored in memory and gets sent to disk only when specific criteria are met. This makes interaction very quick. In fact, not all data stored in Cassandra is designed to persist over time, and data might not get written to disk at all. This means that not all readers or seekers of data may find a specific piece, but in cases like Facebook’s need to store Inbox search data that only has limited time value (like search results, that could be different tomorrow or even ten minutes from now), this might not matter at all. In these cases, both access speed and convenience are more important.

Cassandra is being used by Facebook, Twitter, Reddit, and many others.

Memcached and MemcacheDB

Memcached stores data requested on a system in RAM for a specific period of time to make retrieving that data faster if it is requested again. The time that data persists can be based on a specific setting, memory needs, and other criteria. The goal is to reduce the number of times that data stores must be accessed. Data that is accessed often is held in memory, from where it is much more quickly retrieved. This can alleviate problems such as a page on a blog that has suddenly become popular as a result of the URL being posted on a social networking site. The spike in traffic could be kept manageable because the content of the blog post is being held in memory instead of being requested over and over from the database.

MemcacheDB is an implementation of the Memcached API that uses a key/value format based on Berkeley DB. However, where Memcached is designed as a cache solution to speed up data access from memory, MemcacheDB is designed as a persistent storage engine. Because it uses the same API protocol as Memcached, it is an easy way to add data persistence where caching is already in place with Memcached.

Memcached is used by sites like Twitter, Reddit, YouTube, and Facebook, and it is also supported and often used by websites based on content management systems like Drupal and WordPress.

Redis

Initially released in 2009, Redis is intended for applications where performance and flexibility are more important than persistence and absolute data integrity. It is an open-source key/value store written in C. Keys can contain strings, hashes, lists, sets, and stored sets. Redis works in RAM for speed, occasionally dumping to disk. Because actions are performed in memory, they are done faster. Operations include appending to a string, incrementing a hash value, pushing to a list, set computations, and more. Redis is also designed so that master-slave replication is easy to set up.

Riak

Riak is a fault tolerant, distributed database designed for scalability and use in the cloud. It is masterless, meaning there should be no single point of failure. It is designed for speed, simplicity, and stability. Riak is based on a paper by Amazon describing Dynamo, which is an internal, proprietary system owned by Amazon. The Riak Wiki describes the database in one place as “the most boring database you’ll ever run in production. No sharding required, just horizontal scaling and straightforward capacity planning. The same operational tasks apply to small clusters and large clusters. More machines does not mean more ops.”

Document Stores

Document stores are designed to store data that is already structured in some form of notation like JSON or XML. They typically focus on one specific type of notation and are intended to allow entire objects, including arrays and hashes, to be stored and retrieved at once.

Many times document stores are implemented as a layer between an application and a relational database to hold the output of certain types of queries. For example, it might be convenient to aggregate information that is typically requested together such as a set of user preferences or name and address information and store it as one object. Requesting and retrieving only one object that is already formatted in an object notation like JSON is faster than making many database queries, and it supplies preformatted data for the client application that can be used to both style output and display specific data at the same time.

Data that is stored and served this way does not have to fit database-specific formatting requirements in a NoSQL database. There are no tables to relate, and data may be larger or smaller and include more or less information. This is generally called semistructured data. Listings 31.1 and 31.2 are a quick snippet of two sets of user preferences in JSON—one that includes many user-set preferences and one that includes only one. The client application could be created to assume a set of default preferences that will be used unless specifically overridden by this file.

LISTING 31.1 Sandra’s Preferences

Click here to view code image

{"userpreferences": {
"displayName": "Don'tHitOnMe",
"gender":"DoNotDisplay",
"siteTheme":"Springtime",
"postsDisplayed":"25",
"keepLoggedIn":"True"
}
}

LISTING 31.2 Matthew’s Preferences

Click here to view code image

{"userpreferences": {
"siteTheme":"TieDye",
}
}

CouchDB

CouchDB began in 2005 as a self-funded personal project of Damien Katz. In 2008, it was given to Apache, where development continues. The goal of CouchDB is to provide a database useful for serving web applications. The emphasis is on scalability and fault tolerance while using commodity hardware (Couch is an acronym for cluster of unreliable commodity hardware). This is not an easy task, but when done successfully it lowers costs.

CouchDB uses a RESTful HTTP API that is designed from the beginning to be used on and for the Web. All stored items have a unique uniform resource identifier (URI), and full create, read, update, and delete (CRUD) functions are available directly using standard HTTP calls, making CouchDB very easy to integrate into web applications. These calls can be made from a browser or from a command line using a tool like cURL, which is available on many typical server platforms, including Ubuntu.

A nice feature of CouchDB is that it is designed with the ability to include ACID compliance, unlike many NoSQL options. This makes it possible to use CouchDB with more consistency-sensitive data.

CouchDB is written in Erlang, which is a language designed for concurrency. That makes CouchDB even better suited for use in a concurrent distributed system. CouchDB is designed to store JSON document objects.

CouchDB is used by several software and web applications, including many Facebook games and applications, internal use at the BBC, and Ubuntu One, Canonical’s cloud storage.

MongoDB

MongoDB is similar to CouchDB in that both are designed as document stores for JSON objects and, like Cassandra, is designed for replication and high-availability. It is created and supported by a company called 10gen and is newer, with its first public release in 2009. A unique feature for this open-source database is that the developer offers commercial, enterprise-class support, training, and consulting. This has made adoption of MongoDB much faster than is typical for NoSQL products.

MongoDB supports sharding, which automatically partitions data across servers for increased performance and scalability. This produces a form of load and data balancing and also offers a way to add nodes simply. Sharding is also intended to support an automatic failover system where node data is replicated, allowing no single point of failure.

In addition, MongoDB includes support for indexing in a manner that is more extensive and powerful than most NoSQL solutions.