Information Management: Strategies for Gaining a Competitive Advantage with Data (2014)

Chapter 10. Operational Big Data

Key-Value, Document, and Column Stores: Hash Tables Reborn

A chapter on key-value stores, which store data paired with its key. Document stores are key-value stores with more robust data type capabilities. These are immensely important for serving any data immediately from a very large dataset. Column stores put some requirements on how NoSQL data is stored in NoSQL.

Keywords

NoSQL; key-value; document stores; column stores; big data

In the mid-2000s, several new-economy Silicon Valley companies were facing data challenges at a scale beyond those of the largest companies in the world. Knowing the exploitation of that data was the essence of their business, and knowing the existing relational technology would be too expensive to scale—if it even could scale to their needs—many companies independently set out to build their own solutions. The commonalities were many among the internal solutions and most reached a similar conclusion about the SQL/relational/transactionally-sound marketplace—it did not have a place in the solutions.

As discussions took place across the Valley, a Meetup was formed to share findings and advance the solutions. That Meetup needed a name and a poll occurred. Jumping on this fact that the solutions had all ignored or banished SQL, the name of “No SQL” was decided on and a movement was begun.

Initially, the movement’s players were adamant that the solutions would rule the enterprise,¹ but as initial outreach of this storyline was met with disinterest from enterprises, the “No” came to be known as “NO” as in “Not Only SQL.” I cannot say I like the “No” or “NO” as a label. Any time you name a space something that it is not, it creates issues. Walls form in prospects that believe you are anti- all the software they have in house—and that you will not integrate with it. Prospects also will think of all of the capabilities they have with SQL and believe the “no” solution will not do any of them. This is not true, although there are definitely some capabilities that are not present. Also, the interfaces are so quickly coming close to SQL that it would difficult to imagine the space being named NoSQL today. I’m here to help you with the industry, not create new terms, so let’s talk about NoSQL.

Before we leave semantics, there is a very important term that is very related here. Big data is synonymous with the NoSQL movement because these Valley companies that originated the movement had “big” data challenges. If the data were “small,” they would not have created the space and would have just used relational technology. The literature uses NoSQL and big data synonymously, but really the big data is the data being stored in the NoSQL solutions and what the NoSQL solutions store is, by definition, big data.

This is almost, but not completely, true. Graph databases, discussed in Chapter 12, provide extremely valuable functionality for relationship data—and that data need not be “big.”

When to Yes NoSQL

You may be looking for the quantifiable definition of big data. Again, it’s the workload threshold when relational no longer makes sense—given your unique situation. It’s hard to imagine that under a certain single-digit number of terabytes—unless you have a specialized single-use workload that matches well with the pros of NoSQL and minimizes the cons of NoSQL—that it makes sense.

Many will say that big data/NoSQL² is a Valley phenomenon and your company does not have a need to get out of the SQL comfort zone. Not so fast. NoSQL enterprise deployments are mostly for net new applications in the business—exactly those applications that have big data needs. It’s a perfect storm of need and solution.

Why Big Data Now?

There are a few trends bringing us into the era of big data.

• Storing Data is Cheaper than Ever

Companies increasingly are feeling liberated by less expensive means for extending the competitive advantages of data exploitation into more voluminous data: clicks, weblogs, and sensor reads of small movements. Factors that are triggering the liberation include past-decade exponential decreases in the cost of all computing resources and an explosion in network bandwidth.

• Sensor Networks

Most people don’t think of their cell phone as part of a “sensor network.” However, their increased ubiquity and web connection are impossible to ignore as data collection mechanisms. One day, we may be able to opt in to contributing our air quality, air pressure, noise level, or other “around me” data to our favorite companies we do business with. Today, we contribute our location, web clicks, and application usage. Cell phones are examples of the growing sensor network, with great adoption in supply chains.

• History Data

Many get into data warehousing entertaining the notion that they will remove data as it becomes old and less useful—perhaps at the 3-year point and perhaps to something as cheap and inaccessible as tape. However, as long as the warehouse is successful enough to make it to that point, the users are inevitably interested in ALL the data and need fast access to all the data. Tiering some of it to lower-cost storage, perhaps some of the big data options—like Hadoop, for example—is possible, but inaccessible, slow tape is not.

• Syndicated Data

The third-party data marketplace, discussed in Chapter 7, is booming and dispensing data to the enterprise in many cases as much as, or more than, the enterprise generates internally. This potentially significant additional avenue of interesting data can dramatically expand enterprise information.

Not all of these trends directly correlate to the need for a NoSQL solution, but they are making enterprise information “bigger.”

• Ability to Utilize the Data

Just as successful data warehouses result in user need for all the data they’ve ever seen in the data warehouse, likewise high-end analysts (sometimes referred to as Data Scientists) are finding business uses for all data—including data types they have never utilized before—which usually falls into the big data category.

There are many other aspects of NoSQL that define its best workload other than cheaply storing a large amount of data.

Achieving high availability cheaply would be another important consideration.

Data model flexibility, defined later in the chapter, would be another good indicator that NoSQL is a correct categorization for the workload.

In general, it is the sensor data, full webclick data, social data, and related offshoots, such as those attributed later in the chapter to certain of the models, that make for the best fit in NoSQL stores. Enterprises have mostly been force-fitting snapshots and summaries of this data into relational databases for years and now it’s time to realize the full potential of big data.

NoSQL Attributes

Open Source

Most of the NoSQL movement has an open source component. While most of the players in enterprise decision making are familiar with open source, it remains an enigma in the enterprise itself. Certainly, being able to take on software with an open source origin in the enterprise is a requirement for taking on NoSQL and, consequently, a requirement for success in IT today.

Many NoSQL companies have high triage with the open source software they support, in addition to the support and related software they add on to that software. Still, the support model is not the same as enterprise “closed source” software. An enterprise can make it passable, even excellent, with effort, but will likely find a new set of challenges given the abstraction of support from software owner (the “community” in open source) to a third-party company.

Data Models

NoSQL solutions do not require, or accept, a preplanned data model whereby every record has the same fields and each field of a table has to be accounted for in each record. Though there can be strong similarities³ from record to record, there is no “carryover” from one record to the next and each field is encoded with JavaScript Object Notation.

(JSON) or Extensible Markup Language (XML)—according to the solution’s architecture. A record may look like:

• Book: “War and Peace”: Author: “Tolstoy”

This is what is meant by unstructured data- Data of varying forms loaded without a set schema.⁴

Abstact Programming in Relational Databases

Many attempt to simulate the data model flexibility in a relational system through abstract programming techniques.

1. “Dummy” fields like COLUMN32, COLUMN33, COLUMN34, etc. that are usually large VARCHARs and accept the variable fields of the record

2. Heavy use of subtypes and supertypes in tables

3. Storing column names in fields

4. “Code tables” with a code numbering system that spans codes

These techniques may make sense in relational systems if other factors are involved that make relational best for the workload. However, if a flexible data model is a primary consideration in the workload, consider a NoSQL solution.

Scale Out

It has long been a truism that to achieve scaling to higher and higher levels,⁵ it is less expensive to do it with smaller machines than larger machines. In the early days of computing, as systems began to tackle scaling challenges with larger machines, the costs took off exponentially. Machines were getting much more complex. A 10-terabyte system could cost not 10 times a 1-terabyte system, but nearly 100 times. If you wanted to store 10 terabytes, you paid the price. This is scale up.

NoSQL solutions found a way to multiply that 1-terabyte system 10 times: at 10, not 100, times the cost. The key is the divide and conquer, exemplified by MapReduce, discussed later in the chapter. By dividing the programming across a single data set spread over many machines, NoSQL is able to keep the costs down and the scale up (I mean out!).

The following sections are also traits of NoSQL solutions that are dissimilar to relational databases. However, their nature could differ from NoSQL solution to NoSQL solution. I advise you get your solution into the right category first, then make sure the solution adheres to the characteristics of these traits that are needed by the application.

Sharding

Sharding is a partitioning pattern for the NoSQL age. It’s a partitioning pattern that places each partition in potentially separate servers—potentially all over the world. This scale out works well for supporting people all over the world accessing different parts of the data set with performance. If you are taking an order for someone in Dallas, for example, that order can be placed in the Southwest U.S. data center.

A shard is managed by a single server. Based on the replication schema, the shard is most likely replicated twice on other shards. Shards can be set up to split automatically when they get too large or they can be more directed. Auto-sharding takes a load off of the programming, which would otherwise not only have to manage the placement of data, but also the application code’s data retrieval.

Consistency

ACID (see box) is essential for guaranteeing 100% accuracy when you have many rows spanning many tables in a single operation. The operation should succeed or fail completely and concurrent operations are isolated so partial updates do not commit.

Most NoSQL solutions do not have full ACID compliance like a relational system does. This inhibits its abilities to do critical financial transactions. Each record will be consistent, but transactions are usually guaranteed to be “eventually consistent” which means changes to data could be staggered to the queries of that data for a short period of time.⁶ Some queries are looking at “old” data while some may be looking at “new” data. Consider the business model behind the application to determine the fit of the consistency model of the solution. The trade-off for this downside is improved performance and scalability.

No vendor has been able to articulate the risk of the lack of full ACID compliance of a NoSQL solution. Also, I know of no one who has experienced some sort of transaction, and hence business, failure as a result of overcommitting transactions to NoSQL. Still, it IS theoretically possible that a transaction could fail unless full ACID is guaranteed and, for that reason, critical money matters should not be committed (pun) to NoSQL.

What is ACID?

• Atomicity – full transactions pass or fail

• Consistency – database in valid state after each transaction

• Isolation – transactions do not interfere with one another

• Durability – transactions remain committed no matter what (e.g., crashes)

Though the most important component of ACID is atomicity, the more complex component is consistency. Consistency may be tunable by the NoSQL database administrator. By tuning the replication factor and the number of nodes that will respond to a write, the system can follow a more or less “write tolerant” profile. Not only writes, but also the replication, can have this inconsistency window. The effect of replication on consistency can be profound. Consistency can often be tuned on a session basis.

Commodity Components

“Commodity” is a class of hardware and does not refer to your old Tandy 1000s, TRS-80s, Burroughs B2500s, and the like. It is, however, an unspecialized level of computer, somewhere in the mid-single digit thousands of dollars that can be effectively used in NoSQL systems. The scale-out architectures are built to utilize this class of computer.

MapReduce

The idea of MapReduce has been around quite a while. Lisp Systems had the same concept decades ago, but Google effectively revived it as the primary access mechanism for NoSQL systems. The idea is to take a large problem and divide it into locally processed subproblems. As much of a query as possible will be processed in the node where the data is held. That part of the query is the same for all nodes. That is called the Map function. The same homogeneity is true for the Reduce step, which follows the Map step and the movement of the data within the nodes to where it can be effectively utilized.

Both the Map and the Reduce are programmed using Java (primarily). While many organizations have attempted to marginalize or eliminate programming from their shop for internal “system integrator” types who can install and customize software, with NoSQL, you need to bring those skills back!

Sample MapReduce Code: Find Out How Many Times Each Account Was Accessed In The Last 10 Years⁷

map(key, value)

{

// key=byte offset in log file

// value=a line in the log file

if (value is an account access audit log)

{

account number=parse account from value

output key=account number, value=1

}

}

reduce(key, list of values)

{

// key=account number

// list of values {1,1,1,1……}

for each value

count=count+value

output key, count

}