NoSQL For Dummies (2015)
Part V. Graph and Triple Stores
Chapter 23. Neo4j and Neo Technologies
In This Chapter
Capitalizing on Neo4j
Getting support for Neo4j
Neo4j is the most popular open-source graph store available today. Neo4j allows the storage of nodes and relationships and uses properties to annotate these types of objects. A typing system is provided by tagging nodes with labels.
Sophisticated queries are supported with the feature-rich Cypher Query Language. The syntax of Cypher is easy to pick up if you’re familiar with the Structured Query Language (SQL) of relational database management systems. Also, using the correct color coding in a Cypher syntax-highlighting text editor makes the queries rather pretty in their own right! Lots of nice brackets, parentheses, and arrows.
Neo4j’s commercial offering — Neo4j Enterprise from Neo Technologies — provides functionality for mission-critical applications, including high availability, full and incremental backups, and systems monitoring. Neo4j specializes in providing an embeddable and feature-rich graph store designed for the most complex graph problems and for storing billions of nodes and relationships.
In this chapter, I discuss the open source software, its commercial counterpart, and the company with the same name that provides support for both.
Exploiting Neo4j
Neo4j goes beyond the simpler triple store data and query model to provide support for advanced path-oriented algorithms. A path is any route between nodes or a set of traversals that match a particular query.
In this section, I talk about Neo4j’s key benefits and why you may wish to adopt it for your graph store needs.
Advanced path-finding algorithms
Queries can be performed that return a set of paths. These paths can be traversed by specialist algorithms in order to answer complex graph questions.
A common use for a graph store is to find the shortest path between two nodes, for example:
· For route planning applications for car navigation systems
· For social distance or influence calculations weighted by interactions between people on the network
· For determining the most efficient path to route traffic through a data network
The algorithm that’s most often used for this analysis is Dijkstra’s algorithm, named after Edsger Dijkstra. The Dijkstra algorithm works by analyzing each individual link from the source to the destination, remembering the shortest route found and discarding longer paths, until the shortest possible route is found.
At a high-level, the algorithm works by traversing the graph starting with all nodes related to node A, each with a particular starting distance. The distance from this node to its relations is added to the initial distance. As the links to non-visited nodes expands, nodes with too great a distance are eliminated until eventually the shortest path is discovered.
The complexity of this graph traversal can sometimes be simplified by applying known tests during traversal. Perhaps it’s possible to determine the estimated cost before traversing a whole graph — for example, assuming that highways are quicker than minor roads for long journeys. This combination of assumptions to choose the next path to analyze is formalized in the A* algorithm.
Specifically, the A* algorithm used weighted comparisons based on some property of each connection in the graph.
Both the A* algorithm and Dijkstra’s algorithm are supported by Neo4j. Neo4j also allows you to plug in your own traversal algorithms using its Traversal Framework Java API.
Scaling up versus scaling out
It’s hard to perform well on a distributed cluster when you’re traversing a large number of paths between nodes, rather than pulling back properties on nodes with a limited number of relations in a query. This is why Neo4j stores all data on a single server. Setting up replicas is possible, but each replica contains a full copy of the data, rather than subsets of the data of the whole graph. This approach differs from triple stores, which share the data between each server in a cluster, using a sharding approach instead.
Although using a single node for all data provides good query speed for complex graph operations, this does come at a cost — because for very large graphs, you may need to use high-specification server hardware instead of multiple small and inexpensive commodity servers.
The Neo4j documentation recommends, for instance, that each server use fast Solid State Disks (SSDs) as the primary storage mechanism. Given that SSDs provide less capacity than traditional magnetic spinning disks, you will need more space in your server for SSDs. You’ll also probably want to create a virtual disk array (a RAID array) so that you can spread the write load across all disks.
The I/O subsystem of the servers must also be capable of fully utilizing the number of SSDs attached to it. This typically requires a dedicated I/O controller card in the server.
Requiring high-specification hardware in order to provide greater data storage and query speed is an example of vertical scaling. You basically buy an ever-bigger (taller) server to handle more load.
Buying double the specification of a machine may cost three times as much, whereas buying two servers of the same specification costs only twice as much. For this reason, vertical scaling is more costly than the horizontal scaling of other NoSQL databases.
If you absolutely need high-performance graph operations and are happy to pay for the privilege, then the tradeoff between cost and speed may be worth it.
Complying with open standards
The dominant standards in the triple store and linked (open) data world are
· Resource Description Framework (RDF) for the data model
· Web Ontology Language (OWL) for defining an ontology
· SPARQL Protocol and RDF Query Language (SPARQL) for query
Neo4j doesn’t, out of the box, support any of these standards; however, you can find a variety of third-party plug-ins and approaches that enable you to use Neo4j as a standards-compliant triple store.
These plug-ins may not directly map RDF concepts onto those in Neo4j, so be sure to read about each one in depth before adopting it. Because the plug-ins aren’t officially supported by Neo4j, they aren’t full-featured, nor do they perform as well as the open standards-based support that’s built into other triple stores. You need to test these plugins before implementing them. They are useful as an add-on to an existing Neo4j graph store, though.
Using Neo4j for Linked Data applications is documented on the Neo4j documentation website at www.neo4j.org/develop/linked_data .
Finding Support for Neo4j
Neo Technologies is the primary commercial company behind the development of Neo4j. As well as offering services and support, it offers an Enterprise Edition.
The Enterprise Edition of Neo4j provides the following enhanced features:
· Certification on Windows and Linux
· Emergency patches
· Enterprise lock manager to prevent deadlocks
· High-performance cache for heap sizes greater than 8GB
· Highly available clustering
· Hot backups, both full and incremental
· Advanced system monitoring
· Commercial email and 24/7 phone support
· Ability to embed the Enterprise Edition in commercial, closed source, applications
If you’re using Neo4j for production in mission-critical systems, these features are probably vital, so evaluating the Enterprise Edition is worth the effort.
Clustering
Neo4j’s Enterprise Edition provides support for highly available clustering. A Neo4j master server manages the primary data store for all write operations.
You can add other servers to the cluster to provide for failover in case the master fails and to spread out the query load. These replicas are updated during a write transaction using an optimistic commit. An optimistic commit is one where you assume that the write operation has succeeded without specifically checking for it. This usually happens on secondary replicas.
If one of the replicas is unavailable, it will be updated at a later date. Therefore, Neo4j doesn’t guarantee ACID consistency across an entire cluster. The inconsistency may last for a very short window of time, but timing is important in situations where the master fails before a replica is updated.
High-performance caching
When operating as a cluster, a client API operation can land on any Neo4j instance, which is okay in many cases. If you receive many queries for the same data, you want to cache them in memory — doing so is faster than always fetching data from disk.
Neo4j provides two types of cache:
· File buffer cache: Caches the data on disk in the same, efficiently compressed format.
· This also acts as a write-through cache when you’re entering new data into Neo4j and journaling it to disk.
· Object cache: Stores nodes, relationships and their properties in a format for efficient in-memory graph traversal.
· This supports multiple techniques, defaulting to the high-performance cache (hpc).
By providing these caches, Neo4j helps to increase the speed of both high ingest rates and common read queries.
Cache-based sharding
If your dataset is very large, you need to carefully manage what data is loaded into Neo4j’s memory to be sure you’re not constantly adding and removing data from the caches.
A good way of doing so is to send particular API calls to the same server. You may, for instance, know that user A performs a lot of operations on the same set of nodes. By always directing this user to the same server, you ensure that the user’s data is on only a single server’s cache. This approach in Neo4j is called cache-based sharding. Remember, it’s the cache that is sharded, not the Neo4j database itself.
Finding local support
Neo Technologies has offices in the United States, Europe, and the Far East. In the United States it’s based in San Mateo, California. The company has European offices in Malmö in Sweden; in London in the UK; in Munich and Dresden in Germany; and in France. It also has an office in Malaysia.
Neo Technologies provides 24/7 premium support for enterprise customers. Services are also available for education and for help in the design, implementation, rollout, and production phases of a deployment project.
Finding skills
As I indicated earlier, Neo4j is the dominant graph store in the NoSQL database world. It’s used extensively in both the open-source community edition as well as in production in large enterprises.
Many computer scientists with an interest in graph theory are familiar with Neo4j, both in theory and practice. Therefore, you can find a broad range of developers.
For existing organizations, the requirement to learn Cypher will be a barrier, although its similarity to the SQL of relational databases will help with this requirement. Third-party extensions are available to link Neo4j with more common open standards such as RDF, SPARQL, and OWL.