NoSQL For Dummies (2015)
Part V. Graph and Triple Stores
Chapter 21. Triple Store Use Cases
In This Chapter
Handling unstructured information
Reconstructing processes
Applying inductive logic
Establishing social relationships
Semantics, triples, and graphs offer a whole new world that’s unfamiliar to many people with a database background. This makes it very exciting!
You first need to get hold of the data needed to build your graph, which you can find at from websites that publish Linked Open Data (LOD), or you can extract them from your own content. Two examples of LOD websites are dbpedia (http://dbpedia.org), which provides a semantically modeled extra of Wikipedia data, and geonames (http://geonames.org), which provides a catalogue of places, countries, and geospatial coordinates.
You may then need to track where you found this information, how you acquired it, and the changes made to the information over time. This provenance information can be key in legal and defense industries.
Perhaps you need to merge all the new triples you have added from your own organizations' data with reference information from other systems and sources. Several techniques are available that can help you accomplish this task. For example, you can add extra properties to a Place from imported Geonames data, or you can create your own subclass of Place, such as a Theme Park, and describe that location.
You then need to support providing answers to potentially difficult questions by using the information you store. Because the social graph is the most wide-ranging example of a complex graph model, in this chapter, I spend some time talking about the links between people and other subjects in a triple store, and the types of queries a triple store holding this type of data can address.
In this chapter, I also extract facts from existing data, use additional facts from third-party sources, and use semantic data to store an audit trail of changes to records over time.
Extracting Semantic Facts
Semantic facts are properly called assertions. For most use cases, using the term facts is an easier mental model to picture, so I use the term facts throughout. An assertion is a single triple of subject – predicate – and object. This could be “Adam is a Person” or “Adam is aged 33.”
All database software can easily handle structured information like these facts. This data is effectively a list of fields and their values, as I discussed in Chapter 19. Unstructured text, though, is very difficult for computers, and indeed humans, to deal with.
Free text indexing and proximity searches can help you to some extent. For example, a proximity search displays documents in which matching terms occur, and you can infer a relationship between text within a document by using proximity-search limits such as NEAR queries (for example, “Adam NEAR/3 age” which means searching for where “Adam” is mentioned within three words of “Age”). To do anything useful, though, you need a more comprehensive approach.
In this section, I describe how to analyze the text in unstructured content in order to extract semantic facts, so you can provide a rich query using semantic technology over information extracted from text documents.
Extracting context with subjects
Natural Language Processing (NLP) is the process of looking at words, phrases, and sentences and identifying not only things (people, places, organizations, and so on) but also their relationships.
You can store this information as subjects, properties, and relationships — as you do in triple stores. Several pieces of software are available to help you do so. Here are the most popular ones:
· OpenCalais: An open-source semantic extraction tool
· Smartlogic Semaphore Server: A commercial entity extraction and enrichment tool that can also generate triples
· TEMIS Luxid: Another common commercial entity extraction and enrichment tool
All of these tools provide entity extraction (identify things) with entity enrichment (add extra information about those things). They also have the ability to generate an output report. This report can be in a list of semantic assertions (OpenCalais) in RDF or XML format. You can also place these facts in a separate RDF document, or you can tag text inline in the original document. All these tools generate XML and/or RDF output.
You can then store this output in a document database or a triple store, or you can utilize a hybrid of the two. Doing so facilitates an accurate semantic search. It also allows the information within unstructured text to be combined with other semantic facts ingested from elsewhere (for example, DBpedia, Wikipedia’s RDF version; or GeoNames, a public geospatial database in RDF).
Forward inferencing
Inferencing is the ability to take a set of assertions and extrapolate them to other assertions. You can perform backward inferencing and forward inferencing.
Here’s an example of backward inferencing: If Leo is the son of Adam, and Adam is the son of Cliff, then you can reason that Leo is the grandson of Cliff. You can “prove” a new fact by referencing others in the database. That is, in order to infer the new triple, you use information available earlier in the text.
Forward inferencing, on the other hand, is the process of inferring what may happen in the future (or be true in the future), when after analysing all existing information, all facts that can be inferred are inferred.
Forward inferencing is typically done at ingestion time, which does lead to slower performance, but it also means that you don’t have to do this work at query time. In this respect, forward inferencing is similar in effect to the denormalization approach of other NoSQL databases.
Forward inferencing does allow you to build quite sophisticated webs of stored and inferred facts, which users can employ as helpful shortcuts. A good example is a product recommendations — for instance, “You may like product x because you liked product y, and others who liked product y also liked product x.”
If you don’t create your inferencing rules carefully, you can end up in a situation where inferencing gets out of hand by creating billions of triples that may not, in fact, be accurate! This can lead to aborted triple ingestion, poor performance, or a significant waste of storage.
This proliferation can happen by the inferencing rules clashing. An example is if you infer that all objects with names are people, but in fact find that many cars and pets have names. You will have a lot of incorrect triples generated stating that all these cars and pets are people!
Tracking Provenance
Being able to prove that a decision was made or that a process was followed because of a certain set of facts is a difficult business. It generally takes weeks or months to trawl through content to discover who knew what and when and why things happened as they did.
However, triple stores store assertions so that they can be queried later. Graph stores go one step further by allowing you to take a subgraph and ask for other matching graphs that are stored.
One example of a provenance query is to find collusion in financial markets. If you see a pattern of traders at two investment banks communicating with each other prior to companies’ marketplace events, then you may see a pattern of fraud.
These patterns can be modeled as graphs of interconnected people, organizations, activities, and other semantic relationships such as “Adam called Freddie the Fraudster.”
Auditing data changes over time
The World Wide Web Consortium (W3C) has created a set of standards under the PROV Ontology (PROV-O) banner to track changes to data.
This ontology includes support for a variety of subject types, including people, organizations, computer programs, activities, and entities (typically content used as input or output).
For example, as a police force receives reports over time, an analysis can be updated. Each change is described, and the list of sources for the report increases to reflect the data the report is based on.
This information can, in turn, be used to create an action plan. A police risk assessment and threat analysis as well as an arrest plan can be based on this information prior to a raid.
By using the output of the police report analysis and a semantic query over provenance information, you can prove that the appropriate process was followed to support a raid on the premises of the “guilty” entity.
Building a Web of Facts
You can create a triple store that answers questions by using other peoples' data instead of just your own. The Resource Description Framework (RDF) standard was created to allow this web of interconnected data to be built. This approach is generally referred to as the Semantic Web.
There’s also a growing movement of open-data publishing. This movement has led governments, public sector organizations, research institutions, and some commercial organizations to open up their information and license it to be reused.
Combining the concepts of open-data publishing and RDF and SPARQL allows you to create Linked Open Data (LOD), which is published data under an open license that refers to other sources of linked open data on other websites.
A news article from the BBC might mention a place that’s linked to the GeoNames geographical RDF dataset. It could also mention a person with a profile on DBpedia — the RDF version of Wikipedia.
Over time, these links develop and are changed to produce a global Semantic Web of information. This is the culmination of Sir Tim Berners-Lee’s vision for a web of interconnected data that can be navigated by computers, just as the World Wide Web is navigated by humans using web browsers.
Taking advantage of open data
You can find many great datasets, ranging from the U.S. Geological Survey (USGS) data about places within the United States to information about Medicare spending per patient, or even to names of newborns since 2007 in the United States.
Not all open data is published according to RDF specifications; therefore, it isn’t ready to be stored immediately in a triple store. For example:
· Of the 108,606 datasets on the U.S. government’s open-data website (https://www.data.gov), only 143 are available in RDF, whereas 13,087 are available in XML, 6,149 are available in JSON, and 6,606 are available in CSV.
· At http://data.gov.uk (the UK’s equivalent to data.gov), of the 15,287 published datasets, 167 are available in RDF, 265 are available in XML, 3,186 are available in CSV, and 22 are available in JSON.
Many of these datasets are easy to convert to an RDF format, but the problem is you have to manually figure out how to do this conversion yourself, for every dataset encountered.
Currently, these publishing mechanisms are published only as static data files rather than as a SPARQL endpoint that can be queried and composed in other RDF data stores. You can search for a dataset, but not for the dataset itself. Hopefully, this will change in the future as open-data publishing develops.
Incorporating data from GeoNames
GeoNames is a database of geographic information that covers a vast range of information, including
· Continents, countries, counties, towns, and administrative boundaries
· Mountains, forests, and streams
· Parks
· Roads
· Farms
· Zip Codes
· Undersea locations
All this information is licensed under the Creative Commons Attribution version 3.0 license. This means you’re free to use the information so long as you clearly state who created it.
All the information in GeoNames is linked to a longitude and latitude point on the Earth, allowing for geospatial search and indexing.
Consider a text document that mentions a town. You can link this document to the town within GeoNames, which enables you to search for all documents in a country even if that document doesn’t explicitly contain the country’s name!
Places in GeoNames are also linked to their DBpedia entries, if one exists. This is a practical example of Linked Open Data in action.
Incorporating data from DBpedia
DBpedia contains a semantic extraction of information available within Wikipedia. It's a crowd-sourced effort created by volunteers and made available under both the Creative Commons Attribution-ShareAlike 3.0 license and the Free Documentation License.
As of this writing, DBpedia provides data on 4.58 million entities, with 583 million facts in 125 languages! It’s a huge resource, and it enables you to provide contextual information to help end users navigate through information.
This information is available for download in the N-Triples and N-Quad formats. A description of the DBpedia ontology is available in the Web Ontology Language (OWL) on its website (www.w3.org/2004/OWL).
Linked open-data publishing
Once you have your own data linked to other linked open-data sources, you may want to publish your own information. You can use a triple store to do so. If you have protected internal information, you may want to extract, or replicate, particular named graphs from your internal servers to a publically accessible server hosting “published” information.
You can use this publishing server to host flat file N-Triples for static downloads. You can also use it to provide a live SPARQL query endpoint to the web for others to interactively query and to download subsets of your data as they need it.
SPARQL, as a query language, allows you to join, at query time, remotely stored information from other SPARQL endpoints. This is a great way to perform ad hoc queries across multiple datasets. The following example is a SPARQL 1.1 Federated query to two different endpoints:
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT ?person ?interest ?known
WHERE
{
SERVICE <http://people.example.org/sparql> {
?person foaf:name ?name .
OPTIONAL {
?person foaf:interest ?interest .
SERVICE <http://people2.example.org/sparql> {
?person foaf:knows ?known . } }
}
}
You can find the entire example at www.w3.org/TR/sparql11-federated-query/#optionalTwoServices. The SPARQL 1.1 Federated Query document is a product of the whole of the W3C SPARQL Working Group.
Migrating RDBMS data
When performing entity enrichment on NoSQL databases, it’s common to pull reference information from relational database management systems.
Again, those clever people who contribute to W3C working groups have you covered, by way of a language called R2RML, which stands for RDB to RDF Mapping Language. This enables you to define transforms to convert a relational schema to a target RDF ontology. R2RML and related standards are available here: www.w3.org/2001/sw/rdb2rdf
In some situations, though, you want to quickly ingest information before performing data refactoring or inferencing. For example, you may want to store the original information so that you can assert PROV-O provenance information on the data before reworking it.
In this situation, you can use the Direct Mapping of relational data to RDF. You can find details on this standard here: www.w3.org/TR/rdb-direct-mapping
In the past, I’ve used direct mapping to perform a tongue-in-cheek demo to illustrate ingesting relational data into a triple store for inferencing and querying. You can find this video on YouTube at https://www.youtube.com/watch?v=4cUqMzsu0F4
Managing the Social Graph
Social relationships between people provide a classic example of how a relationship graph is used. It’s been said that all people on the planet are separated by only six degrees — that is, we’re all within six steps of knowing someone who knows someone, who knows someone . . .
Social graphs are important in several instances, including the following:
· Social networks: Include the likes of Facebook, Twitter, and LinkedIn.
· Professional organizations: To identify individuals or groups in large organizations with required expertise. Also can be used for organizational charts.
· Organized crime detection: Police and security organizations perform multi-year million-dollar investigations of organized crime networks with many points of contact and relationships.
· Family tree: Includes many parent, grandparent, uncle, and other relationships. Some assertions from such research may or may not be accurate. One source may list a George Fowler who may or may not be the same George Fowler in another source.
Managing this information has its challenges. Different approaches can be taken, depending on whether you’re describing relationships between people or whether you’re describing individuals and their activities.
Storing social information
You can find ontologies to model social data. One of the oldest, but still useful, models is Friend of a Friend (FOAF).
The FOAF RDF model gives individuals the ability to describe themselves, their interests, their contact details, as well as a list of people they know. This model includes Linked Open Data links to their contacts’ FOAF descriptions, when it’s known.
This data can be discovered and mined to produce spheres of influence around particular subject areas. The downside is that typically only Semantic Web computer scientists use FOAF to describe themselves on their websites!
When describing family trees, most people use the Genealogical Data Communication (GEDCOM) file format. This file format is designed specifically for this information rather than combining with other data. Using a common RDF-based expression would instead have allowed this data to be easily integrated into a triple store.
However, the Defense Advanced Research Projects Agency (DARPA) funded the DARPA Agent Markup Language (DAML) to model family relationships. Despite its spooky name, this project created an RDF expression of GEDCOM. You can find this standard documented atwww.daml.org/2001/01/gedcom/gedcom.
Performing social graph queries
Triple and graph stores can be used to perform queries on a social graph in several ways. For example, to
· Suggest a product to users based on similar items purchased by other people.
· Suggest that users add new friends based on their being friends of your existing friends.
· Suggest groups that users can be a member of based on existing group memberships, friends with similar group memberships, or others with similar likes and dislikes who aren’t in their network.
You can do the preceding by constructing SPARQL to enable users to do the following:
1. Find all customers who share the same or who have similar product purchases.
2. Order customers by descending value of the number of matches.
3. Find all products those people have ordered that the user hasn’t.
4. Order the list of products by descending value according to the total number of orders for each item across all similar customers.
Here’s an example of using SPARQL to find potential new friends in a social network (listing friends of my friends who are not me):
SELECT ?newfriend WHERE {
?me ex:username “adamfowler” .
?me ex:friend ?friend .
?friend ex:friend ?newfriend .
FILTER NOT EXISTS (?me ex:friend ?newfriend) .
FILTER NOT EXISTS (?me = ?newfriend ) .
} limit 10
Selecting friends based on second-level matches is pretty straightforward. To enable users to find the first ten people who are potential friends, you can use SPARQL as you did in the preceding example.
The preceding example shows traversal of a graph of relationships using SPARQL. Group membership suggestions use a similar query, except instead of traversing friends of friends, you traverse memberships of groups.