Elasticsearch: The Definitive Guide (2015)

Preface

The world is swimming in data. For years we have been simply overwhelmed by the quantity of data flowing through and produced by our systems. Existing technology has focused on how to store and structure warehouses full of data. That’s all well and good—until you actually need to make decisions in real time informed by that data.

Elasticsearch is a distributed, scalable, real-time search and analytics engine. It enables you to search, analyze, and explore your data, often in ways that you did not anticipate at the start of a project. It exists because raw data sitting on a hard drive is just not useful.

Whether you need full-text search, real-time analytics of structured data, or a combination of the two, this book introduces you to the fundamental concepts required to start working with Elasticsearch at a basic level. With these foundations laid, it will move on to more-advanced search techniques, which you will need to shape the search experience to fit your requirements.

Elasticsearch is not just about full-text search. We explain structured search, analytics, the complexities of dealing with human language, geolocation, and relationships. We will also discuss how best to model your data to take advantage of the horizontal scalability of Elasticsearch, and how to configure and monitor your cluster when moving to production.

Who Should Read This Book

This book is for anybody who wants to put their data to work. It doesn’t matter whether you are starting a new project and have the flexibility to design the system from the ground up, or whether you need to give new life to a legacy system. Elasticsearch will help you to solve existing problems and open the way to new features that you haven’t yet considered.

This book is suitable for novices and experienced users alike. We expect you to have some programming background and, although not required, it would help to have used SQL and a relational database. We explain concepts from first principles, helping novices to gain a sure footing in the complex world of search.

The reader with a search background will also benefit from this book. Elasticsearch is a new technology that has some familiar concepts. The more experienced user will gain an understanding of how those concepts have been implemented and how they interact in the context of Elasticsearch. Even the early chapters contain nuggets of information that will be useful to the more advanced user.

Finally, maybe you are in DevOps. While the other departments are stuffing data into Elasticsearch as fast as they can, you’re the one charged with stopping their servers from bursting into flames. Elasticsearch scales effortlessly, as long as your users play within the rules. You need to know how to set up a stable cluster before going into production, and then be able to recognize the warning signs at three in the morning in order to prevent catastrophe. The earlier chapters may be of less interest to you, but the last part of the book is essential reading—all you need to know to avoid meltdown.

Why We Wrote This Book

We wrote this book because Elasticsearch needs a narrative. The existing reference documentation is excellent—as long as you know what you are looking for. It assumes that you are intimately familiar with information-retrieval concepts, distributed systems, the query DSL, and a host of other topics.

This book makes no such assumptions. It has been written so that a complete beginner—to both search and distributed systems—can pick it up and start building a prototype within a few chapters.

We have taken a problem-based approach: this is the problem, how do I solve it, and what are the trade-offs of the alternative solutions? We start with the basics, and each chapter builds on the preceding ones, providing practical examples and explaining the theory where necessary.

The existing reference documentation explains how to use features. We want this book to explain why and when to use various features.

Elasticsearch Version

The explanations and code examples in this book target the latest version of Elasticsearch available at the time of going to print—version 1.4.0—but Elasticsearch is a rapidly evolving project. The online version of this book will be updated as Elasticsearch changes.

You can find the latest version of this book online.

You can also track the changes that have been made by visiting the GitHub repository.

How to Read This Book

Elasticsearch tries very hard to make the complex simple, and to a large degree it succeeds in this. That said, search and distributed systems are complex, and sooner or later you have to get to grips with some of the complexity in order to take full advantage of Elasticsearch.

Complexity, however, is not the same as magic. We tend to view complex systems as magical black boxes that respond to incantations, but there are usually simple processes at work within. Understanding these processes helps to dispel the magic—instead of hoping that the black box will do what you want, understanding gives you certainty and clarity.

This is a definitive guide: we help you not only to get started with Elasticsearch, but also to tackle the deeper more, interesting topics. These include Chapter 2, Chapter 4, Chapter 9, and Chapter 11, which are not essential reading but do give you a solid understanding of the internals.

The first part of the book should be read in order as each chapter builds on the previous one (although you can skim over the chapters just mentioned). Later chapters such as Chapter 15 and Chapter 16 are more standalone and can be referred to as needed.

Navigating This Book

This book is divided into seven parts:

§ Chapters 1 through 11 provide an introduction to Elasticsearch. They explain how to get your data in and out of Elasticsearch, how Elasticsearch interprets the data in your documents, how basic search works, and how to manage indices. By the end of this section, you will already be able to integrate your application with Elasticsearch. Chapters 2, 4, 9, and 11 are supplemental chapters that provide more insight into the distributed processes at work, but are not required reading.

§ Chapters 12 through 17 offer a deep dive into search—how to index and query your data to allow you to take advantage of more-advanced concepts such as word proximity, and partial matching. You will understand how relevance works and how to control it to ensure that the best results are on the first page.

§ Chapters 18 through 24 tackle the thorny subject of dealing with human language through effective use of analyzers and queries. We start with an easy approach to language analysis before diving into the complexities of language, alphabets, and sorting. We cover stemming, stopwords, synonyms, and fuzzy matching.

§ Chapters 25 through 35 discuss aggregations and analytics—ways to summarize and group your data to show overall trends.

§ Chapters 36 through 39 present the two approaches to geolocation supported by Elasticsearch: lat/lon geo-points, and complex geo-shapes.

§ Chapters 40 through 43 talk about how to model your data to work most efficiently with Elasticsearch. Representing relationships between entities is not as easy in a search engine as it is in a relational database, which has been designed for that purpose. These chapters also explain how to suit your index design to match the flow of data through your system.

§ Finally, Chapters 44 through 46 discuss moving to production: the important configurations, what to monitor, and how to diagnose and prevent problems.

There are three topics that we do not cover in this book, because they are evolving rapidly and anything we write will soon be out-of-date:

§ Highlighting of result snippets: see Highlighting.

§ Did-you-mean and search-as-you-type suggesters: see Suggesters.

§ Percolation—finding queries which match a document: see Percolators.

Online Resources

Because this book focuses on problem solving in Elasticsearch rather than syntax, we sometimes reference the existing documentation for a complete list of parameters. The reference documentation can be found here:

http://www.elasticsearch.org/guide/

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates emphasis, and new terms or concepts.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

TIP

This icon signifies a tip, suggestion.

NOTE

This icon signifies a general note.

WARNING

This icon indicates a warning or caution.