Learning Qlik Sense The Official Guide (2015)
Chapter 2. What is Data Discovery?
We've seen how the BI market is changing rapidly and how new demands have transformed the way that users look at analysis. In this chapter, we will examine the new discovery-based approach to business intelligence, which is rapidly emerging and has defined Qlik® Sense.
In this chapter, we will cover the following topics:
· The Qlik® philosophy
· The approach to data discovery
· The importance of the empowered user
· How a user really interacts with data
· The difference between traditional BI and data discovery
Why do we need data discovery?
Over the years, there have been many names of the different business intelligence methods and tools, such as:
· Executive Information Systems (EIS)
· Management Information Systems (MIS)
· Online Analytical Processing (OLAP)
· Decision Support Systems (DSS)
· Management reporting
· Ad hoc query and reporting
Do we really need an additional label for something that in principle is the same thing? The answer is yes.
There is a fundamental difference between older technologies and data discovery, and it is in the approach. Most of the preceding tools are oriented towards technology, but data discovery is not. Instead, data discovery is oriented towards people—towards the users who need the information in their daily work.
Most of the preceding tools were developed for a small, select number of decision-makers, but again, data discovery is not. Data discovery is for everyone.
Decisions are made at all levels in a company. Obviously, managers are decision-makers, but we sometimes forget that machine operators and receptionists are also decision-makers, albeit at a more local level. They also need information to make better decisions.
We, at Qlik, believe that information can change the world and that every user can contribute to this transformation. Everyone should easily be able to view data, navigate in data, and analyze data. Everyone should be able to experience that "a-ha" moment of discovery.
Data discovery is not just business intelligence. It is user-centric, dynamic, and empowering. And it is fun!
The empowered user
For the first couple of years in Qlik's history, the company was called QuikTech and the product was called QuikView. It was a game with words: the product name insinuated that you could view things quickly, and at the same time, the letters Q-U-I-K were an abbreviation for what we believed in: Quality, Understanding, Information, and Knowledge. These are shown in the following banner:
The initial QuikTech banner
Qlik believed that a business could improve its processes and product quality by empowering employees and encouraging them to engage in lifelong learning. And Qlik meant all employees—we saw everyone as a decision-maker, not just managers.
To get information from data was an important part of creating understanding, knowledge, and quality. We were inspired by the management trends of the time, especially by employee empowerment as described in the book, Moments of Truth by Jan Carlzon(President and CEO of Scandinavian Airlines), HarperBusiness (Swedish: Riv pyramiderna!).
Thus, the abbreviation was an early attempt to make a statement on values and it was there long before the genesis of the product. What the abbreviation stood for was really the ideological base when founding the company.
The company later changed its name to QlikTech, and the values statement was adjusted accordingly to look like what is shown in the following banner:
The second QlikTech banner
Today Qlik's mission statement is "Simplifying Decisions for Everyone, Everywhere". The words we use to describe our mission have changed slightly, from the general "Knowledge" and "Quality" to the more specific "Decisions", which is the main step in converting knowledge to quality.
The current mission statement is more to the point than our original values statement. Further, it includes the idea that all people are included, which is something we took for granted but failed to express in our initial values statement. In all aspects, the current mission statement is a very good description of what we stood for 20 years ago and what we still stand for today.
Although some things have changed since then, much remains the same: users are still often in a situation where they are unable to analyze their data—data that they have the right to see, or should have the right to see, in order to do a good job. Rigid systems, technical limitations, and poor user interfaces are usually the culprits.
However, people's expectations of software have changed dramatically during the last decade. Applications from Google and Apple invite users to interact with simple, friendly interfaces. Search bars, Like buttons, and touchscreens have transformed the way people explore, consume, and share information. Today, people want the same ease of use from their business tools as they get from their consumer tools at home.
The current trends such as the consumerization of software, performance improvements of hardware, usability improvements of software, mobile devices, social networks, and so on just accelerate this change. All these trends are reshaping user behavior. Yesterday, a user was a passive end user, but the user of tomorrow will be both able and demanding. They will demand tools that are fast, flexible, and dynamic. They will demand tools that they can use themselves. The empowered user is here to stay.
The interaction with data
The classic picture of business intelligence is that the user has one or several questions, and that the data holds the answers. So the problem boils down to creating a tool where the user can enter their questions, and the tool can return the answers.
However, this picture is incorrect. The truth is that the user does not always know the question initially. Or rather, if the user knows the question, they often already know the answer. So, the first thing the tool should do is to help the user find the questions.
Finding the questions is a process that involves exploring the data. It involves testing what you suspect but don't know for sure. It also involves discovering new facts. Further, it involves playing with data, turning it around, scrutinizing the facts, and formulating a possible question. You use your gut feeling as a source of ideas and you use the data to refine the ideas into knowledge; or to discard the ideas, if facts show that the ideas are wrong. You need to be able to play with the data, to turn facts around and look at them from different angles before you can say that you understand the data, and you need to understand the data before you can talk smartly about it.
When you have found a relevant question, you also need to be able to conduct an analysis to get a well-founded answer to the question.
Finally, the process involves presenting the answer to the question to other people as a basis for a decision or an action. The tool must support the entire process of going from ignorance to insight.
Hence, one major difference between data discovery and the more old-fashioned tools is that data discovery software supports the entire process—the process of coming from a blank mind, not knowing what you are looking for, all the way to attaining knowledge and taking action.
This is what data discovery is all about: helping you to prepare before you speak, act, or make a decision. It is the process of going from the darkness to the light, from the unknown to the known, from ignorance to insight. It is the process of going all the way from a blank mind to a substantiated argument.
The traditional business intelligence architecture
It is quite clear that users representing the business want the ability to ask and answer questions on their own so that they can make better decisions, but traditional business intelligence solutions aren't well-suited for user demands. Instead, it is common that the systems are created in a report-centric manner, where governance and system demands set the goals, rather than user demands. The solutions often have preconfigured dashboards, fixed drill-down paths, predefined queries, predefined views, and very little flexibility.
With traditional BI, the creation of the business intelligence solution often belongs to the IT organization, which has to do the following: create data models, establish a semantic layer, build reports and dashboards, and protect and control the data. Often, the creation of business intelligence solutions is not driven by user demands. The following figure depicts the traditional BI architecture:
When analyzing data, you want to set filters so that you can make selections, but with traditional tools, you often need to start at the top of predefined hierarchies. So instead of selecting a customer directly, you may need to answer this question: which market does this customer belong to?, then its country, and only then can you specify the customer.
Further, in the drill-down hierarchy, you are often limited to the choice of one or all. For example, you can look at either a single customer or all of them. The possibility of choosing two or three specific customers doesn't exist, unless this has been specifically predefined by the data model developer.
Numbers are often precalculated to ensure short response times, but this has a drawback that if the developer hasn't anticipated a specific calculation, the tool will not be able to show it.
Further, the architecture of the tool is often made in three layers; the stack. The first layer is the Extract, Transform, Load (ETL) layer, or the data load layer. The second is the Data Store / Engine layer, and the third is the User Interface (UI) layer. The three layers are different pieces of software, sometimes delivered by different software vendors.
These three layers also demand different skillsets. Often the ETL expert knows little or nothing about the UI software, and the UI expert knows little or nothing about the ETL.
The product stack in traditional BI
This architecture also leads to problems. When an application is built, the feedback comes from users trying to use the application. It could be that KPIs are incorrectly calculated or that dimensions or measures are missing. It could also mean that the user realizes that the initial requirements were incorrect or insufficient. The feedback could imply changes in the UI, or in the data model, or even in the ETL component.
This type of feedback is normal—it happens with all business intelligence tools. It only means that the development of applications is a process where you need to be agile and prepared. The expectation that you should be able to define an application completely and correctly prior to a prototype or an intermediate version is just unrealistic.
This is where the architecture leads to problems. In order for a project to be successful, you need to be able to implement change requests and new user demands with short notice, and this is extremely difficult since three different pieces of software and three different groups of people are involved. The distance between the user and the ETL component is just too great for efficient communication. Hence, traditional architecture leads to a broken process.
The Qlik way
Qlik has tried to solve all the drawbacks discussed in the preceding section by doing things differently.
First of all, you click and view. You don't need to formulate your question or tell the system more specifically what you want to look at. You just click, and by that, you say "Tell me more about that…". Then you look at the calculation, KPI, or field that might be interesting.
Color coding
The color coding defines the answer. Some things are associated with what you clicked on, and they remain white. Others that are not associated become gray. The color coding is for simplicity. The user quickly gets an overview and understands how things work.
Showing the excluded reveals the unexpected, creates insight, and creates new questions. Hence, the gray color is an important part of making the Qlik experience an associative one—a data dialog and an information interaction—rather than just a database query. Showing you that something is excluded when you didn't expect it means answering questions you didn't ask. This surprise creates new knowledge in a way that only a true data discovery platform can.
Freedom of data navigation
A user has total freedom to navigate through data and make any combination of selections. Any number of values can be selected. No drill-down paths need to be predefined. This allows the user to follow their own train of thought instead of someone else's. Start anywhere and just follow your intuition.
This total freedom when exploring data is really the core attribute of data discovery.
Calculation on demand
Further, no numbers need to be precalculated. QlikView® and Qlik® Sense calculate everything on demand, usually in a fraction of a second. The short response time allows the user to "have a conversation" with the data, where one answer leads to the next question, which in turn leads to next, and so on. Only this way can you interact with data so that you learn from it.
The developer does not need to anticipate all questions that the user will pose. All they need to do is to create a logical, coherent data model, and Qlik Sense will be able to answer the question correctly:
The stack (ETL-Data Store / Engine-UI) is replaced by a single integrated environment. This makes it possible to develop applications in close cooperation with the users, and it can often be done by the users themselves. Feedback is implemented instantaneously and the changes can be evaluated just seconds later. This shortens the development cycle and ensures that the application meets the user demands much sooner than it would otherwise.
This step-wise implementation is crucial for the success of a business intelligence project. It is also the core of modern agile methodologies that are used in all types of software development.
With Qlik Sense, all BI stack functions are integrated into one tool
Development of business intelligence applications must be done as close to the user as possible to enable user feedback and short development cycles. It does not necessarily imply self-service capability, although it is good if this capability exists.
With the introduction of Qlik Sense, the ground-breaking work continues by enabling a new class of users who are highly mobile and require greater self-service capabilities. In Qlik Sense, the self-service capability has become a core feature. Users can define new graphs and visualizations that the app developer didn't think of. This functionality empowers the users even further.
With Qlik Sense, it has also become easier to share your findings and communicate them. This is something that is necessary in all environments where human interaction is important, which is pretty much everywhere.
Data discovery – the next generation of BI
Data discovery is the future of business intelligence. With data discovery, users pursue their own path to insight, make discoveries collaboratively, and can arrive at a whole new level of decision-making. Users are not limited to predefined paths or precalculated numbers. They do not need to formulate questions ahead of time. They can interact with data, find the questions, ask what they need to ask, and explore up, down, and sideways rather than only drilling down in a predefined hierarchy.
Organizations may still need standardized reporting for many cases, but data discovery is the approach that ultimately fulfills the promise of business intelligence for everyone.
Summary
Data discovery is the inevitable consequence of demands from active users who want information from the ever-increasing amount of data. From the very beginning, the core of the Qlik philosophy was the empowered user. It affects both the view of how BI solutions should be developed and how the user interface of the tool should be designed.
In summary, data discovery is user-centric; it is BI for the empowered user. It means total freedom in how data is explored. It should be simple and have as few limitations as possible. Data discovery means a user-centric development process so that user feedback can be implemented instantaneously.
In the next chapter, we will look some of the other factors behind the development of Qlik Sense.