HDInsight Essentials, Second Edition (2015)
Preface
We live in a connected digital era and we are witnessing unprecedented growth of data. Organizations that are able to analyze big data are demonstrating significant return on investment by detecting fraud, improved operations, and reduced time to analyze with a scale-out architecture such as Hadoop. Azure HDInsight is an enterprise-ready distribution of Hadoop hosted in the cloud and provides advanced integration with Excel and .NET without the need to buy or maintain physical hardware.
This book is your guide to building a modern data architecture using HDInsight to enable your organization to gain insights from various sources, including smart-connected devices, databases, and social media. This book will take you through a journey of building the next generation Enterprise Data Lake that consists of ingestion, transformation, and analysis of big data with a specific use case that can apply to almost any organization.
This book has working code that developers can leverage and extend in order to fit their use cases with additional references for self-learning.
What this book covers
Chapter 1, Hadoop and HDInsight in a Heartbeat, covers the business value and the reason behind the big data hype. It provides a primer on Apache Hadoop, core concepts with HDFS, YARN, and the Hadoop 2.x ecosystem. Next, it discusses the Microsoft HDInsight platform, its key benefits, and deployment options.
Chapter 2, Enterprise Data Lake using HDInsight, covers the main points of the current Enterprise Data Warehouse and provides a path for an enterprise Data Lake based on the Hadoop platform. Additionally, it explains a use case built on the Azure HDInsight service.
Chapter 3, HDInsight Service on Azure, walks you through the steps for provisioning Azure HDInsight. Next, it explains how to explore, monitor, and delete the cluster using the Azure management portal. Next, it provides tools for developers to verify the cluster using a sample program and develop it using HDInsight Emulator.
Chapter 4, Administering Your HDInsight Cluster, covers steps to administer the HDInsight cluster using remote desktop connection to the head node of the cluster. It includes management of Azure Blob storage and introduces you to the Azure scripting environment known as Azure PowerShell.
Chapter 5, Ingest and Organize Data Lake, introduces you to an end-to-end Data Lake solution with a near real life size project and then focuses on various options to ingest data to a HDInsight cluster, including HDFS commands, Azure PowerShell, CloudExplorer, and Sqoop. Next, it provides details on how to organize data using Apache HCatalog. This chapter uses a real life size sample airline project to explain the various concepts.
Chapter 6, Transform Data in the Data Lake, provides you with various options to transform data, including MapReduce, Hive, and Pig. Additionally, it discusses Oozie and Spark, which are also commonly used for transformation. Throughout the chapter, you will be guided with a detailed code for the sample airline project.
Chapter 7, Analyze and Report from Data Lake, provides you with details on how to access and analyze data from the sample airline project using Excel Hive ODBC driver, Excel Power Query, Powerpivot, and PowerMap. Additionally, it discusses RHadoop, Giraph, and Mahout as alternatives to analyze data in the cluster.
Chapter 8, HDInsight 3.1 New Features, provides you with new features that are added to the evolving HDInsight platform with sample use cases for HBase, Tez, and Storm.
Chapter 9, Strategy for a Successful Data Lake Implementation, covers the key challenges for building a production Data Lake and provides guidance on the success path for a sustainable Data Lake. This chapter provides recommendations on architecture, organization, and links to online resources.
What you need for this book
For this book, the following are the prerequisites:
· To build an HDInsight cluster using the Azure cloud service, you will need an Azure account and a laptop with Windows Remote Desktop software to connect to the cluster
· For Excel-based exercises, you will need Office 2013/Excel 2013/Office 365 ProPlus/Office 2010 Professional Plus
· For HDInsight Emulator, which is suited for local development, you will need a Windows laptop with one of these operating systems: Windows 7 Service Pack 1/Windows Server 2008 R2 Service Pack 1/Windows 8/Windows Server 2012.
Who this book is for
This book is designed for data architects, developers, managers, and business users who want to modernize their data architectures leveraging the HDInsight distribution of Hadoop. It guides you through the business values of big data, the main points of current EDW (Enterprise Data Warehouse), steps for building the next generation Data Lake, and development tools with real life examples.
The book explains the journey to a Data Lake with a modular approach for ingesting, transforming, and reporting on a Data Lake leveraging HDInsight platform and Excel for powerful analysis and reporting.
Conventions
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "I have selected hdind and the complete URL is hdind.azurehdinsight.net."
Any command-line input or output is written as follows:
# Import PublishSettingsFile that was saved from last step
Import-AzurePublishSettingsFile "C:\Users\Administrator\Downloads\Pay-As-You-Go-Free Trial-11-21-2014-credentials.publishsettings"
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "You can select the desired configuration Two Head Nodes on an Extra Large (A4) instance included orTwo Head Nodes on a Large (A3) instance included."
Note
Warnings or important notes appear in a box like this.
Tip
Tips and tricks appear like this.