Learning Apache Drill: Query and Analyze Distributed Data Sources with SQL

Get up to speed with Apache Drill, an extensible distributed SQL query engine that reads massive datasets in many popular file formats such as Parquet, JSON, and CSV. Drill reads data in HDFS or in cloud-native storage such as S3 and works with Hive metastores along with distributed databases such as HBase, MongoDB, and relational databases. Drill works everywhere: on your laptop or in your largest cluster.

In this practical book, Drill committers Charles Givre and Paul Rogers show analysts and data scientists how to query and analyze raw data using this powerful tool. Data scientists today spend about 80% of their time just gathering and cleaning data. With this book, you’ll learn how Drill helps you analyze data more effectively to drive down time to insight.

• Use Drill to clean, prepare, and summarize delimited data for further analysis
• Query file types including logfiles, Parquet, JSON, and other complex formats
• Query Hadoop, relational databases, MongoDB, and Kafka with standard SQL
• Connect to Drill programmatically using a variety of languages
• Use Drill even with challenging or ambiguous file formats
• Perform sophisticated analysis by extending Drill’s functionality with user-defined functions
• Facilitate data analysis for network security, image metadata, and machine learning

Table of Contents

Chapter 1. Introduction to Apache Drill
Chapter 2. Installing and Running Drill
Chapter 3. Overview of Apache Drill
Chapter 4. Querying Delimited Data
Chapter 5. Analyzing Complex and Nested Data
Chapter 6. Connecting Drill to Data Sources
Chapter 7. Connecting to Drill
Chapter 8. Data Engineering with Drill
Chapter 9. Deploying Drill in Production
Chapter 10. Setting Up Your Development Environment
Chapter 11. Writing Drill User-Defined Functions
Chapter 12. Writing a Format Plug-in
Chapter 13. Unique Uses of Drill
Appendix A. List of Drill Functions
Appendix B. Drill Formatting Strings