Learning Spark SQL

Key Features

• Learn about the design and implementation of streaming applications, machine learning pipelines, deep learning, and large-scale graph processing applications using Spark SQL APIs and Scala.
• Learn data exploration, data munging, and how to process structured and semi-structured data using real-world datasets and gain hands-on exposure to the issues and challenges of working with noisy and “dirty” real-world data.
• Understand design considerations for scalability and performance in web-scale Spark application architectures.

Book Description

In the past year, Apache Spark has been increasingly adopted for the development of distributed applications. Spark SQL APIs provide an optimized interface that helps developers build such applications quickly and easily. However, designing web-scale production applications using Spark SQL APIs can be a complex task. Hence, understanding the design and implementation best practices before you start your project will help you avoid these problems.

This book gives an insight into the engineering practices used to design and build real-world, Spark-based applications. The book’s hands-on examples will give you the required confidence to work on any future projects you encounter in Spark SQL.

It starts by familiarizing you with data exploration and data munging tasks using Spark SQL and Scala. Extensive code examples will help you understand the methods used to implement typical use-cases for various types of applications. You will get a walkthrough of the key concepts and terms that are common to streaming, machine learning, and graph applications. You will also learn how such systems are architected and deployed for a successful delivery of your project. Finally, you will move on to performance tuning, where you will learn practical tips and tricks to resolve performance issues.

What you will learn

• Familiarize yourself with Spark SQL programming including working with DataFrame/Dataset API and SQL.
• Perform a series of hands-on exercises with different types of data source including CSV, JSON, Avro, MySQL, and MongoDB.
• Perform data quality checks, data visualization, and basic statistical analysis tasks.
• Perform data munging tasks on publically available datasets.
• Learn to use Spark SQL and SparkR for typical data science tasks.
• Learn key performance-tuning tips and tricks in Spark SQL applications
• Learn to identify cases where Spark SQL can be used in large-scale application architectures.

Table of Contents

Chapter 1. Getting Started With Spark Sql
Chapter 2. Using Spark Sql For Processing Structured And Semistructured Data
Chapter 3. Using Spark Sql For Data Exploration
Chapter 4. Using Spark Sql For Data Munging
Chapter 5. Using Spark Sql In Streaming Applications
Chapter 6. Using Spark Sql In Machine Learning Applications
Chapter 7. Using Spark Sql In Graph Applications
Chapter 8. Using Spark Sql With Sparkr
Chapter 9. Developing Applications With Spark Sql
Chapter 10. Using Spark Sql In Deep Learning Applications
Chapter 11. Tuning Spark Sql Components For Performance
Chapter 12. Spark Sql In Large-Scale Application Architectures