Machine Learning with Spark, 2nd Edition

Key Features

• Get to the grips with the latest version of Apache Spark
• Utilize Spark’s machine learning library to implement predictive analytics
• Leverage Spark’s powerful tools to load, analyze, clean, and transform your data

Book Description

This book will teach you about popular machine learning algorithms and their implementation. You will learn how various machine learning concepts are implemented in the context of Spark ML. You will start by installing Spark in a single and multinode cluster. Next you’ll see how to execute Scala and Python based programs for Spark ML. Then we will take a few datasets and go deeper into clustering, classification, and regression. Toward the end, we will also cover text processing using Spark ML.

Once you have learned the concepts, they can be applied to implement algorithms in either green-field implementations or to migrate existing systems to this new platform. You can migrate from Mahout or Scikit to use Spark ML.

By the end of this book, you will acquire the skills to leverage Spark’s features to create your own scalable machine learning applications and power a modern data-driven business.

What you will learn

• Get hands-on with the latest version of Spark ML
• Create your first Spark program with Scala and Python
• Set up and configure a development environment for Spark on your own computer, as well as on Amazon EC2
• Access public machine learning datasets and use Spark to load, process, clean, and transform data
• Use Spark’s machine learning library to implement programs by utilizing well-known machine learning models
• Deal with large-scale text data, including feature extraction and using text data as input to your machine learning models
• Write Spark functions to evaluate the performance of your machine learning models

Table of Contents

1. Getting Up and Running with Spark
1. Installing and setting up Spark locally
2. Spark clusters
3. The Spark programming model
4. SchemaRDD
5. Spark data frame
6. The first step to a Spark program in Scala
7. The first step to a Spark program in Java
8. The first step to a Spark program in Python
9. The first step to a Spark program in R
10. Getting Spark running on Amazon EC2
11. Configuring and running Spark on Amazon Elastic Map Reduce
12. UI in Spark
13. Supported machine learning algorithms by Spark
14. Benefits of using Spark ML as compared to existing libraries
15. Spark Cluster on Google Compute Engine – DataProc
16. Summary

2. Math for Machine Learning
1. Linear algebra
2. Gradient descent
3. Prior, likelihood, and posterior
4. Calculus
5. Plotting
6. Summary

3. Designing a Machine Learning System
1. What is Machine Learning?
2. Introducing MovieStream
3. Business use cases for a machine learning system
4. Types of machine learning models
5. The components of a data-driven machine learning system
6. An architecture for a machine learning system
7. Spark MLlib
8. Performance improvements in Spark ML over Spark MLlib
9. Comparing algorithms supported by MLlib
10. MLlib supported methods and developer APIs
11. MLlib vision
12. MLlib versions compared
13. Summary

4. Obtaining, Processing, and Preparing Data with Spark
1. Accessing publicly available datasets
2. Exploring and visualizing your data
3. Processing and transforming your data
4. Extracting useful features from your data
5. Summary

5. Building a Recommendation Engine with Spark
1. Types of recommendation models
2. Extracting the right features from your data
3. Training the recommendation model
4. Using the recommendation model
5. Evaluating the performance of recommendation models
6. FP-Growth algorithm
7. Summary

6. Building a Classification Model with Spark
1. Types of classification models
2. Extracting the right features from your data
3. Training classification models
4. Using classification models
5. Improving model performance and tuning parameters
6. Additional features
7. Summary

7. Building a Regression Model with Spark
1. Types of regression models
2. Evaluating the performance of regression models
3. Extracting the right features from your data
4. Training and using regression models
5. Improving model performance and tuning parameters
6. Summary

8. Building a Clustering Model with Spark
1. Types of clustering models
2. Extracting the right features from your data
3. K-means – training a clustering model
4. K-means – evaluating the performance of clustering models
5. Effect of iterations on WSSSE
6. Bisecting KMeans
7. Bisecting K-means – training a clustering model
8. Gaussian Mixture Model
9. Summary

9. Dimensionality Reduction with Spark
1. Types of dimensionality reduction
2. Extracting the right features from your data
3. Training a dimensionality reduction model
4. Using a dimensionality reduction model
5. Evaluating dimensionality reduction models
6. Summary

10. Advanced Text Processing with Spark
1. What’s so special about text data?
2. Extracting the right features from your data
3. Using a tf-idf model
4. Evaluating the impact of text processing
5. Text classification with Spark 2.0
6. Word2Vec models
7. Word2Vec with Spark ML on the 20 Newsgroups dataset
8. Summary

11. Real-Time Machine Learning with Spark Streaming
1. Online learning
2. Stream processing
3. Online learning with Spark Streaming
4. Online model evaluation
5. Structured Streaming
6. Summary

12. Pipeline APIs for Spark ML
1. Introduction to pipelines
2. How pipelines work
3. Machine learning pipeline with an example
4. Summary