Spark with Python

Nowadays the internet is an integral part of our life, right from the waking moment we indulge in the world of the internet like creating a Facebook post or watch a YouTube video or so, and in this process we tend to create data. And think of it as the entire human population participating in this process of creating data every day, every minute and every second, now that would be a lot of data. Ok, now storage is an issue but the bigger issue is managing this data, it would be difficult and confusing to handle this data and to get some insights from this data to improvise the user experience and facilitate the society by providing them with the precise information which they require. But the question is how do we handle this data or how do we get the insights from this data?

Before answering that let us virtually visit a hospital and there we see patients waiting in long queues and paying lump some money to avail various medical services, with the amount of medical historical data that is available to us, how can we handle this and get some insights from this data which would, in turn, help the patients in need of these services get it faster and avail it for cheap. We can achieve this by making the diagnostics easier for doctors or making the medical equipments function better or so and all this can be done by handling the respective medical data and finding some insights. In this similar fashion we can go about finding insights for various problems in society and addressing problems in various industries like aviation, transportation, and automobile and so.

Now we understand the importance of data and the need to handle and process it. Hence, in order to handle and process it we need some tools which would help us perform various operations on data and one such powerful tool which can help us in this process is Apache Spark. Therefore, in this book we will learn about Apache Spark, how to handle the data with Apache Spark using Spark’s DataFrames, and also learn how to obtain insights and make predictions using Machine Learning with Spark.

This book is designed in such a manner where it starts from the scratch by understanding the fundamentals, then going through the Step-by-Step installation process of Spark, brushing up our Python Skills for Spark, working with data in Spark and finally entering into the Machine Learning section with Spark.

This book can be easily followed by anyone with or without any programming background, but on the completion of this book, I am sure my readers will be confident to write programs using the python language and would also be in a position to write Machine Learning scripts using python and spark. Since, each and every concept or topic is demonstrated using code snippets and its outputs, it would be really easy to follow and execute the same.