Web Scraping with Python: Collecting Data from the Modern Web

Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once.

Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice.

• Learn how to parse complicated HTML pages
• Traverse multiple pages and sites
• Get a general overview of APIs and how they work
• Learn several methods for storing the data you scrape
• Download, read, and extract data from documents
• Use tools and techniques to clean badly formatted data
• Read and write natural languages
• Crawl through forms and logins
• Understand how to scrape JavaScript
• Learn image processing and text recognition

Table of Contents

Part I. Building Scrapers
Chapter 1. Your First Web Scraper
Chapter 2. Advanced HTML Parsing
Chapter 3. Starting to Crawl
Chapter 4. Using APIs
Chapter 5. Storing Data
Chapter 6. Reading Documents

Part II. Advanced Scraping
Chapter 7. Cleaning Your Dirty Data
Chapter 8. Reading and Writing Natural Languages
Chapter 9. Crawling Through Forms and Logins
Chapter 10. Scraping JavaScript
Chapter 11. Image Processing and Text Recognition
Chapter 12. Avoiding Scraping Traps
Chapter 13. Testing Your Website with Scrapers
Chapter 14. Scraping Remotely
Appendix A. Python at a Glance
Appendix B. The Internet at a Glance
Appendix C. The Legalities and Ethics of Web Scraping