Web Scraping with Python: Collecting More Data from the Modern Web, 2nd Edition

If programming is magic then web scraping is surely a form of wizardry. By writing a simple automated program, you can query web servers, request data, and parse it to extract the information you need. The expanded edition of this practical book not only introduces you web scraping, but also serves as a comprehensive guide to scraping almost every type of data from the modern web.

Part I focuses on web scraping mechanics: using Python to request information from a web server, performing basic handling of the server’s response, and interacting with sites in an automated fashion. Part II explores a variety of more specific tools and applications to fit any web scraping scenario you’re likely to encounter.

• Parse complicated HTML pages
• Develop crawlers with the Scrapy framework
• Learn methods to store data you scrape
• Read and extract data from documents
• Clean and normalize badly formatted data
• Read and write natural languages
• Crawl through forms and logins
• Scrape JavaScript and crawl through APIs
• Use and write image-to-text software
• Avoid scraping traps and bot blockers
• Use scrapers to test your website

Table of Contents

Part I. Building Scrapers
Chapter 1. Your First Web Scraper
Chapter 2. Advanced Html Parsing
Chapter 3. Writing Web Crawlers
Chapter 4. Web Crawling Models
Chapter 5. Scrapy
Chapter 6. Storing Data

Part II. Advanced Scraping
Chapter 7. Reading Documents
Chapter 8. Cleaning Your Dirty Data
Chapter 9. Reading And Writing Natural Languages
Chapter 10. Crawling Through Forms And Logins
Chapter 11. Scraping Javascript
Chapter 12. Crawling Through Apis
Chapter 13. Image Processing And Text Recognition
Chapter 14. Avoiding Scraping Traps
Chapter 15. Testing Your Website With Scrapers
Chapter 16. Web Crawling In Parallel
Chapter 17. Scraping Remotely
Chapter 18. The Legalities And Ethics Of Web Scraping