2. Install Scrapy
3. Write Your First Script
4. Put It To Work!
For this tutorial, we’re going to write a Web Spider to scrape Stack Overflow Jobs.
1. What the Hell is Scraping, Anyway?
Web scraping (web harvesting or web data extraction) is data scraping used for extracting data from websites. Web scraping software may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser.
While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler.
What’s a “Web Spider”?
A Web crawler, sometimes called a spider, is an Internet bot which systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).