I need someone to build a bot to crawl thousands of pages of a classified ads website. This website maybe has an anti-crawl mechanism so our bot would have to use a few different IP adresses, have some kind of time management (to not crawl all pages at a time) and change user-agents as well.
I don't think this would be very difficult for someone who already made this kind of bot. Literally several hundred thousands of pages must be crawled. Each page containing maybe 3-4 fields to crawl.
The bot would run once every month. I don't need real time data.
All data should be fed to a MySQL database.