If you are the best web scraping expert on Freelancer, we want to hear from you!
We need to scrape 2 million products PER HOUR on a leading ecommerce website.
As a web scraping expert, you will come up with the best solution for fetching the most recent price/stock changes for 2 million items and allowing our existing software to access your data easily perhaps via a JSON API.
The solution you provide should:
- Be scalable so that now it will monitor approx 2 million items per hour, but it could grow more in future to 3/4 million maybe.
- It MUST be able to scan that many items within 1 hour. So the technology you use must be scalable, e.g. it could use more threads / more proxy addresses etc to scale accordingly.
- Your system should also be modular / reusable so that it could be used on OTHER websites such as Costco / Walmart / HomeDepo etc.
- Previous experience with this kind of project is desirable.
- The solution you come up with should also be cost effective. E.g. some proxy networks we have used in the past charge fees based on bandwidth and since our target website pages are quite large, it is quickly cost ineffective to scrape alot of data every hour, so your solution needs to take into account the bandwidth needs.
We are looking to hire an expert ASAP so we look forward to your proposals.
Please tell us about any experience and how much time you have available to help with this and similar projects as we also plan to scrape many other stores.