The project is part of a site which is supposed to give a better answer to consumer needs regarding comparing prices and getting to better deals over the web.
The data is supposed to be the most basic corner stone of the site.
Scrapping of 10 sites for the same data.
Entering the scraped data to mango DB format with google translate
The scrapping scripts are supposed to run in the fly like ([url removed, login to view]
) so that the data should be always up to date means retrieved data will be stored in DB.
In addition the data should be achieved without the need for logging into the sites.
Looping the scripts is not part of the requirements for this project.
[url removed, login to view] multithreading or Python + Scrapy for scrapping the sites.
[url removed, login to view] - should be both speed and space.
The scripts should not consume minimal memory and run as fast as possible.
[url removed, login to view] programmer should be smart and think out of the box , take decisions , and make everything work as expected.
4.I will be available and expect consultation in case it is needed.
Time 2- 4 weeks budget 1500
Please do not bid if you do not have experience at least 2 years
33 freelancers are bidding on average $1477 for this job
Hi, Friend. I have enough experience in Java programming. I also have deep understanding about WebScraping. I think I can help you perfectly & asap. Please tell me your details. Thanks.
Hi, I can develop scrapy project for those websites. Each website will have separate crawler. I will add mongodb pipelines for scrapy to save items. Best regards, Ilshat