1) Perform a google search, per website, based on a mass list of websites in the program. Ie site:[login to view URL]
2) Scan through every single link, on every single result page, and make sure it isn't omitting any useful information. (It has to be every html page on the website)
3) Scrape the URL link and the entire page source code for the URL into a sqlite database for later viewing
Basically it needs to search google from a list of sites and scrape the URL and the source code of the URL. It will be handling a massive amount of information so no newbies please.
Python or Ruby preferred.
19 freelancere byder i gennemsnit $136 på dette job
Hii i am an experienced python developer,i would love to work on your [login to view URL] scraping skills are so good,i can easily scrap both internal and external links from a [login to view URL] free to message me ,Thanks
Hello Dear, I am interested in your project and invite you to discuss in details. Please reply so that I can show you my past result. Looking forward to hear from you. Thanks, Page O.