As a company we have some droplets hosted on Digital Ocean and we have already a very long list of websites and domains (and subdomains) that need to be crawled: from every url, the crawler need to fetch webpage title and metatags (if any).
In order to do so, there is a crawler that is already there, [login to view URL] but it is quite hard for our team to deploy it, install it and run it.
This job requires the installation of BuBing crawler into the droplet, the correct setup in order to not get blocked by webpages and run it. After that, for each page the title and metatags need to be saved into a file that will be then parsed by one of our scripts and given to the databases.
Long story short: install, setup and run BuBing given a list of urls.
7 freelancere byder i gennemsnit €184 på dette job
Hello, I can install, configure and setup the BuBING Web Crawler in the digital ocean server. Please let me know if you are interested so we can discuss the project further. Waiting for your reply. Thank you