I have a set of urls that are like this:
[url removed, login to view]
(which redirects to [url removed, login to view])
The valid values for file are not incremental. I have a list (in the millions) of valid ones. I can prepare the list as a text file, sqlite, whatever we want. The idea is to start the script in a threaded manner, download 10,000 files, zip them, download another 10,000, zip them, and so on until the list is exhausted.
Want to get this done asap.
24 freelancers are bidding on average $30/hour for this job
Hello, shouldn't be a problem. What will be done with the urls that failed to download? Written to another file/database/whatever to try later? Any authentication necessary? KR, Oliver
Hello, I have around 9+ yrs of experience in perl/python/C/C++/C++11. I worked in reputed companies like IBM, Alcatel-Lucent. etc. I wish to take up this project and deliver it on time. Thanks Nagaraj
Sir , I have already had a python script for automatically downloading urls(for scratch a ebook site, links are listed in file), I can modify it to fit your demand ASAP Thank you:-)