Software
by
hardcorefs
Example of a multi-threadded distributed web-crawler. 1. >15 million pages a day per computer. 2. Capacity limited only by disk space available. 3. Does not re-crawl known pages, unless asked
About Me
IT and development jobs handled efficiently.
