I am looking for someone to create 1+ multi-threaded spiders/crawlers in C# 2.0. The crawler will operate in a Windows environment. It will need to be able to handle tokens, javascript, login credentials, cookies, etc.
The main purpose of the spider is to crawl the video site and save data like category/title/tag/link/comments/run-time/etc. The database can be MySQL, MS SQL Server 2005 Express, Access, or just a flat-file. I also need it to be able to 1.) determine the total number of threads and 2.) set some type of crawling frequency to prevent being throttled.
This will most likely be the beginning of many spider projects.