I need a multi-web crawler/spider made that will collect information based on the keywords I enter into the search field of the websites,
The crawlers need to be for the following sites:
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
-The first 3 sites require you to enter in a city and state, however I need the crawler to automatically collect information for ALL cities and states nationwide.
-I should be able to enter a keyword (for example: Marketing Services) and the crawler should collect all the companies that it found with that keyword. It needs to collect full information (company name, address, phone number, website url, and anything else)
-For [login to view URL]: I will enter the keyword and it needs to do a WHOIS search of the domains that it found and collect the full whois information (Company Name, Company address, Person's name, Phone number, Fax Number, Email address, and any other information). I should be able to limit the number of pages that it should crawl through by entering X pages to crawl.
The system needs to take out duplicate results based on containing the same PHONE NUMBER. I need to be able to export the results to a TEXT-TAB DELIMITED format in Microsoft Excel/CSV.
The final product needs to be displayed and usable in an easy-to-use GUI. So it needs to be user friendly.