I need a Class developed which I can plug into a large application. This class will need to be able to make an HTTP request to retrieve the HTML from a given URL, and then parse the HTML into a object model I will provide. There is a main page which will need to be crawled, which contains a list of links which will each need to be visited. Additionally, when on each sub-link, you will need to visit one link of each sub-link.
It's a structured navigation pattern, and the problem probably lends itself to some copy-and-paste.
This must be a C#.Net application, compiled against the .Net 4.0 Framework. Also, while not mandatory, it is suggested that the solution use Regular Expressions.