Data scraping solution

Task is to create generic solution that can be extended to scrape data from thousands of sites. In this first phase, solution must: - plan architecture so that scraping data from a single site is treated as microtask - be extensible (ability to add new microtasks super easily is a key requirement) - proved working with static and javascript sites by having at least 50 microtasks configured - websites are commonly used enduser websites and amount of scraped data varies per site but is typically between 10-200 unique data entry points - scraped data saved in structured format (CSV) - support scheduling so that tasks can be configured to run in given interval (defined in seconds) Optional requirement - support parallel execution of micro-tasks so that they can be configured to run parallel in an distributed environment (multiple severs) End result is backend program that can (manually) be executed on command-line on Linux environment which saves scraped data to CSV-file (name given as parameter).