To write SSIS Task Script or Application that crawl to all related links and pages on the websites (Online Store). The extracted data to be split by every 100k records per file and per eCommerce website.
Some of the information that will be extract are: Product name, description, price, currency, detail description, images links, the seller information, product location, and etc.
The scripts should be written in a very effective way, as everyday, millions of records will be required to be extracted .
In total the number of website to be extracted is 13 websites.
The detail of the requirement will be provided if candidate is shortlisted.
The scripts or application will run on auto pilot mode. Triggered by scheduler, and automatically export to csv or excel.