I need someone to write me two pieces Python scripts. The scripts work with mysql database.
This page queries [url removed, login to view] (or other similar sites on web ranking) to discover popular websites and store the website URLs in table websites. An example table entry:
ID site_domain global_rank checked_for_sso
2 [url removed, login to view] 449 no
3 [url removed, login to view] 6476 no
4 [url removed, login to view] 760 no
Requirement: it is able to discover top 1 million websites
This page get a website not analyzed (checked_for_sso field is “no”) from table websites, then visit this website and explore it to figure out what single sign on services it uses. If any single sign on services discovered, store the information in table sso. Also update checked_for_sso field in table websites to “yes” after the analysis is done. An example entry for table sso:
ID login_url supported_SSO
2 [url removed, login to view] facebook
3 [url removed, login to view] google
4 [url removed, login to view] facebook,google,yahoo
Requirement: false positive rate is 0, and false negative rate is less than 1%
11 freelancere byder i gennemsnit $625 for dette job
Hi. I'm able to write super fast crawler that will download thoushands of pages per second. Shall it crawl whole domain (~2000 pages per domain) or just one page?