Crawl the web to find websites which use single sign-on

I need someone to write me two pieces Python scripts. The scripts work with mysql database.

Crawler script:

This page queries [url removed, login to view] (or other similar sites on web ranking) to discover popular websites and store the website URLs in table websites. An example table entry:

ID site_domain global_rank checked_for_sso

2 [url removed, login to view] 449 no

3 [url removed, login to view] 6476 no

4 [url removed, login to view] 760 no

Requirement: it is able to discover top 1 million websites

Analysis script:

This page get a website not analyzed (checked_for_sso field is “no”) from table websites, then visit this website and explore it to figure out what single sign on services it uses. If any single sign on services discovered, store the information in table sso. Also update checked_for_sso field in table websites to “yes” after the analysis is done. An example entry for table sso:

ID login_url supported_SSO

2 [url removed, login to view] facebook

3 [url removed, login to view] google

4 [url removed, login to view] facebook,google,yahoo

Requirement: false positive rate is 0, and false negative rate is less than 1%

Færdigheder: MySQL, Python

Se mere: websites work home, web python freelancer, smartsheet freelancer, smartsheet app, similar sites freelancer, services web freelancer, sears work home, python web freelancer, python script freelancer, python home work, python find, popular freelancer websites, google sites app scripts, google freelancer login, freelancer work find, freelancer web services, freelancer web ranking, freelancer top websites, freelancer similar websites, freelancer sign facebook

Om arbejdsgiveren:
( 12 bedømmelser ) Bloomington, United States

Projekt-ID: #5992764

11 freelancere byder i gennemsnit $625 for dette job


Hello, sir I'm interested with your project. I have good skill and my place is 8th in freelancer. I'll do it for you. I can do it well more than you could guess. :) If you give me t Mere

$773 USD in 10 dage
(85 bedømmelser)

Dear Client, I can help in your project. We have already experience of working on similar projects. Please see below to get idea of our experience: Amazon/Ebay Bots: Mere

$526 USD in 10 dage
(67 bedømmelser)

I am willing to discuss further about the project specifications and deliver the same to your needs .

$526 USD in 10 dage
(32 bedømmelser)

Hi, Hope you are doing well! We have gone through your requirement and we understand that you are looking for highly skilled, qualified, and experience Python development team for your project. Our developer Mere

$773 USD in 15 dage
(10 bedømmelser)

Dear Sir / Madam, Warm Greetings from Genpex IT Solutions. As soon as I saw your posting for a “Crawl the web to find websites which use single sign-on”. It is the perfect position for us .Our AIM to give the bes Mere

$526 USD in 10 dage
(2 bedømmelser)

Hello, I'm Anna - a project manager in a Russian-Canadian web development company - A2 Design. You can check our recent projects on our website We’d love to help you with yo Mere

$350 USD in 5 dage
(1 bedømmelse)

Hi, Iam interested in your project and I'll be happy to do that for you. I have rich experince in scrapping using curl regular expressions Dom and Selenium RC. I worked for and sear Mere

$400 USD in 10 dage
(4 bedømmelser)

Hi, You need a crawler which collects the most popular websites, checks whether the sites use a single sign on or not and stores the data in MySQL. If you are interested I will show you a demo of this project on Mere

$400 USD in 10 dage
(2 bedømmelser)

Hi. I'm able to write super fast crawler that will download thoushands of pages per second. Shall it crawl whole domain (~2000 pages per domain) or just one page?

$1666 USD in 14 dage
(0 bedømmelser)

i can do it. i am a database expert and script/python/c++/c#/java programmer. i have many developing experience of various script projects during five years. also i developed a various of simulation programs. than Mere

$578 USD in 4 dage
(0 bedømmelser)

I have 2 years experience specializing in website scraping with python.Auto post on website . I do a lot of sites like facebook, twitter, google, ... in the normal way or used in conjunction with multi-threading api do Mere

$355 USD in 9 dage
(0 bedømmelser)