Crawl the web to find websites which use single sign-on

I need someone to write me two pieces Python scripts. The scripts work with mysql database.

Crawler script:

This page queries [url removed, login to view] (or other similar sites on web ranking) to discover popular websites and store the website URLs in table websites. An example table entry:

ID site_domain global_rank checked_for_sso

2 [url removed, login to view] 449 no

3 [url removed, login to view] 6476 no

4 [url removed, login to view] 760 no

Requirement: it is able to discover top 1 million websites

Analysis script:

This page get a website not analyzed (checked_for_sso field is “no”) from table websites, then visit this website and explore it to figure out what single sign on services it uses. If any single sign on services discovered, store the information in table sso. Also update checked_for_sso field in table websites to “yes” after the analysis is done. An example entry for table sso:

ID login_url supported_SSO

2 [url removed, login to view] facebook

3 [url removed, login to view] google

4 [url removed, login to view] facebook,google,yahoo

Requirement: false positive rate is 0, and false negative rate is less than 1%

This is small project. My budget is around $300. Please don't bid more than $400.

Færdigheder: MySQL, Python

Se mere: www the freelancer com, www freelancer web page, www freelancer login com, www freelancer in login, www freelancer in facebook, www freelancer id, www freelancer com sign, www freelancer com find freelancer, www freelancer com b, www find freelancer com, what to write on freelancer, what other freelancer sites, websites to work from home, web python freelancer, web freelancer com, web app freelancer, web analysis services, top www freelancer com, to find freelancer, to find a freelancer, smartsheet freelancer, smartsheet app, sites similar freelancer com, similar sites to freelancer, sign out freelancer

Om arbejdsgiveren:
( 12 bedømmelser ) Bloomington, United States

Projekt-ID: #5992764

10 freelancers are bidding on average $630 for this job


Hello, sir I'm interested with your project. I have good skill and my place is 8th in freelancer. I'll do it for you. I can do it well more than you could guess. :) If you give me t Mere

$773 USD in 10 dage
(183 bedømmelser)

Hello, I'm Anna - a project manager in a Russian-Canadian web development company - A2 Design. You can check our recent projects on our website [The administrator removed this message for encouraging communicati Mere

$350 USD in 5 dage
(18 bedømmelser)

Dear Sir / Madam, Warm Greetings from Genpex IT Solutions. As soon as I saw your posting for a “Crawl the web to find websites which use single sign-on”. It is the perfect position for us .Our AIM to give the bes Mere

$526 USD in 10 dage
(37 bedømmelser)

Dear Client, I can help in your project. We have already experience of working on similar projects. Please see below to get idea of our experience: Amazon/Ebay Bots: [url removed, login to view] Mere

$526 USD in 10 dage
(69 bedømmelser)

Hi, Iam interested in your project and I'll be happy to do that for you. I have rich experince in scrapping using curl regular expressions Dom and Selenium RC. I worked for [url removed, login to view] and [url removed, login to view] sear Mere

$400 USD in 10 dage
(12 bedømmelser)

I am willing to discuss further about the project specifications and deliver the same to your needs .

$526 USD in 10 dage
(46 bedømmelser)

Hi, Hope you are doing well! We have gone through your requirement and we understand that you are looking for highly skilled, qualified, and experience Python development team for your project. Our developer Mere

$773 USD in 15 dage
(15 bedømmelser)

Hi, You need a crawler which collects the most popular websites, checks whether the sites use a single sign on or not and stores the data in MySQL. If you are interested I will show you a demo of this project on Mere

$400 USD in 10 dage
(2 bedømmelser)

Hi. I'm able to write super fast crawler that will download thoushands of pages per second. Shall it crawl whole domain (~2000 pages per domain) or just one page?

$1666 USD in 14 dage
(0 bedømmelser)

I have 2 years experience specializing in website scraping with [url removed, login to view] post on website . I do a lot of sites like facebook, twitter, google, ... in the normal way or used in conjunction with multi-threading api do Mere

$355 USD in 9 dage
(0 bedømmelser)