A web crawler that can pick up JS onclick events

I need a web crawler that can find links on a page and list them. Even links that are hidden by javascript onclick events.

It must

1) log the status code of the url given and any urls redirected through - example if given a url that redirects to another url with a 301 status code I need the 301 code and the 200 that it redirects to.

2) List the urls in a redirect chain if there is a chain.

3) Get all the links on the page given even ones hidden in onclick divs or other methods.

4) list all the rel, anchor text and image url elements for each link if they exist

5) follow redirects if required by meta redirects or [url removed, login to view] and list the urls in the redirect

6) We need to be able to run this from command line on a linux machine. I don't care too much what language but we need to be able to use it with php. Previously we were running HTML unit through shell_exec in php and then capturing what was echoed to the command line. Continuing like this is fine.

We had some luck with HTML unit but we have not got enough experience to get all our requirements.

Evner: Web Skrabning

Se mere: web-crawler, what is a web, what is a crawler, hidden web, scraping crawler, pick 3, web scraping image, web scraping linux, web redirect, redirect status code, command web, find hidden url, crawler javascript, scraping experience, javascript image onclick, window events php, web scraping code, html unit, example web scraping, crawler code, scraping web, image onclick javascript, methods events, web crawler html, javascript links given url page

Om arbejdsgiveren:
( 0 bedømmelser ) Stockport, United Kingdom

Projekt ID: #4058819

Tildelt til:


I have lots of experience with writing web automation software, please see PMB for examples of my previous projects related to web automation. Available to start immediately and finish as soon as possible. Best Rega Flere

£500 GBP in 10 dage
(20 bedømmelser)

8 freelancere byder i gennemsnit £390 på dette job


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

£250 GBP in 3 dage
(32 bedømmelser)

I look forward to discuss further and can deliver the project

£520 GBP in 12 dage
(29 bedømmelser)

Hi, Ready to start your work. Eagerly awaiting for your positive reply. Please check your inbox for further details. Thanks, Shaik.

£250 GBP in 5 dage
(25 bedømmelser)

Hello, I can do this work for you and I'm ready to start. Please see pmb for details. Regards Raul

£250 GBP in 7 dage
(11 bedømmelser)

Hello,Understood your scraping [login to view URL] check pmb for [login to view URL]

£250 GBP in 7 dage
(2 bedømmelser)

Scrapping/Parsing/Automated engine Experts here. Check the message with attached samples and contact us. SI Team.

£750 GBP in 10 dage
(2 bedømmelser)

Hi, Please check your PMB regards, Arun

£700 GBP in 5 dage
(0 bedømmelser)

I'm an expert Webbot, Netbot creator and a Professional webscraper. .NET/C# My webscraping skills can be found at [login to view URL] I'll scrape any data from any website.

£400 GBP in 5 dage
(0 bedømmelser)