I gang

scraper to scrape vbulletin forum and record specific data about threads

Need a simple scraper created that will scrape a vbulletin forum.

Actions scraper will complete:

Every 10 minutes: Scrapes a subforum page that lists all threads in that subforum sorted by thread creation date (decending), adds new thread records to a watchlist as they are created.

Every 30 seconds: Scrapes a subforum page that lists all threads in that subforum sorted by last reply (decending), records the time, views, and replies for all threads on the watchlist that appear on that page.

Must include proxy support, rotating through a user entered list


- user entered list of proxies

- user entered URLSs (2) to scrape


-scraper must keep a log of every time it scrapes, recording the time it scraped the page, also the number of views and replies at that time for each thread on the watchlist.

-export the log in csv format.

goal: to be able to record the rate of new posts per 30 seconds, and rate of new replies per 30 seconds, of a thread that is getting tons of views and replies.

Evner: Forum Software, MySQL, Software Arkitektur, vBulletin, Web Skrabning

Se mere: vbulletin scraper threads, web forum scraping, simple scraping software, log page, time record, threads, scraper software, scrape a web, page scrape, recording web page, export data web page, date scraper, forum proxy, web proxies, list web proxies, number scraper, simple web proxy, scraper csv, web scraper proxy, web data scraping software, rate scraping, proxies vbulletin, scraper vbulletin, web page scrape, software data scraping web

Om arbejdsgiveren:
( 65 bedømmelser ) Chicago, United States

Projekt ID: #4078051