Lukket

Write some Software

News Aggregator/Scraper/Bot

We need to build an infrastructure capable of scraping/indexing approximately 50-100 news websites.

The bots/scrapers should be equipped to quickly request, extract and parse news article URLs as they are published to the various news websites - in near 'real time'. The major focus of this project is to ensure that 1) News content/URLs are parsed and entered into a Database as fast as possible (within minutes of publication), and 2) that ALL news content/article URLs are captured (avoid missing content). We are looking to build a robust, easily scaled news aggregator system of bots/scrapers that can easily handle the load of 10,000 - 20,000 news articles per 24 hours, and where adding *new* sources of news (publishers, websites) is reasonably easy to do. We are open to ideas on how this is achieved.

The scraper/bot should capture the following information:

- Unique article URL

- Headline

- Body of text

- Author name

- Publisher URL (IE: [url removed, login to view])

The indexed information should be entered into an SQL database with relevant fields, unique article_ID and tables according to the above indexed information.

A list of the relevant news websites to be indexed can be obtained on request. They are primarily Australian, New Zealand, US and European news websites. All English language, all properly formatted.

We utilise Linode servers (Ubuntu OS) and operate an SQL database. We would prefer the scraper/bots to be completed in Node.js.

We have a Proxy Rotator that can manage 'requests' from the scraper/bots, so as to 'mask' multiples of visits to news websites.

*** Please note: we are seeking developers with strong experience in scraping/content extraction automation. Our primary focus is to ensure that ALL new news articles added to a news websites from the start point are captured, AND that all news articles are captured as quickly as possible. This content needs to be collected 24/7, and within 'minutes' of publication at the news website. Please only post a proposal or express interest if you believe you have the necessary skills to complete this task. ***

Færdigheder: node.js, PHP, Software Arkitektur

Se mere: write content easily, write a news article, write an article of publication, software developers websites, proxy sources, proposal to write content, proposal on how to write a project, project proposal how to write, php software developers, js write, information of how to write an article, how write project proposal, how to write the proposal of a project, how to write the project proposal, how to write proposal project, how to write project proposal, how to write news article, how to write article on believe, how to write a proposal project, how to write a proposal on a project, how to write a project proposal, how to write a news article, how to write an article to be published, how to start to write an article, how do we write article

Om arbejdsgiveren:
( 0 bedømmelser ) Australia

Projekt-ID: #5988770

26 freelancere byder i gennemsnit $3150 for dette job

esolzsales

Need to discuss the project in more details Please let me know if you have some time to discuss Regards ESolz

$5154 USD in 75 dage
(248 bedømmelser)
9.5
buraqtech

1. Vollks Australian Online Store In these days we are complete re-designing and reimplementing an online store for our Australian client in which we are using X-Cart latest stable release to implement his all require Mere

$2551 USD in 35 dage
(88 bedømmelser)
7.9
A2Design

Hello, I'm Anna - a project manager in a Russian-Canadian web development company - A2 Design. You can check our recent projects on our website [url removed, login to view] We excel at what we do becaus Mere

$3000 USD in 30 dage
(39 bedømmelser)
7.8
qualifiedcoders

Hi, We have read your requirements and would like to discuss further as we have strong hands in web, desktop & mobile application development & have in depth experience in Java scripting using Node.js, [url removed, login to view] etc Mere

$2105 USD in 20 dage
(22 bedømmelser)
7.3
workspaceit

Hello, Let me introduce myself. I am Tahsinul Alam, completed Masters in Software Engineering now working as one of the project manager of Workspace Infotech, software/Outsourcing firm located in Dhaka, Bangladesh. Mere

$2319 USD in 30 dage
(36 bedømmelser)
7.0
onelinewebdeUK

Thanks for your consideration, if you would like to discuss the project further do get in touch and I’ll give you a call.

$3157 USD in 40 dage
(14 bedømmelser)
7.0
fattahaabdul

It is possible. i have 8+ years of experience. Can we discuss the project. Please initiate a chat with me so that we can discuss the project at a broader level. Why you should hire me- 1. I have a very good comm Mere

$12371 USD in 30 dage
(18 bedømmelser)
6.9
inavigator

Hello Sir, We have designed and developed many Software,websites,Apps which are currently running online as business. For those websites/project we used ASP.NET,PHP,Wordpress,Joomla,Drupal,Magento etc We have Mere

$3157 USD in 30 dage
(11 bedømmelser)
6.7
helmot

Edit: IF YOU TELL ME ABOUT WHAT EXACTLY YOU WANT TO DO, I may be able to do what you want with just $1500 with unlimited number of news websites! I have +10 years of experience in Python programming. If you take Mere

$1500 USD in 10 dage
(55 bedømmelser)
6.6
daffodilsoftware

Joyent Partners [url removed, login to view] . 1) Learning Management System [url removed, login to view] 2) Pet Social Networking Reuse Market User Authentication Pet Profile Mere

$3000 USD in 25 dage
(15 bedømmelser)
6.3
w3esolutions

Greetings! We have studied your requirements and understand your needs very well. We are well aware of this sort of Scrapping apps to use the online bots to your desired list of websites and do your job with accura Mere

$2736 USD in 45 dage
(11 bedømmelser)
6.2
WorkXpressPaaS

!! **We are a US company with an 13 year history of developing cloud-based custom software solutions for a diverse client list**!! After reviewing your post, I am confident that we can develop and deploy a custom solut Mere

$5154 USD in 30 dage
(1 bedømmelse)
5.6
vijaywebsol

PHP [url removed, login to view] [url removed, login to view] [url removed, login to view] [url removed, login to view] [url removed, login to view] [url removed, login to view] Mere

$2268 USD in 20 dage
(35 bedømmelser)
5.2
drudev

Hello. I am a web developer with 11+-year experience . I work remotely on DigitalRay company - [url removed, login to view] - LA (USA). Technical knowledges: 1PHP JavaScript Ruby RoR ASP.Net C#, Java, Python, Zend, CakePHP, Symph Mere

$1500 USD in 30 dage
(17 bedømmelser)
4.8
gkws

Hello Sir/ Mam, Thanks for posting the project on Freelancer, this project exactly matches with our technical strengths. We have more than 30 plus retail and agency clients and have successfully completed over 100 Mere

$2061 USD in 25 dage
(14 bedømmelser)
4.6
aekpani

Dear Valued Customer, This is a placeholder bid and timeline which we will adjust once we have your precise scope of work which would enable us to assess your requirements and advise an optimum solution that tailore Mere

$2222 USD in 45 dage
(11 bedømmelser)
4.6
rakeshyadav91

offering clones and scripts and dirt cheap prices get started now: School Management - $230 [url removed, login to view] Dating/Match Making - $250 skadate Mail Marketing - $250 Mere

$2157 USD in 30 dage
(9 bedømmelser)
4.1
avkiev

Предложение еще не подано

$3508 USD in 3 dage
(5 bedømmelser)
4.0
phamtech211

My team is really interested in this project as it is exactly within our scope of expertise: We are WEB application design & development experts. Please kindly visit our website [url removed, login to view] to learn more about us and Mere

$6444 USD in 60 dage
(7 bedømmelser)
3.9
umarbsse

Hi As a Software Engineer and programmer i can do this simple scrapping task within 1-5 days.I have 2 years scrapping experience and ready to start that project right [url removed, login to view] you have any questions you may [url removed, login to view]

$1500 USD in 5 dage
(13 bedømmelser)
3.8