I gang

Data collection and connection

We are looking for a freelancer that can support us in requesting huge datasets (up to 500 Mio lines) from APIs, enrich them with more data from other APIs and store it in a database or file. Therefore you should know how to do such things very effective and fast, since we cant wait weeks to get the things performed. We are flexible in which programming language you work, but it should run on a linux server at the end.

The project will be extended and can take a few weeks with up to 200 hours in total. First task is to get all domain lists from [login to view URL] and save them in the format domain | tld in a text file. Then we want all of these domains resolved with their ip adress. Please let me know how you would technically solve this and how many hours it would take you.

Evner: Linux, MySQL, PHP, Python, Software Arkitektur

Se mere: data collection websites, email data collection, data collection cartoons, php, mysql, python, linux, codeigniter, angular.js, cakephp, mongodb, laravel, scrapy, joomla data collection, data collection website, research data collection, web data collection, cdr data collection project, iphone data collection application, data processing scanned images forms freelancer

Om arbejdsgiveren:
( 12 bedømmelser ) Cologne, Germany

Projekt ID: #17494981

Tildelt til:

matmunk

Hi, I'm very interested in working on your project. I have experience working with data sets of this size from previous projects, and I am able to deliver quickly. The first task I would solve by fetching the lis Flere

€25 EUR / time
(1 bedømmelse)
2.2

22 freelancere byder i gennemsnit €26/time for dette job

kchg

hi there, I just checked your proposal. I am Senior PHP and Laravel Developer with 9+ years strong practical experience in building various kinds of web projects. Especially I have good experience with PHP, Laravel, Flere

€30 EUR / time
(434 bedømmelser)
9.6
mmadi

Hi plusserver, I have gone through your project "Data collection and connection" and could assure you the best of the solution. I am taking it as an opportunity and ready to deliver my best. please check our fr Flere

€21 EUR / time
(43 bedømmelser)
7.0
widadsaghir1993

Hello sir. As a full stack developer, i'm glad to see your post. I can help you well and can deliver you perfect result. Please contact me for your job. Regards

€30 EUR / time
(74 bedømmelser)
6.6
ferozstk

Hello, After reading your project details I believe I'm suitable for this project. As I'm expert on this kind of project with more than 5 years experience. Please feel free to contact me. I am looking forward Flere

€18 EUR / time
(86 bedømmelser)
6.3
ambyt

Hello There, Greeting of the day! As per your project requirement, would be able to scrap data from a database. I can use PHP that would be suitable for Linux. Technically I would be able to resolve the issue an Flere

€18 EUR / time
(21 bedømmelser)
5.8
atulawasthi2

We have team of experts have excellent knowledge in Database like mysql and website design Html , Css , Wordpress , Javascript ,Java , Jquery ,PHP and various other website design and development technologies. We h Flere

€18 EUR / time
(20 bedømmelser)
5.5
AlexxxRu

Hello! This is REST API, so there can't any special optimizations. I think the best choice will be using a simple map-reduce algorithm. So basic idea will be: 1. Create multiple threads. In this kind of task, it shoul Flere

€20 EUR / time
(11 bedømmelser)
5.1
omscoders

Hi, We can use Python for grabing data from API, and use mongo DB as destination, the combination give high performance in large data management tasks, it also run in linux server we are an IT company from India, Flere

€30 EUR / time
(20 bedømmelser)
5.3
ozo42

Hi. Pro account is needed for api access (50/month). I do this kind of stuff on aws so it depends on you how much are you willing to pay for bandwidth, storage, etc.

€36 EUR / time
(4 bedømmelser)
4.3
€30 EUR / time
(5 bedømmelser)
4.5
€18 EUR / time
(35 bedømmelser)
4.7
Vedtechnology

I have good team for website design and development, I am terribly excited for the chance to figure with you in accomplishing your goals. We have developed thousands of websites in many programming languages for client Flere

€30 EUR / time
(42 bedømmelser)
4.2
ryancasteel511

Hi, there. I'm a senior software engineer and developer, I'm very interested in your project. I read your description and I understand your requirement exactly. I have rich experience and knowledge in Server m Flere

€30 EUR / time
(2 bedømmelser)
4.3
sku5551ed6fd9643

Hi I have have done it before , Kindly connect with your requirement. I have years of experience and expertise in a gamut of technologies that include: • Php, Magento, Laravel, Wordpress, Android & ios etc • Flere

€30 EUR / time
(11 bedømmelser)
3.3
€30 EUR / time
(9 bedømmelser)
3.0
teamspirit3

Hi, as an expert in software development, I am very interested in your project since I have rich experiences in database management. Almost all of apps use its own database to store data which is needed in the program. Flere

€30 EUR / time
(0 bedømmelser)
0.0
€30 EUR / time
(0 bedømmelser)
0.0
Jebas5161

I am an Web Scrapper doing for many websites as freelancer. delivering into excel database through spider or phyton i can do your project.

€20 EUR / time
(0 bedømmelser)
1.7
dataexplorer

Hi there, I'm a Data entry,Web Scraping, Excel and Digital Marketing Expert. I have read your project details which you have stated in the project "Data collection and connection" . I'll go through and manually copy & Flere

€30 EUR / time
(0 bedømmelser)
0.0
SOrginal

Hello, I can write a PHP script that reads the outcome of api and writes to a database such as mysql. And than another script that uses "$ip = gethostbyname('[login to view URL]');" to get ip. I can create numerous typ Flere

€30 EUR / time
(0 bedømmelser)
0.0