Build a data merging system - python + celery + redis + mongodb

Lukket Opslået 3 måneder siden Betalt ved levering
Lukket Betalt ved levering

I am looking for a skilled Python developer to build a database update system using RabbitMQ and MongoDB. The purpose of the system is to update our database on demand to ensure that it is always up to date.

The objective to merge candidate application data from three data sources - linkedin jobs, naukri and naukri rms. Each job will have a unique job_id and all the applicants from each data source will be tagged to one unique job_id.

[login to view URL] 1- A service will capture the data from each data source and upsert into each collection. eg. linkedin_jobs, naukri, naukri_rms. Since this service can run any number of times, the operation will always be upsert so that duplicates are removed

2. Stage 2- Upon completion of the upsert, there will be a collection called merged_data that will merge data from all three collections for a given job_id. This script will be based on a logic that we will provide. At the end of each upsert into stage1, the system will start a script to update the merged_data.

Your job will be to build and test these python+flask+celery+redis+mongodb scripts locally and then deploy and test them on AWS EC2.

Ideal Skills and Experience:

- Strong proficiency in Python programming

- Experience with Celery and RabbitMQ/ Redis and MongoDB

- Ability to design and implement an efficient and scalable system for small databases

- Familiarity with database optimization techniques

Frequency of Updates:

- The updates will be on demand, so the system should be able to handle updates as soon as they are requested.

Size of Database:

- The database being updated is small (less than 1GB), so the system should be optimized for efficient updates of smaller databases.

Python Elasticsearch Datasøgning NoSQL Couch & Mongo Redis

Projekt ID: #36684826

Om projektet

2 bud Remote projekt Aktiv 2 måneder siden

2 freelancere byder i gennemsnit ₹1040 timen for dette job


i can deliver you this..............Connect to me in chats.................................................

₹1080 INR in 7 dage
(21 bedømmelser)

I understand that you are looking for a skilled Python developer to build a database update system using Celery and RabbitMQ. With my extensive experience in Python, Redis, MongoDB and Celery, I believe I am the perfec Flere

₹1000 INR in 3 dage
(0 bedømmelser)