
Completed
Posted
Paid on delivery
Title: Urgent: MQTT to Hadoop Sales Data Pipeline with Hive Data Warehouse (Small Scope) We need an experienced Big Data developer to build a compact, end-to-end sales data pipeline. The scope is small and well defined, and we need it delivered quickly. We plan to award within hours to the right candidate. The project is fully self-contained. It needs no account access, credentials, invitations or files from our side. You can start right away in your own local environment. PROJECT OVERVIEW Build a small pipeline that ingests real-time sales events through MQTT, stores them in Hadoop, and organizes them in a Hive data warehouse for basic sales reporting. SCOPE OF WORK Part 1 - Data Generation (MQTT Publisher) Write a Python script that publishes simulated sales events to a local Mosquitto broker. Each event should contain order_id, timestamp, region, product, quantity and price. Part 2 - Data Ingestion (MQTT Subscriber to HDFS) Write a subscriber that consumes these events and writes them to HDFS. A single-node Hadoop setup is enough. Part 3 - Data Warehouse (Hive) Design a simple star schema in Hive with these tables: - fact_sales - dim_product - dim_region Load the ingested data into these tables. Part 4 - Sales Reporting Provide three SQL queries: - Total sales by region - Top 5 products by revenue - Daily sales totals DELIVERABLES - Complete source code (zip or rar) - A short README covering setup, execution steps and a brief description of the architecture - Screenshots showing the pipeline running and the query results TIMELINE Delivery is expected within 1 to 2 days of award. Please state your exact delivery time in your proposal. REQUIREMENTS - Hands-on experience with Hadoop (HDFS, Hive) - Working knowledge of MQTT and Python - Understanding of data warehousing concepts (star schema) - Clear, timely communication HOW TO APPLY Begin your proposal with the word "PIPELINE" and include: - A brief summary of your relevant experience - Your proposed approach in 2 to 3 lines - Your delivery timeline Generic or copy-paste proposals will not be considered.
Project ID: 40752050
77 proposals
Remote project
Active 5 days ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

Hi - Adelman here from Texas MQTT TO HADOOP SALES DATA PIPELINE - you need a small working pipeline from simulated MQTT sales events through HDFS and Hive reporting, delivered fast. I’d use Python with Mosquitto for event generation and subscription, write the incoming records into HDFS, then load them into a simple Hive star schema with fact_sales, dim_product, and dim_region. I’ll also provide the three requested Hive queries, README, and screenshots showing the complete flow. I can deliver the complete working package within 2 days, including local single-node Hadoop/Hive setup validation and end-to-end testing. Can you confirm whether you want Hive running on the same local single-node Hadoop environment, or through a separate Hive installation? Looking forward to work with you.
$300 USD in 1 day
3.0
3.0
77 freelancers are bidding on average $406 USD for this job

Hi There, Hope you are well. I have extensive experience with Hadoop, MQTT, and Python, with a solid understanding of data warehousing concepts. My approach would be to swiftly develop and deploy a small-scale data pipeline as per the outlined requirements. Delivery timeline will be provided upon award. "PIPELINE" Thanks in advance for the opportunity to working with you. Best Regards, Serazum
$300 USD in 1 day
7.6
7.6

Hello, With my extensive experience and skill set in Big Data Sales and Software Development, I can guarantee the successful and timely completion of your MQTT-Hadoop project. Over the past 15+ years, I have focused on building scalable, secure, and user-friendly solutions for clients from diverse industries. My expertise spans Hadoop (HDFS, Hive), MQTT, Python, along with proficiency in data warehousing concepts including star schemas. I am well-suited to handle all aspects of this project: from writing a Python script for simulating sales events, to designing a simple yet efficient star schema in Hive to load the data into relevant tables. My approach is always centered around providing a quality solution that meets and often exceeds the client's requirements. Additionally, you can count on me for clear and timely communication throughout the project phase. I aim to make this collaboration an impactful one for you as we successfully create a compact and efficient sales data pipeline to steer your business forward. Let's start straight away toward a timely delivery within your 1-2 day timeline! Thanks!
$350 USD in 4 days
5.3
5.3

PIPELINE The key requirement is a reliable end-to-end MQTT-to-HDFS sales pipeline with Hive star-schema loading and validated reporting queries, delivered within 1-2 days. I have strong hands-on experience with Python and Software Development and have solved similar problems involving event-driven ingestion, data warehousing, and reproducible local development environments. My approach would be to: * Review the required event structure, local Mosquitto flow, Hadoop/HDFS layout, and Hive execution model. * Identify the root cause of any ingestion or schema issues and determine what can be reused versus what needs to be changed. * Implement a clean, maintainable solution using Python, MQTT, HDFS, and Hive, including the publisher, subscriber, star schema, loading logic, and reporting SQL. * Test edge cases, reliability, performance, security, and integration behavior, including malformed events and duplicate delivery. * Deploy or integrate the fix without disrupting the existing system, then provide setup instructions, screenshots, and a concise architecture README. I will deliver the complete source package and verified results within 2 days. MQTT delivery can be at-least-once, so I will make ingestion and Hive loading restart-safe to reduce duplicate sales records. I prefer fixing the actual root cause rather than applying temporary patches, and I will keep the implementation simple, stable, and maintainable. I can start by preparing the local broker, Hadoop/HDFS paths
$600 USD in 2 days
5.1
5.1

PIPELINE The subscriber is the part that will make or break this. MQTT hands you one small message at a time, HDFS wants big files, and writing each sales event straight to HDFS leaves thousands of tiny files that Hive then struggles with. I'd buffer in the subscriber and flush batches into partitioned directories, so fact_sales stays queryable without a compaction step. Mosquitto, single-node Hadoop and Hive all run in Docker Compose, so you can reproduce the whole thing with one command. I'll pin the versions and note them in the README. M1: Compose stack and the Python publisher with simulated sales events, $211 M2: Subscriber writing to HDFS, plus the Hive star schema (fact_sales, dim_product, dim_region), $214 M3: The three reporting queries, README and screenshots, $215 The total is 640 USD. That price assumes Hadoop, Hive and Mosquitto run locally in Docker, with no existing cluster or environment of yours to integrate with. It also assumes you get source code, a README and screenshots from my own setup, not a deployment on your infrastructure, and that the schema and the three queries stay as specified, with no extra dimensions or dashboards. If any of that differs, I'll re-quote before any work starts. Send me the sample event format and the three query definitions and I can start today.
$640 USD in 2 days
4.7
4.7

PIPELINE Hello I can build the complete MQTT-to-Hadoop sales pipeline using Python, Mosquitto, HDFS, Hive, and SQL. I’ll create the MQTT publisher for realistic sales events, a subscriber that consumes and writes events into HDFS, then organize the data into a clean Hive star schema with fact_sales, dim_product, and dim_region. I’ll also provide the three requested reporting queries for regional sales, top 5 products by revenue, and daily totals. The delivery will include organized source code, a concise README with architecture and setup/run commands, plus screenshots showing the pipeline and query results. I can complete this compact scope within 2 days and keep the implementation reproducible in a local single-node Hadoop environment. Looking forward to work with you. Thanks
$300 USD in 2 days
4.3
4.3

Hi, Good day! I'm excited about the opportunity to contribute to MQTT-Hadoop Sales Data Pipeline Development and deliver work that meets the highest standards of quality. I've worked on numerous projects similar to MQTT-Hadoop Sales Data Pipeline Development. I'd be happy to share relevant samples from my portfolio that demonstrate my capabilities in Python, SQL, Big Data Sales, Hadoop, Software Development, SQLite, Hive, Spark, MQTT, Big Data. Looking forward to bringing my Python, SQL, Big Data Sales, Hadoop, Software Development, SQLite, Hive, Spark, MQTT, Big Data expertise to MQTT-Hadoop Sales Data Pipeline Development. I'm ready when you are! Regards, Manisha
$500 USD in 7 days
4.3
4.3

PIPELINE Most of my data work has been Python pipelines in Docker - Celery/Redis ingest workers, time-series data flowing into VictoriaMetrics with Grafana on top, and ESP32 firmware on the device side I'd put Mosquitto, single-node Hadoop and Hive in one docker-compose so the whole thing comes up with one command. The paho publisher would send JSON events, the subscriber would batch them into date-partitioned files on HDFS, and a Hive staging table over that folder would load the dims and the fact table, with the 3 report queries on top Exact delivery time is in the bid details
$340 USD in 1 day
4.5
4.5

Hive and Spark are where small MQTT pipelines usually fall apart under load, not the ingestion side. I would set up an MQTT broker subscriber feeding into HDFS, land it in Hive tables partitioned for your sales queries, and keep Spark jobs lean given the scope. Can start today, working pipeline in 5 days. Budget and timeline above are starting estimates from the post. We will firm them up once we go through the actual data volume and query needs together. Want me to send a quick scope doc?
$450 USD in 10 days
4.2
4.2

PIPELINE To develop the MQTT to Hadoop sales data pipeline, I will initiate by creating a Python script for publishing simulated sales events to a Mosquitto broker, ensuring each event captures the essential details like order_id, timestamp, region, product, quantity, and price. Next, I’ll implement the MQTT subscriber to ingest these events directly into HDFS using a single-node Hadoop setup, followed by constructing a star schema in Hive for organizing the data, which includes tables for fact_sales, dim_product, and dim_region. Finally, I’ll provide the requested SQL queries to deliver insights on total sales by region, top products by revenue, and daily sales totals. I can complete this within 1 day of project award as requested. Regards, Ali
$350 USD in 7 days
3.9
3.9

⚠️ If you're not happy, you don’t pay. ⚠️ Hi Client, thank you for checking my proposal and sharing the detailed project brief. I understand your project needs. I can build your sales data pipeline using Python, MQTT, Hadoop, and Hive with a scalable and efficient design. I will deliver: - Python script for publishing simulated sales events to Mosquitto - MQTT subscriber that writes events to HDFS - Star schema design in Hive with fact_sales, dim_product, and dim_region tables - Three SQL queries for sales reporting: total sales by region, top 5 products by revenue, and daily sales totals - Complete source code in a compressed file - README with setup, execution steps, and architecture overview You will also receive: - Detailed documentation for easy onboarding I am confident I can execute your vision professionally and efficiently. Looking forward to discussing timeline and next steps. Best regards, Chirag Pipal
$400 USD in 14 days
4.0
4.0

Greetings, PIPELINE. The project requires developing a compact sales data pipeline that utilizes MQTT for real-time data ingestion, storing the data in Hadoop, and structuring it within a Hive data warehouse. I would start by creating a Python script to publish simulated sales events, followed by a subscriber that will consume these events and store them in HDFS. Finally, I’ll design a star schema in Hive for effective reporting. I have hands-on experience with Hadoop, MQTT, and Python, which allows me to seamlessly integrate each part of the pipeline. My understanding of data warehousing concepts will ensure that the schema is both efficient and effective for reporting. Looking forward to getting started! Best regards, Saba Ehsan
$450 USD in 4 days
4.1
4.1

Hello, PIPELINE: Your compact sales pipeline requires reliable MQTT event generation, HDFS ingestion, Hive star-schema modeling, and validated reporting queries. I understand the need for a self-contained implementation that can be demonstrated locally without external access. I will build the Python publisher and subscriber around a local Mosquitto broker, write structured events to HDFS, create and populate fact_sales, dim_product, and dim_region in Hive, then validate the three requested reports. The package will include clean source code, setup instructions, architecture notes, and execution screenshots. Similar work includes data ingestion services, Python automation, SQL warehouse modeling, and backend integrations where correctness and reproducible local setup were essential. I will keep the implementation compact and easy to inspect, with clear separation between generation, ingestion, transformation, and reporting. Delivery will be completed within 2 days of award, including testing the full pipeline and documenting each command required to run it. I will provide the complete archive and screenshots of successful execution and query results. Best regards, Bohdan
$650 USD in 2 days
4.0
4.0

Hello. PIPELINE I have hands-on experience building similar Python-based data pipelines involving MQTT, Hadoop/HDFS, Hive, data ingestion, and warehouse reporting, so I can handle this compact scope end-to-end without turning a small pipeline into a science experiment. 1. Will the single-node Hadoop/Hive environment be local with standard HDFS and Hive configurations, or do you have a preferred Hadoop/Hive version? 2. Should the simulated MQTT events run continuously for testing, or would you prefer a fixed dataset/batch size for demonstrating the complete pipeline? Your answers will greatly help me understand the project precisely; I will then implement the flow as MQTT Publisher → Subscriber → HDFS → Hive star schema → reporting queries, with clear setup steps and validation screenshots. I can start immediately and deliver the complete source code, README, working pipeline, Hive tables, three reporting queries, and screenshots within **1 day**, giving you a fast, clean, and fully demonstrable solution. Let's discuss this project further Best regards, Jeff Chong.
$500 USD in 7 days
4.1
4.1

PIPELINE Hi, my background is Python data pipelines and reporting for clients, recent ones were a data feed rebuilt from source for Qlik Sense reports, a Grafana dashboard over VictoriaMetrics for many tenants, and a Celery and Redis queue layer in Docker How I'd do it: Mosquitto, Hadoop and Hive in one docker-compose so it starts with one command on your machine too, a paho-mqtt publisher, and a subscriber that writes small batches into date folders on HDFS. A Hive external table would sit over those raw files and fill the dims and the fact table with INSERT SELECT, then the report queries run on top of that Exact delivery time is in the bid fields, I can start right after award
$260 USD in 1 day
4.2
4.2

Nice to meet you , My name is Anthony Muñoz, I express my interest in working on your project after carefully reading the requirements and concluding that they match my area of knowledge and skills. I am currently the lead engineer for the IT agency DSPro and I have more than 10 years of experience in the field. I have successfully completed a large number of similar jobs and I consider your project to be a challenge in which I would like to work and be able to make it a reality. Please feel free to contact me, it will be my pleasure to help you. I greatly appreciate the time provided and I remain attentive to any questions or concerns. Greetings
$397 USD in 7 days
3.8
3.8

PIPELINE ✌✌✌✌✌ High Quiality ✌✌ Fast Reply Per Day ✌✌ Success Business ✌✌✌✌✌ Hello, I built a past streaming-data pipeline where ingestion latency stayed under 2 seconds and test-event delivery to storage achieved 99.9% reliability. For your project, I’ll build a Python MQTT publisher/subscriber using Mosquitto, write sales events into HDFS, then create the Hive star schema with `fact_sales`, `dim_product`, and `dim_region`. I’ll also provide the 3 requested Hive queries, screenshots, and a concise README covering setup and execution. Delivery timeline: 1 day You’ll receive complete source code, Hive scripts, sample data, README, and screenshots showing the end-to-end pipeline and query results. Thanks, Dejan
$300 USD in 5 days
3.4
3.4

PIPELINE Hi there, I’m Bilal, a Software Engineer with expertise in building scalable, reliable data pipelines and backend solutions. With strong experience in Python, MQTT, Hadoop, HDFS, Hive, SQL, data warehousing, and ETL pipelines, I can build your complete local sales pipeline from MQTT event generation through HDFS ingestion and Hive-based reporting. My focus is always on: * Python MQTT publisher/subscriber with structured sales events * Reliable ingestion into a single-node HDFS environment * Clean Hive star schema using fact_sales, dim_product, and dim_region * Accurate revenue and sales reporting queries * Reproducible setup with clear README instructions * Screenshots demonstrating the running pipeline and query results My proposed approach is to generate simulated events through Mosquitto, consume and persist them into HDFS, then transform/load the data into Hive tables for reporting. I’ll keep the architecture compact, understandable, and easy to run locally. I’d love to discuss the project further and can provide the complete source code, setup documentation, and execution proof. Feel free to contact me anytime—I look forward to collaborating with you! Best regards, Muhammad Bilal
$250 USD in 2 days
3.2
3.2

PIPELINE Hello! I’m excited about the opportunity to build the sales data pipeline you need. I understand that you require a compact solution that ingests real-time sales events via MQTT, stores them in Hadoop, and organizes the data in Hive for reporting. I have extensive experience in Big Data development, particularly with Hadoop, Hive, and MQTT. My background includes building similar data pipelines using Python, which equips me to handle the specific requirements of this project effectively. To achieve your goals, I will: - Develop a Python script to simulate sales events and publish them to a Mosquitto broker. - Create a subscriber to ingest these events into HDFS in a single-node Hadoop setup. - Design a star schema in Hive and load the data into the appropriate tables. - Write the necessary SQL queries for sales reporting. I can deliver the complete source code, README, and documentation within 24 hours of project award. I look forward to discussing any further details and starting this project immediately. Thank you for considering my proposal!
$250 USD in 7 days
3.0
3.0

Hi, PIPELINE. Before I touch code I want to confirm your local setup, what version of Hadoop and Hive you're running, whether Mosquitto is already installed, and whether this is single-node on your machine or a VM I need to spin up myself. I expect to find nothing blocking, this is a scope I've built pieces of before and it's straightforward on a single node. My plan: Python script publishes fake sales events over MQTT, a subscriber picks them up and lands them in HDFS as the raw layer, then Hive tables on top with fact_sales linked to dim_product and dim_region. I'll write the three reporting queries against that schema and grab screenshots of the pipeline running end to end plus the query output. The README will walk through setup and how to rerun it. I can have this done in 2 days. Ready to start now if you confirm the environment details above. Best, Emrah
$450 USD in 2 days
2.8
2.8

Hey , Good afternoon! I’ve carefully checked your requirements and really interested in this job. I’m full stack node.js developer working at large-scale apps as a lead developer with U.S. and European teams. I’m offering best quality and highest performance at lowest price. I can complete your project on time and your will experience great satisfaction with me. I’m well versed in React/Redux, Angular JS, Node JS, Ruby on Rails, html/css as well as javascript and jquery. I have rich experienced in Python, SQL, Software Development, MQTT, SQLite, Big Data Sales, Spark, Hive, Big Data and Hadoop. For more information about me, please refer to my portfolios. I’m ready to discuss your project and start immediately. "PIPELINE" Looking forward to hearing you back and discussing all details.. Best Regards
$555 USD in 6 days
2.8
2.8

Kumasi, Ghana
Payment method verified
Member since Dec 23, 2010
$30-250 USD
$30-250 USD
$30-250 USD
$10-30 USD
$30-250 USD
$10-30 USD
min $50 USD / hour
$20-80 CAD / hour
₹12500-37500 INR
₹750-1250 INR / hour
₹600-1500 INR
₹750-1250 INR / hour
$15-25 AUD / hour
₹400-750 INR / hour
₹75000-150000 INR
$8-15 USD / hour
$30-250 USD
₹2500000-5000000 INR
$30-250 USD
$30-250 USD
₹750-1250 INR / hour
$25-50 USD / hour
$30-250 USD
₹12500-37500 INR
₹750-1250 INR / hour