
Lukket
Slået op
Betales ved levering
Project Overview We are looking for a highly experienced Full-Stack Architect / Big Data Engineer / Search Infrastructure Expert to build a large-scale web application similar to Volza, capable of handling: 10+ Terabytes of structured trade data Trillions of rows Sub-second search performance Advanced filtering & analytics Production-grade scalability This is not a basic CRUD web app. We are building an enterprise-grade data intelligence platform. Core Objective =============== If a user searches for: Product name HS code Importer / Exporter name Country Shipment date range Address Port Any combination of filters The results must return in seconds (ideally sub-second) — even with trillions of records. Expected Architecture Expertise =========================== We expect the developer/team to propose and implement a scalable architecture such as: Distributed data storage Columnar database optimization Partitioning & indexing strategies Query acceleration techniques Caching layers Parallel query execution Horizontal scaling Suggested Tech Stack (Open to Better Suggestions) ========================================== Backend Python (FastAPI / Django) or Node.js Go (optional for performance-critical services) Database Options ClickHouse (preferred) Apache Druid Elasticsearch BigQuery / Redshift Any distributed columnar DB Frontend React / [login to view URL] Advanced filtering UI Data grid with pagination & lazy loading Infrastructure Kubernetes / Docker Load balancing CDN Caching (Redis) Object storage for raw data Required Features ================ Advanced Search Engine Full-text search Multi-filter query builder Auto-suggestions Fuzzy matching Aggregations (sum, count, trends) Data Handling ============ Bulk data ingestion pipelines ETL processing Schema optimization Index optimization Performance Requirements ====================== Query response in seconds Pagination with deep offset handling Parallel query execution Caching for repeated queries Security & Access =============== User authentication Role-based access Paid subscription model (optional phase 2) Dataset Details ============= 10+ TB structured shipment/export-import data Trillions of rows Continually growing dataset Structured but large-volume relational-style data Ideal Candidate ============== Experience building large-scale search platforms Hands-on experience with distributed databases Strong system design background Experience optimizing heavy analytical queries Experience handling 1B+ rows minimum (preferably more) Deliverables =========== Complete system architecture design Scalable backend Optimized database schema High-performance search engine Production-ready deployment Documentation Budget ======= Open to proposals (Fixed / Milestone-based preferred). Serious and experienced teams only. Timeline ======= Phase 1 MVP: 8–12 weeks Full production version: Based on architecture complexity
Projekt-ID: 40258529
29 forslag
Projekt på afstand
Aktiv 10 dage siden
Fastsæt dit budget og din tidsramme
Bliv betalt for dit arbejde
Oprids dit forslag
Det er gratis at skrive sig op og byde på jobs
29 freelancere byder i gennemsnit ₹1.767.241 INR på dette job

As an established full-stack developer with a strong background in data management and search infrastructure optimization, I am confident in my ability to spearhead the construction of your enterprise data intelligence platform. Over the years, I have built scalable applications using Python that handled extensive volumes of structured data, much like you're anticipating. My proficiency includes Elasticsearch, MySQL, and Python, which aligns with some of the technologies you are open to. Moreover, my experience incorporates not only the utilization of columnar databases like ClickHouse but also strategies like partitioning and indexing to enhance query efficiency and minimize response time - two central aspects to your project requirement. Additionally, my expertise in cache implementation can ensure swift retrieval of returned results for subsequent similar searches. With me on board, you gain a partner who can offer advanced filtering infrastructures and high-performance search engines akin to what Volza has accomplished. Lastly, my dedication to producing top-quality deliverables is evident from deploying numerous successful projects throughout my career. On-time delivery and accountability have always been paramount in my work ethos.
₹2.350.000 INR på 90 dage
8,5
8,5

Hi there, I am a strong fit for this project because I have architected distributed data platforms where large-scale analytical queries and low-latency search were core performance requirements. For this system, I would propose a columnar-first architecture using ClickHouse as the primary analytical engine with partitioning by date and country, aggressive indexing strategies, materialized views for pre-aggregations, and Redis for hot-query caching. The backend can be built in FastAPI or Go for performance-critical endpoints, with a query orchestration layer that supports parallel execution, keyset pagination for deep result sets, and controlled filter composition. Bulk ingestion would run through staged ETL pipelines with schema validation, batch processing, and background workers, while Kubernetes-managed horizontal scaling ensures compute elasticity as data volume grows beyond 10 TB. Full-text search and fuzzy matching can be handled via ClickHouse integration or a complementary Elasticsearch layer depending on workload distribution. I focus on compression optimization, data skipping indexes, distributed clusters, and load testing benchmarks to ensure sub-second performance even at multi-trillion row scale. I can deliver a structured architecture blueprint, phased MVP plan within 8–12 weeks, and production-ready DevOps configuration with monitoring and logging. Regards, Chirag
₹1.750.000 INR på 30 dage
4,4
4,4

Hello, I hope you are doing well. I understand that you are looking to build a DATA INTELLIGENCE PLATFORM and I would be happy to help. I have extensive experience in designing and developing Application and have successfully delivered many projects. What I offer: -Development using the latest technologies -Daily project updates and clear communication -On-time delivery within the agreed timeline -Complete source code provided after project completion -One year of support for the developed work -10+ years of experience in web and app development -Fully responsive, fast-loading, and SEO-friendly design -A powerful super admin panel to manage the entire system Please feel free to initiate a chat so we can discuss your requirements in detail. Thank you and regards, Dheeraj K.
₹1.000.000 INR på 90 dage
5,0
5,0

I understand you require a scalable enterprise data intelligence platform capable of handling over 10 terabytes of structured trade data with trillions of rows and sub-second search performance using advanced filtering and analytics. Your focus on distributed data storage, columnar database optimization, and parallel query execution aligns perfectly with the complexity of this project. The need for a production-grade solution with features like full-text search, fuzzy matching, and multi-filter query building is clear. With over 15 years of experience and 200+ projects completed, I specialize in full-stack development using Python, FastAPI, and database design with MongoDB and MySQL. I have extensive expertise in building large-scale distributed systems, optimizing ETL pipelines, and implementing high-performance search engines using Elasticsearch and related technologies. My background in cloud and DevOps ensures smooth deployment and scalability. For your platform, I will design a robust architecture leveraging distributed columnar storage and caching layers, implementing efficient indexing and partitioning strategies. The backend will be built with FastAPI for performance, integrating Elasticsearch for advanced search capabilities, while ensuring optimized schema and query acceleration. I propose an 8–12 week timeline for the MVP phase with iterative testing and documentation. Let’s discuss your project in more detail so I can tailor the approach precisely to your needs.
₹1.100.000 INR på 7 dage
2,0
2,0

Hello,You’re building a true data intelligence platform—not a typical web app—and the architecture must be designed for scale from day one. I’ve worked on large analytical systems handling billions of rows using distributed columnar databases (ClickHouse/Elasticsearch) with optimized partitioning, indexing, and parallel query execution. My proposed approach: ClickHouse for high-speed columnar queries, Elasticsearch for full-text/fuzzy search, FastAPI or Go-based services for performance-critical endpoints, Redis for caching, and Kubernetes for horizontal scaling. I’ll design ingestion pipelines, deep pagination strategy, and sub-second filtered search with aggregation support. Delivery will include full architecture documentation, production deployment, and performance validation benchmarks. Available for the 8–12 week MVP phase and ready to begin immediately. BEST REGARDS SHABAHAT HABIB*
₹1.750.000 INR på 7 dage
3,8
3,8

Hello there, We bring 8 years of experience in large-scale data engineering, distributed search, and Python/FastAPI backends. Sub-second search across trillions of trade data rows with multi-filter queries is exactly our kind of problem. The critical path is pipeline design — how shipment data gets ingested, normalized, and indexed so HS code + country + date range + fuzzy product name resolves under a second. We'd build a FastAPI ingestion service with schema validation, deduplication, and partitioning by date/country feeding into ClickHouse with materialized views for common aggregations. ClickHouse handles analytics, Elasticsearch handles full-text search, auto-suggestions, and fuzzy matching — that separation is non-negotiable at this scale. Redis caches repeated query signatures, and pre-computing top aggregations during off-peak windows cuts compute costs 60-70%. Our team built ETL pipelines processing millions of records with Python, FastAPI, and PostgreSQL on AWS under strict query performance SLAs — directly parallel to your growing dataset with bulk ingestion. For deep pagination across trillions of rows, we'd use keyset pagination with composite sort keys instead of OFFSET, which degrades past a few million rows. Phase 1 MVP in 10 weeks: ingestion pipeline, ClickHouse schema, search API, React frontend with filtering. Phase 2 adds subscriptions, RBAC, and horizontal scaling. Weekly demos, async Slack updates. Naveen Brainstack Technologies
₹1.850.000 INR på 70 dage
0,0
0,0

Patna, India
Medlem siden feb. 25, 2026
$30-250 USD
₹1500-12500 INR
$30-250 USD
₹37500-75000 INR
£10-20 GBP
$5-10 USD / time
$30-250 USD
₹600-1500 INR
£10-15 GBP / time
$30-250 USD
$2-8 USD / time
₹600-1500 INR
$50-450 NZD
$750-1500 USD
$200-350 USD
$750-1500 USD
£20-250 GBP
$250-850 USD
$250-750 USD
$25-40 USD / time