Optimasi Penyimpanan Big Data Lakehouse

$30-250 USD

Closed

Posted about 2 months ago

$30-250 USD

Paid on delivery

Proyek ini bertujuan melakukan optimasi penyimpanan Big Data berbasis kolumnar pada arsitektur data lakehouse untuk meningkatkan performa analitik pada data transaksi/kejadian berskala besar. Fokus utama adalah merancang tata letak fisik data yang efisien sehingga eksekusi kueri menjadi lebih cepat, lebih stabil, dan membaca data lebih sedikit, tanpa mengorbankan kemampuan ingest/append untuk kebutuhan pemantauan berkala (near real-time). Pekerjaan dilakukan melalui benchmarking eksperimen terkontrol, yaitu menjalankan dataset yang sama pada beberapa konfigurasi penyimpanan dan membandingkan hasilnya secara objektif. Konfigurasi yang diuji mencakup kombinasi kompresi (mis. Snappy/ZSTD), strategi partisi (berbasis waktu dan/atau dimensi utama), sorting/clustering pada kolom yang sering difilter, serta encoding per kolom. Workload kueri disusun untuk mewakili kebutuhan umum analitik, seperti rekap periodik, agregasi rolling window, pencarian data selektif berbasis ambang, dan drill-down investigatif pada subset dimensi. Kinerja dievaluasi menggunakan metrik utama: storage footprint, bytes-read/data scanned, latensi kueri p50/p95, throughput ingest/append, serta penggunaan CPU dan RAM, termasuk pengukuran pada kondisi cold-cache dan warm-cache untuk mendapatkan hasil yang adil dan dapat direplikasi. Hasil akhir proyek adalah rekomendasi konfigurasi lakehouse terbaik Output yang Diharapkan (deliverables) [login to view URL] skenario optimasi penyimpanan (bertahap) + parameter konfigurasi. [login to view URL] kueri benchmark (template SQL) + prosedur eksekusi yang replikasi. [login to view URL] hasil benchmark: footprint, bytes-read, p50/p95, throughput, CPU/RAM (cold/warm). [login to view URL] implementasi.

Project ID: 40204223

About the project

5 proposals

Remote project

Active 13 days ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

5 freelancers are bidding on average $178 USD for this job

@immaculetac

I am excited to submit my bid for your project focused on optimizing columnar-based Big Data storage within a data lakehouse architecture. With a strong foundation in data engineering, I have hands-on experience in performance tuning for large-scale analytical workloads. I fully understand the importance of reducing query latency and storage read overhead, without compromising ingest/append capabilities. I am confident in executing controlled benchmarking experiments across different configurations—compression types (e.g., Snappy, ZSTD), partitioning strategies, and clustering techniques—to identify the most efficient layout. I also have experience designing workloads that simulate real-world use cases such as rolling aggregations, selective queries, and dimensional drill-downs. With strong analytical skills, attention to detail, and a deep understanding of modern data architectures, I will deliver measurable improvements and well-documented outcomes. Let’s optimize your data lakehouse for faster, scalable analytics.

$140 USD in 7 days

0.0

(0 reviews)

0.0

@rajdeepdutta1981

As a seasoned data professional well-versed in all aspects of data handling and processing, I am confident that my skills and experience make me an excellent fit for your project. My forte in data entry and management, coupled with my proficiency in tools like Excel, allow me to not just handle but also analyse large datasets with ease and precision.

$180 USD in 7 days

0.0

(0 reviews)

0.0

@labee5

Hello, I’m interested in your Big Data lakehouse optimization project. I understand the goal is to improve analytics performance by optimizing columnar storage, compression, partitioning, and query execution. I will benchmark different configurations on the same dataset and measure key metrics such as storage footprint, query latency (p50/p95), throughput, and CPU/RAM usage. Based on the results, I will recommend the best lakehouse configuration for efficient analytics and reliable ingest/append performance. I’m a beginner freelancer, but I’m detail-focused and committed to delivering accurate, well-documented results. Thank you for considering my proposal.

$140 USD in 7 days