Data Engineer -- 2

$100-150 USD / hour

Closed

Posted 7 days ago

$100-150 USD / hour

Onsite – In person, client site – Washington DC Long Term Salary Range : $100- $150/hour Joining period : Immediate About the Role We are seeking a highly skilled Data Engineer to join our onsite team. In this role, you will design, build, and optimize complex, highly scalable big data pipelines and infrastructure on AWS to support advanced regulatory analytics, surveillance systems, and market research. Key Responsibilities Pipeline Development: Design, develop, and maintain high-performance ETL/ELT data pipelines capable of ingesting and transforming massive volumes of structured and unstructured financial data. Big Data Processing: Utilize PySpark, Apache Spark, and AWS EMR to process billions of records efficiently, ensuring low latency and high reliability. Cloud Architecture: Architect and optimize solutions within FINRA’s AWS data lake and data warehousing environment (S3, Athena, Glue, Snowflake, Presto/Trino). Data Quality & Governance: Implement rigorous data quality checks, validation frameworks, and observability pipelines to ensure data lineage, consistency, and strict regulatory compliance. Collaboration: Partner cross-functionally with data scientists, economists, software engineers, and regulatory stakeholders to translate complex business requirements into robust technical solutions. CI/CD & DevOps: Deploy scalable infrastructure using CI/CD pipelines (Jenkins, Bitbucket, Git) and infrastructure-as-code (Terraform, CloudFormation). Performance Optimization: Troubleshoot production data pipelines, tune PySpark/SQL queries for optimal compute performance, and manage AWS resource costs. Required Qualifications Experience: 5+ years of hands-on, dedicated Data Engineering experience in a production environment. Programming Languages: Expert-level proficiency in Python (specifically for data engineering) and advanced SQL. Big Data Ecosystem: Deep, hands-on experience with Apache Spark (PySpark) and distributed computing architecture. Cloud Platforms: Strong implementation experience within the AWS ecosystem, specifically with EMR, S3, Glue, Athena, IAM, and Step Functions. Data Architecture: Proven experience working with data lakes, modern data warehousing (e.g., Snowflake, Redshift), and data formats (Parquet, Apache Iceberg, Delta Lake). Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a closely related discipline. Work Authorization: Eligible to work in US Preferred Qualifications Prior domain experience in financial services, capital markets, payroll, or regulatory environments. Experience with streaming data technologies such as Apache Kafka or Amazon Kinesis. AWS Certifications (e.g., AWS Certified Data Analytics, AWS Certified Solutions Architect). Familiarity with AI-assisted developer tooling (GitHub Copilot, Amazon Q) and integrating AI into engineering workflows. Excellent communication skills with the ability to explain complex technical concepts to non-technical financial stakeholders.

Project ID: 40742807

About the project

15 proposals

Remote project

Active 3 days ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

15 freelancers are bidding on average $115 USD/hour for this job

@pvsysgroupinc

★★★ TOP 1% IN FREE LANCER WORLD ★★★ ★★★ 20+ Year Experience in IBD being CMD★★★ ★★★ 200+ Country Satisfied Clientele ★★★ ★Linkedin★ ★Data Entry★ ★Business Plans★★★ ★★★ Operational Strategic planner Customer Support 24*7★★★ ★★★Excel/Word Operation★★★ ★★★Chat Support★★★ ★★★Calling Support★★★ ★★★Business Plans / Marketing Strategy ★★★ * Digital Marketing★★★ ★★★Social Media Marketing ★★★ ★★★Internet Marketing ★★★ ★★★Any type of Data Projects★★★ ★★★★★★ Regards, ★★★CMD★★★ ★★★PVSYS GROUP (INDIA)★★★ YOUR HONEST and DEDICATED WORKER ★★★IF YOU THINK THEN I CAN★★★

$150 USD in 40 days

5.0

(4 reviews)

5.1

@AZTURK1

Building scalable PySpark and AWS EMR pipelines for high-volume financial data demands robust partition tuning, memory management, and strict governance. I'm based in Zagreb, Croatia (EU) and am fully available to travel onsite for the deployment/hardware setup if travel is covered, or support remotely across your AWS data lake infrastructure. ✅ Engineering high-performance PySpark / EMR processing workflows, optimizing data skew, shuffle partitions, and Iceberg/Parquet storage layouts. ✅ Automating end-to-end ingestion and transformation pipelines via AWS Glue, Athena, and Snowflake integrated through Terraform and CI/CD. ✅ Establishing automated data quality validation, schema enforcement, and lineage checks with zero pipeline regression. ✅ 10+ years production experience | 400+ completed projects | 320+ five-star reviews ✅ Microsoft® Certified: MCSA | MCSE | MCT & cPanel Certified (CWSA-2) Are your current EMR workloads running on persistent clusters or provisioned dynamically via Step Functions and EMR on EKS? My rate is 100 USD/hour and I can start immediately.

$100 USD in 1 day

5.0

(10 reviews)

4.4

@AhmadXpert2410

Hi there, I’m a Data Scientist / AI Engineer with hands-on experience developing Python-based ETL pipelines, SQL data workflows, data processing systems, and production ML infrastructure. I work extensively with Python, SQL Server, PostgreSQL, Pandas, Docker, AWS, and automated data pipelines, and I’m comfortable working with large structured datasets and production data workflows. I can help design reliable ETL/ELT pipelines with data validation, transformation, monitoring, error handling, and reproducibility in mind. I’m also experienced in optimizing SQL queries and data-processing workflows, integrating multiple data sources, and preparing clean datasets for analytics and machine-learning workloads. My background in telecom has involved production data pipelines and troubleshooting data discrepancies between development and production environments. I should note that my experience is primarily in production data/AI engineering rather than 5+ years of dedicated financial-services data engineering, so I would approach the role based on the transferable technical requirements rather than overstating that experience. I’m comfortable learning domain-specific regulatory requirements and collaborating closely with technical and business stakeholders. Regards, Ahmad

$100 USD in 40 days

5.0

(16 reviews)

4.3

@dataspro

Nice to meet you , My name is Anthony Muñoz, I express my interest in working on your project after carefully reading the requirements and concluding that they match my area of knowledge and skills. I am currently the lead engineer for the IT agency DSPro and I have more than 10 years of experience in the field. I have successfully completed a large number of similar jobs and I consider your project to be a challenge in which I would like to work and be able to make it a reality. Please feel free to contact me, it will be my pleasure to help you. I greatly appreciate the time provided and I remain attentive to any questions or concerns. Greetings

$101 USD in 40 days

5.0

(1 review)

3.8

@abdf2010

Hi, I am a professional web developer and I can do this project "Data Engineer -- 2", I have 5 years of experience in web development. I have done many projects like this. I can do this job for you. I can start right now. Please contact me. Thanks

$100 USD in 2 days

4.8

(2 reviews)

3.9

@muzammilf0

I’m a Python/AI Engineer with 5+ years of production experience, including building scalable data and ML systems with AWS, Python, SQL, Docker, and CI/CD. I can contribute to high-volume data pipelines, ETL/ELT, cloud infrastructure, and performance optimization. I’m available to start immediately. My portfolio: https://www.freelancer.com/u/muzammilf0 Best regards, Muzammil Fareed

$125 USD in 40 days

0.0

(0 reviews)

0.0

@Kyybaindia

# Headquarters – Michigan, working for one of the 3 largest healthcare companies in the USA # Hi there, This is Shree from Kyyba. We can place our direct W-2 data engineering experts onsite in Washington, DC, as needed, with immediate availability. Quick questions 1. Is the engagement fully onsite in DC, hybrid, or flexible? 2. Is US work authorization required for each consultant? 3. Which AWS/Snowflake workloads are the immediate priority? 4. What is the expected onsite schedule? 5. Will interviews include a PySpark/SQL technical assessment? 6. Can you share the expected start date and interview timeline? Approach Build scalable AWS data lake pipelines using Python, PySpark/Spark, S3, Glue,e and EMR. Develop high-volume ETL/ELT for structured/unstructured financial and regulatory datasets. Use Athena/Presto/Trino, Snowflake,ke and advanced SQL for analytics and warehousing. Implement Parquet/Iceberg/Delta-based data models, partitioning, and performance optimization. Add data-quality checks, reconciliation, lineage, monitoring, and failure/retry handling. Automate deployments through Jenkins, Bitbucket, Git, Terraform/CloudFormation. Optimize Spark jobs, SQL execution, cluster sizing and AWS costs. Work closely with regulatory, analytics, and engineering stakeholders. Sample: S3 ingestion - Glue/EMR - PySpark validation/transformation - Parquet/Iceberg - Snowflake/Athena - quality/observability - regulatory analytics. Thanks Shree Kyyba.

$150 USD in 40 days