ETL & Transformation using Synapse
Built a dynamic ingestion pipeline using Azure Synapse Pipelines + Self-hosted IR to extract data from multiple on-prem and cloud sources. Worked on sales data mainly for different sites Performance tuning using partitioning, salting, and caching (e.g., reduced job time from 8 hrs to 30 mins) Landed raw sales data into ADLS Gen2 in Parquet/CSV format, ensuring full traceability. Designed and implemented a layered architecture (RAW → BRONZE → SILVER → GOLD). In the BRONZE Layer, applied data cleansing, quality checks, and deduplication using Delta Lake. In the SILVER Layer, enriched data with dimension tables (e.g., sites, product, time) for analytical models. In the GOLD Layer, created fact-dimension tables and data marts using Synapse Dedicated SQL Pool. Boosted team productivity by 20% through streamlined collaboration using Azure DevOps and agile sprint planning. Implemented secure data practices using Azure Key Vault and Azure Active Directory, ensuring full compliance
About Me
Experienced Azure Data Engineer with expertise in Microsoft Fabric, Azure Synapse, and Data Factory. I design and build robust, scalable data pipelines and implement business logic that improves data quality by up to 99%. I have automated data validation and testing using PySpark, developed dynamic notebooks for reusable transformations, and implemented performance optimization techniques (like SALT and partitioning) to reduce processing time from 8 hours to 30 minutes. Additionally, I led initiatives that reduced cloud resource costs by 20% through efficient Spark job tuning and workload management. My end-to-end solutions—from raw data ingestion to Power BI visualization—enable accurate, timely, and cost-effective business insights. Let’s collaborate to deliver data solutions that are fast, accurate, and efficient.
$ 20 USD/hr
NEW FREELANCER!
