E-Commerce Platform
• Distributed, large scale, analytical and machine learning system for e-commerce domain based on modern data architecture on Databricks ecosystem • Supports common EDA, customer segmentation, customer retention, life time value analysis • AI based churn and periodic sales prediction, product recommendation and sentiment analysis • Built pipeline for fault-tolerant data ingestion including cleaning, de-duplication, transformation and feature engineering. • System was tested on load of 700 Million e-commerce transaction per year
About Me
I am a Senior Backend Engineer specializing in AI-powered automation, cloud-native backend systems, and advanced web scraping. With 15+ years of experience, I design and deliver scalable, resilient and production-ready solutions across data engineering, APIs, microservices, scraping pipelines, and DevOps automation. I build enterprise-grade web scraping systems capable of handling Cloudflare protection, dynamic content, region-locked data, and large-scale extraction. My scrapers use Scrapy, Playwright, proxy rotation, browser fingerprinting, CAPTCHA solving (2Captcha), automated retries, failover logic, session management and resume-from-last-state capabilities. I also build real-time dashboards using Flask/FastAPI with live logs, metrics, job monitoring, and one-click scraping controls. On the backend side, I develop high-throughput microservices, distributed systems, and event-driven architectures using Python, Scala, Java, Akka, and FastAPI. I integrate APIs, build data-heavy automation, and deliver fully containerized applications with CI/CD pipelines. As a data engineer, I create robust ETL/ELT pipelines using Airflow, Databricks, Spark/PySpark, DBT, Lakehouse/Medallion architecture, Delta Lake, and streaming/batch workloads. I help clients modernize legacy systems to high-performing, cloud-optimized platforms. On the DevOps and cloud side, I work extensively with Docker, Kubernetes, Terraform, Helm, GitHub Actions, JenkinsX, and cloud-native logging/monitoring stacks such as Prometheus, Grafana, FluentD. 🧩 Key Capabilities 🔹AI-powered web scraping engines (Scrapy, Playwright, proxies, CAPTCHA solving) 🔹Backend engineering (APIs, microservices, distributed systems) 🔹Data engineering (ETL/ELT, Spark, Airflow, Databricks, DBT) 🔹Dashboard development (Flask/FastAPI) for automation & monitoring 🔹Cloud & DevOps (Kubernetes, Docker, Terraform, CI/CD) 🔹Database & storage integrations (PostgreSQL, Hive, Cassandra, Delta Lake) 🔹Production-grade architecture, testing & deployment ⚙️Technical Skill Set ▸Languages: Python, Scala, Java, SQL ▸Frameworks: Scrapy, SeleniumBase, Flask, FastAPI, Akka ▸Big Data: Spark/PySpark, Databricks, Delta, Airflow, Kafka, Hive, Parquet, Avro, Kubeflow, Strimzi ▸DevOps: Docker, Kubernetes, Helm, Terraform, GitHub Actions ▸Databases: PostgreSQL, DB2, Cassandra ▸OS & Tools → Linux (Debian/Ubuntu), Bash, SonarQube, PMD, Checkstyle ▸Monitoring: Prometheus, Grafana, FluentD ✨Message to Clients I focus on long-term, reliable technical partnerships. You can expect: ✔ On-time delivery ✔ High-quality, maintainable architecture ✔ Clear communication ✔ Free post-delivery support ✔ Strong technical ownership Let’s collaborate to create a solution that is scalable, robust and built to last.
$ 15 USD/hr
NEW FREELANCER!
