Real-Time Spark Data Pipeline

My analytics stack must consume a continuous stream of events from a proprietary source (it isn’t Kafka, Flume, or Kinesis) and process them instantly with Apache Spark Structured Streaming. The job is to wire that source into Spark 3.x, perform the required real-time transformations and aggregations, and deliver the cleaned data to two sinks—Parquet files on S3 for long-term storage and a live dashboard feed for immediate insight. Deliverables • Production-ready Scala or PySpark code that runs on Spark 3.x in either Stand-Alone or YARN mode. • Structured Streaming logic featuring checkpointing, watermarking, and exactly-once guarantees. • Tuning guidelines for executor, memory, and parallelism settings to maintain sub-second end-to-end latency at 50 k events/sec. • A runnable test suite plus a concise README so I can reproduce results on my cluster. Acceptance Criteria 1. Pipeline ingests and writes a sustained 50 k events/sec with no data loss for 10 minutes. 2. Average latency from ingestion to sink remains below one second during that test. 3. Entire build, test, and deploy sequence executes with a single command (e.g., sbt run or spark-submit with a provided shell script). Include any specialized connectors or libraries you rely on and package everything in a Git repo so each commit is clear and reviewable.

Project ID: 39740684