25/06/2026
My Portfolio item: Scalable streaming data pipeline in Databricks (Delta Live Tables)
Project overview
This project details a scalable streaming data platform architecture implemented in Databricks, using Delta Live Tables (DLT) to process streaming data via a medal architecture (bronze, silver, gold layers). The solution is designed for robust, fault-tolerant and high-performance real-time event processing.
Problem Solved
Processing streaming data often involves challenges such as managing schema development, ensuring data quality, managing delayed data, optimizing storage for small files, and dynamically scaling resources. This architecture provides a structured and automated approach to overcome these complexities, transforming raw embedded data into clean, aggregated insights suitable for a variety of downstream applications such as dashboards and ML service.
Technical architecture
Architecture is governed by medallion architecture, which consists of three layers:
• Bronze Layer (Raw Ingestion): This layer captures raw events from various sources (e.g., Kafka, S3/ADLS) for connection only, ensuring full fidelity without transformation. It is designed to be resistant to data loss and allow for efficient trimming over time.
• Silver layer (cleaned and conformed data): The data from the bronze layer is cleaned, deduplicated, and molded at this stage. It involves schema enforcement and resolves bad records by either throwing them away or quarantining them, preventing silent data corruption.
• Gold layer (aggregated and business data): This last layer provides aggregations and summaries at the business level, optimized for consumption by dashboards, ML service or other analytical tools. Examples are daily summaries of the entire platform and real-time user activity metrics.
Key technologies
• Databricks: A unified analytics platform that hosts the entire pipeline.
• Delta Live Tables (DLT): An automated framework built on Databricks to build reliable, maintainable, and testable data pipelines. DLT automatically handles checkpoints, repetitions, and a line of data.
• Apache Kafka Connector: Used to receive streaming events in real-time.
• Auto Loader: Databricks feature to efficiently receive new data files as soon as they arrive in cloud storage (e.g. S3, ADLS).
• PySpark SQL: A basic language for transformations and aggregations of data within a pipeline.
• Unity Catalog: Used for data management and gold layer operation.
Key features and design decisions
The architecture includes several key design decisions for scalability, reliability, and maintainability:
• Ingestion strategy: A hybrid approach using Auto Loader for file-based resources (schema evolution processing and processing exactly once) and a Kafka connector for receiving streams in real-time.
• Processing Framework: Utilizes Delta Live Tables instead of raw structured streaming. This option simplifies pipeline management by automatically handling checkpoints, recurrences, data quality checks, and origin.
• Deduplication: Utilizes dropDuplicatesWithinWatermark to effectively ensure the uniqueness of records across defined time windows, preventing memory issues during full historical tracking.
• Data Quality Enforcement: Integrates .expect_or_drop and expect_or_quarantine rules. This ensures that erroneous records are either deleted or isolated for further investigation, preserving the integrity of the data assets.
• File Compression: Uses the properties of the autoOptimize.optimizeWrite and delta.autoOptimize.autoCompact tables to avoid the "small file problem" common with high-frequency streaming writes, improving query performance.
• Back Pressure Control: maxOffsetsPerTrigger is set to limit the number of records processed per microdose. This mechanism prevents Out-Of-Memory (OOM) errors during traffic peaks, ensuring stable pipeline operation.
• Late Data Processing: Implements watermarks (e.g., 1 hour for silver, 2 hours for gold) to smoothly process delayed data within a defined tolerance, ultimately throwing out extremely delayed data to maintain performance and freshness.
• Splitting strategy: Data is partitioned by date in the bronze layer for efficient time range trimming, and by windows in the gold layer to align with common query patterns.
• Dynamic scaling: Leverages Databricks Auto Scaling clusters to tailor compute resources to workload requirements, scaling during peak reception and scaling down during quieter periods for cost efficiency.
Code structure example
The example provided contains Python code for dlt_pipeline.py file that shows how to define Bronze, Silver, and Gold tables using .table decorators and PySpark SQL functions. It also shows how to externalize configurations for resources like Kafka.
Result and impact
This DLT-based streaming architecture provides a solid foundation for modern data platforms. It automates much of the operational overhead associated with streaming analytics, ensures high data quality, and provides timely, reliable data to drive business decisions and power AI/ML applications. A modular and layered approach increases sustainability and simplifies future expansions.
