28/09/2026
Kafka becomes much easier when you see how the pieces connect.
It is not just about producers sending messages and consumers reading them. Kafka is a streaming system built around logs, partitions, processing state, delivery guarantees, and operational controls.
Here are 15 concepts every Data Engineer should understand:
โ Topic
A named stream where events are published.
โ Partition
An ordered, append-only log that lets Kafka scale across consumers.
โ Offset
A recordโs position within a partition.
โ Consumer Group
Multiple readers share the work across partitions.
โ Consumer Lag
Shows how far a consumer is behind the latest available records.
โ Rebalancing
Partitions are reassigned when consumers join, leave, or fail.
โ Windowing
Groups events into time-based buckets for stream processing.
โ Grace Period
Defines how long late-arriving events are still accepted.
โ State
Stores memory between events for stateful processing.
โ At-least-once
Records may be processed more than once, but are not intentionally skipped.
โ Exactly-once
Processing effects are committed once within supported transactional workflows.
โ Idempotency
Retries can happen without creating duplicate writes.
โ Retention
Controls how long or how much data Kafka keeps.
โ Compaction
Keeps the latest value for each key instead of every historical version.
โ Schema Registry
Manages schema versions and compatibility between producers and consumers.
The easiest way to remember Kafka is to think in five layers:
Publish โ Consume โ Process โ Guarantee โ Operate
Once these concepts click together, Kafka stops looking like a collection of isolated features and starts looking like one connected streaming architecture.