Insights Data Engineering
Real-time data pipelines: streaming, Kafka & CDC
"Real-time" is one of the most over-requested features in data engineering. It sounds obviously better than "eventually", so people ask for it by default, then pay for complexity they never needed. The truth is most analytics are fine on batch, and real-time is a specific tool for a specific job. This guide covers when you genuinely need streaming, how Kafka and change data capture actually work, and how to avoid over-engineering.
In this guide
Batch vs streaming: the real question
A batch pipeline processes data on a schedule, every hour, every night, in chunks. A streaming pipeline processes each event the moment it arrives. The question is never "which is more modern?" It is "how fresh does this data need to be for the decision it feeds?" If a report is read once a morning, nightly batch is perfect and far cheaper. Streaming only earns its cost when staleness has a real price.
When you actually need real-time
Genuine real-time cases share one trait: a decision or reaction must happen within seconds. Examples:
- Fraud detection, you must flag a suspicious transaction before it completes, not tomorrow.
- Live operational dashboards, ops teams reacting to what is happening right now.
- Instant notifications, alerting a user or system the moment an event occurs.
- Dynamic pricing or inventory, where seconds of lag means selling something you no longer have.
If a delay of minutes or hours causes no harm, you do not need real-time, and choosing batch is the more responsible engineering decision.
Real-time is not a badge of sophistication. It is a cost you take on when staleness would genuinely hurt, and a liability you avoid when it would not.
Kafka and CDC, explained
Kafka is a distributed event-streaming platform, think of it as a durable, high-throughput pipe. Producers publish events to it; consumers read them, reliably and in order, even at huge volume. It is the backbone most serious streaming architectures are built around.
Change data capture (CDC) streams the changes in a database, every insert, update, and delete, as a feed of events. Instead of repeatedly reloading an entire table to spot what changed (slow and expensive), CDC lets downstream systems stay in sync by reacting only to the deltas. Together, CDC feeds Kafka, and Kafka feeds everything else, which is how databases power real-time systems without being hammered by full reloads.
The cost of real-time
Streaming systems have to handle events arriving out of order, failures that must not lose data, and the fact that they never stop. That is real operational complexity and real ongoing cost, on top of the reliability discipline any pipeline needs (see data pipelines and data quality). My advice, shaped by building both: adopt real-time only where the business truly needs it, design it carefully with proper monitoring, and keep everything else on batch. The ETL vs ELT decision applies within both.
Wondering if you need real-time or just reliable batch?
Tell me the decision your data feeds and how fresh it must be. I will give you a straight answer, and build whichever pipeline genuinely fits, without the over-engineering.
Get a straight answerFrequently asked questions
What is a real-time data pipeline?
A real-time (or streaming) data pipeline processes data continuously as events arrive, rather than in scheduled batches. Instead of waiting for a nightly job, each new event, a purchase, a sensor reading, a status change, flows through and updates downstream systems within seconds, enabling live dashboards, instant alerts, and immediate reactions.
Do I really need real-time, or is batch enough?
Most reporting and analytics are perfectly served by batch, and batch is simpler and cheaper to run. You need real-time only when a decision or reaction must happen within seconds, fraud detection, live operational dashboards, instant notifications, or dynamic pricing. If a delay of minutes or hours causes no harm, batch is the right, less costly choice.
What are Kafka and change data capture?
Kafka is a distributed event-streaming platform that acts as a durable, high-throughput pipe for events, letting many producers and consumers exchange data reliably in real time. Change data capture (CDC) streams the changes happening in a database, inserts, updates, deletes, as events, so downstream systems stay in sync without expensive full reloads.
Is a streaming pipeline harder to run than batch?
Yes, meaningfully. Streaming systems must handle out-of-order events, failures without data loss, and continuous operation, which adds real operational complexity and cost. That is exactly why you should adopt real-time only when the business genuinely needs it, and design it carefully with monitoring so it stays reliable.