Insights Data Engineering
Data quality & observability: trust your pipelines
The scariest failure in data engineering is not the pipeline that crashes, that one is obvious and gets fixed. It is the pipeline that keeps running while quietly loading wrong data. Dashboards look normal, decisions get made, and the error only surfaces weeks later when someone notices the numbers never added up. Data quality and observability exist to catch that class of failure before it reaches a human, and they are what separate a pipeline you can trust from one you merely hope is working.
In this guide
The silent-failure problem
Bad data that looks fine is more dangerous than an outright outage. An outage announces itself; a subtly wrong number does not. It flows into a dashboard, an executive reads it, a decision follows, and by the time anyone traces it back, real damage is done and trust in the whole data platform takes the hit. Observability is the discipline of noticing the problem yourself, first, instead of hearing about it from the person who acted on the wrong figure.
The four things to monitor
You do not need an expensive platform to get most of the value. Monitor these four, with alerts on each:
- Freshness, did the data arrive when it should have? Late data is often the first sign of a break.
- Volume, is the row count in a normal range? A sudden drop or spike usually means something upstream changed.
- Schema, did columns or types change unexpectedly? Schema drift silently corrupts downstream logic.
- Business rules, are values plausible and within valid ranges? Negative prices and future birthdates should never pass quietly.
The non-negotiable addition is alerting, when any check fails, a human hears about it immediately. A check nobody is notified about is just a log entry.
A pipeline without monitoring is not a pipeline, it is a rumour. You do not actually know it is working; you are hoping.
Quality vs observability
These get used interchangeably but are not the same. Data quality is the goal: accurate, complete, timely data. Data observability is the ongoing practice that protects it, the tests, checks, and monitoring that continuously verify quality and shout when it slips. You cannot achieve durable quality without observability, because data degrades constantly as sources change. This is the same reliability mindset I apply throughout the data pipelines work, and it is what made the OMERS orchestration migration safe to ship.
Quick pipeline health audit
Tick what is true of your current pipelines. Unchecked boxes are exactly where silent failures hide.
Do you actually trust your data?
Tell me about your pipelines and where the numbers have burned you before. I will add the tests, freshness checks, and alerting that turn "I hope it's right" into "I know it is".
Make my data trustworthyFrequently asked questions
What is data observability?
Data observability is the practice of continuously monitoring the health of your data and pipelines, freshness, volume, schema, and value distributions, so you know whether your data is trustworthy at any moment. It is the difference between discovering a data problem yourself, before anyone acts on it, and hearing about it from an executive staring at a wrong number.
How is data quality different from data observability?
Data quality is the goal, data that is accurate, complete, and timely. Data observability is how you keep it: the tests, checks, and monitoring that continuously verify quality and alert you when it slips. Quality is the outcome; observability is the ongoing practice that protects it.
What should I monitor in a data pipeline?
At minimum: freshness (did the data arrive on time), volume (is the row count in a normal range), schema (did columns or types change unexpectedly), and key business rules (are values within valid, plausible ranges). Add alerting so a human is notified the moment any check fails, rather than the failure passing silently downstream.
Why do silent data failures matter so much?
Because bad data that looks fine is worse than an obvious outage. A pipeline that quietly loads incomplete or wrong data keeps feeding dashboards and decisions that look normal but are subtly false, eroding trust and causing real mistakes. Catching failures early, before the data is used, is the whole point of observability.