The worst incidents are silent: the pipeline is green, but the business metric is wrong. Observability for data pipelines must go beyond job status to include data properties and business expectations.
Three layers of observability
- System metrics: runtime, errors, retries, resource usage
- Data metrics: freshness, volume, null rates, distribution shifts
- Business metrics: KPI sanity checks (e.g., revenue not negative)
Freshness: the most important metric
Freshness measures ‘how old is the newest data’. It’s what your consumers actually feel. Track it per dataset and per critical partition.
Volume and anomaly detection
Row counts catch upstream outages, filter regressions, and duplicated loads. Compare volume against rolling baselines and seasonal patterns.
Schema drift: detect and negotiate
- Detect: columns added/removed/renamed, types changed
- Classify: breaking vs non-breaking
- Respond: block deploy, or route to quarantine with alert
Business-level checks
Some problems don’t show up as nulls or counts — they show up as impossible business results. Add small, curated checks that match how stakeholders think.
- Gross margin within expected range
- Orders per day not zero on business days
- Refund rate not 10x normal
Make alerts actionable
Include dataset, partition, last good run, and likely upstream dependency. If the alert doesn’t tell you what to do next, it’s noise.