Data Observability

Parent: Data Analysis · Published reference · snapshot 2026-09-08 · skill da-data-engineering-platform/references/da-19-data-observability.md

↓ Facts as markdown all context files

Data observability is the discipline of measuring and maintaining the health and reliability of data across its lifecycle — the ability to fully understand the state of the data in your systems, so yo

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Overview

Data observability is the discipline of measuring and maintaining the health and reliability of data across its lifecycle - the ability to fully understand the state of the data in your systems, so you can detect, triage, resolve, and prevent data incidents before they reach downstream consumers. The term was popularized by Monte Carlo (Barr Moses, 2019) as an explicit analog to software observability (the logs/metrics/traces discipline for services): instead of asking "is my service up?", data observability asks "is my data trustworthy, and if not, why?" [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]
It is a distinct discipline, not a synonym for any of its neighbors: [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]
- Data quality is the target state (accurate, complete, trustworthy data) and the broad discipline of achieving it. Observability is one modern tactic for sustaining quality at scale. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]
- Data testing (e.g. dbt tests, Great Expectations) is assertion-based: you declare expected conditions up front and check them. It catches the failures you anticipated. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]
- Monitoring watches individual pipeline/job metrics. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]
- Observability is broader than all three: it continuously watches the behavior of data and systems, surfaces unexpected anomalies (not just the ones you wrote tests for), and ties them to lineage and context so you can answer "what broke, where, who is affected, and why." Testing tells you about known-unknowns; observability surfaces unknown-unknowns. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]
Why it matters now: as data stacks fragment (ingestion, warehouse, transform, BI, reverse-ETL, ML/AI features), the surface area for silent failure explodes, and bad data quietly poisons dashboards, ML features, and AI products. Gartner projects ~50% of enterprises with distributed data architectures will adopt data observability tooling by 2026 (up from ~20% in 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#overview]

The Five Pillars

Monte Carlo's canonical framework decomposes data health into five measurable pillars. The first four are table-/field-level signals; the fifth is the connective tissue. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#the-five-pillars]
- Freshness - Is the data up to date? How recently was a table updated, and does the update cadence match expectations? Stale tables (e.g. a daily table that didn't load by 9 AM) are the most common, most user-visible incident. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#the-five-pillars]
- Volume - Is the amount of data as expected? Row-count completeness over time. Sudden drops (a partial load) or spikes (a duplicate load / fan-out bug) signal problems. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#the-five-pillars]
- Schema - Has the structure changed? Added/removed/retyped columns, dropped or renamed tables. Schema drift from upstream producers is a leading root cause of breakage ("who changed this column type?"). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#the-five-pillars]
- Distribution / Quality - Is the data within expected ranges at the field level? Null rates, uniqueness, value ranges, category cardinality, percent-zero, format conformance. This pillar is where field-level data-quality signals live. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#the-five-pillars]
- Lineage - The map of upstream sources and downstream consumers for every asset, ideally to column level. Lineage is the holistic pillar: it turns a freshness/volume/schema/distribution anomaly into an impact assessment ("which 14 dashboards and 2 ML features depend on this broken table, and what upstream job caused it"). Lineage powers both root-cause (upstream) and blast-radius (downstream) analysis. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#the-five-pillars]

Data Downtime

Data downtime is the period when data is partial, erroneous, missing, or otherwise wrong. It is the headline reliability metric and is modeled as: [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-downtime]
> Data downtime ≈ N × (TTD + TTR) [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-downtime]
where N = number of incidents, TTD = time-to-detection, TTR = time-to-resolution. Observability programs drive downtime down by reducing all three: fewer incidents (shift-left/contracts), faster detection (automated monitors vs. "an exec noticed the dashboard was wrong"), faster resolution (lineage-accelerated root cause). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-downtime]

Data SLAs / SLOs / SLIs

Borrowed from SRE and applied to data products: [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-slas-slos-slis]
- SLI (indicator) - the actual measurement: e.g. % of days the table landed by 9 AM; null rate of customer_email; row-count vs. 7-day baseline. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-slas-slos-slis]
- SLO (objective) - the internal target on an SLI: e.g. "freshness ≤ 6h for 99.5% of loads"; "null rate of key columns < 0.1%". [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-slas-slos-slis]
- SLA (agreement) - the external, formal commitment to stakeholders/consumers, often with consequences/escalation if breached. A data SLA bundles measurable targets for freshness, availability, volume, and quality of a data product. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-slas-slos-slis]
- Common starting target: 99.9% ("three nines") ≈ 43.8 min downtime/month; pick SLOs per data product's criticality, not one blanket number. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-slas-slos-slis]

Data Incident Management

Treating data issues with the rigor of software incidents. The lifecycle: detection → triage/severity → ownership → resolution → retrospective. Key practices: explicitly declared incidents (a human or system declares it, giving a clean basis for uptime/SLA math), severity levels tied to blast radius (use lineage), an on-call/owner model for data assets, blameless postmortems, and tracking MTTD/MTTR trends. Incident data is the empirical basis for SLAs - you can't promise reliability you don't measure. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#data-incident-management]

Anomaly Detection on Pipelines

Observability detection ranges from simple to ML-driven: [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anomaly-detection-on-pipelines]
- Rule/threshold monitors - fixed bounds ("row count must be > 1M"). Cheap, explicit, but brittle on seasonal data: they false-alarm at peaks and miss off-peak drops. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anomaly-detection-on-pipelines]
- Statistical / time-series monitors - learn a baseline from history and flag deviations (z-score/IQR on a metric's history, moving averages, STL/Prophet-style seasonal decomposition). Handle daily/weekly seasonality, trend, and growth. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anomaly-detection-on-pipelines]
- ML/automated monitors - platform-trained models that auto-tune per table, adapt to growth/seasonality, and self-onboard new assets without one-by-one config. Teams report ~40–60% fewer false positives vs. static thresholds. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anomaly-detection-on-pipelines]
- Practical guidance: each metric/anomaly type may warrant a different method; reserve ML for high-cardinality, seasonal, or numerous assets and keep cheap thresholds for hard invariants (e.g. PK uniqueness). Watch timezone/DST shifts - they silently break seasonal models. (See da-16-anomaly-detection for the underlying algorithms; this skill covers their application to pipeline/table health.) [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anomaly-detection-on-pipelines]

Lineage & OpenLineage / Marquez

OpenLineage is the open standard (LF AI & Data) for collecting lineage metadata, modeling Run / Job / Dataset entities with consistent naming; it instruments jobs as they run and supports column-level lineage (notably via the Spark integration, with facets for inputs/outputs). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#lineage-openlineage-marquez]
Marquez is the reference implementation/metadata server that collects, aggregates, and visualizes the lineage graph; it added column-level lineage and job-to-job lineage. Integrations exist for Airflow, dbt, Spark, etc. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#lineage-openlineage-marquez]
Column-level lineage enables fine-grained root cause and sensitive-data tracking (where does PII flow?), valuable for governance/compliance. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#lineage-openlineage-marquez]

Shift-Left & Data Contracts

The 2024–2026 evolution of the discipline: stop catching issues only after the warehouse load ("shift right") and move checks earlier - into dbt runs, CI/CD, and at the producer boundary. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#shift-left-data-contracts]
- Data contracts formalize the producer↔consumer interface: schema, freshness SLA, volume expectations, semantic/quality thresholds, and versioning. They prevent incidents at the source rather than detecting them downstream, and bridge testing (pre-merge assertions) with observability (runtime monitoring). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#shift-left-data-contracts]
- Shift-left emphasizes prevention and "design for trust from the start," pairing producer-side contracts with leadership-visible quality dashboards. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#shift-left-data-contracts]

Tools / Frameworks

Selection heuristics: dbt-centric stack → Elementary (then Soda/GX for extra checks). Need vendor-neutral lineage → OpenLineage + Marquez. Enterprise, many sources, want automated full-coverage ML monitoring + incident mgmt → Monte Carlo / Bigeye / Anomalo. Lightweight, CI-embedded checks in code → Soda Core or Great Expectations. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#tools-frameworks]

Methodology

Inventory & prioritize assets. You can't (and shouldn't) monitor everything equally. Rank tables/data products by downstream blast radius (use lineage) and business criticality. Define tiers. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Establish lineage. Stand up column/table lineage (OpenLineage+Marquez or a platform) so every later step has impact context. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Baseline the five pillars. Turn on freshness, volume, schema monitors broadly (cheap, high ROI); add distribution/quality monitors on key fields of high-tier assets. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Set SLIs → SLOs → SLAs per tier. Define indicators, internal objectives, and (for the top tier) external agreements with consumers. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Choose detection per metric. Thresholds for hard invariants; statistical/ML monitors for seasonal/high-cardinality metrics. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Wire incident management. Route alerts to owners, set severities by blast radius, declare incidents, track TTD/TTR. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Shift left. Add data contracts and CI/dbt checks at producer boundaries to cut incident count (N) at the source. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]
Measure & iterate. Track data downtime (N×(TTD+TTR)), false-positive rate, SLO attainment; retune monitors and prune noise. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#methodology]

Practical Patterns

Lineage-first triage. When a metric breaks, walk lineage up for root cause and down for blast radius before touching code. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]
Tier your coverage. Broad cheap metadata monitors (freshness/volume/schema) everywhere; expensive value-level + ML monitors only on tier-1 assets. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]
Freshness SLO as the first win. It's the most common, most visible incident and the cheapest to monitor - start here to build trust in the program. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]
Contract the producer boundary. A schema/freshness contract on the most unstable upstream source prevents the largest class of incidents. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]
Declare incidents explicitly. Gives clean SLA/uptime math and a paper trail for postmortems. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]
Co-locate observability with transformation. dbt-native (Elementary) keeps monitors versioned alongside the models that produce the data. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]
Track the false-positive rate as a first-class metric. Alert fatigue kills observability programs faster than missed incidents. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#practical-patterns]

Anti-Patterns

Treating observability as "more dbt tests." Tests cover known-unknowns; observability must also surface unknown-unknowns via anomaly detection + lineage. Don't conflate the two. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
Static thresholds on seasonal data. Guarantees false alarms at peaks and misses during troughs; use seasonal/ML monitors for cyclical metrics. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
Monitoring everything equally. Untiered, full-coverage value-level monitors produce overwhelming noise and cost; nobody acts on the alerts. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
No lineage. Without lineage, every incident is a manual archaeology dig; triage time (and TTR) balloons. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
Detection without resolution ownership. Alerts with no owner, severity, or incident process just become a noisy dead-letter channel. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
SLAs you don't measure. Promising freshness/quality with no SLIs/incident tracking is theater. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
Shift-right only. Catching everything post-load means consumers see bad data first; push contracts/CI checks upstream. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]
Ignoring timezone/DST in seasonal models. Silent drift in seasonal baselines → spurious anomalies. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#anti-patterns]

Troubleshooting

Flood of false-positive alerts → metrics are seasonal/trending under static thresholds, or monitors lack enough history to baseline. Switch to seasonal/ML monitors, widen training windows, and add DST/timezone handling. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]
Incidents found by consumers, not monitors → coverage gap. Add freshness/volume monitors on the affected lineage path; this is a TTD problem. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]
Long TTR → missing/incomplete lineage. Stand up column-level lineage so root cause and blast radius are immediate. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]
Schema-drift breakage recurs from one source → reactive monitoring only; add a producer-side data contract + CI check (shift-left). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]
"Is this a quality or an observability problem?" → If you can write the assertion up front, it's a test (GX/dbt/Soda check). If you need to detect a deviation you didn't anticipate and trace its impact, it's observability. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]
OpenLineage events missing column lineage → confirm the integration emitting them supports column facets (Spark integration is the most complete); not all integrations emit column-level yet. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]
Tool sprawl → consolidate: dbt shops standardize on Elementary + one platform; vendor-neutral lineage standardizes on OpenLineage to avoid lock-in. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#troubleshooting]

References

Monte Carlo - What Is Data Observability? 5 Key Pillars (montecarlo.ai, updated 2025/2026). https://montecarlo.ai/blog-what-is-data-observability/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Barr Moses / Monte Carlo - Introducing the 5 Pillars of Data Observability (Medium / Towards Data Science, 2021; foundational). https://medium.com/data-science/introducing-the-five-pillars-of-data-observability-e73734b263d5 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
TechTarget - 5 pillars of data observability bolster the data pipeline (2024). https://www.techtarget.com/searchdatamanagement/tip/Pillars-of-data-observability-bolster-data-pipeline [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Monte Carlo - 12 Data Quality Metrics That Actually Matter (data downtime, TTD/TTR) (2024). https://www.montecarlodata.com/blog-data-quality-metrics/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
dbt Labs - What are data SLAs? Best practices for reliable pipelines (2024). https://www.getdbt.com/blog/data-slas-best-practices [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Bigeye - The complete guide to understanding data SLAs (2024). https://www.bigeye.com/blog/the-complete-guide-to-understanding-data-slas [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
SYNQ - Why incidents must be the basis of data reliability SLAs (2024). https://www.synq.io/blog/2024-03-incident-management [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
SYNQ - Data Observability Guide 2025. https://www.synq.io/blog/data-observability-guide [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
incident.io - SLOs, SLAs, and SLIs: a complete guide (2024/2025). https://incident.io/blog/slo-sla-sli [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Atlassian - SLA vs SLO vs SLI (2024). https://www.atlassian.com/incident-management/kpis/sla-vs-slo-vs-sli [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
OpenLineage - Getting Started + standard docs (2024/2025). https://openlineage.io/getting-started/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
OpenLineage - GitHub (open standard for lineage metadata). https://github.com/OpenLineage/OpenLineage [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Marquez Project - Trying Out the New Column Lineage Feature (2024). https://marquezproject.ai/blog/column-lineage-demo/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Atlan - Marquez (WeWork): Architecture, Features & Use Cases (2025). https://atlan.com/marquez-wework-open-source/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Branch Boston - Great Expectations vs Deequ vs Soda compared (2024/2025). https://branchboston.com/great-expectations-vs-deequ-vs-soda-data-quality-testing-tools-compared/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
DataKitchen - The 2026 Open-Source Data Quality and Data Observability Landscape. https://datakitchen.io/blog/the-2026-open-source-data-quality-and-data-observability-landscape/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Elementary - How anomaly detection works (docs, 2024/2025). https://docs.elementary-data.com/data-tests/how-anomaly-detection-works [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Elementary - GitHub (dbt-native observability). https://github.com/elementary-data/elementary [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Atlan - Top 14 Data Observability Tools in 2026: Features & Pricing (Bigeye, Anomalo, etc.). https://atlan.com/know/data-observability-tools/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Conduktor - Data Quality vs Data Observability: Key Differences (2025). https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Metaplane - Data quality vs data observability: how they differ but work together (2024/2025). https://www.metaplane.dev/blog/data-quality-vs-data-observability [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Gable - The Shift Left Data Manifesto (data contracts) (2024/2025). https://www.gable.ai/blog/shift-left-data-manifesto [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Sifflet - Data Observability: Five Years In, Why the Old Playbook Fails (2025). https://www.siffletdata.com/blog/data-observability-five-years-in-why-the-old-playbook-doesnt-work-anymore [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
VictoriaMetrics - Anomaly Detection Handbook (seasonality, thresholds, ML) (2024). https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Promethium - Data Observability Metrics That Matter in 2026: Core KPIs. https://promethium.ai/guides/data-observability-metrics-that-matter-2026/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]
Better Stack - MTTR and other incident metrics explained (2024). https://betterstack.com/community/guides/incident-management/mttr-and-other-incident-metrics/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-19-data-observability/#references]

Where this helps

Diagnosing why a downstream dashboard shows stale or wrong numbers before a stakeholder notices, by checking freshness and volume metrics on the upstream tables first. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Onboarding a new data pipeline where automated freshness, volume, and schema checks are needed instead of manual spot-checks. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Root-causing a broken ML feature pipeline where an upstream schema change silently altered a column's type. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Negotiating data SLAs with consuming teams during a platform migration, so both sides agree on what "healthy" data means. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Project ideas

Build a lightweight freshness-and-volume monitor on top of query metadata (last-updated timestamps, row counts) for your top critical tables, alerting on threshold breach. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Wire OpenLineage events from your orchestrator (Airflow, dbt) into a lineage graph store like Marquez to get automatic upstream/downstream impact analysis. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Implement schema-diff checks in CI (dbt tests, Great Expectations) that block a PR when a producer changes a column's type or removes a field consumers depend on. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Stand up a data-incident runbook (page the owner, log the blast radius, root-cause template) modeled on SRE incident response. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Common mistakes

Treating data observability as a tool purchase rather than a practice — buying a monitoring platform without first defining which tables are tier-1/critical just produces alert noise. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Setting freshness and volume thresholds once at rollout and never revisiting them as the pipeline's actual usage pattern evolves. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Alerting the platform team on every anomaly instead of routing to the data producer who can actually fix the root cause, which trains everyone to ignore the channel. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Skipping shift-left data contracts and instead reacting after each schema break, so every producer change becomes a downstream fire drill. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Known issues

Automated anomaly detection on pipeline metrics is prone to false positives around known seasonal or business-cycle swings (e.g. month-end batch spikes) unless the detector is tuned to expected patterns. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Full column-level lineage is expensive to compute and maintain across a many-hop pipeline, and many lineage tools advertise column-level support but only reliably deliver table-level lineage. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Data observability tooling only monitors what it's told to monitor — it won't catch a semantic bug (correct schema, correct volume, wrong join logic) that never manifests as a pipeline metric anomaly. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Data SLAs are only useful if someone owns enforcement; publishing an SLA without an on-call rotation or escalation path just creates false confidence. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Children

Five Pillars (freshness, volume, schema, distribution, lineage) (frontier)
Data Downtime (frontier)
Data SLAs/SLOs/SLIs (frontier)
Data Incident Management (frontier)
Anomaly Detection on Pipelines (frontier)
Data Lineage and OpenLineage/Marquez (frontier)
Observability Tooling Landscape (frontier)
Shift-Left and Data Contracts (frontier)
Observability vs Quality vs Testing vs Monitoring (frontier)

Frontier under this node: Anomaly Detection on Pipelines, Data Downtime, Data Incident Management, Data Lineage and OpenLineage/Marquez, Data SLAs/SLOs/SLIs, Five Pillars (freshness, volume, schema, distribution, lineage), Observability Tooling Landscape, Observability vs Quality vs Testing vs Monitoring, Shift-Left and Data Contracts

← the whole tree · 3D view· how to read this page