Question-led guide · diagnostic
How do I detect drift in an evolving semantic layer?
Monitor source, definition, mapping, and usage changes separately so a data agent does not keep producing plausible answers from stale meaning.
Direct answer
Track semantic drift as several distinct events: source schema or population changes, mapping failures, policy revisions, and new user questions outside the approved definition. Compare observed data and agent decisions with versioned invariants and owner-reviewed examples. Route each detected difference to the right repair path. A changing answer is not always an ontology defect, and stable query execution is not proof that meaning stayed valid.
A successful query can still be stale
Schema compatibility is a low bar. A status column can keep the same type while its codes or population change. An agent may continue producing clean tables and confident explanations from an obsolete mapping. Monitor the semantic assumptions that connect data to business meaning, not only query failures. Preserve when the source changed and which definition revision consumed it.
Classify the changing layer before repair
A source drift changes input data or schema. A mapping drift changes the relation between source and concept. A policy drift changes the approved business rule. A usage drift introduces questions beyond the old scope. Each has a different owner and rollback path. Updating the ontology for every anomalous total can hide a broken source pipeline; fixing a pipeline cannot settle a disputed policy.
The new status code quietly alters totals
In a constructed customer table, active status was encoded as A. A migration adds P for provisionally active accounts. Existing SQL excludes P but still executes. Weekly active-customer totals fall by 12%, and an agent calls it a decline in engagement. Source-change and mapping checks show the new code. The owner then decides whether P belongs in the metric; the agent must not make that policy choice silently.
Keep a drift triage ledger
Record the symptom and the evidence needed to assign responsibility.
| Drift class | Signal | Owner question |
|---|---|---|
| Source | New code, null rate, or dataset revision | Is input valid? |
| Mapping | Unmapped values or changed joins | Does the projection still hold? |
| Policy | Approved rule or effective date changed | Which definition applies? |
| Usage | New question family or segment | Is scope sufficient? |
| Agent | Cached revision or wrong retrieval | Which rule was actually used? |
Run invariants and sampled answer review
Use data-quality assertions for structural changes and competency questions for meaning. Compare representative answers before and after a source or semantic revision, including boundary cases. Monitor unresolved definitions, retrieval conflicts, and owner corrections. An alert should name the likely layer and source revision so responders can investigate rather than receiving an undifferentiated “ontology drift” score.
Close drift with a versioned decision
Repair the source, mapping, retrieval path, or definition according to the evidence. Keep old semantics available for historical reports and record affected outputs. Re-run the relevant task portfolio and monitor the new baseline after release. If the policy owner cannot decide a disputed population, mark the result as unresolved instead of allowing a model-generated rule to become production truth.
Evidence boundary for semantic drift
- OpenLineage object model: OpenLineage distinguishes dataset metadata, jobs, and run events. Those events do not determine the right business policy for a new code.
- W3C SHACL: SHACL defines data-graph validation against shapes. A valid graph can still carry stale or wrong business meaning.
The 12% change is illustrative. Drift thresholds and repair decisions need the real source, owner, and affected reports.
Evidence
Dataset and job metadata can reveal changes in data production.
OpenLineage distinguishes dataset metadata, jobs, and run events.
Primary source · official-doc · checked Oct 7, 2026
Limit: Those events do not determine the right business policy for a new code.
Validation constraints can catch specified shape violations.
SHACL defines data-graph validation against shapes.
Primary source · standard · checked Oct 7, 2026
Limit: A valid graph can still carry stale or wrong business meaning.
Limitations
The ledger supports triage, not automatic policy resolution. Real drift detection requires known invariants and representative questions.
FAQ
- Does a passing query mean the metric is still valid?
- No. Input codes, population, mapping, or policy may change while syntax remains valid.
- Should every drift alert update the ontology?
- No. First identify whether the issue is source, mapping, policy, usage, or agent retrieval behavior.
Related guides
Continue within Evolving semantic layers for data agents, or use one of these adjacent diagnostics:
Editorial QA: automated native-English, structure, source-presence, and link checks completed . This record is not an independent expert endorsement. Review boundary.