Question-led guide · evaluation
How do I evaluate a semantic-layer change before release?
Compare a proposed definition against source support, old and new business questions, data-agent outcomes, and rollback obligations.
Direct answer
Evaluate the candidate at three separate levels: whether its semantic assertion has source and owner support, whether data-agent tasks improve under the revision, and whether the benefit transfers beyond the motivating case. Run old and new questions against both revisions, inspect changed rows and query plans, and require a disposition for historical results. Release behind versioned adoption and retain a reversal path if neighboring decisions regress.
Make the claim testable before running tasks
Write the proposed definition in terms of population, grain, time, source, and owner. State which assertion changed and why the old one was insufficient. A task success score alone cannot establish that the new rule is supported by business evidence. Validate the record against declared constraints and have the domain owner examine positive and counterexample cases.
Run a paired old-and-new comparison
Freeze input data and execute representative business questions under both semantic revisions. Record changed SQL, row populations, totals, and explanations. Include the motivating task and neighboring questions that depended on the old meaning. A definition that improves one answer while silently altering a finance report is not an unqualified gain. Distinguish an intended correction from an unintended regression.
The corrected dashboard breaks the board pack
In a constructed enterprise, a new active-customer rule fixes a weekly adoption dashboard by excluding trials. A monthly board report had intentionally included trials during the launch period. Both reports referenced the same semantic identifier. The candidate changes the board pack’s historical totals. The release team creates a scoped new revision and explicit migration rather than rewriting the old report without explanation.
Use a release contrast matrix
The matrix makes each acceptance claim visible.
| Gate | Old revision | Candidate revision | Decision |
|---|---|---|---|
| Source rule | Approved policy | New evidence and owner | Supported? |
| Motivating task | Wrong population | Corrected rows | Improved? |
| Neighboring task | Existing report | Changed or stable | Intended? |
| History | Interpretable output | Migration note | Preserved? |
| Recovery | Old resolver | New resolver | Reversible? |
Measure transfer beyond the motivating case
Hold out questions from another team, time period, or data source that use the same concept. Inspect whether the candidate still resolves the intended meaning and whether the agent cites the correct revision. One curated example can hide overfitting. Record uncertainty when a transfer context has a different business policy; “same label” is not enough to infer shared semantics.
Release with a migration and a stop condition
Name consumers, adoption order, cache invalidation, and historical interpretation before promotion. Retain both revisions while dependent reports migrate. Monitor changed answer distributions and owner corrections, then stop or reverse when a severe neighboring decision regresses. Reversal of a definition does not automatically repair reports already delivered under it; keep an affected-output ledger for follow-up.
Evidence boundary for semantic-layer releases
- W3C SHACL: SHACL defines validation against RDF data graphs. Passing a shape does not prove business-task improvement.
- dbt semantic models: dbt documents semantic model components and their analytical use. It does not approve the fictional adoption-policy migration.
The board-pack example is invented. Actual changed reports and owners must be identified before release.
Evidence
Shape constraints can validate structural conditions on semantic data.
SHACL defines validation against RDF data graphs.
Primary source · standard · checked Oct 7, 2026
Limit: Passing a shape does not prove business-task improvement.
Semantic-model structures define measures and entities consumed by analytics.
dbt documents semantic model components and their analytical use.
Primary source · official-doc · checked Oct 7, 2026
Limit: It does not approve the fictional adoption-policy migration.
Limitations
The matrix is an evaluation framework. It depends on valid source rules, representative data, consumer inventory, and release authority.
FAQ
- Is one fixed task enough to approve a change?
- No. Test the motivating case and neighboring decisions that may share the definition, plus a held-out transfer context.
- Can rollback restore already published reports?
- No. It changes future resolution; previously delivered outputs need an affected-output review and possible correction.
Related guides
Continue within Evolving semantic layers for data agents, or use one of these adjacent diagnostics:
Editorial QA: automated native-English, structure, source-presence, and link checks completed . This record is not an independent expert endorsement. Review boundary.