Can an AI agent really find root cause from OpenTelemetry data?

Scope

Use this method for production services where the agent can query traces, logs, metrics, resource attributes, and at least one source of topology or change evidence. It applies to assisted diagnosis, not to autonomous remediation. Before implementation, the incident owner must define what “root cause” means for the organization and which claim levels require human approval.

Why it happens

Telemetry is attractive because it looks objective and machine-readable. The trap is moving from “this span was slow” to “this component caused the incident” without a discriminating test. A trace may omit unsampled requests, work outside the instrumented boundary, downstream state, or a control-plane change. Logs may record application interpretation rather than physical fact. Metrics aggregate populations and can hide the request that matters.

An agent amplifies this problem when it is rewarded for a complete narrative. If the tool contract only asks for “the root cause,” the model has no representation for partial localization, competing hypotheses, counterevidence, or an unobservable boundary.

Diagnosis

1. Name the impact and interval. State the affected user outcome, population, region, service, and evidence window. Do not start from a component name.
2. Check observation health. Record sampling, collector loss, clock uncertainty, missing resource attributes, and instrumentation changes.
3. Localize without claiming cause. Identify where the failing population diverges from a healthy comparison by path, version, resource, dependency, or change cohort.
4. Write competing mechanisms. Require at least one plausible alternative when impact is material.
5. Run a discriminating query. Ask what observation should differ if hypothesis A is true and hypothesis B is false.
6. Search for counterevidence. Healthy requests through the suspected component or failures that bypass it can weaken the claim.
7. Declare the claim level. Use observed symptom, localized fault, supported mechanism, contributing factor, root-cause claim, or unknown.

Solution

Give the agent narrow tools that return typed evidence rather than a bulk telemetry dump. Each tool response should include the query, source, time range, resource identity, sampling or completeness note, and a stable reference. The agent should maintain a hypothesis ledger with supporting evidence, counterevidence, tests attempted, and unresolved observation gaps.

Make stopping behavior part of the contract. The agent must hand over when two material hypotheses remain indistinguishable, when evidence quality falls below policy, when the suspected mechanism crosses an unobserved boundary, or when the next test would require unsafe production action.

Artifact

Use this root-cause claim checklist before accepting the final statement:

Check	Pass condition
Impact	Affected outcome, population, and interval are explicit.
Identity	Every cross-signal join uses a documented resource or request identity.
Mechanism	The claim explains how the suspected condition produced the impact.
Alternatives	Material competing hypotheses were listed and tested.
Counterevidence	Evidence that weakens the preferred explanation is visible.
Observation gaps	Sampling, missing signals, clock, and instrumentation limits are stated.
Reproduction	Another operator can rerun the decisive queries from saved references.
Authority	The claim does not itself grant permission to remediate or close the incident.

Common mistakes

• Treating the alert, last error, slowest span, latest deployment, or largest metric change as the answer.
• Joining data by display name when deployment, namespace, instance, or tenant identity differs.
• Asking the model for confidence without defining evidence thresholds or calibration data.
• Hiding missing telemetry so the narrative appears complete.
• Evaluating only whether the final diagnosis matches a label instead of grading evidence use and abstention.