Why does my RCA bot always blame the last error in the trace?

Scope

Use this guide when an incident assistant repeatedly selects the terminal exception, the reddest span, or the most recent deployment and presents it as root cause. The method is designed for diagnosis of distributed services with request traces plus at least one comparison signal. It does not authorize remediation and does not replace the broader evidentiary test for a root-cause claim.

Why it happens

Trace viewers are arranged in time, so the last visible error feels like the conclusion of a story. Models strengthen that bias: the last exception is compact, legible, and close to the final answer, while an initiating queue, exhausted pool, stale configuration, or dependency slowdown may be represented only indirectly.

The trace is also conditioned on what was recorded. A downstream span may contain an explicit timeout because the caller knows it failed; the overloaded dependency may emit no error at all. Retries can place a late error after the condition that consumed the original latency budget. A bot trained or prompted to return one cause will compress those ambiguities into a confident narrative.

Diagnosis

Start with a counterexample test. Select one incident where the bot blamed the final error, then collect a healthy comparison from the same route and version.

1. State the affected user outcome and time window before naming a component.
2. Mark every error as initiating candidate, propagated consequence, recovery action, or unclassified.
3. Compare failing and healthy flows at each boundary: queue, service, datastore, external API, and control plane.
4. Ask whether healthy requests also contain the alleged cause and whether failed requests can bypass it.
5. Identify the earliest divergence in behavior, which is different from the earliest error timestamp.
6. Require one competing hypothesis and one observation that would discriminate between the two.

If the bot cannot state what evidence would change its mind, it is producing a summary, not conducting an investigation.

Solution

Change the agent contract from find_root_cause(trace) to an impact-led loop. The first tool call should retrieve a bounded affected population. Subsequent calls should compare cohorts, inspect boundaries, and add observations to a ledger. Do not let the ranker use recency or span status as sufficient evidence; treat them as candidate features.

Give the agent explicit claim states: symptom, candidate location, supported mechanism, and unresolved. Promote a hypothesis only after a test produces the predicted difference and the agent has searched for material counterevidence. Store query references so an incident commander can reproduce the result.

Artifact

Use one row per live hypothesis:

Field	Required entry
Impact boundary	User outcome, affected cohort, interval, and error budget consequence
Hypothesis	A falsifiable mechanism, not a component label
Supporting observations	Stable query or trace references and what each observation establishes
Counterevidence	Healthy paths, bypass failures, timing conflicts, or data that weakens the mechanism
Predicted difference	What should be observable in affected versus control populations
Next test	The safest query or experiment that separates this hypothesis from a competitor
Observation gap	Missing signal, sampling, clock, topology, or change evidence
Claim state	candidate, weakened, supported, rejected, or unresolved

The last trace error may remain in the ledger. It simply receives no privileged status because of its position.

Common mistakes

• Replacing “last error” with “first error” without changing the reasoning method.
• Comparing requests from different releases, regions, tenants, or traffic classes.
• Treating a component with many error spans as the component that initiated the failure.
• Asking for a confidence percentage without a calibrated relationship between evidence and correctness.
• Deleting rejected hypotheses, which removes the audit trail that explains why the final candidate survived.