How do I test an AIOps diagnosis against counterevidence?

Write the claim at the level evidence supports

A spike in errors is an observation. A changed dependency is context. A statement that one caused the other is a mechanism claim. Keep those levels distinct in the incident record. Ask what the proposed cause predicts that a competing explanation would not. If the answer is only that two events happened near each other, the system has a lead, not a diagnosis.

Prefer discriminating checks over more summary

A model can collect many similar graphs without reducing uncertainty. Select one query that changes the relative credibility of two explanations: compare affected and unaffected cohorts, inspect dependency errors before deployment, or identify the first failing hop. Preserve query scope and timestamps. If telemetry sampling hides the needed population, label the check inconclusive instead of filling the gap with a fluent narrative.

The last rollout is an attractive distraction

In a fictional shop, version 42 was deployed at 11:02. Checkout failures rose at 11:05. The payment provider’s timeout rate, however, began climbing at 10:58, while an unchanged region also failed. The deployment remains relevant because it may change retry behavior, but it is not yet the initiating fault. A region comparison and provider timeline narrow the next check.

Keep a contrast ledger during the incident

Write down the prediction before running each query so the result cannot be reinterpreted after the fact.

Candidate	Predicted observation	Contrary observation	Next check
Version 42	Only updated cohort fails	Unchanged region fails	Compare retry paths
Provider outage	Timeout precedes rollout	Healthy provider path	Provider status and traces
Data gap	Errors absent because traces dropped	Collector healthy	Coverage counters

Choose a safe stopping point

The team may need to mitigate before it knows a final cause. Record a bounded action and the evidence that supports it, along with the uncertainty left open. Keep diagnosis confidence separate from authority to modify production. After impact is controlled, test the remaining alternatives and update the incident account without rewriting the earlier decision state.

Review where the reasoning could fail

Check clock alignment, sample coverage, deployment cohort mapping, and whether a dependency status page describes the same region and time. An operator should be able to reopen the source trace or metric rather than trusting a generated summary. The ledger is most useful when it captures a contradicted attractive story; a system that never records counterevidence is not performing diagnosis.

Evidence boundary for AIOps diagnosis

• Google SRE incident response: Google SRE explains structured response, roles, working records, and learning from incidents. Its incident examples do not validate the invented checkout timeline.
• OpenTelemetry signals: OpenTelemetry defines distinct trace, metric, and log signals. The signal definitions do not infer causality from temporal order.

All shop identities and times are invented. The ledger shows a method for questioning a claim, not the actual cause of a real outage.