How do I replay incidents and grade an AI investigation?

Scope

Use incident replay to compare agent versions, test a control change, reproduce a known diagnostic failure, or exercise handoff behavior. The target is the investigation, not a cinematic reconstruction of the outage. Begin with read-only environments; remediation evaluation requires a separate sandbox and effect model.

Why it happens

Teams often convert a postmortem into a question and expected root-cause label. That leaks hindsight and removes the operational work. The agent never has to discover the impact window, cope with missing signals, choose a safe query, test a competing mechanism, or decide that evidence is insufficient.

Distributed incidents also depend on state: release versions, routing, caches, queues, topology, credentials, clocks, and provider behavior. A static bundle of logs may produce a repeatable text task while failing to reproduce the decision environment.

Diagnosis

Audit an existing replay with four tests:

1. Leakage test: search prompts, tool descriptions, filenames, expected answers, and telemetry annotations for post-incident language.
2. State test: run twice after reset and compare starting resources, data, clocks, credentials, and tool behavior.
3. Observability test: verify that information becomes available according to the replay timeline rather than all at once.
4. Alternative-path test: determine whether a safe investigation that differs from the author path can still receive full credit.

If the final label can be guessed from task metadata, the replay measures recognition. If the world cannot be reset, score changes may reflect environment drift.

Solution

Build a replay package with immutable evidence snapshots where possible and controlled simulators where necessary. Define an observation clock so queries cannot see future data. Version topology, service builds, runbooks, policy, tool schemas, grader code, and model configuration. Provide deterministic reset and health checks.

Create multiple grades. Outcome grading asks whether the agent localized the issue or supported a mechanism. Process grading checks evidence references, hypothesis discrimination, counterevidence, and observation gaps. Safety grading covers permissions and forbidden effects. Efficiency grading covers elapsed time, queries, load, tokens, and human effort. An unsupported correct guess should not pass the evidence grade.

Run more than once because the agent is stochastic. Preserve every run, including tool errors and grader versions, and report results by incident slice instead of only an average.

Artifact

The incident-replay manifest should include:

Manifest group	Required fields
Task identity	Replay ID, incident family, owner, purpose, version, and leakage review
Initial world	Services, builds, data, topology, traffic, queues, clocks, and external dependencies
Fault model	Trigger, affected population, duration, recovery behavior, and hidden ground truth
Observation surface	Signals, sampling, delays, gaps, retention, and time-release rules
Tools and authority	Schemas, versions, scopes, rate limits, costs, forbidden calls, and approvals
Reset proof	Cleanup steps, seed, health checks, and acceptable nondeterminism
Grading	Outcome, evidence, process, safety, efficiency, handoff, and serious failures
Trials	Model/config versions, run count, stopping rule, and uncertainty report
Provenance	Source incident, transformed artifacts, reviewers, and change history

Common mistakes

• Naming fixtures after the historical root cause.
• Giving the agent the final postmortem chronology at time zero.
• Grading only exact diagnosis text or only similarity to one trace path.
• Allowing state, credentials, or external APIs to drift between model comparisons.
• Averaging an unauthorized production-like action into an otherwise good score.