What makes an agent test environment reproducible?

Scope

Use this guide for regression suites, capability studies, red-team exercises, and pre-release evaluations of agents that act in stateful environments. It covers the world around the agent. Task selection and trial-count decisions are separate evaluation problems.

Why it happens

Agent tests fail for reasons that ordinary input-output fixtures do not capture. A prior run changed a record, the clock crossed a deadline, a test user lost a role, a tool schema changed, a browser retained a session, or an external API returned new data. The model appears to regress or improve when the environment actually moved.

Teams sometimes solve this by mocking everything. The suite becomes repeatable but loses the latency, permission, state, and failure behavior that determines real agent performance. Reproducibility requires a deliberate boundary between controlled fidelity and declared approximation.

Diagnosis

Run the same test twice after reset and compare a state digest before the agent starts. Include databases, files, queues, browser storage, identities, resource versions, clocks, feature flags, caches, and external fixtures. If the digest differs, identify whether the field is relevant or allowed nondeterminism.

Then replay the same recorded action sequence without the model. Differences expose environment drift. Finally, run two different model configurations against the same prepared snapshot and verify that one run cannot contaminate the other.

Solution

Build environments from versioned recipes and seed packages. Give every test an isolated tenant or namespace and a deterministic setup/teardown lifecycle. Freeze time where the task permits; otherwise record it and define acceptable time behavior. Use short-lived test identities with explicit scopes.

Pin tool contracts and adapter versions. For external dependencies, choose among a pinned simulator, recorded contract fixture, dedicated test service, or live sandbox, and label the resulting fidelity. Inject faults through controlled mechanisms rather than hoping for natural errors.

Before scoring, run health and reset checks. After the test, capture end-state evidence and clean up effects. Quarantine the result if setup, observation, or reset checks fail.

Artifact

The test-environment manifest contains:

Area	Required record
Environment identity	Image, code, schema, seed, region, and configuration versions
Initial state	Resource inventory and pre-run state digest
Time and randomness	Clock policy, timezone, seeds, and allowed nondeterminism
Identity and policy	Test actors, roles, tokens, target scopes, and policy version
Tools	Schemas, adapter builds, rate limits, errors, and side-effect semantics
Dependencies	Simulator/live status, contract version, data snapshot, and known drift
Faults	Injection point, trigger, duration, cleanup, and observability
Observations	Screens, APIs, telemetry, logs, and hidden evaluator state
Reset	Procedure, verification digest, timeout, and quarantine behavior
Provenance	Builder, capture time, change history, and artifact hashes

Common mistakes

• Resetting the database but retaining browser sessions, queues, caches, or files.
• Pinning model settings while tool and dependency versions drift.
• Letting failed setup runs enter the score denominator as agent failures.
• Mocking away permissions and latency that the agent must handle in production.
• Allowing one stochastic run to change fixtures for the next trial.