Question-led guide · evaluation
How many trials does a stochastic agent evaluation need?
A decision-driven trial plan using uncertainty, task slices, correlated runs, serious failures, stopping rules, and paired comparisons.
Direct answer
There is no universal trial count. Start from the decision, smallest meaningful difference, acceptable uncertainty, serious-failure tolerance, and task slices. Use repeated independent trials for stochastic tasks, paired configurations on the same prepared environments, and confidence intervals rather than raw success rates. Predefine maximum trials and stopping rules; rare high-impact failures need targeted stress tests, not only a larger average-case sample.
Scope
Use this guide when repeated executions of the same agent task can differ and the team needs a regression, routing, or release decision. It does not provide a single online calculator because the correct model depends on the metric, dependence structure, and decision.
Why it happens
Teams often choose three, ten, or one hundred trials by convention. A raw success rate then appears precise even when the denominator is small, tasks are correlated, or the result is dominated by easy cases. Conversely, teams spend large budgets repeating low-risk cases while serious failure modes receive no targeted coverage.
The desired claim matters. Detecting a large regression needs fewer observations than distinguishing two nearly equal configurations. Showing occasional capability differs from showing reliable repeated success. A release veto for one unauthorized effect has different logic from estimating an average style score.
Diagnosis
Write the decision sentence first: “We will release configuration B if…” Include the primary metric, task slices, minimum practical improvement or non-inferiority margin, serious failures, and uncertainty criterion.
Inspect dependence. Runs sharing the same user, repository, seed package, simulator, day, or external service are not automatically independent. Task-level variation may exceed trial-level variation, so adding tasks can be more valuable than repeating one task.
Run a pilot only to estimate variability and test the harness. Keep pilot data out of the final decision if it influenced task, prompt, threshold, or grader design, unless the statistical plan explicitly handles adaptation.
Solution
Use paired comparisons: prepare the same environment snapshot for each configuration and randomize execution order where practical. Repeat tasks enough to estimate stochastic variation, but report results hierarchically by task and slice.
Predefine interim checks and a maximum evaluation budget. Stop early only under the declared rule—for overwhelming regression, serious failure, futility, or sufficient precision. Preserve every valid run and every harness failure separately; do not stop after a favorable streak.
For rare high-impact risks, author targeted adversarial and fault-injection tasks. More ordinary trials cannot substitute for threat-specific coverage.
Artifact
The trial-analysis plan contains:
| Field | Required decision |
|---|---|
| Decision and owner | Release, rollback, routing, or further research |
| Unit of analysis | Run, task, user, environment, repository, or session |
| Primary metric | Definition, denominator, observation window, and oracle |
| Practical difference | Improvement or non-inferiority margin that changes action |
| Slices | Risk, task family, language, tool, customer, and difficulty |
| Serious failures | Veto rule and targeted coverage |
| Pairing/blocking | Shared environment, task, seed, order, and contamination control |
| Trial range | Minimum, maximum, and allocation across tasks/slices |
| Uncertainty | Interval or decision criterion and assumptions |
| Stopping rule | Success, regression, futility, harness failure, and budget exhaustion |
| Reporting | All runs, exclusions, versions, missingness, and sensitivity checks |
Common mistakes
- Publishing a percentage without the denominator or interval.
- Repeating one task many times while covering too few tasks.
- Treating simulator or external-service runs as independent when they share state.
- Stopping after the result crosses a desired threshold without a predeclared rule.
- Using average-case sample size to claim absence of rare serious failures.
Evidence
Agent evaluation should account for stochasticity through repeated trials and report task-level behavior rather than rely on one run.
Anthropic's agent-evaluation guidance discusses trials, pass rates, pass@k/pass^k concepts, task suites, and sources of variability.
Primary source · official-doc · checked Aug 26, 2026
Limit: The article provides practical guidance, not a sample-size formula or independent validation for every evaluation design.
Repeated reliability can reveal that success on one trial differs from consistent success across multiple opportunities.
Tau-bench evaluates tool-agent-user interaction and reports pass^k to characterize consistency across repeated runs in its domains.
Primary source · paper · checked Aug 26, 2026
Limit: The domains, simulators, metrics, and model versions are bounded; pass^k is not the only reliability measure for production agents.
Trial planning should be documented before results are observed and tied to an explicit decision and uncertainty target.
The trial-analysis plan below reduces post-hoc stopping and makes scarce evaluation budget auditable.
Signal Studio author framework · reviewed Aug 26, 2026
Limit: A qualified statistician should review high-stakes designs, clustered data, adaptive experiments, and rare-event claims.
Limitations
Confidence intervals do not correct biased tasks, drifting environments, invalid graders, correlated runs, or unrepresentative slices. Rare failures may remain unseen even after many trials. The plan bounds sampling uncertainty, not total evaluation validity.
FAQ
- Is one run enough for deterministic settings?
- Only if the complete system is deterministic and the claim is limited to that exact configuration and state. Hosted models, tools, networks, clocks, and simulators often retain variability even at fixed generation settings.
- Should every task receive the same number of trials?
- Not necessarily. Allocate more trials to variable, high-impact, decision-critical slices while preserving a predeclared design and enough coverage to avoid hiding weak groups.
Related guides
Continue within AI agent evaluation, or use one of these adjacent diagnostics:
Editorial QA: automated native-English, structure, source-presence, and link checks completed . This record is not an independent expert endorsement. Review boundary.