Question-led guide · comparison
Should I grade the final outcome or the agent trace?
A grader design that makes verified outcomes primary while using trace checks selectively for policy, safety, evidence, and diagnosis.
Direct answer
Grade the verified final outcome as the primary measure whenever the environment exposes one. Add trace checks for behavior that outcomes cannot reveal: unauthorized tools, fabricated evidence, duplicate effects, budget breaches, required approvals, and serious safety violations. Avoid rewarding imitation of one reference path. Use traces diagnostically and constrain only paths whose policy or causal relevance is explicit.
Scope
Use this guide when an agent can take several valid paths to a goal and the evaluation team is deciding whether to score end state, trajectory, or both. It applies to tool use, research, coding, support, and operational workflows.
Why it happens
Outcome-only scoring is attractive because it is close to user value. It can hide an unauthorized read, fabricated citation, repeated side effect, or a lucky guess. Trace scoring exposes behavior, but often rewards conformity to the task author’s path. Agents that discover a shorter safe route are penalized, while verbose imitation earns points.
The choice becomes clearer when each grade has a job. Outcomes answer whether the intended world state or deliverable exists. Trace constraints answer whether prohibited or required behaviors occurred. Other trace fields are diagnostics, useful for improvement but not necessarily part of the pass decision.
Diagnosis
For each current trace check, ask:
- Could the final state hide the behavior this check detects?
- Is the behavior required by policy, safety, evidence, or resource limits?
- Are multiple safe trajectories possible?
- Can the check be deterministic from tool and effect records?
- Would a model judge be evaluating style instead of a real constraint?
Move a check out of the gate when its only rationale is “the reference trace did it this way.” Keep it as a diagnostic label if it still helps engineering.
Solution
Define outcome oracles from authoritative state: files, records, messages, tests, permissions, or verified user results. Separate required outcomes from forbidden outcomes.
Define trace policy as a small set of invariants: permitted tools and resources, required confirmation, evidence references, effect identity, budgets, and serious failures. Score deterministic facts deterministically. Use rubric or model graders only where interpretation is unavoidable, and calibrate them against qualified human review.
Report a grade vector rather than an opaque weighted number: outcome, policy/safety, evidence, efficiency, and handoff. A serious failure can veto release; a harmless alternate path should not.
Artifact
Create one grader card per task family:
| Card field | Definition |
|---|---|
| Decision | Regression alert, release gate, routing, or research comparison |
| Outcome oracle | Authoritative end-state query and observation window |
| Required outcome | Conditions that must hold |
| Forbidden outcome | Effects that must remain absent |
| Trace invariants | Permissions, approvals, evidence, budgets, and required milestones |
| Serious failures | Individually blocking events; no averaging |
| Alternate paths | Known valid strategies and policy for unseen safe paths |
| Diagnostic signals | Non-gating steps, loops, plans, and error categories |
| Grader type | Deterministic, rubric, model-assisted, or human |
| Audit | Calibration set, disagreement review, version, owner, and expiry |
Common mistakes
- Using exact sequence match for an open-ended task.
- Grading model-written rationale instead of observed tool and effect records.
- Averaging an unauthorized effect with stylistic or efficiency points.
- Treating one delayed read as proof that an effect failed.
- Changing trace instrumentation without versioning the grader contract.
Evidence
Agent evaluations can combine outcome, transcript, trajectory, and environment checks according to the task and grader design.
Anthropic's engineering guidance discusses tasks, trials, environments, graders, outcome checks, trajectories, and capability versus regression evaluations.
Primary source · official-doc · checked Aug 26, 2026
Limit: It is first-party practice guidance and does not validate a universal weighting between outcome and trace grades.
Tool-agent evaluation can use stateful environments, milestone checks, and minefields rather than exact path matching alone.
ToolSandbox defines stateful tool-use tasks with milestones and minefields for partial credit and critical failures.
Primary source · paper · checked Aug 26, 2026
Limit: Manually authored milestones and minefields can be incomplete, and benchmark tasks do not cover every production policy.
A grader card should state which decision each check supports and which alternative safe trajectories remain valid.
The artifact below separates success, required constraints, serious failures, and diagnostic trace signals.
Signal Studio author framework · reviewed Aug 26, 2026
Limit: Domain reviewers must identify real outcome oracles and avoid encoding stylistic preferences as safety requirements.
Limitations
Some outcomes are delayed, partially observable, or subjective, and some reasoning is not visible in traces. Instrumentation can change behavior or omit actions. Grader validity depends on task-specific oracles, policy coverage, and ongoing audit.
FAQ
- Can a bad trace produce a correct outcome?
- Yes. It may guess correctly, use unauthorized data, create duplicate effects, or waste unsafe resources. Outcome success does not erase serious trace-level violations.
- Can a good trace receive a failing outcome grade?
- Yes. A plausible process can still fail the task. Use the trace to diagnose where, but do not award full success for following expected steps without achieving the required effect.
Related guides
Continue within AI agent evaluation, or use one of these adjacent diagnostics:
Editorial QA: automated native-English, structure, source-presence, and link checks completed . This record is not an independent expert endorsement. Review boundary.