How do I build a task portfolio instead of a benchmark shrine?

Scope

This method is for product teams maintaining an evaluation program across releases. It assumes task examples can be governed, versioned, and connected to environments. It does not require all production data to be copied into an evaluation set.

Why it happens

Benchmarks accumulate prestige and technical debt. Teams tune toward what is scored, preserve old cases because trend lines look stable, and add examples without removing obsolete tasks. The result is a high score on a world the product no longer inhabits.

Diagnosis

Create a coverage matrix and look for empty cells across:

• task family and user segment;
• common, uncommon, and rare-but-serious frequency bands;
• low, medium, high, and prohibited risk;
• normal, degraded, ambiguous, and adversarial environments;
• read-only, proposed effect, approved effect, and human-review paths;
• easy, boundary, compositional, and long-horizon difficulty;
• supported and accessibility-critical locales or modalities.

Then trace every portfolio weight to demand evidence, risk ownership, or an explicit conservative assumption.

Solution

Maintain separate sets: development for iteration, calibration for grader and threshold work, and decision sets protected from routine tuning. Store task cards in version control and bind each run to task, environment, model, prompt, tool, policy, and grader versions.

Use two result views. The prevalence view estimates common product behavior with frequency-aware weights. The risk view applies hard gates and deliberate sampling to serious failures. Add production failures through a review process that removes private data, confirms the failure family, and prevents near-duplicate inflation.

Artifact

Download the agent evaluation task card. It contains synthetic data and fields for initial state, outcomes, tools, budgets, fault injection, resets, deterministic checks, rubric checks, trace checks, and serious failures.

At the portfolio level, record:

Field	Purpose
task_family	Stable product or workflow grouping
segment	User, environment, locale, or risk slice
frequency_evidence	Source and date for prevalence assumptions
risk_owner	Person or function accepting the serious-failure boundary
set_role	development, calibration, or decision
weight	Prevalence or deliberate risk weight, clearly labeled
introduced_by	Product requirement, incident, red team, support, or research
review_or_retire	Date and condition for maintenance

Common mistakes

• Treating one benchmark as the product definition.
• Copying production traffic weights into a safety gate without rare-event sampling.
• Tuning on the protected decision set.
• Adding every failure as a new task without clustering the underlying mechanism.
• Deleting retired tasks and making historical results impossible to reproduce.