How do I build a judge corpus that exposes serious misses?

Sample for the decision, not the easiest collection path

A production log may contain many straightforward approved answers and few documented severe errors. Random sampling alone can therefore produce a high-looking score while revealing little about the failure the judge is meant to prevent. Define the unacceptable mistake and seek cases that test it directly. Keep prevalence estimates separate from a deliberately enriched challenge set.

Make case identity and evidence reproducible

Each case needs the input, answer, permitted evidence snapshot, rubric version, human reference process, and collection reason. Remove customer secrets or use governed synthetic substitutes where possible. A case copied from a later postmortem can leak the answer into the judge input. Label the evidence actually available at decision time and record transformations that could change interpretation.

A high agreement score hides one expensive miss

Consider a fictional set of 100 support answers: 99 routine replies and one unsupported promise of premium access. A judge that passes all 100 obtains 99% agreement if the routine labels are all pass. It fails the only disqualifying case. The team expands the challenge set with varied false promises and records the severe-miss rate separately, while still keeping a representative sample for prevalence.

Use a coverage matrix before splitting data

A small matrix makes missing families visible before test numbers appear.

Case family	Collection route	Critical check
Ordinary correct	Representative production sample	False alarm
Known false claim	Expert-authored or confirmed case	Severe miss
Ambiguous evidence	Dated reference conflict	Abstention
Changed policy	Old and new effective dates	Version handling
New wording	Held-out paraphrase	Shortcut resistance

Protect the holdout from prompt iteration

Choose and freeze a subset before tuning. If a prompt author repeatedly inspects failures and rewrites the judge, those cases have become development data. Keep a separate untouched evaluation set or refresh it under a recorded process. Report the exact corpus version and sampling weights; a weighted production estimate and an enriched challenge-set rate answer different questions.

Audit what the corpus still cannot show

A finite set cannot cover every future product rule or adversarial wording. Inspect subgroups, source types, response lengths, and missing-data conditions. When a new policy launches, add a dated case family and re-evaluate instead of assuming the old corpus transfers. A good corpus is an operating asset whose gaps remain visible, not a one-time benchmark.

Evidence boundary for judge corpora

• NIST AI evaluation toolbox: NIST analyzes evaluation validity and statistical assumptions for AI measurement. It does not prescribe this support corpus composition.
• Judging LLM-as-a-Judge research: The paper examines agreement and bias in model judges on specific benchmarks. Its benchmark tasks do not determine support-case prevalence.

The 100-case arithmetic is invented and demonstrates denominator choice, not observed judge performance.