Question-led guide · decision
What can an LLM judge score actually support?
Attach a judge score to its rubric, corpus, uncertainty, and intended decision so it is not mistaken for universal answer quality.
Direct answer
A judge score supports only the tested claim: a defined verdict under a specific rubric, evidence set, case population, threshold, and model revision. It may help triage review or compare a bounded release candidate. It does not prove overall truthfulness, safety, or customer satisfaction. Publish the denominator, severe-failure count, uncertainty, and untested groups beside the score, then choose an action no broader than those observations justify.
Translate the number into a sentence
“Accuracy 0.87” leaves out what was classified, which label was positive, and how cases were sampled. Replace it with a claim such as: on a held-out set of dated support answers, the judge agreed with the adjudicated rule on the stated fraction, with specified severe misses. That sentence still needs uncertainty and subgroup limits. The publication decision comes after those facts, not inside the number.
Separate ranking from gating
A judge may rank two answers usefully yet fail a binary disqualifying-claim gate. A high correlation with human preferences does not establish acceptable miss rate for a rare serious error. Conversely, a conservative screening judge may produce many false alarms but still reduce risk if review capacity is available. Align the metric with the actual downstream action and its error cost.
The broad release claim outruns the sample
A fictional team reports 87% agreement on 300 English support questions about two products. It proposes deployment to all languages and products. The tested corpus contains no billing disputes or missing-policy cases. The score may support a bounded pilot on similar English questions, but cannot justify those untested populations. The team writes the proposed rollout scope on the claim card and marks the missing groups.
Attach a claim card to the score
A compact card prevents an isolated number from escaping its conditions.
| Field | Required statement |
|---|---|
| Target decision | Triage, compare, block, or release |
| Standard | Rubric and reference revision |
| Population | Sampling frame and excluded groups |
| Errors | Misses, false alarms, abstentions |
| Uncertainty | Count, interval, and contested labels |
| Action | Exact rollout the evidence permits |
Review the untested boundary first
List case families absent from the set, changed policies, new products, languages, input lengths, and adversarial attempts. The point is not to add every possible category immediately; it is to prevent an unobserved category from being treated as a passing result. Choose targeted tests for categories whose failure would change the release decision. Preserve an independent human review path during initial deployment.
Update the claim when the system changes
A new judge model, prompt, retrieval source, rubric, or product policy can alter the target being measured. Re-run the relevant evaluation and give the new result its own identity. Do not overwrite the earlier score or silently combine unlike test sets into a trend. A score is a dated measurement under stated conditions, not a permanent badge of reliability.
Evidence boundary for judge scores
- NIST AI evaluation toolbox: NIST analyzes assumptions and the scope of statistical conclusions in AI evaluation. It does not authorize any particular support-assistant rollout.
- Model Cards for Model Reporting: The model-card paper proposes documenting intended uses, metrics, and caveats. Documentation cannot compensate for missing tests or weak reference labels.
The 0.87 score and sample are constructed. A real claim card needs exact evaluation records and uncertainty estimates.
Evidence
Statistical validity limits what an AI evaluation result can support.
NIST analyzes assumptions and the scope of statistical conclusions in AI evaluation.
Primary source · paper · checked Oct 7, 2026
Limit: It does not authorize any particular support-assistant rollout.
Evaluation results need intended-use and limitation context.
The model-card paper proposes documenting intended uses, metrics, and caveats.
Primary source · paper · checked Oct 7, 2026
Limit: Documentation cannot compensate for missing tests or weak reference labels.
Limitations
The claim card describes evidentiary scope. It cannot establish population transfer or compensate for an invalid rubric.
FAQ
- Can a high agreement score prove an assistant is safe?
- No. It measures a bounded comparison under a specific standard and sample; safety depends on consequence-specific failures and deployment controls.
- May a score support a limited pilot?
- Yes when the pilot population and action are inside the tested scope, with monitoring and a withdrawal rule.
Related guides
Continue within LLM judge evaluation, or use one of these adjacent diagnostics:
Editorial QA: automated native-English, structure, source-presence, and link checks completed . This record is not an independent expert endorsement. Review boundary.