How do I write an LLM judge rubric for one decision?

Name the action the verdict will influence

A judge cannot be calibrated for “good writing” in the abstract when its score will block a release. State whether it triages review, recommends a revision, or makes a pass/fail gate. A support team may care most about a false product promise; style and warmth can be recorded separately. The higher-consequence decision needs the clearest error definition and an accountable owner.

Make evidence eligibility part of the rule

List what the judge may inspect: customer request, approved product documentation, proposed answer, and perhaps a dated policy note. Exclude hidden release notes or later customer outcomes if the production judge cannot see them. Describe how to handle absent, conflicting, or outdated references. If the evidence set is unbounded, experts can reach different verdicts without either being careless.

The polished answer with one false promise

In a fictional support case, an answer accurately explains five setup steps but promises that every account includes a feature available only in a premium tier. A broad helpfulness score might pass it. The rubric says an unsupported entitlement claim is a disqualifying error even when the rest is useful. If the tier documentation is missing, the correct state is “review needed,” not an invented pass.

Write a compact decision sheet

The sheet turns a preference into an inspectable rule.

Field	Specific entry
Decision	Send for human review or permit normal release
Evidence	Versioned entitlement documentation and answer
Fail condition	Unsupported feature availability claim
Abstain	Evidence absent or conflicting
Owner	Product policy expert and evaluation lead

Test disagreement before prompting a model

Ask two experts to label the same mixed cases independently. Compare their rationales and identify whether the wording, evidence, or underlying policy caused disagreement. Rewrite the criterion and preserve the earlier rubric version. The goal is not to pressure every reviewer into agreement; it is to make the boundary and legitimate exceptions visible enough that a machine verdict can be evaluated.

Keep the output narrower than the rubric

Require a verdict, cited evidence identifier, triggered criterion, and uncertainty state. Do not let a model produce a confident reason unsupported by the allowed materials. Review both false passes and false alarms against expert decisions. A clear rubric is necessary for evaluation, but the actual judge still needs task-specific calibration on held-out cases.

Evidence boundary for judge rubrics

• NIST AI Risk Management Framework: NIST frames AI risks through governance, mapping, measurement, and management. It does not supply the support-answer rubric in this example.
• Judging LLM-as-a-Judge research: The paper studies model-judge agreement and several judgment biases. Its benchmark results cannot validate this support policy.

The entitlement case is invented. Domain experts must approve the real policy and labeled examples before deployment.