How do I stress-test an LLM judge against shortcuts?

Hold the rule-bearing fact constant

A shortcut probe changes one irrelevant feature at a time while preserving the content that should decide the verdict. If the rubric checks unsupported claims, keep the same claim and supporting documents. Rewrite tone, layout, or response length. A changed verdict suggests the judge may be using presentation as a proxy. First verify that the transformation did not accidentally alter meaning.

Probe known comparison biases explicitly

For pairwise judgments, swap candidate positions and repeat the trial. For single-answer classification, vary length, confident wording, citation style, and benign formatting. The model-judge literature documents position and verbosity bias in particular settings; that evidence motivates a test, not a conclusion that every judge has the same bias. Record model revision and sampling settings for each run.

A beautiful answer promises a nonexistent feature

Consider a fictional support reply with six polished paragraphs and a fake premium entitlement. Its paired version is a terse but accurate two-sentence answer. A judge that rewards the long reply because it looks comprehensive violates the disqualifying-claim rubric. The team constructs another pair where the long answer is accurate, ensuring the judge is not merely trained to distrust length.

Use a pairing sheet rather than anecdotes

Record each controlled change and expected invariant.

Pair	Held constant	Changed cue	Expected result
A/B	Unsupported entitlement	Length	Both fail
B/A	Same answer contents	Display order	Same preference
C/D	Supported claim	Confidence wording	Both pass
E/F	Missing policy evidence	Format	Both abstain

Count inconsistency by case family

Do not stop at one dramatic example. Test multiple paraphrases, product policies, and evidence states. Report the fraction of pairs whose decision changes, plus the severe errors among those changes. Run the same controlled suite after prompt, model, or rubric updates. A change that fixes position sensitivity may introduce a new false-alarm pattern, so retain the independent reference labels.

Know what stress tests cannot estimate

An enriched adversarial suite is deliberately unlike ordinary traffic. Its inconsistency rate is a robustness finding, not a production incident rate. Use representative cases for prevalence and paired cases for mechanisms. When a shortcut is found, repair the prompt or pipeline, then test the repair on a holdout and on cases where the suspicious cue is genuinely relevant.

Evidence boundary for judge shortcuts

• Judging LLM-as-a-Judge research: The paper analyzes several biases in model-judge benchmarks. The reported effects are task and model specific.
• Systematic position-bias study: The study examines position consistency and related factors across model judges. Its pairwise benchmarks do not determine this support judge’s failure rate.

The support pairs are invented. Stress tests must preserve the actual rubric-bearing facts for a valid comparison.