LLM-as-judge — using a model to grade the outputs of other models — is the only evaluation method that scales to open-ended generation. It is also a biased, gameable instrument that will quietly corrupt your measurements unless you calibrate it like any other measuring device.
The documented biases form a checklist. Position bias: judges prefer the first (or sometimes last) answer presented — MT-Bench and follow-up studies measured swings of tens of percentage points by swap order alone, hence the standard of evaluating both orderings. Verbosity bias: longer answers score higher independent of quality, which quietly trains your system to be verbose. Self-preference: models grade their own outputs higher — same-family judging is a systematic error, not an evaluation. Eloquence over truth: confident, fluent wrong answers beat hedged correct ones more often than teams expect.
The mitigation stack is now standard practice: rubric-bound criteria instead of "rate quality 1–10"; reference answers wherever one exists; pairwise comparison with position swapped, or with the reference anchoring; judge ensembles over a single call; and — the non-negotiable —periodic calibration against human labels, measuring judge agreement per rubric dimension and retiring judge prompts that drift. A judge without a calibration history is not an evaluation; it is an opinion with a temperature setting.
Where Judging Fits
- Good fit: open-ended quality (tone, helpfulness, summarization fidelity), large-scale A/B comparison, and pre-filtering candidate pools before human review concentrates on the finalists.
- Poor fit: anything checkable deterministically — schema, arithmetic, security properties. Never spend a judge on what an assert statement can decide.
- Danger zone: using the same model family as generator and judge on your production evals — the numbers will look great and mean little.
Used within those bounds, LLM-as-judge is genuinely transformative: it is what made evaluating helpfulness at 10,000 samples a Tuesday instead of a hiring plan. The discipline is to remember what it is — a cheap, scalable, proxy measurement that earns its keep only while it stays calibrated against the ground truth you quietly maintain.