Definition
LLM-as-a-judge
LLM-as-a-judge is an evaluation method where a strong language model grades or ranks the outputs of another model, replacing or supplementing human judges. It is fast and cheap at scale, but carries documented biases (position, verbosity, self-preference) and must be validated against human preference data to be trustworthy.
Strengths and failure modes
For clearly-specified criteria (factuality against a reference, format compliance, instruction following), LLM judges correlate well with humans and cost orders of magnitude less. For subjective quality (aesthetics, tone, creative merit, cultural fit), correlation drops sharply, and the judge's systematic biases compound silently: preferring longer answers, preferring its own model family's style, favoring the first option shown.
Calibrating against humans
The emerging standard practice is hybrid: run the LLM judge on everything, and continuously validate it against human preference judgments on a sample. Measure judge–human agreement per task type; where agreement is high, trust the judge and spend human budget elsewhere; where it is low or drifting, the humans are the eval. This makes periodic human preference collection a permanent fixture of any serious eval stack, not a one-time bootstrap.
Common questions
For objective criteria, largely yes. For subjective quality, no: LLM judges must be periodically calibrated against human preference data, or their biases (verbosity, position, self-preference) silently distort your metrics.
Related terms