Definition

Human evaluation

Human evaluation measures an AI model's quality by having people judge its outputs through side-by-side comparisons, ratings, or task-success assessments, rather than relying on automated metrics. It is the gold standard for subjective qualities like helpfulness, realism, and aesthetics, and the benchmark against which automated evaluators are calibrated.

Common designs

Side-by-side preference (which output is better) is the workhorse: robust to calibration drift, directly comparable across model versions, and aggregatable into Elo-style rankings. Absolute rating on a rubric suits single-output monitoring. Task-success evaluation (did the agent complete the job?) fits agentic systems. Across designs, the levers are the same: clear instructions, multiple independent judges per item, and agreement statistics reported alongside the verdict.

Who the judges are

The choice of judges is part of the experiment. Expert annotators excel at technical rubrics but their taste diverges from everyday users; representative consumers reflect the population your product serves. For consumer-facing generative models, evaluating against real end users, targeted by market and demographic, answers the question that matters commercially: will the people who use this actually prefer it?