Definition

Side-by-side evaluation

Side-by-side (SxS) evaluation presents a judge with two outputs for the same input, typically from two models or two versions of one model, and asks which better satisfies the task. It is the standard human-evaluation design for model comparisons, powering launch decisions and public leaderboards alike.

Design details that matter

Randomize left/right position per judgment (position bias is real for humans and severe for LLM judges); blind the judge to which system produced which output; allow ties when outputs are genuinely equivalent; and collect several independent judgments per pair so you report a vote distribution, not a single opinion. Win rate with confidence intervals, not raw counts, is the decision metric.

Where SxS fits

SxS answers "is B better than A?" directly, which is usually the launch question. It scales from a quick 500-vote sanity check on a prompt-change to standing multi-model tournaments aggregated into Elo ratings. For subjective domains like image aesthetics, video realism, ad creative, and landing pages, SxS with real consumers is frequently the only evaluation that predicts market response.