Definition
Elo rating (for AI models)
An Elo rating converts pairwise comparison outcomes into a single skill score per competitor, updating after each head-to-head result. Borrowed from chess, it is now the standard way AI leaderboards such as Chatbot Arena and generative-model benchmarks turn millions of human preference votes into a global model ranking.
From votes to rankings
Each human preference vote is a "match" between two models; Elo (or its batch cousin, the Bradley–Terry model) finds the scores that best explain all observed wins and losses. The elegance is sparsity: you never need every model to face every other on every prompt: a well-mixed sample of pairings recovers a stable ranking, with confidence intervals shrinking as votes accumulate.
Running your own leaderboard
Internal Elo leaderboards are increasingly how teams track model versions: route a stream of fresh prompts to side-by-side human votes between the incumbent and each challenger, update ratings continuously, and ship when a challenger’s interval separates. Datapoint runs a public human-preference leaderboard across image, video, and code models built exactly this way.
Related terms