# Pairwise ranking

> Score a group of 2 to 16 candidates from one-pair-at-a-time human votes.

Pairwise ranking turns a group of candidates into a score for each. Raters see one pair at a time and pick the better of the two, and all of the group's votes are fitted into one ranking.

## When to use it

- Scoring a group of rollouts for the same prompt, for group-relative advantages (GRPO) on a [stream](/docs/streams)
- Ranking more candidates than one rater could order at once
- When you want a full win matrix as well as an order

For exactly two candidates, [comparison](/docs/task-types/comparison) asks the same question and adds consensus and confidence fields. To have one rater order a small set in one go, use [ranking](/docs/task-types/ranking).

## Datapoint shape

```json
{
  "media": {
    "candidates": [
      { "url": "dp://b6a1cd3f1234/a.mp4", "type": "video" },
      { "url": "dp://c8f3ae9b5678/b.mp4", "type": "video" },
      { "url": "dp://a3e901fd8c67/c.mp4", "type": "video" }
    ],
    "reference": [{ "url": "dp://d41f7a2c9e10/prompt.png", "type": "image" }]
  }
}
```

<ParamTable
  title="media"
  params={[
    { name: "candidates", type: "array (2–16)", required: true, description: "All text, `{\"text\": \"...\"}`, or all one media type, `{\"url\": ...}` or `{\"media_id\": ...}`. A candidate is identified by its position: the first is `0`." },
    { name: "reference", type: "array (0–10)", required: false, description: "Image, video or audio items shown with every pair and never chosen. Media candidates only." }
  ]}
/>

Text candidates:

```json
{
  "media": {
    "candidates": [
      { "text": "Water it every day." },
      { "text": "Give it six hours of sun and let the soil dry between waterings." },
      { "text": "Basil grows anywhere." },
      { "text": "Use a pot with drainage and pinch off the flowers." }
    ]
  }
}
```

Pairwise ranking is for standalone jobs and streams: it takes no chain `steps` and no `dimensions`.

## Vote budget

`max_responses_per_datapoint` (on a stream, `max_responses_per_item`) is the total number of votes a datapoint collects, B. A group of G candidates has P = G·(G−1)/2 pairs, which are put in rounds: round 1 pairs each candidate i with i+1, round 2 with i+2, and so on, wrapping around after the last candidate, so round 1 is a cycle through every candidate. Each pair in that order gets ⌊B/P⌋ votes and the first B mod P pairs one more, so shares differ by at most one and add up to B. A pair with no share is never shown.

The minimum budget is G (1 for two candidates), which covers round 1, so every candidate is compared. The default budget of 10 is below the minimum for 11 to 16 candidates: set it for groups that large.

| Candidates (G) | Pairs (P) | Minimum budget | Budget (B) | Votes per pair |
|---|---|---|---|---|
| 2 | 1 | 1 | 10 | 10 |
| 4 | 6 | 4 | 6 | 1 for each pair |
| 4 | 6 | 4 | 10 | 2 for 4 pairs, 1 for 2 |
| 8 | 28 | 8 | 100 | 4 for 16 pairs, 3 for 12 |
| 16 | 120 | 16 | 48 | 1 for the first 48 pairs in round order; the other 72 are never shown |

- Billing follows the budget, not the number of pairs: a datapoint bills at most B responses.
- Raters see one pair at a time, left and right in random order, and may answer several pairs of the same datapoint, each once. Use a larger budget when you want more independent raters per group.
- With fewer votes than pairs, the pairs early in round order get them, so the allocation depends on candidate positions. Shuffle your candidates if position shouldn't matter.

## Result shape

Keys are candidate positions, and every position is present. Values below are for four candidates that 6 votes ranked 0, 1, 2, 3.

```json
{
  "total_responses": 6,
  "comparisons": { "0": 3, "1": 3, "2": 3, "3": 3 },
  "wins": [[0, 1, 1, 1], [0, 0, 1, 1], [0, 0, 0, 1], [0, 0, 0, 0]],
  "win_rate": { "0": 1.0, "1": 0.667, "2": 0.333, "3": 0.0 },
  "scores": { "0": 1.2264, "1": 0.3783, "2": -0.3783, "3": -1.2264 },
  "ranking_order": ["0", "1", "2", "3"],
  "weighted_wins": [[0.0, 1.0, 1.0, 1.0], [0.0, 0.0, 1.0, 1.0], [0.0, 0.0, 0.0, 1.0], [0.0, 0.0, 0.0, 0.0]],
  "weighted_scores": { "0": 1.2264, "1": 0.3783, "2": -0.3783, "3": -1.2264 },
  "weighted_ranking_order": ["0", "1", "2", "3"]
}
```

<ParamTable
  params={[
    { name: "total_responses", type: "int", description: "Votes counted." },
    { name: "comparisons", type: "{ [position]: int }", description: "Votes each candidate took part in." },
    { name: "wins", type: "int[][]", description: "A G×G matrix: `wins[i][j]` counts how often candidate i beat candidate j." },
    { name: "win_rate", type: "{ [position]: float | null }", description: "Wins over comparisons, rounded to 3 decimals." },
    { name: "scores", type: "{ [position]: float | null }", description: "Bradley–Terry log-strengths fitted over all of the datapoint's pairs: centred on 0, rounded to 4 decimals, higher is better." },
    { name: "ranking_order", type: "array of positions", description: "Compared candidates sorted by score, ties going to the lower position." },
    { name: "weighted_wins", type: "float[][]", description: "Like `wins`, summing each vote's reliability weight, rounded to 3 decimals." },
    { name: "weighted_scores", type: "{ [position]: float | null }", description: "The same fit over the votes with a positive reliability weight, the weights rescaled to a mean of 1." },
    { name: "weighted_ranking_order", type: "array of positions", description: "Candidates sorted by `weighted_scores`." }
  ]}
/>

- A candidate with no votes yet has `comparisons` 0, null `win_rate` and `scores`, and no place in either ranking order.
- Until the voted pairs connect every candidate, scores rest mostly on the fit's prior.
- A vote with zero reliability weight is still billed and counted in `comparisons`, `wins`, `win_rate`, `scores` and `ranking_order`, but the weighted fit leaves it out.
- Keys stay positions with `?labels=true`, because pairwise candidates have no labels.
- On a stream, an item whose media was never stored, such as one that expired before it was ingested, has no positions at all: `comparisons`, `win_rate`, `scores` and `weighted_scores` are `{}`, and `wins` and `weighted_wins` are `[]`.

## Example

<CodeTabs>
  <CodeTab lang="curl" label="curl">
```bash
curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "video-group-ranking-v1",
    "instruction": "Which video looks more realistic?",
    "task_type": "pairwise_ranking",
    "max_responses_per_datapoint": 12,
    "datapoints": [
      {
        "media": {
          "candidates": [
            { "url": "dp://b6a1cd3f1234/a.mp4", "type": "video" },
            { "url": "dp://c8f3ae9b5678/b.mp4", "type": "video" },
            { "url": "dp://a3e901fd8c67/c.mp4", "type": "video" },
            { "url": "dp://e5b2d8f04a91/d.mp4", "type": "video" }
          ]
        }
      }
    ]
  }'
```
  </CodeTab>
</CodeTabs>

## Common errors

| Status | Cause |
|---|---|
| `400` | Fewer than 2 or more than 16 candidates. |
| `400` | Candidates mix text and media, or media of different types. |
| `400` | A candidate has no `text`, `url` or `media_id`, or its `text` is empty. |
| `400` | Text candidates with a `reference`, or more than 10 references. |
| `400` | A budget below the minimum for the datapoint's number of candidates. |
| `400` | Chain `steps` or `dimensions` on a pairwise ranking job. |

On a stream, the item checks above run at each [push](/docs/api/streams#push-items), with the stream's `max_responses_per_item` as the budget. A push with a bad item is refused whole with `400 invalid_items`, and `detail.errors` lists each bad item as `{"item", "error"}`, where `item` is its position in `items`. A stream takes no `steps` or `dimensions` at all: sending either to `POST /streams` is a `422`.

## Next

- [Online RL with Streams](/docs/streams): pairwise ranking for GRPO groups
- [Ranking](/docs/task-types/ranking): one rater orders the whole set
- [API Reference: Jobs](/docs/api/jobs)