Pairwise ranking

Pairwise ranking turns a group of candidates into a score for each. Raters see one pair at a time and pick the better of the two, and all of the group's votes are fitted into one ranking.

When to use it

• Scoring a group of rollouts for the same prompt, for group-relative advantages (GRPO) on a stream
• Ranking more candidates than one rater could order at once
• When you want a full win matrix as well as an order

For exactly two candidates, comparison asks the same question and adds consensus and confidence fields. To have one rater order a small set in one go, use ranking.

Datapoint shape

{
  "media": {
    "candidates": [
      { "url": "dp://b6a1cd3f1234/a.mp4", "type": "video" },
      { "url": "dp://c8f3ae9b5678/b.mp4", "type": "video" },
      { "url": "dp://a3e901fd8c67/c.mp4", "type": "video" }
    ],
    "reference": [{ "url": "dp://d41f7a2c9e10/prompt.png", "type": "image" }]
  }
}

Text candidates:

{
  "media": {
    "candidates": [
      { "text": "Water it every day." },
      { "text": "Give it six hours of sun and let the soil dry between waterings." },
      { "text": "Basil grows anywhere." },
      { "text": "Use a pot with drainage and pinch off the flowers." }
    ]
  }
}

Pairwise ranking is for standalone jobs and streams: it takes no chain steps and no dimensions.

Vote budget

max_responses_per_datapoint (on a stream, max_responses_per_item) is the total number of votes a datapoint collects, B. A group of G candidates has P = G·(G−1)/2 pairs, which are put in rounds: round 1 pairs each candidate i with i+1, round 2 with i+2, and so on, wrapping around after the last candidate, so round 1 is a cycle through every candidate. Each pair in that order gets ⌊B/P⌋ votes and the first B mod P pairs one more, so shares differ by at most one and add up to B. A pair with no share is never shown.

The minimum budget is G (1 for two candidates), which covers round 1, so every candidate is compared. The default budget of 10 is below the minimum for 11 to 16 candidates: set it for groups that large.

• Billing follows the budget, not the number of pairs: a datapoint bills at most B responses.
• Raters see one pair at a time, left and right in random order, and may answer several pairs of the same datapoint, each once. Use a larger budget when you want more independent raters per group.
• With fewer votes than pairs, the pairs early in round order get them, so the allocation depends on candidate positions. Shuffle your candidates if position shouldn't matter.

Result shape

Keys are candidate positions, and every position is present. Values below are for four candidates that 6 votes ranked 0, 1, 2, 3.

{
  "total_responses": 6,
  "comparisons": { "0": 3, "1": 3, "2": 3, "3": 3 },
  "wins": [[0, 1, 1, 1], [0, 0, 1, 1], [0, 0, 0, 1], [0, 0, 0, 0]],
  "win_rate": { "0": 1.0, "1": 0.667, "2": 0.333, "3": 0.0 },
  "scores": { "0": 1.2264, "1": 0.3783, "2": -0.3783, "3": -1.2264 },
  "ranking_order": ["0", "1", "2", "3"],
  "weighted_wins": [[0.0, 1.0, 1.0, 1.0], [0.0, 0.0, 1.0, 1.0], [0.0, 0.0, 0.0, 1.0], [0.0, 0.0, 0.0, 0.0]],
  "weighted_scores": { "0": 1.2264, "1": 0.3783, "2": -0.3783, "3": -1.2264 },
  "weighted_ranking_order": ["0", "1", "2", "3"]
}

• A candidate with no votes yet has comparisons 0, null win_rate and scores, and no place in either ranking order.
• Until the voted pairs connect every candidate, scores rest mostly on the fit's prior.
• A vote with zero reliability weight is still billed and counted in comparisons, wins, win_rate, scores and ranking_order, but the weighted fit leaves it out.
• Keys stay positions with ?labels=true, because pairwise candidates have no labels.
• On a stream, an item whose media was never stored, such as one that expired before it was ingested, has no positions at all: comparisons, win_rate, scores and weighted_scores are {}, and wins and weighted_wins are [].

Example

curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "video-group-ranking-v1",
    "instruction": "Which video looks more realistic?",
    "task_type": "pairwise_ranking",
    "max_responses_per_datapoint": 12,
    "datapoints": [
      {
        "media": {
          "candidates": [
            { "url": "dp://b6a1cd3f1234/a.mp4", "type": "video" },
            { "url": "dp://c8f3ae9b5678/b.mp4", "type": "video" },
            { "url": "dp://a3e901fd8c67/c.mp4", "type": "video" },
            { "url": "dp://e5b2d8f04a91/d.mp4", "type": "video" }
          ]
        }
      }
    ]
  }'

Common errors

On a stream, the item checks above run at each push, with the stream's max_responses_per_item as the budget. A push with a bad item is refused whole with 400 invalid_items, and detail.errors lists each bad item as {"item", "error"}, where item is its position in items. A stream takes no steps or dimensions at all: sending either to POST /streams is a 422.

Next

• Online RL with Streams: pairwise ranking for GRPO groups
• Ranking: one rater orders the whole set
• API Reference: Jobs