Ranking

Order multiple media items from best to worst.

Ranking tasks ask annotators to order multiple items from most to least preferred. Use it when the full ordering matters, not just a winner.

When to use it

Comparing 3 or more model variants simultaneously
Ordering a slate of recommendations
Preference elicitation where the full ranking carries information (e.g. for learning-to-rank)

If you only have two items, use comparison instead — it's a simpler judgement for the annotator and returns a direct winner.

Datapoint shape

{
  "media": {
    "candidates": [
      { "url": "dp://b6a1cd3f1234/a.mp4", "type": "video" },
      { "url": "dp://c8f3ae9b5678/b.mp4", "type": "video" },
      { "url": "dp://a3e901fd8c67/c.mp4", "type": "video" }
    ]
  }
}

media.candidates

Field	Type	Description
`url`*	`string`	A dp:// reference returned by POST /media, or a public https URL.
`type`	`"video" \| "image" \| "audio"`	Determines which player the annotator sees. Inferred from the uploaded media, or from the URL's file extension — only required when a public URL has no extension to infer from. All candidates must share one type.

The list must contain at least 2 candidates. Practical upper bound is 5–7; more items make the task cognitively harder and slow collection.

Text-only ranking

For ranking free-form text items (factors, claims, options) without media, use {id, text} candidates instead of {url, type}:

{
  "media": {
    "candidates": [
      { "id": "factor_price", "text": "Price" },
      { "id": "factor_brand", "text": "Brand reputation" },
      { "id": "factor_reviews", "text": "Customer reviews" }
    ]
  }
}

text candidates

Field	Type	Description
`id`*	`string`	Stable identifier you choose; appears verbatim as the result key for this item. Must be a non-empty string, unique within the datapoint, with no commas and no leading or trailing whitespace.
`text`*	`string`	Non-empty display text shown to the annotator.

A datapoint's candidates must be uniformly text or uniformly media. Mixing the two within a single datapoint is rejected. For text candidates, the result envelope's average_ranks and ranking_order use your own id values directly.

Result shape

{
  "datapoint_index": 0,
  "average_ranks": {
    "video_1": 1.8,
    "video_2": 2.1,
    "video_3": 2.1
  },
  "ranking_order": ["video_1", "video_2", "video_3"],
  "total_responses": 10,
  "weighted_average_ranks": {
    "video_1": 1.6,
    "video_2": 2.2,
    "video_3": 2.2
  },
  "weighted_ranking_order": ["video_1", "video_2", "video_3"],
  "media": [
    {
      "media_id": "b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d",
      "type": "video",
      "role": "candidates",
      "label": null,
      "url": "/media/v2/b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d?sig=...",
      "thumbnail_url": "/media/v2/b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d?preview=1&sig=..."
    }
  ]
}

Field	Type	Description
`average_ranks`	`{ [item_id: string]: float }`	Mean rank across responses per item (lower is better; rank 1 is best), rounded to 3 decimals. For media candidates the item_ids are type_index (e.g. video_1, video_2); for text candidates they are your own supplied ids.
`ranking_order`	`array of item_ids`	Overall order sorted by ascending average_rank. Ties break by the item's first appearance across responses.
`total_responses`	`int`	Valid ranking responses counted.
`weighted_average_ranks`	`{ [item_id: string]: float } \| null`	Mean rank per item where each response is weighted by a platform-computed reliability score instead of counted equally (lower is better), rounded to 3 decimals. Empty or null when there are no eligible responses.
`weighted_ranking_order`	`array of item_ids \| null`	Overall order sorted by ascending weighted_average_rank.
`media`	`array`	The datapoint's media items, so you can tie each result key back to the asset it refers to. Empty for text-only ranking. See the fields below.

For media candidates, each entry in the media array describes one item on the datapoint:

media[]

Field	Type	Description
`media_id`	`string`	Stable identifier for the media item.
`type`	`"video" \| "image" \| "audio"`	The media item's type.
`role`	`"candidates"`	Where the item sits on the datapoint. Ranking candidates are always `candidates`.
`label`	`string \| null`	Not populated for ranking — always `null`. Map result keys to assets by array position (see the note below), not by this field.
`url`	`string`	Signed, relative proxy URL for the media; valid for roughly one hour.
`thumbnail_url`	`string`	Signed, relative preview URL; valid for roughly one hour.

Example

curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "slate-ranking-v1",
    "instruction": "Order these from most to least realistic.",
    "task_type": "ranking",
    "max_responses_per_datapoint": 10,
    "datapoints": [
      {
        "media": {
          "candidates": [
            { "url": "dp://b6a1cd3f1234/a.mp4", "type": "video" },
            { "url": "dp://c8f3ae9b5678/b.mp4", "type": "video" },
            { "url": "dp://a3e901fd8c67/c.mp4", "type": "video" }
          ]
        }
      }
    ]
  }'

Common errors

Status	Cause
`400`	`media.candidates` has fewer than 2 entries.
`400`	Candidates mix text and media items within one datapoint.
`400`	Media candidates are not all the same media type.
`400`	A text candidate has a missing/blank `id` or `text`, or two candidates share the same `id`.
`400`	A text candidate's `id` contains a comma or has leading or trailing whitespace.
`422`	`media` is not an object (ranking requires object-keyed media with a `candidates` list), or a required top-level field is missing or malformed.

Comparison: when you only have two items
API Reference: Jobs: full endpoint reference

← PreviousRating Next →Multiple choice