Datapoint

Comparison

Pairwise A/B comparison of two media items.

Comparison is the default and most-used task type. Each datapoint has exactly two candidate media items and annotators pick the one that answers your question better.

When to use it

  • Preference comparisons between two model outputs ("which generation looks better?")
  • A/B tests on ad creatives, logos, or design variants
  • Pairwise preference data for RLHF-style training

If you have 3+ items to compare, use ranking instead. Splitting into pairwise comparisons would be noisier and more expensive.

Datapoint shape

{
  "media": {
    "candidates": [
      { "url": "dp://b6a1cd3f1234/video_a.mp4", "type": "video" },
      { "url": "dp://c8f3ae9b5678/video_b.mp4", "type": "video" }
    ]
  }
}

media.candidates

FieldTypeDescription
url*stringA dp:// reference returned by POST /media, or a public https URL.
type"video" | "image" | "audio"Determines which player the annotator sees. Inferred from the uploaded media, or from the URL's file extension — only required when a public URL has no extension to infer from. All candidates must share one type.

The list must contain exactly two items; 1 or 3+ rejects with 400 Bad Request.

Optional reference media

Add media.reference to show a third item alongside the candidates for context (e.g. an input image both generations came from):

{
  "media": {
    "reference": [{ "url": "dp://5ce7b1a94f28/prompt.png", "type": "image" }],
    "candidates": [
      { "url": "dp://b6a1cd3f1234/gen_a.mp4", "type": "video" },
      { "url": "dp://c8f3ae9b5678/gen_b.mp4", "type": "video" }
    ]
  }
}

Reference items are shown but not voted on.

Result shape

{
  "datapoint_index": 0,
  "votes": { "A": 7, "B": 3 },
  "total_responses": 10,
  "consensus": "A",
  "confidence": 0.7,
  "agreement_rate": 0.7,
  "weighted_votes": { "A": 6.1, "B": 2.4 },
  "weighted_consensus": "A",
  "weighted_confidence": 0.718,
  "media": [
    {
      "media_id": "b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d",
      "type": "video",
      "role": "candidates",
      "label": null,
      "url": "/media/v2/b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d?sig=...",
      "thumbnail_url": "/media/v2/b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d?preview=1&sig=..."
    },
    {
      "media_id": "c8f3ae9b-5678-4d3e-8f9a-1b2c3d4e5f60",
      "type": "video",
      "role": "candidates",
      "label": null,
      "url": "/media/v2/c8f3ae9b-5678-4d3e-8f9a-1b2c3d4e5f60?sig=...",
      "thumbnail_url": "/media/v2/c8f3ae9b-5678-4d3e-8f9a-1b2c3d4e5f60?preview=1&sig=..."
    }
  ]
}
FieldTypeDescription
votes{ A: int, B: int }Vote count per candidate.
total_responsesintTotal annotator responses received.
consensus"A" | "B" | "tie" | nullThe winning option, or `tie` on exact match, or null if no responses yet.
confidencefloatWinning votes / total, rounded to 3 decimals (0.5 on a tie).
agreement_ratefloatSame as confidence for comparison tasks.
weighted_votes{ A: float, B: float }Vote totals where each response is weighted by a platform-computed reliability score instead of counted as 1. Rounded to 3 decimals.
weighted_consensus"A" | "B" | "tie" | nullWinner by `weighted_votes`, or null when there are no eligible responses.
weighted_confidencefloat | nullWinning weighted votes / total weighted votes, rounded to 3 decimals, or null when there are no eligible responses.
mediaarrayThe datapoint's media items, so you can tie each vote key back to the asset it refers to. See the fields below.

Each entry in the media array describes one media item on the datapoint:

media[]

FieldTypeDescription
media_idstringStable identifier for the media item.
type"video" | "image" | "audio"The media item's type.
role"candidates" | "reference"Where the item sits on the datapoint — `candidates` are the two voted-on items; `reference` is context-only.
labelstring | nullA per-item vote key, present only for task types that store one on the media. Comparison candidates don't carry one here, so this is `null` — map candidates back to `A`/`B` by submission order instead (see the note below). Always null for reference media.
urlstringSigned, relative proxy URL for the media; valid for roughly one hour.
thumbnail_urlstringSigned, relative preview URL; valid for roughly one hour.

Example request

curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "realism-bake-off-v1",
    "instruction": "Which video looks more realistic?",
    "task_type": "comparison",
    "max_responses_per_datapoint": 15,
    "datapoints": [
      {
        "media": {
          "candidates": [
            { "url": "dp://b6a1cd3f1234/gen_a.mp4", "type": "video" },
            { "url": "dp://c8f3ae9b5678/gen_b.mp4", "type": "video" }
          ]
        }
      }
    ]
  }'

Multi-dimension comparisons

Replace instruction with dimensions to ask the same pair of candidates multiple questions, for example to judge realism and coherence independently:

{
  "name": "gen-eval-v1",
  "task_type": "comparison",
  "dimensions": {
    "realism": "Which video looks more realistic?",
    "coherence": "Which video has better motion coherence?"
  },
  "dimension_weights": { "realism": 0.6, "coherence": 0.4 },
  "max_responses_per_datapoint": 10,
  "datapoints": [...]
}

dimension_weights is optional — omit it and every dimension is weighted equally (each gets 1/N). If you supply it, its keys must match your dimensions keys exactly and the values must sum to 1.0 (within a 0.01 tolerance). Results include a combined weighted_score and per-dimension breakdowns under dimensions.

Common errors

StatusCause
400Candidates list has 0, 1, or 3+ entries.
400Candidates are not all the same media type.
400instruction and dimensions both set, or neither set.
400dimension_weights keys don't match dimensions keys.
400dimension_weights don't sum to 1.0 (within 0.01 tolerance).
422max_responses_per_datapoint outside [1, 10000], or a required top-level field is missing or malformed.

Next

← PreviousBillingNext →Image-to-video comparison