Task types

Every job has a task_type. It's optional and defaults to comparison. The task type determines the payload shape each datapoint expects, what annotators see, and the structure of the aggregated result. Six types are supported today. A seventh value, chain, is never something you set directly — it's what the API reports as a chain job's task_type in its results.

Overview

Choosing between them

• Comparison vs. ranking. Use comparison when you have exactly two candidates. Switch to ranking only when there are 3+ items and the full order matters; ranking tasks are cognitively harder for annotators, which slows collection.
• Rating vs. comparison. Ratings are absolute judgments and suffer from scale drift across annotators. Comparisons are relative and more reliable when you want to know which of two things is better. Use rating when you care about calibrated quality rather than a head-to-head winner.
• Multiple choice. Use when the answer space is discrete and known in advance (e.g. "which of these tags apply?"). Use multi mode if an item can legitimately have several labels. Multiple choice isn't text-only: each datapoint may also carry a single pre-uploaded media subject (image, video, or audio) rendered above the text options, so annotators can classify what they see.
• Text answer vs. multiple choice. Use text answer when you can't (or don't want to) enumerate the answers in advance — the response is a written verbatim rather than a countable vote. Like multiple choice, each datapoint may carry a single pre-uploaded media subject to describe or react to. For a structured question that just needs an optional written aside, prefer a structured type with the free_text add-on instead.

Defining the question

A standalone job has either an instruction (one question for the whole job) or a set of dimensions (multiple questions run against the same datapoints) — exactly one of the two, never both and never neither. (This choice applies to standalone jobs only; chain jobs set neither and give each step its own instruction instead.) Dimensions turn a comparison survey into a weighted composite score:

{
  "dimensions": {
    "realism": "Which video looks more realistic?",
    "coherence": "Which video has better motion coherence?"
  },
  "dimension_weights": { "realism": 0.6, "coherence": 0.4 }
}

dimension_weights is optional. Omit it and every dimension is weighted equally (1/N). When you do supply it, its keys must match your dimensions keys exactly and its values must sum to 1.0 (within a small rounding tolerance). Every dimension runs as its own set of tasks, and once an annotator has been served a datapoint, later serves exclude its other dimensions — so covering N dimensions at M responses each needs up to N×M distinct annotators per datapoint (e.g. a 10-response cap with 2 dimensions means up to 20 responses per datapoint). Billing scales with the dimension count.

rating and multiple_choice need one more piece: a response_options object. Rating requires a scale (a list of 2 or more values — numeric is recommended, e.g. [1, 2, 3, 4, 5]), and multiple choice requires a mode ("single" or "multi"). For multiple_choice, the question annotators actually see is each datapoint's own context string; the job-level question (instruction or dimensions) is still required but isn't shown to them. The dedicated task-type pages document each response_options shape.

Media payload shapes

Each task type expects a task-specific media object per datapoint. The dedicated task-type pages document the exact shape.

Chaining steps

Need a follow-up question that depends on the previous answer? Wrap 1 to 5 task-type steps into a chain. You can gate a rating behind a watchability check, follow a tag selection with a confidence rating, or end the survey early on a specific answer. Each step keeps its own task type, instruction, and aggregation, and you get per-step results in a single response.

Result envelope

All task types share the same DatapointResult wrapper under GET /jobs/{id}/results. The backend always emits every declared field on that model. Fields that aren't relevant to your task type come back as null. The per-task-type examples below only show the populated fields to keep them readable; assume a sibling null for every other field (mean, average_ranks, dimensions, etc.). The model is also permissive, so you may occasionally see additional forward-compatible keys not listed here — treat any unknown key as optional.

Every result also includes:

• datapoint_index (int): zero-based position of the datapoint in your original datapoints[].
• context (string | null): the datapoint's context if you supplied one, otherwise null.
• media (array | null): for standalone jobs, the datapoint's media items as signed proxy URLs. Each entry names its role, so you can see which part of the task the asset played. Entries are returned in the order you submitted them, which is how you tie a result key back to an asset. (A label field is also present, but is null for media uploaded through the API.)

Weighted fields

Two different families of fields share the weighted_ prefix and mean different things.

Reliability-weighted verdict. Alongside the equal-vote results above, each aggregation also carries a reliability-weighted verdict in which each response is weighted by a platform-computed reliability score. These additive fields sit beside their equal-vote counterparts and never change them:

For chain jobs these appear on each step's aggregation; for multi-dimension jobs they appear on each dimension's aggregation.

Dimension composite. Multi-dimension jobs additionally return weighted_score and overall_consensus. These are unrelated to the reliability weighting above: they combine the per-dimension results into a single score using your dimension_weights (i.e. how much each dimension matters), and overall_consensus is the winning key of that composite.

Next

• Comparison tasks
• Image-to-video comparison tasks
• Rating tasks
• Ranking tasks
• Multiple-choice tasks
• Text-answer tasks
• Chains: compose multiple steps into a single survey