Rating

Rating tasks ask annotators to score one media or text subject on a predefined scale, like the classic Likert-style 1-to-5 or 1-to-7 question. You can also show one non-selectable media reference when the score depends on fidelity to an original.

When to use it

• Scoring a model output on perceived quality
• Checking whether an edit preserves the identity or features of a reference image
• Likert-scale surveys (agreement, satisfaction, confidence)
• Jobs where you want the full distribution of scores, not just a winner

Datapoint shape

{
  "media": {
    "reference": [{ "url": "dp://e7d921ab3c45/shoe1.png", "type": "image" }],
    "subject": [{ "url": "dp://e7d921ab3c45/shoe2.png", "type": "image" }]
  }
}

Each datapoint can have at most one media subject and one optional reference. The reference is context only: annotators always score the subject, and the submitted answer remains a single numeric rating. The two media roles may use different supported media types. When there is no media subject, a reference is valid if the datapoint has non-empty context containing the text to rate.

Omit subject for a text rating, retaining reference only when the text is being judged against that media.

Text rating

For rating an abstract question with no specific media subject, send an empty media object — "media": {} — with no subject array. The annotator sees the prompt and scale only.

{
  "task_type": "rating",
  "instruction": "How likely are you to recommend this product to a friend?",
  "response_options": { "scale": [1, 2, 3, 4, 5] },
  "datapoints": [{ "media": {} }]
}

When you have several text passages to rate against the same question, put each passage in the datapoint's context, reference it with {context} in the job-level instruction, and submit one datapoint per passage:

{
  "instruction": "Rate how much you agree: {context}",
  "datapoints": [
    { "media": {}, "context": "Climate change is human-caused." },
    { "media": {}, "context": "Lemurs are reptiles." }
  ]
}

The {context} placeholder is substituted per datapoint at delivery time. Without it, context is stored on the datapoint but never shown to annotators; they would only see the bare instruction.

To rate a description against an image, keep the text in context and send the image under media.reference. The annotator sees the rendered question, the non-selectable reference image, and the rating scale; no media subject card is created.

{
  "task_type": "rating",
  "instruction": "How accurately does this description match the reference image?\n\nDescription: {context}",
  "response_options": {
    "scale": [1, 2, 3, 4, 5],
    "labels": { "1": "Not accurate", "5": "Fully accurate" }
  },
  "datapoints": [
    {
      "media": {
        "reference": [
          { "url": "dp://e7d921ab3c45/shoe1.png", "type": "image" }
        ]
      },
      "context": "A red shoe with white laces."
    }
  ]
}

Scale configuration

Rating jobs require a response_options object on the job body:

{
  "task_type": "rating",
  "response_options": {
    "scale": [1, 2, 3, 4, 5],
    "labels": {
      "1": "Very poor",
      "5": "Excellent"
    }
  },
  ...
}

Result shape

{
  "datapoint_index": 0,
  "mean": 4.0,
  "median": 4,
  "distribution": { "3": 2, "4": 6, "5": 2 },
  "total_responses": 10,
  "weighted_mean": 4.059,
  "weighted_distribution": { "3": 1.4, "4": 5.2, "5": 1.9 },
  "media": [
    {
      "media_id": "e7d921ab-3c45-4a2f-9b71-0d1e2f3a4b5c",
      "type": "video",
      "role": "subject",
      "label": null,
      "url": "/media/v2/e7d921ab-3c45-4a2f-9b71-0d1e2f3a4b5c?cid=...&exp=...&sig=...",
      "thumbnail_url": "/media/v2/e7d921ab-3c45-4a2f-9b71-0d1e2f3a4b5c?cid=...&exp=...&sig=...&preview=1"
    }
  ]
}

Each entry in the media array describes one media item on the datapoint:

Example

curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "edit-identity-preservation-v1",
    "instruction": "Do people, animals, or objects maintain their original identity and features after the edit?",
    "task_type": "rating",
    "response_options": {
      "scale": [1, 2, 3, 4, 5],
      "labels": { "1": "Not preserved", "3": "Partially preserved", "5": "Fully preserved" }
    },
    "max_responses_per_datapoint": 10,
    "datapoints": [
      {
        "media": {
          "reference": [
            { "url": "dp://e7d921ab3c45/shoe1.png", "type": "image" }
          ],
          "subject": [
            { "url": "dp://e7d921ab3c45/shoe2.png", "type": "image" }
          ]
        }
      }
    ]
  }'

Common errors

Next

• Comparison: head-to-head preference
• Multiple choice: categorical labels