Datapoint

Image-to-video comparison

Pairwise A/B comparison of two candidate videos against a reference image.

Image-to-video comparison (i2v_comparison) is a purpose-built task type for evaluating generated video against the source image it was conditioned on. Each datapoint pairs exactly one reference image with exactly two candidate videos, and annotators pick the video that better matches the reference. The reference image is shown alongside both videos.

When to use it

  • Ranking image-to-video model outputs by fidelity to the conditioning image
  • Comparing two generations from the same prompt frame
  • Preference data for image-to-video model training and evaluation

For same-modality pairs — two images, two videos, or two audio clips — use plain comparison instead; it accepts an optional reference item too.

Datapoint shape

{
  "media": {
    "reference": [{ "url": "dp://5ce7b1a94f28/prompt.png", "type": "image" }],
    "candidates": [
      { "url": "dp://b6a1cd3f1234/gen_a.mp4", "type": "video" },
      { "url": "dp://c8f3ae9b5678/gen_b.mp4", "type": "video" }
    ]
  }
}

media

FieldTypeDescription
reference*arrayExactly one item, which must be an image. Shown for context; not voted on.
candidates*arrayExactly two items, both of which must be videos. These are the A/B options annotators choose between.

Each item is a { url, type } object: url is a dp:// reference from POST /media (or a public https:// URL). type is optional — it's inferred from the uploaded media, or from the URL's file extension. Supply it explicitly only when a public URL has no extension to infer from.

Result shape

Results use the same A/B shape as comparison:

{
  "datapoint_index": 0,
  "votes": { "A": 8, "B": 2 },
  "total_responses": 10,
  "consensus": "A",
  "confidence": 0.8,
  "agreement_rate": 0.8,
  "weighted_votes": { "A": 7.3, "B": 1.6 },
  "weighted_consensus": "A",
  "weighted_confidence": 0.82,
  "media": [
    {
      "media_id": "5ce7b1a9-4f28-4c1a-8b7e-2d3f4a5b6c7d",
      "type": "image",
      "role": "reference",
      "label": null,
      "url": "/media/v2/5ce7b1a9-4f28-4c1a-8b7e-2d3f4a5b6c7d?sig=...",
      "thumbnail_url": "/media/v2/5ce7b1a9-4f28-4c1a-8b7e-2d3f4a5b6c7d?preview=1&sig=..."
    },
    {
      "media_id": "b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d",
      "type": "video",
      "role": "candidates",
      "label": null,
      "url": "/media/v2/b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d?sig=...",
      "thumbnail_url": "/media/v2/b6a1cd3f-1234-4a2b-9c8d-0e1f2a3b4c5d?preview=1&sig=..."
    },
    {
      "media_id": "c8f3ae9b-5678-4d3e-8f9a-1b2c3d4e5f60",
      "type": "video",
      "role": "candidates",
      "label": null,
      "url": "/media/v2/c8f3ae9b-5678-4d3e-8f9a-1b2c3d4e5f60?sig=...",
      "thumbnail_url": "/media/v2/c8f3ae9b-5678-4d3e-8f9a-1b2c3d4e5f60?preview=1&sig=..."
    }
  ]
}
FieldTypeDescription
votes{ A: int, B: int }Vote count per candidate video.
total_responsesintTotal annotator responses received.
consensus"A" | "B" | "tie" | nullThe winning video, or `tie` on exact match, or null if no responses yet.
confidencefloatWinning votes / total, rounded to 3 decimals.
agreement_ratefloatSame as confidence for comparison tasks.
weighted_votes{ A: float, B: float }Vote totals where each response is weighted by a platform-computed reliability score instead of counted as 1. Rounded to 3 decimals.
weighted_consensus"A" | "B" | "tie" | nullWinner by `weighted_votes`, or null when there are no eligible responses.
weighted_confidencefloat | nullWinning weighted votes / total weighted votes, rounded to 3 decimals, or null when there are no eligible responses.
mediaarrayThe datapoint's media items — the reference image and both candidate videos — so you can tie each vote key back to the asset it refers to. See the fields below.

Each entry in the media array describes one media item on the datapoint:

media[]

FieldTypeDescription
media_idstringStable identifier for the media item.
type"video" | "image"The media item's type — `video` for candidates, `image` for the reference.
role"candidates" | "reference"Where the item sits on the datapoint — `candidates` are the two voted-on videos; `reference` is the context-only source image.
labelstring | nullA per-item vote key, present only for task types that store one on the media. i2v candidates don't carry one here, so this is `null` for every entry, reference included — map candidates back to `A`/`B` by submission order instead (see the note below).
urlstringSigned, relative proxy URL for the media; valid for roughly one hour.
thumbnail_urlstringSigned, relative preview URL; valid for roughly one hour.

Multi-dimension scoring works the same way as comparison: replace instruction with dimensions to ask several questions about the same pair, and results include a combined weighted_score and per-dimension breakdowns.

Example request

curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "i2v-fidelity-v1",
    "instruction": "Which video better matches the reference image?",
    "task_type": "i2v_comparison",
    "max_responses_per_datapoint": 15,
    "datapoints": [
      {
        "media": {
          "reference": [{ "url": "dp://5ce7b1a94f28/prompt.png", "type": "image" }],
          "candidates": [
            { "url": "dp://b6a1cd3f1234/gen_a.mp4", "type": "video" },
            { "url": "dp://c8f3ae9b5678/gen_b.mp4", "type": "video" }
          ]
        }
      }
    ]
  }'

Common errors

StatusCause
422media is not a JSON object (it must be an object with reference and candidates).
400reference does not contain exactly one item.
400candidates does not contain exactly two items.
400The reference item is not an image.
400A candidate item is not a video.
422max_responses_per_datapoint outside [1, 10000], or a required top-level field is missing or malformed.

Next

← PreviousComparisonNext →Rating