Skills and qualification

A skill is a set of questions with known answers. When a job or stream requires a skill, each annotator answers a short assessment from it first, and only responses from annotators whose score is above your threshold count. You choose the skills and thresholds; Datapoint serves and grades the assessments and leaves out responses from annotators who don't qualify.

You can require a shared skill listed by GET /skills, or create your own from examples you have labelled. Skills you create are private to your account and need no approval from our team: once a set of examples passes the automated checks, you can require the skill straight away.

This page covers the workflow and behavior. The Skills API has every request and response field.

1. Create a skill

curl -X POST https://api.trydatapoint.com/data-labelling/v1/skills \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Artifact spotting",
    "description": "Finds the generated image with a visible rendering artifact."
  }'

The response's skill field is the identifier you use everywhere else. Use it as returned rather than building it from the name; it doesn't change if you rename the skill. The skill starts as draft and can't be required until it has a set of examples.

2. Add labelled examples

Upload the images, video, or audio your examples use, then send 20–500 labelled examples to POST /skills/{skill}/calibration-sets. Each set uses one task type:

• comparison: two uploaded candidates per example, with the correct one as the answer.
• multiple_choice: a question in context, its options, optional uploaded media subjects, and the correct option or options.

Every example also carries provenance: where it came from and the licence you use it under. See Add labelled examples for the request and answer formats.

The set is checked when you send it. If it passes, its questions go live in the same call, and a draft skill becomes active. A later set adds more questions; it doesn't reactivate a skill you paused. The checks refuse the whole set with a 400 when:

• an example's media isn't one of your uploads that passed content review (public URLs aren't accepted);
• two examples have the same media and text;
• an answer isn't one of its example's candidates or options.

Good examples:

• Test the criterion your real task needs, with clear instructions and one defensible answer.
• Need attention, but not trivia, ambiguous wording, or unstated assumptions.
• Cover the range of cases your real task will show, with answers reviewed by someone other than the person who labelled them.
• Use material you own or are licensed to use, with accurate provenance.
• For multiple choice, vary which option is correct, so an annotator can't qualify by always picking the same one.
• Come in quantity. Each annotator sees each question at most once, so a larger set supports more re-checks. An annotator who still needs an assessment but has seen every question isn't offered the job.

A set is served only in the environments its serving_environment allows. It defaults to the environment of the API you call, which is prod on the production API. A prod job can use prod and all sets, a sandbox job can use sandbox and all sets, and an all job can use only all sets. To try a skill in sandbox before going live, publish its set with all.

Creating a skill or a set doesn't start anything on its own. Assessments are served only to annotators working on a job or stream that requires the skill.

3. Require skills on a job or stream

Add these fields to POST /jobs, for standalone and chain jobs alike, or to POST /streams:

Scores run from 0 to 1. A threshold is at least 0 and below 1, and a score must be strictly above it: with reject_at_or_below: 0.7, a score of 0.7 fails and 0.8 passes. A threshold doesn't guarantee the accuracy of your final dataset.

{
  "name": "realism-eval-run-3",
  "task_type": "comparison",
  "instruction": "Which image looks more realistic?",
  "max_responses_per_datapoint": 10,
  "skill_gate_mode": "adaptive",
  "skill_requirements": [
    { "skill": "REPLACE_WITH_RETURNED_SKILL", "reject_at_or_below": 0.7 }
  ],
  "skill_budget_credits": 2000,
  "datapoints": [
    {
      "media": {
        "candidates": [
          { "url": "dp://b6a1cd3f1234/model_a_017.png", "type": "image" },
          { "url": "dp://c8f3ae9b5678/model_b_017.png", "type": "image" }
        ]
      }
    }
  ]
}

The budget above is illustrative; size yours from your assessment rate. A skill must be active, available to your account, and have a set for the job's serving_environment; otherwise the request is a 400 naming it. For a sandbox job, omit skill_budget_credits: sandbox assessments are free.

Skill settings are fixed when the job is created. Re-sending a job's name returns the original job with its original skill_requirements, without an error, so check the returned settings. (A stream returns 409 config_mismatch instead.)

Choose a mode

Both modes apply every threshold, and an annotator with no score never passes.

How scores work

An annotator's score on a skill reflects their recent graded answers on it, and every job that requires the skill shares it; for a shared skill, that includes other clients' jobs. If a re-check drops an annotator below your threshold, their responses stop counting until they qualify again.

What annotators see

Assessment questions come before real work. When a job requires several skills, the assessments can be spread over more than one session before real work appears. In a chain, they come before the chain and never between its steps.

Only responses from annotators who meet every requirement count. Responses that don't count are left out of your results and raw responses, don't fill your response target, and aren't charged as responses. Answered assessments of skills you created are still charged.

Requirements also affect how fast you collect:

• More requirements and higher thresholds shrink the eligible audience and slow collection.
• A longer skill-gated chain reaches fewer annotators. Keep skill-gated chains as short as you can.
• A datapoint keeps being served until enough responses count, up to a serving limit. If many annotators don't qualify, it can close with fewer counted responses than max_responses_per_datapoint.
• When a job's assessment budget runs out, annotators who still need an assessment aren't offered the job until you raise the budget.

4. Monitor quality and spend

Skill performance reports show accuracy and completion per set or per question across your jobs. For enforced jobs they also show how often annotators met the job's threshold. Accuracy and that pass rate measure different things, and adaptive jobs have no pass rate. GET /jobs/{job_id}/skill-quality shows a job's thresholds and its assessment counts.

Check sample sizes before drawing conclusions. Low accuracy on a question can mean unclear instructions, a wrong answer key, real difficulty, or an audience mismatch; it doesn't by itself prove bad-faith answers. High accuracy can mean the question is clear, but doesn't show that it separates the annotators you want from the rest.

On a job with skill requirements, the skill scores decide which responses count. They also set each response's trust.estimate in raw responses and its weight in the weighted_* result fields: both use the annotator's lowest score across the job's required skills.

Shared skills and sandbox jobs have no assessment charge. Assessments of skills you created are charged at your account's assessment rate, fixed when the job is created. GET /jobs/{job_id} reports skill_cost_credits separately from response_cost_credits; see Billing for account-level totals.

5. Maintain your skills

PATCH /skills/{skill} renames a skill, edits its description, or sets its status to paused, retired, or active. Setting active needs at least one accepted set, and a retired skill takes no new sets.

Pausing or retiring a skill stops every job and stream that requires it from serving new work, including to annotators who already qualified, until you set it back to active. Those jobs stay active and report no error.

A job's requirements, thresholds, and mode can't be changed. To change them, create a new job with a new name. A job's assessment budget can be raised while it runs.

Next

• Skills API: request fields, answer formats, and reports
• Jobs API: create jobs and track progress
• Billing: assessment rates, budgets, and spend