# Skills and qualification

> Require annotators to pass an assessment built from your own labelled examples before their responses count.

A skill is a set of questions with known answers. When a job or [stream](/docs/streams) requires a skill, each annotator answers a short assessment from it first, and only responses from annotators whose score is above your threshold count. You choose the skills and thresholds; Datapoint serves and grades the assessments and leaves out responses from annotators who don't qualify.

You can require a shared skill listed by `GET /skills`, or create your own from examples you have labelled. Skills you create are private to your account and need no approval from our team: once a set of examples passes the automated checks, you can require the skill straight away.

This page covers the workflow and behavior. The [Skills API](/docs/api/skills) has every request and response field.

## 1. Create a skill

```bash
curl -X POST https://api.trydatapoint.com/data-labelling/v1/skills \
  -H "X-API-Key: $DATAPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Artifact spotting",
    "description": "Finds the generated image with a visible rendering artifact."
  }'
```

The response's `skill` field is the identifier you use everywhere else. Use it as returned rather than building it from the name; it doesn't change if you rename the skill. The skill starts as `draft` and can't be required until it has a set of examples.

## 2. Add labelled examples

[Upload](/docs/api/media) the images, video, or audio your examples use, then send **20–500 labelled examples** to `POST /skills/{skill}/calibration-sets`. Each set uses one task type:

- **`comparison`**: two uploaded candidates per example, with the correct one as the answer.
- **`multiple_choice`**: a question in `context`, its options, optional uploaded media subjects, and the correct option or options.

Every example also carries `provenance`: where it came from and the licence you use it under. See [Add labelled examples](/docs/api/skills#add-labelled-examples) for the request and answer formats.

The set is checked when you send it. If it passes, its questions go live in the same call, and a `draft` skill becomes `active`. A later set adds more questions; it doesn't reactivate a skill you paused. The checks refuse the whole set with a `400` when:

- an example's media isn't one of your uploads that passed content review (public URLs aren't accepted);
- two examples have the same media and text;
- an answer isn't one of its example's candidates or options.

Good examples:

- Test the criterion your real task needs, with clear instructions and one defensible answer.
- Need attention, but not trivia, ambiguous wording, or unstated assumptions.
- Cover the range of cases your real task will show, with answers reviewed by someone other than the person who labelled them.
- Use material you own or are licensed to use, with accurate `provenance`.
- For multiple choice, vary which option is correct, so an annotator can't qualify by always picking the same one.
- Come in quantity. Each annotator sees each question at most once, so a larger set supports more re-checks. An annotator who still needs an assessment but has seen every question isn't offered the job.

<DocCallout type="note" title="Passing the checks isn't a quality guarantee">
The automated checks confirm that the set can be served. They don't confirm that your answers are right or that the questions are as hard as you need. Review the questions yourself and watch how they perform once jobs use them.
</DocCallout>

A set is served only in the environments its `serving_environment` allows. It defaults to the environment of the API you call, which is `prod` on the production API. A `prod` job can use `prod` and `all` sets, a `sandbox` job can use `sandbox` and `all` sets, and an `all` job can use only `all` sets. To try a skill in sandbox before going live, publish its set with `all`.

Creating a skill or a set doesn't start anything on its own. Assessments are served only to annotators working on a job or stream that requires the skill.

## 3. Require skills on a job or stream

Add these fields to [`POST /jobs`](/docs/api/jobs#create-a-job), for standalone and [chain](/docs/chains) jobs alike, or to [`POST /streams`](/docs/api/streams#skill-requirements):

| Field | Meaning |
|---|---|
| `skill_requirements` | 1–10 skills, each `{ "skill": ..., "reject_at_or_below": ... }`. Each skill may appear once, and an annotator must pass all of them. |
| `skill_gate_mode` | `adaptive` (the default) or `enforced`. See [Choose a mode](#choose-a-mode). |
| `skill_budget_credits` | Jobs only. The most the job may spend on assessments of skills you created, reserved on top of its responses. Required for those skills outside sandbox; see [Billing](/docs/billing#set-a-budget). Streams pay for assessments as they go instead. |

Scores run from 0 to 1. A threshold is at least `0` and below `1`, and a score must be **strictly above** it: with `reject_at_or_below: 0.7`, a score of `0.7` fails and `0.8` passes. A threshold doesn't guarantee the accuracy of your final dataset.

```json
{
  "name": "realism-eval-run-3",
  "task_type": "comparison",
  "instruction": "Which image looks more realistic?",
  "max_responses_per_datapoint": 10,
  "skill_gate_mode": "adaptive",
  "skill_requirements": [
    { "skill": "REPLACE_WITH_RETURNED_SKILL", "reject_at_or_below": 0.7 }
  ],
  "skill_budget_credits": 2000,
  "datapoints": [
    {
      "media": {
        "candidates": [
          { "url": "dp://b6a1cd3f1234/model_a_017.png", "type": "image" },
          { "url": "dp://c8f3ae9b5678/model_b_017.png", "type": "image" }
        ]
      }
    }
  ]
}
```

The budget above is illustrative; size yours from your [assessment rate](/docs/billing#skill-assessment-pricing). A skill must be active, available to your account, and have a set for the job's `serving_environment`; otherwise the request is a `400` naming it. For a sandbox job, omit `skill_budget_credits`: sandbox assessments are free.

Skill settings are fixed when the job is created. Re-sending a job's `name` returns the original job with its original `skill_requirements`, without an error, so check the returned settings. (A [stream](/docs/api/streams) returns `409 config_mismatch` instead.)

### Choose a mode

| Mode | Use it when | What happens |
|---|---|---|
| `adaptive` | You're happy to reuse qualification earned on other jobs that require the same skill. | Annotators without a passing score are assessed before real work. Annotators who already pass start real work, but are re-checked from time to time. An annotator below the threshold can be assessed again later. |
| `enforced` | You want every annotator assessed for this job. | Each annotator is assessed on every required skill for this job, even with an existing score, and must score above the threshold. An annotator who doesn't qualify isn't assessed again for this job. |

Both modes apply every threshold, and an annotator with no score never passes.

### How scores work

An annotator's score on a skill reflects their recent graded answers on it, and every job that requires the skill shares it; for a shared skill, that includes other clients' jobs. If a re-check drops an annotator below your threshold, their responses stop counting until they qualify again.

### What annotators see

Assessment questions come before real work. When a job requires several skills, the assessments can be spread over more than one session before real work appears. In a chain, they come before the chain and never between its steps.

Only responses from annotators who meet every requirement count. Responses that don't count are left out of your results and raw responses, don't fill your response target, and aren't charged as responses. Answered assessments of skills you created are still charged.

Requirements also affect how fast you collect:

- More requirements and higher thresholds shrink the eligible audience and slow collection.
- A longer skill-gated chain reaches fewer annotators. Keep skill-gated chains as short as you can.
- A datapoint keeps being served until enough responses count, up to a serving limit. If many annotators don't qualify, it can close with fewer counted responses than `max_responses_per_datapoint`.
- When a job's assessment budget runs out, annotators who still need an assessment aren't offered the job until you [raise the budget](/docs/billing#increase-a-jobs-skill-budget).

## 4. Monitor quality and spend

[Skill performance reports](/docs/api/skills#check-how-a-skill-is-doing) show accuracy and completion per set or per question across your jobs. For enforced jobs they also show how often annotators met the job's threshold. Accuracy and that pass rate measure different things, and adaptive jobs have no pass rate. `GET /jobs/{job_id}/skill-quality` shows a job's thresholds and its assessment counts.

Check sample sizes before drawing conclusions. Low accuracy on a question can mean unclear instructions, a wrong answer key, real difficulty, or an audience mismatch; it doesn't by itself prove bad-faith answers. High accuracy can mean the question is clear, but doesn't show that it separates the annotators you want from the rest.

On a job with skill requirements, the skill scores decide which responses count. They also set each response's `trust.estimate` in [raw responses](/docs/api/jobs#get-raw-responses) and its weight in the `weighted_*` [result fields](/docs/task-types): both use the annotator's lowest score across the job's required skills.

Shared skills and sandbox jobs have no assessment charge. Assessments of skills you created are charged at your account's assessment rate, fixed when the job is created. `GET /jobs/{job_id}` reports `skill_cost_credits` separately from `response_cost_credits`; see [Billing](/docs/billing#separate-response-and-skill-spend) for account-level totals.

## 5. Maintain your skills

`PATCH /skills/{skill}` renames a skill, edits its description, or sets its `status` to `paused`, `retired`, or `active`. Setting `active` needs at least one accepted set, and a retired skill takes no new sets.

Pausing or retiring a skill stops every job and stream that requires it from serving new work, including to annotators who already qualified, until you set it back to `active`. Those jobs stay active and report no error.

A job's requirements, thresholds, and mode can't be changed. To change them, create a new job with a new `name`. A job's assessment budget can be [raised](/docs/billing#increase-a-jobs-skill-budget) while it runs.

## Next

- [Skills API](/docs/api/skills): request fields, answer formats, and reports
- [Jobs API](/docs/api/jobs): create jobs and track progress
- [Billing](/docs/billing#skill-assessment-pricing): assessment rates, budgets, and spend