Annotator evaluation
Beyond aggregate agreement metrics on the project as a whole, AfriAnnotate can score individual annotators against the project's ground-truth tasks and act on the score. Three related features:
- Calibration ramp — new annotators see ground-truth tasks first until they've done N of them
- Minimum-score pause — annotators whose ground-truth agreement drops below a floor are auto-paused on the project
- Low-agreement auto-routing — tasks where annotators disagree get another annotator or a review-queue item
Configure under Project → Settings → Quality — see Quality settings for the card-by-card tour of the tab.
Everything here is measured against ground truth (annotations marked as ground truth on a task — see Review annotations). Reviewer accept/reject verdicts don't feed these scores.
Calibration ramp
While an annotator has fewer than N ground-truth tasks on the project, their labelling queue is restricted to tasks that carry a ground truth (as long as any remain unsolved for them). Graduation is purely by count: after N GT tasks the restriction lifts, whatever the annotator scored. There is no calibration pass mark — the minimum-score pause below is what acts on the score.
| Setting | Default |
|---|---|
Calibration GT tasks (calibration_gt_tasks) | 0 (disabled) |
Worked example with Calibration GT tasks = 20:
- New annotator joins the project
- Their stream serves only ground-truth-bearing tasks; the queue label reads Calibration (GT first, 3/20) and so on
- After 20 GT tasks the stream opens up to every task
- If the project also has evaluation on, those 20 tasks are what the first evaluation score is computed from
Useful when:
- Onboarding new annotators on a new project / domain
- Quality matters more than time-to-first-label
- You have enough ground-truth tasks to draw from — the ramp does nothing once the GT pool is exhausted for that annotator
Minimum-score pause
With annotator evaluation enabled, every submission re-scores the annotator: for each task that has a ground truth and an annotation by them, their latest answer is compared with the ground truth using the project's agreement metric (auto-picked per control tag when none is set; a Custom Python metric is skipped in favour of auto-pick). The score is the mean over all such tasks — not a rolling window.
| Setting | Default |
|---|---|
Evaluate annotators against ground truth (annotator_evaluation_enabled) | Off |
Minimum agreement score (evaluation_min_score) | Blank (no pause; scores are still recorded) |
Ground-truth tasks required (evaluation_min_ground_truth) | 0 (evaluate from the first GT task) |
The pause trips when all three hold: a minimum score is set, the annotator has completed at least the required number of GT tasks, and their mean score is below the minimum.
Worked example with Minimum score = 0.7 + GT tasks required = 20:
- Annotator has 19 GT tasks — score computed and stored, no action
- Hits 20 — mean GT agreement is 0.85 → they keep working
- Later the mean falls to 0.65 → auto-paused on this project:
- their project membership is disabled; opening a task returns "Your access to this project has been paused."
- they get a notification titled "You were paused on a project" with their score, the floor, and the GT count, pointing them to the project owner
- org owners/admins and the project creator are notified once
- every GT task they scored below 1.0 on is raised as an
evaluation_failitem in the review queue so a manager can see the specific disagreements
- A manager reviews and either Re-enables them from Project → Members (the evaluation flag is cleared so they aren't re-paused on the next submission unless the score is still under the floor), removes them from the project, or leaves it to the org-wide auto-suspend rules
Every evaluation also appends a history snapshot; annotators see it under Profile → Analytics → Evaluation history, and the org Quality page rolls up who is currently paused.
Useful when:
- Live monitoring of crowd quality
- Crowd projects where you can't afford low-quality contributors filling the pool
Distinct from auto-suspend rules — this is a per-project pause; auto-suspend is org-wide.
Low-agreement auto-routing
When a task's freshly computed agreement (across annotators) falls below a threshold, one of three strategies fires:
| Strategy | Behaviour |
|---|---|
| None (default) | No automatic action; the task just carries a low score |
| Assign additional annotator | The task's overlap grows by 1 (capped at annotators-per-task + 2) and the task is marked not-yet-labelled, so the next annotator picks it up from the normal queue |
| Move to review queue | Raises a low_agreement item in the review queue: the task is exempt from review sampling and comes first in the review stream when QC priority is on. The task keeps its labelled status — it is not reopened to annotators |
| Setting | Default |
|---|---|
Low-agreement threshold (low_agreement_threshold) | Blank (disabled) |
Strategy (low_agreement_strategy) | None |
There is no combined "review + add annotator" option; pick one.
Worked example with threshold 0.5 + Move to review queue:
- Three annotators label a task — agreement score 0.45
- The system raises a low-agreement item (one open item per task and reason; repeated triggers refresh its score rather than duplicating)
- A reviewer works it in the stream; under the default
single_truthconflict policy the annotation they accept becomes the task's ground truth
Separately, when three or more annotators work a task, an annotator
whose mean agreement with their peers is below 0.5 and at least 0.25
under the group mean is flagged with an outlier item.
Embedding API for semantic similarity
For projects whose agreement metric is Semantic Similarity, AfriAnnotate embeds each text through an external endpoint and scores the pair by cosine similarity (rescaled to 0–1):
| Setting | What |
|---|---|
Embedding API URL (embedding_api_url) | Your hosted endpoint; public HTTP(S) only — loopback and private addresses are refused |
Auth token (embedding_api_token) | Sent as Authorization: Bearer … if set |
Timeout (ms) (embedding_api_timeout_ms) | Default 3000 (clamped to 250 ms – 30 s) |
The endpoint contract: one text per request — POST {url} with body
{"input": ["..."]} — and a response in any of these shapes:
{"embeddings": [[...]]}, OpenAI-style
{"data": [{"embedding": [...]}]}, or {"embedding": [...]}. Results
are memoised per (URL, token, timeout, text), so repeated texts don't
re-hit the endpoint. If the URL is blank or the call fails, the metric
falls back to character edit-distance rather than skipping the pair.
The Quality API's embedding-health call checks the endpoint
end-to-end.
Useful for free-text annotations (summaries, descriptions, translations) where exact-match metrics aren't meaningful.
Where the per-annotator numbers show up
- Project → Members — GT-task count per member; a paused badge for evaluation pauses; Pause / Resume / Re-enable.
- Project Analytics → Metrics — per-annotator annotations, average lead time, average agreement for the selected range.
- Profile → Analytics — evaluation history (score per evaluation), recent submissions, and a paused / good-standing pill in the title row; managers can open another user's analytics.
- Organization → Quality — currently paused annotators across projects.
Custom agreement metric (Python sandbox)
A custom agreement metric changes how pairs of annotators are scored. It is not used for ground-truth evaluation — that path auto-picks a built-in metric per control tag.
When NOT to use these features
Annotator evaluation has friction. Skip it when:
- Your team is small (< 10 annotators) — manager-eyeball is faster
- Your task type doesn't have a clean "right answer" — subjective labelling won't have agreement that maps to quality
- You're early in a project — ramps and pauses need ground-truth tasks, which you don't have yet
To observe first, enable evaluation and leave the minimum score blank: scores and history are recorded without pausing anyone. Set the floor once you've seen what the distribution looks like.
Counting reviewer verdicts
By default evaluation is ground-truth only: reviewer accept/reject does not move an annotator's evaluation score (rejection rate feeds the organisation's auto-suspend policy instead). Turn on Count reviewer verdicts too (Settings → Quality → Annotator evaluation) to add review outcomes to the sample: an accepted or fixed annotation counts as 1.0, a rejected one as 0.0; an annotation with an open dispute is skipped until the re-review lands. The minimum-sample floor then counts ground-truth tasks and reviewed annotations together.
What's next
- Agreement metrics → — built-in + custom metrics that power the scoring
- Auto-suspend rules → — org-wide escalation for persistent quality issues
- Review workflow → — how reviewers see and act on routed tasks