Quality settings
Project → Settings → Quality (/settings/quality).
Controls how annotations are distributed, scored, and resolved: five cards, each with its own Save button — saving one card doesn't touch the others. Like the rest of Settings, the tab is manager-only.
This page documents the tab itself. For what the numbers mean, see Agreement metrics and Annotator evaluation.
Overlap of annotations
How many different annotators must label each task.
- Annotators per task — a task is marked complete once it has this many annotations (default 1). Deliberately named "annotators", not "annotations": the similarly-named field on Annotation settings counts regions per annotation, this one counts people per task.
- Apply to this % of tasks — appears when annotators per task is > 1. Only that percentage of tasks (a sampled cohort) gets the full overlap; the rest keep a single annotation. Slider, 0–100% in steps of 5.
Raising overlap is what makes every agreement feature below (and the Insights dashboards) light up — agreement needs at least 2 annotations on the same task. Completion mechanics live in Task lifecycle.
Tasks per annotator limit
Cap how many tasks a single annotator may claim on this project. Toggle off (default) = unlimited.
Enforced when an annotator tries to open a new task: once they've annotated the configured number of distinct tasks, further tasks are refused with "You've reached your task limit". They can still revisit tasks they already annotated.
See also Automatically pause annotators.
Annotator evaluation
Continuously score every annotator against the project's ground-truth tasks and auto-pause those below the bar.
| Setting | Default | Effect |
|---|---|---|
| Evaluate all annotators against ground truth | Off | Master toggle (annotator_evaluation_enabled) |
| Minimum agreement score | 80% | Floor (0–1) the annotator's mean GT agreement must stay above |
| Ground-truth tasks required | 5 | Evaluation only fires after the annotator has completed this many GT tasks |
When an annotator's score drops below the floor (after enough GT tasks), their project membership is paused, they and the project admins are notified, and each GT task they got wrong is raised in the review queue. Re-enable them from Project → Members. Scoring uses the same per-control metric machinery as agreement (the Custom Python metric is skipped for GT scoring — auto-select is used instead).
Concepts, calibration ramp, and worked examples: Annotator evaluation.
Agreement
How the per-task agreement score is computed. Two steps, presented in the card in this order:
Step 1 — Metric
How two annotations on the same task are compared. Three modes, picked via a tab strip:
| Mode | What happens |
|---|---|
| Auto (default) | Server picks a sensible metric per control tag in your labeling config — exact/Jaccard for Choices, IoU for boxes, span overlap for NER, text similarity/WER for TextArea |
| Built-in | Lock in one specific metric from the server registry. The dropdown groups metrics "recommended for this label config" above the rest, and an instant compatibility check warns when the chosen metric can't score any control in your config (Save and Recompute are blocked until it can) |
| Custom Python | Write your own def agreement(annotation_1, annotation_2, per_label=False) -> float returning a score in [0, 1] |
Per-metric semantics (what affects each score, thresholds, examples) are in Agreement metrics.
Semantic similarity metrics add an Embedding endpoint
sub-card: an HTTP(S) URL that accepts { input: ["text"] } and
returns { embeddings: [[...]] } (or the OpenAI data[].embedding
shape), an optional bearer token, and a request timeout in ms
(default 3000). A Test endpoint button runs a one-shot health
check. Leave the URL blank and semantic similarity falls back to
edit distance. See
Embedding API.
Custom Python runs sandboxed: only pure-Python stdlib imports
(math, statistics, itertools, re, json, difflib, …),
no exec/eval/open/dunder access, CPU capped at 2 seconds per
pair; a failing pair is skipped, not fatal. The editor toolbar's
Annotation info dialog shows your config's controls, a sample
annotation JSON, and the full sandbox rules; Test on first
scorable task runs your code against a real annotation pair
before you save.
Step 2 — Methodology
How per-pair scores roll up into one number per task (and project). Radio cards, one of:
| Value | Needs | Notes |
|---|---|---|
| Pairwise averaging (default) | Any data | Averages the Step-1 metric across all annotator pairs |
| Consensus (plurality) | Any data | Fraction of annotators on the plurality answer, averaged across fields |
| Cohen's kappa | 2 raters, categorical | Chance-corrected; averaged across pairs when > 2 |
| Fleiss' kappa | > 2 raters, categorical | Standard multi-rater kappa |
| Randolph's free-marginal kappa | ≥ 2 raters, categorical | Chance floor fixed at 1/q; robust to skewed marginals |
| Krippendorff's alpha | ≥ 2 raters, categorical | Handles missing answers natively |
| Scott's pi | 2 raters, categorical | Pooled marginals |
| Gwet's AC1 | ≥ 2 raters, categorical | Avoids the kappa paradox on imbalanced labels |
| Gwet's AC2 (ordinal) | ≥ 2 raters, ordinal numeric | Distance-weighted AC1 for Rating / Number |
| Weighted kappa (linear) | 2 raters, ordinal numeric | Disagreement penalised by distance |
| Weighted kappa (quadratic) | 2 raters, ordinal numeric | Squared-distance penalties (MOS-style ratings) |
The chance-corrected coefficients (everything from Cohen's kappa down) compare labels directly and ignore the Step-1 metric — the card shows a notice when your selection makes the metric moot. Only Pairwise (and loosely Consensus) consume the metric.
Recompute all tasks
The Recompute all tasks button (in the metric card's tab row) re-scores every task with the current settings and reports a breakdown: scored / no shape match (metric doesn't apply to your controls — wrong metric, not missing work) / waiting for more annotators (fewer than 2 eligible annotations). The stored per-task scores drive the Metrics dashboard, the Insights panels, and the resolution strategy below.
Low agreement resolution
What to do when a task's agreement score lands below a threshold. Toggle on, then pick:
| Strategy | Effect |
|---|---|
| None (default) | Nothing happens automatically |
| Assign additional annotator | The task's overlap grows by 1 so the next available annotator picks it up from the normal queue (capped at annotators-per-task + 2) |
| Move to review queue | Raises a low-agreement item in the review queue and sends the task back into the annotator queue |
Minimum agreement — the threshold (default 60%). The strategy fires whenever a task's freshly computed score falls below it. More on the routing behaviour: Low-agreement auto-routing.
What's next
- Project insights → — where the scores this tab configures become dashboards
- Agreement metrics → — how every built-in metric scores each control tag
- Annotator evaluation → — the concepts behind the evaluation and resolution cards