Review annotations

After annotators have labelled tasks, review their output before it becomes the dataset. This page is the quality-side companion to the Review section: it covers what reviewers do, how the slice they see is chosen, how annotators get paused, and how ground truth fits in. The knobs themselves live in Project → Settings → Review (Review settings) and Project → Settings → Quality (Quality settings).

Who reviews

Anyone with the Reviewer org role (or higher) on the project, plus project managers and owners — see Roles. Members are added from Project → Members; managers can additionally pre-assign reviewers from Settings → Review → Assign reviewers. Assigned reviewers see their assigned tasks first and receive a digest of pending review work; unassigned tasks follow unless the project restricts the stream to assignments only.

Review annotated tasks

1. Open the project dashboard and press Review — it opens the review stream. The stream serves one task at a time, with every non-cancelled annotation on it.
2. For each annotation:
   • Correct → Accept.
   • Mostly correct → edit it in the editor, then Fix & Accept. The annotator's original is snapshotted, the change summary is shown wherever the annotation appears, and credit stays with the annotator.
   • Wrong → Reject, optionally with a comment (the project can require one). Rejecting does not delete anything and does not put the task back into the labelling queue by itself. The annotator sees the rejection on their project dashboard (Your rejected work) and can request one re-review with a reason; the item then re-enters the reviewer's stream for a fresh verdict (rejection disputes).
3. Keep going until the stream reports it is empty. Verdicts can be changed later from the same card (accept ↔ reject, amend a fix).

When several annotators produced identical results on a task, they cluster into one card and one verdict applies to all of them.

What reaches the stream

The stream only surfaces tasks that have reached the project's annotations-per-task quota and still carry at least one unreviewed (or disputed) annotation. Partially annotated tasks never appear.

Settings that shape the slice (Settings → Review):

Setting	Behaviour
Task ordering	Sequential (by task ID) or Random. Random shuffles tasks; annotations within a task keep creation order. The other ordering choices offered in the settings currently fall back to task-ID order in the stream.
Task limit (%)	Applies only when ordering is Random. A stable random N% of all project tasks (seeded by project ID) is reviewable; the rest never surfaces. Tasks with an open review-queue item are always included, whatever the percentage. Filters such as "show only finished tasks" don't change which tasks are in the sample. There is no way to bypass the cap for a hand-picked selection.
Show only finished tasks	Hide tasks not yet marked completed.
Tie-breaker only	Surface only tasks whose annotators produced different results; tasks where everyone agrees are skipped.
Require comment on reject	Reject is refused without a comment.
Conflicting annotations	single_truth (default): accepting one annotation marks it the task's ground truth and clears ground truth on its siblings. warn: accepting a second, differing annotation asks for confirmation; both stay accepted.
QC priority (Settings → Audio QC)	Tasks with an open review-queue item come first. On by default.

There is no "task is reviewed after all annotations are reviewed" switch: a task stays in the stream while any of its annotations is unreviewed, and the stream shows all of a task's annotations together.

Review annotator activity

The project's Analytics view (from the Data Manager toolbar) has two tabs:

• Review queue — every open / resolved / dismissed escalation with its reason, score, and notes; resolve or dismiss items here.
• Metrics — throughput over time, a per-annotator table (annotations, average lead time, average agreement), and the agreement histogram across tasks. Deeper cuts — per-label agreement, confusion matrices, the annotator-vs-annotator agreement matrix, the hardest tasks — are on the Insights page.

Project → Members shows each member's GT-task count and a paused badge when annotator evaluation has suspended them. Individuals see their own evaluation history and recent submissions under Profile → Analytics.

Pause an annotator

Manually

Anyone who can manage the project can Pause a member from Project → Members (any role; confirmation required) and Resume them later. A paused member cannot open tasks or save annotations on the project — they get "Your access to this project has been paused." — and work already submitted is unaffected. The member is notified when their access state changes.

Automatically

• Annotator evaluation pauses annotators whose ground-truth agreement drops below the project's minimum. These members show a paused badge on the Members page with a fixed tooltip explaining it was the automatic evaluation pause (no custom message or actor is stored); the button to lift it reads Re-enable so managers can tell it apart from a manual pause.
• The tasks-per-annotator limit caps how many tasks one annotator may claim; it's a ceiling, not a pause.
• Org-wide auto-suspend rules (review rejection rate, inactivity, low throughput, …) act at the organization level.

Behaviour-based triggers (spam / bot detection) are not shipped.

Ground-truth annotations

Ground truth is the yardstick for annotator evaluation: annotators are scored against the ground-truth annotation on each task they both worked. Predictions are not compared against ground truth.

• Mark an annotation as ground truth from the editor (the annotation's ground-truth toggle). A task keeps one ground truth: marking a new one clears the previous.
• Under the single_truth conflict policy, a reviewer's Accept also makes that annotation the task's ground truth.
• The Data Manager's Ground Truth column / filter shows which tasks already have one.
• Unmark from the same toggle in the editor.

Use the calibration ramp to route new annotators onto ground-truth tasks first.