Review settings

Project → Settings → Review.

Controls how reviewers receive tasks in the review stream and how their decisions (accept / fix & accept / reject) are recorded.

There is no on/off switch — every project has a review stream. Anyone who can review (project Reviewer, or Owner / Admin / Manager, the project creator, staff) can open it; projects that don't need review simply never send anyone there.

Visible to Owner, Admin, Manager.

Reviewer instructions

• Instructions — HTML shown to reviewers
• Show before reviewing — open the instructions automatically when a reviewer enters the stream

Reviewing options

Setting	Default	Effect
Show only manually assigned tasks	Off	On: reviewers see only tasks they were assigned to. Off: assigned tasks are served first; once the reviewer's assigned queue drains, unassigned tasks follow
Review workflow (enable review)	On	Off: no review stream, no Review button on the dashboard; annotations are final as submitted. Audio QC and crowd validation are independent and keep working
Show only finished tasks	Off	On: tasks not yet marked complete (is_labeled) are hidden
Tie-breaker mode	Off	On: only tasks whose annotators produced different results are served — the reviewer breaks the tie. Tasks where everyone agrees are skipped (the stream scans up to 200 candidates per request)
Task ordering	Sequential	sequential / random / agreement / model_confidence — see below
Conflicting annotations	Single ground truth	What happens when more than one annotation on a task is accepted — see Conflict policy
Task limit (%)	100	Share of project tasks included in the stream when ordering is Random — see below

Regardless of these settings a task only enters the stream once it has at least maximum_annotations non-cancelled annotations and at least one of them is still unreviewed (or carries an open re-review dispute). Partially-annotated tasks never surface.

Task ordering

Order	Behaviour
Sequential (by Task ID)	Lowest task ID first
Random	Reshuffled on every request
By agreement (uncertain first)	Stored on the project; the stream currently serves these in task-ID order — use Tie-breaker mode to surface disagreements
By model confidence	Stored on the project; currently served in task-ID order

QC priority. When the project's QC priority in review toggle (Settings → Audio QC) is on, tasks with an open review-queue item (QC failure, low agreement, outlier, eval failure, manual flag) are served first, ahead of whichever ordering is picked.

Task limit

Applies only when ordering is Random. A value below 100 caps the pool to that percentage of the project's tasks:

• The subset is picked once per project (seeded by project ID) and is the same for every reviewer; only the order reshuffles per request.
• Tasks with an open review-queue item are always included, even when they fall outside the sampled subset.
• 0 is a no-op — nothing auto-accepts; the full pool is served.
• The limit is ignored when reviewers open tasks directly from the Data Manager (selection or quick view).

Conflict policy

Governs accepting several annotations on the same task:

Policy	Effect
Single ground truth (accept one) (default)	Accepting an annotation makes it the task's ground truth and clears ground truth on its siblings
Allow, but warn on conflict	Allowed; the review UI confirms when a second, differing annotation is accepted
Allow all (keep for agreement)	Every accepted annotation stands — use when the goal is inter-annotator agreement and a single label is produced later at export

Reject options

• Reviewers must leave a comment on reject — enforced in the UI and by the server (a reject without a comment is refused).

Data Manager for reviewers

• Show the Data Manager to reviewers (default on)
• Show unused task-data columns to reviewers
• Show agreement to reviewers in the Data Manager

Accept / reject behaviour

A verdict sets review_state on the annotation (accepted, fixed, rejected) plus reviewed_by / reviewed_at and the optional comment.

Decision	Effect
Accept	review_state = accepted; annotator credited in the contribution ledger
Fix & Accept	Reviewer's edits saved on the annotation; the annotator's original is snapshotted so the UI can show a "fixed in review" diff; review_state = fixed
Reject	review_state = rejected. The annotation stays in the database and is not re-served to annotators. The annotator can ask for a re-review (dispute); an open dispute puts the item back in the stream

Reviewers can revisit reviewed items and change a verdict; the annotator is notified again and any credit that no longer applies is reversed. Annotators learn outcomes through a per-reviewer digest flushed when the reviewer's queue empties.

Bulk shortcut

Managers can skip per-item review on wholesale-trusted projects: Project → Settings → Danger Zone → Accept All Pending Annotations accepts every unreviewed annotation and credits the annotators, after a typed confirmation showing the exact counts. Tasks with an open review-queue item are always skipped — see Accept all pending.

Assigning reviewers

Settings → Review → Assign reviewers… attaches reviewers to tasks in bulk:

1. Pick reviewers from the project's members
2. Choose the scope — all tasks, only tasks with unreviewed annotations, or specific task IDs
3. Choose add (keep existing reviewers) or replace

Assignees get an in-app / email notification with a link to the review stream. Assignment only affects ordering (and hard-restricts when Show only manually assigned tasks is on); reviewers can always enter the stream without pre-assignment.

Review stream UX

• Task opens with its annotations; each shows review_state, the reviewer's comment, any open dispute reason, and the fix diff
• Accept / Fix + Accept (the label switches when the reviewer has edited) / Reject
• Keyboard: Ctrl+Enter accept, Ctrl+Space reject (see Hotkeys)
• After a verdict the next task loads; reviewed items can be revisited from the reviewed list

What's next

• Review workflow overview → — the end-to-end reviewer journey
• Agreement metrics → — how agreement is scored
• Annotation settings → — the annotator-side companion settings