The reviewer's workflow
Assigning reviewers​
Project managers assign reviewers explicitly from Project → Settings → Review → Assign reviewers; assigned reviewers receive a digest of pending review work. Any user with the reviewer org role or project management rights can also review without a per-project assignment — the stream serves a reviewer's assigned tasks first and falls back to unassigned ones (or stays restricted to assignments when the project enables only assigned).
The review stream​
Reviewers work a stream, not a list: the project serves the next annotation needing review, the reviewer decides, the next one loads — the same focused rhythm annotators get, tuned for throughput.
The stream only serves tasks that have reached the project's annotations-per-task quota and still carry at least one unreviewed (or disputed) annotation. Ordering respects the project's settings: escalated work (tasks with an open review-queue item — QC-flagged, crowd-reported, low-agreement, and so on) comes first when QC priority is on; the rest follows in sequential or random order, with random sampling applied if the project caps the review slice.
Deciding​
For each annotation:
- Accept — one click; the annotation is marked reviewed and stands.
- Fix & accept — edit the annotation in the editor, then accept; the corrected version stands with full history (who annotated, who fixed).
- Reject — the annotation is sent back. Rejection is a signal, not a deletion: the annotator sees the outcome, and on speech projects the clip earns no ledger credit.
Reviewer actions are recorded per annotation — acceptance state, who reviewed, when, and the comment. They do not feed annotator evaluation, which scores annotators against ground truth only; the consequence path for verdicts is the org-level auto-suspend rule set, which offers a review-rejection-rate rule.
Review on speech projects​
On crowd-validated projects, votes resolve routine clips and the review queue carries the escalations — reports with structured reasons (wrong language, misread, offensive, bad audio) and crowd rejections for double-checking — which lead the stream when QC priority is on. In crowd mode a reviewer's accept/reject here is a review verdict on the annotation, independent of the crowd's resolution. In hybrid mode reviewers also hold a real override: the Listen page's Re-check crowd verdicts queue lists crowd-rejected, escalated and reported clips, and Validate / Reject there overrules the crowd, closes the open report and credits the speaker when it validates. A clip already credited through crowd validation is never credited twice.
Rejection disputes (request a re-review)​
Annotators have recourse when they believe a rejection is wrong:
- The project dashboard shows the annotator a "Your rejected work" card — each rejection with the reviewer's comment.
- Request re-review asks for a reason (required) and sends the item back into the reviewer's stream. The reviewer who rejected is notified.
- In the stream, the disputed annotation leads with the annotator's reason above the verdict buttons, which are open for a fresh judgment — accept, fix & accept, or uphold the rejection.
- Whatever the new verdict, the annotator is notified of the outcome. An upheld rejection is final: one request per verdict, so the loop can't ping-pong.
Accepting a disputed rejection credits the annotator's ledger exactly like any other acceptance.
After a verdict​
A reviewed card never shows the raw buttons again — it shows its verdict state (green Accepted, amber Fixed & Accepted with the change summary, red Rejected) plus two actions:
- Change verdict — flip a decision (accept ↔ reject); the annotator is re-notified of the corrected outcome.
- Amend fix — re-open the editor to adjust a fix. The annotator's original snapshot is never overwritten, so the diff always compares reviewer-vs-annotator no matter how many amendments happen.
Identical annotations cluster into one card ("Consensus, same labelling across N annotators") and one verdict applies to every member — each annotator keeps their own review record and ledger credit.
For the annotator: an accepted or fixed annotation opens read-only with a banner explaining the lock and, for fixes, what the reviewer changed. Rejections can be disputed once per verdict (request re-review).