The review queue — escalations
Sampling reviews a fraction of ordinary work. The review queue is the other door: work that something — a machine, the crowd, a quality signal, a human — flagged. While a queue item is open, its task is exempt from review sampling, so flagged work always stays reachable in the stream. Managers see the queue itself (with resolve / dismiss) in the Review queue section of the project's Analytics page.
Escalation reasons
Each queue item carries a machine-readable reason:
| Reason | Raised by | Meaning |
|---|---|---|
low_agreement | Agreement engine | Annotators disagree on this task beyond the project's threshold. |
outlier | Consensus analysis | One annotator dissents from an otherwise-consistent consensus. |
evaluation_fail | Annotator evaluation | The annotator's evaluation score dropped below the bar. |
manual | A human or an integrated flow | Explicit flags — including crowd reports (wrong language / misread / offensive / bad audio, with the reason in the item's notes) and crowd rejections ("N no-votes vs M yes") from Listen validation, and audio-QC failures when the project's action on fail is route to review or require re-record (the failing check names and the lowest check score ride along on the item). |
Items are deduplicated per task and reason — an event that fires repeatedly as more annotators submit doesn't flood the queue.
Item lifecycle
Open → resolved (a reviewer dealt with it) or open → dismissed (flag judged spurious). Resolution is recorded with the reviewer and timestamp.
Priorities
With QC priority on (Audio QC → action-on-fail section; on by default), tasks with an open queue item surface at the head of the review stream — reviewer attention goes to probable problems before ordinary work. With it off, open items are still exempt from sampling but take their normal place in the ordering.
Why this design
Escalation is what makes the platform's automation trustworthy: machines and crowds are allowed to act because their contentious calls always reach a human. Every automated rejection path on the platform terminates in this queue, not in silence.