Review — the second pair of eyes
Annotation produces judgments; review decides which of them become the dataset. AfriAnnotate's review layer is built around one idea: reviewer attention is the scarcest resource on the platform, so everything about review exists to point it at the work that needs it most.
The model in one diagram
Escalations that jump the sampling gate: automated audio-QC failures, crowd validation reports and rejections, low inter-annotator agreement, consensus outliers, annotator evaluation failures, and manual flags.
Who reviews
Reviewing takes the reviewer org role or project management rights. Managers can assign reviewers to a project explicitly; assigned reviewers get a digest of what's waiting.
What enters review
Two doors:
- Sampling — the project's review settings
set a sampling rate (
task_limit_percent). It only applies when the review ordering is Random: 100% means everything is reviewed; 20% means a stable random fifth of the project's tasks (seeded per project, so the subset doesn't shift between sessions). Under sequential ordering every reviewable task surfaces. This is the budget knob. - Escalation — tasks with an open review queue item are exempt from sampling: work that a machine check, the crowd, or a quality signal flagged always stays in the stream, and surfaces first when the project enables QC priority.
The outcomes
- Accept — the annotation stands.
- Fix & accept — the reviewer corrects it in the editor and the fix completes automatically on save: the annotator's original is snapshotted, the verdict is recorded, and everywhere the annotation appears you'll see what changed — an amber note with before → after chips in the review stream, and the changed regions highlighted on the editor canvas. Attribution and credit stay with the annotator.
- Reject — the work goes back; on speech projects a rejected clip earns nothing, because credit lands at validation.
Accept all pending — the Danger Zone bulk action
On projects you trust wholesale, per-item review is optional. Project → Settings → Danger Zone → Accept All Pending Annotations (the dashboard's Danger button jumps straight there) accepts every unreviewed annotation in one action and credits each annotator in the contribution ledger — straight to the pay process.
| Rule | Behaviour |
|---|---|
| Who | Managers and above (Owner / Admin / Manager, or the project creator). Reviewers don't see the card. |
| Escalation guardrail | Annotations on tasks with an open review-queue item — crowd reports, QC routing, low agreement — are skipped, never blanket-accepted. The result toast reports how many flagged items still need review. |
| Crediting | Each accepted annotation credits its annotator once (idempotent). Work already paid through crowd validation isn't paid again, and your own annotations are accepted but never self-credited. |
| Untouched | Skipped/cancelled annotations and already-reviewed annotations. |
The confirmation dialog shows the exact pending / flagged counts and requires typing "accept" before it runs, and the action can't be bulk-undone — the skipped escalations remain in the review queue for a per-item decision.
How review composes with the rest of the platform
- Automated audio QC condemns corrupt clips before humans see them and routes marginal ones here — machines filter, humans judge (Audio QC).
- Crowd validation resolves the easy majority by agreement; its reports and rejections escalate here — the crowd scales, reviewers arbitrate (Listen validation).
- Quality measurement (agreement metrics, annotator evaluation) feeds the escalation reasons and is documented under Ship datasets.
- Reviewers on speech projects work the escalations the crowd produces. In crowd mode votes resolve clips and reports / crowd rejections land in the review queue for a normal review verdict. In hybrid mode reviewers additionally get the Re-check crowd verdicts queue on the Listen page and an explicit override (Validate / Reject) that overrules the crowd's resolution, closes the open report and credits the speaker when it validates.
Pages in this section: the reviewer's workflow and the review queue.