Listen — crowd validation
Two reviewers can't validate a community's output. Listen is AfriAnnotate's crowd-validation surface — the Common Voice model adapted to this platform: contributors judge each other's clips in a fast, focused loop, agreement resolves clips, and anything contentious escalates to human reviewers automatically.
Listen appears as a project tab on speech projects (and on prompt-collection projects, where the same loop judges submitted sentences).
Who validates
Project Settings → Audio QC → Corpus purpose & validation:
| Mode | Who votes |
|---|---|
| Reviewers only | Managers/reviewers — classic review, the default. |
| Crowd | Any eligible contributor or org member votes; N agreeing votes resolve a clip. |
| Hybrid | Same voter pool and resolution rules as Crowd, plus reviewer override: project reviewers (and managers) get a Re-check crowd verdicts queue on the Listen page — crowd-rejected, escalated and reported clips — and can overrule any verdict with Validate / Reject. The override is recorded on the clip (who, when, note), closes the open crowd report, and credits the speaker when it validates (never twice). Managers and staff can override in every mode; reviewers also see the override buttons while voting normally. |
You also set the vote thresholds — votes to validate and votes to reject (Common Voice uses 2 — the default for both). Managers and staff can always vote, whatever the mode.
The judging loop
The page serves one clip at a time with total focus:
- Player with autoplay, replay, and playback-speed toggle.
- Yes / No / Skip / Report — with keyboard shortcuts (Y/N/S/R, Space to replay) for high throughput; on touch devices, thumb-sized two-up Yes/No targets.
- Reports are structured: wrong language, misread, offensive, bad audio, other. A report is a No-vote that escalates immediately.
- An always-visible criteria sheet states exactly what "Yes" means —
derived from the project's QC profile, so voters enforce the same bar as
the machines and the recorder hints:
- Every profile: every word audible; matches the prompt exactly; one main speaker; no cut-offs.
- Robust ASR adds: background noise is fine — even background voices — as long as the speech is clear. (Rejecting natural noise throws away exactly the audio that makes ASR robust.)
- Clean TTS adds: quiet background required; steady volume.
- Spontaneous projects swap the verbatim criterion for language-and-audibility judging (corpus engineering).
- An opt-in leaderboard panel shows the project's top validators (goals & progress).
Rotation rules — where the fairness lives
Enforced server-side, not by UI politeness:
- You never see your own clips.
- You never see a clip twice.
- Resolved clips leave the pool.
- Clips the automated QC hard-failed are skipped — machines already condemned them; human attention is never spent confirming corruption.
- Speakers never record the same prompt twice, so the pool stays diverse (corpus engineering).
Resolution and escalation
- N yes-votes → validated. The speaker is credited in the ledger at this moment — credit lands at validation, not submission.
- N no-votes → rejected, and the clip escalates to the project's review queue so a human can double-check the crowd.
- Any report → immediate escalation with the reason attached.
- Every vote also credits the voter — judging is contribution too.
Offline validation (mobile)
"Download for offline" caches a validation pack (a batch of clips with prewarmed audio). Offline, Listen keeps serving the next unjudged cached clip; votes queue in the outbox and replay on reconnect. Replay is safe by construction — one vote per person per clip is a database constraint, and the queue treats a duplicate rejection as success. See Mobile architecture.