The community speech-corpus pipeline
Building a speech corpus with a community is normally five tools duct-taped together: a recording app, a validation spreadsheet, a consent PDF folder, a payment ledger someone maintains by hand, and a pile of export scripts. Every seam loses data, money, or trust.
AfriAnnotate ships the entire pipeline as one system — its recording and validation studied from what works at scale (Mozilla Common Voice's validation and sentence-collector model) and built consent-first throughout. Field campaigns run through Facilitators, a platform-wide capability described there.
The pipeline at a glance
| Stage | What happens | Where it's documented |
|---|---|---|
| 1. Prompts | The community submits candidate sentences with a required license attestation; peers vote them in; accepted prompts publish as recording tasks. Or go scriptless with spontaneous topics. | Community prompt collector |
| 2. Recording | Contributors read (or speak freely) into a live-metered recorder capturing at 48 kHz, tuned by the project's declared corpus purpose. | Audio QC · Corpus engineering |
| 3. Machine QC | Every submission runs automated checks server-side (clipping, loudness, SNR, silences, and more); corruption-level failures never reach human reviewers. | Audio QC |
| 4. Speaker profile | Speakers complete an all-fields-optional demographics step once; details attach to released data only with separate, versioned, revocable consent. | Speaker profiles |
| 5. Crowd validation | The Listen page serves clips for fast Yes/No judging against criteria derived from the project's QC profile; N agreeing votes resolve a clip; problems escalate to reviewers. | Listen validation |
| 6. Payment | Validated work credits an append-only ledger automatically; owners declare rate cards and export payout CSVs — grouped by field coordinator where campaigns run through facilitators. | Contributions & payment · Facilitators |
| 7. Motivation | Personal goals and streaks, opt-in leaderboards, and public per-language progress keep contribution visible and voluntary. | Goals & progress |
| 8. Publication | Export to the Hugging Face Hub with embedded 16 kHz ASR-ready derivatives and speaker+prompt-disjoint splits — the leakage standard of serious corpus releases. | Corpus engineering · Hugging Face Hub |
Everything crowd-shaped starts at the Discover catalog, where public projects recruit contributors with one-click joining and consent gates enforced in the join flow itself.
Why this design earns trust
Three rules run through every stage:
- Consent is enforced, not filed. Nothing leaves a device, enters the corpus, or appears on a leaderboard without an explicit, versioned, recorded, revocable choice — enforced server-side at the API, not by UI politeness. See Consent and Speaker profiles.
- Machines and humans enforce the same bar. The recorder's live hints, the automated QC pipeline, and the human validation criteria all read one declared corpus purpose — so a robust-ASR project keeps the natural background noise that makes models robust, and a TTS project stays studio-clean, consistently at every stage.
- Credit lands at validation, not submission. Contributors earn when the community accepts their work — which keeps the ledger honest and the incentive aligned with quality, not volume.
Works offline, like everything here
Contributors on the mobile app can download a validation pack and keep judging clips with no connectivity; votes queue and replay safely on reconnect (duplicates are impossible by construction). Recording and annotation work offline on the desktop app with deliberate sync. See Mobile and Offline sync.