Corpus engineering
Collection produces recordings; engineering produces a corpus. This page covers the decisions AfriAnnotate makes (or lets you make) so the dataset that leaves the platform survives peer review.
Corpus purpose: QC profiles​
The single most consequential setting on a speech project. Audio QC → Corpus purpose & validation → Audio profile:
| Profile | Philosophy | What changes |
|---|---|---|
robust_asr — real-world speech | Natural background noise is a feature, not a defect. A clip is good when every word is intelligible — the Common Voice lesson. | Silence gate off; browser noise-suppression off (keep the real ambience); noise/loudness findings (SNR, loudness, MOS, minor clipping, speaker consistency, reading speed) warn but never block; hard rejection only for corruption-level problems — too short, critically low speech ratio, critical clipping; a severely mismatched transcript is a soft reject (re-record), because verbatim accuracy still matters for scripted speech. |
clean_tts — studio speech | TTS and pronunciation corpora need clean signal. | Silence gate on; marginal SNR forces a re-record; strict noise floor. |
custom | Your thresholds. | Plain defaults plus every individual knob in the Audio QC tab. |
The profile drives three enforcement surfaces at once — the recorder's live hints, the automated QC pipeline, and the Listen criteria sheet — so contributors, machines, and validators enforce the same bar. Declaring the purpose is one dropdown; misaligned enforcement stops being possible.
Formats: capture high, derive down​
- Capture at 48 kHz (the platform default). Originals are stored at capture quality and never destroyed or downsampled in place — the information ceiling of your corpus is set at the microphone, once.
- Derive at export. The Hugging Face push offers per-clip 16 kHz mono WAV derivatives (the standard ASR training format) embedded as a typed audio column — optionally RMS-normalized to a speech-corpus working level. A clip that fails to decode exports its row without embedded audio rather than failing the push; the original URLs are always present regardless.
Leakage-free splits​
Serious corpus releases split train/dev/test with non-overlapping speakers and non-overlapping sentences — leaking either across splits inflates evaluation. The Hugging Face push offers this as a one-click option:
- Speaker + prompt disjoint (80/10/10) — a deterministic partition where no speaker and no prompt appears in two splits.
- Rows whose speaker and prompt land in different splits — the unavoidable price of double disjointness — are dropped and reported in the push result, never silently.
- Prompt identity follows provenance: a published community sentence keeps its identity across projects, so the same sentence can't leak between splits even via republication.
Assignment-time hygiene​
Diversity is enforced while collecting, not patched afterwards:
- One speaker, one prompt, once. No speaker ever records the same prompt twice — the task rotation avoids it and a server-side wall rejects direct submissions (skipping a prompt and recording it later is fine).
- Per-prompt recording caps. Publishing prompts can cap total recordings per sentence, because ten speakers reading ten sentences beats one speaker reading a hundred. The cap rides the platform's standard task-overlap machinery — a capped task simply leaves rotation when full.
Spontaneous speech​
Scripted read-aloud is only half of speech data. Set a project's prompt style to spontaneous (Audio QC → Corpus purpose) and:
- Tasks present topics or questions instead of scripts (a Spontaneous speech template ships in the voice-collection group).
- The Listen criteria automatically drop the verbatim-match check and judge what actually matters for unscripted speech: clearly audible, in the project's language, one main speaker, no cut-offs — still flavored by your QC profile's noise philosophy.
Validation ergonomics​
The details that make thousand-clip validation sessions humane are built into Listen: keyboard shortcuts, playback speed control, autoplay, the always-visible criteria sheet, mobile-sized touch targets — and the project's clip duration bounds (a global QC setting, the same for every profile; clips below the minimum hard-fail) keep clips judgeable in one breath. See Listen validation.