Hugging Face Hub — import + push

AfriAnnotate treats Hugging Face Hub as a first-class source and sink for annotation data. This page walks through both directions: importing a dataset into a project, and pushing annotations back.

What you can do

• Import any public HF dataset into a project. Private and gated datasets work too if you paste a Hugging Face access token with read scope.
• Push back three ways: fresh (a brand-new dataset), enrich-source (add annotation columns onto the source repo — you need write access), or fork (source + annotations to your own repo).
• Bind multiple HF datasets to one project. Each binding has its own Unbind + Push controls; pushes can go to different repos independently.
• Aggregate multi-annotator work on push — pick majority, first, adjudicator, or raw. Output is a real datasets.Dataset with typed columns per label control-tag, not a raw JSON blob.
• Everything provenance-stamped. Every imported task carries Task.meta.hf_dataset_id, hf_config, hf_split, and hf_row_index. Every pushed dataset carries an SPDX license, consent summary, and annotator-language metadata in its dataset card.

Who can use Hugging Face

Access is decided per organisation, then per member:

1. Your own token (Account Settings → Hugging Face) always works for import and push.
2. The organisation's token, entered once by an owner or admin under Organization → Settings → Hugging Face. By default the roles that can create projects (owners, admins, managers) may import with it and owners, admins and managers may push. Owners can widen either to more roles and grant individual members import or push on top ("by role" marks what a role already covers).
3. Nobody else: the Import tab and the Push button show a disabled state with one sentence saying who can switch it on, instead of asking annotators for tokens they do not have.

Org policies still apply to pushes on top of this: exports restricted to managers, dataset licence before export, Hugging Face pushes disabled, and data stays in the organisation's own buckets.

A fine-grained Hugging Face token scoped to your organisation's repositories is the right choice for the organisation token: it limits what any member can do with it to those repositories.

One-time setup — save your HF token

Only needed when your organisation has not set a token, or when you want to push to repositories of your own.

Third-party tokens are stored encrypted-at-rest via a Fernet field keyed off HF_TOKEN_ENCRYPTION_KEY. API responses never return the raw value — only a boolean has_hf_token flag.

1. Go to huggingface.co/settings/tokens and create an access token with read scope (add write if you plan to enrich-source or fork into a repo you own).
2. In AfriAnnotate, open the Contribute editor for any project, or your Profile → Security & access tab (the Hugging Face card).
3. Paste the token into the Hugging Face token field and click Save. The audit log records the set with your actor, IP, timestamp, and the token's 3-char prefix — never the raw value.

You can also paste a per-project override token later — the token input is inline on the Project → Settings → Hugging Face tab.

Import a dataset into a project

The importer runs as a 3-tab wizard at project creation time — you can also add a new binding to an existing project via Project → Settings → Hugging Face → + Add binding.

Tab 1 — Dataset

1. Type the dataset ID (e.g. masakhane/mafand, abumafrim/lafand-mt, or google/xtreme).
2. Pick a config (if the dataset has multiple).
3. Pick a split — train, validation, test, or any custom split.
4. Set a row limit (leave blank for the full split). Useful for a pilot pass before you commit to labelling 50,000 rows.

Tab 2 — Column mapping

The wizard shows a table of the HF dataset's columns on the left, and your project's label config field names on the right. Map them inline — rename the HF column, or point a HF column at a different field name.

• Media columns (audio, image, video) — pick the HF column that holds the file / URL / bytes payload.
• Text columns — pick the HF column that holds the source text.
• Metadata columns — anything else can flow into Task.meta and be filtered on in the Data Manager.

Tab 3 — Confirm

The wizard shows a preview of the first few rows with your column mapping applied. Confirm to create the tasks. Each imported task carries the provenance stamps (Task.meta.hf_dataset_id, hf_config, hf_split, hf_row_index) so you can round-trip cleanly on push.

Annotate normally

Once tasks are imported, they behave like any other AfriAnnotate task — RBAC, review queue, agreement metrics, auto-suspend rules, consent flow, all apply. Nothing HF-specific in the annotation UI.

Push annotations back

Push controls live on Project → Settings → Hugging Face below each binding.

Push mode

Pick one of three:

Mode	What happens	When to use
fresh	Push to a brand-new HF dataset repo (either yours or an org's) with only the annotated rows and the annotation columns.	Fresh corpus release. The safe default.
enrich-source	Push annotation columns directly onto the source repo — the imported rows come back with their new labels attached. Requires write access on the source repo.	You own or co-own the source dataset and want the labelled version to be the canonical one.
fork	Push the source rows plus annotation columns to a new repo owned by you.	You don't own the source but want a labelled derivative under your own account.

Aggregation strategy

If the project is single-annotator, pick raw. For multi-annotator projects, choose how to reduce N annotations per task into one column value:

Strategy	Behaviour
majority	Modal value per label control-tag. Ties broken by the earliest-submitted annotation.
first	First submitted annotation wins.
adjudicator	Reviewer-adjudicated value, if a reviewer has weighed in; else falls back to majority.
raw	No aggregation. Every task carries an array of per-annotator annotations, one row per annotator per task.

Optional: enable _annotations sidecar column to preserve the full per-annotator provenance alongside the aggregated column — useful for downstream disagreement analysis without sacrificing the clean labelled column.

Speech-corpus options (fresh pushes)

Speech projects get two additional controls on fresh pushes:

• Audio format — original (URL columns only, the default), or embed a per-clip 16 kHz mono WAV derivative as a typed audio column (wav16k), optionally RMS-normalized (wav16k_norm). Originals are never modified; a clip that fails to decode exports its row without embedded audio rather than failing the push.
• Splits — Speaker + prompt disjoint generates 80/10/10 train/validation/test with no speaker and no sentence appearing in two splits (the leakage standard of serious speech releases). Rows that would straddle splits are dropped and the count is reported in the push result — never silently.

Speaker demographic columns (_speaker_age, _speaker_gender, _speaker_accents) attach only for single-speaker rows whose speaker has granted the sharing consent — no consent, no columns. Background and rationale: Corpus engineering.

Column shape

Push output is a real datasets.Dataset with typed columns per label control-tag. A <Labels> tag becomes a Sequence[str] column; a <Choices> tag becomes a ClassLabel column; a <KBRef> tag becomes a Struct({external_id, label}) column; a <TypedFeature> layer becomes a Struct(<per-feature dtypes>) column. No raw JSON blobs — the schema is what a downstream datasets consumer expects.

Dataset card

Every push writes a README.md (dataset card) to the target repo, assembled automatically from metadata the platform already holds:

• YAML frontmatter — language code, the SPDX license (when the Licensing wizard has been completed), and task_categories: automatic-speech-recognition for speech projects.
• License — SPDX identifier, name, URL, and the four constraint flags (attribution / commercial / derivatives / share-alike).
• Consent — which consent template(s) govern the project and how many distinct people signed each. Counts only — never signer names.
• Speakers — aggregate demographics (age bands, gender, language varieties) over contributors who granted the sharing consent. Aggregates only; no per-person data.
• Quality control — which audio-QC checks were enabled and their thresholds, the QC profile, and the crowd-validation vote rules.
• Collection method — read-aloud vs spontaneous prompt style, and whether prompts were community-collected.
• Volume — clip counts, total hours, and the collection date range.
• Splits — sizes plus the speaker/prompt-disjointness guarantee and the dropped-row count, when the disjoint split strategy is used.
• Provenance — aggregation strategy, sidecar column, source dataset link (enrich/fork modes).

Sections whose underlying data doesn't exist (no consent templates, no speaker profiles, QC off, …) are simply omitted — the card never invents metadata. The same generator produces the DATA_CARD.md included in standard export archives.

Unbind and clean up

Each binding has an Unbind button on the same settings tab. Unbind is destructive by default — it deletes the tasks and their annotations that were imported through that binding. If you want to keep the annotations, push first, then unbind.

For platform admins: a management command cleanup_orphan_hf_tasks sweeps orphaned tasks left behind by mid-workflow failures. See Management commands for the runbook.

Related pages

• <KBRef> — attach concepts from a Knowledge Base to imported HF text tasks for named-entity linking.
• <TypedFeature> — attach dtype'd attribute forms on top of HF-imported spans.
• Licensing wizard — pick the SPDX license that ships on your pushed dataset's card.
• Consent library — which consent template's summary lands in the pushed dataset's card.
• Python SDK — same import / push flow via client.hub.import_() and client.hub.push().