Skip to main content

Active learning

Active learning flips the labeling order: instead of annotating tasks in sequence, annotators are served the tasks the model is least sure about — where a human label adds the most value.

Three pieces make the loop, and AfriAnnotate wires all three:

  1. Uncertainty sampling — with Task sampling → Active learning (General settings), the next task served is the unlabeled task with the lowest prediction score for the project's current model version
  2. Retraining — with the loop enabled, AfriAnnotate counts new annotations and calls the connected ML backend's /train endpoint every N annotations (the Retrain every knob, default 25)
  3. Prediction refresh — after a successful retrain request, AfriAnnotate asks the backend to re-predict a batch of still-unlabeled tasks (up to 200 per cycle), so the sampling in step 1 ranks on the new model's uncertainty, not stale scores

Without steps 2–3, uncertainty sampling quietly degrades: the ordering keeps ranking on the scores of whatever model version last predicted, no matter how many labels have arrived since.

Requirements (honest ones)​

  • A deployed ML backend that implements the Label Studio fit/predict contract — POST /train and POST /predict at minimum. AfriAnnotate triggers training; it does not host it. If your backend's /train is a no-op, the loop degrades to plain uncertainty sampling on whatever scores exist.
  • Task sampling set to Active learning on Settings → General. With Sequential or Random sampling the retrain loop would burn GPU time without changing task order, so the toggle stays disabled until sampling is switched.
  • The backend should return a score with each prediction and a model_version — scores drive the ordering; the version keeps fresh predictions distinguishable from stale ones.

Turning it on​

Project → Settings → Model → Active learning card:

  1. Connect an ML backend (same page) — the card needs one
  2. Switch Task sampling to Active learning on General settings (the card links the exact hint if you haven't)
  3. Enable the toggle, set Retrain every (default 25)
  4. Save

The card shows a status line — Last retrained <time> · <n> annotations since — so you can see the loop actually cycling.

How a cycle runs​

On each submitted annotation (ground-truth annotations excluded) the project's counter increments. At the threshold:

  1. POST /train is sent to every connected backend with all annotated tasks (the standard Label Studio training payload)
  2. If at least one backend accepts, the counter resets and the timestamp is stamped; if all reject, the counter stays put and the next annotation retries
  3. Up to 200 unlabeled tasks are sent for fresh predictions. Tasks beyond the batch keep their previous scores until the next cycle (or a manual Batch predictions run from the Data manager); the skip is logged server-side

Cycles run on the background queue when Redis + django-rq workers are available (the reference Cloud Run deployment), and inline best-effort otherwise. Either way the loop is failure-tolerant by contract: a down, slow, or erroring backend never blocks or rolls back an annotation submit — failures are logged and the loop retries at the next threshold.

Choosing Retrain every​

SituationSuggested value
Fast incremental trainer (logistic head, adapters)10–25
Full fine-tune per cycle (minutes of GPU)100+
Several annotators submitting in parallelHigher — cycles overlap otherwise; the backend should also guard against concurrent trains

What's next​

  • ML backend → — deploying and connecting the model service this loop drives
  • Predictions → — how prediction scores and model versions are stored and displayed