Auto-suspend rules

Auto-suspend rules automatically suspend org members when their behaviour crosses an operator-configured threshold. Useful at scale where reviewing every annotator's metrics by hand isn't feasible.

Suspended members stop receiving tasks from the org. Work they've already submitted stays in the database. An Owner or Admin can unsuspend them at any time from Organization → Members; the reason (which rule, which threshold) is recorded in the audit log and in the notification the member receives.

This is AfriAnnotate-specific — built for crowdsourced data labelling where annotator quality varies wildly and manual review of every contributor doesn't scale.

Where it lives

Organization → Settings → Auto-suspend. Visible to Owner and Admin (the rules API, /api/organizations/<id>/auto-suspend-rules, also accepts Managers).

Rules are evaluated by a background job that reschedules itself every 24 hours and starts with the server — no operator setup. To evaluate immediately, or to preview without writing anything:

python manage.py run_auto_suspend_rules [--dry-run] [--org <id>]

Anatomy of a rule

Field	Meaning
rule_type	One of the seven types below
label	Free-text name shown in the rules list
enabled	Disabled rules are skipped by the job
params	Type-specific thresholds (JSON). Missing keys fall back to the defaults listed below
grace_period_days	0 = suspend on the first hit. Otherwise the member is warned first and suspended only if still over threshold at a later run once that warning is older than the grace period

A rule evaluates every active member of the org — anyone not removed and not already suspended. There is no per-rule project or role scope, and no separate "action" field: a rule either warns (inside its grace period) or suspends.

The seven rule types

rule_type	Over threshold when…	params (defaults)
inactivity	the member has had no activity for more than max_idle_days	max_idle_days 90
low_agreement	mean ground-truth accuracy over the window is below min_agreement, with at least min_tasks gold comparisons (a small sample never triggers)	min_agreement 0.7, window_days 30, min_tasks 10
rejection_rate	reviewer rejection rate over the window is at or above max_rate, with at least min_reviews reviews	max_rate 0.4, window_days 14, min_reviews 20
low_throughput	fewer than min_tasks annotations submitted in the window	min_tasks 10, window_days 7
high_incomplete_rate	share of cancelled / abandoned annotations in the window is at or above max_rate, with at least min_assigned assignments	max_rate 0.5, window_days 14, min_assigned 10
quota_exceeded	total annotations reach the member's per-member quota, or fallback_limit when the member has none (0 = skip)	fallback_limit 0
custom	a library metric, compared with comparator (gt / gte / lt / lte) against threshold over window_days	metric inactivity_days, comparator gt, threshold 0, window_days 30, min_sample 0

Notes:

• Low agreement is a ground-truth signal: it reads the per-annotator evaluations produced when annotators are compared against gold tasks (see Annotator evaluation), so a project with gold tasks must be in the mix.
• Rejection rate is annotator-vs-reviewer; agreement is annotator-vs-gold. Both signals correlate but cover different failure modes.
• Low throughput is useful for contractors paid per task — it flags people who started but never delivered. Give it a grace period so legitimate breaks get a warning first.
• Custom picks from a curated metric library rather than free-form expressions: inactivity_days, annotations_count, rejection_rate, incomplete_rate, ground_truth_accuracy. A metric that can't be computed for a member (insufficient sample) skips that member instead of firing.

Warnings, suspension, and the grace period

On each run, for every enabled rule and every member it returns:

1. If the rule has a grace period and the member hasn't been warned within that many days, they get a warning notification (auto_suspend.warning: what's wrong and how many days they have).
2. Otherwise — no grace period, or the warning has aged past it and they're still over threshold — the member is suspended (suspended_at set on the membership) and notified with the reason.

Audit trail

• Each automatic suspension logs member.suspend.auto with the member and the reason text (rule + threshold).
• Creating, updating or deleting a rule logs org.auto_suspend.create / .update / .delete, and the org owner is notified whenever someone else changes a rule — a policy that can lock members out never changes silently.
• Manual suspensions / unsuspensions from the Members page log member.suspend / member.unsuspend.

Admins reviewing a suspension later see exactly why the system pulled the trigger.

Unsuspending

Only manually: an Owner or Admin opens Organization → Members, finds the member, and clicks Unsuspend. There is no self-appeal flow and no automatic unsuspension when the metric recovers — a member who is unsuspended but still over threshold will be picked up again on the next run unless the rule is adjusted or disabled.

When NOT to use auto-suspend

Auto-suspend is a blunt instrument. Skip it when:

• You have < 20 annotators — manual review is fine, auto-suspend is over-engineering
• Your annotators are vetted internal staff — a false-positive suspension is more disruptive than the labour saved
• Your project has high natural variance — diverse-perspective labelling (subjective tasks) won't have "agreement" that maps to "quality"

For a monitoring-only rollout, run run_auto_suspend_rules --dry-run to see who would be hit, or give rules a long grace period so members only receive warnings while you tune thresholds.

What's next

• Annotator evaluation → — the ground-truth metrics the low_agreement rule reads
• Members → — manual suspend / unsuspend / remove controls
• Roles → — what each role can see + do, including who can configure auto-suspend