Testing & QA

AfriAnnotate is tested in five layers that back each other up:

Layer	What it is	Where it lives
Manual suites	Human checklist steps, one per behaviour, grouped by area	warakacommunity/afriannotate-testing → suites/
In-app reports	Any signed-in tester files a failed step or suggestion from inside the app	see Testing programme
Automated tests	Playwright smoke + accessibility + API authorization, run in CI	testing repo → automation/
Nightly CI	Runs the automated suites against production every night and files failures as issues	testing repo → GitHub Actions
Code checks	Lint, security scan and migration check on every push; Python and frontend unit tests nightly, on pull requests and on demand	app repo → .github/workflows/ci.yml, see below

The manual suites and the automated suites share one identifier scheme (OB-22, A11Y-13, AN-NER-07, …), so a checklist step, an in-app report, an automated failure and a GitHub issue all point at the same thing.

Manual test suites

559 checklist steps across 39 suites in 14 areas. Each step is a "do this → you should see that" line with its own Report a problem link (which pre-fills the issue form). Suites are generated, never hand-edited, so IDs stay stable.

Area	Prefix	Steps
Onboarding & auth	OB	24
Annotation editors (26 types)	AN-…	409
Review & quality	RQ	15
Project management	PM	18
Organization & platform	OP	16
Speech & sign language	SP	10
UI & design	UI	13
Accessibility	A11Y	14
Mobile	MB	10
Desktop & offline	DT	7
Security & privacy	SEC	10
Localization	L10N	4
Docs	DOC	4
Performance	PERF	5

The annotation editors area covers every labeling type on its own suite: ASR, audio classification, audio segments, bounding box, brush/mask, classification, dialogue, HTML, image classification, keypoints, multi-type, NER, OCR, pairwise, PDF, polygons, POS, QA, ranking, rating, relations, text area, taxonomy, time series, video classification and video events.

A testing round opens these as tracked checklists on GitHub (one issue per area) so a cohort can work through them and tick items off.

In-app tester reports

Testers do not need a GitHub account. While the Testing programme module is on, anyone signed in can Report a test step (avatar menu, the floating Report button, or Alt+Shift+R) with a screenshot; each report becomes a GitHub issue in the testing repo, pre-filled with the page, build, client and browser. Retire it with one toggle in Platform → Testing when a round ends.

Automated tests

All under automation/ in the testing repo, written with Playwright. Every test runs in two projects — desktop Chromium and a mobile (phone) viewport — and is role-aware: a step the signed-in account cannot legitimately reach (for example a project it has no access to) is skipped with a reason, never failed.

Smoke — the critical paths

Spec	Steps	Checks
smoke/auth.spec.ts	OB-20/21/22	Logged-out lands on the login page with no avatar; a wrong password shows a neutral error and stays on login (no traceback); a correct sign-in lands on Home with the avatar and no console errors
smoke/navigation.spec.ts	OB-23/24	Home, Projects, Account, Organization → Members and a project's Data Manager all render without the error screen; the label stream opens the editor with a task (or a healthy "nothing left")

Accessibility

a11y/axe.spec.ts (A11Y-13) runs an axe-core scan tagged WCAG 2.0/2.1 A & AA + best-practice on the login, home, projects, account, members and Data Manager pages. Serious and critical violations fail the run; moderate and minor ones are attached to the report for review. The full axe JSON is saved per page.

API authorization probes

scripts/authz-probe*.mjs log in as each role (owner, admin, manager, reviewer, annotator) and assert that privileged actions are refused for the wrong role and allowed for the right one — org role changes and member removal, mass-assignment of ownership/quota/freeze, platform admin endpoints, and project/task/annotation/import/export write gates. These backed the authorization hardening described in the Security model and are run against a dev or staging deployment (the one destructive check targets a throwaway project it creates).

Continuous integration

Workflow	When	What
Automation	Nightly (03:00 UTC) + on demand	Runs smoke + accessibility against production (or any base_url), uploads the report (best-effort), and files each failure as a GitHub issue (source:automation; a recurrence comments on the open issue instead of opening a new one)
Open a testing round	On demand	Opens the round's per-area checklist issues
Triage	On new reports	Labels reports (status:needs-triage, module:…, severity:…, surface:…) and adds them to the project board

What we have done

• API authorization audit. A full pass for broken access control (BAC/IDOR) across organizations, projects, tasks, annotations, imports, exports, webhooks, consent and session policy, closing the gaps with object-level permission checks and verifying every one with the role probes above. See the Security model.

• First automation round. Shook out the nightly suite end to end. It found and fixed two harness flakes (an execution-context-destroyed race and unbounded login waits) and, in doing so, surfaced a real product bug: a fresh visitor's service worker reloaded the login page as it took control, resetting the two-step sign-in form. Both the harness and the product bug are fixed; the suite now runs green.

• Owner-deletion incident (12 September 2026). A platform admin deleted the account that owned an organisation with no other members. The organisation was left ownerless (created_by is SET NULL) and the feature-flag user context dereferenced the owner's email in a context processor that runs on every page, so the one account still pointed at that organisation got "Uh oh, something went wrong" everywhere. Found from production logs within minutes; fixed the same hour in two places: the dereference is now null-safe, and deleting an owner can no longer leave an organisation ownerless (empty organisations go with the account, organisations with data pass to the deleting staff user). Both are pinned by core/tests/test_ownerless_org.py, and the organisation suite gained steps OP-15 and OP-16 so a testing round exercises owner deletion deliberately, and the two-step confirm was replaced by a single delete modal that shows the full impact, requires a decision per owned organisation and a typed confirmation (A11Y-14 covers its keyboard and screen-reader behaviour). Lesson recorded in Users → Delete and the Operator FAQ.

• Select tag shipped untested (17 September 2026). The new <Select> control (searchable language / country / script / currency lists) was deployed after unit tests only. Annotators hit three bugs in the first hour: a crash when the tag had no children yet, a raw translation key in the dropdown, and answers stored as nested arrays because <Choice> did not recognise the new parent tag. Driving the real app with a browser script then found two more that unit tests could not see: options that arrive after a task opens were counted as an edit (every task opened "dirty" and autosaved a draft), and on a saved annotation that early default answer made the lazy loader think the annotation was already hydrated, so the stored answer never appeared. All five are fixed. Lesson: a tag or setting that touches the editor is browser-tested against a running stack (sign in, open a task, use it, submit, reopen) before it is deployed; the scripts for this live in the testing repository under automation/tests/editor/ and automation/tests/settings/ (npm run test:editor, npm run test:settings).

• Same audio on every task (21 September 2026). Annotators in one organisation reported that every audio task played the same clip, across two projects and for everyone using that browser. The files in the bucket were fine (every upload had a different checksum; the identical sizes were fixed-length chunks). The cause was the app's offline service worker: it cached media with ignoreSearch, and on cloud storage every uploaded file is served from one path (/storage-data/uploaded/?filepath=…) where only the query names the file, so the first clip cached was served for all of them. The service worker (v12) now keys media on the full file identity, dropping only cache-buster and presign parameters, and the version bump discards the poisoned cache on the next page load. The same release split file import into one request per file with its own progress bar and error card, lowered the resumable-chunk threshold to 24 MB (Cloud Run refuses single requests above 32 MB), added a pre-upload storage check (GET /api/projects/<id>/import-limits), and gave organisations a Plan and usage strip with a multi-resource Request an increase flow reviewed item by item under Platform → Organizations, editable organisation tiers with a default for new organisations, and a community inbox for platform-directed notices (data_import/tests/test_import_limits.py, organizations/tests/test_quota_request.py). Lesson: any cache that ignores the query string must be proven safe against every URL shape the deployment serves, not only the local one; the browser script for the import page (progress + distinct tasks) now runs before deploys.

• Audio / video splitter (21 September 2026). Built with tests first: data_import/tests/test_media_split.py covers probing, silence detection, budgeted cutting into uploads and tasks, deleting the original, validation, permissions, the storage cap mid-run, and a real ffmpeg video split. A browser script then drove the import-page flow (upload, suggest cuts, add a manual cut, cut, delete the original) and the Data Manager entry point before anything was deployed.

• One submit, three annotations (21 September 2026, local SQLite). Submitting a clip task in the Data Manager saved three identical annotations a second apart. The browser had sent one request. The cause was the SQLite "database is locked" retry middleware: a background sync push held the lock, a write after the annotation insert failed, the request answered 500, and the middleware re-ran the whole request twice more, each run inserting again. Production runs Postgres, where that middleware is off, but the desktop build is SQLite. Fix: annotation creation is now idempotent. When the client sends no unique id, the server derives one from the task, author, draft, lead time and result, so any repeat of the same request returns the existing annotation (tasks/tests/test_annotation_idempotency.py). The test run also switches the sync threads off (AFRI_SYNC_AUTOPUSH_DISABLE, AFRI_SYNC_AUTOPULL_DISABLE) so they cannot contend for the lock during tests.

• Data residency policy and organisation Hugging Face access (21 September 2026). Both shipped with tests first (organizations/tests/test_org_policies.py::TestOwnBucketOnly, organizations/tests/test_hf_access.py, the bucket write-back case in data_import/tests/test_media_split.py) and a browser walk-through of the organisation Hugging Face tab, the disabled and enabled import states, and the project access view.

What is next

• Seed a [TEST] project with tasks so the Data Manager, label stream and per-editor annotation steps run instead of skipping.
• Per-editor annotation smoke — submit one annotation for each of the 26 editor types.
• Review, verdict and org-management write flows in CI (currently manual-only).
• Fold the authorization probes into CI against a staging deployment, on every change.
• Visual regression — screenshot diffs on the key pages.
• Performance budgets — turn the PERF suite into automated timing thresholds.

Code checks on every change

The application repository (warakacommunity/afriannotate) runs a CI workflow. The repository is private, so its minutes are counted, and the checks are split by cost:

• On every push to production (about 9 minutes): the two linters, the security lint, the secrets scan and the migration check on SQLite.
• Every night at 02:30 UTC, when the day had commits; on every pull request; and by hand (Actions → CI → Run workflow): everything below, with the Python tests and the frontend unit tests (about 45 minutes).

The full Python suite is also run on the developer's machine before a push. A change cannot land with any of these red:

Check	What it verifies	Run it locally
Python lint (ruff)	Lint rules and formatting for all Python code	ruff check . && ruff format --check .
Frontend lint (biome)	Lint, accessibility rules and formatting for the web code	cd web && yarn biome check .
Security lint (bandit)	Common Python security mistakes (unsafe XML, missing timeouts, weak hashes, raw RQ enqueues)	bandit -r label_studio/ -ll
Secrets scan (gitleaks)	No credentials or tokens in the commits being pushed	git diff HEAD | gitleaks stdin
Migrations up to date	makemigrations produces nothing new, on SQLite and on PostgreSQL	python label_studio/manage.py makemigrations --check
Python tests	The full pytest suite on SQLite (Python 3.11, as in production) and on PostgreSQL (Python 3.13)	cd label_studio && pytest -n auto
Frontend unit tests	Jest suites for the editor, Data Manager and app	cd web && yarn test:unit

The checks are separate reusable workflows under .github/workflows/; ci.yml is the entry point that runs them together. Results appear on the commit and pull request in GitHub, and under Actions → CI. The Windows run of the Python suite is opt-in (Actions → Tests → Run workflow with windows ticked) because the server only ships on Linux.

Findings that are intentional are marked in the code with a reason (# nosec B615 - user-chosen dataset or biome-ignore lint/a11y/useMediaCaption: …), so a reviewer can see why a rule was waived rather than silently disabled.

Running it yourself

The suites and automation live in the private warakacommunity/afriannotate-testing repository.

• Automated run: GitHub → Actions → Automation → Run workflow (optionally set a base_url). It needs the repo secrets AFRI_EMAIL / AFRI_PASSWORD (a member account).
• Locally: in automation/, npm ci && npx playwright install --with-deps chromium, then AFRI_URL=… AFRI_EMAIL=… AFRI_PASSWORD=… npm test.
• A testing round: GitHub → Actions → Open a testing round.