Skip to main content

Annotation — Question answering (span selection)

Ticks are recorded live

Work these steps on the in-app checklist — account menu → My testing checklist — where every Pass / Fail / Skip is recorded the moment you press it, and Fail files the report for you. This page (and its PDF) is the same list for reading, printing, or working offline; if you used paper, transfer your ticks into the app afterwards.

Project fixture: fixtures/projects/17-qa-span-selection.json (20 tasks). Steps are the same shape for every data type so a tester can compare behaviour across types.

Each step: Do this, then check You should see.

Setup (manager does this once)​

  • AN-QA-01 Create a project from fixtures/projects/17-qa-span-selection.json (scripts/seed.py --only 17 or Create project → paste the label config → Import the JSON). You should see: 20 tasks in the Data Manager; the labeling interface preview renders without red warnings.

  • AN-QA-02 Publish the project and add tester3 (annotator) and tester4 (reviewer). Set annotators per task = 2. You should see: the project appears for tester3 on Home → Continue working and on the Projects page.

Label​

  • AN-QA-03 As tester3 open the project and label the first task: highlight the answer span in the passage. You should see: the region/answer appears; Submit enables; Ctrl/Cmd+Enter submits and loads the next task.

  • AN-QA-04 Label 2 more tasks, using hotkeys shown in the tooltips where possible. You should see: each submit moves on; the counter in the bottom bar advances.

  • AN-QA-05 Press Undo (Ctrl/Cmd+Z) after an action, then Redo. You should see: the exact last action reverts and re-applies.

  • AN-QA-06 Skip a task. You should see: the next task loads; the skipped one is marked skipped in the Data Manager.

  • AN-QA-07 Open a task you already submitted and change something, then Update. You should see: the change is saved; the history panel shows the update.

Regions toolbar​

  • AN-QA-08 Select a region. You should see: a floating toolbar appears next to it with Delete (destructive tint), Duplicate, Lock, Hide, and nudge arrows; each tooltip shows the hotkey.

  • AN-QA-09 Delete a region from the toolbar. You should see: it disappears and a toast offers Undo; Undo brings it back.

  • AN-QA-10 Lock a region, then try to move it. You should see: it does not move; the lock shows in the Regions panel too.

  • AN-QA-11 On a phone-sized window (or the phone app), select a region. You should see: the toolbar docks at the bottom with labels, not floating off-screen.

Review round-trip​

  • AN-QA-12 As tester4 open the review stream for this project. You should see: tester3's annotations appear one at a time with Accept / Fix & accept / Reject.

  • AN-QA-13 Click Fix & accept on one. You should see: the editor opens THAT task with THAT annotation (same text/media, regions visible); saving returns you to the stream.

  • AN-QA-14 Reject one with a comment, then as tester3 open My work → rejected. You should see: the rejection and comment are visible; 'request re-review' is offered.

Export​

  • AN-QA-15 As manager: Export → Preview. Switch the format between JSON. You should see: the record pane shows the task exactly as the file will; non-JSON formats show the file output and mark any results that will not survive.

  • AN-QA-16 Choose fields… and untick all bookkeeping fields, Apply, then Export. You should see: the downloaded file has only the chosen fields; the count in the button matches the Data Manager view.

Mobile / small screen​

  • AN-QA-17 Repeat one labeling task at phone width (or in the phone app if the project allows mobile). You should see: the interface is usable without horizontal scrolling; controls are at least finger-sized.