Skip to main content

Review and quality

Ticks are recorded live

Work these steps on the in-app checklist — account menu → My testing checklist — where every Pass / Fail / Skip is recorded the moment you press it, and Fail files the report for you. This page (and its PDF) is the same list for reading, printing, or working offline; if you used paper, transfer your ticks into the app afterwards.

Review stream, verdicts, disputes, agreement, annotator evaluation, flags. Needs a project with 2+ annotators' work (use the NER or BBOX fixture after the annotation suite).

Each step: Do this, then check You should see.

A. Review stream​

  • RQ-01 As tester4 open Review on a project where tester3 and tester3b both annotated. You should see: tasks with annotations needing review, newest first or per the ordering setting; a progress count.

  • RQ-02 Accept, then open the Reviewed list. You should see: the item is there with your name and time; the annotator sees 'Accepted' on My work.

  • RQ-03 Reject with a required comment (turn on 'require comment' in Settings → Review first). You should see: you cannot reject without a comment; the annotator sees the comment.

  • RQ-04 Fix & accept: change a region/answer and save. You should see: the annotation is marked fixed, the original is kept in history, the annotator sees 'Fixed and accepted' with the changes highlighted.

  • RQ-05 Settings → Review → turn the review workflow OFF, then reopen the project. You should see: no Review button on the dashboard; the review stream says review is switched off; turning it back on restores both.

  • RQ-06 Settings → Review → 'show only finished tasks' ON, then review. You should see: only tasks with the full number of annotations appear.

B. Disputes​

  • RQ-07 As tester3, on a rejected annotation click 'request re-review' with a note. You should see: the status becomes disputed; tester4 sees it first in the stream.

  • RQ-08 As tester4 resolve the dispute. You should see: the annotator is notified; the dispute disappears from the queue.

C. Agreement​

  • RQ-09 Settings → Quality → agreement method 'pairwise' with the default metric; have two annotators label the same 5 tasks (two spans each on NER). You should see: the Data Manager agreement column shows a score per task; multi-span tasks score sensibly (not only the last span).

  • RQ-10 Switch the method to 'consensus'. You should see: scores update; spans/boxes still get a score (not only choices).

  • RQ-11 Set a low-agreement strategy to 'review queue' with a threshold above the scores. You should see: those tasks appear in the review queue as 'low agreement' and stay marked labelled (they do NOT reopen for annotators).

D. Annotator evaluation​

  • RQ-12 Turn on annotator evaluation with 3 ground-truth tasks and a 70% floor; label the GT tasks badly as tester3b. You should see: tester3b is paused in the project with a notice; the manager sees it on Members and the dashboard 'Needs attention'.

  • RQ-13 Turn on 'Count reviewer verdicts too', reject two of tester3's annotations. You should see: tester3's evaluation score drops accordingly on their profile Analytics → Quality.

E. Flags and escalation​

  • RQ-14 On an audio-QC project, submit a clip that fails a hard rule (e.g. silence). You should see: it is flagged; with 'route to review' on, it appears in the review queue with the reason.

  • RQ-15 Turn on 'QC priority in review'. You should see: flagged items come first in the review stream and are never sampled out.