Quality
Organization → Quality answers two questions without opening projects one by one: which projects need attention? and who has been evaluated (or paused) where? It has two inner tabs — Quality by project and Evaluations — and a row of headline chips.
Visible to org Owners and Admins (and platform staff); the tab is hidden and the route refused for other roles. The evaluation endpoints themselves also require Owner / Admin / staff.
Headline chips
Three counters on the title row (labels in the tooltip):
| Chip | Meaning |
|---|---|
| Currently paused annotators | Project memberships across the org that are paused by evaluation right now (one person paused in two projects counts twice) |
| Projects affected | Number of projects with at least one such pause |
| Evaluations | Snapshots matching the current time window (all time when the window is All time) |
Quality by project
A ranked table of every project in the org — worst mean agreement first; projects with no score sort last. Clicking a row opens that project's Analytics → Insights. Columns:
| Column | How it is computed |
|---|---|
| Agreement | Mean of the per-task agreement score over the project's scored tasks. A task is scored by the project's agreement method and metric (pairwise or consensus, see Built-in metrics) from its non-cancelled, non-ground-truth annotations; tasks with fewer than two such annotations have no score. — when nothing is scored yet. Green ≥ 80 %, amber ≥ 60 %, red below. |
| Coverage | Labeled tasks ÷ total tasks |
| Open | Review-queue items currently open for that project |
| Velocity | Non-cancelled, non-GT annotations created in the last 14 days ÷ 14, per day |
Rank numbers are global across pages; only the true worst project carries the highlight. 10 per page by default.
Evaluations
A feed of annotator-evaluation snapshots across every project in the org, newest first. A snapshot is written each time a project with annotator evaluation enabled re-scores an annotator; rows are never updated, so this is the history, not just the latest state.
Each row shows the annotator and project, the score (coloured against
the floor in force at the time), N GT, M compared (ground-truth tasks the
annotator completed and how many comparisons the score is based on), the
floor if one was set, a status pill, and when. Click a row to open the
annotator's profile.
| Pill | Meaning |
|---|---|
| Currently paused | The annotator is paused by evaluation in that project right now |
| Was paused, since re-enabled | The snapshot recorded a pause, but the membership has since been re-enabled |
| Active | No pause recorded on this snapshot |
Filters:
- Only paused snapshots — snapshots where the annotator was auto-paused at the moment of evaluation.
- Window — last 30 / 90 days (default), last year, or all time.
Export CSV downloads every snapshot matching the current filters
(disabled when there are none). Columns: created_at, project_id,
project_title, user_id, user_email, user_name, score,
gt_completed, comparisons, min_score, min_gt, suspended. The
export streams, so large orgs are fine.
How the score is computed
For every task in the project that has a ground-truth annotation and an
annotation by the annotator, their latest non-cancelled answer is compared
with the ground truth; the score is matches ÷ comparisons. min_score and
min_gt on a row are the project's thresholds at that time, so a row
tells you whether the person was passing by the rules then in force even
if the settings have changed since. Full rules, including the calibration
ramp and when a pause trips, are in
Annotator evaluation.
Gotchas
- No evaluations? The feed is empty until a project turns on Evaluate annotators against ground truth in its Quality settings and has ground-truth annotations; every new submission is scored from then on.
- Agreement
—is not a bad score — it means no task in the project has two or more comparable annotations, or the control types return no score. - The paused chips are live, the feed is history. A person can show as Currently paused with an older row that reads Active.
- Auto-suspend rules are separate from evaluation pauses — an auto-suspend suspension is an org-membership action and does not appear here.
What's next
- Annotator evaluation → — the scoring rules and the pause
- Built-in metrics → — the agreement metrics behind the per-project score
- Metrics dashboard → — volume and velocity across the org
- Auto-suspend rules → — acting on quality signals automatically