Skip to main content

Project insights

Project → Analytics is one page with a strip of sections: Overview, Quality, Readiness, Delivery, People and Review queue. The title row carries the period (7 days, 30 days, 90 days, 1 year, all time, or custom), which the sections with a time axis (Overview, People) use. Each section opens with its figures in a summary band, then cards in two columns (one on a phone). Read-only; nothing here changes project state. Anyone who can view the project can load the underlying data.

Every percentage is colour-coded the same way: green 80% and above, amber 60 to 79%, red under 60%.

The Quality and Readiness cards are built from stored per-task agreement scores: tasks with at least 2 annotations, scored with the metric and methodology configured on Quality settings. No overlap, no insights: the empty line in the card says to raise the overlap and recompute.

Overview​

The project's figures for the period (complete, annotations, active annotators, average lead time, average agreement, open review items), then Annotations over time, Completion and Label distribution. An installed extension may add cards here.

Quality​

Opens with the agreement figure (the chance-corrected coefficient when the method has one, else the observed mean), the observed mean and median, the scored tasks, the raters compared and the hardest task. Then the Agreement distribution (per-task observed agreement) and the cards below.

Field agreement​

Mean pairwise agreement per labeling field, each field scored with the metric that fits its control type — text similarity / WER for transcripts, IoU for boxes, span overlap for NER, exact / Jaccard for choices (or the project's configured metric, if one is locked in). Each row shows the metric used and how many annotation pairs it averaged; lowest-agreement fields sort first.

Label agreement​

How consistently annotators land on each individual label — most contentious first, top 24 shown. Agreement per label is the average fraction of annotators who selected it, over the tasks where anyone selected it (1.0 = whenever one person used the label, everyone did); support is how many tasks that was. Covers Choices and the label dimension of span / region tasks. Low bars are where your guidelines need work.

Confusion, one heatmap per field​

For single-choice fields with ≤ 12 classes: an annotator-vs-annotator confusion matrix. Cell (A, B) counts rater pairs where one picked A and the other B. The green diagonal is agreement; bright red off-diagonal cells are the class pairs annotators mix up. Multi-select fields are skipped (confusion isn't well defined there).

Agreement by annotator​

Pairwise agreement (%) between every pair of annotators, scored with the same per-control metric as Field agreement, plus a Peer avg column (each annotator's mean agreement with everyone else). "—" means two annotators never labelled the same task. A consistently low row flags a miscalibrated or drifting annotator. Shown once the project has ≥ 2 raters on overlapping tasks.

Hardest tasks​

The 15 lowest-agreement tasks, with their score and annotation count — the best targets for review or guideline clarification.

Readiness​

Is the dataset good enough to export / train on?

Dataset readiness​

One 0–1 gauge: a weighted blend of the factors below, each shown with its own bar and weight. Missing factors drop out and the weights renormalise.

FactorWeightWhat it measures
Coverage35%Labeled tasks / total tasks
Agreement35%Mean per-task agreement score
Ground-truth coverage15%Tasks with a ground-truth annotation / total tasks
Class balance15%Mean per-field balance score (below)

Class balance​

Per-field class distribution scored by normalised entropy: 1 = perfectly even split, → 0 as one class dominates. Most-imbalanced fields first; each row shows the class count and the top class's share. Computed over a sample of annotations (up to 4000); single-class fields are skipped. Heavy skew hurts model training.

Accuracy vs ground truth​

Only appears when the project has ground-truth annotations: per-class precision / recall / F1 of regular annotations against the gold answers, with support counts and the macro-F1 in the panel subtitle. Each comparison is one task × annotator × field label-set match. Ground-truth workflow: Review and quality overview.

Delivery​

Opens with complete, remaining, velocity and finish, and, when the project has a plan, Target: how many of the planned tasks have an annotation. A Plan card then shows the goal, the target with the tasks started per day, the date the target is reached at that pace and whether that is on track for the planned date. The plan is set in the new-project wizard and on Settings, General.

The PM / ops view.

Completion forecast​

Projected from the last 14 days of annotation velocity:

  • Complete — annotations done vs required (required = the sum of every task's overlap, so raising annotators-per-task raises the target)
  • Remaining — annotations still needed
  • Velocity — annotations per day over the window
  • ETA — remaining ÷ velocity, as days and a projected date; "needs more velocity" when nothing landed in the window

Review queue​

Backlog of the review queue: open items awaiting triage, age of the oldest open item, and how many items were resolved (or dismissed) in the last 7 days.

Labour​

Effort spent annotating, from per-annotation lead time: total hours, number of annotations, active annotators, and average hours per annotator.

People​

Who annotated in the period: active annotators, annotations (and the average per person), average lead time and average agreement, then the Contributors table (annotations with a share bar, lead time, agreement; a row opens the person's profile), 10 to a page. An installed extension may add cards here.

Review queue​

The flagged work, with filters for state and reason above the list. Each row shows a state mark, the task and the reason, then a quiet line with the figures, the annotator and when it was flagged; Resolve and Dismiss beside open items. See The review queue.

What's next​