Built-in metrics

The following metrics are available for control tags out-of-the-box in AfriAnnotate.

Low-agreement resolution

Low agreement between annotators is a signal, not a verdict. AfriAnnotate gives you three levers to resolve it:

• Review the conflicting tasks. Route low-agreement tasks to a reviewer, who accepts one annotation as ground truth. The project's review conflict policy controls what happens on accept — the default (single_truth) keeps exactly one accepted answer and clears the siblings.
• Flag weak annotators automatically. Org auto-suspend can pause annotators whose ground-truth accuracy (measured against gold/honeypot tasks) falls below a threshold, so persistent low agreement stops propagating.
• Tune the metric. If the disagreement is a scoring artifact (near-miss spans, taxonomy depth), adjust the metric or its threshold in quality settings before treating it as a real conflict.

How the metric is applied

The Methodology column below is indicative. Only the pairwise method runs the metric you pick: for every pair of annotators it pairs result items by control tag (from_name), scores each shared control, and averages. One item per control per annotation is kept — if a control produced several regions (several spans from one Labels tag, several boxes from one RectangleLabels tag), only the last one is compared; there is no region-to-region matching. Consensus (plurality) ignores the metric registry altogether: it takes the most common answer per control over classification-shaped values (choices, labels, taxonomy paths, ratings) and averages the plurality share. The chance-corrected coefficients (Cohen's / Fleiss' / Randolph's κ, Krippendorff's α, Scott's π, Gwet's AC, weighted κ) also bypass the registry. Ground-truth annotator evaluation uses the pairwise path against the ground-truth annotation.

All control tags

Metric	Description	Tags	Methodology
Exact Match	Evaluates whether annotation results exactly match, with optional label weights	All tags	Pairwise, Consensus

Choices and taxonomy

Categorical metrics are used for categorical control tags, such as Choices and Taxonomy.

Metric	Description	Tags	Methodology
Common Subtree Matches	Evaluates common subtree matches for a taxonomy of choices. Computes IoU over the subtree of selected taxonomy nodes.	Taxonomy	Pairwise
Common Subtree Matches (Threshold)	Evaluates common subtree matches for a taxonomy of choices, returns binarized match based on threshold	Taxonomy	Pairwise, Consensus
Jaccard Similarity	Evaluates common label matches using set intersection divided by set union	Choices (multi-select) Labels Taxonomy	Pairwise
Jaccard Similarity (Threshold)	Evaluates common label matches, returns binary match based on a 0.5 threshold	Choices (multi-select) Labels Taxonomy	Pairwise, Consensus

On Taxonomy values every selected path is expanded to its ancestors before comparison, so Jaccard Similarity and Common Subtree Matches score taxonomy selections identically, and Exact Match compares the expanded sets — equivalent to requiring the same set of selected paths. Jaccard is the auto-picked default for Choices and Taxonomy.

Numeric

Metric	Description	Tags	Methodology
Numeric Difference	Evaluates how similar two numeric values are based on their absolute difference	Number Rating	Pairwise
Numeric Difference (Threshold)	Evaluates whether two numeric values match within a specified tolerance	Number Rating	Pairwise, Consensus

Rectangles

Metric	Description	Tags	Methodology
Intersection over Union	Evaluates overlap between bounding box regions, returns raw IoU score	RectangleLabels Rectangle	Pairwise
Intersection over Union (Threshold)	Evaluates overlap between bounding box regions, returns binarized match based on threshold	RectangleLabels Rectangle	Pairwise, Consensus
Bounding Box Labels Similarity	Evaluates bbox overlap with Jaccard similarity for label matching, returns raw Jaccard score	Choices*	Pairwise
Bounding Box Labels Similarity (Threshold)	Evaluates bbox overlap with Jaccard similarity for label matching, returns binary match based on threshold	Choices*	Pairwise, Consensus
Bounding Box Text Similarity	Evaluates bounding box overlap with text similarity for text matching, returns raw similarity score	TextArea*	Pairwise
Bounding Box Text Similarity (Threshold)	Evaluates bounding box overlap with text similarity for text matching, returns binary match based on threshold	TextArea*	Pairwise, Consensus

* Nested Choices or TextArea tags inside RectangleLabels/Rectangle tags

Polygons

Metric	Description	Tags	Methodology
Intersection over Union for Polygons	Evaluates overlap between polygon regions, returns raw IoU score	PolygonLabels Polygon	Pairwise
Intersection over Union for Polygons (Threshold)	Evaluates overlap between polygon regions, returns binarized match based on threshold	PolygonLabels Polygon	Pairwise, Consensus
Polygon Labels Similarity	Evaluates polygon overlap with Jaccard similarity for label matching, returns raw Jaccard score	Choices*	Pairwise
Polygon Labels Similarity (Threshold)	Evaluates polygon overlap with Jaccard similarity for label matching, returns binary match based on threshold	Choices*	Pairwise, Consensus
Polygon Text Similarity	Evaluates polygon overlap with text similarity for text matching, returns raw similarity score	TextArea*	Pairwise
Polygon Text Similarity (Threshold)	Evaluates polygon overlap with text similarity for text matching, returns binary match based on threshold	TextArea*	Pairwise, Consensus

* Nested Choices or TextArea tags inside PolygonLabels/Polygon tags

Brush

Metric	Description	Tags	Methodology
Brush Intersection over Union	Evaluates pixel overlap between brush mask regions, returns raw IoU score	BrushLabels Brush	Pairwise
Brush Intersection over Union (Threshold)	Evaluates pixel overlap between brush mask regions, returns binarized match based on threshold	BrushLabels Brush	Pairwise, Consensus

Span and segment

Metric	Description	Tags	Methodology
Span Overlap	Evaluates overlap between one-dimensional labeled regions, returns raw IoU score	Labels ParagraphLabels TimeSeriesLabels TimelineLabels	Pairwise
Span Overlap (Threshold)	Evaluates overlap between labeled regions, returns binarized match based on threshold	Labels ParagraphLabels TimeSeriesLabels TimelineLabels	Pairwise, Consensus
Span Labels Similarity	Evaluates span overlap with Jaccard similarity for label matching, returns raw Jaccard score	Choices*	Pairwise
Span Labels Similarity (Threshold)	Evaluates span overlap with Jaccard similarity, returns binary match based on threshold	Choices*	Pairwise, Consensus
Span Text Similarity	Evaluates span overlap with text edit distance, returns raw similarity score	TextArea*	Pairwise
Span Text Similarity (Threshold)	Evaluates span overlap with text similarity, returns binary match based on threshold	TextArea*	Pairwise, Consensus
Unordered Naive Comparison for Timeline Labels	Compares timeline label annotations without regard to label order	TimelineLabels	Pairwise, Consensus

* Nested Choices or TextArea tags inside Labels tags

HTML spans

Metric	Description	Tags	Methodology
Overlap over HTML Spans	Evaluates whether two given hypertext spans have points in common	HyperTextLabels	Pairwise
Overlap over HTML Spans (Threshold)	Evaluates HTML span overlap, returns binarized match based on threshold	HyperTextLabels	Pairwise, Consensus
HTML Span Labels Similarity	Evaluates HTML span overlap with Jaccard similarity for label matching, returns raw Jaccard score	Choices*	Pairwise
HTML Span Labels Similarity (Threshold)	Evaluates HTML span overlap with Jaccard similarity, returns binary match based on threshold	Choices*	Pairwise, Consensus
HTML Span Text Similarity	Evaluates HTML span overlap with text edit distance, returns raw similarity score	TextArea*	Pairwise
HTML Span Text Similarity (Threshold)	Evaluates HTML span overlap with text similarity, returns binary match based on threshold	TextArea*	Pairwise, Consensus

* Nested Choices or TextArea tags inside HyperTextLabels tags

Text

Metric	Description	Tags	Methodology
Text Similarity	Uses the edit distance algorithm to calculate how similar two text annotations are to one another	TextArea	Pairwise
Text Similarity (Threshold)	Uses the edit distance algorithm to determine if two text annotations match based on a similarity threshold	TextArea	Pairwise, Consensus
Semantic Similarity	Evaluates text similarity by comparing semantic meaning using embeddings	User-defined	Pairwise, Consensus

Video

Metric	Description	Tags	Methodology
Exact Frames Matching for Video	Evaluates video annotations by comparing exact frame matches	VideoRectangle	Pairwise
Exact Frames Matching for Video (Threshold)	Evaluates video annotations by comparing exact frame matches, returns binarized match based on threshold	VideoRectangle	Pairwise, Consensus
Video Tracking	Evaluates video tracking by comparing bounding boxes using IoU score across frames	User-defined	Pairwise, Consensus

Keypoints

Metric	Description	Tags	Methodology
Keypoint Distance	Evaluates keypoint annotations by checking if corresponding labeled keypoints are within a coordinate distance threshold	KeypointLabels Keypoint	Pairwise

Examples

Exact Match

Exact Match is the simplest agreement metric. It checks whether two annotators gave the exact same answer, and returns either a perfect score (1.0) or zero (0.0). There is no partial credit.

This is the default metric for Choices, Taxonomy, Pairwise, and DateTime tags. It is also available as an alternative metric for many other tag types.

What affects the score

Scenario	Score
Annotators give the exact same answer	1.0
Annotators give different answers	0.0
Neither annotator provides an answer	1.0
One annotator provides an answer, the other doesn't	0.0

Unlike Span Overlap or IoU, there is no partial credit with Exact Match. The annotators either agree completely or they don't.

Single-select classification

When a labeling configuration uses a single-select Choices tag (e.g., sentiment analysis), each annotator picks one option. Exact Match compares the two selections directly.

Example: A project classifies customer reviews as Positive, Negative, or Neutral.

Annotator A	Annotator B	Score
Positive	Positive	1.0
Positive	Negative	0.0
Neutral	Neutral	1.0

Multi-select classification

When a Choices tag allows multiple selections, each annotator's response is a list. Exact Match compares the two selections as sets: they must contain the same items, in any order.

Example: A project tags articles with topics: Sports, Politics, Technology, Health.

Annotator A	Annotator B	Score
[Sports, Politics]	[Sports, Politics]	1.0
[Sports, Politics]	[Politics, Sports]	1.0 (order doesn't matter)
[Sports, Politics]	[Sports]	0.0 (different selections)

note

If you want partial credit for multi-select classifications (e.g., matching 2 out of 3 selected items), use Jaccard Similarity instead of Exact Match.

Taxonomy

For Taxonomy tags, the annotator's selection is a specific path through the taxonomy tree. Exact Match requires the full path to be identical.

Example: A project uses a taxonomy to classify animals.

Annotator A	Annotator B	Score
Animals > Dogs > Labrador	Animals > Dogs > Labrador	1.0
Animals > Dogs > Labrador	Animals > Dogs > Poodle	0.0
Animals > Dogs > Labrador	Animals > Dogs	0.0

note

If you want partial credit for taxonomy paths that share a common prefix, use Common Subtree Matches (or Jaccard, which behaves the same on taxonomy values) instead.

DateTime

For DateTime tags, Exact Match compares the two date/time values. They must be identical to score 1.0.

Annotator A	Annotator B	Score
2025-03-19	2025-03-19	1.0
2025-03-19	2025-03-20	0.0

Span Overlap

Span Overlap measures how much two annotators agree on the position of a labeled span in text (or other one-dimensional data like audio segments or time series). It is available for Labels, ParagraphLabels, TimeSeriesLabels, and TimelineLabels tags. When no metric is chosen, those tags auto-pick the label-aware variant, Span Labels Similarity, described below.

What affects the score

Scenario	Effect on score
Annotators highlight exactly the same characters	1.0
Spans overlap partially	Between 0.0 and 1.0, proportional to the overlap (IoU)
Spans don't touch	0.0
Spans have different labels but identical positions	1.0 — Span Overlap is geometry only. Use Span Labels Similarity to penalise label mismatch
Only one annotator used the control at all	That control isn't shared by the pair, so it contributes no score (the pair is skipped, not zeroed)

What counts as a "span"

When annotators highlight a region of text and assign it a label, AfriAnnotate records the character positions where the highlight starts and ends, along with the label. For example, in the sentence:

Dr. Maria Chen presented her findings at the Berlin conference.

If an annotator highlights Dr. Maria Chen and labels the span as "Person", the span is recorded as characters 0 through 14.

Step 1: Calculate IoU (Intersection over Union)

For two spans, the metric calculates how much they overlap:

IoU = Intersection length / Union length

• Intersection is the region where both spans overlap.
• Union is the total region covered by either span.

Example: Consider two annotators labeling the same sentence:

Dr. Maria Chen presented her findings at the Berlin conference.

	Span	Characters	Label
Annotator A	"Dr. Maria Chen"	0–14	Person
Annotator B	"Maria Chen"	4–14	Person

• Intersection: characters 4–14 (the overlap) = 10 characters
• Union: characters 0–14 (the combined extent) = 14 characters
• IoU = 10 / 14 = 0.71

Span Overlap returns 0.71 whether or not the labels match.

Step 2: Bring labels in — Span Labels Similarity

Span Labels Similarity averages the positional IoU with the Jaccard similarity of the two label sets:

score = (IoU + Jaccard(labels_A, labels_B)) / 2

Labels	IoU	Jaccard	Score
Person vs Person	0.71	1.0	0.86
Person vs Organization	0.71	0.0	0.36

So identical positions with a different label score 0.5 rather than 0.0 — a label mismatch halves the score, it doesn't zero it.

Several spans per annotation

The scorer keeps one result item per control tag per annotation. If an annotator drew several spans with the same Labels control, only the last one is compared with the other annotator's last one — there is no region-to-region matching, and unmatched spans are not penalised. Span Overlap is therefore a faithful signal when each annotation carries one span per control (one highlight per question, one segment per clip). For multi-entity tasks — NER with many spans per document — write a custom metric that does its own matching; the result list handed to it carries every span.

How the threshold variant works

The base Span Overlap metric returns a continuous score (0.71 in the example above). The Span Overlap (Threshold) variant converts this into a simple yes-or-no match against a 0.5 threshold:

• Score 0.71 ≥ 0.5 → 1.0 (match)
• A score of 0.45 → 0.0 (no match)

Span Labels Similarity (Threshold) does the same on the label-aware score. The threshold variants are the ones to pick when you want a binary per-task match rather than a graded score.

Intersection over Union (for bounding boxes)

Intersection over Union (IoU) measures how much two annotators agree on the position and size of bounding boxes drawn on an image. It is available for RectangleLabels and Rectangle tags; when no metric is chosen, those tags auto-pick the label-aware variant, Bounding Box Labels Similarity.

What affects the score

Scenario	Effect on score
Boxes in exactly the same position and size	1.0
Boxes overlap partially	Between 0.0 and 1.0, proportional to the overlapping area
Boxes don't overlap	0.0
Boxes have different labels but identical positions	1.0 — IoU is geometry only. Use Bounding Box Labels Similarity to penalise label mismatch
Only one annotator drew with the control at all	That control isn't shared by the pair, so it contributes no score (skipped)

What counts as a bounding box

When an annotator draws a rectangle on an image and assigns it a label, AfriAnnotate records the box's position (x, y coordinates of the top-left corner) and its width and height, as percentages of the image. For example, if an annotator draws a box around a dog in a photo and labels it Dog, AfriAnnotate stores the box's coordinates along with that label.

Step 1: Calculate IoU

IoU = Intersection area / Union area

• Intersection is the area where both boxes overlap (the region covered by both).
• Union is the total area covered by either box (both boxes combined, not double-counting the overlap).

Another way to express this:

IoU = Intersection area / (Area of box A + Area of box B − Intersection area)

Example: Two annotators are labeling dogs in a photo.

Annotator A draws a box around the dog:

• Position: x=10, y=20
• Size: 100 wide × 80 tall
• Label: Dog

Annotator B draws a slightly different box around the same dog:

• Position: x=30, y=20

• Size: 100 wide × 80 tall

• Label: Dog

• Box A covers x=10 to 110, y=20 to 100

• Box B covers x=30 to 130, y=20 to 100

• Intersection: x=30 to 110, y=20 to 100 = 80 wide × 80 tall = 6,400 sq. units

• Area of Box A = 100 × 80 = 8,000 sq. units

• Area of Box B = 100 × 80 = 8,000 sq. units

• Union = 8,000 + 8,000 − 6,400 = 9,600 sq. units

• IoU = 6,400 / 9,600 = 0.67

IoU returns 0.67 whether or not the labels match.

Step 2: Bring labels in — Bounding Box Labels Similarity

Bounding Box Labels Similarity averages the IoU with the Jaccard similarity of the two label sets:

Labels	IoU	Jaccard	Score
Dog vs Dog	0.67	1.0	0.83
Dog vs Cat	0.67	0.0	0.33

Bounding Box Text Similarity does the same with a nested TextArea (e.g. OCR transcription inside the box), averaging IoU with the text edit-distance similarity.

Several boxes per annotation

As with spans, the scorer keeps one result item per control tag per annotation: if an annotator drew several boxes with the same RectangleLabels control, only the last one is compared. There is no greedy box-to-box matching and unmatched boxes are not penalised. For multi-object detection tasks, use a custom metric that matches boxes itself.

How the threshold variant works

The base IoU metric returns a continuous score (0.67 in the example above). The Intersection over Union (Threshold) variant converts this into a binary match/no-match result against a 0.5 threshold:

• Score 0.67 ≥ 0.5 → 1.0 (match)
• A score of 0.4 → 0.0 (no match)

Bounding Box Labels Similarity (Threshold) binarises the label-aware score the same way.

IoU for other region types

The IoU concept applies to more than just rectangular bounding boxes. AfriAnnotate adapts the same core idea for different annotation types:

• Intersection over Union for Polygons works the same way, but calculates the overlapping area between polygon shapes instead of rectangles. This is useful when annotators draw freeform outlines around irregular objects.
• Brush Intersection over Union compares brush masks at run-length level — how much of the painted runs overlap versus the total painted by either annotator — which is a good agreement signal for same-sized masks without a full pixel decode. This is useful for segmentation tasks where annotators paint regions rather than drawing shapes.

In all cases the base metric is geometry only, the Labels Similarity variant averages the overlap with the Jaccard similarity of the labels, and one region per control tag per annotation is compared (the last one when there are several). Polygon IoU is computed with the shoelace formula and Sutherland–Hodgman clipping; brush IoU is a coarse run-length comparison — it treats each RLE run as a span on the flat pixel index and computes span-level intersection over union, without a full bitmap decode.

Jaccard Similarity

Jaccard Similarity measures how much two annotators' selections overlap when they can each choose multiple items from a list. Unlike Exact Match, it gives partial credit when annotators agree on some selections but not all of them.

This metric is dynamically available for Choices tags with multi-select enabled.

What affects the score

Scenario	Score
Annotators select the exact same items	1.0
Annotators share some but not all selections	Between 0.0 and 1.0, based on the overlap
Annotators select completely different items	0.0
Neither annotator selects anything	1.0 (agreement that nothing applies)
One annotator selects items, the other selects nothing	0.0

The formula

Jaccard Similarity treats each annotator's selections as a set and computes:

Jaccard Similarity = number of shared selections / total number of distinct selections

Or more formally:

J(A, B) = |A ∩ B| / |A ∪ B|

• Intersection (A ∩ B) is the set of items both annotators selected.
• Union (A ∪ B) is the set of all items selected by either annotator.

A simple example

A project asks annotators to tag articles with all relevant topics from a list: Sports, Politics, Technology, Health, Science.

Annotator A selects: Sports, Politics, Technology Annotator B selects: Politics, Technology, Health

• Shared selections (intersection): Politics, Technology = 2 items
• All distinct selections (union): Sports, Politics, Technology, Health = 4 items
• Jaccard Similarity = 2 / 4 = 0.50

How it compares to Exact Match

Using the same example above:

Metric	Annotator A	Annotator B	Score
Exact Match	[Sports, Politics, Technology]	[Politics, Technology, Health]	0.0 (not identical)
Jaccard Similarity	[Sports, Politics, Technology]	[Politics, Technology, Health]	0.50 (2 out of 4 distinct items match)

This is why Jaccard Similarity is useful for multi-select tasks — it recognizes that agreeing on 2 out of 4 topics is meaningfully different from agreeing on 0.

More examples

Annotator A	Annotator B	Intersection	Union	Score
[Sports]	[Sports]	1	1	1.0
[Sports, Politics]	[Sports, Politics]	2	2	1.0
[Sports]	[Politics]	0	2	0.0
[Sports, Politics, Technology]	[Sports]	1	3	0.33
[Sports, Politics]	[Sports, Politics, Technology, Health]	2	4	0.50

Note that order does not matter. [Sports, Politics] and [Politics, Sports] produce the same score because both are treated as sets.

How the threshold variant works

The base Jaccard Similarity metric returns a continuous score (like 0.50). The Jaccard Similarity (Threshold) variant converts this into a binary match/no-match result.

For example, with a threshold of 0.5:

• Score 0.50 >= 0.5 → 1.0 (match)

With a threshold of 0.75:

• Score 0.50 < 0.75 → 0.0 (no match)

The threshold variant is required when using the consensus methodology.

Text Similarity

Text Similarity measures how closely two free-text annotations match at the surface level — that is, how similar the actual characters or words are. It is the default metric for TextArea tags.

The core idea

Text Similarity uses edit distance algorithms to calculate how many changes (insertions, deletions, substitutions) would be needed to transform one text into the other. Fewer changes means higher similarity.

It uses the Levenshtein algorithm at character level: the minimum number of single-character insertions, deletions, and substitutions needed. The distance is divided by the length of the longer text and subtracted from 1, giving a score between 0.0 and 1.0 where 1.0 means the texts are identical. There are no algorithm or granularity settings; for word-level or n-gram comparison use the Word error rate, Character error rate, chrF, or BLEU metrics from the registry instead.

Examples

Annotator A	Annotator B	Score	Why
"The cat sat on the mat"	"The cat sat on the mat"	1.0	Identical text
"The cat sat on the mat"	"The cat sat on the mat."	~0.96	One extra character (period)
"The cat sat on the mat"	"The cat sit on the mat"	~0.96	One character substitution (a→i)
"The cat sat on the mat"	"A dog lay on the rug"	~0.43	Many differences
"hello"	"world"	~0.20	Almost entirely different

Multiple text fields

When a TextArea tag allows multiple lines of text, each annotator's response is a list of strings. Text Similarity joins the lines into one string (separated by spaces) before comparing — there is no per-line alignment or averaging.

Example: Annotators are asked to transcribe three lines of text from an image.

	Line 1	Line 2	Line 3
Annotator A	"John Smith"	"123 Main St"	"New York"
Annotator B	"John Smith"	"123 Main Street"	"New York"

• Compared as: "John Smith 123 Main St New York" vs "John Smith 123 Main Street New York"
• Four extra characters against a 35-character string → ~0.89

If one annotator writes more lines than the other, the extra text simply counts as insertions. A completely empty answer on both sides is skipped (no score) rather than scored 1.0.

How the threshold variant works

The base Text Similarity metric returns a continuous score. The Text Similarity (Threshold) variant converts this into a binary match/no-match result. The default threshold for text similarity is 0.85 (compared to 0.5 for most other metrics), reflecting the expectation that free-text annotations should be fairly close to count as agreement.

Semantic Similarity

Semantic Similarity measures whether two text annotations convey the same meaning, regardless of how they are worded. Instead of comparing characters or words directly, it compares the underlying meaning using AI embeddings.

The core idea

Semantic Similarity works in three steps:

1. Convert each text to an embedding — A numerical representation (a vector) that captures the meaning of the text.
2. Compute cosine similarity between the two vectors — This measures how close the two meanings are on a scale from 0.0 to 1.0.
3. Rescale to 0–1 — the cosine is mapped onto a 0–1 score. The threshold variant then treats ≥ 0.85 as a match.

When meaning matters more than wording

The key difference from Text Similarity is that Semantic Similarity understands that different words can express the same idea.

Annotator A	Annotator B	Text Similarity	Semantic Similarity
"The cat is on the mat"	"The cat is on the mat"	1.0 (identical)	~1.0 (identical meaning)
"The cat is on the mat"	"A feline is sitting on the rug"	~0.35 (very different words)	~0.90 (very similar meaning)
"The patient has a fever"	"The patient's temperature is elevated"	~0.42 (different words)	~0.88 (same clinical meaning)
"The cat is on the mat"	"The stock market crashed today"	~0.25 (different words)	~0.10 (completely different meaning)

Text Similarity would score the second and third rows low because the actual words are quite different. Semantic Similarity recognizes that the underlying meaning is essentially the same.

Configuration

Semantic Similarity needs an embedding endpoint — Embedding API URL, optional bearer token, and timeout under Project → Settings → Quality → Agreement (contract in Annotator evaluation). If no URL is set, or the endpoint is unreachable, the metric silently falls back to character edit distance so the pair still gets a score. The 0.85 cut-off of the threshold variant is fixed; there is no per-project threshold setting.

Choosing between Text Similarity and Semantic Similarity

Consider	Text Similarity	Semantic Similarity
How it compares	Character-by-character edit distance	Meaning and intent (embeddings)
Best for	Tasks where exact wording matters	Tasks where meaning matters more than wording
Handles synonyms	No — "couch" vs "sofa" scores low	Yes — "couch" vs "sofa" scores high
Handles paraphrasing	No — rephrased text scores low	Yes — same meaning scores high
Catches typos	Yes — small edits produce high scores	Yes — typos rarely change meaning

Use Text Similarity when:

• Annotators are transcribing text (OCR, audio transcription)
• Annotators are entering structured data (names, addresses, codes)
• Exact wording is important to the task
• You want to detect and measure minor typos or formatting differences

Use Semantic Similarity when:

• Annotators are writing descriptions, summaries, or captions
• Multiple valid phrasings exist for the same answer
• You care about whether annotators understood the content the same way, not whether they used the same words
• Annotators may be working in slightly different styles or vocabularies

---

Numeric Difference

Numeric Difference measures how close two numeric values are. Unlike Exact Match, which only cares whether two numbers are identical, Numeric Difference gives partial credit when values are close but not equal. It is the default metric for Number and Rating tags.

The formula

Numeric Difference converts the absolute difference between two values into a similarity score using this formula:

score = 1 / (1 + difference)

This produces a score between 0.0 and 1.0. Identical values score 1.0, and the score decreases smoothly as the difference increases.

Examples with a 1–5 rating scale

A project asks annotators to rate the quality of customer support responses on a scale from 1 to 5.

Annotator A	Annotator B	Difference	Score	Interpretation
4	4	0	1.0	Perfect agreement
4	5	1	0.50	Close, one step apart
4	3	1	0.50	Close, one step apart
5	3	2	0.33	Moderate disagreement
5	1	4	0.20	Strong disagreement

Examples with continuous numeric values

A project asks annotators to estimate the age of a person in a photo.

Annotator A	Annotator B	Difference	Score
30	30	0	1.0
30	32	2	0.33
30	35	5	0.17
30	50	20	0.05

Notice that the score drops quickly for larger differences. A difference of 1 already halves the score, and a difference of 5 brings it down to 0.17. This makes the base metric most useful for detecting whether annotators are in close agreement.

How the threshold variant works

The threshold variant, Numeric Difference (Threshold), works differently from other threshold variants. Instead of binarizing the similarity score, it uses a maximum difference parameter (default: 1.0). If the absolute difference between two values is within this tolerance, the score is 1.0; otherwise it's 0.0.

Example: With a max difference of 1.0 on a 1–5 rating scale:

Annotator A	Annotator B	Difference	Within tolerance?	Score
4	4	0	Yes (0 <= 1.0)	1.0
4	5	1	Yes (1 <= 1.0)	1.0
4	2	2	No (2 > 1.0)	0.0

This is often more intuitive for rating scales: "Annotators agree if they're within 1 point of each other."

Taxonomy metrics

AfriAnnotate offers Common Subtree Matches for Taxonomy tags as the partial-credit alternative to Exact Match, which requires the selected paths to be identical. It recognizes that two annotators who selected nearby items in a taxonomy are in closer agreement than two who selected completely unrelated items. (Jaccard Similarity applied to a taxonomy value behaves the same way, because taxonomy paths are expanded to their ancestors before the set comparison.)

Understanding taxonomy paths

A taxonomy is a tree structure. When an annotator makes a selection, it is recorded as a path from the root to the selected node. For example, in this taxonomy:

Selecting "Labrador" produces the path: Animals > Dogs > Labrador Selecting "Siamese" produces the path: Animals > Cats > Siamese

Annotators can also select multiple paths (e.g., both "Labrador" and "Eagle").

Common Subtree Matches

Common Subtree Matches gives partial credit for shared ancestry in the taxonomy tree. Before comparing, it expands each selected path into all of its ancestor prefixes. Then it computes Jaccard Similarity over these expanded sets.

How expansion works: The path "Animals > Dogs > Labrador" expands into:

• Animals
• Animals > Dogs
• Animals > Dogs > Labrador

This means that even if two annotators selected different leaf nodes, they'll still get credit for agreeing on the parent categories.

Example: Comparing "Labrador" vs. "Poodle":

	Expanded nodes
Annotator A: Labrador	{Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}
Annotator B: Poodle	{Animals}, {Animals > Dogs}, {Animals > Dogs > Poodle}

• Shared nodes (intersection): {Animals}, {Animals > Dogs} = 2 nodes
• All distinct nodes (union): {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}, {Animals > Dogs > Poodle} = 4 nodes
• Score = 2 / 4 = 0.50

Compare this to "Labrador" vs. "Siamese":

	Expanded nodes
Annotator A: Labrador	{Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}
Annotator B: Siamese	{Animals}, {Animals > Cats}, {Animals > Cats > Siamese}

• Shared nodes: {Animals} = 1 node
• All distinct nodes: {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}, {Animals > Cats}, {Animals > Cats > Siamese} = 5 nodes
• Score = 1 / 5 = 0.20

And "Labrador" vs. "Eagle":

	Expanded nodes
Annotator A: Labrador	{Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}
Annotator B: Eagle	{Animals}, {Animals > Birds}, {Animals > Birds > Eagle}

• Shared nodes: {Animals} = 1 node
• All distinct nodes: 5 nodes
• Score = 1 / 5 = 0.20

The scores reflect how closely related the selections are in the taxonomy tree. Two dog breeds (0.50) score higher than a dog and a cat (0.20), which makes intuitive sense.

Comparing the taxonomy metrics

Using the taxonomy above, here's how the two approaches score the same pairs:

Annotator A	Annotator B	Exact Match	Common Subtree Matches (= Jaccard)
Labrador	Labrador	1.0	1.0
Labrador	Poodle	0.0	0.50
Labrador	Siamese	0.0	0.20
Labrador	Eagle	0.0	0.20
[Labrador, Eagle]	[Labrador, Sparrow]	0.0	0.67

• Exact Match is strictest — any difference at all scores 0.0.
• Common Subtree Matches recognizes that selecting two items in the same branch of the taxonomy represents partial agreement. In the last row the two annotations share four of six expanded nodes (Animals, Animals > Dogs, Animals > Dogs > Labrador, Animals > Birds).

How each number is computed, and how it was checked

Audited 2026-10-11: every number on the Quality and Insights pages was read against its definition in the literature and given a known-answer test (projects/tests/test_metric_audit.py), so what you pick for your data means what the page says. What each one is:

Chance-corrected coefficients (the headline number on the Quality page when you choose one). Computed across every overlapping task of the project at once, per control, then averaged over controls; a unit is one control on one task.

• Cohen's κ: (p_o − p_e)/(1 − p_e) with each rater's own marginals; for more than two raters, every pair's κ over the tasks both did, averaged weighted by how many tasks they shared. Checked against the textbook 2×2 example (κ = 0.40).
• Scott's π: the same with pooled marginals.
• Weighted κ (linear, quadratic): for ordered numeric categories (Rating, Number); weights 1 − d or 1 − d² over the rank distance. Checked by hand on a three-point scale.
• Fleiss' κ: P̄ is the mean per-task share of agreeing rater pairs, P̄e = Σ π_k² with π_k the mean within-task share of category k (Gwet's generalisation, which allows a different number of raters per task). Checked against Fleiss' own worked example (κ = 0.210).
• Randolph's free-marginal κ: p_e = 1/q, where q is the number of categories the control offers in the labeling config (fixed in the audit: it used to count only the categories seen in the answers, which overstates agreement when a category is never picked).
• Krippendorff's α (nominal): from the coincidence matrix, 1 − D_o/D_e, every ordered pair of values in a task counted 1/(m−1); missing raters are no trouble. Checked by hand on a four-rater table with missing values (α = 0.797). Only the nominal distance is implemented: for ordered scales use weighted κ or Gwet's AC2.
• Gwet's AC1 / AC2: p_e = Σ π_k(1 − π_k)/(q − 1) (AC1), the weighted form for AC2; robust when one category dominates. Checked by hand.

Per task these coefficients cannot be computed (one task is one item), so the per-task distribution and heat map show observed agreement for them: the share of agreeing rater pairs, with the ordinal weights for AC2 and weighted κ. The page says so.

Pairwise metrics (the registry; the score of one pair of raters on one control, averaged over pairs and controls):

• Exact match, Jaccard, subtree, numeric difference: as named.
• Intersection over Union: rectangles by area (rotation is ignored); polygons by clipping when both are convex, and on a 160×160 grid when either is concave (fixed in the audit: the clipping was wrong for a concave second polygon; the grid is within about a percent); spans by length. Only the last region of a control in each annotation is compared; there is no region-to-region matching.
• Text similarity: 1 − edit distance / length of the longer.
• Word error rate (agreement): 1 − WER, but with the longer of the two texts as the denominator so the score is the same whichever annotator is first; it is not the one-sided WER of speech evaluation (for "a b c" against "a b c d e" it gives 0.6 where one-sided WER gives 0.33). Character error rate likewise.
• chrF: character n-grams 1 to 6 with spaces kept (sacrebleu drops them), precision and recall averaged over the orders that exist, F with β = 2. BLEU: sentence BLEU, orders 1 to 4, add-one smoothing on every order, brevity penalty, averaged in both directions.
• Semantic similarity: embeddings, when a model is configured.

Readiness (Insights): a weighted mean of what is known, with weights coverage 0.35, agreement 0.35, ground-truth coverage 0.15, class balance 0.15, renormalised over the factors present. Class balance is the normalised entropy of the label counts (1 even, 0 one class). Accuracy against ground truth is precision, recall and F1 per class, with macro and micro F1, every annotator's answer against the ground-truth answer.

Metrics tab: annotations over time are counts per day or week; the average agreement is the mean of tasks' stored agreement (consensus or the chosen metric), not a chance-corrected coefficient.

Custom metrics

If none of the built-in metrics fits your data, write your own in Python. Each project has a single custom-code slot — a function named agreement that receives two full annotations and returns a 0..1 score:

def agreement(annotation_1, annotation_2, per_label=False) -> float:
    # annotation_1 / annotation_2 are dicts with:
    #   - id      — the annotation's ID
    #   - result  — its result list; each item carries
    #               from_name, to_name, type, and a tag-specific value
    r1 = annotation_1["result"]
    r2 = annotation_2["result"]
    # ... your comparison logic ...
    return some_float_between_0_and_1

Return rules:

Return	Effect
float	The pair's score; values outside [0, 1] are clamped
dict of {label: float}	The numeric values are averaged
None	Skip this pair — no score, no error

Add a custom metric

Project → Settings → Quality → Agreement → Step 1 (Metric) → Custom Python tab (see Quality settings):

1. Pick the Custom Python tab in the metric picker. A CodeMirror editor (Python highlighting) opens, pre-filled with a starter snippet.
2. Write your agreement(...) function. The Annotation info button next to the editor opens a dialog with your config's control tags, a sample annotation JSON, and the full sandbox rules.
3. Test on first scorable task — runs the code without saving against the first task that has 2+ non-cancelled annotations and shows the score and pair used, or the exact validation/runtime error.
4. Save, then Recompute all tasks to re-score the project.

The code is stored on the project (agreement_custom_code) and is used only while the project's metric is set to Custom Python. There are no separate name/description fields and no org-level shared metrics — to reuse a metric in another project, paste the code there.

Sandbox + safety

Custom code is AST-validated at save/compile time and runs with restricted builtins:

• Imports — 19 whitelisted pure-Python stdlib modules only: math, statistics, itertools, functools, collections, collections.abc, json, re, string, datetime, fractions, decimal, random, heapq, bisect, operator, textwrap, difflib, unicodedata. Nine of them (math, statistics, itertools, functools, collections, json, re, string, datetime) are pre-imported, and mean / median / stdev helpers are available at top level.
• Rejected at parse time — everything else: import os / subprocess / socket; calls to exec, eval, compile, open, getattr, globals; dunder / underscore-prefixed attribute access; with statements; async/await; class definitions; decorators.
• CPU cap — 2 seconds per pairwise comparison (POSIX SIGALRM-based; skipped on Windows).
• Failure isolation — a failing pair is skipped and logged, never fatal to a recompute. The Test-run surface reports the exact error type (sandbox / missing key / type / timeout / runtime).

If your metric needs real semantic similarity, don't fight the sandbox — pick a Semantic Similarity built-in and configure the project's embedding endpoint (Quality settings).

Quality API

Programmatic access, all under /api/projects/<id>/quality/:

Endpoint	Method	Purpose
metrics	GET	The metric catalogue — every registry entry as {key, label, description, applies_to}, whether custom code is present, and the custom pseudo-entry
schema	GET	The label-config-derived schema (controls + sample annotation JSON) shown in the UI's Annotation info dialog
recompute	POST	Re-scores every task's precomputed_agreement with the current settings. Synchronous — returns {scanned, scored, mismatch, skipped}
test	POST	Runs candidate custom code ({"code": "..."}) against the first scorable task without persisting it
embedding-health	POST	One-shot health check of a semantic-similarity embedding endpoint

Recompute when you've changed the metric or methodology, imported historical annotations, or fixed a custom metric. For very large projects prefer the management command — same logic, more visibility:

python manage.py backfill_agreement --project <id>

The agreement-metrics registry (developer reference)

The full list of built-in metrics lives in the REGISTRY dict in label_studio/projects/agreement_metrics.py. Each entry carries:

• key — stable identifier persisted on Project.agreement_metric
• label — what appears in the UI dropdown
• description — one-liner shown under the dropdown option
• applies_to — control-tag types the metric is defined for (["*"] = any). Any metric is still selectable on any project; the UI only uses this to group "recommended for this label config" above the rest, and non-matching shapes score None at runtime rather than erroring
• callable — the implementation. Built-in callables receive the value dicts of two result items; the scorer pairs items by matching from_name (keeping the last item per from_name when a control produced several) and averages the per-control scores

The current registry covers 40+ metrics across classification, spans, HTML spans, rectangles, polygons, brush masks, keypoints, video, timeline, text (edit distance, WER, CER, chrF, BLEU, semantic similarity), and the taxonomy variants documented above.

To contribute a new metric upstream, add to the registry in agreement_metrics.py and open a PR to https://github.com/warakacommunity/afriannotate.