Skip to main content

<VideoTiers>

Category: Object tag · AfriAnnotate-specific

The VideoTiers tag is the video analog of <AudioTextAlign>: given a video (a sign-language clip, a gesture recording, a lecture), the annotator marks time-aligned segments across one or more stacked, hierarchical tiers — the model EAF uses for sign-language corpora:

Sign gloss   ▐──HOUSE──▌  ▐──BIG──▌   ▐──RED──▌
Mouthing         ▐─house─▌     ▐─red─▌
Non-manual    ▐───brow-raise (question)───▌
Translation  ▐──────"the big red house?"──────▌

Each tier holds time-aligned segments; a child tier's segments are time-locked inside their parent segment's [start, end] interval (a mouthing can't extend past the sign it belongs to). The single-region <Video> tag marks one track of events; sign languages are inherently multi-tier (manual and non-manual channels run in parallel), so a corpus needs the tier hierarchy.

VideoTiers reuses AudioTextAlign's tier / hierarchy / time-lock engine unchanged and swaps only the media layer: a plain HTML5 <video> player with a compact, frame-accurate control bar (play/pause · ◀| frame step |▶ · scrubber · time + frame readout · 0.25×–2× speed) in place of the waveform. The saved value is structurally identical to AudioTextAlign's, so the same .eaf export / import path serves both.

Use with the following data types: video (a URL in task data, or a clip captured by <VideoRecord> on the same page). Optionally pre-computed pose keypoints.

Attributes​

AttributeTypeRequiredDefaultDescription
namestringyes—Name of the element
valuestringno—Video URL (or $var). When empty, the tag falls back to the toName tag's resolved video, then to the task-data keys video, video_url, url, recording, recording_url (first non-empty string wins).
toNamestringno—Name of the <Video> / <VideoRecord> tag this annotation references. A <VideoRecord> sibling supplies its captured clip URL; a <Video> supplies its resolved value.
transcriptstringno—Optional reference gloss / prompt text (or $var) shown above the video. In span mode the text is selectable — highlight a fragment to lift it into a segment. In token mode it is whitespace-split to pre-fill segment chips.
lexiconstringno—Numeric id of a Knowledge Base holding the ID-gloss lexicon (Organization → Resources → Knowledge bases). Segments on the lexicon tiers then autocomplete against it and store the matched entry's stable external_id beside the label, which is what lets the corpus be searched by lemma rather than by label string. The id is written to .eaf exports as EXT_REF. A gloss that is not in the lexicon may still be typed — it simply carries no id — and the suggestion list offers to add it. Empty → free text.
lexiconTiersstringno—Comma-separated tiers the lexicon applies to. Empty → every tier whose name starts with rh, lh or gloss; mouthing, non-manual and translation tiers are not drawn from a sign lexicon.
tiersstringno—Comma-separated tier hierarchy, e.g. utterance,rh:utterance,lh:utterance,mouthing:rh,brow:utterance,head:utterance,translation:utterance. Each entry is name[:parent[:derivation]]; without an explicit :parent the entry is a child of the previous one; a missing derivation defaults to subdivision for child tiers and alignable for root tiers. Duplicate names are ignored. Empty → a single implicit tier and no tier fields in the output. Tier tabs and stacked tracks only appear when two or more tiers are declared.
timelockon,offnoautoEnforce child-in-parent time containment. Auto = on when 2+ tiers are declared, off otherwise (on/true/1 and off/false/0 are accepted). When on, boundary edits on a child segment are clamped into its parent's interval (hard containment — stricter than the advisory-only audio tag), violating rows are painted red, and a warning counts them.
frameratestringno—Frames per second of the video. When set (> 0), every captured or typed boundary snaps to the nearest frame, the ◀| |▶ buttons step exactly one frame, and the time readout shows the current frame number (f123). Unset → times stay continuous and the step buttons move 0.1 s. The tag measures the clip's real frame rate as it plays and uses that in preference to this value, so an annotator sent a video by someone else does not have to know it — this attribute is the fallback for browsers without requestVideoFrameCallback, and the starting point before playback begins. The rate in use is shown beside the frame-step buttons, marked when it disagrees with this setting.
modespan,tokennospanspan — the annotator segments freely (highlight the transcript or use + row on a tier). token — chips are pre-filled from tokens / the whitespace-split transcript for the rare case where a known gloss sequence exists.
tokensstringno—$var pointing to a pre-tokenized reference list (array of strings or {text} objects, or a string split on whitespace). Only consulted in token mode; only $var form is resolved.
granularitystringnoglossFree-text granularity label stored in the result and sent as context.granularity to the auto-segmentation backend.
autoalignstringno—model_version of a registered ML backend to POST to for pre-segmentation. Empty → the Auto-segment button is hidden.
posestringno—$var pointing to pre-computed pose keypoints (an array of frames). Drawn as a canvas overlay synced to the video; the tag does not run pose estimation in the browser. Accepted frame shapes: [{t?, keypoints|landmarks|points: [[x,y,score?], …]}] or [{landmarks: [{x, y, visibility?}, …]}]; coordinates may be normalized 0–1 or pixels in the video's intrinsic space (auto-detected). A ⣿ pose toggle hides/shows the overlay.
posefpsstringno—Frames-per-second of the pose stream when frames carry no per-frame t. Falls back to framerate, then 25.
totextstringno—Accepted for config parity with AudioTextAlign; not read by VideoTiers — use transcript.

How annotators work​

  1. Find the boundary — play, scrub, change speed, or nudge one frame at a time with ◀| / |▶ until the exact handshape transition is on screen.
  2. Add a segment — in span mode, highlight a fragment of the transcript and click + Add segment, or click + row on the active tier tab to name a segment at the playhead (a new child segment auto-links to whichever parent segment contains the playhead). In token mode the chips are already there.
  3. Capture — on the segment chip press ↧ start / ↧ end to write the current (frame-snapped) time, or type seconds into the numeric inputs (step 0.001). end is clamped to >= start.
  4. Check — the chip under the playhead highlights; ▶ on a chip plays just that segment; the footer reads N of M segments timed. With 2+ tiers, the stacked tracks under the player show every tier at once so the parent/child relationship stays legible, and red bars / a warning flag time-lock violations.
  5. Clear all zeroes every segment's times (segments stay). Export .eaf downloads the saved annotation as an .eaf file (GET /api/tasks/<id>/text-align/export-eaf?annotation=<id>) — submit first, or you get "Nothing to export yet".

Result format​

One result per tag; from_name and to_name are both the VideoTiers tag itself, and type is videotiers. The result is created as soon as at least one segment has end > start.

{
  "from_name": "signtiers",
  "to_name": "signtiers",
  "type": "videotiers",
  "value": {
    "videotiers": {
      "alignments": [
        { "word": "HOUSE", "start": 0.48, "end": 1.12, "confidence": 1,
          "tier": "gloss", "rowId": "k2x9q1m7" },
        { "word": "house", "start": 0.6, "end": 1.08, "confidence": 1,
          "tier": "mouthing", "parentId": "k2x9q1m7", "rowId": "p0a3z8ru" }
      ],
      "granularity": "gloss",
      "tiers": [
        { "name": "gloss", "parent": null, "derivation": "alignable" },
        { "name": "mouthing", "parent": "gloss", "derivation": "subdivision" }
      ],
      "timelock": "on"
    }
  }
}

word carries the segment's label (gloss, mouthing token, translation fragment). tier, parentId, rowId, tiers, and timelock appear only when two or more tiers are declared. Untimed segments (start == end) are kept in alignments so chips survive a reload. The payload is the same shape as result.value.audiotextalign, so a tier editor import/export and downstream tooling written for AudioTextAlign work unchanged.

Auto-segmentation (autoalign)​

With autoalign="<model_version>" an Auto-segment button appears. On click the tag lists the project's ML backends (GET /api/ml?project=<id>), picks the one whose model_version matches (case-insensitive), and POSTs to /api/ml/<backend>/interactive-annotating with:

{ "task": { "data": { "video": "<resolved video URL>", "transcript": "<transcript text>" } },
  "context": { "granularity": "gloss" } }

The response must carry an alignments array — at alignments, result.alignments, or predictions[0].result[0].value.alignments — in the row shape above; it replaces the current segments. No matching backend → "No registered ML backend with model_version=… Add one from Settings → Models."

Examples​

Standalone multi-tier sign-language annotation​

<View>
  <VideoTiers
    name="signtiers"
    value="$video_url"
    tiers="utterance,rh:utterance,lh:utterance,mouthing:rh,brow:utterance,head:utterance,translation:utterance"
    mode="span"
  />
</View>

Record → tier-annotate on one page​

<View>
  <Text name="prompt" value="$prompt"/>
  <VideoRecord name="clip" toName="prompt"/>
  <VideoTiers name="signtiers" toName="clip" tiers="utterance,rh:utterance,lh:utterance,translation:utterance" framerate="30"/>
</View>

The tiers surface shows "No video URL on this task" until a clip has been recorded.

Pre-computed pose overlay + reference gloss​

<View>
  <VideoTiers
    name="gesture"
    value="$video"
    transcript="$reference_gloss"
    pose="$pose_frames"
    posefps="30"
    framerate="30"
    tiers="stroke,hold:stroke"
  />
</View>

The pts frame shape written by <VideoRecord skeleton="…"> sidecars is accepted directly, so recorded clips can be reviewed with their own skeleton without conversion.

Config preview

Without a task (Labeling Setup preview) the tag renders its controls and the "No video URL" note; nothing can be segmented until the config runs against task data or a sibling <VideoRecord> captures a clip.

Gotchas​

  • Hard containment. Unlike <AudioTextAlign>, a child boundary that would leave its parent's interval is clamped, not merely flagged. Set the parent segment first, then the children.
  • Token mode pre-fill happens once, only when the tag has no segments yet — editing transcript after segments exist won't add chips.
  • tokens must be a $var. A literal comma-separated list is not parsed (it is cleared).
  • One result per tag. Duplicate results for the same tag are removed on save; the newest state wins.
  • Export .eaf reads the saved annotation, not the in-progress one.
  • <AudioTextAlign> — the audio counterpart; same tier spec, same payload.
  • <VideoRecord> — captures the clip (and optional skeleton sidecar) that VideoTiers can annotate via toName.
  • <Video> + <TimelineLabels> — single-track frame classification when you don't need a hierarchy.
  • <MultiModalCanvas> — accepts VideoTiers as a child when you need one shared playhead across several time-based tags.