Skip to main content

Annotating sign video

Two shipped shapes cover most sign-language annotation; both work on any video URL in task data or on a clip captured by <VideoRecord> on the same page.

Multi-tier: <VideoTiers>​

Sign languages run several channels in parallel, so a corpus needs stacked tiers, not one track. <VideoTiers> is the video analog of <AudioTextAlign> — same tier / hierarchy engine, a frame-accurate video player instead of a waveform:

<VideoTiers name="tiers" value="$video"
            tiers="utterance,rh:utterance,lh:utterance,mouthing:rh,brow:utterance,head:utterance,translation:utterance"
            lexicon="3"
            timelock="on" mode="span"/>

How it behaves (details on the tag page):

  • Tiers are declared as name[:parent]; root tiers are alignable, child tiers time-locked hard inside their parent segment — a mouthing cannot extend past the sign it belongs to (child boundaries are clamped, violations painted red and counted). Set the parent first, then the children.
  • Frame accuracy. With framerate set, every boundary snaps to a frame, the step buttons move one frame, and the readout shows the frame number — so a gloss starts on the exact handshape transition.
  • Segmenting. span mode: add a row on the active tier at the playhead, or highlight part of the reference transcript to lift it into a segment; capture start/end from the playhead or type seconds. token mode pre-fills chips from a known gloss sequence.
  • Pose overlay. pose="$var" draws pre-computed keypoints synced to the playhead. The tag never runs pose estimation; it displays what the task provides — including the sidecar a <VideoRecord skeleton="…"> capture wrote, with no conversion.
  • Auto-segment. autoalign="<model_version>" posts the video and transcript to a registered ML backend and replaces the segments with what comes back.
  • One result per tag (type: videotiers), structurally identical to audiotextalign: alignments rows with word, start, end, tier, parentId, plus the tiers declaration and timelock.

Record and annotate on one page​

<VideoTiers toName="clip"> next to <VideoRecord name="clip"> annotates the clip just captured — the tiers surface shows No video URL on this task until a take exists. Useful for a signer-annotator glossing their own production; for a two-stage corpus, keep collection and annotation as separate projects and feed exported clips in as $video.

.eaf round-trip​

<AudioTextAlign> and <VideoTiers> share one payload, so one .eaf path serves both (full reference):

  • Export — Export .eaf on the tag, or GET /api/tasks/<task_id>/text-align/export-eaf?annotation=<id>. Reads the saved annotation (submit first); every row becomes an ALIGNABLE_ANNOTATION with its own time slots, so child boundaries survive; child tiers carry their parent reference.
  • Import — POST /api/tasks/<task_id>/audio-text-align/import-eaf (multipart file, 25 MB max) parses an .eaf file into the shared tiers + alignments payload and returns it without creating or changing an annotation. Write it into result.value.videotiers to seed a task. Tier names are not checked against the tag's tiers= — declare matching tiers or the rows land in tracks the tag does not render.

Single timeline: <Video> + <TimelineLabels>​

When one track is enough — isolated signs, or a first segmentation pass — the Gloss and translate template marks segments on the video's timeline and attaches per-segment fields:

<Video name="video" value="$video" frameRate="25"/>
<TimelineLabels name="segments" toName="video">
  <Label value="Sign"/> <Label value="Fingerspelling"/> <Label value="Non-manual"/>
</TimelineLabels>
<TextArea name="gloss" toName="video" perRegion="true"/>
<TextArea name="translation" toName="video" perRegion="true"/>
<Choices name="handshape" toName="video" perRegion="true" choice="single"/>

Each timeline region gets an ID-gloss, a spoken-language translation, and one handshape from the starter inventory (19 citation forms — extend it for your language). The <Video> frameRate must match the footage for frame-true scrubbing. This produces ordinary timeline regions, not the tier payload — there is no .eaf export for it. Tag pages: <Video> · <TimelineLabels>.

Spatial tracking on video​

For hand, face or body regions over time rather than temporal segments, the video object tags apply: <VideoRectangle> (tracked bounding boxes — the Video Object Tracking template) and <VideoVector> / <VideoVectorLabels> (tracked polylines and keypoints). Neither shipped sign template uses them; pair them with a sign config when you need annotator-drawn trajectories on top of — or instead of — model-produced skeletons.

Conventions the templates assume​

  • ID-glosses in UPPERCASE (HOUSE, INDEX-1); hyphenate multi-word glosses.
  • Declare the tier hierarchy and frame rate to match your corpus and footage; the shipped values (25 fps, gloss → mouthing / non-manual, translation) are starting points.
  • Pose data is optional everywhere: drop the pose attribute if a task has none.