Terms you will hear
These words appear in the checklists, the workbook and this guide. They are grouped by where you will meet them.
Things we test
| Term | Meaning |
|---|---|
| Step | One instruction plus the result it should produce. Example: "Sign up again with the same email" should give "a friendly already-registered message with a link to sign in". A step is the smallest thing we record. |
| Step ID | The short code on each step, such as OB-04 or AN-NER-07. The letters name the suite (OB is onboarding, AN-NER is annotation, named-entity recognition), the number is its position. Use the ID whenever you talk about a step; it is unambiguous where "the sign-up one" is not. |
| Suite | A group of steps about one feature, run in order. There are 39. Examples: 01-onboarding-auth (24 steps), 02-annotation-bbox (bounding boxes, 19 steps), 09-mobile (10 steps). |
| Section | A sub-heading inside a suite. Annotation suites all follow the same sections: Setup, Label, Regions toolbar, Review round-trip, Export, Mobile / small screen. The section tells you which role does that part. |
| Module | The area of the product a suite belongs to: annotation, onboarding-auth, mobile, security-privacy and so on. The Dashboard groups results by module so we can see where the risk is. |
| Tier | How urgent a step is. Smoke steps prove the basic path works and are run first. Core steps are the full functional checks. Extended steps are small-screen repeats, appearance, accessibility, translation and documentation. Run in that order. |
| Fixture / seed script | Ready-made sample projects and data that the setup steps load, for example fixtures/projects/15-named-entity-recognition.json. They mean every tester labels the same material, so results can be compared. |
Results
Every step ends in exactly one of these.
| Status | Use it when | Example |
|---|---|---|
| Not Run | Nobody has tried the step yet. The starting state of every row. | Monday morning. |
| In Progress | You have started it but stopped part-way, for example a step that needs someone else to reject your work first. | You submitted an annotation and are waiting for tester4 to review it. |
| Pass | What happened matches the expected result. Not "roughly", but as written. If the expected text is not quite what you saw but the meaning is the same, Pass and put the detail in Result Notes. | The password rule message appeared and the form did not submit. |
| Fail | What happened does not match the expected result, or the app errored or froze. Every Fail needs a defect report. | Signing up with a used email showed a stack trace instead of a friendly message. |
| Blocked | You could not even attempt the step because something earlier is broken, or something you need is missing (a build, an account, a device). Say what is blocking you in Result Notes. | You cannot test export because the project would not publish in the setup step. |
| Skipped | You chose not to do it this round, and the reason is not a bug. No time, no suitable device, or the lead told you to leave it. | The iPad split-view step, when nobody has an iPad. |
| N/A | The step does not apply to this deployment at all. Use rarely and only after asking the lead. | "Open the verification email" when this server does not send them. |
| Obsolete | The step no longer makes sense because the feature changed. Only the lead sets this. It replaces deleting a row. | A button the step names has been removed on purpose. |
Blocked is not Skipped
Blocked means "the product stopped me" and points at hidden risk. Skipped means "we decided not to". Do not use Skipped to avoid writing a bug report.
Bugs
| Term | Meaning |
|---|---|
| Defect | The formal word for a bug: one thing that is wrong with the product. One defect can make many steps fail. Ten steps failing because the project would not publish is one defect, referenced from ten steps, not ten defects. |
| Defect ID | The row number on the Defects tab: DEF-001, DEF-002 and so on. It goes back into the Defect ID column of every step it explains. |
| Severity | How bad the bug is for a user, judged by the reporter. Blocker: stops a whole suite or corrupts data. Major: a feature is unusable but there is a way round it. Minor: wrong but you can carry on. Cosmetic: looks wrong, works fine. |
| Priority | How soon it gets fixed, set by the lead at triage. P0: fix now, stops release. P1: fix this round. P2: next round. P3: backlog. Severity and priority usually agree but need not: a cosmetic typo on the sign-up page could be P1 because everyone sees it. |
| Triage | The lead's daily pass through new defects: confirm it is real, set Severity and Priority, assign it to a developer, mark duplicates. Triage is what turns a pile of reports into a plan. |
| Reproduce | To make the bug happen again by following the same steps. A developer's first question is always "can I reproduce it?", which is why Steps to Reproduce is the most important field in a report. |
| Evidence | A screenshot, screen recording or exported file that shows the failure. Put a link in the Evidence Link column. A picture saves three messages of back-and-forth. |
| Retest / verify | Running a failed step again after a developer says it is fixed. When it passes, the defect is Verified. If it still fails, the defect is Reopened. The Dashboard's "retest queue" is the list of defects waiting for this. |
| Regression | Something that used to work and now does not, usually because a fix elsewhere broke it. Round 2 exists partly to catch these. |
| Duplicate | A defect that describes the same underlying problem as an existing one. Not a failure on the reporter's part; the lead marks it Duplicate and points to the original. |
Builds and environments
| Term | Meaning |
|---|---|
| Build | One specific version of the software, identified by a code such as afri-cloud-00204-sg7 or 88270403. Two testers on different builds can see different behaviour for the same step, which is why every result names its build. |
| Environment | Where you are testing: the cloud website, the Android app, the iPhone or iPad app, the Mac desktop app or the Windows desktop app. Each has an ID on the Environments tab: ENV-WEB, ENV-AND, ENV-IOS, ENV-MAC, ENV-WIN. |
| Surface | Same idea as environment, used on the Defects tab and in the in-app report form: web, android, ios, mac, windows. |
| TestFlight / Firebase App Distribution | Apple's and Google's services for handing out pre-release apps. The iPhone build comes through TestFlight, the Android build through Firebase. You need an invitation for each; ask the lead. |
| SmartScreen | Windows' warning when you install an app that has not been signed by a registered publisher. The Windows build is unsigned, so this warning is expected once. Click More info then Run anyway. This is not a bug. |
| Offline / sync | The phone and desktop apps can work without internet and send results later. "Go offline" in a step means switch on airplane mode or unplug the network on purpose. |
Process
| Term | Meaning |
|---|---|
| Round | One complete pass through the steps against a fixed set of builds. Round 1 (R1) started on 2 September 2026. Round 2 re-runs what failed plus anything the fixes might have disturbed. |
| Entry criteria | What must be true before a round starts: builds available, accounts working, testers onboarded. Starting without them wastes everyone's first day. |
| Exit criteria | What must be true before we call the round finished. For R1: zero blockers open, at least 95% of steps executed, at least 90% of executed steps passing. Not meeting them is a decision to record, not a formality to skip. |
| Execution % | Executed steps divided by total steps. How far through the round we are. |
| Pass rate % | Passed steps divided by executed steps. How healthy the build is. Always report the two together: 100% pass on 5% executed tells you nothing. |
| Sign-off | The lead's written decision at the end of the round, on the Round Log tab, with the numbers at close and the reason for the decision. |
| Coverage | Which parts of the product have actually been exercised. A module at 0% executed is unknown territory, not a pass. |