> For the complete documentation index, see [llms.txt](https://docs.unitlab.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.unitlab.ai/documentation/qa/qa-overview.md).

# QA Overview

Manage annotation quality through workflow stages, outcome routes, benchmark evidence, and expert review before release.

Quality assurance checks whether annotations are accurate, consistent, and complete enough for their intended use. Unitlab connects independent annotation, approved reference comparisons, human review, validation findings, and rework through the project workflow.

Use the project **QA** area to inspect quality evidence. Use **Workflows** to configure how that evidence affects the next stage. Use the **Task Queue** and Workbench to perform assigned annotation and review work.

## See QA in action

The demo opens the project QA page, expands benchmark results, and inspects a submission against its approved reference in the Workbench.

{% embed url="<https://homepage-files.s3.us-east-2.amazonaws.com/hero-videos/hero/annotation-quality-assurance-2-20260918.mp4>" %}

[Open the demo in a new tab](https://homepage-files.s3.us-east-2.amazonaws.com/hero-videos/hero/annotation-quality-assurance-2-20260918.mp4).

## QA Workflows

QA is handled through the project's **Workflows**. Stages define which annotation and quality checks an item must complete; routes define what happens after each result. Configure the process in Workflows, then use the **QA** page to inspect its agreement, benchmark, and validation evidence.

<figure><img src="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGjVLUz4wthGkGlRKM6rM%2Fuploads%2FN1OFd9SvuQyIoC49zbCM%2Fqa-workflows-annotation-quality-stages.png?alt=media&#x26;token=ae1585fb-9ba6-41da-812e-23bff713535f" alt="Illustrative QA workflow: Annotate, Consensus, Quality Gate, Review, and Complete. Green routes advance accepted work; orange routes illustrate correction loops. The text below explains the required Consensus review and Quality Gate Not evaluated routes."><figcaption></figcaption></figure>

*An example of layered quality controls: successful outcomes move work forward, while failed checks and rejected reviews lead to correction.*

1. **Annotate:** eligible annotators create or correct labels using the project ontology and instructions. Assignment settings determine who can perform the work.
2. [**Consensus**](https://docs.unitlab.ai/documentation/qa/consensus)**:** collect independent submissions and compare agreement using the configured vote and similarity requirements. Agreement advances a representative result; disagreement requires review.
3. [**Quality Gate**](https://docs.unitlab.ai/documentation/qa/quality-gate)**:** compare a submission with an approved answer key. Connect **Pass**, **Fail**, and **Not evaluated** to explicit destinations. A missing or unusable key is not a passing evaluation.
4. [**Review**](https://docs.unitlab.ai/documentation/qa/review-stages)**:** assigned reviewers inspect the source, labels, and quality evidence. Approval moves the item to the configured next stage; rejection returns it for correction with actionable feedback.
5. **Complete:** work reaches the end of its configured process after the required checks and decisions. Create a separate annotation release when the accepted output is ready for delivery.

**Read the illustration as a process overview.** In the workflow editor, connect **Consensus Fail to a Review stage**; the reviewer can then reject work back to Annotate. The illustration simplifies that correction path and omits the Quality Gate **Not evaluated** route, which must also be configured.

Choose the stages your acceptance policy requires. Consensus collects human annotations itself, so a preceding Annotate stage is optional. Use reviewer eligibility, calibrated thresholds, and explicit outcome routes to make each quality decision enforceable. Pilot passing, failing, disputed, and unavailable-comparison cases before applying the process at scale.

Open [**QA Workflows**](https://docs.unitlab.ai/documentation/qa/qa-workflows) for workflow patterns, a routing example, and pilot checks. See [**Stages and routes**](https://docs.unitlab.ai/documentation/workflows/stages-and-routes) for the stage catalog and editor controls.

## Find your way around the QA page

![Project QA Benchmarks dashboard showing approved references and answer-key readiness.](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGjVLUz4wthGkGlRKM6rM%2Fuploads%2Frxh68Kv1WOPLXNFTVd73%2Fqa-benchmarks-overview.webp?alt=media\&token=ed7acc1b-6402-4897-b22f-34944925c5af)

*The project QA page brings reference results, agreement evidence, and items needing attention into one place. Values shown are from the recorded demo.*

| View               | What it tells you                                                                               | What to do next                                                                                  |
| ------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| **Benchmarks**     | Which answer keys are active or still need approval, and how submitted work compares with them. | Author and approve keys, inspect evaluations, and investigate failed or unavailable comparisons. |
| **Consensus**      | How independent submissions agree, which items need review, and what happened to each attempt.  | Open comparisons and use the assigned review route to resolve disagreements.                     |
| **Quality checks** | Which work units have validation problems or unresolved issues.                                 | Open the affected item, inspect its context, and correct or resolve the finding.                 |

QA reporting and benchmark management depend on your project and workspace permissions. An annotator's task view does not expose hidden answer keys. If management controls are absent, have the appropriate project manager review access and configuration.

## Choose the right quality control

| Control            | Question it answers                                                        | Configure it with                                                                  |
| ------------------ | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| **Consensus**      | Do independent annotators produce sufficiently similar answers?            | Vote count, required agreement count, answer similarity, and a review route.       |
| **Quality Gate**   | Does this submission meet the threshold against an approved answer key?    | Approved benchmarks, one gate threshold, and Pass, Fail, and Not evaluated routes. |
| **Review stage**   | Does an accountable reviewer accept the work against the instructions?     | Reviewer eligibility, acceptance criteria, and Approve/Reject destinations.        |
| **Quality checks** | Are required values, validation findings, or open issues still unresolved? | Ontology rules, issue handling, and inspection of the affected items.              |

Agreement, reference similarity, and human approval are distinct evidence. A high agreement score can still reflect a shared misunderstanding. A passed benchmark only describes a comparison with its approved reference. Review quality depends on clear policy and qualified reviewers.

## Set up a QA process

1. **Define the acceptance policy.** Specify required labels, boundaries or timing rules, properties, relations, and treatment of ambiguous or empty items in Project Instructions.
2. **Calibrate representative examples.** Include ordinary cases, edge cases, and examples of incorrect work. Resolve policy disagreements before scaling assignments.
3. **Choose controls for the task.** Add Consensus for independent judgments, Quality Gate for approved references, and Review stages for human decisions as needed.
4. **Connect every outcome.** Give failed checks, unavailable comparisons, and rejected work an explicit destination and owner.
5. **Pilot the workflow.** Exercise passing, failing, disputed, and unscorable cases with the roles that will operate it.
6. **Monitor and correct.** Inspect QA findings and queue ownership, then confirm corrections before creating a release.

## Read metrics in context

* **Active benchmarks** counts approved answer keys ready for scoring; **Needs answer key** identifies preparation work.
* **Pass rate** covers completed evaluations with a Quality Gate verdict. Unavailable comparisons do not count as passes.
* **Average score** summarizes evaluated submissions. It is a similarity measure, not a universal estimate of dataset accuracy.
* **Consensus results** show vote completion, similarity, differences, and outcomes for original work items. Review decisions remain part of the evidence.
* **Quality checks** identify work needing attention, validation problems, and open issues. One item can have more than one problem, so category counts need not be additive.

Interpret percentages alongside the number and representativeness of evaluated items, the active policy, and the comparison version. Investigate changes in benchmark coverage or task composition before treating a score trend as an improvement in the whole dataset.

## Adapt checks to the modality

Review geometry and coverage for visual data; identity and timing for video or audio; spans and relation direction for text, HTML, and tabular records; and X values, interval boundaries, and channel scope for sensor data. In a Data Group, inspect every required panel while keeping the group as one work unit.

## Continue with a focused guide

* [Consensus](https://docs.unitlab.ai/documentation/qa/consensus): configure independent votes and resolve disagreements.
* [Quality Gate](https://docs.unitlab.ai/documentation/qa/quality-gate): create approved references and route comparison outcomes.
* [Review Stages](https://docs.unitlab.ai/documentation/qa/review-stages): assign reviewers, correct labels, and approve or reject work.
* [QA Workflows](https://docs.unitlab.ai/documentation/qa/qa-workflows): assemble and validate the full process.

Explore the [Unitlab annotation quality assurance overview](https://unitlab.ai/en/annotation-quality-assurance).