> For the complete documentation index, see [llms.txt](https://docs.unitlab.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.unitlab.ai/documentation/qa/consensus.md).

# Consensus

Collect independent annotations, configure agreement requirements, inspect differences, and resolve disputed results.

Consensus collects independent annotations of the same work item and compares the submitted results. It moves an agreed representative result forward when the configured requirement is met and sends disagreement to review.

Use Consensus when a task is ambiguous, when calibrating a new policy or team, or when independent judgments are part of the acceptance process. Use [Quality Gate](https://docs.unitlab.ai/documentation/qa/quality-gate) when the requirement is comparison with an approved answer key.

![Annotator 1 and Annotator 2 show matching cyclist boxes; Annotator 3 has a narrower box missing part of the rear wheel, marked mismatch with an orange issue callout.](https://content.gitbook.com/content/GjVLUz4wthGkGlRKM6rM/blobs/AP07SHmbwCU9xSrgtzII/6aaac92e99d64c6ea603a7c0_unitlab%20qa%20consensus%20user%20approved.png)

*Independent annotations can agree on an object class while differing in boundary coverage. This illustration shows the difference a reviewer needs to inspect.*

## Before you begin

* Define the unit of work and acceptance rules in the project instructions.
* Ensure enough eligible annotators are available to supply the required independent votes.
* Assign reviewers who can resolve differences against the same ontology and policy.
* Plan destinations for both **Pass** (agreement) and **Fail** (needs review) before applying the workflow.

## Configure a Consensus stage

1. Open the project's **Workflows** editor and add **Consensus** from **Add Stages**.
2. Connect the upstream stage to Consensus, then connect **Pass** to the intended next stage and **Fail** to a **Review** stage.
3. Select the node and configure its eligible members and assignment behavior.
4. Set **Required votes**, **Must agree**, and **Answer similarity** in the stage settings.
5. Use **Save & Apply**, inspect any impact information, and pilot the stage with representative items.

| Setting               | Meaning                                                                                                                       | Example configuration          |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| **Required votes**    | Number of independent annotators, from 2 to 10. Each provides one vote.                                                       | 3 votes.                       |
| **Must agree**        | Number of submitted answers that must support the representative. It cannot exceed Required votes. The default is a majority. | 2 of 3 answers.                |
| **Answer similarity** | Minimum similarity for an answer to support the representative. Higher values demand closer agreement.                        | 80%, chosen after calibration. |

The example values illustrate configuration, not a recommended threshold for every task. Calibrate using real examples and the annotation differences that matter to the downstream use.

## What happens during voting

Each annotator works in an isolated assignment. Submitting a vote freezes that submission for comparison. For a Data Group, the vote must include the required panels.

Consensus waits for **all** required votes. It then selects a representative submitted answer using the similarities between the submissions and checks how many answers meet the similarity requirement against it. **Must agree** is not an early-exit rule: a 2-of-3 configuration still waits for the third vote.

When the requirement is met, Unitlab publishes the representative result and follows the configured agreement route. When agreement is insufficient, the **Fail** route sends the item to Review. Do not interpret the representative as a newly synthesized answer or assume every matching annotation was independently proven correct.

## Inspect results in project QA

Open **QA → Consensus** to review the latest attempt for each original item or group. Check completed versus required votes, available similarity, difference categories, and the outcome. Use stage, folder, type, date, or outcome filters where available to focus the investigation.

Open the comparison to inspect frozen submissions against the original source. Looking at report evidence does not itself claim an assignment or approve a result. When action is required, enter the assigned review work from its queue or review route.

Differences can include missing or extra objects, class disagreement, geometry differences, and properties. Read the actual labels alongside the score; a percentage does not explain which correction is needed.

## Resolve disagreement in Review and Refine

1. Open the assigned Consensus review and inspect the submissions in source context.
2. Compare each disputed label with the instructions. Correct geometry, text spans, timing, classes, or properties as needed.
3. Explicitly approve the submitted objects that should be retained. Editing a submitted object does not automatically approve it.
4. Confirm Item Properties and approve the intended values. Check the selected tags and relation endpoints as part of the final result.
5. Review the complete draft and use the available approval action to publish the reviewed result, or reject it through the configured correction path.

Reviewer changes affect the review result. Frozen original votes remain available as evidence. Newly drawn reviewer objects are handled separately from imported submitted objects, so inspect the final approved set rather than assuming every visible proposal will be published.

## Handle stalled or unexpected results

| Situation                         | What to inspect                                                                                           |
| --------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Waiting for votes                 | Required vote count, completed assignments, member eligibility, and unclaimed work.                       |
| Low agreement                     | Source ambiguity, unclear policy, missing labels, class definitions, boundaries, and required properties. |
| Comparison unavailable or errored | The reported issue and source availability. Do not treat unavailable evidence as agreement.               |
| Reviewer cannot edit              | The active review assignment, stage, permissions, and whether the view is an inspection-only comparison.  |
| Settings cannot be changed        | Active Consensus work can lock runtime settings. Assess in-flight work before changing the graph.         |

Changing eligibility affects future claims; it does not silently replace existing assignments or erase submitted votes. Treat reassignment and workflow changes as explicit operating decisions.

## Next steps

* Add reference-based checks with [Quality Gate](https://docs.unitlab.ai/documentation/qa/quality-gate).
* Define human acceptance in [Review Stages](https://docs.unitlab.ai/documentation/qa/review-stages).
* Validate both routes using [QA Workflows](https://docs.unitlab.ai/documentation/qa/qa-workflows).