Quality Gate

Prepare hidden approved answer keys and route submissions through Pass, Fail, or Not evaluated outcomes.

A Quality Gate compares a submission with the item's approved benchmark answer key and routes it using one configured threshold. It has three outcomes: Pass, Fail, and Not evaluated.

Use it to evaluate work against trusted references. The answer key remains hidden from annotators and is managed separately from the working item. The term gold benchmark refers to that approved reference; marking an item as a benchmark does not by itself approve its answer key.

Understand the three outcomes

Outcome	Meaning	Typical destination
Pass	A comparable approved key exists and the score meets the gate threshold.	The next required check, Review, or Complete according to policy.
Fail	The submission was compared and its score missed the threshold.	Annotate for correction or a reviewer for investigation.
Not evaluated	A valid comparison was unavailable, such as no approved key or incompatible comparison data.	Review or another explicitly defined path for unbenchmarked work.

Not evaluated is not a pass. Keep this route separate from a successful benchmark result, even if your policy eventually sends both paths to the same stage.

Prepare approved benchmarks

1. Open the project QA → Benchmarks view with benchmark-management access.

2. Choose representative benchmark items. Include the kinds of boundaries, labels, properties, and difficult cases that your process needs to assess.

3. Open each answer key and inspect the complete source. A copied annotation is a draft, not automatically a trusted reference.

4. Have the qualified author or reviewer correct the key and complete the required annotations and properties.

5. Explicitly approve the answer key. Confirm that it is active before relying on it for gate scoring.

The key is authored on a hidden copy of the source work item. Approval freezes the reference used for comparison. Later edits to ordinary working annotations do not rewrite that approved reference. If the key itself is edited, approve it again to publish the intended update.

Benchmarks view listing active references and items that still need an answer key.

Check benchmark readiness before using reference scores as an acceptance signal.

Benchmark management does not imply automatic random insertion of hidden test items or automatic suspension of workers. Design benchmark coverage and any follow-up actions explicitly.

Configure the Quality Gate stage

1. Open Workflows and add Quality Gate after the work you want to evaluate.

2. Select the gate and set its passing threshold. Choose the value by reviewing representative correct and incorrect submissions.

3. Connect Pass, Fail, and Not evaluated to their intended stages.

4. Check that downstream annotation or review stages have eligible owners.

5. Use Save & Apply and test all three outcomes before applying the process at volume.

Quality Gate settings showing the passing threshold and benchmark controls.

The gate combines a configured threshold with explicit routes. The displayed threshold is a demo setting, not a universal quality recommendation.

The gate computes its comparison when the item enters the stage. It does not route solely from a previously displayed score or from a manually supplied confidence value.

Inspect a submission against the key

In QA → Benchmarks, expand an item to inspect its evaluated submissions and available diagnostics. Open a comparison to see the selected submission and its frozen reference in the relevant viewer.

Benchmark and submission comparison with complete video frame and annotation timeline.

Inspect the source, reference, submitted labels, and timeline together. A score must be interpreted through the actual annotation differences.

Review class mismatches, missing or extra annotations, boundary or timing differences, and property discrepancies. A diagnostic such as object overlap explains one part of the comparison; it is not necessarily the full gate score. Compare the same evaluation and key version when discussing a result.

For grouped data, the benchmark covers the group and its required panels. Missing or changed required comparison context can make the group Not evaluated. Do not infer successful coverage from the panels that happened to be available.

Operate and maintain the check

• Investigate failures before changing thresholds. Determine whether the submission, key, or instructions are wrong.

• Keep reference policy current. Update and reapprove keys when acceptance criteria change, and preserve which version produced earlier results.

• Track coverage. A good pass rate on a small or easy benchmark set does not establish quality for every item.

• Resolve unavailable comparisons. Separate missing-key preparation from processing or annotation problems.

• Verify corrections. Send reworked items through the intended check or review path again.

Troubleshooting

Situation	Recommended action
Item needs an answer key	Open the draft, finish it, and explicitly approve it.
Gate returns Not evaluated	Check key readiness, source compatibility, required group panels, and diagnostic details.
A correct-looking item fails	Inspect the exact selected submission, key version, class and property differences, and the calibrated threshold.
A key was edited after approval	Review the change and approve the new key; do not assume the active reference changed automatically.
Benchmarks show scores but no gate verdict	Confirm whether a Quality Gate was configured and the item reached it. A comparison score and a routing verdict are different records.

Next steps

• Compare independent judgments with Consensus.

• Resolve exceptions using Review Stages.

• Connect the complete process in QA Workflows.