QA Workflows
Connect annotation, consensus, benchmark gates, review, and rework with explicit owners and outcome routes.
A QA workflow connects quality controls to the work they govern. Choose the checks the task requires, assign accountable owners, and define what happens when work passes, fails, lacks comparison evidence, or needs a human decision.
Use QA Overview to select controls. Use Stages and routes for the full stage catalog and editor behavior.
See the workflow in action
The demo shows a workflow containing Annotate, Consensus, Quality Gate, Review, and Complete, with correction routes and gate configuration.
Routing requirement: connect Consensus Fail to a Review stage. Configure Quality Gate Pass, Fail, and Not evaluated separately, as shown in the example below.
Start with the smallest sufficient process
Project → Annotate → Review → Complete
Every item needs human acceptance.
Reject returns to Annotate.
Project → Consensus → Review → Complete
Independent judgments and expert resolution are required.
Disagreement reaches an eligible reviewer.
Project → Annotate → Quality Gate → Review → Complete
Approved references should inform the process.
Fail and Not evaluated each have an explicit destination.
Project → Annotate → Consensus → Quality Gate → Review → Complete
The process requires prepared labels, independent comparison, reference checks, and final review.
Disputed, failed, and unavailable results remain owned and observable.
These are example designs. Consensus itself collects human annotations; an earlier Annotate stage is optional. Existing annotations can seed the independent assignments, so decide whether seeded work fits the independence required by your quality policy.
Example: layered checks with final review
In this example, reviewers accept the final result whether it reached them through agreement, a passed reference check, or unavailable benchmark evidence. Failed benchmark work and rejected reviews return to Annotate. If your policy requires every reviewed Consensus result to pass a gate before completion, configure that additional route explicitly and pilot it; the example above does not imply it.
Build the workflow
Write the policy first. Define label quality, required checks, who can accept work, and how unbenchmarked items should be treated.
Prepare inputs. Confirm source readiness, ontology, instructions, and representative approved answer keys where needed.
Add stages. Create only stages that represent a distinct responsibility or automatic check.
Assign people. Set annotator and reviewer eligibility and verify the available self-assignment or manager controls.
Set check parameters. Configure Consensus votes and similarity, and the Quality Gate threshold. Calibrate them with actual examples.
Connect every outcome. Include the disagreement route, gate Pass/Fail/Not evaluated, and review approval/rejection paths.
Save and pilot. Inspect the impact of Save & Apply, then validate the operating scenarios below.
Validate the process before scaling
Complete, acceptable annotation
Reaches the required next stage with the right owner.
Insufficient Consensus votes
Waits for the remaining votes rather than passing early.
Consensus agreement
Publishes the representative result and follows the agreement route.
Consensus disagreement
Opens the intended review path with the submitted evidence available.
Benchmark score above threshold
Uses the configured Pass route.
Benchmark score below threshold
Uses the configured Fail route.
No usable approved key
Uses Not evaluated rather than reporting a successful benchmark check.
Rejected review
Returns to correction with actionable feedback and can be reviewed again.
Grouped work
Treats the group as one task while preserving required-panel evidence.
Inspect both the queue and the Workbench after transitions. A correct-looking graph is insufficient if work cannot be claimed, a reviewer lacks access, or a correction loop does not return to the required check.
Monitor the process
Use QA → Benchmarks to track key readiness and evaluations, QA → Consensus for agreement and disputed results, and QA → Quality checks for validation problems and open issues. Use queues and item history to identify the current owner and the sequence of attempts.
Automatic Quality Gate evaluation is not a human assignment queue. If an item appears stalled, inspect the current stage and any processing or comparison issue before assigning it to a person.
Change a running process deliberately
Review in-flight items before removing stages, changing routes, altering Consensus requirements, or replacing a project workflow. Preserve active vote and review work where the product allows it; some runtime settings are locked while Consensus items are active. Changing eligibility is different from explicitly reassigning an existing task.
Follow Change a live workflow, record the intended handling of active work, and repeat the affected pilot scenarios after applying the change.
Release the accepted result
Confirm that the intended cohort has completed all required checks and that unresolved findings have an explicit disposition. A dataset membership version and an annotation release serve different purposes: the former defines selected source data; the latter freezes reviewed project output. Verify representative exported items, ontology values, relations, and source identity before delivery.
Next steps
Last updated