Review Stages

Configure reviewer ownership, inspect work in context, approve accepted labels, and return rejected items for correction.

A Review stage assigns responsibility for a human quality decision. Reviewers inspect the original source, labels, properties, relations, and feedback, then approve the work or reject it through the configured route.

Use one Review stage for a single acceptance decision or several named Review stages when distinct specialists must check different requirements. A stage named Expert Review or Final Review uses the same Review stage type with its own eligible members and routes.

Cyclist annotation in a review card with explicit Reject and Approve controls.

Reviewers make an explicit Approve or Reject decision while inspecting the annotation in context.

Before you begin

• Publish clear acceptance criteria and examples in Project Instructions.

• Identify eligible reviewers and the expertise needed for the modality.

• Define the correction owner and the destination of rejected work.

• Decide whether another independent reviewer is required before completion.

Configure the stage

1. Open the project Workflows editor and add a Review stage.

2. Give it a meaningful name when it represents a specialist or final acceptance responsibility.

3. Configure eligible reviewers and the assignment options available for that stage.

4. Connect its approval route to the next required stage and its rejection route to annotation or the intended correction stage.

5. Apply the workflow and verify the routes using the actual reviewer and annotator roles.

A stage name alone does not enforce reviewer independence. Confirm eligibility and assignment rules when the same person must not perform both the annotation and its acceptance review.

Review one work item

1. Open assigned work. Enter through the queue or project route that resolves to your active review assignment.

2. Inspect the complete source. Cover the relevant frames, pages, channels, fields, or required panels rather than only the currently visible region.

3. Check the annotation policy. Review label coverage, class choice, geometry or timing, required properties, and relations.

4. Correct or explain. Make permitted corrections directly, or leave specific feedback that identifies what needs to change.

5. Resolve the decision. Approve when the work meets the stage criteria. Reject when corrective work is needed, using the configured return route.

6. Verify the handoff. Confirm the next stage and owner. Saving edits alone does not move the work through the workflow.

Make feedback actionable

Anchor comments or issues to the relevant label and describe the expected correction. For example: “Extend the Vehicle box to include the rear wheel in frames 14–18” gives the annotator a concrete task. A broad comment such as “bad annotation” does not.

An open issue anchored to a cyclist bounding box with a request to tighten the annotation and a Resolve action.

An issue identifies the affected object and the specific correction, with resolution tracked beside the source.

Distinguish discussion from completion: resolving an issue does not itself approve the item or move it to Complete. Likewise, approval of one review stage does not imply that later required stages have finished.

Review Consensus disagreements

When Review receives a disputed Consensus result, use its Review and Refine workspace. Submitted proposals remain comparison evidence while the reviewer prepares the accepted result.

Explicitly approve the submitted objects to retain, including objects you edited. Review and approve Item Properties, check selected tags, and confirm that relations connect retained objects. Frozen original votes remain unchanged. See Consensus for the complete procedure.

Design rework and escalation

Decision	Route and ownership
Meets the acceptance criteria	Follow the approval route to the next check or Complete.
Specific, correctable defect	Reject to the assigned annotation or rework stage with actionable feedback.
Requires specialist judgment	Use a configured specialist Review stage or the authorized manager handoff.
Instructions or ontology are ambiguous	Escalate to the policy owner, resolve the rule, and inspect other affected items.
Source cannot support the task	Apply the project's invalid-source policy and configured set-aside route.

Rework is commonly another visit to an Annotate stage, or an Annotate stage named for corrections. Escalation is a configured operating path; do not assume a universal Escalate button exists in every Workbench.

Review across modalities

For images, inspect boundaries and missed instances. For video and audio, check identity and timing over the full relevant sequence. For text, HTML, and tabular records, check exact spans and relation endpoints. For sensor recordings, verify X values, intervals, and channel scope. For medical and pathology tasks, assign the required domain expertise and inspect the relevant slices or regions. In grouped work, inspect all required panels before deciding on the group.

Next steps

• Configure stage connections in Stages and routes.

• Check assignment behavior in Assignment and stage actions.

• Build a repeatable correction cycle using QA Workflows.