> For the complete documentation index, see [llms.txt](https://docs.unitlab.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.unitlab.ai/api-sdk-cli/mcp/workflows.md).

# MCP workflows and automation

Use Unitlab MCP for folder uploads, multimodal annotation, AI labeling, model workflows, review, quality checks, and dataset exports.

This guide shows how Unitlab tools fit together from data preparation to a versioned export. Use the [quickstart](https://docs.unitlab.ai/api-sdk-cli/mcp/quickstart) for a first image-labeling example and the [tool reference](https://docs.unitlab.ai/api-sdk-cli/mcp/tools) for the complete catalog.

At the start of a session, the assistant should call `get_context`, find the project with `list_projects`, and inspect it with `get_project`. This establishes the connected workspace, permissions, classes, data families, and active workflow.

## Upload an image folder and label it

A complete folder workflow has four parts: create the project, define the classes, transfer the files, and label the ready items.

Example prompt:

> Create a Road scenes project from my image folder. Add bounding-box classes for Car, Person, and Bicycle. Upload all images, detect these classes across the dataset, and leave the results for review. Report failed uploads and images that need attention.

The assistant coordinates this sequence:

1. **Check access.** Read `get_context`. Project setup, classes, uploads, and annotation each require the appropriate permissions and write access.
2. **Create or identify the project.** Use `list_projects` first and `create_project` if a new project is intended. Use a fresh `idempotency_key` for the creation.
3. **Define the ontology.** Add classes with `create_ontology` and read their integer values with `get_ontology`. Add properties or required fields with `update_ontology` when needed.
4. **Inspect the workflow.** Read `get_workflow` before uploading. Items that reach an existing Model stage start a model run automatically.
5. **Prepare and transfer the files.** Request the upload destination with `prepare_upload`, including the batch's `file_count`. Send each file according to the returned transfer mode.
6. **Confirm readiness.** Read `get_upload_status` and `list_items`. Resolve failures and wait for processing before labeling.
7. **Create labels.** Read the item schema and current annotation revision. Call the appropriate AI or annotation tool for each item and class.
8. **Review.** Inspect saved results and validation problems before advancing tasks.

The assistant must be able to access the actual files and send their bytes. The MCP upload tools prepare and finalize the transfer; they do not read a folder from your computer by themselves. If your client cannot transfer files, upload through Unitlab or reuse library assets.

For per-item AI detection, `detect_objects` processes one requested class on one item or frame. The assistant repeats the operation across the dataset. For a model that runs automatically as tasks arrive, use a [Model stage](#batch-auto-labeling-with-a-model-stage).

## Uploads and reusable data

### Choose the destination

* Upload to a **project** when files should become annotation items immediately.
* Upload to a **library folder** when files will be organized, reused across projects, or combined into Data Groups.
* Reuse existing assets with `list_assets` and `attach_data` instead of uploading another copy unnecessarily.

Inspect the project workflow before attaching library data as well. Entering a Model stage can start runs and consume the applicable model quota.

For the web-app upload experience, see [Data upload](https://docs.unitlab.ai/documentation/data/data-upload). For organization and reuse, see [Data curation](https://docs.unitlab.ai/documentation/data/data-curation) and [Dataset versions](https://docs.unitlab.ai/documentation/datasets/dataset-versions-and-history).

### Transfer modes

`prepare_upload` returns the mode and the information required to transfer bytes:

| Mode        | Transfer                                                                                                                                           | Completion                                                                                                                                                    |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `form`      | Send one `multipart/form-data` POST per file to `upload_url`, using the `file` field and all returned `extra_fields`.                              | Ordinary image uploads do not require an explicit completion call. Medical form uploads into a project require completion after the session's files are sent. |
| `multipart` | Split the file into `part_size` chunks and PUT each part to its URL. Keep each response's `ETag`. Request additional part URL batches when needed. | Call `complete_upload` with the upload token and every part number and ETag.                                                                                  |
| `folder`    | For a LiDAR scene, PUT its files to the returned path-specific URLs. Request additional URLs as needed.                                            | Call `complete_upload` after every scene file has been sent.                                                                                                  |

Provide the real filename and extension. Use the exact `size_bytes` for large files and where required, including geospatial/pathology files, TIFFs, and LiDAR scenes. The server selects the applicable mode and validates size constraints.

Use the returned URLs exactly and within their expiry time. They are temporary credentials, so do not include them in shared documentation or logs.

### Wait for processing

Poll `get_upload_status` using the returned item, asset, or upload-session identifier. An accepted transfer is not always a ready annotation item.

* Images and some simple file types can be ready immediately after a form upload.
* Video, medical volumes, HTML, tabular data, LiDAR scenes, and some multipart uploads need background processing.
* Geospatial and pathology imagery also needs its tiled processing state to become `ready`.
* Large raster or slide processing can take substantially longer than a small image batch.

If a CSV needs configuration, complete that step before annotating it. A CSV may represent a time series or tabular rows; specify the intended data type where the extension is ambiguous.

Ask `get_help` for `upload-files` or `upload-a-lidar-scene` for the current format, structure, and completion rules.

### Library organization and Data Groups

Library tools can organize files into folders, collections, and datasets. A dataset can be published into a version and attached to a project. Cloud-linked folders can register existing storage objects by reference in the library. `start_cloud_import` imports selected storage object paths directly into a project by reference; the bytes remain in the connected storage and project processing starts in the background.

Use `group_assets` when multiple related files should form one annotation task, such as several views of the same subject. The assistant can then inspect the group with `get_group` and annotate its individual items using their own schemas.

Detaching data, archiving it, trashing library assets, and permanently deleting it have different effects. Read the action's tool guidance and impact before applying it. Clearing annotations during detach is permanent, and restoring a library asset does not necessarily restore a deleted project copy.

## Multimodal data and coordinates

Use `get_annotation_schema` for the exact item before writing labels. The modality links below explain the corresponding Unitlab annotation interface. It describes object types, usable classes, coordinate units, properties, and examples. `get_help` accepts the data-family key for a detailed guide.

| Data family                                                                                    | Guide key    | How the assistant reads and annotates it                                                                              |
| ---------------------------------------------------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------- |
| [Images](https://docs.unitlab.ai/documentation/annotations/image-annotation)                   | `img`        | Read the image or a region; use the item's supported spatial shapes and properties.                                   |
| [Video](https://docs.unitlab.ai/documentation/annotations/video-annotation)                    | `video`      | Read selected frames; create tracks and keyframes using zero-based frame indices.                                     |
| [Medical](https://docs.unitlab.ai/documentation/annotations/medical-annotation)                | `medical`    | Read slices and volume context; use supported planes, tracks, and slice indices.                                      |
| [Geospatial](https://docs.unitlab.ai/documentation/annotations/geospatial-annotation)          | `geospatial` | Read an overview or a focused region; spatial annotations use level-0 pixels.                                         |
| [Pathology](https://docs.unitlab.ai/documentation/annotations/pathology-annotation)            | `pathology`  | Read slide regions at the required detail; annotations use level-0 pixels.                                            |
| [PDF documents](https://docs.unitlab.ai/documentation/annotations/document-and-pdf-annotation) | `document`   | Read a page or its embedded text; page indices start at one and spatial units come from the schema.                   |
| [Text](https://docs.unitlab.ai/documentation/annotations/text-annotation)                      | `text`       | Read character windows and annotate supported spans and properties.                                                   |
| [HTML](https://docs.unitlab.ai/documentation/annotations/html-annotation)                      | `html`       | Read content and supported states; use the schema's text and structural context.                                      |
| [Tabular](https://docs.unitlab.ai/documentation/annotations/tabular-annotation)                | `tabular`    | Read selected columns and cell windows; annotate supported row/cell content.                                          |
| [Audio](https://docs.unitlab.ai/documentation/annotations/audio-annotation)                    | `audio`      | Read an audio clip and annotate events or transcripts in the schema's time units.                                     |
| [Time series](https://docs.unitlab.ai/documentation/annotations/sensor-annotation)             | `timeseries` | Read channels and time windows; annotate supported intervals or instants.                                             |
| [LiDAR](https://docs.unitlab.ai/documentation/annotations/lidar-annotation)                    | `pointcloud` | Read top-down point-cloud views or camera images; annotate 3D objects, tracks, linked camera boxes, and point labels. |

### Convert preview coordinates

Visual previews may be resized or cropped. `read_content` returns `origin` and `scale` so the assistant can convert a point in the preview to the original annotation coordinates:

```
original_x = origin.x + preview_x / scale.x
original_y = origin.y + preview_y / scale.y
```

Do not save preview pixels directly as original coordinates. For geospatial and pathology, the original space is level-0 pixels. LiDAR coordinates are metres with z up; the top-down view can have a negative y scale.

Read content in windows when necessary. A response can include up to eight video frames or medical slices, four LiDAR top views, 20,000 text characters, or a 60-second audio clip. Follow continuation information and pagination.

For grouped files and shared context, see [Multimodal annotation](https://docs.unitlab.ai/documentation/multimodal-annotations/multimodal-overview).

## Save and validate annotations

The normal editing sequence is:

```
get_instructions + get_ontology
→ read_content + get_annotation_schema
→ get_annotations
→ validate_annotations
→ save_annotations
→ read_content with annotations shown
```

`get_annotations` returns a `revision`. Pass that value as `expected_revision` for the next write, or `null` when no saved annotation version exists. If another annotator changes the item first, Unitlab refuses the stale revision. Read the latest state and reconcile the change before retrying.

`save_annotations` takes selected objects to add or change and selected IDs to delete. Objects omitted from the request are retained. Use the class, property, and option values returned by the ontology, rather than inventing numeric values.

`validate_annotations` checks the proposed save without storing it. A successful save can still return `problems`, such as missing required properties. Inspect and resolve those problems before treating an item as complete.

For task work, claim an eligible task before editing so another annotator does not take it while the assistant reads. `get_next_task` finds work but does not claim it.

Project classes and properties are explained in [Classes and annotation types](https://docs.unitlab.ai/documentation/ontologies/classes-and-annotation-types).

## AI labeling on individual items

AI tools use the item's ontology and current annotation revision. Most can save results directly, or return a preview with `save=false`. A preview can still require write access and consume quota.

| Tool                   | Use                                                               | Important limits                                                                                                    |
| ---------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `segment_object`       | Outline one object from a box, positive/negative clicks, or both. | Works on images, video, axial medical slices, geospatial, and pathology.                                            |
| `detect_objects`       | Find instances of one named class.                                | Default threshold 0.5; default maximum 100 instances, configurable up to 500. Video/medical needs a frame or slice. |
| `find_similar_objects` | Find objects resembling one existing object or boxed example.     | Searches the same item or frame. Cross-item references use a workflow Model stage.                                  |
| `refine_objects`       | Improve existing polygons or masks.                               | Geometric mode smooths outlines. Smart mode re-fits supported imagery and may return a job to poll.                 |
| `track_object`         | Follow an existing supported object through frames or slices.     | Always saves. Up to 100 frames per call; continuation information is returned when more work can continue.          |

Segmentation, detection, and Find Similar do not operate on PDF, text, audio, HTML, tabular, time-series, or point-cloud items. Geometric refinement also supports PDF geometry. Smart refinement supports images, video, geospatial, and pathology, while medical smart refinement is available through the editor rather than this tool.

For large geospatial or pathology imagery, whole-image per-item detection uses a downscaled overview. Use focused regions to inspect detail. Do not assume a single call processes every tiny object in a gigapixel image. The workflow model's processing behavior is separate from this per-item tool.

AI results should be checked. Use `read_content` with annotations displayed, remove false detections, add missing objects, and fill required properties. Detecting the requested classes does not establish that every object in the image was found.

For the corresponding web tools, see [Prompt labeling](https://docs.unitlab.ai/documentation/auto-labeling/prompt-labeling), [Find Similar](https://docs.unitlab.ai/documentation/auto-labeling/find-similar), and [Detect Anything](https://docs.unitlab.ai/documentation/auto-labeling/detect-anything-sam-1-sam-3).

### Tracking

Start from a saved keyframe on the object. `track_object` supports the appropriate video and medical tracks, with frame counts and direction. Medical masks can be tracked on the axial plane; supported boxes and polygons can use other available planes.

A call is bounded by its frame limit and processing window. When `continue_from` is returned, use the new revision and continuation values for the next call. If the tracker loses the object, inspect the affected frames and add or correct keyframes.

For LiDAR, `track_object` follows an existing cuboid through a calibrated camera that can see it. This is camera-assisted cuboid tracking, not text-prompted point-cloud detection. Review frames reported as absent.

### LiDAR point labeling

Read the scene schema and camera list first. Use `save_annotations` for supported 3D geometry. Create a painted-object record, then use `paint_points` with a supported region such as a cuboid, sphere, ground polygon, or existing object.

Point painting operates on selected frames, with a maximum of 20 per call. Each point belongs to at most one painted object; other objects' assignments are protected unless overwrite is explicitly selected.

## Batch auto-labeling with a Model stage

A workflow **Model stage** runs its model automatically when a task enters that stage. This provides a dataset pipeline for pre-labeling before human annotation or review, including newly uploaded or attached data.

Example prompt:

> Set up a model stage to pre-label cars and people before annotation in Road scenes. Show the model options and preview how many existing tasks would start a run before applying the workflow.

The assistant should:

1. Call `list_models` for the data type and inspect `available`, `needs_prompts`, `needs_references`, and the model's classes.
2. Read `get_ontology` to identify target class values.
3. Read `get_workflow` with stage-type information.
4. Build a complete graph that connects the data stage, Model stage, and the intended annotation or review stages.
5. Call `configure_workflow` with `dry_run=true`.
6. Inspect `impact`, `ontology_changes`, and `model_runs`, including the number of existing tasks that would run.
7. Apply the intended graph and follow each task with `get_model_run`.

Applying the workflow makes it live immediately. Tasks moved into the Model stage can start at once. New files uploaded later also run when their tasks enter that stage.

The web workflow is documented in [Model stages](https://docs.unitlab.ai/documentation/workflows/model-stages) and [Batch auto-labeling with prompts](https://docs.unitlab.ai/documentation/auto-labeling/batch-auto-labeling-with-prompts).

### Model class mapping

| Model type        | Configuration                                                                  |
| ----------------- | ------------------------------------------------------------------------------ |
| Fixed class list  | Map `model_class_value` from the model to `target_class_value` in the project. |
| Text-prompt model | Map a `text_prompt` to a project `target_class_value`.                         |
| Reference model   | Publish example objects for the stage; do not send class-mapping rows.         |

Use the model's reported capabilities and the current tool schema. Prompt limits and supported shapes depend on the model. Geospatial model settings differ from other imagery, so do not copy unsupported options between model types.

### Reference-based batch labeling

For a model marked `needs_references`, choose representative items from the project and create reference objects on their hidden example copies. The source item's annotations are preserved.

The sequence is:

```
get_references
→ save_reference_examples: draw, approve, publish
→ get_references: verify published state
→ let tasks enter the Model stage
```

An object must be drawn, approved, and published before the stage uses it. At most eight approved objects are kept per example, and a run uses up to eight published examples per class and eight classes. A `needs_update` example continues using its previously published references until the revised examples are published.

Save the stage before adding references, then publish references before sending tasks through it. A task that reaches a reference model with no published examples can fail without producing labels. After correcting the examples, move eligible tasks back to the Model stage if another run is intended.

This is the cross-image batch workflow. `find_similar_objects` on an individual item is a different operation. For a visual walkthrough, see [Batch auto-labeling with Find Similar](https://docs.unitlab.ai/documentation/auto-labeling/batch-auto-labeling-with-find-similar).

### Follow, retry, or cancel a model run

Use `get_model_run` to read the actual state:

* **Queued or processing:** wait and poll the same run.
* **Done:** inspect the saved annotations. A successful run may have found no new objects.
* **Failed:** read the error and `can_start`. Retry only when the tool says the task is eligible.
* **Skipped:** the model did not run for this task, for example because its data type did not match.

`start_model_run` operates on an eligible task at its current Model stage. If the task has already moved to another stage, preview moving it back with `move_tasks` instead of repeatedly calling start.

`cancel_model_run` cancels a background run using its current `run_id`. A synchronous model has no background run to cancel. Annotations already imported are kept. Wait for cancellation to be confirmed before attempting a new run.

Custom model endpoints can be registered with `create_model`, then tested and confirmed in Unitlab's **Models** web interface before use. Registering a model does not train or deploy it.

## Tasks, review, and collaboration

A task represents an item or group at a workflow stage. Keep its `task_id` separate from the item's `datasource_id`.

* Use `list_tasks` or `get_next_task` to find work.
* Read `get_task` for the assignee, claim eligibility, available actions, and rejection feedback.
* Use `claim_task` before editing an eligible free task.
* Save labels with annotation tools.
* Use `perform_task_action` to complete, approve, reject, or apply another currently available action.

Saving labels does not complete a task. `perform_task_action` uses the current stage and available actions, so read the task again if its stage has changed. Rejected tasks carry feedback that the assistant should inspect before making corrections.

Managers with the appropriate permissions can assign, release, or move tasks. Preview moves where supported, especially when the destination is a Model stage.

Comments can be pinned to locations on items. Use `list_comments`, `add_comment`, and `resolve_comment` for discussion, and notification tools to follow responses. Review issues and validation problems are complementary: a technically valid annotation can still require human judgment.

See [Review, rework, and escalation](https://docs.unitlab.ai/documentation/workflows/review-rework-and-escalation) for the web workflow.

## Consensus and benchmark quality

### Consensus

Consensus work uses a private `assignment_id` rather than editing the final item directly:

1. Find the Consensus assignment and claim it with `claim_consensus_task`.
2. Read and save annotations using that assignment's private-copy context.
3. Submit the vote with `submit_consensus_vote`.
4. A reviewer reads `get_consensus_review`, selects approvals with `set_consensus_approval`, and completes the result with `finish_consensus_review`.

Use `get_consensus_report` to inspect available comparison reporting. A comparison metric does not replace review of the underlying objects.

### Benchmarks

Eligible items can be marked as benchmarks with the item management tools. `list_gold_labels` exposes their answer-key copies. Edit the key using the appropriate annotation context, then approve it with `update_gold_label`. Use `get_gold_results` and `get_quality_issues` to inspect results and problems.

Deleting a benchmark removes its answer key, versions, and recorded benchmark results. Read the tool's guidance before selecting that action.

For the corresponding product guides, see [Consensus](https://docs.unitlab.ai/documentation/qa/consensus) and [Quality gate](https://docs.unitlab.ai/documentation/qa/quality-gate).

## Releases and exports

After the intended annotation and review work, create a release to version and export the dataset.

```
list_releases
→ create_release if needed
→ get_release until ready
→ get_release_download
```

Only one export is prepared per project at a time. Reuse a pending or exporting release rather than creating another one. If an export fails, read its error and use `retry_release` after fixing the cause.

### Supported release formats

| Data                                          | Formats                    |
| --------------------------------------------- | -------------------------- |
| Images, video, medical, geospatial, pathology | COCO, YOLOv8, YOLOv5, UUEF |
| Text, HTML, time series, tabular              | JSONL, UUEF                |
| Audio                                         | Audio JSON, RTTM, UUEF     |
| Geospatial                                    | GeoJSON is also available  |
| PDF and LiDAR                                 | UUEF                       |
| Mixed data families                           | UUEF or `standard_bundle`  |

A Standard Bundle uses the applicable family formats. The exact choices and model-specific limits are available through tool help and the release schema.

Split percentages must be whole numbers summing to 100. The default is 80% train, 10% validation, and 10% test. Small datasets may have empty splits.

### Check what will be included

Export eligibility is based on non-archived data, active classes, and valid saved annotations. **Workflow stage is not an export filter.** Do not assume a release contains only approved tasks. Use the supported data-type or upload-session filters and inspect the resulting item counts.

`get_release_download` returns a temporary signed URL. Retrieve a new one if it expires. This tool requires write access because the download is recorded.

A ready release can be cloned into a new project with `clone_release`. Publishing a release publicly is a separate explicit operation and requires the appropriate license selection.

See [Create a release](https://docs.unitlab.ai/documentation/releases/create-a-release) and [Export formats and source inclusion](https://docs.unitlab.ai/documentation/releases/export-formats-and-source-inclusion) for the web export workflow.

## Quotas and processing time

Different operations use different workspace quotas:

| Operation                                          | Quota behavior                                                                                                                    |
| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Segmentation, per-item detection, smart refinement | One AI inference unit per call, including previews; model-service failures refund the unit.                                       |
| Find Similar on an item, geometric refinement      | No AI inference quota.                                                                                                            |
| Tracking                                           | AI inference units per frame in the processing window.                                                                            |
| Hosted workflow model                              | Model quota per item, or per video frame/medical slice, as applicable. Workspace-owned models do not use that hosted-model quota. |

Availability, plans, and workspace limits still apply. A large image, many objects, long video, or large release can take longer to process. Use status tools and progress fields instead of assuming a fixed completion time. `get_context` reports the connection's request limits.

For blocked work, see [Troubleshooting](https://docs.unitlab.ai/api-sdk-cli/mcp/troubleshooting). For the full operation list, see [MCP tool reference](https://docs.unitlab.ai/api-sdk-cli/mcp/tools).