> For the complete documentation index, see [llms.txt](https://docs.unitlab.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.unitlab.ai/documentation/auto-labeling/prompt-labeling.md).

# Prompt Labeling

Describe a visual concept in natural language and create editable, class-bound SAM 3 proposals.

Prompt Labeling lets an annotator describe a visual concept in natural language and create editable, class-bound proposals on the current image or frame. It is the prompt-authoring workflow behind the current SAM 3 **Detect all objects** action.

![Prompt Auto-Labeling identifies a described concept in the current image](https://292810646-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGjVLUz4wthGkGlRKM6rM%2Fuploads%2FhruSGNY8k1TbrOPBnHXn%2Fprompt-auto-labeling.webp?alt=media)

### Supported scope

| Dimension | Current behavior                                             |
| --------- | ------------------------------------------------------------ |
| Data      | Image and current video frame                                |
| Geometry  | Bounding box, polygon, segmentation mask, cuboid / 3D box    |
| Class     | Output is written to the active ontology class               |
| Prompt    | Defaults to the class name; up to 300 characters             |
| Result    | Editable candidate annotations                               |
| Quota     | One AI-inference unit is consumed after a successful request |

Prompt Labeling is not a dataset-wide text search and does not create a new ontology class. It uses the prompt to find instances, then applies the selected class and geometry.

### Write effective prompts

A good prompt names one visible concept and, only when necessary, adds a short disambiguator.

| Prompt quality | Example                       | Why                                                      |
| -------------- | ----------------------------- | -------------------------------------------------------- |
| Strong         | **yellow safety helmet**      | Concrete object and distinguishing state                 |
| Strong         | **white delivery van**        | Object plus visible attribute                            |
| Strong         | **tumor region in the liver** | Region plus anatomical context                           |
| Weak           | **all important things**      | Ambiguous and not visually testable                      |
| Weak           | **person, helmet, vehicle**   | Mixes several ontology classes                           |
| Weak           | **unsafe**                    | Describes a judgment rather than a stable visible object |

Use project Instructions to standardize prompts when several operators work on the same class. Record class-specific examples, known exclusions, and failure cases.

### Run Prompt Labeling

{% stepper %}
{% step %}

#### Select the target class

Choose the ontology class and its supported geometry. The class should represent the semantic label you intend to store.
{% endstep %}

{% step %}

#### Open Auto-labeling

Select the wand action or press **S**. The popup shows the active class and pre-fills the prompt with its name.
{% endstep %}

{% step %}

#### Refine the prompt

Keep the class name when it is precise. Otherwise add a short visible qualifier. Use the reset action to return to the class name.
{% endstep %}

{% step %}

#### Detect all objects

Run the request on the current image or frame. Wait for the proposal set to appear before navigating away.
{% endstep %}

{% step %}

#### Review the full result set

Check missed instances, false positives, boundary quality, duplicate overlap, and the effect of the chosen geometry.
{% endstep %}

{% step %}

#### Correct and submit

Edit proposals with the standard Workbench tools, complete required properties, and follow the project’s Annotate/Review workflow.
{% endstep %}
{% endstepper %}

### Prompt-to-geometry design

| Downstream task           | Recommended geometry | Prompt guidance                                                 |
| ------------------------- | -------------------- | --------------------------------------------------------------- |
| Object detection          | Bounding box         | Name one countable object                                       |
| Instance segmentation     | Mask                 | Name one object or region with visible boundaries               |
| Boundary-aware labeling   | Polygon              | Use the same semantic class as the polygon ontology             |
| Oriented spatial labeling | Cuboid / 3D box      | Name the physical object; review perspective and depth manually |

### Common failure modes

| Symptom                               | Likely cause                     | Recovery                                                              |
| ------------------------------------- | -------------------------------- | --------------------------------------------------------------------- |
| Correct concept, wrong class          | Active class was not changed     | Clear proposals, select the correct class, rerun                      |
| Good large objects, missed small ones | Scale or visibility limits       | Use a clearer frame, lower threshold where available, or add manually |
| Background regions are included       | Prompt is too broad              | Add visible context or use mask correction                            |
| Several concepts are mixed            | Prompt contains multiple classes | Run one class at a time                                               |
| Video results vary by frame           | Each request evaluates one frame | Use a reliable seed frame and continue with tracking                  |

{% hint style="warning" %}
Do not encode hidden business rules only in a prompt. Inclusion, exclusion, geometry, occlusion, and property policy belong in the project Instructions and ontology.
{% endhint %}

### Prompt governance

For production programs, retain a small prompt register with:

* project and ontology version;
* class and geometry;
* approved prompt;
* representative examples and exclusions;
* known domain failures;
* accepted-proposal and correction rate;
* owner and last validation date.

### Related guides

* [Detect Anything (SAM 1–SAM 3)](/documentation/auto-labeling/detect-anything-sam-1-sam-3.md)
* [Find Similar](/documentation/auto-labeling/find-similar.md)
* [Image Annotation](/documentation/annotations/image-annotation.md)
* [Project Instructions](/documentation/projects/project-instructions.md)