Prompt Labeling

Describe a visual concept in natural language and create editable, class-bound SAM 3 proposals.

Prompt Labeling lets an annotator describe a visual concept in natural language and create editable, class-bound proposals on the current image or frame. It is the prompt-authoring workflow behind the current SAM 3 Detect all objects action.

Prompt Auto-Labeling identifies a described concept in the current image

Supported scope

Dimension	Current behavior
Data	Image and current video frame
Geometry	Bounding box, polygon, segmentation mask, cuboid / 3D box
Class	Output is written to the active ontology class
Prompt	Defaults to the class name; up to 300 characters
Result	Editable candidate annotations
Quota	One AI-inference unit is consumed after a successful request

Prompt Labeling is not a dataset-wide text search and does not create a new ontology class. It uses the prompt to find instances, then applies the selected class and geometry.

Write effective prompts

A good prompt names one visible concept and, only when necessary, adds a short disambiguator.

Prompt quality	Example	Why
Strong	yellow safety helmet	Concrete object and distinguishing state
Strong	white delivery van	Object plus visible attribute
Strong	tumor region in the liver	Region plus anatomical context
Weak	all important things	Ambiguous and not visually testable
Weak	person, helmet, vehicle	Mixes several ontology classes
Weak	unsafe	Describes a judgment rather than a stable visible object

Use project Instructions to standardize prompts when several operators work on the same class. Record class-specific examples, known exclusions, and failure cases.

Run Prompt Labeling

1

Select the target class

Choose the ontology class and its supported geometry. The class should represent the semantic label you intend to store.

2

Open Auto-labeling

Select the wand action or press S. The popup shows the active class and pre-fills the prompt with its name.

3

Refine the prompt

Keep the class name when it is precise. Otherwise add a short visible qualifier. Use the reset action to return to the class name.

4

Detect all objects

Run the request on the current image or frame. Wait for the proposal set to appear before navigating away.

5

Review the full result set

Check missed instances, false positives, boundary quality, duplicate overlap, and the effect of the chosen geometry.

6

Correct and submit

Edit proposals with the standard Workbench tools, complete required properties, and follow the project’s Annotate/Review workflow.

Prompt-to-geometry design

Downstream task	Recommended geometry	Prompt guidance
Object detection	Bounding box	Name one countable object
Instance segmentation	Mask	Name one object or region with visible boundaries
Boundary-aware labeling	Polygon	Use the same semantic class as the polygon ontology
Oriented spatial labeling	Cuboid / 3D box	Name the physical object; review perspective and depth manually

Common failure modes

Symptom	Likely cause	Recovery
Correct concept, wrong class	Active class was not changed	Clear proposals, select the correct class, rerun
Good large objects, missed small ones	Scale or visibility limits	Use a clearer frame, lower threshold where available, or add manually
Background regions are included	Prompt is too broad	Add visible context or use mask correction
Several concepts are mixed	Prompt contains multiple classes	Run one class at a time
Video results vary by frame	Each request evaluates one frame	Use a reliable seed frame and continue with tracking

Do not encode hidden business rules only in a prompt. Inclusion, exclusion, geometry, occlusion, and property policy belong in the project Instructions and ontology.

Prompt governance

For production programs, retain a small prompt register with:

• project and ontology version;

• class and geometry;

• approved prompt;

• representative examples and exclusions;

• known domain failures;

• accepted-proposal and correction rate;

• owner and last validation date.

Related guides

• Detect Anything (SAM 1–SAM 3)

• Find Similar

• Image Annotation

• Project Instructions