> For the complete documentation index, see [llms.txt](https://docs.unitlab.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.unitlab.ai/api-sdk-cli/apis/annotation-json-reference.md).

# Annotation JSON reference

The JSON shapes Unitlab writes and reads for annotations - UUEF per item, GeoJSON for geospatial models, and the prediction format of image models.

Unitlab uses three JSON shapes for annotations:

| Shape                                                          | Where you meet it                                                                                                                            |
| -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **UUEF** (one JSON document per item)                          | Release exports in UUEF format, annotation imports, Agent stage reads and saves                                                              |
| **GeoJSON (geospatial models)**, an RFC 7946 FeatureCollection | The answer a geospatial model gives a Model stage. Agent saves accept it as `format="geojson"`.                                              |
| **Prediction format (image models)**                           | The predictions an image model connected under **My AI Models** sends back to Unitlab. Agent saves accept the same JSON as `format="model"`. |

For exports in other formats, see [Export formats and source inclusion](https://docs.unitlab.ai/documentation/releases/export-formats-and-source-inclusion). To bring existing labels into a project, see [Annotation imports](https://docs.unitlab.ai/api-sdk-cli/sdks/annotation-imports.md). To connect your own model, see [Bring your own Models](https://docs.unitlab.ai/documentation/auto-labeling/bring-your-own-models).

{% hint style="info" %}
**Coordinates** are in pixels of the original media: x to the right, y down, (0, 0) at the top-left corner. This holds for images, video frames, medical slices, PDF pages and large rasters and slides at full resolution. A geospatial model can answer in map coordinates instead (see [GeoJSON (geospatial models)](#geojson-for-geospatial-models)).
{% endhint %}

## UUEF document

Each item of a UUEF release is one JSON document. Every UUEF release archive includes the JSON Schema as `uuef.schema.json` at its root.

```json
{
  "version": "1.0",
  "media": {
    "filename": "street_001.jpg",
    "media_type": "Image",
    "origin": {"media_id": "…", "release_id": "…", "release_version": "v1"},
    "source_url": "https://…",
    "width": 1920,
    "height": 1080
  },
  "labels": [
    {
      "id": "b7c1…",
      "value": 1,
      "geometry": {"type": "bbox", "points": [[100, 50], [300, 50], [300, 200], [100, 200]]},
      "properties": [{"id": 3, "name": "Occluded", "type": "boolean", "value": false}],
      "attributes": {}
    }
  ],
  "classes": [
    {"name": "Car", "value": 1, "markup_type": "bbox", "color": "#e5484d", "attributes": [], "properties": []}
  ],
  "item_property_definitions": [],
  "item_properties": [],
  "data_tags": [],
  "relations": []
}
```

### Top-level fields

| Field                                          | Meaning                                                                                                                                                                                                                                                                                                                         |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version`                                      | Always `"1.0"`.                                                                                                                                                                                                                                                                                                                 |
| `media`                                        | The item: `filename`, `media_type`, `origin` (the item and release it came from), `source_url` (a link to the original file) and, depending on the data type, `width`/`height`, `total_frames`/`fps`/`duration`, `page_count`, or, for a geospatial raster, `geospatial` (`crs`, `transform`, `bounds_native`, `bounds_wgs84`). |
| `classes`                                      | The project's classes: `name`, `value` (the class number labels refer to), `markup_type`, `color`, and the class's `attributes` and `properties` definitions.                                                                                                                                                                   |
| `labels`                                       | The objects on the item (below).                                                                                                                                                                                                                                                                                                |
| `item_property_definitions`, `item_properties` | Item-level properties: their definitions and the values set on this item.                                                                                                                                                                                                                                                       |
| `data_tags`                                    | Tags on the item.                                                                                                                                                                                                                                                                                                               |
| `relations`                                    | Links between labels (below).                                                                                                                                                                                                                                                                                                   |
| `group`                                        | Present only for an item that is a panel of a Data Group: the group and the panel.                                                                                                                                                                                                                                              |

### Labels

Every label has:

| Field         | Meaning                                                                                       |
| ------------- | --------------------------------------------------------------------------------------------- |
| `id`          | The object's id. It stays the same across versions and releases.                              |
| `value`       | The class number: the `value` of an entry in `classes`.                                       |
| `properties`  | A list of `{id, name, type, value}`. A selection carries the chosen option's `id` and `name`. |
| `attributes`  | The class's attribute values.                                                                 |
| `page_number` | PDF only: the page, starting at 1.                                                            |

Plus exactly one shape:

| Shape        | JSON                                                                                                                  |
| ------------ | --------------------------------------------------------------------------------------------------------------------- |
| Bounding box | `"geometry": {"type": "bbox", "points": [[x, y], [x, y], [x, y], [x, y]]}`: four corners, clockwise from the top-left |
| Polygon      | `"polygon": {"paths": [[[x, y], …], [[x, y], …]]}`: the outline first, then one path per hole                         |
| Mask         | `"mask": {"rle": […], "left": 10, "top": 20, "width": 64, "height": 48}` (see below)                                  |
| Point        | `"point": [x, y]`                                                                                                     |
| Line         | `"polyline": {"point": [[x, y], …]}`                                                                                  |
| Skeleton     | `"skeleton": {"point": [[x, y], …]}`                                                                                  |
| Cuboid       | `"cuboid": {"point": [[x, y], …]}`: eight corners, in the editor's order                                              |

**Mask encoding.** `left`, `top`, `width` and `height` give the box the mask fills. `rle` covers that box row by row, from the top-left pixel, as alternating counts of empty and filled pixels. The first count is empty pixels, and it is `0` when the first pixel is filled. Example: `[3, 2, 5]` means 3 empty pixels, 2 filled, then 5 empty.

Text, audio and time series labels have a `type` instead of a shape:

| Data type   | Label                                                                                                                                                          |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Text        | `"type": "entity"` with `span: {start, end, text}` in character offsets; `"type": "classification"` for a whole-text class                                     |
| Audio       | `"type": "event"` with `interval: {start, end}` in seconds; `"type": "classification"` for a whole-file class                                                  |
| Time series | `"type": "interval"` with `interval: {start, end}`, or `"type": "instant"` with `instant: {x}`, in the units of the X column (dates as UTC epoch milliseconds) |

### Video and medical: tracks

On a video or a medical volume, each label is a track: one object followed across frames (slices for medical).

```json
{
  "id": "car-1",
  "value": 1,
  "ranges": [[0, 3]],
  "track": {
    "0": {"keyframe": true,  "geometry": {"type": "bbox", "points": [[10, 10], [50, 10], [50, 40], [10, 40]]}},
    "1": {"keyframe": false, "geometry": {"type": "bbox", "points": [[12, 10], [52, 10], [52, 40], [12, 40]]}},
    "2": {"keyframe": true,  "geometry": {"type": "bbox", "points": [[14, 10], [54, 10], [54, 40], [14, 40]]}}
  },
  "properties": [],
  "attributes": {}
}
```

| Field           | Meaning                                                                                                                                            |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ranges`        | Where the object exists, as `[start, end]` frame numbers: `start` is included and `end` is not. Frames start at 0.                                 |
| `track`         | One entry per frame number, each with the same shapes as above.                                                                                    |
| `keyframe`      | `true` for a frame someone (a person or a model) placed. `false` for a frame filled in between keyframes. A release lists every frame in `ranges`. |
| `hidden_ranges` | Optional: frame ranges where the object is switched off.                                                                                           |
| `plane`         | Medical only: the view the track is drawn on (`sagittal` or `coronal`). When missing, the view is axial. Frame numbers are that view's slices.     |

### Relations

```json
{"id": "r1", "type_id": 2, "type": "drives", "source_id": "person-1", "target_id": "car-1"}
```

`source_id` and `target_id` are label `id`s. A relation whose source or target is not in `labels` is left out.

## GeoJSON (geospatial models) <a href="#geojson-for-geospatial-models" id="geojson-for-geospatial-models"></a>

A custom geospatial model answers a Model stage with a GeoJSON `FeatureCollection` (RFC 7946). It can also wrap it as `{"status": "complete", "result": {…}}`.

```json
{
  "type": "FeatureCollection",
  "classes": ["building", "road"],
  "features": [
    {
      "type": "Feature",
      "geometry": {"type": "Polygon", "coordinates": [[[69.2401, 41.2995], [69.2410, 41.2995], [69.2410, 41.3001], [69.2401, 41.3001], [69.2401, 41.2995]]]},
      "properties": {"class_name": "building", "markup_type": "polygon", "confidence": 0.93}
    }
  ]
}
```

| Field                    | Meaning                                                                                                                     |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| `classes`                | The model's classes: names, or objects `{name, markup_type}`. A class's number is its position in this list, starting at 0. |
| `properties.class_name`  | The class by name. Alternatively, `properties.class` gives its number.                                                      |
| `properties.markup_type` | Optional: `bbox`, `polygon`, `mask`, `line` or `point`. If omitted, it follows from the geometry.                           |
| `properties.confidence`  | Optional, 0 to 1.                                                                                                           |

Geometry and the shape it becomes:

| Geometry       | Becomes                                                                   |
| -------------- | ------------------------------------------------------------------------- |
| `Point`        | point                                                                     |
| `LineString`   | line                                                                      |
| `Polygon`      | polygon (default), bounding box or mask. Rings after the first are holes. |
| `MultiPolygon` | one polygon or mask per part                                              |

Coordinates can be longitude/latitude (WGS84), the raster's own map units, or raster pixels. Unitlab works out which from the whole answer. Shapes that extend past the raster are clipped to its edge.

An Agent stage save (`format="geojson"`, for geospatial and pathology items) takes the same features without the `classes` list: `properties.class_name` is the name of a project class, or `properties.class` a project class value, and a feature's `id` becomes the object's id. Pathology slides use pixels.

## Prediction format (image models) <a href="#prediction-format-image-models" id="prediction-format-image-models"></a>

The predictions an image model connected under **My AI Models** sends back to Unitlab. Agent saves accept the same JSON as `format="model"`. [Bring your own Models](https://docs.unitlab.ai/documentation/auto-labeling/bring-your-own-models#representative-image-request) is the authoritative description of the request and the answer, and its [Validation](https://docs.unitlab.ai/documentation/auto-labeling/bring-your-own-models#validation) step checks a model's answer; the keys below are a short summary.

Unitlab sends the model a JSON POST with `src`, a temporary link to the image, plus the model's saved parameters. The model answers with one list of objects per shape it produces, plus `classes`:

```json
{
  "classes": ["car", "person"],
  "bboxes": [[{"point": [[100, 50], [300, 50], [300, 200], [100, 200]], "class": 0}]],
  "polygons": [{"point": [[10, 10], [80, 12], [60, 90]], "class": 1}]
}
```

| Key         | Each object                                                                                                                                                                                                                             |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `classes`   | The model's class names. `class` in every object is a position in this list, starting at 0.                                                                                                                                             |
| `bboxes`    | A list of lists of `{point, class}`. `point` is four `[x, y]` corners.                                                                                                                                                                  |
| `polygons`  | `{point, class}` with at least three `[x, y]` points.                                                                                                                                                                                   |
| `brush`     | A pixel mask `{rle, left, top, right, bottom, value}` (`right` and `bottom` are the last filled column and row; `rle` as in [Mask encoding](#labels)), or an outline `{point, holes, value, vector_brush: true}`. `value` is the class. |
| `points`    | `{point: [x, y], class}`                                                                                                                                                                                                                |
| `lines`     | `{point: [[x, y], …], class}`                                                                                                                                                                                                           |
| `skeletons` | `{point: [[x, y], …], class}`                                                                                                                                                                                                           |

All coordinates are pixels of the original image. A Model stage maps the model's classes to project classes.

In an Agent stage save, `classes` names project classes, matched by name and the object's shape, and the saved objects get new ids.
