Documentation
HomeBlogsGet Started
  • ๐Ÿ’กIntroduction
  • ๐Ÿš€Unitlab Annotate
  • Workspace Management
    • ๐Ÿ Create a Workspace
    • ๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆInvite Members
    • โ›ณRole-Based Access
  • Project Management
    • ๐Ÿ“ŒSetup a Project
    • ๐Ÿ–Œ๏ธAnnotation
    • ๐Ÿ“‰Performance Analytics
  • Auto Labeling
    • ๐Ÿช„Segment Anything (SAM)
    • ๐Ÿค–Batch Auto-Annotation
    • โน๏ธCrop Auto-Annotation
  • Ai models
    • ๐Ÿง Model Integration
    • ๐Ÿ”งModel Management
    • ๐ŸŽกHow To Use?
  • AUTOMATION WORKFLOW
    • โ›“๏ธAutomation Workflow
  • Dataset Management
    • ๐Ÿ›ฐ๏ธRelease Datasets
    • โš™๏ธManage Datasets
    • ๐ŸŒ€Clone Datasets
  • CLI/Python SDK
    • ๐Ÿ› ๏ธGet started
    • ๐Ÿ’ปUnitlab CLI
    • ๐Ÿ›ก๏ธUnitlab Python SDK
  • How to integrate
    • ๐Ÿ”‘Create an API key
Powered by GitBook
On this page
  • Bring your own pre-trained models
  • Advantages of BYO Models
  • Integrating BYO Models.
  • Valid JSON schemes and format examples

Was this helpful?

  1. Ai models

Model Integration

Bring your own pre-trained models to Unitlab Annotate

PreviousCrop Auto-AnnotationNextModel Management

Last updated 10 months ago

Was this helpful?

Unitlab Annotate's lets you use both built-in and your own pre-trained AI models to annotate data quickly and easily. This guide shows you how to add custom AI models to Unitlab and use them effectively.

Bring your own pre-trained models

Unitlab Annotate enables users to integrate their BYO (Bring Your Own) models for annotation purposes, allowing them to leverage domain-specific models to enhance their data annotation workflow.

Advantages of BYO Models

  • Model In The Loop: Simply annotate data with your model, review and refine it, train your model with a new dataset, and integrate the updated model into Unitlab Annotate. Repeat this process to continually enhance performance.

  • Time & Cost Savings: Automating annotations with custom models reduces manual effort, saving time and costs associated with large-scale data processing.

  • Tailored Solutions: Custom models allow users to address unique annotation requirements, ensuring the platform fits their exact needs.

Integrating BYO Models.

To integrate a BYO AI model into our platform, ensure you have created an account and are signed in.

Prepare the Model: Train and deploy your AI model using your preferred framework or libraries. Ensure the smooth operation of your model's endpoint on your servers, whether they are hosted in the cloud or on-premises."

After signing into Unitlab Annotate, navigate to the AI Models page. You will see the following page

To start integrating a model into our platform, click on the "Integrate Model" box. To successfully complete the integration, you are required to complete 4 steps as follows:

  1. Registration

  2. Validation

  3. Integration

  4. Confirmation.

In our example, let's integrate YOLOv8 - Vehicle Segmentation into Unitlab Annotate. We have already prepared the full source code for YOLOv8 Segmentation deployment in our GitHub repository.

This demo AI model deployment repository can be used for smooth integration with Unitlab Annotate, as shown in the screenshot below.

In the Registration step, the following inputs are required:

  • Model details: Specify the model name and its description.

  • Model type: Select the model data type, such as Image Bounding Box, Segmentation, etc.

  • Model endpoint: Enter the working model endpoint.

  • Headers and Params (optional): Your endpoint may require you to set headers and parameters. Enter parameters and headers as key-value pairs.

After you completing the Registration step, click on the next and move to Validation step.

In the Validation step, we validate your endpoint's health status and response format. During validation, our platform sends a request to your AI model service, using the test data you provided to check its health and JSON format compatibility.

To seamlessly integrate your AI model into Unitlab Annotate, your AI model's service should return a 200 OK response code with the COCO JSON format.

When the validation process is successfully completed, the 'Next' button will be enabled, allowing users to proceed to the next step.

In the Integration step, users must define the model category, related tags, and classes to ensure the seamless integration of their AI model service into the platform.

You can also add extra classes after you have organized the original classes in sequence.

In the final Confirmation step, carefully review all the previous steps to ensure they have been executed correctly. After your review, simply click the Confirm button to finalize the integration.

Valid JSON schemes and format examples

This guide presents the valid COCO JSON structure for integration with UnitLab Annotate. The following examples cover all four model types: Bounding Box, Segmentation, Polygon, and others.

Image Bounding Box

Scheme JSON
{
    "type": "object",
    "properties": {
        "bboxes": {
            "type": "array",
            "items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "point": {
                            "type": "array",
                            "items": {
                                "type": "array",
                                "items": {
                                    "type": "number",
                                    "minimum": 0
                                },
                                "minItems": 2,
                                "maxItems": 2
                            },
                            "minItems": 4,
                            "maxItems": 4
                        },
                        "recognition": {
                            "type": "string"
                        },
                        "class": {
                            "type": "number",
                            "minimum": 0
                        },
                        "text_recognition_confs": {
                            "type": "string"
                        }
                    },
                    "required": [
                        "point",
                        "recognition",
                        "class"
                    ]
                }
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "bboxes",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "bboxes": [
        [
            {
                "class": 0,
                "point": [
                    [
                        1307,
                        1614
                    ],
                    [
                        1307,
                        420
                    ],
                    [
                        1858,
                        420
                    ],
                    [
                        1858,
                        1614
                    ]
                ]
            },
            {
                "class": 0,
                "point": [
                    [
                        1,
                        2482
                    ],
                    [
                        1,
                        95
                    ],
                    [
                        1140,
                        95
                    ],
                    [
                        1140,
                        2482
                    ]
                ]
            },
            {
                "class": 1,
                "point": [
                    [
                        565,
                        577
                    ],
                    [
                        565,
                        152
                    ],
                    [
                        878,
                        152
                    ],
                    [
                        878,
                        577
                    ]
                ],
                "recognition": "face"
            },
            {
                "class": 1,
                "point": [
                    [
                        1453,
                        623
                    ],
                    [
                        1453,
                        439
                    ],
                    [
                        1600,
                        439
                    ],
                    [
                        1600,
                        623
                    ]
                ],
                "recognition": "face"
            },
            {
                "class": 0,
                "point": [
                    [
                        46,
                        1328
                    ],
                    [
                        46,
                        869
                    ],
                    [
                        274,
                        869
                    ],
                    [
                        274,
                        1328
                    ]
                ]
            },
            {
                "class": 0,
                "point": [
                    [
                        0,
                        1732
                    ],
                    [
                        0,
                        542
                    ],
                    [
                        169,
                        542
                    ],
                    [
                        169,
                        1732
                    ]
                ]
            }
        ]
    ],
    "classes": [
        "person",
        "face"
    ],
    "predicted_classes": [
        0,
        1
    ]
}

Image Segmentation

Scheme JSON
{
    "type": "object",
    "properties": {
        "annotations": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "segmentation": {
                        "anyOf": [
                            {
                                "type": "array",
                                "items": {
                                    "type": "array",
                                    "items": {
                                        "type": "number"
                                    }
                                },
                                "minItems": 1
                            },
                            {
                                "type": "object",
                                "properties": {
                                    "counts": {
                                        "type": "array",
                                        "items": {
                                            "type": "integer"
                                        }
                                    },
                                    "size": {
                                        "type": "array",
                                        "items": {
                                            "type": "integer"
                                        }
                                    }
                                },
                                "required": [
                                    "counts",
                                    "size"
                                ]
                            }
                        ]
                    },
                    "category_id": {
                        "type": "integer"
                    }
                },
                "required": [
                    "segmentation",
                    "category_id"
                ]
            },
            "minItems": 1
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "annotations",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "annotations": [
        {
            "category_id": 2,
            "segmentation": [
                [
                    101.0,
                    32.9,
                    101.0,
                    34.9,
                    100.0,
                    75.8,
                    416.0,
                    75.8,
                    394.0,
                    64.8,
                    58.9,
                    129.0,
                    58.9,
                    126.0,
                    55.9,
                    125.0,
                    55.9,
                    102.0,
                    32.9
                ]
            ]
        },
        {
            "category_id": 1,
            "segmentation": [
                [
                    192.0,
                    319.3,
                    191.0,
                    320.3,
                    189.0,
                    320.3,
                    188.0,
                    321.3,
                    187.0,
                    321.3,
                    8.0,
                    388.1,
                    8.0,
                    167.0,
                    565.7,
                    322.3,
                    235.0,
                    322.3,
                    234.0,
                    321.3,
                    233.0,
                    321.3,
                    232.0,
                    320.3,
                    230.0,
                    320.3,
                    229.0,
                    319.3
                ]
            ]
        }
    ],
    "classes": [
        "cat",
        "dog",
        "horse",
        "sheep",
        "cow"
    ],
    "predicted_classes": [
        1,
        2
    ]
}

Image Instance Segmentation

Scheme JSON
{
    "type": "object",
    "properties": {
        "annotations": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "segmentation": {
                        "anyOf": [
                            {
                                "type": "array",
                                "items": {
                                    "type": "array",
                                    "items": {
                                        "type": "number"
                                    }
                                },
                                "minItems": 1
                            },
                            {
                                "type": "object",
                                "properties": {
                                    "counts": {
                                        "type": "array",
                                        "items": {
                                            "type": "integer"
                                        }
                                    },
                                    "size": {
                                        "type": "array",
                                        "items": {
                                            "type": "integer"
                                        }
                                    }
                                },
                                "required": [
                                    "counts",
                                    "size"
                                ]
                            }
                        ]
                    },
                    "category_id": {
                        "type": "integer"
                    }
                },
                "required": [
                    "segmentation",
                    "category_id"
                ]
            },
            "minItems": 1
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "annotations",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "annotations": [
        {
            "category_id": 2,
            "segmentation": [
                [
                    101.0,
                    32.9,
                    101.0,
                    34.9,
                    100.0,
                    75.8,
                    416.0,
                    75.8,
                    394.0,
                    64.8,
                    58.9,
                    129.0,
                    58.9,
                    126.0,
                    55.9,
                    125.0,
                    55.9,
                    102.0,
                    32.9
                ]
            ]
        },
        {
            "category_id": 1,
            "segmentation": [
                [
                    192.0,
                    319.3,
                    191.0,
                    320.3,
                    189.0,
                    320.3,
                    188.0,
                    321.3,
                    187.0,
                    321.3,
                    8.0,
                    388.1,
                    8.0,
                    167.0,
                    565.7,
                    322.3,
                    235.0,
                    322.3,
                    234.0,
                    321.3,
                    233.0,
                    321.3,
                    232.0,
                    320.3,
                    230.0,
                    320.3,
                    229.0,
                    319.3
                ]
            ]
        }
    ],
    "classes": [
        "cat",
        "dog",
        "horse",
        "sheep",
        "cow"
    ],
    "predicted_classes": [
        1,
        2
    ]
}

Image Polygon

Schema JSON
{
    "type": "object",
    "properties": {
        "polygons": {
            "type": "array",
            "items": {
                "type": "object",
                "items": {
                    "type": "object",
                    "properties": {
                        "point": {
                            "type": "array",
                            "items": {
                                "type": "array",
                                "items": {
                                    "type": "number",
                                    "minimum": 0
                                },
                                "minItems": 2,
                                "maxItems": 2
                            }
                        },
                        "class": {
                            "type": "integer",
                            "minimum": 0
                        }
                    },
                    "required": [
                        "point",
                        "class"
                    ]
                }
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "polygons",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "classes": [
        "person"
    ],
    "polygons": [
        {
            "point": [
                [
                    113.1,
                    94.4
                ],
                [
                    111.5,
                    96
                ],
                [
                    111.5,
                    98.7
                ],
                [
                    110.9,
                    99.2
                ],
                [
                    107.7,
                    99.2
                ],
                [
                    107.2,
                    99.8
                ],
                [
                    106.7,
                    99.8
                ],
                [
                    106.1,
                    100.2
                ],
                [
                    104.5,
                    100.2
                ],
                [
                    104,
                    100.8
                ],
                [
                    99.8,
                    100.8
                ],
                [
                    99.2,
                    101.3
                ],
                [
                    98.1,
                    101.3
                ],
                [
                    97.6,
                    101.9
                ],
                [
                    96.5,
                    101.9
                ]
            ],
            "class": 0
        }
    ],
    "predicted_classes": [
        0
    ]
}

Image Skeleton

Schema JSON
{
    "type": "object",
    "properties": {
        "skeletons": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "point": {
                        "anyOf": [
                            {
                                "type": "array",
                                "items": {
                                    "type": "array",
                                    "items": {
                                        "type": "number",
                                        "minimum": 0
                                    },
                                    "minItems": 2,
                                    "maxItems": 2
                                },
                                "minItems": 2
                            },
                            {
                                "type": "array",
                                "items": {
                                    "type": "number",
                                    "minimum": 0
                                },
                                "minItems": 2,
                                "maxItems": 2
                            }
                        ]
                    },
                    "class": {
                        "type": "integer",
                        "minimum": 0
                    }
                },
                "required": [
                    "point",
                    "class"
                ]
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "skeletons",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "classes": [
        "head",
        "upper body",
        "lower body",
        "torso"
    ],
    "predicted_classes": [
        0,
        1,
        2,
        3
    ],
    "skeletons": [
        {
            "class": 0,
            "point": [
                [
                    4100,
                    956
                ],
                [
                    3889,
                    935
                ],
                [
                    3889,
                    935
                ],
                [
                    3512,
                    1237
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    4132,
                    1037
                ],
                [
                    4100,
                    956
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    3903,
                    1370
                ],
                [
                    3512,
                    1237
                ],
                [
                    3512,
                    1237
                ],
                [
                    3125,
                    1661
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    3903,
                    1370
                ],
                [
                    3942,
                    1903
                ],
                [
                    3942,
                    1903
                ],
                [
                    4391,
                    2189
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    3125,
                    1661
                ],
                [
                    3347,
                    1965
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    2993,
                    3847
                ],
                [
                    3666,
                    3235
                ],
                [
                    3666,
                    3235
                ],
                [
                    3634,
                    2305
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    3977,
                    4077
                ],
                [
                    3900,
                    3059
                ],
                [
                    3900,
                    3059
                ],
                [
                    3473,
                    2269
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    3903,
                    1370
                ],
                [
                    3634,
                    2305
                ],
                [
                    3634,
                    2305
                ],
                [
                    3473,
                    2269
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    3512,
                    1237
                ],
                [
                    3473,
                    2269
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    6346,
                    1244
                ],
                [
                    6392,
                    1200
                ],
                [
                    6392,
                    1200
                ],
                [
                    6295,
                    1199
                ],
                [
                    6295,
                    1199
                ],
                [
                    6228,
                    1224
                ],
                [
                    6228,
                    1224
                ],
                [
                    6123,
                    1522
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    6346,
                    1244
                ],
                [
                    6295,
                    1199
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    6392,
                    1200
                ],
                [
                    6461,
                    1226
                ],
                [
                    6461,
                    1226
                ],
                [
                    6515,
                    1510
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    6515,
                    1510
                ],
                [
                    6123,
                    1522
                ],
                [
                    6123,
                    1522
                ],
                [
                    5998,
                    1862
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    5998,
                    1862
                ],
                [
                    6268,
                    1873
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    6211,
                    3049
                ],
                [
                    6467,
                    2603
                ],
                [
                    6467,
                    2603
                ],
                [
                    6378,
                    2193
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    5790,
                    3215
                ],
                [
                    5967,
                    2741
                ],
                [
                    5967,
                    2741
                ],
                [
                    6098,
                    2212
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    6515,
                    1510
                ],
                [
                    6378,
                    2193
                ],
                [
                    6378,
                    2193
                ],
                [
                    6098,
                    2212
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    6123,
                    1522
                ],
                [
                    6098,
                    2212
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    1448,
                    1041
                ],
                [
                    1491,
                    984
                ],
                [
                    1491,
                    984
                ],
                [
                    1389,
                    985
                ],
                [
                    1389,
                    985
                ],
                [
                    1278,
                    996
                ],
                [
                    1278,
                    996
                ],
                [
                    1129,
                    1334
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    1448,
                    1041
                ],
                [
                    1389,
                    985
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    1609,
                    1333
                ],
                [
                    1129,
                    1334
                ],
                [
                    1129,
                    1334
                ],
                [
                    886,
                    1710
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    1609,
                    1333
                ],
                [
                    1688,
                    1772
                ],
                [
                    1688,
                    1772
                ],
                [
                    1813,
                    2031
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    886,
                    1710
                ],
                [
                    1189,
                    1791
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    1681,
                    3519
                ],
                [
                    1698,
                    2847
                ],
                [
                    1698,
                    2847
                ],
                [
                    1570,
                    2216
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    790,
                    3404
                ],
                [
                    1042,
                    2858
                ],
                [
                    1042,
                    2858
                ],
                [
                    1215,
                    2220
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    1609,
                    1333
                ],
                [
                    1570,
                    2216
                ],
                [
                    1570,
                    2216
                ],
                [
                    1215,
                    2220
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    1129,
                    1334
                ],
                [
                    1215,
                    2220
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    2644,
                    824
                ],
                [
                    2486,
                    908
                ],
                [
                    2486,
                    908
                ],
                [
                    2728,
                    1250
                ]
            ]
        },
        {
            "class": 0,
            "point": [
                [
                    2708,
                    882
                ],
                [
                    2644,
                    824
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    2128,
                    1285
                ],
                [
                    2728,
                    1250
                ],
                [
                    2728,
                    1250
                ],
                [
                    3112,
                    1609
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    2128,
                    1285
                ],
                [
                    1674,
                    1831
                ],
                [
                    1674,
                    1831
                ],
                [
                    1795,
                    1849
                ]
            ]
        },
        {
            "class": 1,
            "point": [
                [
                    3112,
                    1609
                ],
                [
                    3577,
                    1660
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    3102,
                    3921
                ],
                [
                    2838,
                    3056
                ],
                [
                    2838,
                    3056
                ],
                [
                    2344,
                    2322
                ]
            ]
        },
        {
            "class": 2,
            "point": [
                [
                    1453,
                    3453
                ],
                [
                    2269,
                    3347
                ],
                [
                    2269,
                    3347
                ],
                [
                    2535,
                    2343
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    2128,
                    1285
                ],
                [
                    2344,
                    2322
                ],
                [
                    2344,
                    2322
                ],
                [
                    2535,
                    2343
                ]
            ]
        },
        {
            "class": 3,
            "point": [
                [
                    2728,
                    1250
                ],
                [
                    2535,
                    2343
                ]
            ]
        }
    ]
}

Image Line

Schema JSON
{
    "type": "object",
    "properties": {
        "lines": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "point": {
                        "type": "array",
                        "items": {
                            "type": "array",
                            "items": {
                                "type": "number",
                                "minimum": 0
                            },
                            "minItems": 2,
                            "maxItems": 2
                        },
                        "minItems": 1
                    },
                    "class": {
                        "type": "integer",
                        "minimum": 0
                    }
                },
                "required": [
                    "point",
                    "class"
                ]
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "lines",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "classes": [
        "noun",
        "verb"
    ],
    "lines": [
        {
            "point": [
                [
                    102.2,
                    2510
                ],
                [
                    617.1,
                    2481.3
                ]
            ],
            "class": 0
        },
        {
            "point": [
                [
                    1279.1,
                    1686.5
                ],
                [
                    1421.3,
                    1666.9
                ],
                [
                    1642,
                    1617.8
                ],
                [
                    1730.2,
                    1539.3
                ],
                [
                    1813.6,
                    1617.8
                ]
            ],
            "class": 0
        },
        {
            "point": [
                [
                    1931.3,
                    2535.3
                ],
                [
                    2058.8,
                    2608.9
                ],
                [
                    2431.5,
                    2662.9
                ],
                [
                    2490.3,
                    2471.5
                ],
                [
                    2436.4,
                    2265.4
                ],
                [
                    2250,
                    2201.7
                ],
                [
                    2049,
                    2236
                ],
                [
                    2019.6,
                    2393
                ],
                [
                    1960.7,
                    2481.3
                ]
            ],
            "class": 1
        },
        {
            "point": [
                [
                    514.1,
                    484.4
                ],
                [
                    519,
                    126.3
                ],
                [
                    1112.4,
                    234.2
                ],
                [
                    960.3,
                    489.3
                ]
            ],
            "class": 0
        }
    ],
    "predicted_classes": [
        0,
        1
    ]
}

Image Point

Schema JSON
{
    "type": "object",
    "properties": {
        "points": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "point": {
                        "type": "array",
                        "items": {
                            "items": {
                                "type": "number",
                                "minimum": 0
                            }
                        },
                        "minItems": 2,
                        "maxItems": 2
                    },
                    "class": {
                        "type": "integer",
                        "minimum": 0
                    }
                },
                "required": [
                    "point",
                    "class"
                ]
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "points",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "classes": [
        "nail",
        "finger"
    ],
    "points": [
        {
            "point": [
                554.2,
                376.1
            ],
            "class": 0
        },
        {
            "point": [
                226.9,
                862.2
            ],
            "class": 0
        },
        {
            "point": [
                911.6,
                331.9
            ],
            "class": 0
        },
        {
            "point": [
                1144.5,
                460.5
            ],
            "class": 0
        },
        {
            "point": [
                1269,
                725.6
            ],
            "class": 0
        },
        {
            "point": [
                313.2,
                1014.9
            ],
            "class": 1
        },
        {
            "point": [
                594.4,
                719.6
            ],
            "class": 1
        },
        {
            "point": [
                590.3,
                583
            ],
            "class": 1
        },
        {
            "point": [
                554.2,
                432.4
            ],
            "class": 1
        },
        {
            "point": [
                281.1,
                940.6
            ],
            "class": 1
        }
    ],
    "predicted_classes": [
        0,
        1
    ]
}

Image Classification

Schema JSON
{
    "type": "object",
    "properties": {
        "categories": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "tag": {
                        "type": "string"
                    },
                    "class": {
                        "type": "integer",
                        "minimum": 0
                    },
                    "id": {
                        "type": "integer"
                    }
                },
                "required": [
                    "tag",
                    "class",
                    "id"
                ]
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "categories",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "categories": [
        {
            "tag": "spam",
            "class": 0,
            "id": 0
        }
    ],
    "classes": [
        "spam",
        "non-spam"
    ],
    "predicted_classes": [
        0
    ]
}

Image OCR

Schema JSON
{
    "type": "object",
    "properties": {
        "points": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "point": {
                        "type": "array",
                        "items": {
                            "items": {
                                "type": "number",
                                "minimum": 0
                            }
                        },
                        "minItems": 2,
                        "maxItems": 2
                    },
                    "class": {
                        "type": "integer",
                        "minimum": 0
                    }
                },
                "required": [
                    "point",
                    "class"
                ]
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "points",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "bboxes": [
        [
            {
                "class": 0,
                "point": [
                    [
                        55.98997389316995,
                        50.75949633573195
                    ],
                    [
                        336.0434088277967,
                        50.75949633573195
                    ],
                    [
                        336.0434088277967,
                        256.8694140492915
                    ],
                    [
                        55.98997389316995,
                        256.8694140492915
                    ]
                ],
                "recognition": "The"
            },
            {
                "class": 0,
                "point": [
                    [
                        36.3723060590093,
                        257.49953428034837
                    ],
                    [
                        349.6248711745234,
                        257.49953428034837
                    ],
                    [
                        349.6248711745234,
                        386.64783202912366
                    ],
                    [
                        36.3723060590093,
                        386.64783202912366
                    ]
                ],
                "recognition": "OCR"
            },
            {
                "class": 0,
                "point": [
                    [
                        366.85455649602386,
                        259.0085856522069
                    ],
                    [
                        824.9760533099553,
                        259.0085856522069
                    ],
                    [
                        824.9760533099553,
                        395.7021402602747
                    ],
                    [
                        366.85455649602386,
                        395.7021402602747
                    ]
                ],
                "recognition": "example"
            }
        ]
    ],
    "classes": [
        "text"
    ],
    "predicted_classes": [
        0
    ]
}

Image Captioning

Schema Json
{
    "type": "object",
    "properties": {
        "bboxes": {
            "type": "array",
            "items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "point": {
                            "type": "array",
                            "items": {
                                "type": "array",
                                "items": {
                                    "type": "number",
                                    "minimum": 0
                                },
                                "minItems": 2,
                                "maxItems": 2
                            },
                            "minItems": 4,
                            "maxItems": 4
                        },
                        "class": {
                            "type": "integer",
                            "minimum": 0
                        }
                    },
                    "required": [
                        "point",
                        "class"
                    ]
                }
            }
        },
        "captioning_text": {
            "type": "array",
            "items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "text": {
                            "type": "string"
                        },
                        "class": {
                            "type": "integer"
                        }
                    },
                    "required": [
                        "text"
                    ]
                }
            }
        },
        "classes": {
            "type": "array",
            "items": {
                "type": "string",
                "minLength": 1
            }
        },
        "predicted_classes": {
            "type": "array",
            "items": {
                "type": "integer"
            }
        }
    },
    "required": [
        "bboxes",
        "captioning_text",
        "classes",
        "predicted_classes"
    ]
}
An Example
{
    "bboxes": [
        [
            {
                "class": 0,
                "point": [
                    [
                        39.191678601999406,
                        167.6251193080077
                    ],
                    [
                        293.2911627522001,
                        167.6251193080077
                    ],
                    [
                        293.2911627522001,
                        860.8851919449099
                    ],
                    [
                        39.191678601999406,
                        860.8851919449099
                    ]
                ]
            },
            {
                "class": 1,
                "point": [
                    [
                        307.15407157693585,
                        152.738319698289
                    ],
                    [
                        452.57991857619015,
                        152.738319698289
                    ],
                    [
                        452.57991857619015,
                        832.6002726864442
                    ],
                    [
                        307.15407157693585,
                        832.6002726864442
                    ]
                ]
            },
            {
                "class": 2,
                "point": [
                    [
                        473.8862272057852,
                        173.57983915189516
                    ],
                    [
                        635.68755377573,
                        173.57983915189516
                    ],
                    [
                        635.68755377573,
                        850.4644322181067
                    ],
                    [
                        473.8862272057852,
                        850.4644322181067
                    ]
                ]
            }
        ]
    ],
    "captioning_text": [
        [
            {
                "text": "The"
            },
            {
                "text": "image"
            },
            {
                "text": "shows"
            },
            {
                "text": "a"
            },
            {
                "text": "group"
            },
            {
                "text": "of"
            },
            {
                "text": "seven"
            },
            {
                "text": "people",
                "class": 0
            },
            {
                "text": "standing"
            },
            {
                "text": "in"
            },
            {
                "text": "a"
            },
            {
                "text": "row"
            },
            {
                "text": "against"
            },
            {
                "text": "a"
            },
            {
                "text": "grey"
            },
            {
                "text": "wall."
            }
        ],
        [
            {
                "text": "The"
            },
            {
                "text": "first"
            },
            {
                "text": "individual",
                "class": 0
            },
            {
                "text": "is"
            },
            {
                "text": "a"
            },
            {
                "text": "young"
            },
            {
                "text": "man"
            },
            {
                "text": "wearing"
            },
            {
                "text": "a"
            },
            {
                "text": "long-sleeve"
            },
            {
                "text": "dark"
            },
            {
                "text": "blue"
            },
            {
                "text": "shirt,",
                "class": 1
            },
            {
                "text": "blue"
            },
            {
                "text": "jeans,",
                "class": 2
            },
            {
                "text": "and"
            },
            {
                "text": "black"
            },
            {
                "text": "shoes.",
                "class": 3
            }
        ]
    ],
    "classes": [
        "person",
        "shirt",
        "jeans",
        "shoes"
    ],
    "predicted_classes": [
        0,
        1,
        2,
        3
    ]
}

Learn the JSON schema in the section.

If you follow our and use , Unitlab Annotate will automatically fill in the classes based on the test response from your AI service, as shown in the screenshot below.

After successfully integrating your BYO model, you can test it, learn how to test and manage it on the , or simply get started using it to annotate your data. Find instructions on how to use it on the '' page.

๐Ÿง 
Model Management Page
How to Use
Valid JSON Schemes
demo repository
valid JSON response formats
Unitlab Annotate: AI Models Page
Unitlab Annotate: The registration step of AI Models.
Unitlab Annotate: The validation step of AI Models.
Unitlab Annotate: The Validation Step of AI Models - Endpoint Health and Format Validated
Unitlab Annotate: The Integration Step of AI Models
Unitlab Annotate: The Confirmation Step of AI Models
LogoGitHub - teamunitlab/yolo8-segmentation-deploy: This is a production-ready deployment of YOLO8 Segmentation with TensorRT and ONNX, supporting both CPU and GPU options. Moreover, it can demonstrate how to integrate your AI models into Unitlab Annotate.GitHub
YOLO8 Segmentation Deployment: A Demo for Model Integration into Unitlab Annotate.