🧠Model Integration
Bring your own pre-trained models to Unitlab Annotate
Last updated
Bring your own pre-trained models to Unitlab Annotate
Last updated
Unitlab Annotate's lets you use both built-in and your own pre-trained AI models to annotate data quickly and easily. This guide shows you how to add custom AI models to Unitlab and use them effectively.
Unitlab Annotate enables users to integrate their BYO (Bring Your Own) models for annotation purposes, allowing them to leverage domain-specific models to enhance their data annotation workflow.
Model In The Loop: Simply annotate data with your model, review and refine it, train your model with a new dataset, and integrate the updated model into Unitlab Annotate. Repeat this process to continually enhance performance.
Time & Cost Savings: Automating annotations with custom models reduces manual effort, saving time and costs associated with large-scale data processing.
Tailored Solutions: Custom models allow users to address unique annotation requirements, ensuring the platform fits their exact needs.
To integrate a BYO AI model into our platform, ensure you have created an account and are signed in.
Prepare the Model: Train and deploy your AI model using your preferred framework or libraries. Ensure the smooth operation of your model's endpoint on your servers, whether they are hosted in the cloud or on-premises."
After signing into Unitlab Annotate, navigate to the AI Models page. You will see the following page
To start integrating a model into our platform, click on the "Integrate Model" box. To successfully complete the integration, you are required to complete 4 steps as follows:
Registration
Validation
Integration
Confirmation.
In our example, let's integrate YOLOv8 - Vehicle Segmentation into Unitlab Annotate. We have already prepared the full source code for YOLOv8 Segmentation deployment in our GitHub repository.
This demo AI model deployment repository can be used for smooth integration with Unitlab Annotate, as shown in the screenshot below.
In the Registration step, the following inputs are required:
Model details: Specify the model name and its description.
Model type: Select the model data type, such as Image Bounding Box, Segmentation, etc.
Model endpoint: Enter the working model endpoint.
Headers and Params (optional): Your endpoint may require you to set headers and parameters. Enter parameters and headers as key-value pairs.
After you completing the Registration step, click on the next and move to Validation step.
In the Validation step, we validate your endpoint's health status and response format. During validation, our platform sends a request to your AI model service, using the test data you provided to check its health and JSON format compatibility.
To seamlessly integrate your AI model into Unitlab Annotate, your AI model's service should return a 200 OK response code with the COCO JSON format.
Learn the JSON schema in the Valid JSON Schemes section.
When the validation process is successfully completed, the 'Next' button will be enabled, allowing users to proceed to the next step.
In the Integration step, users must define the model category, related tags, and classes to ensure the seamless integration of their AI model service into the platform.
If you follow our demo repository and use valid JSON response formats, Unitlab Annotate will automatically fill in the classes based on the test response from your AI service, as shown in the screenshot below.
You can also add extra classes after you have organized the original classes in sequence.
In the final Confirmation step, carefully review all the previous steps to ensure they have been executed correctly. After your review, simply click the Confirm button to finalize the integration.
After successfully integrating your BYO model, you can test it, learn how to test and manage it on the Model Management Page, or simply get started using it to annotate your data. Find instructions on how to use it on the 'How to Use' page.
This guide presents the valid COCO JSON structure for integration with UnitLab Annotate. The following examples cover all four model types: Bounding Box, Segmentation, Polygon, and others.
{
"type": "object",
"properties": {
"bboxes": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"point": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number",
"minimum": 0
},
"minItems": 2,
"maxItems": 2
},
"minItems": 4,
"maxItems": 4
},
"recognition": {
"type": "string"
},
"class": {
"type": "number",
"minimum": 0
},
"text_recognition_confs": {
"type": "string"
}
},
"required": [
"point",
"recognition",
"class"
]
}
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"bboxes",
"classes",
"predicted_classes"
]
}{
"bboxes": [
[
{
"class": 0,
"point": [
[
1307,
1614
],
[
1307,
420
],
[
1858,
420
],
[
1858,
1614
]
]
},
{
"class": 0,
"point": [
[
1,
2482
],
[
1,
95
],
[
1140,
95
],
[
1140,
2482
]
]
},
{
"class": 1,
"point": [
[
565,
577
],
[
565,
152
],
[
878,
152
],
[
878,
577
]
],
"recognition": "face"
},
{
"class": 1,
"point": [
[
1453,
623
],
[
1453,
439
],
[
1600,
439
],
[
1600,
623
]
],
"recognition": "face"
},
{
"class": 0,
"point": [
[
46,
1328
],
[
46,
869
],
[
274,
869
],
[
274,
1328
]
]
},
{
"class": 0,
"point": [
[
0,
1732
],
[
0,
542
],
[
169,
542
],
[
169,
1732
]
]
}
]
],
"classes": [
"person",
"face"
],
"predicted_classes": [
0,
1
]
}{
"type": "object",
"properties": {
"annotations": {
"type": "array",
"items": {
"type": "object",
"properties": {
"segmentation": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number"
}
},
"minItems": 1
},
{
"type": "object",
"properties": {
"counts": {
"type": "array",
"items": {
"type": "integer"
}
},
"size": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"counts",
"size"
]
}
]
},
"category_id": {
"type": "integer"
}
},
"required": [
"segmentation",
"category_id"
]
},
"minItems": 1
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"annotations",
"classes",
"predicted_classes"
]
}{
"annotations": [
{
"category_id": 2,
"segmentation": [
[
101.0,
32.9,
101.0,
34.9,
100.0,
75.8,
416.0,
75.8,
394.0,
64.8,
58.9,
129.0,
58.9,
126.0,
55.9,
125.0,
55.9,
102.0,
32.9
]
]
},
{
"category_id": 1,
"segmentation": [
[
192.0,
319.3,
191.0,
320.3,
189.0,
320.3,
188.0,
321.3,
187.0,
321.3,
8.0,
388.1,
8.0,
167.0,
565.7,
322.3,
235.0,
322.3,
234.0,
321.3,
233.0,
321.3,
232.0,
320.3,
230.0,
320.3,
229.0,
319.3
]
]
}
],
"classes": [
"cat",
"dog",
"horse",
"sheep",
"cow"
],
"predicted_classes": [
1,
2
]
}{
"type": "object",
"properties": {
"annotations": {
"type": "array",
"items": {
"type": "object",
"properties": {
"segmentation": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number"
}
},
"minItems": 1
},
{
"type": "object",
"properties": {
"counts": {
"type": "array",
"items": {
"type": "integer"
}
},
"size": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"counts",
"size"
]
}
]
},
"category_id": {
"type": "integer"
}
},
"required": [
"segmentation",
"category_id"
]
},
"minItems": 1
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"annotations",
"classes",
"predicted_classes"
]
}{
"annotations": [
{
"category_id": 2,
"segmentation": [
[
101.0,
32.9,
101.0,
34.9,
100.0,
75.8,
416.0,
75.8,
394.0,
64.8,
58.9,
129.0,
58.9,
126.0,
55.9,
125.0,
55.9,
102.0,
32.9
]
]
},
{
"category_id": 1,
"segmentation": [
[
192.0,
319.3,
191.0,
320.3,
189.0,
320.3,
188.0,
321.3,
187.0,
321.3,
8.0,
388.1,
8.0,
167.0,
565.7,
322.3,
235.0,
322.3,
234.0,
321.3,
233.0,
321.3,
232.0,
320.3,
230.0,
320.3,
229.0,
319.3
]
]
}
],
"classes": [
"cat",
"dog",
"horse",
"sheep",
"cow"
],
"predicted_classes": [
1,
2
]
}{
"type": "object",
"properties": {
"polygons": {
"type": "array",
"items": {
"type": "object",
"items": {
"type": "object",
"properties": {
"point": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number",
"minimum": 0
},
"minItems": 2,
"maxItems": 2
}
},
"class": {
"type": "integer",
"minimum": 0
}
},
"required": [
"point",
"class"
]
}
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"polygons",
"classes",
"predicted_classes"
]
}{
"classes": [
"person"
],
"polygons": [
{
"point": [
[
113.1,
94.4
],
[
111.5,
96
],
[
111.5,
98.7
],
[
110.9,
99.2
],
[
107.7,
99.2
],
[
107.2,
99.8
],
[
106.7,
99.8
],
[
106.1,
100.2
],
[
104.5,
100.2
],
[
104,
100.8
],
[
99.8,
100.8
],
[
99.2,
101.3
],
[
98.1,
101.3
],
[
97.6,
101.9
],
[
96.5,
101.9
]
],
"class": 0
}
],
"predicted_classes": [
0
]
}{
"type": "object",
"properties": {
"skeletons": {
"type": "array",
"items": {
"type": "object",
"properties": {
"point": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number",
"minimum": 0
},
"minItems": 2,
"maxItems": 2
},
"minItems": 2
},
{
"type": "array",
"items": {
"type": "number",
"minimum": 0
},
"minItems": 2,
"maxItems": 2
}
]
},
"class": {
"type": "integer",
"minimum": 0
}
},
"required": [
"point",
"class"
]
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"skeletons",
"classes",
"predicted_classes"
]
}{
"classes": [
"head",
"upper body",
"lower body",
"torso"
],
"predicted_classes": [
0,
1,
2,
3
],
"skeletons": [
{
"class": 0,
"point": [
[
4100,
956
],
[
3889,
935
],
[
3889,
935
],
[
3512,
1237
]
]
},
{
"class": 0,
"point": [
[
4132,
1037
],
[
4100,
956
]
]
},
{
"class": 1,
"point": [
[
3903,
1370
],
[
3512,
1237
],
[
3512,
1237
],
[
3125,
1661
]
]
},
{
"class": 1,
"point": [
[
3903,
1370
],
[
3942,
1903
],
[
3942,
1903
],
[
4391,
2189
]
]
},
{
"class": 1,
"point": [
[
3125,
1661
],
[
3347,
1965
]
]
},
{
"class": 2,
"point": [
[
2993,
3847
],
[
3666,
3235
],
[
3666,
3235
],
[
3634,
2305
]
]
},
{
"class": 2,
"point": [
[
3977,
4077
],
[
3900,
3059
],
[
3900,
3059
],
[
3473,
2269
]
]
},
{
"class": 3,
"point": [
[
3903,
1370
],
[
3634,
2305
],
[
3634,
2305
],
[
3473,
2269
]
]
},
{
"class": 3,
"point": [
[
3512,
1237
],
[
3473,
2269
]
]
},
{
"class": 0,
"point": [
[
6346,
1244
],
[
6392,
1200
],
[
6392,
1200
],
[
6295,
1199
],
[
6295,
1199
],
[
6228,
1224
],
[
6228,
1224
],
[
6123,
1522
]
]
},
{
"class": 0,
"point": [
[
6346,
1244
],
[
6295,
1199
]
]
},
{
"class": 0,
"point": [
[
6392,
1200
],
[
6461,
1226
],
[
6461,
1226
],
[
6515,
1510
]
]
},
{
"class": 1,
"point": [
[
6515,
1510
],
[
6123,
1522
],
[
6123,
1522
],
[
5998,
1862
]
]
},
{
"class": 1,
"point": [
[
5998,
1862
],
[
6268,
1873
]
]
},
{
"class": 2,
"point": [
[
6211,
3049
],
[
6467,
2603
],
[
6467,
2603
],
[
6378,
2193
]
]
},
{
"class": 2,
"point": [
[
5790,
3215
],
[
5967,
2741
],
[
5967,
2741
],
[
6098,
2212
]
]
},
{
"class": 3,
"point": [
[
6515,
1510
],
[
6378,
2193
],
[
6378,
2193
],
[
6098,
2212
]
]
},
{
"class": 3,
"point": [
[
6123,
1522
],
[
6098,
2212
]
]
},
{
"class": 0,
"point": [
[
1448,
1041
],
[
1491,
984
],
[
1491,
984
],
[
1389,
985
],
[
1389,
985
],
[
1278,
996
],
[
1278,
996
],
[
1129,
1334
]
]
},
{
"class": 0,
"point": [
[
1448,
1041
],
[
1389,
985
]
]
},
{
"class": 1,
"point": [
[
1609,
1333
],
[
1129,
1334
],
[
1129,
1334
],
[
886,
1710
]
]
},
{
"class": 1,
"point": [
[
1609,
1333
],
[
1688,
1772
],
[
1688,
1772
],
[
1813,
2031
]
]
},
{
"class": 1,
"point": [
[
886,
1710
],
[
1189,
1791
]
]
},
{
"class": 2,
"point": [
[
1681,
3519
],
[
1698,
2847
],
[
1698,
2847
],
[
1570,
2216
]
]
},
{
"class": 2,
"point": [
[
790,
3404
],
[
1042,
2858
],
[
1042,
2858
],
[
1215,
2220
]
]
},
{
"class": 3,
"point": [
[
1609,
1333
],
[
1570,
2216
],
[
1570,
2216
],
[
1215,
2220
]
]
},
{
"class": 3,
"point": [
[
1129,
1334
],
[
1215,
2220
]
]
},
{
"class": 0,
"point": [
[
2644,
824
],
[
2486,
908
],
[
2486,
908
],
[
2728,
1250
]
]
},
{
"class": 0,
"point": [
[
2708,
882
],
[
2644,
824
]
]
},
{
"class": 1,
"point": [
[
2128,
1285
],
[
2728,
1250
],
[
2728,
1250
],
[
3112,
1609
]
]
},
{
"class": 1,
"point": [
[
2128,
1285
],
[
1674,
1831
],
[
1674,
1831
],
[
1795,
1849
]
]
},
{
"class": 1,
"point": [
[
3112,
1609
],
[
3577,
1660
]
]
},
{
"class": 2,
"point": [
[
3102,
3921
],
[
2838,
3056
],
[
2838,
3056
],
[
2344,
2322
]
]
},
{
"class": 2,
"point": [
[
1453,
3453
],
[
2269,
3347
],
[
2269,
3347
],
[
2535,
2343
]
]
},
{
"class": 3,
"point": [
[
2128,
1285
],
[
2344,
2322
],
[
2344,
2322
],
[
2535,
2343
]
]
},
{
"class": 3,
"point": [
[
2728,
1250
],
[
2535,
2343
]
]
}
]
}{
"type": "object",
"properties": {
"lines": {
"type": "array",
"items": {
"type": "object",
"properties": {
"point": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number",
"minimum": 0
},
"minItems": 2,
"maxItems": 2
},
"minItems": 1
},
"class": {
"type": "integer",
"minimum": 0
}
},
"required": [
"point",
"class"
]
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"lines",
"classes",
"predicted_classes"
]
}{
"classes": [
"noun",
"verb"
],
"lines": [
{
"point": [
[
102.2,
2510
],
[
617.1,
2481.3
]
],
"class": 0
},
{
"point": [
[
1279.1,
1686.5
],
[
1421.3,
1666.9
],
[
1642,
1617.8
],
[
1730.2,
1539.3
],
[
1813.6,
1617.8
]
],
"class": 0
},
{
"point": [
[
1931.3,
2535.3
],
[
2058.8,
2608.9
],
[
2431.5,
2662.9
],
[
2490.3,
2471.5
],
[
2436.4,
2265.4
],
[
2250,
2201.7
],
[
2049,
2236
],
[
2019.6,
2393
],
[
1960.7,
2481.3
]
],
"class": 1
},
{
"point": [
[
514.1,
484.4
],
[
519,
126.3
],
[
1112.4,
234.2
],
[
960.3,
489.3
]
],
"class": 0
}
],
"predicted_classes": [
0,
1
]
}{
"type": "object",
"properties": {
"points": {
"type": "array",
"items": {
"type": "object",
"properties": {
"point": {
"type": "array",
"items": {
"items": {
"type": "number",
"minimum": 0
}
},
"minItems": 2,
"maxItems": 2
},
"class": {
"type": "integer",
"minimum": 0
}
},
"required": [
"point",
"class"
]
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"points",
"classes",
"predicted_classes"
]
}{
"classes": [
"nail",
"finger"
],
"points": [
{
"point": [
554.2,
376.1
],
"class": 0
},
{
"point": [
226.9,
862.2
],
"class": 0
},
{
"point": [
911.6,
331.9
],
"class": 0
},
{
"point": [
1144.5,
460.5
],
"class": 0
},
{
"point": [
1269,
725.6
],
"class": 0
},
{
"point": [
313.2,
1014.9
],
"class": 1
},
{
"point": [
594.4,
719.6
],
"class": 1
},
{
"point": [
590.3,
583
],
"class": 1
},
{
"point": [
554.2,
432.4
],
"class": 1
},
{
"point": [
281.1,
940.6
],
"class": 1
}
],
"predicted_classes": [
0,
1
]
}{
"type": "object",
"properties": {
"categories": {
"type": "array",
"items": {
"type": "object",
"properties": {
"tag": {
"type": "string"
},
"class": {
"type": "integer",
"minimum": 0
},
"id": {
"type": "integer"
}
},
"required": [
"tag",
"class",
"id"
]
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"categories",
"classes",
"predicted_classes"
]
}{
"categories": [
{
"tag": "spam",
"class": 0,
"id": 0
}
],
"classes": [
"spam",
"non-spam"
],
"predicted_classes": [
0
]
}{
"type": "object",
"properties": {
"points": {
"type": "array",
"items": {
"type": "object",
"properties": {
"point": {
"type": "array",
"items": {
"items": {
"type": "number",
"minimum": 0
}
},
"minItems": 2,
"maxItems": 2
},
"class": {
"type": "integer",
"minimum": 0
}
},
"required": [
"point",
"class"
]
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"points",
"classes",
"predicted_classes"
]
}{
"bboxes": [
[
{
"class": 0,
"point": [
[
55.98997389316995,
50.75949633573195
],
[
336.0434088277967,
50.75949633573195
],
[
336.0434088277967,
256.8694140492915
],
[
55.98997389316995,
256.8694140492915
]
],
"recognition": "The"
},
{
"class": 0,
"point": [
[
36.3723060590093,
257.49953428034837
],
[
349.6248711745234,
257.49953428034837
],
[
349.6248711745234,
386.64783202912366
],
[
36.3723060590093,
386.64783202912366
]
],
"recognition": "OCR"
},
{
"class": 0,
"point": [
[
366.85455649602386,
259.0085856522069
],
[
824.9760533099553,
259.0085856522069
],
[
824.9760533099553,
395.7021402602747
],
[
366.85455649602386,
395.7021402602747
]
],
"recognition": "example"
}
]
],
"classes": [
"text"
],
"predicted_classes": [
0
]
}{
"type": "object",
"properties": {
"bboxes": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"point": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number",
"minimum": 0
},
"minItems": 2,
"maxItems": 2
},
"minItems": 4,
"maxItems": 4
},
"class": {
"type": "integer",
"minimum": 0
}
},
"required": [
"point",
"class"
]
}
}
},
"captioning_text": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"text": {
"type": "string"
},
"class": {
"type": "integer"
}
},
"required": [
"text"
]
}
}
},
"classes": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"predicted_classes": {
"type": "array",
"items": {
"type": "integer"
}
}
},
"required": [
"bboxes",
"captioning_text",
"classes",
"predicted_classes"
]
}{
"bboxes": [
[
{
"class": 0,
"point": [
[
39.191678601999406,
167.6251193080077
],
[
293.2911627522001,
167.6251193080077
],
[
293.2911627522001,
860.8851919449099
],
[
39.191678601999406,
860.8851919449099
]
]
},
{
"class": 1,
"point": [
[
307.15407157693585,
152.738319698289
],
[
452.57991857619015,
152.738319698289
],
[
452.57991857619015,
832.6002726864442
],
[
307.15407157693585,
832.6002726864442
]
]
},
{
"class": 2,
"point": [
[
473.8862272057852,
173.57983915189516
],
[
635.68755377573,
173.57983915189516
],
[
635.68755377573,
850.4644322181067
],
[
473.8862272057852,
850.4644322181067
]
]
}
]
],
"captioning_text": [
[
{
"text": "The"
},
{
"text": "image"
},
{
"text": "shows"
},
{
"text": "a"
},
{
"text": "group"
},
{
"text": "of"
},
{
"text": "seven"
},
{
"text": "people",
"class": 0
},
{
"text": "standing"
},
{
"text": "in"
},
{
"text": "a"
},
{
"text": "row"
},
{
"text": "against"
},
{
"text": "a"
},
{
"text": "grey"
},
{
"text": "wall."
}
],
[
{
"text": "The"
},
{
"text": "first"
},
{
"text": "individual",
"class": 0
},
{
"text": "is"
},
{
"text": "a"
},
{
"text": "young"
},
{
"text": "man"
},
{
"text": "wearing"
},
{
"text": "a"
},
{
"text": "long-sleeve"
},
{
"text": "dark"
},
{
"text": "blue"
},
{
"text": "shirt,",
"class": 1
},
{
"text": "blue"
},
{
"text": "jeans,",
"class": 2
},
{
"text": "and"
},
{
"text": "black"
},
{
"text": "shoes.",
"class": 3
}
]
],
"classes": [
"person",
"shirt",
"jeans",
"shoes"
],
"predicted_classes": [
0,
1,
2,
3
]
}
