Auxiliary AI Service Guide

• Overview
• Design Principles
  • Authentication
  • Request and Response Schemas
• AI Features and API Endpoints
  • Overall description on image load
  • Overall description enhancement
  • Region description on polygon draw
  • Polygon refinement on double click
  • Region referring segmentation on press Enter
  • Chat with AI through OpenAI API proxy

Overview

The MetaDent App supports integration with an Auxiliary AI Service for automatic label generation.

This AI-assisted labeling feature is fully optional. If enabled, the frontend will send structured requests to a user-provided backend service.

INFO

We do not provide a ready-to-use AI backend service, as:

1. The implementation can vary greatly depending on your specific use case and the models you want to use.
2. Our model is not ready for public release yet. We may open-source it in the future after further development and testing.

Instead, we provide a clear API specification and design principles for you to develop your own custom AI service that best fits your needs.

Design Principles

• The frontend does NOT transmit image data, only the image ID is transmitted.
• All endpoints use HTTP POST method and exchange data in body as JSON format. (except the OpenAI proxy endpoint which should follow the OpenAI API spec)

The AI features are designed in a modular way, and the frontend will only call the endpoints that are supported by your backend. Which can be configured in the user settings. This allows you to implement only the features that you need without worrying about the rest.

Below are the API specifications copied from the backend fastapi's pydantic models and endpoint definitions. You can refer to these specifications when implementing your own backend service.

Authentication

The frontend sends a Bearer token at request headers for authentication:

Authorization: Bearer <aiBackendToken>

Your backend may validate this token from this.

Request and Response Schemas

from metadent_tools.model import Polygon
from pydantic import BaseModel, Field

class OverallDescriptionRequest(BaseSchema):
    image_id: str

class OverallDescriptionEnhanceRequest(BaseSchema):
    image_id: str
    overall_description: str

class RegionDescriptionContext(BaseSchema):
    class _Item(BaseSchema):
        description: str
        contours: list[Polygon]
        low_confidence: bool = False

    items: list[_Item] = []
    overall_description: str = ""

class RegionDescriptionRequest(BaseSchema):
    image_id: str
    contours: list[Polygon]
    context: RegionDescriptionContext = Field(default_factory=RegionDescriptionContext)

class RegionRefineRequest(BaseSchema):
    image_id: str
    contours: list[Polygon]

class RegionReferringRequest(BaseSchema):
    image_id: str
    prompt: str

class RegionResponse(BaseSchema):
    image_id: str
    contours: list[Polygon]

class LLMResponse(BaseSchema):
    image_id: str
    choices: list[str]

AI Features and API Endpoints

Below are the API endpoint specifications, using the above request and response schemas.

The first line is the endpoint path, and the following two lines are the request body and response body schema respectively.

Overall description on image load

Automatically generates an overall description for the image on image load.

POST /overall-description 
OverallDescriptionRequest
LLMResponse

Overall description enhancement

This toggles a more advanced overall description generation that takes the initial overall description as input and try to enhance it. It contains two endpoints, one for generating a more complex description, and the other one is for simplifying the overall description into a more concise version.

POST /overall-description-complexify
OverallDescriptionEnhanceRequest
LLMResponse

POST /overall-description-simplify
OverallDescriptionEnhanceRequest
LLMResponse

Region description on polygon draw

Generates a description for the user-selected region after drawing a polygon, with optional overall context information.

POST /region-description
RegionDescriptionRequest
LLMResponse

Polygon refinement on double click

Refines the polygon contour drawn by the user, typically using SAM or similar segmentation models.

POST /region-refine
RegionRefineRequest
RegionResponse

Region referring segmentation on press Enter

Refers to a specific region based on the user prompt, and return the polygon contour for that region.

POST /region-referring
RegionReferringRequest
RegionResponse

Chat with AI through OpenAI API proxy

If your backend provides the OpenAI API proxy endpoint, the frontend can open up a chat interface for users to directly chat with the AI and get responses from the OpenAI API.

[METHOD] /proxy/openai-v1/*
...

Any OpenAI compatible API route should be proxied through this endpoint, for example:

POST /proxy/openai-v1/chat/completions
...

WARNING

Currently the image for chat is send using lfss image url with token included !! So must use a self hosted OpenAI compatible server to avoid potential data leak. Will add a safer image proxy for this in the future.