Segmentation

Name a thing - "person", "red car" - and get back every one of them in the picture, each with a mask, a box and a score. Or click on something and get that one object. Meta's SAM 3, served locally.

Two Different Questions

The endpoint does two jobs, told apart by what the prompt holds, and mixing them is a 400 rather than a guess.

Prompt	Answer
`text`, `boxes`, or both	Every instance of that concept, best first
`points`, `object_box`, or both	The one object under them, as up to three candidate masks

The first is what makes this different from a click-to-select tool: one request finds all of them, not the one you pointed at.

Finding Every One

curl -X POST localhost:11540/v1/masks \
  -H 'Content-Type: application/json' \
  -d '{
    "image": "data:image/jpeg;base64,...",
    "text": "red car",
    "threshold": 0.5
  }'

{
  "instances": [
    {
      "score": 0.94,
      "box": [412.0, 188.5, 735.2, 402.1],
      "area": 48120,
      "mask": {"size": [1080, 1920], "counts": [...]}
    }
  ],
  "timings": {"encode_ms": 61.2, "prompt_ms": 3.1, "image_reused": false}
}

Masks come back as COCO RLE, so anything that already reads COCO annotations reads these. The box is in the picture's own pixels, and the picture is turned upright by its EXIF orientation first, so the coordinates line up with the image as you see it rather than as it happens to be stored.

Field	Meaning
`image`	The picture as a base64 `data:` URI. Up to 24 MP.
`text`	A short noun phrase. Up to 30 tokens.
`boxes`	Example boxes, `{"xyxy": [...], "positive": true}`. Up to 16. A negative box says what not to include.
`threshold`	Keep instances scoring above this. Default 0.5.
`max_instances`	Cap the count, best first. Default: everything kept.

A concept longer than 30 tokens is refused, not truncated. Meta's own processor cuts it silently, which means a long prompt quietly becomes a different prompt. It is also a finder rather than a reasoner: "red car" works, "the car that is parked badly" does not.

Clicking On One Thing

Pass points instead and you get the single object under them, as up to three candidate masks ranked by the model's own predicted overlap. A negative point says "not this".

{
  "image": "data:image/png;base64,...",
  "points": [{"xy": [640.0, 380.0], "positive": true},
             {"xy": [710.0, 395.0], "positive": false}]
}

Note the score means different things between the two tasks. On a concept it is the model's confidence that this is one of the things you named; on clicks it is the model's predicted accuracy of the mask itself.

Asking Again Costs Almost Nothing

Encoding the picture is nearly all the work. Send the same bytes again with a different prompt and the encoding is reused: timings.image_reused says so, and encode_ms is zero. A new prompt on a picture already loaded comes back in milliseconds, which is what makes interactive use practical.

Following Things Through Video

A session runs the detector on every frame and carries each object from frame to frame under one id, so object 3 in frame 90 is the same thing as object 3 in frame 12.

# start, tracking up to four concepts at once
curl -X POST localhost:11540/v1/masks/sessions \
  -d '{"texts": ["person", "bicycle"]}'

# then a frame at a time, every frame the first one's size
curl -X POST localhost:11540/v1/masks/sessions/{id}/frames -d '{"image": "data:..."}'

# end it; the frames still held come back final
curl -X POST localhost:11540/v1/masks/sessions/{id}/finish
curl -X DELETE localhost:11540/v1/masks/sessions/{id}

Output lags the frame you just sent, and deliberately. A frame is final only once 15 more have passed over it, because an object that turns out to be noise in that window is then removed from the earlier frames it appeared on as well. A live view gets the preview meanwhile; the final answer is the one that has had its second look.

Four sessions can be live at once, and one idle for two minutes is dropped.

SAM 3 and SAM 3.1

The engine reads either checkpoint and works out which it has from the weights themselves, so you do not tell it. The two score differently and it accounts for that: SAM 3 puts its overlap head through a sigmoid and SAM 3.1 does not, which makes 3.1's click scores unbounded rather than between 0 and 1. Compare a click score against another from the same checkpoint, not across the two.

Video sessions need SAM 3. SAM 3.1 drops the fixed no-object pointer its tracker would need, and that tracker is not in this build, so a session on a 3.1 checkpoint is refused by name. Single pictures work on both. The catalog ships SAM 3.

Getting The Weights

SAM 3's weights are gated and we do not redistribute them. Accept Meta's licence on the model's Hugging Face page, create a token, and add it under Manager > Settings. The download then comes straight from Hugging Face. A refused download says it was refused rather than reporting a missing file.

Limits

One picture up to 24 MP, a prompt up to 30 tokens, 16 example boxes.
A video session tracks up to four concepts at once.
NVIDIA GPUs only for now.

In the Studio

The Masks page does this without writing a request: drop, paste or open a picture, or turn on the camera and track up to four named things at once, each in its own colour, keeping snapshots of the moments you want. Pictures are kept in a side panel the way Tables and Reads keep theirs, and the result exports as COCO JSON, cut-outs or mask PNGs. See Studio.