Compare commits

...
3 Commits
Author SHA1 Message Date
mohitandGitHub c6ceed5a8f Merge ee823f8488 into cbf6d032cb 2026-06-25 23:45:25 +00:00
Mohit Varikuti ee823f8488 fix(genai): drop twelvelabs SDK dep, call Marengo via REST 2026-06-25 16:45:21 -07:00
Josh HawkinsandGitHub cbf6d032cb Add optional docs link to config field messages (#23569)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
* add optional docs link to config field messages

* docs tweaks

* add field messages for model dimensions
2026-06-25 17:25:59 -06:00
10 changed files with 182 additions and 84 deletions
-1
View File
@@ -50,7 +50,6 @@ transformers == 4.45.*
google-genai == 1.58.*
ollama == 0.6.*
openai == 1.65.*
twelvelabs == 1.2.*
# push notifications
py-vapid == 1.9.*
pywebpush == 2.0.*
@@ -73,6 +73,22 @@ This does not affect using hardware for accelerating other tasks such as [semant
:::
### Choosing a model size
Along with picking a detector for your hardware, you will choose a model's **input resolution** (such as `320x320` or `640x640`) and, for model families like YOLOv9, a **variant size** (`tiny`, `small`, etc.). Both affect the balance between accuracy and the inference time your hardware can sustain.
**Resolution (320x320 vs 640x640):** Frigate is optimized for `320x320` models, and `320x320` is the best choice for the vast majority of setups. Frigate is specifically designed to compensate for the smaller model by cropping a region of motion from the full frame and zooming into it before running detection, so a `320x320` model is actually _better_ at small and distant objects — not worse. A `640x640` model is slower and uses more resources, and its main benefit is fitting more objects into a single inference when many objects are spread across a large area. Recent versions of Frigate have improved support for `640x640` models, but `320x320` remains the recommended starting point for nearly all setups.
**Variant size (tiny/small/medium):** Larger variants are gradually more accurate but slower. Whether the difference is noticeable depends on your specific cameras and scenes. A good rule of thumb is to use the largest model your hardware can run without skipping detections, which you can monitor on the <NavPath path="System > Metrics > Cameras" /> page in the UI — better accuracy only helps if your detector keeps up with the detection load across all cameras.
**Acceptable inference time depends on your hardware.** Inference time alone does not tell the whole story, because different hardware has different capacity. A GPU can run multiple instances of the same model concurrently, so an inference time around 30ms can still keep up with several cameras. A Google Coral runs only a single instance of the model, so it needs a much lower inference time (around 10ms) to keep up.
:::tip
The best detection accuracy comes from a model trained on images that look like what Frigate actually sees — security camera footage cropped to regions of interest. You can train or fine-tune your own model on images like this and run it as a custom model (see the per-detector sections below), but [Frigate+](/plus) makes this much easier by handling the training for you on images submitted from your own cameras. For YOLOv9, the `s` (small) variant at `320x320` resolution is a good place to start.
:::
# Officially Supported Detectors
Frigate provides a number of builtin detector types. By default, Frigate will use a single CPU detector. Other detectors may require additional configuration as described below. When using multiple detectors they will run in dedicated processes, but pull from a common queue of detection requests from across all cameras.
+1 -1
View File
@@ -29,7 +29,7 @@ If you see **"No recordings found for this time"**, the most common causes are:
A toggle (a drawer on mobile) switches the side panel between three modes:
- **Timeline** — a scrubbable vertical timeline of the selected camera, annotated with a motion line, review-item markers, and gaps where no recording exists.
- **Timeline** — a scrubbable vertical timeline of the selected camera. Horizontal lines down the center represent motion, with longer lines indicating more motion at that moment. Review items are marked as shaded areas (**red** for alerts, **orange** for detections), and sections with no colored background are times when no recording exists.
- **Events** — a scrollable list of the camera's review items for the time range; clicking one seeks the player to it.
- **Detail** — the [tracking details inspector](#the-detail-view) for the objects in view.
+46 -27
View File
@@ -20,6 +20,7 @@ upstream and is consistent for both text and image inputs.
import logging
import numpy as np
import requests
from frigate.config import GenAIProviderEnum
from frigate.genai import GenAIClient, register_genai_provider
@@ -29,26 +30,28 @@ logger = logging.getLogger(__name__)
# Default Marengo model. Overridable via the `model` config field.
DEFAULT_MODEL = "marengo3.0"
# Marengo embed REST endpoint. No SDK is needed — this is a plain multipart POST
# made through Frigate's existing `requests` dependency.
EMBED_URL = "https://api.twelvelabs.io/v1.3/embed"
@register_genai_provider(GenAIProviderEnum.twelvelabs)
class TwelveLabsClient(GenAIClient):
"""GenAI client for Frigate using TwelveLabs Marengo embeddings."""
def _init_provider(self):
"""Initialize the TwelveLabs SDK client."""
try:
from twelvelabs import TwelveLabs
except ImportError:
logger.error(
"The twelvelabs package is required for the TwelveLabs provider."
)
return None
"""Validate config for the TwelveLabs REST provider.
The provider is just an HTTPS API, so there is no client object to
build — the API key is the only thing required. A non-None sentinel is
returned so the shared ``ensure_provider``/initialization machinery
treats the provider as available.
"""
if not self.genai_config.api_key:
logger.error("TwelveLabs provider requires an api_key.")
return None
return TwelveLabs(api_key=self.genai_config.api_key)
return self.genai_config.api_key
@property
def _model(self) -> str:
@@ -69,6 +72,8 @@ class TwelveLabsClient(GenAIClient):
sent one at a time. Returns one 512-dim float32 vector per input, in
order (texts first, then images). The shared GenAIEmbedding adapter
pads these to Frigate's 768-dim search schema.
Calls the Marengo REST endpoint directly via ``requests`` — no SDK.
"""
if self.provider is None:
logger.warning(
@@ -93,28 +98,42 @@ class TwelveLabsClient(GenAIClient):
def _embed_one(
self, text: str | None = None, image: bytes | None = None
) -> np.ndarray | None:
"""Embed a single text or image input, returning a float32 vector."""
try:
if text is not None:
response = self.provider.embed.create(
model_name=self._model,
text=text,
request_options={"timeout_in_seconds": self.timeout},
)
result = response.text_embedding
else:
response = self.provider.embed.create(
model_name=self._model,
image_file=image,
request_options={"timeout_in_seconds": self.timeout},
)
result = response.image_embedding
"""Embed a single text or image input, returning a float32 vector.
if result is None or not result.segments:
Posts a multipart form to the Marengo embed endpoint (``model_name`` plus
either a ``text`` or an ``image_file`` part). The endpoint requires
multipart/form-data, so every field — including text — is passed via
``files`` (the ``(None, value)`` form makes requests emit a multipart
text part). ``self.provider`` holds the validated API key. The 512-dim
vector is at ``<text|image>_embedding.segments[0].float`` in the JSON
response.
"""
headers = {"x-api-key": self.provider}
files: dict = {"model_name": (None, self._model)}
if text is not None:
files["text"] = (None, text)
result_key = "text_embedding"
else:
files["image_file"] = ("image.jpg", image, "image/jpeg")
result_key = "image_embedding"
try:
response = requests.post(
EMBED_URL,
headers=headers,
files=files,
timeout=self.timeout,
)
response.raise_for_status()
result = response.json().get(result_key) or {}
segments = result.get("segments") or []
if not segments:
logger.warning("TwelveLabs returned no embedding for input.")
return None
return np.array(result.segments[0].float_, dtype=np.float32)
return np.array(segments[0]["float"], dtype=np.float32)
except Exception as e:
logger.warning("TwelveLabs returned an error: %s", e)
return None
+46 -52
View File
@@ -3,7 +3,7 @@
import io
import os
import unittest
from unittest.mock import MagicMock
from unittest.mock import MagicMock, patch
import numpy as np
@@ -24,91 +24,85 @@ def _make_config(model: str = "") -> GenAIConfig:
)
def _segment(values):
"""Mimic the SDK BaseSegment shape (a `float_` list per segment)."""
seg = MagicMock()
seg.float_ = values
return seg
def _response(key: str, values):
"""Mimic the Marengo REST JSON: ``{<key>: {segments: [{float: [...]}]}}``."""
resp = MagicMock()
resp.raise_for_status.return_value = None
resp.json.return_value = {key: {"segments": [{"float": values}]}}
return resp
class TestTwelveLabsEmbedNoNetwork(unittest.TestCase):
"""Unit tests with the SDK client mocked — no network access."""
"""Unit tests with ``requests`` mocked — no network access, no SDK."""
def _client_with_provider(self, provider) -> TwelveLabsClient:
def _client(self) -> TwelveLabsClient:
client = TwelveLabsClient.__new__(TwelveLabsClient)
client.genai_config = _make_config()
client.timeout = 120
client.provider = provider
client.provider = "test-key"
return client
def test_text_embedding_returns_vector(self):
provider = MagicMock()
response = MagicMock()
response.text_embedding.segments = [_segment([0.1, 0.2, 0.3])]
provider.embed.create.return_value = response
@patch("frigate.genai.plugins.twelvelabs.requests.post")
def test_text_embedding_returns_vector(self, post):
post.return_value = _response("text_embedding", [0.1, 0.2, 0.3])
client = self._client_with_provider(provider)
out = client.embed(texts=["a person walking a dog"])
out = self._client().embed(texts=["a person walking a dog"])
self.assertEqual(len(out), 1)
self.assertIsInstance(out[0], np.ndarray)
self.assertEqual(out[0].dtype, np.float32)
np.testing.assert_allclose(out[0], [0.1, 0.2, 0.3], rtol=1e-6)
_, kwargs = provider.embed.create.call_args
self.assertEqual(kwargs["model_name"], DEFAULT_MODEL)
self.assertEqual(kwargs["text"], "a person walking a dog")
_, kwargs = post.call_args
self.assertEqual(kwargs["files"]["model_name"][1], DEFAULT_MODEL)
self.assertEqual(kwargs["files"]["text"][1], "a person walking a dog")
self.assertEqual(kwargs["headers"]["x-api-key"], "test-key")
self.assertNotIn("image_file", kwargs["files"])
def test_image_embedding_uses_image_file(self):
provider = MagicMock()
response = MagicMock()
response.image_embedding.segments = [_segment([1.0, 2.0])]
provider.embed.create.return_value = response
@patch("frigate.genai.plugins.twelvelabs.requests.post")
def test_image_embedding_uses_image_file(self, post):
post.return_value = _response("image_embedding", [1.0, 2.0])
client = self._client_with_provider(provider)
out = client.embed(images=[b"\xff\xd8\xff jpeg bytes"])
out = self._client().embed(images=[b"\xff\xd8\xff jpeg bytes"])
self.assertEqual(len(out), 1)
_, kwargs = provider.embed.create.call_args
self.assertEqual(kwargs["image_file"], b"\xff\xd8\xff jpeg bytes")
self.assertNotIn("text", kwargs)
_, kwargs = post.call_args
self.assertEqual(kwargs["files"]["image_file"][1], b"\xff\xd8\xff jpeg bytes")
self.assertNotIn("text", kwargs["files"])
def test_custom_model_name_is_used(self):
provider = MagicMock()
response = MagicMock()
response.text_embedding.segments = [_segment([0.0])]
provider.embed.create.return_value = response
@patch("frigate.genai.plugins.twelvelabs.requests.post")
def test_custom_model_name_is_used(self, post):
post.return_value = _response("text_embedding", [0.0])
client = self._client_with_provider(provider)
client = self._client()
client.genai_config = _make_config(model="marengo-custom")
client.embed(texts=["x"])
_, kwargs = provider.embed.create.call_args
self.assertEqual(kwargs["model_name"], "marengo-custom")
_, kwargs = post.call_args
self.assertEqual(kwargs["files"]["model_name"][1], "marengo-custom")
def test_empty_segments_are_skipped(self):
provider = MagicMock()
response = MagicMock()
response.text_embedding = None
provider.embed.create.return_value = response
@patch("frigate.genai.plugins.twelvelabs.requests.post")
def test_empty_segments_are_skipped(self, post):
resp = MagicMock()
resp.raise_for_status.return_value = None
resp.json.return_value = {"text_embedding": {"segments": []}}
post.return_value = resp
client = self._client_with_provider(provider)
self.assertEqual(client.embed(texts=["x"]), [])
self.assertEqual(self._client().embed(texts=["x"]), [])
def test_api_error_is_swallowed(self):
provider = MagicMock()
provider.embed.create.side_effect = RuntimeError("boom")
@patch("frigate.genai.plugins.twelvelabs.requests.post")
def test_api_error_is_swallowed(self, post):
post.side_effect = RuntimeError("boom")
client = self._client_with_provider(provider)
self.assertEqual(client.embed(texts=["x"]), [])
self.assertEqual(self._client().embed(texts=["x"]), [])
def test_no_provider_returns_empty(self):
client = self._client_with_provider(None)
client = self._client()
client.provider = None
self.assertEqual(client.embed(texts=["x"]), [])
def test_no_inputs_returns_empty(self):
client = self._client_with_provider(MagicMock())
self.assertEqual(client.embed(), [])
self.assertEqual(self._client().embed(), [])
@unittest.skipUnless(
@@ -1924,6 +1924,10 @@
"resolutionHigh": "This detect resolution is higher than recommended and may cause increased resource usage without improving detection accuracy. A detect resolution at or below 1080p is recommended for most cameras.",
"globalResolutionMultipleCameras": "A global detect resolution is set while multiple cameras are configured. Unless all cameras share the same resolution and aspect ratio, the detect width and height should be defined per camera to match each camera's native aspect ratio."
},
"model": {
"optimizedFor320": "Frigate is optimized for a 320x320 model, which is the best choice for most setups. A 640x640 model is slower and only helps in specific scenarios.",
"inputDimensionsNotDetectResolution": "Model input width and height are the input dimensions of the object detection model, not your camera's detect resolution. They should match the dimensions of the model you're using — typically a square size like 320x320 or 640x640."
},
"ffmpeg": {
"hwaccelManualNotRecommended": "Manual hardware acceleration arguments are not recommended. Unless a specific requirement exists, select the preset that matches your hardware."
},
@@ -1,6 +1,13 @@
import { useTranslation } from "react-i18next";
import { Link } from "react-router-dom";
import { Alert, AlertDescription } from "@/components/ui/alert";
import { LuInfo, LuTriangleAlert, LuCircleAlert } from "react-icons/lu";
import {
LuInfo,
LuTriangleAlert,
LuCircleAlert,
LuExternalLink,
} from "react-icons/lu";
import { useDocDomain } from "@/hooks/use-doc-domain";
import type { MessageSeverity } from "./section-configs/types";
const severityVariantMap: Record<
@@ -28,13 +35,18 @@ function SeverityIcon({ severity }: { severity: string }) {
type ConfigFieldMessageProps = {
messageKey: string;
severity: string;
values?: Record<string, unknown>;
docLink?: string;
};
export function ConfigFieldMessage({
messageKey,
severity,
values,
docLink,
}: ConfigFieldMessageProps) {
const { t } = useTranslation("views/settings");
const { t } = useTranslation(["views/settings", "common"]);
const { getLocaleDocUrl } = useDocDomain();
return (
<Alert
@@ -42,7 +54,22 @@ export function ConfigFieldMessage({
className="flex items-center [&>svg+div]:translate-y-0 [&>svg]:static [&>svg~*]:pl-2"
>
<SeverityIcon severity={severity} />
<AlertDescription>{t(messageKey)}</AlertDescription>
<AlertDescription>
<div className="flex flex-col gap-1">
<span>{t(messageKey, values)}</span>
{docLink && (
<Link
to={getLocaleDocUrl(docLink)}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center underline"
>
{t("readTheDocumentation", { ns: "common" })}
<LuExternalLink className="ml-2 inline-flex size-3" />
</Link>
)}
</div>
</AlertDescription>
</Alert>
);
}
@@ -3,6 +3,39 @@ import type { SectionConfigOverrides } from "./types";
const model: SectionConfigOverrides = {
base: {
sectionDocs: "/configuration/object_detectors#model",
fieldMessages: [
{
key: "model-optimized-for-320",
field: "width",
position: "before",
messageKey: "configMessages.model.optimizedFor320",
severity: "info",
docLink: "/configuration/object_detectors#choosing-a-model-size",
condition: (ctx) => {
const width = ctx.formData?.width as number | null | undefined;
const height = ctx.formData?.height as number | null | undefined;
return width === 640 || height === 640;
},
},
{
key: "model-input-dimensions-not-detect-resolution",
field: "height",
position: "after",
messageKey: "configMessages.model.inputDimensionsNotDetectResolution",
severity: "warning",
condition: (ctx) => {
const width = ctx.formData?.width as number | null | undefined;
const height = ctx.formData?.height as number | null | undefined;
if (typeof width !== "number" || typeof height !== "number") {
return false;
}
if (width <= 0 || height <= 0) {
return false;
}
return width > 640 || height > 640;
},
},
],
restartRequired: [
"path",
"labelmap_path",
@@ -26,6 +26,8 @@ export type ConditionalMessage = {
condition: (ctx: MessageConditionContext) => boolean;
/** Optional interpolation values passed to t() for {{var}} substitution */
values?: Record<string, unknown>;
/** Optional documentation path (e.g. "/configuration/object_detectors#model"). */
docLink?: string;
};
/** Field-level conditional message, adds field targeting */
@@ -406,6 +406,8 @@ export function FieldTemplate(props: FieldTemplateProps) {
key={m.key}
messageKey={m.messageKey}
severity={m.severity}
values={m.values}
docLink={m.docLink}
/>
))}
</div>
@@ -418,6 +420,8 @@ export function FieldTemplate(props: FieldTemplateProps) {
key={m.key}
messageKey={m.messageKey}
severity={m.severity}
values={m.values}
docLink={m.docLink}
/>
))}
</div>