Compare commits

..
Author SHA1 Message Date
Josh Hawkins 3f91ca32bb Keep the input path when switching stream source modes and reset mode overrides across cameras 2026-07-20 16:35:44 -05:00
Josh Hawkins c98efd4a02 warn when a camera input points at a missing go2rtc stream 2026-07-20 14:00:04 -05:00
Josh Hawkins 38d0fa8b06 clarify transcribe button 2026-07-20 12:29:54 -05:00
Josh Hawkins d7403c1010 tweak 2026-07-20 11:28:09 -05:00
Josh Hawkins 3e0f6c85de update debug info in face rec docs 2026-07-20 11:12:09 -05:00
Nicolas Mowen 45d3c6389e update docs 2026-07-19 17:25:16 -06:00
Nicolas Mowen 761a431c55 Use manual context size if set 2026-07-19 17:22:33 -06:00
Josh Hawkins ce0197cc8f fix semantic search model_size showing dirty for genai embeddings providers
The resolved config always reports model_size (schema default), so clearing it
whenever a provider was selected falsely marked the field dirty on load and sent
a delete for a YAML key that isn't there (Error updating config: 'model_size').
Only clear a non-default value, which is the only case actually present in the
config file.
2026-07-19 06:47:08 -05:00
12 changed files with 874 additions and 465 deletions
+2 -2
View File
@@ -272,7 +272,7 @@ If you have CUDA hardware, you can experiment with the `large` `whisper` model o
#### Transcription and translation of `speech` audio events
Any `speech` events in Explore can be transcribed and/or translated through the Transcribe button in the Tracked Object Details pane.
Any `speech` events in Explore can be transcribed and/or translated through the Transcribe button (the microphone icon) in the Tracked Object Details pane.
In order to use transcription and translation for past events, you must enable audio detection and define `speech` as an audio type to listen for. To have `speech` events translated into the language of your choice, set the `language` config parameter with the correct [language code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10).
@@ -294,7 +294,7 @@ Recorded `speech` events will always use a `whisper` model, regardless of the `m
Because transcription is **serialized (one event at a time)** and speech events can be generated far faster than they can be processed, an auto-transcribe toggle would very quickly create an ever-growing backlog and degrade core functionality. For the amount of engineering and risk involved, it adds **very little practical value** for the majority of deployments, which are often on low-powered, edge hardware.
If you hear speech that's actually important and worth saving/indexing for the future, **just press the transcribe button in Explore** on that specific `speech` event - that keeps things explicit, reliable, and under your control.
If you hear speech that's actually important and worth saving/indexing for the future, **just press the transcribe button (the microphone icon) in Explore** on that specific `speech` event - that keeps things explicit, reliable, and under your control.
Other options are being considered for future versions of Frigate to add transcription options that support external `whisper` Docker containers. A single transcription service could then be shared by Frigate and other applications (for example, Home Assistant Voice), and run on more powerful machines when available.
+16 -2
View File
@@ -232,7 +232,21 @@ Once front-facing images are performing well, start choosing slightly off-angle
Start with the [Usage](#usage) section and re-read the [Model Requirements](#model-requirements) above.
1. Ensure `person` is being _detected_. A `person` will automatically be scanned by Frigate for a face. Any detected faces will appear in the Recent Recognitions tab in the Frigate UI's Face Library.
1. Enable debug logs to see exactly what Frigate is doing.
- Enable debug logs for face recognition by adding `frigate.data_processing.real_time.face: debug` to your `logger` configuration. Restart Frigate after this change.
```yaml
logger:
default: info
logs:
# highlight-next-line
frigate.data_processing.real_time.face: debug
```
- These logs report where the pipeline stopped for each `person` object, such as no face being found within the person's bounding box, the detected face being smaller than `min_area`, or a face being recognized but scoring too low.
- If you see no face-related messages at all, also add `frigate.embeddings.maintainer: debug` to confirm that the face processor was created at startup and that `person` updates are reaching it.
2. Ensure `person` is being _detected_. A `person` will automatically be scanned by Frigate for a face. Any detected faces will appear in the Recent Recognitions tab in the Frigate UI's Face Library.
If you are using a Frigate+ or `face` detecting model:
- Watch the [debug view](/usage/live#the-single-camera-view) to ensure that `face` is being detected along with `person`.
@@ -242,7 +256,7 @@ Start with the [Usage](#usage) section and re-read the [Model Requirements](#mod
- Check your `detect` stream resolution and ensure it is sufficiently high enough to capture face details on `person` objects.
- You may need to lower your `detection_threshold` if faces are not being detected.
2. Any detected faces will then be _recognized_.
3. Any detected faces will then be _recognized_.
- Make sure you have trained at least one face per the recommendations above.
- Adjust `recognition_threshold` settings per the suggestions [above](#advanced-configuration).
+4 -2
View File
@@ -78,7 +78,7 @@ All llama.cpp native options can be passed through `provider_options`, including
- Set **Provider** to `llamacpp`
- Set **Base URL** to your llama.cpp server address (e.g., `http://localhost:8080`)
- Set **Model** to the name of your model
- Under **Provider Options**, set `context_size` to tell Frigate your context size so it can send the appropriate amount of information
- Optionally, under **Provider Options**, set `context_size` to override the context size Frigate detects from the server
</TabItem>
<TabItem value="yaml">
@@ -89,12 +89,14 @@ genai:
base_url: http://localhost:8080
model: your-model-name
provider_options:
context_size: 16000 # Tell Frigate your context size so it can send the appropriate amount of information.
context_size: 16000 # Optional, overrides the context size reported by the server.
```
</TabItem>
</ConfigTabs>
Frigate queries the llama.cpp server for the model's context size at startup and logs it along with the other detected capabilities. If `context_size` is set in `provider_options`, that value is always used instead, even when the server reports its own.
### Ollama
[Ollama](https://ollama.com/) allows you to self-host large language models and keep everything running locally. It is highly recommended to host this server on a machine with an Nvidia graphics card, or on a Apple silicon Mac for best performance.
+1 -1
View File
@@ -192,7 +192,7 @@ class LlamaCppClient(GenAIClient):
logger.info(
"llama.cpp model '%s' initialized — context: %s, vision: %s, audio: %s, tools: %s, reasoning: %s",
configured_model,
self._context_size or "unknown",
self.get_context_size(),
self._supports_vision,
self._supports_audio,
self._supports_tools,
+28
View File
@@ -491,6 +491,34 @@ class TestLlamaCppProvider(unittest.TestCase):
final = _final_message(self._run_with_lines(client, lines, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
def _validated_client(self, server_context_size, provider_options=None):
"""Build a client as if the server reported the given context size."""
cfg = GenAIConfig(
provider="llamacpp",
model="m",
base_url="http://localhost:9999",
provider_options=provider_options or {},
)
info = {
"context_size": server_context_size,
"supports_vision": False,
"supports_audio": False,
"supports_tools": False,
"supports_reasoning": False,
"media_marker": "<__media__>",
}
cls = PROVIDERS[GenAIProviderEnum.llamacpp]
with patch.object(cls, "_get_model_info", return_value=info):
return cls(cfg, timeout=5)
def test_server_context_size_used_without_override(self):
client = self._validated_client(4096)
self.assertEqual(client.get_context_size(), 4096)
def test_provider_options_context_size_overrides_server(self):
client = self._validated_client(4096, {"context_size": 32768})
self.assertEqual(client.get_context_size(), 32768)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,129 @@
/**
* Semantic Search settings tests -- MEDIUM tier.
*
* Focuses on the model_size field, which is unused when a GenAI embeddings
* provider is selected as the semantic search model. The resolved config always
* reports model_size (it has a schema default of "small"), even when the YAML
* file has no such key. Clearing model_size for a provider used to run
* unconditionally, which falsely marked the section dirty on load and asked the
* backend to delete a key that wasn't in the config file (KeyError: 'model_size').
*/
import { readFileSync } from "node:fs";
import { resolve, dirname } from "node:path";
import { fileURLToPath } from "node:url";
import { test, expect } from "../../fixtures/frigate-test";
import type { Page } from "@playwright/test";
import { configFactory } from "../../fixtures/mock-data/config";
const __dirname = dirname(fileURLToPath(import.meta.url));
const CONFIG_SCHEMA = JSON.parse(
readFileSync(
resolve(__dirname, "../../fixtures/mock-data/config-schema.json"),
"utf-8",
),
);
const PROVIDER = "llama_cpp";
const SETTINGS_URL = "/settings?page=integrationSemanticSearch";
const NOT_APPLICABLE = "Not applicable for GenAI providers";
const UNSAVED = "You have unsaved changes";
type SemanticSearch = {
enabled?: boolean;
model?: string;
model_size?: string;
};
async function installRoutes(page: Page, semanticSearch: SemanticSearch) {
const config = configFactory({
genai: { [PROVIDER]: { provider: PROVIDER, roles: ["embeddings"] } },
semantic_search: semanticSearch,
});
let lastSavedConfig: unknown = null;
await page.route("**/api/config/schema.json", (route) =>
route.fulfill({ json: CONFIG_SCHEMA }),
);
await page.route("**/api/config", (route) => {
if (route.request().method() === "GET") {
return route.fulfill({ json: config });
}
return route.fulfill({ json: { success: true } });
});
await page.route("**/api/config/set", async (route) => {
lastSavedConfig = route.request().postDataJSON();
await route.fulfill({ json: { success: true, require_restart: false } });
});
await page.route("**/api/config/raw_paths", (route) =>
route.fulfill({ json: { semantic_search: semanticSearch } }),
);
return { capturedConfig: () => lastSavedConfig };
}
test.describe("semantic search model_size @medium", () => {
test("a provider with a defaulted model_size is not dirty on load", async ({
frigateApp,
}) => {
// model_size stays at its schema default ("small"), i.e. it is not present
// in the YAML. This mirrors the reported bug: selecting a GenAI provider and
// returning to the page.
await installRoutes(frigateApp.page, {
enabled: true,
model: PROVIDER,
});
await frigateApp.goto(SETTINGS_URL);
// The provider path is active: model_size shows "Not applicable".
await expect(frigateApp.page.getByText(NOT_APPLICABLE)).toBeVisible();
// Give any clearing effect time to fire, then confirm the section stayed
// clean (no phantom unsaved-changes banner, Save disabled).
await frigateApp.page.waitForTimeout(1000);
await expect(frigateApp.page.getByText(UNSAVED)).toBeHidden();
await expect(
frigateApp.page.getByRole("button", { name: "Save", exact: true }),
).toBeDisabled();
});
test("switching from a configured non-default model_size clears it", async ({
frigateApp,
}) => {
// A genuinely configured non-default model_size ("large") can only come from
// the YAML, so switching to a provider must still remove it.
const capture = await installRoutes(frigateApp.page, {
enabled: true,
model: "jinav2",
model_size: "large",
});
await frigateApp.goto(SETTINGS_URL);
// Starts clean on a Jina model.
await expect(frigateApp.page.getByText(UNSAVED)).toBeHidden();
// Switch the model to the GenAI provider.
await frigateApp.page
.getByRole("combobox", { name: /Semantic search model/ })
.click();
await frigateApp.page.getByRole("option", { name: PROVIDER }).click();
// The change is now dirty and model_size is no longer applicable.
await expect(frigateApp.page.getByText(NOT_APPLICABLE)).toBeVisible();
await expect(frigateApp.page.getByText(UNSAVED)).toBeVisible();
await frigateApp.page
.getByRole("button", { name: "Save", exact: true })
.click();
// The saved payload removes model_size (empty string = "remove" key).
await expect
.poll(() => capture.capturedConfig(), { timeout: 5_000 })
.toMatchObject({
config_data: {
semantic_search: { model: PROVIDER, model_size: "" },
},
});
});
});
+643 -436
View File
File diff suppressed because it is too large Load Diff
+4 -4
View File
@@ -112,14 +112,14 @@
"@types/react": "^19.2.14",
"@types/react-dom": "^19.2.3",
"@types/strftime": "^0.9.8",
"@typescript-eslint/eslint-plugin": "^8.65.0",
"@typescript-eslint/parser": "^8.65.0",
"@typescript-eslint/eslint-plugin": "^7.5.0",
"@typescript-eslint/parser": "^7.5.0",
"@vitejs/plugin-react-swc": "^3.8.0",
"@vitest/coverage-v8": "^3.0.7",
"autoprefixer": "^10.4.20",
"eslint": "^10.7.0",
"eslint": "^8.57.0",
"eslint-config-prettier": "^9.1.0",
"eslint-plugin-jest": "^29.15.5",
"eslint-plugin-jest": "^28.2.0",
"eslint-plugin-prettier": "^5.0.1",
"eslint-plugin-react-hooks": "^5.2.0",
"eslint-plugin-react-refresh": "^0.4.8",
+2 -1
View File
@@ -1936,7 +1936,8 @@
"inputDimensionsNotDetectResolution": "Model input width and height are the input dimensions of the object detection model, not your camera's detect resolution. They should match the dimensions of the model you're using — typically a square size like 320x320 or 640x640."
},
"ffmpeg": {
"hwaccelManualNotRecommended": "Manual hardware acceleration arguments are not recommended. Unless a specific requirement exists, select the preset that matches your hardware."
"hwaccelManualNotRecommended": "Manual hardware acceleration arguments are not recommended. Unless a specific requirement exists, select the preset that matches your hardware.",
"inputsMissingGo2rtcStream": "An input below points at a go2rtc restream that no longer exists. Select an existing restream or enter the camera's URL manually, otherwise this camera will fail to connect."
},
"objects": {
"genaiNoDescriptionsProvider": "You must configure a GenAI provider with the 'descriptions' role for descriptions to be generated."
@@ -1,3 +1,4 @@
import { parseRestreamStreamName } from "../theme/fields/streamSource";
import type { SectionConfigOverrides } from "./types";
const arrayAsTextWidget = {
@@ -42,6 +43,29 @@ const ffmpeg: SectionConfigOverrides = {
return false;
},
},
{
key: "inputs-missing-go2rtc-stream",
field: "inputs",
position: "before",
messageKey: "configMessages.ffmpeg.inputsMissingGo2rtcStream",
severity: "warning",
docLink: "/configuration/restream",
condition: (ctx) => {
const streams = ctx.fullConfig?.go2rtc?.streams;
const inputs = ctx.formData?.inputs;
if (!Array.isArray(inputs)) {
return false;
}
return inputs.some((input) => {
const path = (input as { path?: unknown } | null)?.path;
const streamName = parseRestreamStreamName(
typeof path === "string" ? path : undefined,
);
return streamName !== undefined && !(streamName in (streams ?? {}));
});
},
},
],
fieldDocs: {
hwaccel_args: "/configuration/ffmpeg_presets#hwaccel-presets",
@@ -170,6 +170,12 @@ export function CameraInputsField(props: FieldProps) {
[go2rtcStreamNames],
);
useEffect(() => {
setSourceModeByIndex((previous) =>
Object.keys(previous).length > 0 ? {} : previous,
);
}, [formContext?.cameraName]);
useEffect(() => {
setOpenByIndex((previous) => {
const next: Record<number, boolean> = {};
@@ -222,18 +228,12 @@ export function CameraInputsField(props: FieldProps) {
const handleSourceModeChange = useCallback(
(index: number, nextMode: StreamSourceMode) => {
const input = inputs[index];
const currentPath =
typeof input?.path === "string" ? input.path : undefined;
if (nextMode === "manual") {
// Only revert the preset we set ourselves; never clobber custom args.
if (input?.input_args === RESTREAM_PRESET) {
handleFieldValuesChange(index, { input_args: undefined });
}
} else if (!parseRestreamStreamName(currentPath)) {
// Entering restream with a non-restream path: clear it so the dropdown
// shows its placeholder until a stream is chosen.
handleFieldValuesChange(index, { path: undefined });
// Only revert the preset we set ourselves; never clobber custom args.
// The path is left alone until a stream is picked, so switching modes
// never discards a typed URL or empties a required field.
if (nextMode === "manual" && input?.input_args === RESTREAM_PRESET) {
handleFieldValuesChange(index, { input_args: undefined });
}
setSourceModeByIndex((previous) => ({ ...previous, [index]: nextMode }));
@@ -24,15 +24,19 @@ export function SemanticSearchModelSizeWidget(props: WidgetProps) {
model !== "jinav1" &&
model !== "jinav2";
// Clear model_size while on a provider (buildOverrides converts to ""
// which the backend treats as "remove"). Restore the schema default
// when returning to a Jina model so the field isn't left empty.
// model_size is unused on a GenAI provider. Only clear it (which the backend
// treats as "remove") for a non-default value, which can only come from the
// config file. A defaulted value is indistinguishable from unset in the
// resolved config, so clearing it would falsely dirty the field and delete a
// YAML key that isn't there. Restore the default when returning to a Jina model.
const { value, onChange, schema } = props;
const schemaDefault = schema?.default as string | undefined;
useEffect(() => {
if (isProvider && value !== undefined) {
onChange(undefined);
} else if (!isProvider && value === undefined && schemaDefault) {
if (isProvider) {
if (value !== undefined && value !== schemaDefault) {
onChange(undefined);
}
} else if (value === undefined && schemaDefault) {
onChange(schemaDefault);
}
}, [isProvider, value, onChange, schemaDefault]);