mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-07-21 11:19:02 +03:00
Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f91ca32bb | ||
|
|
c98efd4a02 | ||
|
|
38d0fa8b06 | ||
|
|
d7403c1010 | ||
|
|
3e0f6c85de | ||
|
|
45d3c6389e | ||
|
|
761a431c55 | ||
|
|
ce0197cc8f |
@@ -272,7 +272,7 @@ If you have CUDA hardware, you can experiment with the `large` `whisper` model o
|
||||
|
||||
#### Transcription and translation of `speech` audio events
|
||||
|
||||
Any `speech` events in Explore can be transcribed and/or translated through the Transcribe button in the Tracked Object Details pane.
|
||||
Any `speech` events in Explore can be transcribed and/or translated through the Transcribe button (the microphone icon) in the Tracked Object Details pane.
|
||||
|
||||
In order to use transcription and translation for past events, you must enable audio detection and define `speech` as an audio type to listen for. To have `speech` events translated into the language of your choice, set the `language` config parameter with the correct [language code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10).
|
||||
|
||||
@@ -294,7 +294,7 @@ Recorded `speech` events will always use a `whisper` model, regardless of the `m
|
||||
|
||||
Because transcription is **serialized (one event at a time)** and speech events can be generated far faster than they can be processed, an auto-transcribe toggle would very quickly create an ever-growing backlog and degrade core functionality. For the amount of engineering and risk involved, it adds **very little practical value** for the majority of deployments, which are often on low-powered, edge hardware.
|
||||
|
||||
If you hear speech that's actually important and worth saving/indexing for the future, **just press the transcribe button in Explore** on that specific `speech` event - that keeps things explicit, reliable, and under your control.
|
||||
If you hear speech that's actually important and worth saving/indexing for the future, **just press the transcribe button (the microphone icon) in Explore** on that specific `speech` event - that keeps things explicit, reliable, and under your control.
|
||||
|
||||
Other options are being considered for future versions of Frigate to add transcription options that support external `whisper` Docker containers. A single transcription service could then be shared by Frigate and other applications (for example, Home Assistant Voice), and run on more powerful machines when available.
|
||||
|
||||
|
||||
@@ -232,7 +232,21 @@ Once front-facing images are performing well, start choosing slightly off-angle
|
||||
|
||||
Start with the [Usage](#usage) section and re-read the [Model Requirements](#model-requirements) above.
|
||||
|
||||
1. Ensure `person` is being _detected_. A `person` will automatically be scanned by Frigate for a face. Any detected faces will appear in the Recent Recognitions tab in the Frigate UI's Face Library.
|
||||
1. Enable debug logs to see exactly what Frigate is doing.
|
||||
- Enable debug logs for face recognition by adding `frigate.data_processing.real_time.face: debug` to your `logger` configuration. Restart Frigate after this change.
|
||||
|
||||
```yaml
|
||||
logger:
|
||||
default: info
|
||||
logs:
|
||||
# highlight-next-line
|
||||
frigate.data_processing.real_time.face: debug
|
||||
```
|
||||
|
||||
- These logs report where the pipeline stopped for each `person` object, such as no face being found within the person's bounding box, the detected face being smaller than `min_area`, or a face being recognized but scoring too low.
|
||||
- If you see no face-related messages at all, also add `frigate.embeddings.maintainer: debug` to confirm that the face processor was created at startup and that `person` updates are reaching it.
|
||||
|
||||
2. Ensure `person` is being _detected_. A `person` will automatically be scanned by Frigate for a face. Any detected faces will appear in the Recent Recognitions tab in the Frigate UI's Face Library.
|
||||
|
||||
If you are using a Frigate+ or `face` detecting model:
|
||||
- Watch the [debug view](/usage/live#the-single-camera-view) to ensure that `face` is being detected along with `person`.
|
||||
@@ -242,7 +256,7 @@ Start with the [Usage](#usage) section and re-read the [Model Requirements](#mod
|
||||
- Check your `detect` stream resolution and ensure it is sufficiently high enough to capture face details on `person` objects.
|
||||
- You may need to lower your `detection_threshold` if faces are not being detected.
|
||||
|
||||
2. Any detected faces will then be _recognized_.
|
||||
3. Any detected faces will then be _recognized_.
|
||||
- Make sure you have trained at least one face per the recommendations above.
|
||||
- Adjust `recognition_threshold` settings per the suggestions [above](#advanced-configuration).
|
||||
|
||||
|
||||
@@ -78,7 +78,7 @@ All llama.cpp native options can be passed through `provider_options`, including
|
||||
- Set **Provider** to `llamacpp`
|
||||
- Set **Base URL** to your llama.cpp server address (e.g., `http://localhost:8080`)
|
||||
- Set **Model** to the name of your model
|
||||
- Under **Provider Options**, set `context_size` to tell Frigate your context size so it can send the appropriate amount of information
|
||||
- Optionally, under **Provider Options**, set `context_size` to override the context size Frigate detects from the server
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="yaml">
|
||||
@@ -89,12 +89,14 @@ genai:
|
||||
base_url: http://localhost:8080
|
||||
model: your-model-name
|
||||
provider_options:
|
||||
context_size: 16000 # Tell Frigate your context size so it can send the appropriate amount of information.
|
||||
context_size: 16000 # Optional, overrides the context size reported by the server.
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</ConfigTabs>
|
||||
|
||||
Frigate queries the llama.cpp server for the model's context size at startup and logs it along with the other detected capabilities. If `context_size` is set in `provider_options`, that value is always used instead, even when the server reports its own.
|
||||
|
||||
### Ollama
|
||||
|
||||
[Ollama](https://ollama.com/) allows you to self-host large language models and keep everything running locally. It is highly recommended to host this server on a machine with an Nvidia graphics card, or on a Apple silicon Mac for best performance.
|
||||
|
||||
@@ -192,7 +192,7 @@ class LlamaCppClient(GenAIClient):
|
||||
logger.info(
|
||||
"llama.cpp model '%s' initialized — context: %s, vision: %s, audio: %s, tools: %s, reasoning: %s",
|
||||
configured_model,
|
||||
self._context_size or "unknown",
|
||||
self.get_context_size(),
|
||||
self._supports_vision,
|
||||
self._supports_audio,
|
||||
self._supports_tools,
|
||||
|
||||
@@ -491,6 +491,34 @@ class TestLlamaCppProvider(unittest.TestCase):
|
||||
final = _final_message(self._run_with_lines(client, lines, MULTIMODAL_MESSAGES))
|
||||
self.assertEqual(final["content"], "ok")
|
||||
|
||||
def _validated_client(self, server_context_size, provider_options=None):
|
||||
"""Build a client as if the server reported the given context size."""
|
||||
cfg = GenAIConfig(
|
||||
provider="llamacpp",
|
||||
model="m",
|
||||
base_url="http://localhost:9999",
|
||||
provider_options=provider_options or {},
|
||||
)
|
||||
info = {
|
||||
"context_size": server_context_size,
|
||||
"supports_vision": False,
|
||||
"supports_audio": False,
|
||||
"supports_tools": False,
|
||||
"supports_reasoning": False,
|
||||
"media_marker": "<__media__>",
|
||||
}
|
||||
cls = PROVIDERS[GenAIProviderEnum.llamacpp]
|
||||
with patch.object(cls, "_get_model_info", return_value=info):
|
||||
return cls(cfg, timeout=5)
|
||||
|
||||
def test_server_context_size_used_without_override(self):
|
||||
client = self._validated_client(4096)
|
||||
self.assertEqual(client.get_context_size(), 4096)
|
||||
|
||||
def test_provider_options_context_size_overrides_server(self):
|
||||
client = self._validated_client(4096, {"context_size": 32768})
|
||||
self.assertEqual(client.get_context_size(), 32768)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* Semantic Search settings tests -- MEDIUM tier.
|
||||
*
|
||||
* Focuses on the model_size field, which is unused when a GenAI embeddings
|
||||
* provider is selected as the semantic search model. The resolved config always
|
||||
* reports model_size (it has a schema default of "small"), even when the YAML
|
||||
* file has no such key. Clearing model_size for a provider used to run
|
||||
* unconditionally, which falsely marked the section dirty on load and asked the
|
||||
* backend to delete a key that wasn't in the config file (KeyError: 'model_size').
|
||||
*/
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { resolve, dirname } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { test, expect } from "../../fixtures/frigate-test";
|
||||
import type { Page } from "@playwright/test";
|
||||
import { configFactory } from "../../fixtures/mock-data/config";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const CONFIG_SCHEMA = JSON.parse(
|
||||
readFileSync(
|
||||
resolve(__dirname, "../../fixtures/mock-data/config-schema.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
|
||||
const PROVIDER = "llama_cpp";
|
||||
const SETTINGS_URL = "/settings?page=integrationSemanticSearch";
|
||||
const NOT_APPLICABLE = "Not applicable for GenAI providers";
|
||||
const UNSAVED = "You have unsaved changes";
|
||||
|
||||
type SemanticSearch = {
|
||||
enabled?: boolean;
|
||||
model?: string;
|
||||
model_size?: string;
|
||||
};
|
||||
|
||||
async function installRoutes(page: Page, semanticSearch: SemanticSearch) {
|
||||
const config = configFactory({
|
||||
genai: { [PROVIDER]: { provider: PROVIDER, roles: ["embeddings"] } },
|
||||
semantic_search: semanticSearch,
|
||||
});
|
||||
|
||||
let lastSavedConfig: unknown = null;
|
||||
|
||||
await page.route("**/api/config/schema.json", (route) =>
|
||||
route.fulfill({ json: CONFIG_SCHEMA }),
|
||||
);
|
||||
await page.route("**/api/config", (route) => {
|
||||
if (route.request().method() === "GET") {
|
||||
return route.fulfill({ json: config });
|
||||
}
|
||||
return route.fulfill({ json: { success: true } });
|
||||
});
|
||||
await page.route("**/api/config/set", async (route) => {
|
||||
lastSavedConfig = route.request().postDataJSON();
|
||||
await route.fulfill({ json: { success: true, require_restart: false } });
|
||||
});
|
||||
await page.route("**/api/config/raw_paths", (route) =>
|
||||
route.fulfill({ json: { semantic_search: semanticSearch } }),
|
||||
);
|
||||
|
||||
return { capturedConfig: () => lastSavedConfig };
|
||||
}
|
||||
|
||||
test.describe("semantic search model_size @medium", () => {
|
||||
test("a provider with a defaulted model_size is not dirty on load", async ({
|
||||
frigateApp,
|
||||
}) => {
|
||||
// model_size stays at its schema default ("small"), i.e. it is not present
|
||||
// in the YAML. This mirrors the reported bug: selecting a GenAI provider and
|
||||
// returning to the page.
|
||||
await installRoutes(frigateApp.page, {
|
||||
enabled: true,
|
||||
model: PROVIDER,
|
||||
});
|
||||
await frigateApp.goto(SETTINGS_URL);
|
||||
|
||||
// The provider path is active: model_size shows "Not applicable".
|
||||
await expect(frigateApp.page.getByText(NOT_APPLICABLE)).toBeVisible();
|
||||
|
||||
// Give any clearing effect time to fire, then confirm the section stayed
|
||||
// clean (no phantom unsaved-changes banner, Save disabled).
|
||||
await frigateApp.page.waitForTimeout(1000);
|
||||
await expect(frigateApp.page.getByText(UNSAVED)).toBeHidden();
|
||||
await expect(
|
||||
frigateApp.page.getByRole("button", { name: "Save", exact: true }),
|
||||
).toBeDisabled();
|
||||
});
|
||||
|
||||
test("switching from a configured non-default model_size clears it", async ({
|
||||
frigateApp,
|
||||
}) => {
|
||||
// A genuinely configured non-default model_size ("large") can only come from
|
||||
// the YAML, so switching to a provider must still remove it.
|
||||
const capture = await installRoutes(frigateApp.page, {
|
||||
enabled: true,
|
||||
model: "jinav2",
|
||||
model_size: "large",
|
||||
});
|
||||
await frigateApp.goto(SETTINGS_URL);
|
||||
|
||||
// Starts clean on a Jina model.
|
||||
await expect(frigateApp.page.getByText(UNSAVED)).toBeHidden();
|
||||
|
||||
// Switch the model to the GenAI provider.
|
||||
await frigateApp.page
|
||||
.getByRole("combobox", { name: /Semantic search model/ })
|
||||
.click();
|
||||
await frigateApp.page.getByRole("option", { name: PROVIDER }).click();
|
||||
|
||||
// The change is now dirty and model_size is no longer applicable.
|
||||
await expect(frigateApp.page.getByText(NOT_APPLICABLE)).toBeVisible();
|
||||
await expect(frigateApp.page.getByText(UNSAVED)).toBeVisible();
|
||||
|
||||
await frigateApp.page
|
||||
.getByRole("button", { name: "Save", exact: true })
|
||||
.click();
|
||||
|
||||
// The saved payload removes model_size (empty string = "remove" key).
|
||||
await expect
|
||||
.poll(() => capture.capturedConfig(), { timeout: 5_000 })
|
||||
.toMatchObject({
|
||||
config_data: {
|
||||
semantic_search: { model: PROVIDER, model_size: "" },
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
Generated
+643
-436
File diff suppressed because it is too large
Load Diff
+4
-4
@@ -112,14 +112,14 @@
|
||||
"@types/react": "^19.2.14",
|
||||
"@types/react-dom": "^19.2.3",
|
||||
"@types/strftime": "^0.9.8",
|
||||
"@typescript-eslint/eslint-plugin": "^8.65.0",
|
||||
"@typescript-eslint/parser": "^8.65.0",
|
||||
"@typescript-eslint/eslint-plugin": "^7.5.0",
|
||||
"@typescript-eslint/parser": "^7.5.0",
|
||||
"@vitejs/plugin-react-swc": "^3.8.0",
|
||||
"@vitest/coverage-v8": "^3.0.7",
|
||||
"autoprefixer": "^10.4.20",
|
||||
"eslint": "^10.7.0",
|
||||
"eslint": "^8.57.0",
|
||||
"eslint-config-prettier": "^9.1.0",
|
||||
"eslint-plugin-jest": "^29.15.5",
|
||||
"eslint-plugin-jest": "^28.2.0",
|
||||
"eslint-plugin-prettier": "^5.0.1",
|
||||
"eslint-plugin-react-hooks": "^5.2.0",
|
||||
"eslint-plugin-react-refresh": "^0.4.8",
|
||||
|
||||
@@ -1936,7 +1936,8 @@
|
||||
"inputDimensionsNotDetectResolution": "Model input width and height are the input dimensions of the object detection model, not your camera's detect resolution. They should match the dimensions of the model you're using — typically a square size like 320x320 or 640x640."
|
||||
},
|
||||
"ffmpeg": {
|
||||
"hwaccelManualNotRecommended": "Manual hardware acceleration arguments are not recommended. Unless a specific requirement exists, select the preset that matches your hardware."
|
||||
"hwaccelManualNotRecommended": "Manual hardware acceleration arguments are not recommended. Unless a specific requirement exists, select the preset that matches your hardware.",
|
||||
"inputsMissingGo2rtcStream": "An input below points at a go2rtc restream that no longer exists. Select an existing restream or enter the camera's URL manually, otherwise this camera will fail to connect."
|
||||
},
|
||||
"objects": {
|
||||
"genaiNoDescriptionsProvider": "You must configure a GenAI provider with the 'descriptions' role for descriptions to be generated."
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { parseRestreamStreamName } from "../theme/fields/streamSource";
|
||||
import type { SectionConfigOverrides } from "./types";
|
||||
|
||||
const arrayAsTextWidget = {
|
||||
@@ -42,6 +43,29 @@ const ffmpeg: SectionConfigOverrides = {
|
||||
return false;
|
||||
},
|
||||
},
|
||||
{
|
||||
key: "inputs-missing-go2rtc-stream",
|
||||
field: "inputs",
|
||||
position: "before",
|
||||
messageKey: "configMessages.ffmpeg.inputsMissingGo2rtcStream",
|
||||
severity: "warning",
|
||||
docLink: "/configuration/restream",
|
||||
condition: (ctx) => {
|
||||
const streams = ctx.fullConfig?.go2rtc?.streams;
|
||||
const inputs = ctx.formData?.inputs;
|
||||
if (!Array.isArray(inputs)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return inputs.some((input) => {
|
||||
const path = (input as { path?: unknown } | null)?.path;
|
||||
const streamName = parseRestreamStreamName(
|
||||
typeof path === "string" ? path : undefined,
|
||||
);
|
||||
return streamName !== undefined && !(streamName in (streams ?? {}));
|
||||
});
|
||||
},
|
||||
},
|
||||
],
|
||||
fieldDocs: {
|
||||
hwaccel_args: "/configuration/ffmpeg_presets#hwaccel-presets",
|
||||
|
||||
@@ -170,6 +170,12 @@ export function CameraInputsField(props: FieldProps) {
|
||||
[go2rtcStreamNames],
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
setSourceModeByIndex((previous) =>
|
||||
Object.keys(previous).length > 0 ? {} : previous,
|
||||
);
|
||||
}, [formContext?.cameraName]);
|
||||
|
||||
useEffect(() => {
|
||||
setOpenByIndex((previous) => {
|
||||
const next: Record<number, boolean> = {};
|
||||
@@ -222,18 +228,12 @@ export function CameraInputsField(props: FieldProps) {
|
||||
const handleSourceModeChange = useCallback(
|
||||
(index: number, nextMode: StreamSourceMode) => {
|
||||
const input = inputs[index];
|
||||
const currentPath =
|
||||
typeof input?.path === "string" ? input.path : undefined;
|
||||
|
||||
if (nextMode === "manual") {
|
||||
// Only revert the preset we set ourselves; never clobber custom args.
|
||||
if (input?.input_args === RESTREAM_PRESET) {
|
||||
handleFieldValuesChange(index, { input_args: undefined });
|
||||
}
|
||||
} else if (!parseRestreamStreamName(currentPath)) {
|
||||
// Entering restream with a non-restream path: clear it so the dropdown
|
||||
// shows its placeholder until a stream is chosen.
|
||||
handleFieldValuesChange(index, { path: undefined });
|
||||
// Only revert the preset we set ourselves; never clobber custom args.
|
||||
// The path is left alone until a stream is picked, so switching modes
|
||||
// never discards a typed URL or empties a required field.
|
||||
if (nextMode === "manual" && input?.input_args === RESTREAM_PRESET) {
|
||||
handleFieldValuesChange(index, { input_args: undefined });
|
||||
}
|
||||
|
||||
setSourceModeByIndex((previous) => ({ ...previous, [index]: nextMode }));
|
||||
|
||||
@@ -24,15 +24,19 @@ export function SemanticSearchModelSizeWidget(props: WidgetProps) {
|
||||
model !== "jinav1" &&
|
||||
model !== "jinav2";
|
||||
|
||||
// Clear model_size while on a provider (buildOverrides converts to ""
|
||||
// which the backend treats as "remove"). Restore the schema default
|
||||
// when returning to a Jina model so the field isn't left empty.
|
||||
// model_size is unused on a GenAI provider. Only clear it (which the backend
|
||||
// treats as "remove") for a non-default value, which can only come from the
|
||||
// config file. A defaulted value is indistinguishable from unset in the
|
||||
// resolved config, so clearing it would falsely dirty the field and delete a
|
||||
// YAML key that isn't there. Restore the default when returning to a Jina model.
|
||||
const { value, onChange, schema } = props;
|
||||
const schemaDefault = schema?.default as string | undefined;
|
||||
useEffect(() => {
|
||||
if (isProvider && value !== undefined) {
|
||||
onChange(undefined);
|
||||
} else if (!isProvider && value === undefined && schemaDefault) {
|
||||
if (isProvider) {
|
||||
if (value !== undefined && value !== schemaDefault) {
|
||||
onChange(undefined);
|
||||
}
|
||||
} else if (value === undefined && schemaDefault) {
|
||||
onChange(schemaDefault);
|
||||
}
|
||||
}, [isProvider, value, onChange, schemaDefault]);
|
||||
|
||||
Reference in New Issue
Block a user