Compare commits

..
Author SHA1 Message Date
Josh Hawkins 9ab9239461 Merge branch 'dev' into birdseye-layout
Resolve conflict in frigate/output/birdseye.py in favor of the single
pass layout algorithm from this branch. Dev's conflicting hunk was the
Optional to union type modernization of the two pass algorithm that this
branch removes.

Convert the new algorithm's Optional hints to unions, since dev dropped
Optional from the typing import in the same change.
2026-07-15 13:24:35 -05:00
a8eca68438 Miscellaneous fixes (0.18 beta) (#23718)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / AMD64 Extra Build (push) Blocked by required conditions
* Cleanup llama.cpp and use api key when configured

* don't report auto-populated object and audio filters as camera overrides

* derive stale replay cameras from bounded directory listings to avoid scanning all clips at startup

* fix tests

* add -vaapi_device to the birdseye vaapi encode preset so hwupload can initialize on ffmpeg 8

---------

Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
2026-07-14 18:49:03 -06:00
81b53b7835 Miscellaneous fixes (0.18 beta) (#23716)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
* resolve zone friendly names against the correct camera

* Improve handling of zone names in chat prompt

* show a numeric keyboard for numeric config form fields on mobile

* Specify english only for semantic search tool when model is JinaV1

* resolve export hwaccel args global value against the correct config path

---------

Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
2026-07-14 08:42:26 -06:00
Josh HawkinsandGitHub c2e739b4bc bound REGEXP evaluation with a timeout to prevent ReDoS on the database thread (#23714)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / AMD64 Extra Build (push) Blocked by required conditions
The recognized_license_plate event filter passed attacker-controlled patterns to re.search on the single serialized SQLite queue thread, letting any authenticated user freeze the whole application with a catastrophic regex. This swaps stdlib re for the regex module with a per-evaluation timeout so a pathological pattern is aborted instead of stalling every database operation.
2026-07-14 06:27:00 -05:00
nulledyandGitHub 62d4e87e5d Add logout endpoint to Nginx configuration to prevent a new token on logout (#23678)
* Add logout endpoint to Nginx configuration to prevent logout from silently generating a new frigate_token cookie

* Change JWT cookie expiration to use max_age and have the appropriate expiration time based on JWT_SESSION_LENGTH

* ruff formatting
2026-07-14 02:35:51 -08:00
Nicolas MowenandGitHub 775ce22204 Miscellaneous Fixes (#23709)
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / AMD64 Build (push) Waiting to run
2026-07-13 19:55:00 -08:00
Jozef HuscavaandGitHub 6f24f5a595 Convert face crops to RGB before embedding (#23712)
* Convert face crops to RGB before embedding

Face crops flow through the cv2 pipeline as BGR arrays, but
_process_image passes ndarrays to PIL without any channel conversion,
so the FaceNet and ArcFace embedders receive BGR input while both
models expect RGB. The error is symmetric between enrollment and
recognition so it partially cancels, but it still costs accuracy.

* Move BGR to RGB conversion into a shared helper

Deduplicate the channel swap from both _preprocess_inputs methods
into a BaseEmbedding._bgr_to_rgb static helper, as suggested in
review.
2026-07-13 16:45:29 -06:00
Nicolas MowenandGitHub 65af0b1351 GenAI Fixes (#23708)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
* Fix Gemini tool calling

* Catch openai bug

* Implement tool calling tests for GenAI

* Expose if embeddings are supported for a given provider
2026-07-13 07:33:15 -06:00
Josh HawkinsandGitHub fcd05ec7bc UI improvements and fixes (#23690)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
* add ability to edit enabled and save_attempts for classification models in the UI

* add state motion and interval configs to edit dialog

* fix preview playback rate for motion previews

* add docs note about environment vars and go2rtc

* update live view faq
2026-07-13 06:30:15 -06:00
Josh Hawkins b5a360be39 add test 2026-04-17 17:18:11 -05:00
Josh Hawkins 54a7c5015e fix birdseye layout calculation
replace the two pass layout with a single pass pixel space algorithm
2026-04-17 17:18:04 -05:00
51 changed files with 2004 additions and 411 deletions
@@ -274,6 +274,13 @@ http {
include proxy.conf;
}
location /api/logout {
auth_request off;
rewrite ^/api(/.*)$ $1 break;
proxy_pass http://frigate_api;
include proxy.conf;
}
# Allow unauthenticated access to the first_time_login endpoint
# so the login page can load help text before authentication.
location /api/auth/first_time_login {
@@ -67,6 +67,12 @@ This section can be used to set environment variables for those unable to modify
Variables prefixed with `FRIGATE_` can be referenced in config fields that support environment variable substitution (such as MQTT host and credentials, camera stream URLs, and ONVIF host and credentials) using the `{FRIGATE_VARIABLE_NAME}` syntax.
:::note
The `go2rtc` section is an exception. go2rtc runs as a separate process, so its stream definitions can only be substituted with variables that exist in the container's environment (set via Docker `-e`, the `environment:` section of `docker-compose.yml`, or Docker secrets). Variables defined in the `environment_vars` block above are not available to go2rtc streams. Home Assistant app users, who cannot set container environment variables, must instead put credentials directly in their go2rtc stream URLs.
:::
<ConfigTabs>
<TabItem value="ui">
+121 -65
View File
@@ -6,6 +6,7 @@ title: Live View
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
import FaqItem from "@site/src/components/FaqItem";
Frigate intelligently displays your camera streams on the Live view dashboard. By default, Frigate employs "smart streaming" where camera images update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion or active objects are detected, cameras seamlessly switch to a live stream.
@@ -341,100 +342,155 @@ When your browser runs into problems playing back your camera streams, it will l
## Live view FAQ
1. **Why don't I have audio in my Live view?**
### Getting Live View Working
You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending PCMA/PCMU or AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
<FaqItem id="why-dont-i-have-audio-in-my-live-view" question="Why don't I have audio in my Live view?">
Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio when in low bandwidth mode, even if you've set up go2rtc.
You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending PCMA/PCMU or AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
2. **Frigate shows that my live stream is in "low bandwidth mode". What does this mean?**
If the audio controls don't appear in the UI at all, verify that the Live view is actually using your go2rtc stream. If your go2rtc stream names don't match your Frigate camera name, you must map them with the `live -> streams` config (see [Setting Streams For Live UI](#setting-streams-for-live-ui) above); otherwise the UI falls back to the video-only jsmpeg player.
Frigate intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio when in low bandwidth mode, even if you've set up go2rtc.
When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI labels as "low bandwidth mode". On Live dashboards, the mode will automatically reset when smart streaming is configured and activity stops. Continuous streaming mode does not have an automatic reset mechanism, but you can use the _Reset_ option to force a reload of your stream.
</FaqItem>
If you are using continuous streaming or you are loading more than a few high resolution streams at once on the dashboard, your browser may struggle to begin playback of your streams before the timeout. Frigate always prioritizes showing a live stream as quickly as possible, even if it is a lower quality jsmpeg stream. You can use the "Reset" link/button to try loading your high resolution stream again.
<FaqItem id="i-have-unmuted-some-cameras-on-my-dashboard-but-i-do-not-hear-sound-why" question="I have unmuted some cameras on my dashboard, but I do not hear sound. Why?">
Errors in stream playback (e.g., connection failures, codec issues, or buffering timeouts) that cause the fallback to low bandwidth mode (jsmpeg) are logged to the browser console for easier debugging. These errors may include:
- Network issues (e.g., MSE or WebRTC network connection problems).
- Unsupported codecs or stream formats (e.g., H.265 in WebRTC, which is not supported in some browsers).
- Buffering timeouts or low bandwidth conditions causing fallback to jsmpeg.
- Browser compatibility problems (e.g., iOS Safari limitations with MSE).
If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
To view browser console logs:
1. Open the Frigate Live View in your browser.
2. Open the browser's Developer Tools (F12 or right-click > Inspect > Console tab).
3. Reproduce the error (e.g., load a problematic stream or simulate network issues).
4. Look for messages prefixed with the camera name.
</FaqItem>
These logs help identify if the issue is player-specific (MSE vs. WebRTC) or related to camera configuration (e.g., go2rtc streams, codecs). If you see frequent errors:
- Verify your camera's H.264/AAC settings (see [Frigate's camera settings recommendations](#camera-settings-recommendations)).
- Check go2rtc configuration for transcoding (e.g., audio to AAC/OPUS).
- Test with a different stream via the UI dropdown (if `live -> streams` is configured).
- For WebRTC-specific issues, ensure port 8555 is forwarded and candidates are set (see [WebRTC Extra Configuration](#webrtc-extra-configuration)).
- If your cameras are streaming at a high resolution, your browser may be struggling to load all of the streams before the buffering timeout occurs. Frigate prioritizes showing a true live view as quickly as possible. If the fallback occurs often, change your live view settings to use a lower bandwidth substream.
<FaqItem id="my-live-view-shows-a-black-screen-or-doesnt-load-but-the-debug-view-works-why" question="My live view shows a black screen or doesn't load, but the debug view works. Why?">
3. **It doesn't seem like my cameras are streaming on the Live dashboard. Why?**
The debug view plays the `detect` stream processed by Frigate itself, while the Live view plays your go2rtc stream directly in the browser. If the debug view works but the Live view doesn't, your browser usually can't decode what the camera is sending, most often H.265 video or an incompatible audio track.
On the default Live dashboard ("All Cameras"), your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity is detected, cameras seamlessly switch to a full-resolution live stream. If you want to customize this behavior, use a camera group.
Work through the [go2rtc troubleshooting guide](/troubleshooting/go2rtc#live-view-is-black-buffering-or-stuck-in-low-bandwidth-mode) to isolate the problem. Two fixes resolve the majority of cases:
4. **I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?**
1. Restream through go2rtc's FFmpeg module by prefixing your source with `ffmpeg:`, for example `- ffmpeg:rtsp://user:password@192.168.1.5:554/stream`.
2. If that doesn't help, transcode to compatible codecs: `- ffmpeg:rtsp://user:password@192.168.1.5:554/stream#video=h264#audio=aac#hardware`.
This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard (4:3, 16:9, or 32:9, etc) aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
</FaqItem>
5. **How does "smart streaming" work?**
<FaqItem id="how-do-i-get-the-best-live-view-experience-in-home-assistant" question="How do I get the best live view experience in Home Assistant?">
Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity (motion or object/audio detection) occurs, cameras seamlessly switch to a live stream.
For a full-resolution, low-latency live view in Home Assistant dashboards, use the [Advanced Camera Card](https://card.camera) with the [go2rtc live provider](https://card.camera/#/configuration/cameras/live-provider?id=go2rtc), which streams directly from Frigate's bundled go2rtc. This also supports audio and [two-way talk](#two-way-talk) on capable cameras. See the [Home Assistant integration docs](/integrations/home-assistant) for setup.
This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second so you can see the activity until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
</FaqItem>
Smart streaming depends on having your camera's motion `threshold` and `contour_area` config values dialed in. Use the Motion Tuner in Settings in the UI to tune these values in real-time.
### Streaming Behavior
This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
<FaqItem id="how-does-smart-streaming-work" question={'How does "smart streaming" work?'}>
6. **I have unmuted some cameras on my dashboard, but I do not hear sound. Why?**
Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity (motion or object/audio detection) occurs, cameras seamlessly switch to a live stream.
If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second so you can see the activity until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
7. **My camera streams have lots of visual artifacts / distortion.**
Smart streaming depends on having your camera's motion `threshold` and `contour_area` config values dialed in. Use the Motion Tuner in Settings in the UI to tune these values in real-time.
Some cameras don't include the hardware to support multiple connections to the high resolution stream, and this can cause unexpected behavior. In this case it is recommended to [restream](./restream.md) the high resolution stream so that it can be used for live view and recordings.
This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
8. **Why does my camera stream switch aspect ratios on the Live dashboard?**
</FaqItem>
Your camera may change aspect ratios on the dashboard because Frigate uses different streams for different purposes. With go2rtc and Smart Streaming, Frigate shows a static image from the `detect` stream when no activity is present, and switches to the live stream when motion is detected. The camera image will change size if your streams use different aspect ratios.
<FaqItem id="it-doesnt-seem-like-my-cameras-are-streaming-on-the-live-dashboard-why" question="It doesn't seem like my cameras are streaming on the Live dashboard. Why?">
To prevent this, make the `detect` stream match the go2rtc live stream's aspect ratio (resolution does not need to match, just the aspect ratio). You can either adjust the camera's output resolution or set the `width` and `height` values in your config's `detect` section to a resolution with an aspect ratio that matches.
On the default Live dashboard ("All Cameras"), your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity is detected, cameras seamlessly switch to a full-resolution live stream. If you want to customize this behavior, use a camera group.
Example: Resolutions from two streams
- Mismatched (may cause aspect ratio switching on the dashboard):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x352 (~1.82:1, not 16:9)
</FaqItem>
- Matched (prevents switching):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x360 (16:9)
<FaqItem id="frigate-shows-that-my-live-stream-is-in-low-bandwidth-mode-what-does-this-mean" question={'Frigate shows that my live stream is in "low bandwidth mode". What does this mean?'}>
You can update the detect settings in your camera config to match the aspect ratio of your go2rtc live stream. For example:
Frigate intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
```yaml
cameras:
front_door:
detect:
width: 640
height: 360 # set this to 360 instead of 352
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/front_door # main stream 1920x1080
roles:
- record
- path: rtsp://127.0.0.1:8554/front_door_sub # sub stream 640x352
roles:
- detect
```
When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI labels as "low bandwidth mode". On Live dashboards, the mode will automatically reset when smart streaming is configured and activity stops. Continuous streaming mode does not have an automatic reset mechanism, but you can use the _Reset_ option to force a reload of your stream.
The same applies to your `record` stream: if its aspect ratio differs from your `detect` stream, your recordings will appear in a different shape than the live view. For consistent framing across live view and recordings, use the same aspect ratio for all of a camera's streams (the resolution can still differ).
If you are using continuous streaming or you are loading more than a few high resolution streams at once on the dashboard, your browser may struggle to begin playback of your streams before the timeout. Frigate always prioritizes showing a live stream as quickly as possible, even if it is a lower quality jsmpeg stream. You can use the "Reset" link/button to try loading your high resolution stream again.
9. **Why does Frigate prefer MSE over WebRTC for live view?**
Errors in stream playback (e.g., connection failures, codec issues, or buffering timeouts) that cause the fallback to low bandwidth mode (jsmpeg) are logged to the browser console for easier debugging. These errors may include:
Frigate prefers MSE because it delivers a better out-of-the-box experience than WebRTC on nearly every axis that matters for a security camera system. MSE is an open standard optimized and supported by all modern browsers, works without any extra configuration (WebRTC requires port forwarding and candidate setup, and lacks H.265 support in some browsers), and requires no internet access for NAT traversal. More importantly, MSE runs over TCP, so every frame arrives and is decoded in order, so nothing is ever silently skipped. WebRTC optimizes for latency over UDP by discarding late or incomplete frames, which works against you on cellular or spotty Wi-Fi: you can end up with frozen video, visual corruption, or gaps in the feed without ever knowing you missed something. Frigate's enhanced MSE player has adaptive speed playback and has been tuned for latency and connection robustness that meets or exceeds WebRTC, so you get near-real-time playback with a guarantee that when the video plays, every frame is actually there - which, for an NVR whose whole purpose is letting you see what happened, matters more than shaving fractions of a second off a latency number. That's why Frigate defaults to MSE and reserves WebRTC for cases that require it, like two-way talk.
- Network issues (e.g., MSE or WebRTC network connection problems).
- Unsupported codecs or stream formats (e.g., H.265 in WebRTC, which is not supported in some browsers).
- Buffering timeouts or low bandwidth conditions causing fallback to jsmpeg.
- Browser compatibility problems (e.g., iOS Safari limitations with MSE).
To view browser console logs:
1. Open the Frigate Live View in your browser.
2. Open the browser's Developer Tools (F12 or right-click > Inspect > Console tab).
3. Reproduce the error (e.g., load a problematic stream or simulate network issues).
4. Look for messages prefixed with the camera name.
These logs help identify if the issue is player-specific (MSE vs. WebRTC) or related to camera configuration (e.g., go2rtc streams, codecs). If you see frequent errors:
- Verify your camera's H.264/AAC settings (see [Frigate's camera settings recommendations](#camera-settings-recommendations)).
- Check go2rtc configuration for transcoding (e.g., audio to AAC/OPUS).
- Test with a different stream via the UI dropdown (if `live -> streams` is configured).
- For WebRTC-specific issues, ensure port 8555 is forwarded and candidates are set (see [WebRTC Extra Configuration](#webrtc-extra-configuration)).
- If your cameras are streaming at a high resolution, your browser may be struggling to load all of the streams before the buffering timeout occurs. Frigate prioritizes showing a true live view as quickly as possible. If the fallback occurs often, change your live view settings to use a lower bandwidth substream.
</FaqItem>
<FaqItem id="why-is-my-live-view-delayed-or-lagging-behind-real-time" question="Why is my live view delayed or lagging behind real time?">
A delay when a stream first starts is usually caused by your camera's I-frame (keyframe) interval. Playback cannot begin until a keyframe arrives, so an interval set higher than your camera's frame rate makes the stream take longer to start. Set the I-frame interval to match the frame rate (or "1x" on Reolink) per the [camera settings recommendations](#camera-settings-recommendations).
A stream that starts on time but falls further behind live is buffering, which is usually the browser struggling to decode too many high-resolution streams at once. Select a lower-bandwidth substream for your dashboards (see [Setting Streams For Live UI](#setting-streams-for-live-ui)), reduce the number of streams open at once, or improve the network connection between your browser and Frigate. Frigate's player automatically speeds up playback to catch up to live after buffering, and falls back to low bandwidth mode if it stalls for too long. The _Reset_ option forces a fresh connection at the live edge.
</FaqItem>
<FaqItem id="why-does-frigate-prefer-mse-over-webrtc-for-live-view" question="Why does Frigate prefer MSE over WebRTC for live view?">
Frigate prefers MSE because it delivers a better out-of-the-box experience than WebRTC on nearly every axis that matters for a security camera system. MSE is an open standard optimized and supported by all modern browsers, works without any extra configuration (WebRTC requires port forwarding and candidate setup, and lacks H.265 support in some browsers), and requires no internet access for NAT traversal. More importantly, MSE runs over TCP, so every frame arrives and is decoded in order, so nothing is ever silently skipped. WebRTC optimizes for latency over UDP by discarding late or incomplete frames, which works against you on cellular or spotty Wi-Fi: you can end up with frozen video, visual corruption, or gaps in the feed without ever knowing you missed something. Frigate's enhanced MSE player has adaptive speed playback and has been tuned for latency and connection robustness that meets or exceeds WebRTC, so you get near-real-time playback with a guarantee that when the video plays, every frame is actually there - which, for an NVR whose whole purpose is letting you see what happened, matters more than shaving fractions of a second off a latency number. That's why Frigate defaults to MSE and reserves WebRTC for cases that require it, like two-way talk.
</FaqItem>
### Video Quality Issues
<FaqItem id="i-see-a-strange-diagonal-line-on-my-live-view-but-my-recordings-look-fine-how-can-i-fix-it" question="I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?">
This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard (4:3, 16:9, or 32:9, etc) aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
</FaqItem>
<FaqItem id="my-camera-streams-have-lots-of-visual-artifacts-or-distortion" question="My camera streams have lots of visual artifacts / distortion.">
Some cameras don't include the hardware to support multiple connections to the high resolution stream, and this can cause unexpected behavior. In this case it is recommended to [restream](./restream.md) the high resolution stream so that it can be used for live view and recordings.
</FaqItem>
<FaqItem id="why-does-my-camera-stream-switch-aspect-ratios-on-the-live-dashboard" question="Why does my camera stream switch aspect ratios on the Live dashboard?">
Your camera may change aspect ratios on the dashboard because Frigate uses different streams for different purposes. With go2rtc and Smart Streaming, Frigate shows a static image from the `detect` stream when no activity is present, and switches to the live stream when motion is detected. The camera image will change size if your streams use different aspect ratios.
To prevent this, make the `detect` stream match the go2rtc live stream's aspect ratio (resolution does not need to match, just the aspect ratio). You can either adjust the camera's output resolution or set the `width` and `height` values in your config's `detect` section to a resolution with an aspect ratio that matches.
Example: Resolutions from two streams
- Mismatched (may cause aspect ratio switching on the dashboard):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x352 (~1.82:1, not 16:9)
- Matched (prevents switching):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x360 (16:9)
You can update the detect settings in your camera config to match the aspect ratio of your go2rtc live stream. For example:
```yaml
cameras:
front_door:
detect:
width: 640
height: 360 # set this to 360 instead of 352
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/front_door # main stream 1920x1080
roles:
- record
- path: rtsp://127.0.0.1:8554/front_door_sub # sub stream 640x352
roles:
- detect
```
The same applies to your `record` stream: if its aspect ratio differs from your `detect` stream, your recordings will appear in a different shape than the live view. For consistent framing across live view and recordings, use the same aspect ratio for all of a camera's streams (the resolution can still differ).
</FaqItem>
+2
View File
@@ -9,6 +9,8 @@ The best way to integrate with Home Assistant is to use the [official integratio
### Preparation
Frigate itself must be installed and running before setting up the integration. See the [installation documentation](../frigate/installation.md) for details.
The Frigate integration requires the `mqtt` integration to be installed and
manually configured first.
+3 -1
View File
@@ -39,7 +39,9 @@ The per-clip variation is typically quite low and is mostly an artifact of keyfr
Debug Replay lets you re-run Frigate's detection pipeline against a section of recorded video without manually configuring a dummy camera. It automatically extracts the recording, creates a temporary camera with the same detection settings as the original, and loops the clip through the pipeline so you can observe detections in real time.
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real-time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame.
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
### When to use
+159 -42
View File
@@ -3,6 +3,8 @@ id: recordings
title: Recordings Errors
---
import FaqItem from "@site/src/components/FaqItem";
## Why are my recordings not working? (empty Recordings, "No recordings found for this time")
If Frigate shows live video but the History view is empty, or you see "No recordings found for this time", the cause is almost always in one of the three categories below. Segments are first written to the RAM cache and are only moved to disk if they match a retention policy _and_ the camera's `record` stream is producing valid, storable video. Work through the categories in order: retention configuration is by far the most common cause.
@@ -19,7 +21,7 @@ A healthy camera logs lines like `Copied /media/frigate/recordings/{segment_path
### Retention configuration issues
#### Recording is enabled, but nothing is saved
<FaqItem id="recording-is-enabled-but-nothing-is-saved" question="Recording is enabled, but nothing is saved">
This is the single most common cause. Setting `record.enabled: True` on its own does **not** keep any footage: **continuous recording is disabled by default**, and segments in the cache are only moved to disk if they match a configured retention policy. You must configure at least one of `continuous`, `motion`, `alerts`, or `detections` retention.
@@ -34,7 +36,9 @@ record:
See [Recording](/configuration/record) for the full set of common configurations, including reduced-storage and alerts-only setups.
#### Motion or event-only recording keeps less than you expect
</FaqItem>
<FaqItem id="motion-or-event-only-recording-keeps-less-than-you-expect" question="Motion or event-only recording keeps less than you expect">
If you only configured `motion`, `alerts`, or `detections` retention (with no `continuous`), Frigate keeps footage selectively based on the retention `mode`:
@@ -44,20 +48,26 @@ If you only configured `motion`, `alerts`, or `detections` retention (with no `c
If you expected continuous footage but only configured motion/event retention, add a `continuous` retention period as shown above. To verify motion is actually being detected, watch the motion boxes in the debug view or the Motion Tuner in the UI.
#### Alert and detection recordings require working object detection
</FaqItem>
<FaqItem id="alert-and-detection-recordings-require-working-object-detection" question="Alert and detection recordings require working object detection">
`alerts` and `detections` retention only keep footage that overlaps a tracked object, so they depend on object detection running:
- **Detection must be enabled.** If `detect: enabled: False`, no alerts or detections are ever created, so alert/detection retention keeps nothing. (Continuous and motion retention still work with detection disabled.)
- **The object must be supported by your model.** If you track an object your model doesn't support (for example `deer` or `license_plate` on the default model), Frigate never detects it and never records for it. Check your logs for warnings such as `... is configured to track ['deer'] objects, which are not supported by the current model` and remove unsupported objects or switch to a model (e.g. [Frigate+](/plus/)) that includes them.
#### You're following an outdated guide
</FaqItem>
<FaqItem id="youre-following-an-outdated-guide" question="You're following an outdated guide">
Configuration keys change between major versions. The old `clips` config, for example, has not existed for a long time. If you copied a config from an old blog post or video, verify every key against the current [reference config](/configuration/advanced/reference).
</FaqItem>
### Camera and stream issues
#### Incompatible audio codec (recordings silently fail to save)
<FaqItem id="incompatible-audio-codec-recordings-silently-fail-to-save" question="Incompatible audio codec (recordings silently fail to save)">
Frigate stores recordings in an MP4 container, and some camera audio codecs (most commonly `pcm_alaw`, `pcm_mulaw`, or other G.711 variants) **cannot be placed in an MP4 container**. When this happens, ffmpeg fails to write the segment and no recording is saved, even though the live view works fine. This is a frequent cause on Tapo, TP-Link VIGI, and some Reolink cameras.
@@ -72,7 +82,9 @@ cameras:
# or preset-record-generic to record with no audio
```
#### The record stream isn't connecting
</FaqItem>
<FaqItem id="the-record-stream-isnt-connecting" question="The record stream isn't connecting">
A message like `No new recording segments were created for <camera> in the last 120s` means ffmpeg cannot read the `record` stream. To diagnose:
@@ -81,17 +93,11 @@ A message like `No new recording segments were created for <camera> in the last
- Test the exact RTSP URL (with the correct path, port, and credentials) in VLC or `ffplay`.
- If you restream through go2rtc, make sure the `record` input path points at the correct go2rtc stream name. Copying a config between cameras without updating the stream name is a common mistake.
#### Recordings play back with no video (or won't play at all)
Frigate copies the `record` stream directly without re-encoding, so playback depends on your browser supporting the camera's codec. H265/HEVC recordings may not be playable in some browsers. If recordings appear as audio-only or a black screen, your camera is likely sending a codec your browser can't decode. Configure the camera to output **H264** for maximum compatibility.
#### Segments are only ~1 second long
If the record stream uses a "Smart Codec"/H.264+ mode or changes encoding parameters mid-stream, corrupt timestamps cause segments to be split far too frequently and fill the cache. This produces the "Too many unprocessed recording segments" warning. See [that section below](#i-see-the-message-warning--too-many-unprocessed-recording-segments-in-cache-for-camera-this-likely-indicates-an-issue-with-the-detect-stream) for the full diagnosis.
</FaqItem>
### Storage and mounting issues
#### The storage volume isn't mounted correctly
<FaqItem id="the-storage-volume-isnt-mounted-correctly" question="The storage volume isn't mounted correctly">
If the recordings volume (`/media/frigate`) points at the wrong location, isn't writable, or a network/encrypted mount failed to mount at boot, Frigate cannot save recordings, or it silently writes to the boot drive and then purges aggressively because the drive appears far smaller than expected.
@@ -100,21 +106,114 @@ If the recordings volume (`/media/frigate`) points at the wrong location, isn't
- For a mount that may fail intermittently, protecting the mount point with `chattr +i` on an empty directory forces Frigate to error out (rather than silently writing to the boot drive) when the mount is missing.
- Check `dmesg` and system logs for filesystem or I/O errors around the time recordings disappeared.
If recordings _are_ being written but the copy is too slow to keep up, see the ["Unable to keep up with recording segments"](#i-see-the-message-warning--unable-to-keep-up-with-recording-segments-in-cache-for-camera-keeping-the-5-most-recent-segments-out-of-6-and-discarding-the-rest) section below.
If recordings _are_ being written but the copy is too slow to keep up, see the ["Unable to keep up with recording segments"](#i-see-the-message-warning--unable-to-keep-up-with-recording-segments-in-cache-for-camera-keeping-the-5-most-recent-segments-out-of-6-and-discarding-the-rest) question below.
## I have Frigate configured for motion recording only, but it still seems to be recording even with no motion. Why?
</FaqItem>
You'll want to:
## Recordings won't play back
- Make sure your camera's timestamp is masked out with a motion mask. Even if there is no motion occurring in your scene, your motion settings may be sensitive enough to count your timestamp as motion.
- If you have audio detection enabled, keep in mind that audio that is heard above `min_volume` is considered motion.
- [Tune your motion detection settings](/configuration/motion_detection) either by editing your config file or by using the UI's Motion Tuner.
<FaqItem id="pipeline-error-decode" question={"Recordings won't play back: \"PIPELINE_ERROR_DECODE\" (or \"Media failed to decode\")"}>
## I see the message: WARNING : Unable to keep up with recording segments in cache for camera. Keeping the 5 most recent segments out of 6 and discarding the rest...
When a recording refuses to play in the Frigate UI and you see an error like `Failed to play recordings (error 3): PIPELINE_ERROR_DECODE`, the message is coming from **your browser**, not from Frigate. `PIPELINE_ERROR_DECODE` is emitted exclusively by the media pipeline in **Chromium-based browsers** (Chrome, Edge, Brave, Vivaldi, Opera, Arc, and the Android WebView used by many in-app browsers) when the browser cannot decode a video or audio packet in the recording. WebKit browsers (Safari) report the same underlying problem with a different message, usually `Media failed to decode` or `DECODER_ERROR_NOT_SUPPORTED`.
Frigate copies the `record` stream to disk **without re-encoding it**, so the browser must decode exactly what your camera produced, and Chromium's decoder is far stricter about malformed or nonstandard media than VLC or ffmpeg.
:::warning
The same recording playing perfectly in VLC, decoding cleanly with `ffprobe`/`ffmpeg`, or having a valid MP4 container does **not** mean the browser can decode it. VLC and ffmpeg are much more tolerant of codec quirks and damaged packets than a browser's media pipeline, so a "valid" file can still trigger `PIPELINE_ERROR_DECODE`. This is outside of Frigate's control, because Frigate never modifies the recording stream.
:::
#### Step 1: Confirm it is a browser issue
Open the same recording in **Firefox** or **Safari**. Firefox and Safari both use a different media engine and cannot produce `PIPELINE_ERROR_DECODE`, so if playback works there you have confirmed a client-side codec or decoder problem rather than a bad recording. Switching browsers is a workaround, not a fix; the remaining steps address the root cause so that Chromium browsers work too.
#### Step 2: Rule out H.265 / HEVC
Browser support for H.265 (HEVC) is limited and depends on the operating system, GPU, hardware acceleration, and browser version, which makes it the most common cause of this error. Options, in order of reliability:
- **Record H.264 instead.** Configure the camera's `record`/main stream to output H.264, the most compatible codec across all browsers. See [camera settings recommendations](/configuration/live#camera-settings-recommendations).
- **Transcode to H.264 with go2rtc.** If you must keep HEVC on the camera, have go2rtc re-encode the recording stream. This increases CPU usage; add `#hardware` to use the GPU where available:
```yaml
go2rtc:
streams:
your_camera:
# transcode video to h264 and audio to aac; #hardware uses the GPU if available
- "ffmpeg:rtsp://user:password@CAMERA_IP:554/stream#video=h264#audio=aac#hardware"
cameras:
your_camera:
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/your_camera
input_args: preset-rtsp-restream
roles:
- record
```
The `#video=h264` parameter only takes effect with the `ffmpeg:` source module; adding it to a plain `rtsp://` go2rtc source does nothing.
- **Keep HEVC but improve compatibility.** If your browser and OS do support HEVC, set [`apple_compatibility`](/configuration/camera_specific#h265-cameras-via-safari) on the camera. Some players (Safari and other clients) require a specific HEVC stream format that this option corrects:
```yaml
cameras:
your_camera:
ffmpeg:
apple_compatibility: true
```
You may also need to enable HEVC and hardware decoding in the browser itself (for example, Chrome's Settings → System → "Use hardware acceleration when available"). HEVC hardware support varies widely by GPU, OS, and browser version.
#### Step 3: Clean up damaged packets from the camera
If the error is **intermittent** (the same recording plays after a page refresh, or fails only after playing for a while), the camera is most likely emitting occasional corrupt or malformed packets. Some camera models are more prone to this than others. Routing the stream through go2rtc's `ffmpeg` module often "cleans up" the stream enough for the browser to decode it, even without changing the codec:
```yaml
go2rtc:
streams:
your_camera:
- "ffmpeg:rtsp://user:password@CAMERA_IP:554/stream#video=h264#audio=aac"
```
#### Step 4: Fix incompatible or corrupt audio
Audio is one of the most common culprits, and a decode failure on the audio track fails the whole recording. Make sure the camera outputs **AAC** audio, transcode the audio to AAC with go2rtc (`#audio=aac`), or drop audio entirely. See [Incompatible audio codec](#incompatible-audio-codec-recordings-silently-fail-to-save) for a preset-based way to do this.
#### Step 5: Avoid "smart" / "+" codecs and check the keyframe interval
- Disable any **"Smart Codec"**, **"H.264+"**, or **"H.265+"** feature in the camera. These nonstandard modes drop keyframes and change encoding parameters mid-stream, producing exactly the kind of packets a browser refuses to decode. (They also cause [short recording segments](#segments-are-only-1-second-long).)
- Set the camera's **I-frame (keyframe) interval equal to the frame rate** (for example `20` for a 20 fps stream). Long keyframe intervals slow the start of playback and make decode errors more likely.
#### Step 6: Consider bitrate and the client hardware
The browser decodes the video locally, so a stream that is too demanding can fail on one device while playing on another:
- A **very high bitrate or resolution** (for example a 4K/8MP HEVC main stream) can overwhelm a low-power tablet, phone, or SBC and stall the decoder. Test the same recording on a desktop; if it plays there, lower the camera's bitrate or record a lower-resolution profile.
- Errors that name the client's GPU decoder, such as `VaapiVideoDecoder: failed Initialize()ing the frame pool`, indicate a browser hardware-decode problem. Toggling the browser's "Use hardware acceleration" setting (on or off) often resolves these.
</FaqItem>
<FaqItem id="recordings-play-back-with-no-video-or-wont-play-at-all" question="Recordings play back with no video (or won't play at all)">
Frigate copies the `record` stream directly without re-encoding, so playback depends on your browser supporting the camera's codec. H265/HEVC recordings may not be playable in some browsers. If recordings appear as audio-only or a black screen, your camera is likely sending a codec your browser can't decode. Configure the camera to output **H264** for maximum compatibility.
If playback instead fails with an explicit `PIPELINE_ERROR_DECODE` or `Media failed to decode` error, see [Recordings won't play back with "PIPELINE_ERROR_DECODE"](#pipeline-error-decode) above.
</FaqItem>
## Recording cache warnings and errors
<FaqItem id="segments-are-only-1-second-long" question="Segments are only ~1 second long">
If the record stream uses a "Smart Codec"/H.264+ mode or changes encoding parameters mid-stream, corrupt timestamps cause segments to be split far too frequently and fill the cache. This produces the "Too many unprocessed recording segments" warning. See [that question below](#i-see-the-message-warning--too-many-unprocessed-recording-segments-in-cache-for-camera-this-likely-indicates-an-issue-with-the-detect-stream) for the full diagnosis.
</FaqItem>
<FaqItem id="i-see-the-message-warning--unable-to-keep-up-with-recording-segments-in-cache-for-camera-keeping-the-5-most-recent-segments-out-of-6-and-discarding-the-rest" question="I see the message: WARNING : Unable to keep up with recording segments in cache for camera. Keeping the 5 most recent segments out of 6 and discarding the rest...">
This warning means the recording maintainer cannot move recording segments from the RAM cache to disk fast enough. When the cache fills up, Frigate discards the oldest segments to avoid running out of memory and crashing, so you lose recorded footage. This is almost always a storage throughput or system resource problem. Work through the steps below to identify which.
### Step 1: Enable recording debug logging
#### Step 1: Enable recording debug logging
The first step is to measure how long each segment takes to move from the RAM cache to disk. Enable debug logging for the recording maintainer:
@@ -132,14 +231,14 @@ DEBUG : Copied /media/frigate/recordings/{segment_path} in 0.2 seconds.
Let this run until the warnings begin to appear, so you can confirm whether the disk is actually slowing down at the moment the error occurs.
### Step 2: Interpret the copy times
#### Step 2: Interpret the copy times
The copy duration tells you which direction to investigate:
- **Consistently longer than ~1 second**: your storage cannot keep up with the incoming recordings. Continue with Steps 35 to diagnose the slow storage.
- **Consistently well under 1 second**: storage is fast enough, and the problem is more likely CPU or resource contention. Skip to Step 6.
### Step 3: Check RAM, swap, cache, and disk utilization
#### Step 3: Check RAM, swap, cache, and disk utilization
If CPU, RAM, disk throughput, or bus I/O is insufficient, nothing inside Frigate will help. Review each aspect of available system resources while the warnings are occurring.
@@ -175,19 +274,21 @@ services:
NOTE: These are hard limits for the container, so be sure there is enough headroom above what `docker stats` shows for your container. It will immediately halt if it hits `<MAXRAM>`. In general, keeping all cache and tmp filespace in RAM is preferable to disk I/O where possible.
### Step 4: Check your storage type
#### Step 4: Check your storage type
Mounting a network share is a popular option for storing recordings, but it can lead to reduced copy times and cause problems. Some users have found that using `NFS` instead of `SMB` considerably decreased copy times and fixed the issue. It is also important to ensure that the network connection between the device running Frigate and the network share is stable and fast. A saturated or unreliable link will stall copies.
### Step 5: Check your mount options
#### Step 5: Check your mount options
Some users found that mounting a drive via `fstab` with the `sync` option dramatically reduced performance and led to this issue. Using `async` instead greatly reduced copy times.
### Step 6: Rule out CPU load
#### Step 6: Rule out CPU load
If the copy times are consistently under 1 second but you still see the warning, the machine's CPU load is likely too high for Frigate to have the resources to keep up. Try temporarily shutting down other services, and any resource-intensive Frigate features, to see if the issue improves.
## I see the message: WARNING : Too many unprocessed recording segments in cache for camera. This likely indicates an issue with the detect stream...
</FaqItem>
<FaqItem id="i-see-the-message-warning--too-many-unprocessed-recording-segments-in-cache-for-camera-this-likely-indicates-an-issue-with-the-detect-stream" question="I see the message: WARNING : Too many unprocessed recording segments in cache for camera. This likely indicates an issue with the detect stream...">
This warning means that the detect stream for the affected camera has fallen behind or stopped processing frames. Frigate's recording cache holds segments waiting to be analyzed by the detector. When more than 6 segments pile up without being processed, Frigate discards the oldest ones to prevent the cache from filling up.
@@ -197,11 +298,11 @@ This error is a **symptom**, not the root cause. The actual cause is always logg
:::
### Step 1: Get the full logs
#### Step 1: Get the full logs
Collect complete Frigate logs from startup through the first occurrence of the error. Look for errors or warnings that appear **before** the "Too many unprocessed" messages begin. That is where the root cause will be found.
### Step 2: Check the cache directory
#### Step 2: Check the cache directory
Exec into the Frigate container and inspect the recording cache:
@@ -211,7 +312,7 @@ docker exec -it frigate ls -la /tmp/cache
Each camera should have a small number of `.mp4` segment files. If one camera has significantly more files than others, that camera is the source of the problem. A problem with a single camera can cascade and cause all cameras to show this error.
### Step 3: Verify segment duration
#### Step 3: Verify segment duration
Recording segments should be approximately 10 seconds long. Run `ffprobe` on segments in the cache to check:
@@ -233,7 +334,7 @@ You don't have to run `ffprobe` by hand to catch this. Open a camera's **Camera
:::
### Step 4: Check for a stuck detector
#### Step 4: Check for a stuck detector
If the detect stream is not processing frames, segments will accumulate. Common causes:
@@ -242,7 +343,7 @@ If the detect stream is not processing frames, segments will accumulate. Common
- **Model too large**: Use smaller model variants (e.g., YOLO `s` or `t` size, not `e` or `x`). Use 320x320 input size rather than 640x640 unless you have a powerful dedicated detector.
- **Virtualization**: Running Frigate in a VM (especially Proxmox) can cause the detector to hang or stall. This is a known issue with GPU/TPU passthrough in virtualized environments and is not something Frigate can fix. Running Frigate in Docker on bare metal is recommended.
### Step 5: Check for GPU hangs
#### Step 5: Check for GPU hangs
On the host machine, check `dmesg` for GPU-related errors:
@@ -252,7 +353,7 @@ dmesg | grep -i -E "gpu|drm|reset|hang"
Messages like `trying reset from guc_exec_queue_timedout_job` or similar GPU reset/hang messages indicate a driver or hardware issue. Ensure your kernel and GPU drivers (especially Intel) are up to date.
### Step 6: Verify hardware acceleration configuration
#### Step 6: Verify hardware acceleration configuration
An incorrect `hwaccel_args` preset can cause ffmpeg to fail silently or consume excessive CPU, starving the detector of resources.
@@ -260,11 +361,11 @@ An incorrect `hwaccel_args` preset can cause ffmpeg to fail silently or consume
- For h265 cameras, use the corresponding h265 preset (e.g., `preset-intel-qsv-h265`).
- Note that `hwaccel_args` are only relevant for the detect stream. Frigate does not decode the record stream.
### Step 7: Verify go2rtc stream configuration
#### Step 7: Verify go2rtc stream configuration
Ensure that the ffmpeg source names in your go2rtc configuration match the correct camera stream. A misconfigured stream name (e.g., copying a config from one camera to another without updating the stream reference) will cause the wrong stream to be used or the stream to fail entirely.
### Step 8: Check system resources
#### Step 8: Check system resources
If none of the above apply, the issue may be a general resource constraint. Monitor the following on your host:
@@ -275,7 +376,9 @@ If none of the above apply, the issue may be a general resource constraint. Moni
Try temporarily disabling resource-intensive features like `genai` and `face_recognition` to see if the issue resolves. This can help isolate whether the detector is being starved of resources.
## I see the message: ERROR : Error occurred when attempting to maintain recording cache
</FaqItem>
<FaqItem id="i-see-the-message-error--error-occurred-when-attempting-to-maintain-recording-cache" question="I see the message: ERROR : Error occurred when attempting to maintain recording cache">
This message means the recording maintainer hit an error while moving segments from the cache to disk. It is a **generic wrapper**: the actual cause is always logged on the **very next line**. Frigate usually recovers and keeps running, but any affected segments are lost, so it is worth resolving.
@@ -287,27 +390,41 @@ Always read the line immediately following this message. `Error occurred when at
Because these are operating-system-level errors, they must be resolved on the **host**, not within Frigate's configuration. The most common underlying errors are below.
### [Errno 28] No space left on device
#### [Errno 28] No space left on device
The filesystem Frigate is writing to is full. Things to check:
- **The recordings volume is genuinely full.** Check free space on the host with `df -h` for the path mapped to `/media/frigate`, and review the **Storage** page in the Frigate UI.
- **The disk shows free space but is still "full".** This usually means the filesystem has run out of **inodes** (check with `df -i`), or recordings are landing on a different, smaller filesystem than you expect because of an incorrect bind mount. See [The storage volume isn't mounted correctly](#the-storage-volume-isnt-mounted-correctly) above.
- **`/tmp/cache` is full.** If you mounted `/tmp/cache` as a small `tmpfs`, a backlog of segments can fill it. Increase the tmpfs size, or address whatever is causing segments to pile up (see the [Too many unprocessed recording segments](#i-see-the-message-warning--too-many-unprocessed-recording-segments-in-cache-for-camera-this-likely-indicates-an-issue-with-the-detect-stream) section above).
- **`/tmp/cache` is full.** If you mounted `/tmp/cache` as a small `tmpfs`, a backlog of segments can fill it. Increase the tmpfs size, or address whatever is causing segments to pile up (see the [Too many unprocessed recording segments](#i-see-the-message-warning--too-many-unprocessed-recording-segments-in-cache-for-camera-this-likely-indicates-an-issue-with-the-detect-stream) question above).
- **The host blocks writes before Frigate can purge.** On some systems (for example Unraid with a fill-up threshold), the host stops writes before Frigate's emergency cleanup can run. Leave more headroom on the volume, or lower your retention so Frigate purges sooner.
### [Errno 17] File exists (with ffmpeg "Error writing trailer" or "unable to re-open output file")
#### [Errno 17] File exists (with ffmpeg "Error writing trailer" or "unable to re-open output file")
Errors like `[Errno 17] File exists: '/media/frigate/recordings/.../<camera>'`, often alongside ffmpeg errors such as `Unable to re-open ... output file for shifting data` or `Error writing trailer: No such file or directory`, are a hallmark of an **unreliable network share** (NFS or SMB). The mount is dropping, serving stale directory entries, or mishandling file locking.
- Confirm the network connection to the NAS is stable and fast. An intermittent link produces these errors sporadically.
- Prefer **NFS over SMB** for the recordings mount; several users have found NFS more reliable and faster.
- Review your `fstab`/mount options for settings that hurt consistency or performance (see the `sync` vs `async` note in the [Unable to keep up with recording segments](#i-see-the-message-warning--unable-to-keep-up-with-recording-segments-in-cache-for-camera-keeping-the-5-most-recent-segments-out-of-6-and-discarding-the-rest) section above).
- Review your `fstab`/mount options for settings that hurt consistency or performance (see the `sync` vs `async` note in the [Unable to keep up with recording segments](#i-see-the-message-warning--unable-to-keep-up-with-recording-segments-in-cache-for-camera-keeping-the-5-most-recent-segments-out-of-6-and-discarding-the-rest) question above).
- Enable `frigate.record.maintainer` debug logging to confirm whether the errors line up with the share becoming unavailable.
### Errors referencing a camera you manually renamed or removed
#### Errors referencing a camera you manually renamed or removed
If the next-line error references a camera name that no longer exists in your config, orphaned data is left over from a rename or removal in a persistent `/tmp/cache` volume.
- Using a `tmpfs` mount for `/tmp/cache` as recommended in the [installation docs](/frigate/installation#storage) prevents stale cache files under the old camera name from surviving a restart, which avoids this issue entirely.
- If errors persist, stop Frigate and remove any leftover segments for the old camera name from `/tmp/cache`.
</FaqItem>
## Other recording questions
<FaqItem id="i-have-frigate-configured-for-motion-recording-only-but-it-still-seems-to-be-recording-even-with-no-motion-why" question="I have Frigate configured for motion recording only, but it still seems to be recording even with no motion. Why?">
You'll want to:
- Make sure your camera's timestamp is masked out with a motion mask. Even if there is no motion occurring in your scene, your motion settings may be sensitive enough to count your timestamp as motion.
- If you have audio detection enabled, keep in mind that audio that is heard above `min_volume` is considered motion.
- [Tune your motion detection settings](/configuration/motion_detection) either by editing your config file or by using the UI's Motion Tuner.
</FaqItem>
+1 -1
View File
@@ -55,7 +55,7 @@ export default function FaqItem({ id, question, children }) {
aria-controls={`${id}-content`}
onClick={toggle}
>
{question}
<span className={styles.question}>{question}</span>
</button>
</Heading>
<div id={`${id}-content`} className={styles.content}>
@@ -44,6 +44,11 @@
transform: rotate(45deg);
}
.question {
flex: 1;
min-width: 0;
}
.content {
display: none;
padding: 0 0 0.85rem;
@@ -61,7 +66,7 @@
/* Desktop: render as a normal expanded heading + answer. */
@media (min-width: 997px) {
.heading {
margin: 1.75rem 0 0.5rem;
margin: 1.75rem 0 0.85rem;
}
.toggle {
@@ -78,7 +83,8 @@
.content {
display: block;
padding: 0;
padding: 0 0 0.5rem 1rem;
border-left: 2px solid var(--ifm-color-emphasis-200);
}
.heading :global(.hash-link) {
+13 -5
View File
@@ -415,7 +415,7 @@ def create_encoded_jwt(user, role, expiration, secret):
)
def set_jwt_cookie(response: Response, cookie_name, encoded_jwt, expiration, secure):
def set_jwt_cookie(response: Response, cookie_name, encoded_jwt, max_age, secure):
# TODO: ideally this would set secure as well, but that requires TLS
# SameSite is intentionally left unset (browsers default to Lax). Setting
# SameSite=Lax/Strict would stop the cookie from being sent in cross-origin
@@ -427,7 +427,7 @@ def set_jwt_cookie(response: Response, cookie_name, encoded_jwt, expiration, sec
key=cookie_name,
value=encoded_jwt,
httponly=True,
expires=expiration,
max_age=max_age,
secure=secure,
)
@@ -762,7 +762,7 @@ def auth(request: Request):
success_response,
JWT_COOKIE_NAME,
new_encoded_jwt,
new_expiration,
JWT_SESSION_LENGTH,
JWT_COOKIE_SECURE,
)
@@ -875,7 +875,11 @@ def login(request: Request, body: AppPostLoginBody):
encoded_jwt = create_encoded_jwt(user, role, expiration, request.app.jwt_token)
response = Response("", 200)
set_jwt_cookie(
response, JWT_COOKIE_NAME, encoded_jwt, expiration, JWT_COOKIE_SECURE
response,
JWT_COOKIE_NAME,
encoded_jwt,
JWT_SESSION_LENGTH,
JWT_COOKIE_SECURE,
)
# Clear admin_first_time_login flag after successful admin login so the
# UI stops showing the first-time login documentation link.
@@ -1037,7 +1041,11 @@ async def update_password(
)
# Set new JWT cookie on response
set_jwt_cookie(
response, JWT_COOKIE_NAME, encoded_jwt, expiration, JWT_COOKIE_SECURE
response,
JWT_COOKIE_NAME,
encoded_jwt,
JWT_SESSION_LENGTH,
JWT_COOKIE_SECURE,
)
return response
+28 -7
View File
@@ -7,7 +7,7 @@ import operator
import time
from datetime import datetime
from functools import reduce
from typing import Any
from typing import Any, Literal
import cv2
from fastapi import APIRouter, Body, Depends, HTTPException, Request
@@ -37,6 +37,7 @@ from frigate.api.defs.response.chat_response import (
from frigate.api.defs.tags import Tags
from frigate.api.event import _build_attribute_filter_clause, events
from frigate.config import FrigateConfig
from frigate.config.classification import SemanticSearchModelEnum
from frigate.genai.prompts import (
build_chat_system_prompt,
get_attribute_classifications,
@@ -86,10 +87,23 @@ def get_tools(request: Request) -> JSONResponse:
tools = get_tool_definitions(
semantic_search_enabled=semantic_search_enabled,
attribute_classifications=attribute_classifications,
embeddings_language=_embeddings_language(config),
)
return JSONResponse(content={"tools": tools})
def _embeddings_language(config: FrigateConfig) -> Literal["english", "multi"]:
"""Return the language capability of the configured embeddings model.
JinaV1 is English-only; every other option (JinaV2 or a GenAI embeddings
provider) handles multiple languages.
"""
if config.semantic_search.model == SemanticSearchModelEnum.jinav1:
return "english"
return "multi"
def _resolve_zones(
zones: list[str],
config: FrigateConfig,
@@ -98,11 +112,14 @@ def _resolve_zones(
"""Map zone names to their canonical config keys, case-insensitively.
LLMs frequently echo a user's casing ("Front Yard") instead of the
configured key ("front_yard"). The downstream zone filter is a SQLite GLOB
over the JSON-encoded zones column, which is case-sensitive — so an
unnormalized name silently returns zero matches. Build a lookup over the
relevant cameras' configured zones and substitute when we find a match;
unknown names pass through so behavior matches what the model asked for.
configured key ("front_yard"), or fall back to a zone's friendly name
("Front Walkway") instead of its ID ("front_walk"). The downstream zone
filter is a SQLite GLOB over the JSON-encoded zones column, which stores
config keys and is case-sensitive — so an unnormalized name silently
returns zero matches. Build a lookup over the relevant cameras' configured
zones, keyed by both the config key and the friendly name, and substitute
when we find a match; unknown names pass through so behavior matches what
the model asked for.
"""
if not zones:
return zones
@@ -112,8 +129,11 @@ def _resolve_zones(
camera_config = config.cameras.get(camera_id)
if camera_config is None:
continue
for zone_name in camera_config.zones.keys():
for zone_name, zone_config in camera_config.zones.items():
lookup.setdefault(zone_name.lower(), zone_name)
lookup.setdefault(
zone_config.get_formatted_name(zone_name).lower(), zone_name
)
return [lookup.get(z.lower(), z) for z in zones]
@@ -1134,6 +1154,7 @@ async def chat_completion(
tools = get_tool_definitions(
semantic_search_enabled=semantic_search_enabled,
attribute_classifications=attribute_classifications,
embeddings_language=_embeddings_language(config),
)
conversation = []
+3
View File
@@ -400,6 +400,9 @@ def verify_objects_track(
)
camera_config.objects.track = valid_objects
for label in invalid_objects:
camera_config.objects.filters.pop(label, None)
def verify_lpr_and_face(
frigate_config: FrigateConfig, camera_config: CameraConfig
+7 -3
View File
@@ -1,9 +1,11 @@
import re
import sqlite3
from typing import Any
import regex
from playhouse.sqliteq import SqliteQueueDatabase
REGEXP_TIMEOUT_SECONDS = 1.0
class SqliteVecQueueDatabase(SqliteQueueDatabase):
def __init__(
@@ -34,8 +36,10 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
if item is None:
return False
try:
return re.search(expr, item) is not None
except re.error:
return (
regex.search(expr, item, timeout=REGEXP_TIMEOUT_SECONDS) is not None
)
except (regex.error, TimeoutError):
return False
conn.create_function("REGEXP", 2, regexp)
+8 -8
View File
@@ -21,8 +21,6 @@ from frigate.config.camera.updater import (
CameraConfigUpdateTopic,
)
from frigate.const import (
CLIPS_DIR,
RECORD_DIR,
REPLAY_CAMERA_PREFIX,
REPLAY_DIR,
THUMB_DIR,
@@ -331,12 +329,14 @@ def cleanup_replay_cameras() -> None:
"""
stale_cameras: set[str] = set()
# Scan filesystem for leftover replay artifacts to derive camera names
for dir_path in [RECORD_DIR, CLIPS_DIR, THUMB_DIR]:
if os.path.isdir(dir_path):
for entry in os.listdir(dir_path):
if entry.startswith(REPLAY_CAMERA_PREFIX):
stale_cameras.add(entry)
# Derive stale camera names from THUMB_DIR (per-camera dirs) and
# REPLAY_DIR (the session's source clip); both listings are bounded by
# camera count. cleanup_camera_files below removes any remaining
# per-camera artifacts (snapshots, thumbnails, LPR images, etc.) by name.
if os.path.isdir(THUMB_DIR):
for entry in os.listdir(THUMB_DIR):
if entry.startswith(REPLAY_CAMERA_PREFIX):
stale_cameras.add(entry)
if os.path.isdir(REPLAY_DIR):
for entry in os.listdir(REPLAY_DIR):
+14 -12
View File
@@ -376,20 +376,22 @@ class EmbeddingMaintainer(threading.Thread):
logger.info(f"Disabled classification processor for model: {model_name}")
return
# Check if processor already exists
for processor in self.realtime_processors:
if isinstance(
processor,
(
CustomStateClassificationProcessor,
CustomObjectClassificationProcessor,
),
if (
isinstance(
processor,
(
CustomStateClassificationProcessor,
CustomObjectClassificationProcessor,
),
)
and processor.model_config.name == model_name
):
if processor.model_config.name == model_name:
logger.debug(
f"Classification processor for model {model_name} already exists, skipping"
)
return
processor.model_config = model_config
logger.debug(
f"Updated config for classification processor: {model_name}"
)
return
if model_config.state_config is not None:
processor = CustomStateClassificationProcessor(
@@ -57,6 +57,12 @@ class BaseEmbedding(ABC):
def _preprocess_inputs(self, raw_inputs: Any) -> Any:
pass
@staticmethod
def _bgr_to_rgb(frame: Any) -> Any:
if isinstance(frame, np.ndarray) and frame.ndim == 3:
return np.ascontiguousarray(frame[:, :, ::-1])
return frame
def _process_image(self, image, output: str = "RGB") -> Image.Image:
if isinstance(image, str):
if image.startswith("http"):
+2 -2
View File
@@ -73,7 +73,7 @@ class FaceNetEmbedding(BaseEmbedding):
self.tensor_output_details = self.runner.get_output_details()
def _preprocess_inputs(self, raw_inputs):
pil = self._process_image(raw_inputs[0])
pil = self._process_image(self._bgr_to_rgb(raw_inputs[0]))
# handle images larger than input size
width, height = pil.size
@@ -159,7 +159,7 @@ class ArcfaceEmbedding(BaseEmbedding):
)
def _preprocess_inputs(self, raw_inputs):
pil = self._process_image(raw_inputs[0])
pil = self._process_image(self._bgr_to_rgb(raw_inputs[0]))
# handle images larger than input size
width, height = pil.size
+5 -1
View File
@@ -150,7 +150,11 @@ PRESETS_HW_ACCEL_SCALE["preset-rk-h265"] = PRESETS_HW_ACCEL_SCALE[FFMPEG_HWACCEL
PRESETS_HW_ACCEL_ENCODE_BIRDSEYE = {
"preset-rpi-64-h264": "{0} -hide_banner {1} -c:v h264_v4l2m2m {2}",
"preset-rpi-64-h265": "{0} -hide_banner {1} -c:v hevc_v4l2m2m {2}",
FFMPEG_HWACCEL_VAAPI: "{0} -hide_banner -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device {3} {1} -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -an -vf format=vaapi|nv12,hwupload {2}",
# -vaapi_device is required in addition to -hwaccel_device: this is the only
# birdseye preset that uses hwupload, and ffmpeg 8 initializes filters before
# the decoder creates a device, so hwupload cannot see an -hwaccel_device one.
# See https://github.com/AlexxIT/go2rtc/issues/1984
FFMPEG_HWACCEL_VAAPI: "{0} -hide_banner -vaapi_device {3} -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device {3} {1} -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -an -vf format=vaapi|nv12,hwupload {2}",
"preset-intel-qsv-h264": "{0} -hide_banner {1} -c:v h264_qsv -g 50 -bf 0 -profile:v high -level:v 4.1 -async_depth:v 1 {2}",
"preset-intel-qsv-h265": "{0} -hide_banner {1} -c:v h264_qsv -g 50 -bf 0 -profile:v main -level:v 4.1 -async_depth:v 1 {2}",
FFMPEG_HWACCEL_NVIDIA: "{0} -hide_banner {1} -c:v h264_nvenc -g 50 -profile:v high -level:v auto -preset:v p2 -tune:v ll {2}",
+5
View File
@@ -281,6 +281,11 @@ class GenAIClient:
"""Whether the configured model exposes a per-request thinking toggle."""
return False
@property
def supports_embeddings(self) -> bool:
"""Whether the configured model can generate embeddings via embed()."""
return False
def list_models(self) -> list[str]:
"""Return the list of model names available from this provider.
+1
View File
@@ -121,5 +121,6 @@ class GenAIClientManager:
"models": client.list_models(),
"roles": [r.value for r in genai_cfg.roles],
"supports_toggleable_thinking": client.supports_toggleable_thinking,
"supports_embeddings": client.supports_embeddings,
}
return result
+62 -60
View File
@@ -38,6 +38,37 @@ def _encode_thought_signature(signature: bytes | None) -> str | None:
return base64.b64encode(signature).decode("ascii")
def _decode_data_uri(url: str) -> tuple[str, bytes] | None:
"""Decode a ``data:`` URI into ``(mime_type, bytes)``; None if not a data URI."""
if not isinstance(url, str) or not url.startswith("data:"):
return None
try:
header, b64 = url.split(",", 1)
mime = header[len("data:") :].split(";")[0] or "image/jpeg"
return mime, base64.b64decode(b64)
except (ValueError, binascii.Error):
return None
def _parts_from_content(content: Any) -> list[types.Part]:
"""Convert OpenAI-style message content (str or multimodal list) to Gemini parts."""
if isinstance(content, list):
parts: list[types.Part] = []
for item in content:
if not isinstance(item, dict):
continue
if item.get("type") == "text":
parts.append(types.Part.from_text(text=item.get("text") or ""))
elif item.get("type") == "image_url":
decoded = _decode_data_uri((item.get("image_url") or {}).get("url", ""))
if decoded is not None:
mime, data = decoded
parts.append(types.Part.from_bytes(data=data, mime_type=mime))
# Gemini rejects empty parts; fall back to a single space.
return parts or [types.Part.from_text(text=" ")]
return [types.Part.from_text(text=content or "")]
def _stats_from_gemini_usage(usage: Any) -> dict[str, Any] | None:
"""Build a stats dict from a Gemini usage_metadata object."""
prompt_tokens = getattr(usage, "prompt_token_count", None)
@@ -227,9 +258,7 @@ class GeminiClient(GenAIClient):
)
else: # user
gemini_messages.append(
types.Content(
role="user", parts=[types.Part.from_text(text=content)]
)
types.Content(role="user", parts=_parts_from_content(content))
)
# Convert tools to Gemini format
@@ -485,9 +514,7 @@ class GeminiClient(GenAIClient):
)
else: # user
gemini_messages.append(
types.Content(
role="user", parts=[types.Part.from_text(text=content)]
)
types.Content(role="user", parts=_parts_from_content(content))
)
# Convert tools to Gemini format
@@ -553,7 +580,7 @@ class GeminiClient(GenAIClient):
# Use streaming API
content_parts: list[str] = []
reasoning_parts: list[str] = []
tool_calls_by_index: dict[int, dict[str, Any]] = {}
tool_calls_accum: list[dict[str, Any]] = []
finish_reason = "stop"
usage_stats: dict[str, Any] | None = None
@@ -600,7 +627,11 @@ class GeminiClient(GenAIClient):
content_parts.append(part.text)
yield ("content_delta", part.text)
elif part.function_call:
# Handle function call
# Gemini streams complete function calls (not partial
# argument deltas), so each part is a distinct tool
# call. Append rather than accumulate by name — the
# latter concatenated parallel/repeated calls into one
# invalid arguments string (e.g. `{...}{...}`).
try:
arguments = (
dict(part.function_call.args)
@@ -610,40 +641,16 @@ class GeminiClient(GenAIClient):
except Exception:
arguments = {}
# Store tool call
tool_call_id = part.function_call.name or ""
tool_call_name = part.function_call.name or ""
# Check if we already have this tool call
found_index = None
for idx, tc in tool_calls_by_index.items():
if tc["name"] == tool_call_name:
found_index = idx
break
if found_index is None:
found_index = len(tool_calls_by_index)
tool_calls_by_index[found_index] = {
"id": tool_call_id,
"name": tool_call_name,
"arguments": "",
"thought_signature": None,
tool_calls_accum.append(
{
"id": part.function_call.name or "",
"name": part.function_call.name or "",
"arguments": arguments,
"thought_signature": getattr(
part, "thought_signature", None
),
}
# Accumulate arguments
if arguments:
tool_calls_by_index[found_index]["arguments"] += (
json.dumps(arguments)
if isinstance(arguments, dict)
else str(arguments)
)
# Capture latest thought_signature for this call
chunk_sig = getattr(part, "thought_signature", None)
if chunk_sig:
tool_calls_by_index[found_index][
"thought_signature"
] = chunk_sig
)
# Build final message
full_content = "".join(content_parts).strip() or None
@@ -651,25 +658,20 @@ class GeminiClient(GenAIClient):
# Convert tool calls to list format
tool_calls_list = None
if tool_calls_by_index:
tool_calls_list = []
for tc in tool_calls_by_index.values():
try:
# Try to parse accumulated arguments as JSON
parsed_args = json.loads(tc["arguments"])
except (json.JSONDecodeError, Exception):
parsed_args = tc["arguments"]
tool_calls_list.append(
{
"id": tc["id"],
"name": tc["name"],
"arguments": parsed_args,
"thought_signature": _encode_thought_signature(
tc.get("thought_signature")
),
}
)
if tool_calls_accum:
tool_calls_list = [
{
"id": tc["id"],
"name": tc["name"],
"arguments": tc["arguments"]
if isinstance(tc["arguments"], dict)
else {},
"thought_signature": _encode_thought_signature(
tc.get("thought_signature")
),
}
for tc in tool_calls_accum
]
finish_reason = "tool_calls"
if usage_stats is not None:
+51 -31
View File
@@ -76,29 +76,6 @@ def _parse_launch_arg(args: list[str], flag: str) -> str | None:
return args[idx + 1]
def _fetch_llama_props(base_url: str, model: str) -> dict[str, Any]:
"""Fetch /props from a llama.cpp server, with llama-swap fallback.
Raises the underlying RequestException if both endpoints fail; callers
decide how to surface the failure.
"""
try:
response = requests.get(
f"{base_url}/props",
params={"model": model},
timeout=10,
)
response.raise_for_status()
return cast(dict[str, Any], response.json())
except Exception:
response = requests.get(
f"{base_url}/upstream/{model}/props",
timeout=10,
)
response.raise_for_status()
return cast(dict[str, Any], response.json())
def _to_jpeg(img_bytes: bytes) -> bytes | None:
"""Convert image bytes to JPEG. llama.cpp/STB does not support WebP."""
try:
@@ -128,6 +105,48 @@ class LlamaCppClient(GenAIClient):
_text_baseline_tokens: int | None
_media_marker: str
@property
def supports_embeddings(self) -> bool:
"""llama.cpp exposes an /embeddings endpoint for any loaded model."""
return True
def _auth_headers(self) -> dict | None:
"""Bearer auth header when an API key is configured, else None."""
if self.genai_config.api_key:
return {"Authorization": "Bearer " + self.genai_config.api_key}
return None
def _get(self, url: str, **kwargs: Any) -> requests.Response:
"""GET with the configured auth headers injected."""
return requests.get(url, headers=self._auth_headers(), **kwargs)
def _post(self, url: str, **kwargs: Any) -> requests.Response:
"""POST with the configured auth headers injected."""
return requests.post(url, headers=self._auth_headers(), **kwargs)
def _fetch_llama_props(self, base_url: str, model: str) -> dict[str, Any]:
"""Fetch /props from a llama.cpp server, with llama-swap fallback.
Raises the underlying RequestException if both endpoints fail; callers
decide how to surface the failure.
"""
try:
response = self._get(
f"{base_url}/props",
params={"model": model},
timeout=10,
)
response.raise_for_status()
return cast(dict[str, Any], response.json())
except Exception:
response = self._get(
f"{base_url}/upstream/{model}/props",
timeout=10,
)
response.raise_for_status()
return cast(dict[str, Any], response.json())
def _init_provider(self) -> str | None:
"""Initialize the client and query model metadata from the server."""
self.provider_options = {
@@ -211,7 +230,7 @@ class LlamaCppClient(GenAIClient):
model_entry: dict[str, Any] | None = None
try:
response = requests.get(f"{base_url}/v1/models", timeout=10)
response = self._get(f"{base_url}/v1/models", timeout=10)
response.raise_for_status()
models_data = response.json()
@@ -272,7 +291,7 @@ class LlamaCppClient(GenAIClient):
info["supports_tools"] = True
try:
props = _fetch_llama_props(base_url, configured_model)
props = self._fetch_llama_props(base_url, configured_model)
if info["context_size"] is None:
default_settings = props.get("default_generation_settings", {})
@@ -358,7 +377,7 @@ class LlamaCppClient(GenAIClient):
if self.supports_toggleable_thinking:
payload["chat_template_kwargs"] = {"enable_thinking": enable_thinking}
response = requests.post(
response = self._post(
f"{self.provider}/v1/chat/completions",
json=payload,
timeout=self.timeout,
@@ -408,7 +427,7 @@ class LlamaCppClient(GenAIClient):
if base_url is None:
return []
try:
response = requests.get(f"{base_url}/v1/models", timeout=10)
response = self._get(f"{base_url}/v1/models", timeout=10)
response.raise_for_status()
models = []
for m in response.json().get("data", []):
@@ -511,7 +530,7 @@ class LlamaCppClient(GenAIClient):
"messages": [{"role": "user", "content": content}],
"max_tokens": 1,
}
response = requests.post(
response = self._post(
f"{self.provider}/v1/chat/completions",
json=payload,
timeout=60,
@@ -621,7 +640,7 @@ class LlamaCppClient(GenAIClient):
if self.provider is None:
return False
try:
props = _fetch_llama_props(self.provider, self.genai_config.model)
props = self._fetch_llama_props(self.provider, self.genai_config.model)
except Exception as e:
logger.warning("Failed to refresh llama.cpp media marker: %s", e)
return False
@@ -682,7 +701,7 @@ class LlamaCppClient(GenAIClient):
return content
def post_embeddings() -> requests.Response:
return requests.post(
return self._post(
f"{self.provider}/embeddings",
json={"model": self.genai_config.model, "content": build_content()},
timeout=self.timeout,
@@ -786,7 +805,7 @@ class LlamaCppClient(GenAIClient):
stream=False,
enable_thinking=enable_thinking,
)
response = requests.post(
response = self._post(
f"{self.provider}/v1/chat/completions",
json=payload,
timeout=self.timeout,
@@ -867,6 +886,7 @@ class LlamaCppClient(GenAIClient):
"POST",
f"{self.provider}/v1/chat/completions",
json=payload,
headers=self._auth_headers(),
) as response:
response.raise_for_status()
async for line in response.aiter_lines():
+12 -3
View File
@@ -423,9 +423,18 @@ class OpenAIClient(GenAIClient):
for tc in tool_calls_by_index.values():
try:
# Parse accumulated arguments as JSON
parsed_args = json.loads(tc["arguments"])
except (json.JSONDecodeError, Exception):
parsed_args = tc["arguments"]
parsed_args = json.loads(tc["arguments"] or "{}")
except (json.JSONDecodeError, ValueError):
logger.warning(
"Failed to parse streamed tool call arguments for %s",
tc["name"],
)
parsed_args = {}
# Downstream (ToolCall model) requires a dict; never leak a
# partial/invalid arguments string.
if not isinstance(parsed_args, dict):
parsed_args = {}
tool_calls_list.append(
{
+20 -6
View File
@@ -6,7 +6,7 @@ transport.
"""
import datetime
from typing import Any
from typing import Any, Literal
from playhouse.shortcuts import model_to_dict
@@ -249,6 +249,7 @@ def get_attribute_classifications(config: FrigateConfig) -> list[dict[str, Any]]
def get_tool_definitions(
semantic_search_enabled: bool = False,
attribute_classifications: list[dict[str, Any]] | None = None,
embeddings_language: Literal["english", "multi"] = "multi",
) -> list[dict[str, Any]]:
"""
Get OpenAI-compatible tool definitions for Frigate.
@@ -258,7 +259,9 @@ def get_tool_definitions(
tool exposes an additional `semantic_query` parameter for descriptive
queries (e.g. "person riding a lawn mower") and find_similar_objects is
included. When attribute classification models are configured, an
`attribute` parameter is exposed for filtering by their labels.
`attribute` parameter is exposed for filtering by their labels. When the
embeddings model only understands English (JinaV1), the `semantic_query`
description instructs the model to write the query in English.
"""
search_objects_properties: dict[str, Any] = {
"camera": {
@@ -349,6 +352,14 @@ def get_tool_definitions(
"When set, combine with label/time/camera/zone filters as "
"usual (e.g. label='person', semantic_query='riding a lawn "
"mower', after='2024-05-01T00:00:00Z')."
+ (
" The configured embeddings model only understands "
"English, so always write semantic_query in English, "
"translating the user's description if they phrased it "
"in another language."
if embeddings_language == "english"
else ""
)
),
}
@@ -682,14 +693,17 @@ def build_chat_system_prompt(
if camera_config.friendly_name
else camera_id.replace("_", " ").title()
)
zone_names = list(camera_config.zones.keys())
zone_descriptors = [
f"{zone_config.get_formatted_name(zone_name)} (ID: {zone_name})"
for zone_name, zone_config in camera_config.zones.items()
]
if not has_speed_zone:
has_speed_zone = any(
zone.distances for zone in camera_config.zones.values()
)
if zone_names:
if zone_descriptors:
cameras_info.append(
f" - {friendly_name} (ID: {camera_id}, zones: {', '.join(zone_names)})"
f" - {friendly_name} (ID: {camera_id}, zones: {', '.join(zone_descriptors)})"
)
else:
cameras_info.append(f" - {friendly_name} (ID: {camera_id})")
@@ -699,7 +713,7 @@ def build_chat_system_prompt(
cameras_section = (
"\n\nAvailable cameras:\n"
+ "\n".join(cameras_info)
+ "\n\nWhen users refer to cameras by their friendly name (e.g., 'Back Deck Camera'), use the corresponding camera ID (e.g., 'back_deck_cam') in tool calls."
+ "\n\nWhen users refer to cameras or zones by their friendly name (e.g., 'Back Deck Camera', 'Front Walkway'), use the corresponding ID (e.g., 'back_deck_cam', 'front_walk') in tool calls. Tool results also identify zones by their ID, so when presenting cameras or zones back to the user, translate the ID to its friendly name."
)
speed_units_section = ""
+83 -103
View File
@@ -595,112 +595,92 @@ class BirdsEyeFrameManager:
) -> list[list[Any]] | None:
"""Calculate the optimal layout for 2+ cameras."""
def map_layout(
camera_layout: list[list[Any]], row_height: int
) -> tuple[int, int, list[list[Any]] | None]:
"""Map the calculated layout."""
candidate_layout = []
starting_x = 0
x = 0
max_width = 0
y = 0
def find_available_x(
current_x: int,
width: int,
reserved_ranges: list[tuple[int, int]],
max_width: int,
) -> int | None:
"""Find the first horizontal slot that does not collide with reservations."""
x = current_x
for row in camera_layout:
final_row = []
max_width = max(max_width, x)
x = starting_x
for cameras in row:
camera_dims = self.cameras[cameras[0]]["dimensions"].copy()
camera_aspect = cameras[1]
for reserved_start, reserved_end in sorted(reserved_ranges):
if x >= reserved_end:
continue
if camera_dims[1] > camera_dims[0]:
scaled_height = int(row_height * 2)
scaled_width = int(scaled_height * camera_aspect)
starting_x = scaled_width
else:
scaled_height = row_height
scaled_width = int(scaled_height * camera_aspect)
if x + width <= reserved_start:
return x
# layout is too large
if (
x + scaled_width > self.canvas.width
or y + scaled_height > self.canvas.height
):
return x + scaled_width, y + scaled_height, None
x = max(x, reserved_end)
final_row.append((cameras[0], (x, y, scaled_width, scaled_height)))
x += scaled_width
if x + width <= max_width:
return x
y += row_height
candidate_layout.append(final_row)
if max_width == 0:
max_width = x
return max_width, y, candidate_layout
canvas_aspect_x, canvas_aspect_y = self.canvas.get_aspect(coefficient)
camera_layout: list[list[Any]] = []
camera_layout.append([])
starting_x = 0
x = starting_x
y = 0
y_i = 0
max_y = 0
for camera in cameras_to_add:
camera_dims = self.cameras[camera]["dimensions"].copy()
camera_aspect_x, camera_aspect_y = self.canvas.get_camera_aspect(
camera, camera_dims[0], camera_dims[1]
)
if camera_dims[1] > camera_dims[0]:
portrait = True
else:
portrait = False
if (x + camera_aspect_x) <= canvas_aspect_x:
# insert if camera can fit on current row
camera_layout[y_i].append(
(
camera,
camera_aspect_x / camera_aspect_y,
)
)
if portrait:
starting_x = camera_aspect_x
else:
max_y = max(
max_y,
camera_aspect_y,
)
x += camera_aspect_x
else:
# move on to the next row and insert
y += max_y
y_i += 1
camera_layout.append([])
x = starting_x
if x + camera_aspect_x > canvas_aspect_x:
return None
camera_layout[y_i].append(
(
camera,
camera_aspect_x / camera_aspect_y,
)
)
x += camera_aspect_x
if y + max_y > canvas_aspect_y:
return None
row_height = int(self.canvas.height / coefficient)
total_width, total_height, standard_candidate_layout = map_layout(
camera_layout, row_height
)
def map_layout(row_height: int) -> tuple[int, int, list[list[Any]] | None]:
"""Lay out cameras row by row while reserving portrait spans for the next row."""
candidate_layout: list[list[Any]] = []
reserved_ranges: dict[int, list[tuple[int, int]]] = {}
current_row: list[Any] = []
row_index = 0
row_y = 0
row_x = 0
max_width = 0
max_height = 0
for camera in cameras_to_add:
camera_dims = self.cameras[camera]["dimensions"].copy()
camera_aspect_x, camera_aspect_y = self.canvas.get_camera_aspect(
camera, camera_dims[0], camera_dims[1]
)
portrait = camera_dims[1] > camera_dims[0]
scaled_height = row_height * 2 if portrait else row_height
scaled_width = int(scaled_height * (camera_aspect_x / camera_aspect_y))
while True:
x = find_available_x(
row_x,
scaled_width,
reserved_ranges.get(row_index, []),
self.canvas.width,
)
if x is not None and row_y + scaled_height <= self.canvas.height:
current_row.append(
(camera, (x, row_y, scaled_width, scaled_height))
)
row_x = x + scaled_width
max_width = max(max_width, row_x)
max_height = max(max_height, row_y + scaled_height)
if portrait:
reserved_ranges.setdefault(row_index + 1, []).append(
(x, row_x)
)
break
if current_row:
candidate_layout.append(current_row)
current_row = []
row_index += 1
row_y = row_index * row_height
row_x = 0
if row_y + scaled_height > self.canvas.height:
overflow_width = max(max_width, scaled_width)
overflow_height = row_y + scaled_height
return overflow_width, overflow_height, None
if current_row:
candidate_layout.append(current_row)
return max_width, max_height, candidate_layout
row_height = max(1, int(self.canvas.height / coefficient))
total_width, total_height, standard_candidate_layout = map_layout(row_height)
if not standard_candidate_layout:
# if standard layout didn't work
@@ -709,9 +689,9 @@ class BirdsEyeFrameManager:
total_width / self.canvas.width,
total_height / self.canvas.height,
)
row_height = int(row_height / scale_down_percent)
row_height = max(1, int(row_height / scale_down_percent))
total_width, total_height, standard_candidate_layout = map_layout(
camera_layout, row_height
row_height
)
if not standard_candidate_layout:
@@ -725,8 +705,8 @@ class BirdsEyeFrameManager:
1 / (total_width / self.canvas.width),
1 / (total_height / self.canvas.height),
)
row_height = int(row_height * scale_up_percent)
_, _, scaled_layout = map_layout(camera_layout, row_height)
row_height = max(1, int(row_height * scale_up_percent))
_, _, scaled_layout = map_layout(row_height)
if scaled_layout:
return scaled_layout
+156 -2
View File
@@ -1,11 +1,64 @@
"""Test camera user and password cleanup."""
"""Tests for Birdseye canvas sizing and layout behavior."""
import unittest
from multiprocessing import Event
from frigate.output.birdseye import get_canvas_shape
from frigate.config import FrigateConfig
from frigate.output.birdseye import BirdsEyeFrameManager, get_canvas_shape
class TestBirdseye(unittest.TestCase):
def _build_manager(
self, camera_dimensions: dict[str, tuple[int, int]]
) -> BirdsEyeFrameManager:
config = {
"mqtt": {"host": "mqtt"},
"birdseye": {"width": 1280, "height": 720},
"cameras": {},
}
for order, (camera, dimensions) in enumerate(
camera_dimensions.items(), start=1
):
config["cameras"][camera] = {
"ffmpeg": {
"inputs": [
{
"path": f"rtsp://10.0.0.1:554/{camera}",
"roles": ["detect"],
}
]
},
"detect": {
"width": dimensions[0],
"height": dimensions[1],
"fps": 5,
},
"birdseye": {"order": order},
}
return BirdsEyeFrameManager(FrigateConfig(**config), Event())
def _assert_no_overlaps(
self, layout: list[list[tuple[str, tuple[int, int, int, int]]]]
):
rectangles = [position for row in layout for _, position in row]
for index, rect in enumerate(rectangles):
x1, y1, width1, height1 = rect
for other in rectangles[index + 1 :]:
x2, y2, width2, height2 = other
overlap = (
x1 < x2 + width2
and x2 < x1 + width1
and y1 < y2 + height2
and y2 < y1 + height1
)
self.assertFalse(
overlap,
msg=f"Overlapping rectangles found: {rect} and {other}",
)
def test_16x9(self):
"""Test 16x9 aspect ratio works as expected for birdseye."""
width = 1280
@@ -45,3 +98,104 @@ class TestBirdseye(unittest.TestCase):
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width # width will be the same
assert canvas_height != height
def test_portrait_camera_does_not_overlap_next_row(self):
"""Portrait cameras should reserve their real horizontal position on the next row."""
manager = self._build_manager(
{
"cam_a": (1280, 720),
"cam_p": (360, 640),
"cam_b": (1280, 720),
"cam_c": (640, 480),
}
)
layout = manager.calculate_layout(["cam_a", "cam_p", "cam_b", "cam_c"], 3)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
cam_c = [
position for row in layout for camera, position in row if camera == "cam_c"
][0]
self.assertEqual(cam_c[0], 0)
def test_portrait_reservation_only_applies_to_next_row(self):
"""Portrait reservations should not push later rows after the span ends."""
manager = self._build_manager(
{
"cam_a": (1280, 720),
"cam_p": (360, 640),
"cam_b": (1280, 720),
"cam_c": (1280, 720),
"cam_d": (1280, 720),
"cam_e": (1280, 720),
}
)
layout = manager.calculate_layout(
["cam_a", "cam_p", "cam_b", "cam_c", "cam_d", "cam_e"],
3,
)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
cam_e = [
position for row in layout for camera, position in row if camera == "cam_e"
][0]
self.assertEqual(cam_e[0], 0)
def test_multiple_portraits_reserve_distinct_ranges(self):
"""Multiple portrait cameras in one row should reserve separate spans below them."""
manager = self._build_manager(
{
"cam_a": (640, 480),
"cam_p1": (360, 640),
"cam_p2": (360, 640),
"cam_b": (640, 480),
"cam_c": (1280, 720),
"cam_d": (640, 480),
}
)
layout = manager.calculate_layout(
["cam_a", "cam_p1", "cam_p2", "cam_b", "cam_c", "cam_d"],
4,
)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
def test_two_landscapes_then_portrait_then_two_landscapes(self):
"""A portrait after two landscapes should reserve only its own tail span."""
manager = self._build_manager(
{
"cam_a": (1280, 720),
"cam_b": (1280, 720),
"cam_p": (360, 640),
"cam_c": (1280, 720),
"cam_d": (1280, 720),
}
)
layout = manager.calculate_layout(
["cam_a", "cam_b", "cam_p", "cam_c", "cam_d"],
3,
)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
cam_c = [
position for row in layout for camera, position in row if camera == "cam_c"
][0]
cam_d = [
position for row in layout for camera, position in row if camera == "cam_d"
][0]
self.assertEqual(cam_c[0], 0)
self.assertEqual(cam_d[0], cam_c[0] + cam_c[2])
+37
View File
@@ -397,6 +397,43 @@ class TestConfig(unittest.TestCase):
assert "dog" in frigate_config.cameras["back"].objects.filters
assert frigate_config.cameras["back"].objects.filters["dog"].threshold == 0.7
def test_unsupported_tracked_object_pruned_from_track_and_filters(self):
# "unicorn" is not in the model labelmap, so it must be removed from the
# tracked objects AND from the object filters, otherwise a stale filter
# entry lingers in the parsed config.
config = {
"mqtt": {"host": "mqtt"},
"cameras": {
"back": {
"ffmpeg": {
"inputs": [
{"path": "rtsp://10.0.0.1:554/video", "roles": ["detect"]}
]
},
"detect": {
"height": 1080,
"width": 1920,
"fps": 5,
},
"objects": {
"track": ["person", "unicorn"],
"filters": {
"person": {"threshold": 0.7},
"unicorn": {"threshold": 0.7},
},
},
}
},
}
frigate_config = FrigateConfig(**config)
objects = frigate_config.cameras["back"].objects
assert "unicorn" not in objects.track
assert "unicorn" not in objects.filters
# supported entries are left untouched
assert "person" in objects.track
assert "person" in objects.filters
def test_global_object_mask(self):
config = {
"mqtt": {"host": "mqtt"},
+496
View File
@@ -0,0 +1,496 @@
"""Smoke tests for GenAI chat providers.
Each provider's ``chat_with_tools_stream`` is driven with a canned "test
response" so the two conversion layers are exercised without any network:
1. Frigate (OpenAI-style) messages -> provider-native request format
2. provider-native response -> Frigate ``("kind", value)`` stream events
These guard against regressions such as tool-call arguments arriving as raw
strings instead of dicts (which crash the ``ToolCall`` model), and multimodal
user content (a list of text/image parts, as injected by ``get_live_context``)
crashing message conversion.
"""
import asyncio
import base64
import json
import unittest
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
from frigate.config import GenAIConfig, GenAIProviderEnum
from frigate.genai import PROVIDERS, load_providers
load_providers()
# A minimal but valid JPEG data URI, mirroring what get_live_context injects.
_TINY_JPEG = base64.b64encode(b"\xff\xd8\xff\xd9").decode("ascii")
_IMAGE_DATA_URI = f"data:image/jpeg;base64,{_TINY_JPEG}"
# Conversation ending in a multimodal user message (text + live image), the
# exact shape the chat endpoint builds after a get_live_context tool result.
MULTIMODAL_MESSAGES = [
{"role": "system", "content": "You are a test assistant."},
{"role": "user", "content": "what do you see on the front camera?"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_1",
"type": "function",
"function": {
"name": "get_live_context",
"arguments": json.dumps({"camera": "front"}),
},
}
],
},
{
"role": "tool",
"tool_call_id": "call_1",
"name": "get_live_context",
"content": json.dumps({"camera": "front"}),
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Here is the current live image from camera 'front'.",
},
{"type": "image_url", "image_url": {"url": _IMAGE_DATA_URI}},
],
},
]
SIMPLE_MESSAGES = [
{"role": "system", "content": "You are a test assistant."},
{"role": "user", "content": "hello"},
]
TOOLS = [
{
"type": "function",
"function": {
"name": "search_objects",
"description": "Search tracked objects",
"parameters": {
"type": "object",
"properties": {"label": {"type": "string"}},
},
},
}
]
def _make_client(provider: str, **cfg_overrides):
"""Build a provider client offline (no model validation, no network)."""
cfg = GenAIConfig(provider=provider, **cfg_overrides)
cls = PROVIDERS[GenAIProviderEnum(provider)]
return cls(cfg, timeout=5, validate_model=False)
def _collect(client, messages, tools=TOOLS):
"""Drain chat_with_tools_stream into a list of (kind, value) events."""
async def _run():
events = []
async for event in client.chat_with_tools_stream(
messages=messages, tools=tools, tool_choice="auto"
):
events.append(event)
return events
return asyncio.run(_run())
def _final_message(events) -> dict:
messages = [value for (kind, value) in events if kind == "message"]
assert messages, f"stream produced no final message: {events}"
return messages[-1]
def _assert_tool_args_are_dicts(final: dict) -> None:
"""Every returned tool call must expose arguments as a dict, never a string."""
for tool_call in final.get("tool_calls") or []:
assert isinstance(tool_call["arguments"], dict), (
f"tool call arguments must be a dict, got "
f"{type(tool_call['arguments']).__name__}: {tool_call['arguments']!r}"
)
# ---------------------------------------------------------------------------
# OpenAI
# ---------------------------------------------------------------------------
def _openai_tc(index, id=None, name=None, arguments=None):
return SimpleNamespace(
index=index,
id=id,
function=SimpleNamespace(name=name, arguments=arguments),
)
def _openai_chunk(content=None, tool_calls=None, finish_reason=None, usage=None):
delta = SimpleNamespace(
content=content,
tool_calls=tool_calls,
reasoning_content=None,
reasoning=None,
)
choice = SimpleNamespace(delta=delta, finish_reason=finish_reason)
return SimpleNamespace(choices=[choice], usage=usage)
class TestOpenAIProvider(unittest.TestCase):
def _client(self):
return _make_client(
"openai", model="gpt-4o", api_key="k", base_url="http://localhost:9999/v1"
)
def test_stream_tool_call_arguments_are_dict(self):
# Arguments arrive split across chunks, as the real API streams them.
chunks = [
_openai_chunk(
tool_calls=[
_openai_tc(0, id="c1", name="search_objects", arguments='{"label":')
]
),
_openai_chunk(tool_calls=[_openai_tc(0, arguments=' "person"}')]),
_openai_chunk(finish_reason="tool_calls"),
]
client = self._client()
client.provider.chat.completions.create = MagicMock(return_value=iter(chunks))
final = _final_message(_collect(client, SIMPLE_MESSAGES))
self.assertEqual(final["finish_reason"], "tool_calls")
self.assertEqual(len(final["tool_calls"]), 1)
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
def test_stream_content_response(self):
chunks = [
_openai_chunk(content="hel"),
_openai_chunk(content="lo"),
_openai_chunk(finish_reason="stop"),
]
client = self._client()
client.provider.chat.completions.create = MagicMock(return_value=iter(chunks))
events = _collect(client, SIMPLE_MESSAGES)
deltas = [v for (k, v) in events if k == "content_delta"]
self.assertEqual("".join(deltas), "hello")
self.assertEqual(_final_message(events)["content"], "hello")
def test_multimodal_message_does_not_crash(self):
client = self._client()
client.provider.chat.completions.create = MagicMock(
return_value=iter([_openai_chunk(content="ok", finish_reason="stop")])
)
# Passing the OpenAI-native multimodal list through must not raise.
final = _final_message(_collect(client, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
# ---------------------------------------------------------------------------
# Gemini
# ---------------------------------------------------------------------------
def _gemini_part(text=None, thought=False, function_call=None, thought_signature=None):
return SimpleNamespace(
text=text,
thought=thought,
function_call=function_call,
thought_signature=thought_signature,
)
def _gemini_chunk(parts, finish_reason=None, usage_metadata=None):
candidate = SimpleNamespace(
content=SimpleNamespace(parts=parts), finish_reason=finish_reason
)
return SimpleNamespace(candidates=[candidate], usage_metadata=usage_metadata)
def _gemini_stream(chunks):
async def _agen(*args, **kwargs):
for chunk in chunks:
yield chunk
return _agen
class TestGeminiProvider(unittest.TestCase):
def _client(self):
return _make_client("gemini", model="gemini-2.5-flash", api_key="k")
def _patch_stream(self, client, chunks):
client.provider = MagicMock()
client.provider.aio.models.generate_content_stream = AsyncMock(
side_effect=_gemini_stream(chunks)
)
def test_stream_parallel_tool_calls_stay_separate_dicts(self):
# Regression: Gemini streams complete function calls. Two calls to the
# same tool must NOT be merged into one concatenated arguments string.
from google.genai.types import FinishReason
chunks = [
_gemini_chunk(
parts=[
_gemini_part(
function_call=SimpleNamespace(
name="search_objects", args={"label": "person"}
)
),
_gemini_part(
function_call=SimpleNamespace(
name="search_objects", args={"limit": 1}
)
),
],
finish_reason=FinishReason.STOP,
),
]
client = self._client()
self._patch_stream(client, chunks)
final = _final_message(_collect(client, SIMPLE_MESSAGES))
self.assertEqual(final["finish_reason"], "tool_calls")
self.assertEqual(len(final["tool_calls"]), 2)
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
self.assertEqual(final["tool_calls"][1]["arguments"], {"limit": 1})
def test_stream_content_response(self):
from google.genai.types import FinishReason
chunks = [
_gemini_chunk(parts=[_gemini_part(text="hel")]),
_gemini_chunk(
parts=[_gemini_part(text="lo")], finish_reason=FinishReason.STOP
),
]
client = self._client()
self._patch_stream(client, chunks)
events = _collect(client, SIMPLE_MESSAGES)
deltas = [v for (k, v) in events if k == "content_delta"]
self.assertEqual("".join(deltas), "hello")
self.assertEqual(_final_message(events)["content"], "hello")
def test_multimodal_message_converts_without_crash(self):
# Regression: a user message with list content (text + image_url) used
# to be handed to Part.from_text(text=<list>) and raise ValidationError.
from google.genai.types import FinishReason
client = self._client()
self._patch_stream(
client,
[
_gemini_chunk(
parts=[_gemini_part(text="ok")], finish_reason=FinishReason.STOP
)
],
)
final = _final_message(_collect(client, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
# ---------------------------------------------------------------------------
# Ollama
# ---------------------------------------------------------------------------
class TestOllamaProvider(unittest.TestCase):
def _client(self):
return _make_client("ollama", model="llama3", base_url="http://localhost:9999")
def _run_with_response(self, client, response, messages):
# Ollama uses a non-streaming call when tools are present, via an
# internally-constructed async client.
fake_async = MagicMock()
fake_async.chat = AsyncMock(return_value=response)
with patch(
"frigate.genai.plugins.ollama.OllamaAsyncClient",
return_value=fake_async,
):
return _collect(client, messages)
def test_tool_call_arguments_are_dict(self):
response = {
"message": {
"content": "",
"tool_calls": [
{
"function": {
"name": "search_objects",
"arguments": {"label": "person"},
}
}
],
},
"done": True,
"done_reason": "stop",
"eval_count": 5,
"prompt_eval_count": 3,
"eval_duration": 1_000_000,
}
client = self._client()
final = _final_message(
self._run_with_response(client, response, SIMPLE_MESSAGES)
)
self.assertEqual(final["finish_reason"], "tool_calls")
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
def test_multimodal_message_normalizes_image(self):
# Ollama needs content as a string with images pulled into a separate
# field; the normalizer must extract both without crashing.
response = {
"message": {"content": "ok"},
"done": True,
"done_reason": "stop",
}
client = self._client()
final = _final_message(
self._run_with_response(client, response, MULTIMODAL_MESSAGES)
)
self.assertEqual(final["content"], "ok")
def test_normalize_multimodal_content(self):
from frigate.genai.plugins.ollama import _normalize_multimodal_content
text, images = _normalize_multimodal_content(MULTIMODAL_MESSAGES[-1]["content"])
self.assertIn("live image", text)
self.assertEqual(len(images), 1)
self.assertEqual(images[0], b"\xff\xd8\xff\xd9")
# ---------------------------------------------------------------------------
# llama.cpp
# ---------------------------------------------------------------------------
class _FakeStreamResponse:
def __init__(self, lines):
self._lines = lines
def raise_for_status(self):
return None
async def aiter_lines(self):
for line in self._lines:
yield line
class _FakeStreamCtx:
def __init__(self, lines):
self._resp = _FakeStreamResponse(lines)
async def __aenter__(self):
return self._resp
async def __aexit__(self, *exc):
return False
class _FakeAsyncClient:
def __init__(self, lines):
self._lines = lines
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return False
def stream(self, method, url, json=None, headers=None):
return _FakeStreamCtx(self._lines)
class TestLlamaCppProvider(unittest.TestCase):
def _client(self):
return _make_client("llamacpp", model="m", base_url="http://localhost:9999")
def _run_with_lines(self, client, lines, messages):
with patch(
"frigate.genai.plugins.llama_cpp.httpx.AsyncClient",
return_value=_FakeAsyncClient(lines),
):
return _collect(client, messages)
def test_stream_tool_call_arguments_are_dict(self):
lines = [
"data: "
+ json.dumps(
{
"choices": [
{
"delta": {
"tool_calls": [
{
"index": 0,
"id": "c1",
"function": {
"name": "search_objects",
"arguments": '{"label":',
},
}
]
}
}
]
}
),
"data: "
+ json.dumps(
{
"choices": [
{
"delta": {
"tool_calls": [
{
"index": 0,
"function": {"arguments": ' "person"}'},
}
]
}
}
]
}
),
"data: "
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "tool_calls"}]}),
"data: [DONE]",
]
client = self._client()
final = _final_message(self._run_with_lines(client, lines, SIMPLE_MESSAGES))
self.assertEqual(final["finish_reason"], "tool_calls")
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
def test_stream_content_response(self):
lines = [
"data: " + json.dumps({"choices": [{"delta": {"content": "hel"}}]}),
"data: " + json.dumps({"choices": [{"delta": {"content": "lo"}}]}),
"data: "
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "stop"}]}),
"data: [DONE]",
]
client = self._client()
events = self._run_with_lines(client, lines, SIMPLE_MESSAGES)
deltas = [v for (k, v) in events if k == "content_delta"]
self.assertEqual("".join(deltas), "hello")
self.assertEqual(_final_message(events)["content"], "hello")
def test_multimodal_message_does_not_crash(self):
lines = [
"data: " + json.dumps({"choices": [{"delta": {"content": "ok"}}]}),
"data: "
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "stop"}]}),
"data: [DONE]",
]
client = self._client()
final = _final_message(self._run_with_lines(client, lines, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
if __name__ == "__main__":
unittest.main()
+70 -12
View File
@@ -40,18 +40,18 @@ class TestGpuStats(unittest.TestCase):
"driver": "i915",
"pid": "100",
"engines": {
"render": (1_000_000_000, 0),
"video": (5_000_000_000, 0),
"video-enhance": (200_000_000, 0),
"compute": (0, 0),
"render": (1_000_000_000, 0, 1),
"video": (5_000_000_000, 0, 1),
"video-enhance": (200_000_000, 0, 1),
"compute": (0, 0, 1),
},
},
("0000:00:02.0", "2", "200"): {
"driver": "i915",
"pid": "200",
"engines": {
"render": (0, 0),
"compute": (2_000_000_000, 0),
"render": (0, 0, 1),
"compute": (2_000_000_000, 0, 1),
},
},
}
@@ -60,18 +60,18 @@ class TestGpuStats(unittest.TestCase):
"driver": "i915",
"pid": "100",
"engines": {
"render": (1_200_000_000, 0),
"video": (5_500_000_000, 0),
"video-enhance": (300_000_000, 0),
"compute": (0, 0),
"render": (1_200_000_000, 0, 1),
"video": (5_500_000_000, 0, 1),
"video-enhance": (300_000_000, 0, 1),
"compute": (0, 0, 1),
},
},
("0000:00:02.0", "2", "200"): {
"driver": "i915",
"pid": "200",
"engines": {
"render": (0, 0),
"compute": (2_100_000_000, 0),
"render": (0, 0, 1),
"compute": (2_100_000_000, 0, 1),
},
},
}
@@ -92,6 +92,64 @@ class TestGpuStats(unittest.TestCase):
},
}
@patch("frigate.stats.intel_gpu_info.intel_gpu_name_resolver.get_names")
@patch("frigate.util.services.time.sleep")
@patch("frigate.util.services.time.monotonic")
@patch("frigate.util.services._read_intel_drm_fdinfo")
def test_intel_gpu_stats_xe_capacity(
self, read_fdinfo, monotonic, sleep, get_names
):
# Xe engines report cumulative cycles paired with total cycles, plus a
# per-class capacity. drm-cycles-* is summed across every instance of a
# class, so on Battlemage (capacity 2 for vcs/vecs) busy/total must be
# divided by capacity to land in 0-100%.
monotonic.side_effect = [0.0, 1.0]
get_names.return_value = {"0000:03:00.0": "Intel Arc"}
# Deltas over the window (busy, total): render 200/1000 cap 1 = 20%,
# video 800/1000 cap 2 = 40%, video-enhance 400/1000 cap 2 = 20%,
# compute 100/1000 cap 1 = 10%. Without the capacity divisor video/
# video-enhance would read 80%/40% and dec would clamp at 100%.
snapshot_a = {
("0000:03:00.0", "1", "300"): {
"driver": "xe",
"pid": "300",
"engines": {
"render": (0, 0, 1),
"video": (0, 0, 2),
"video-enhance": (0, 0, 2),
"compute": (0, 0, 1),
},
},
}
snapshot_b = {
("0000:03:00.0", "1", "300"): {
"driver": "xe",
"pid": "300",
"engines": {
"render": (200, 1000, 1),
"video": (800, 1000, 2),
"video-enhance": (400, 1000, 2),
"compute": (100, 1000, 1),
},
},
}
read_fdinfo.side_effect = [snapshot_a, snapshot_b]
intel_stats = get_intel_gpu_stats(None)
assert intel_stats == {
"0000:03:00.0": {
"name": "Intel Arc",
"vendor": "intel",
"gpu": "90.0%",
"mem": "-%",
"compute": "30.0%",
"dec": "60.0%",
"clients": {"300": "90.0%"},
},
}
@patch("frigate.util.services._read_intel_drm_fdinfo")
def test_intel_gpu_stats_no_clients(self, read_fdinfo):
read_fdinfo.return_value = {}
+54
View File
@@ -0,0 +1,54 @@
"""Tests for the REGEXP function registered on the main Frigate database.
Regression coverage for GHSA-q8jx-q884-jcq9: an attacker-controlled
catastrophic (ReDoS) pattern reaching the REGEXP sink must not be able to
stall the serialized database worker thread.
"""
import sqlite3
import time
import unittest
from frigate.db.sqlitevecq import REGEXP_TIMEOUT_SECONDS, SqliteVecQueueDatabase
class TestRegexpFunction(unittest.TestCase):
def setUp(self) -> None:
# autostart=False keeps the queue worker thread from spinning up; we
# only need the REGEXP registration, exercised on our own connection.
self.db = SqliteVecQueueDatabase(":memory:", autostart=False)
self.conn = sqlite3.connect(":memory:")
self.db._register_regexp(self.conn)
def tearDown(self) -> None:
self.conn.close()
def _regexp(self, value: str | None, pattern: str) -> int | None:
# SQLite maps "value REGEXP pattern" to regexp(pattern, value).
return self.conn.execute("SELECT ? REGEXP ?", (value, pattern)).fetchone()[0]
def test_normal_patterns_still_match(self) -> None:
self.assertTrue(self._regexp("ABC123", "^ABC"))
self.assertTrue(self._regexp("ABC123", "ABC.*"))
self.assertTrue(self._regexp("ABC123", "[0-9]+$"))
self.assertFalse(self._regexp("ABC123", "^XYZ"))
def test_null_value_does_not_match(self) -> None:
self.assertFalse(self._regexp(None, ".*"))
def test_invalid_pattern_does_not_raise(self) -> None:
self.assertFalse(self._regexp("ABC123", "(unclosed"))
def test_catastrophic_pattern_is_time_bounded(self) -> None:
# Without the timeout this evaluation backtracks for minutes to hours
# and wedges the whole database thread (GHSA-q8jx-q884-jcq9).
catastrophic = "(a{2,})+c"
subject = "a" * 4000
start = time.monotonic()
result = self._regexp(subject, catastrophic)
elapsed = time.monotonic() - start
# The pattern does not match; the guarantee is that it returns quickly.
self.assertFalse(result)
self.assertLess(elapsed, REGEXP_TIMEOUT_SECONDS + 2.0)
+24 -5
View File
@@ -360,7 +360,7 @@ def _read_intel_drm_fdinfo(target_pdev: str | None) -> dict:
if key in snapshot:
continue
engines: dict[str, tuple[int, int]] = {}
engines: dict[str, tuple[int, int, int]] = {}
if driver == "i915":
for fkey, engine in _I915_ENGINE_KEYS.items():
@@ -368,19 +368,34 @@ def _read_intel_drm_fdinfo(target_pdev: str | None) -> dict:
if not raw:
continue
try:
engines[engine] = (int(raw.split()[0]), 0)
engines[engine] = (int(raw.split()[0]), 0, 1)
except (ValueError, IndexError):
continue
else:
for suffix, engine in _XE_ENGINE_KEYS.items():
busy_raw = fields.get(f"drm-cycles-{suffix}")
total_raw = fields.get(f"drm-total-cycles-{suffix}")
if not (busy_raw and total_raw):
continue
# drm-cycles-* is summed across every instance of the engine
# class while drm-total-cycles-* tracks a single instance, so
# busy/total scales up to the capacity (e.g. Battlemage
# reports 2 for vcs/vecs). Capture it to divide back out;
# absent means a single engine, so default to 1.
capacity_raw = fields.get(f"drm-engine-capacity-{suffix}")
try:
capacity = int(capacity_raw.split()[0]) if capacity_raw else 1
except (ValueError, IndexError):
capacity = 1
try:
engines[engine] = (
int(busy_raw.split()[0]),
int(total_raw.split()[0]),
max(1, capacity),
)
except (ValueError, IndexError):
continue
@@ -450,11 +465,13 @@ def get_intel_gpu_stats(
pid_pct = per_pdev_pid_pct.setdefault(pdev, {})
client_total = 0.0
for engine, (busy_b, total_b) in data_b["engines"].items():
for engine, (busy_b, total_b, capacity) in data_b["engines"].items():
if engine not in engine_pct:
continue
busy_a, total_a = data_a["engines"].get(engine, (busy_b, total_b))
busy_a, total_a, _ = data_a["engines"].get(
engine, (busy_b, total_b, capacity)
)
if data_b["driver"] == "i915":
delta = max(0, busy_b - busy_a)
@@ -464,7 +481,9 @@ def get_intel_gpu_stats(
delta_total = total_b - total_a
if delta_total <= 0:
continue
pct = min(100.0, delta_busy / delta_total * 100.0)
# Normalize by capacity so a class with N engine instances
# (busy summed across all N) reports 0-100%, not 0-N*100%.
pct = min(100.0, delta_busy / (delta_total * capacity) * 100.0)
engine_pct[engine] += pct
client_total += pct
@@ -1,5 +1,6 @@
{
"documentTitle": "Classification Models - Frigate",
"disabled": "Disabled",
"details": {
"scoreInfo": "Score represents the average classification confidence across all detections of this object.",
"none": "None",
@@ -64,7 +65,20 @@
"title": "Edit Classification Model",
"descriptionState": "Edit the classes for this state classification model. Changes will require retraining the model.",
"descriptionObject": "Edit the object type and classification type for this object classification model.",
"stateClassesInfo": "Note: Changing state classes requires retraining the model with the updated classes."
"enabled": "Enabled",
"enabledDesc": "Run this model. When disabled, it stops running and no longer classifies.",
"saveAttempts": "Save Attempts",
"saveAttemptsDesc": "Number of classification attempt images to keep for the recent classifications UI.",
"motion": "Run on Motion",
"motionDesc": "Run classification when motion is detected within the configured crop.",
"interval": "Interval",
"intervalDesc": "Seconds between periodic classification runs. Leave empty to run only on motion.",
"intervalPlaceholder": "No interval",
"stateClassesInfo": "Model updated. Retrain the model for the class changes to take effect.",
"errors": {
"saveAttemptsInvalid": "Save attempts must be a whole number of 0 or greater",
"intervalInvalid": "Interval must be a whole number greater than 0"
}
},
"deleteDatasetImages": {
"title": "Delete Dataset Images",
@@ -9,6 +9,7 @@ import {
import {
Form,
FormControl,
FormDescription,
FormField,
FormItem,
FormLabel,
@@ -17,6 +18,7 @@ import {
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group";
import { Switch } from "@/components/ui/switch";
import {
Select,
SelectContent,
@@ -50,14 +52,25 @@ type ClassificationModelEditDialogProps = {
type ObjectClassificationType = "sub_label" | "attribute";
type ObjectFormData = {
enabled: boolean;
saveAttempts: number;
objectLabel: string;
objectType: ObjectClassificationType;
};
type StateFormData = {
enabled: boolean;
saveAttempts: number;
motion: boolean;
interval?: number;
classes: string[];
};
const DEFAULT_SAVE_ATTEMPTS = {
object: 200,
state: 100,
} as const;
export default function ClassificationModelEditDialog({
open,
model,
@@ -71,6 +84,10 @@ export default function ClassificationModelEditDialog({
const isStateModel = model.state_config !== undefined;
const isObjectModel = model.object_config !== undefined;
const defaultSaveAttempts = isObjectModel
? DEFAULT_SAVE_ATTEMPTS.object
: DEFAULT_SAVE_ATTEMPTS.state;
const objectLabels = useMemo(() => {
if (!config) return [];
@@ -93,8 +110,17 @@ export default function ClassificationModelEditDialog({
// Define form schema based on model type
const formSchema = useMemo(() => {
const sharedFields = {
enabled: z.boolean(),
saveAttempts: z.coerce
.number({ message: t("edit.errors.saveAttemptsInvalid") })
.int(t("edit.errors.saveAttemptsInvalid"))
.min(0, t("edit.errors.saveAttemptsInvalid")),
};
if (isObjectModel) {
return z.object({
...sharedFields,
objectLabel: z
.string()
.min(1, t("wizard.step1.errors.objectLabelRequired")),
@@ -103,6 +129,17 @@ export default function ClassificationModelEditDialog({
} else {
// State model
return z.object({
...sharedFields,
motion: z.boolean(),
interval: z.preprocess(
(val) =>
val === "" || val === null || val === undefined ? undefined : val,
z.coerce
.number({ message: t("edit.errors.intervalInvalid") })
.int(t("edit.errors.intervalInvalid"))
.positive(t("edit.errors.intervalInvalid"))
.optional(),
),
classes: z
.array(z.string())
.min(1, t("wizard.step1.errors.classRequired"))
@@ -129,12 +166,18 @@ export default function ClassificationModelEditDialog({
resolver: zodResolver(formSchema),
defaultValues: isObjectModel
? ({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
objectLabel: model.object_config?.objects?.[0] || "",
objectType:
(model.object_config
?.classification_type as ObjectClassificationType) || "sub_label",
} as ObjectFormData)
: ({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
motion: model.state_config?.motion ?? false,
interval: model.state_config?.interval,
classes: [""], // Will be populated from dataset
} as StateFormData),
mode: "onChange",
@@ -151,6 +194,8 @@ export default function ClassificationModelEditDialog({
if (open) {
if (isObjectModel) {
form.reset({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
objectLabel: model.object_config?.objects?.[0] || "",
objectType:
(model.object_config
@@ -158,6 +203,10 @@ export default function ClassificationModelEditDialog({
} as ObjectFormData);
} else {
form.reset({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
motion: model.state_config?.motion ?? false,
interval: model.state_config?.interval,
classes: [""],
} as StateFormData);
}
@@ -166,7 +215,15 @@ export default function ClassificationModelEditDialog({
mutateDataset();
}
}
}, [open, isObjectModel, isStateModel, model, form, mutateDataset]);
}, [
open,
isObjectModel,
isStateModel,
model,
form,
mutateDataset,
defaultSaveAttempts,
]);
// Update form with classes from dataset when loaded
useEffect(() => {
@@ -233,6 +290,7 @@ export default function ClassificationModelEditDialog({
setIsSaving(true);
try {
if (isObjectModel) {
// object model save
const objectData = data as ObjectFormData;
// Update the config
@@ -243,9 +301,10 @@ export default function ClassificationModelEditDialog({
classification: {
custom: {
[model.name]: {
enabled: model.enabled,
enabled: objectData.enabled,
name: model.name,
threshold: model.threshold,
save_attempts: objectData.saveAttempts,
object_config: {
objects: [objectData.objectLabel],
classification_type: objectData.objectType,
@@ -260,7 +319,34 @@ export default function ClassificationModelEditDialog({
position: "top-center",
});
} else {
// state model save
const stateData = data as StateFormData;
const stateConfig: { motion: boolean; interval?: number | null } = {
motion: stateData.motion,
};
if (stateData.interval != null) {
stateConfig.interval = stateData.interval;
} else if (model.state_config?.interval != null) {
stateConfig.interval = null;
}
await axios.put("/config/set", {
requires_restart: 0,
update_topic: `config/classification/custom/${model.name}`,
config_data: {
classification: {
custom: {
[model.name]: {
enabled: stateData.enabled,
save_attempts: stateData.saveAttempts,
state_config: stateConfig,
},
},
},
},
});
const newClasses = stateData.classes.filter(
(c) => c.trim().length > 0,
);
@@ -307,11 +393,11 @@ export default function ClassificationModelEditDialog({
if (renamePromises.length > 0) {
await Promise.all(renamePromises);
await mutate(`classification/${model.name}/dataset`);
toast.success(t("toast.success.updatedModel"), {
toast.success(t("edit.stateClassesInfo"), {
position: "top-center",
});
} else {
toast.info(t("edit.stateClassesInfo"), {
toast.success(t("toast.success.updatedModel"), {
position: "top-center",
});
}
@@ -359,6 +445,29 @@ export default function ClassificationModelEditDialog({
<div className="space-y-6">
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
<FormField
control={form.control}
name="enabled"
render={({ field }) => (
<FormItem className="flex flex-row items-center justify-between gap-4">
<div className="space-y-0.5">
<FormLabel className="text-primary-variant">
{t("edit.enabled")}
</FormLabel>
<FormDescription className="text-xs">
{t("edit.enabledDesc")}
</FormDescription>
</div>
<FormControl>
<Switch
checked={field.value}
onCheckedChange={field.onChange}
/>
</FormControl>
</FormItem>
)}
/>
{isObjectModel && (
<>
<FormField
@@ -520,6 +629,77 @@ export default function ClassificationModelEditDialog({
</div>
)}
{isStateModel && (
<>
<FormField
control={form.control}
name="motion"
render={({ field }) => (
<FormItem className="flex flex-row items-center justify-between gap-4">
<div className="space-y-0.5">
<FormLabel className="text-primary-variant">
{t("edit.motion")}
</FormLabel>
<FormDescription className="text-xs">
{t("edit.motionDesc")}
</FormDescription>
</div>
<FormControl>
<Switch
checked={field.value}
onCheckedChange={field.onChange}
/>
</FormControl>
</FormItem>
)}
/>
<FormField
control={form.control}
name="interval"
render={({ field }) => (
<FormItem>
<FormLabel className="text-primary-variant">
{t("edit.interval")}
</FormLabel>
<FormControl>
<Input
className="h-8"
inputMode="numeric"
placeholder={t("edit.intervalPlaceholder")}
{...field}
value={field.value ?? ""}
/>
</FormControl>
<FormDescription className="text-xs">
{t("edit.intervalDesc")}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
</>
)}
<FormField
control={form.control}
name="saveAttempts"
render={({ field }) => (
<FormItem>
<FormLabel className="text-primary-variant">
{t("edit.saveAttempts")}
</FormLabel>
<FormControl>
<Input className="h-8" inputMode="numeric" {...field} />
</FormControl>
<FormDescription className="text-xs">
{t("edit.saveAttemptsDesc")}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
<div className="flex flex-col gap-3 pt-3 sm:flex-row sm:justify-end sm:gap-4">
<Button
type="button"
@@ -169,6 +169,13 @@ const detect: SectionConfigOverrides = {
resolution: ["width", "height", "fps"],
tracking: ["min_initialized", "max_disappeared"],
},
uiSchema: {
annotation_offset: {
"ui:options": {
signed: true,
},
},
},
hiddenFields: ["enabled_in_config"],
advancedFields: [
"min_initialized",
@@ -57,6 +57,7 @@ const record: SectionConfigOverrides = {
"ui:options": {
suppressMultiSchema: true,
ffmpegPresetField: "hwaccel_args",
ffmpegGlobalFieldPath: "export.hwaccel_args",
},
},
},
@@ -17,3 +17,4 @@ export {
isSubtreeModified,
} from "./overrides";
export { getSizedFieldClassName } from "./fieldSizing";
export { getNumericInputMode } from "./inputMode";
@@ -0,0 +1,51 @@
import type { RJSFSchema } from "@rjsf/utils";
type NumericInputOptions = {
signed?: boolean;
};
/**
* Derive the on-screen keyboard hint for a schema field.
*
* Numeric config fields render as text inputs because RJSF's NumberField
* relies on the widget echoing raw strings back, so that trailing "." and "0"
* characters survive while a value is being typed. That means the numeric
* keypad has to be requested explicitly. Desktop browsers ignore inputMode, so
* this only affects virtual keyboards.
*
* Fields accepting negative values opt out, since the iOS numeric and decimal
* keypads have no minus key. Most numeric fields declare no minimum even
* though they are non-negative, so signed fields are marked explicitly with
* ui:options.signed.
*
* Args:
* schema: The JSON schema for the field being rendered
* options: The resolved ui:options for the field
*
* Returns:
* The inputMode to apply, or undefined to leave the keyboard alone
*/
export function getNumericInputMode(
schema: RJSFSchema,
options: unknown,
): "numeric" | "decimal" | undefined {
const types = Array.isArray(schema.type) ? schema.type : [schema.type];
const isInteger = types.includes("integer");
if (!isInteger && !types.includes("number")) {
return undefined;
}
const numericOptions =
typeof options === "object" && options !== null
? (options as NumericInputOptions)
: undefined;
const minimum = schema.minimum ?? schema.exclusiveMinimum;
if (numericOptions?.signed || (minimum ?? 0) < 0) {
return undefined;
}
return isInteger ? "numeric" : "decimal";
}
@@ -120,6 +120,12 @@ export function FfmpegArgsWidget(props: WidgetProps) {
id,
} = props;
const presetField = options?.ffmpegPresetField as PresetField | undefined;
// Path to this field within its config section. This is usually the same as
// the preset field, but the two diverge when the field sits below the
// section root: record.export.hwaccel_args uses the hwaccel_args preset list
// while living at export.hwaccel_args inside the record section.
const globalFieldPath =
(options?.ffmpegGlobalFieldPath as string | undefined) ?? presetField;
const allowInherit = options?.allowInherit === true;
const hideDescription = options?.hideDescription === true;
const useSplitLayout = options?.splitLayout !== false;
@@ -131,11 +137,18 @@ export function FfmpegArgsWidget(props: WidgetProps) {
// Extract the global value for this specific field to detect inheritance
const globalFieldValue = useMemo(() => {
if (!showUseGlobalSetting || !formContext?.globalValue || !presetField) {
if (
!showUseGlobalSetting ||
!formContext?.globalValue ||
!globalFieldPath
) {
return undefined;
}
return get(formContext.globalValue as Record<string, unknown>, presetField);
}, [showUseGlobalSetting, formContext?.globalValue, presetField]);
return get(
formContext.globalValue as Record<string, unknown>,
globalFieldPath,
);
}, [showUseGlobalSetting, formContext?.globalValue, globalFieldPath]);
const { data } = useSWR<FfmpegPresetResponse>("ffmpeg/presets");
@@ -1,8 +1,10 @@
import type { WidgetProps } from "@rjsf/utils";
import { useMemo } from "react";
import { useEffect, useMemo } from "react";
import { useTranslation } from "react-i18next";
import useSWR from "swr";
import { Switch } from "@/components/ui/switch";
import type { ConfigFormContext } from "@/types/configForm";
import type { GenAIModelsResponse } from "@/types/chat";
const GENAI_ROLES = ["embeddings", "descriptions", "chat"] as const;
@@ -37,10 +39,24 @@ export function GenAIRolesWidget(props: WidgetProps) {
const selectedRoles = useMemo(() => normalizeValue(value), [value]);
const providerKey = useMemo(() => getProviderKey(id), [id]);
// Compute occupied roles directly from formData. The computation is
// trivially cheap (iterate providers × 3 roles max) so we skip an
// intermediate memoization layer whose formData dependency would
// never produce a cache hit (new object reference on every change).
const { data: genaiInfo } = useSWR<GenAIModelsResponse>("genai/models", {
revalidateOnFocus: false,
});
const embeddingsSupported = useMemo(() => {
if (!providerKey) return true;
const info = genaiInfo?.[providerKey];
return info ? info.supports_embeddings : true;
}, [genaiInfo, providerKey]);
const availableRoles = useMemo(
() =>
embeddingsSupported
? GENAI_ROLES
: GENAI_ROLES.filter((role) => role !== "embeddings"),
[embeddingsSupported],
);
const occupiedRoles = useMemo(() => {
const occupied = new Set<string>();
const fd = formContext?.formData;
@@ -64,6 +80,12 @@ export function GenAIRolesWidget(props: WidgetProps) {
return occupied;
}, [formContext?.formData, providerKey]);
useEffect(() => {
if (!embeddingsSupported && selectedRoles.includes("embeddings")) {
onChange(selectedRoles.filter((role) => role !== "embeddings"));
}
}, [embeddingsSupported, selectedRoles, onChange]);
const toggleRole = (role: string, enabled: boolean) => {
if (enabled) {
if (!selectedRoles.includes(role)) {
@@ -78,7 +100,7 @@ export function GenAIRolesWidget(props: WidgetProps) {
return (
<div className="rounded-lg border border-secondary-highlight bg-background_alt p-2 pr-0 md:max-w-md">
<div className="grid gap-2">
{GENAI_ROLES.map((role) => {
{availableRoles.map((role) => {
const checked = selectedRoles.includes(role);
const roleDisabled = !checked && occupiedRoles.has(role);
const label = t(`configForm.genaiRoles.options.${role}`, {
@@ -2,7 +2,7 @@
import type { WidgetProps } from "@rjsf/utils";
import { Input } from "@/components/ui/input";
import { cn } from "@/lib/utils";
import { getSizedFieldClassName } from "../utils";
import { getNumericInputMode, getSizedFieldClassName } from "../utils";
export function TextWidget(props: WidgetProps) {
const {
@@ -28,6 +28,7 @@ export function TextWidget(props: WidgetProps) {
id={id}
className={cn(fieldClassName)}
type="text"
inputMode={getNumericInputMode(schema, options)}
value={value ?? ""}
disabled={disabled || readonly}
placeholder={placeholder || (options.placeholder as string) || ""}
@@ -127,8 +127,12 @@ export default function ObjectTrackOverlay({
},
);
const getZonesFriendlyNames = (zones: string[], config: FrigateConfig) => {
return zones?.map((zone) => resolveZoneName(config, zone)) ?? [];
const getZonesFriendlyNames = (
zones: string[],
config: FrigateConfig,
cameraId?: string,
) => {
return zones?.map((zone) => resolveZoneName(config, zone, cameraId)) ?? [];
};
const timelineResults = useMemo(() => {
@@ -151,7 +155,7 @@ export default function ObjectTrackOverlay({
data: {
...event.data,
zones_friendly_names: config
? getZonesFriendlyNames(event.data?.zones, config)
? getZonesFriendlyNames(event.data?.zones, config, event.camera)
: [],
},
}));
@@ -61,7 +61,11 @@ export function ObjectPath({
...pos.lifecycle_item?.data,
zones_friendly_names: pos.lifecycle_item?.data.zones.map(
(zone) => {
return resolveZoneName(config, zone);
return resolveZoneName(
config,
zone,
pos.lifecycle_item?.camera,
);
},
),
},
@@ -301,11 +301,19 @@ export function TrackingDetails({
[recordings, actualVideoStart],
);
eventSequence?.map((event) => {
event.data.zones_friendly_names = event.data?.zones?.map((zone) => {
return resolveZoneName(config, zone);
});
});
const sequence = useMemo(
() =>
eventSequence?.map((item) => ({
...item,
data: {
...item.data,
zones_friendly_names: item.data?.zones?.map((zone) =>
resolveZoneName(config, zone, item.camera),
),
},
})),
[eventSequence, config],
);
// Use manualOverride (set when seeking in image mode) if present so
// lifecycle rows and overlays follow image-mode seeks. Otherwise fall
@@ -849,9 +857,9 @@ export function TrackingDetails({
</div>
<div className="mt-2">
{!eventSequence ? (
{!sequence ? (
<ActivityIndicator className="size-2" size={2} />
) : eventSequence.length === 0 ? (
) : sequence.length === 0 ? (
<div className="py-2 text-muted-foreground">
{t("detail.noObjectDetailData", { ns: "views/events" })}
</div>
@@ -871,7 +879,7 @@ export function TrackingDetails({
/>
)}
<div className="space-y-2">
{eventSequence.map((item, idx) => {
{sequence.map((item, idx) => {
return (
<div
key={`${item.timestamp}-${item.source_id ?? ""}-${idx}`}
+26 -2
View File
@@ -27,7 +27,7 @@ import axios from "axios";
import { toast } from "sonner";
import useSWR from "swr";
import { FrigateConfig } from "@/types/frigateConfig";
import { reviewQueries } from "@/utils/zoneEdutUtil";
import { removeRequiredZoneQuery, reviewQueries } from "@/utils/zoneEdutUtil";
import IconWrapper from "../ui/icon-wrapper";
import { buttonVariants } from "@/components/ui/button";
import { Trans, useTranslation } from "react-i18next";
@@ -153,6 +153,30 @@ export default function PolygonItem({
cameraConfig?.review.alerts.required_zones || [],
cameraConfig?.review.detections.required_zones || [],
);
const genaiQueries = removeRequiredZoneQuery(
polygon.name,
polygon.camera,
"objects.genai",
cameraConfig?.objects.genai.required_zones || [],
);
const snapshotQueries = removeRequiredZoneQuery(
polygon.name,
polygon.camera,
"snapshots",
cameraConfig?.snapshots.required_zones || [],
);
const mqttQueries = removeRequiredZoneQuery(
polygon.name,
polygon.camera,
"mqtt",
cameraConfig?.mqtt.required_zones || [],
);
const autotrackQueries = removeRequiredZoneQuery(
polygon.name,
polygon.camera,
"onvif.autotracking",
cameraConfig?.onvif.autotracking.required_zones || [],
);
// Also delete from profiles that have overrides for this zone
let profileQueries = "";
if (allProfileNames && cameraConfig) {
@@ -165,7 +189,7 @@ export default function PolygonItem({
}
}
}
url = `cameras.${polygon.camera}.zones.${polygon.name}${alertQueries}${detectionQueries}${profileQueries}`;
url = `cameras.${polygon.camera}.zones.${polygon.name}${alertQueries}${detectionQueries}${genaiQueries}${snapshotQueries}${mqttQueries}${autotrackQueries}${profileQueries}`;
}
await axios
+5 -1
View File
@@ -108,7 +108,11 @@ export default function ZoneEditPane({
}
const inRequiredZones =
cam.review.alerts.required_zones.includes(polygon.name) ||
cam.review.detections.required_zones.includes(polygon.name);
cam.review.detections.required_zones.includes(polygon.name) ||
cam.objects.genai.required_zones.includes(polygon.name) ||
cam.snapshots.required_zones.includes(polygon.name) ||
cam.mqtt.required_zones.includes(polygon.name) ||
cam.onvif.autotracking.required_zones.includes(polygon.name);
const hasProfileOverride = Object.values(cam.profiles ?? {}).some(
(profile) => profile?.zones && polygon.name in profile.zones,
);
+1 -1
View File
@@ -1032,7 +1032,7 @@ function ObjectTimeline({
data: {
...event.data,
zones_friendly_names: event.data?.zones?.map((zone) =>
resolveZoneName(config, zone),
resolveZoneName(config, zone, event.camera),
),
},
}));
+117 -8
View File
@@ -102,6 +102,91 @@ function stripAutoDerivedMissingFromGlobal(
return cloned;
}
/**
* Sections carrying a `filters` map that the backend materializes per-camera:
* every label in the camera's label list (`objects.track`, `audio.listen`)
* without an explicit filter gets a default entry. The global map gets no
* equivalent treatment for those labels, so a label the camera picks up exists
* on the camera side alone.
*/
const AUTO_POPULATED_FILTER_SECTIONS = ["objects", "audio"];
/**
* Resolve the schema defaults for a single `<section>.filters` entry, in the
* normalized/collapsed shape config values are compared in. Returns undefined
* when the schema hasn't loaded or doesn't describe the filters map.
*/
function getDefaultFilter(
sectionPath: string,
schema?: RJSFSchema,
): JsonValue | undefined {
if (!schema) return undefined;
const sectionSchema = extractSectionSchema(schema, sectionPath, "camera");
const filtersSchema = sectionSchema?.properties?.filters as
| RJSFSchema
| undefined;
if (!filtersSchema) return undefined;
// An optional map (`dict[str, X] | None`, as `audio.filters` is declared)
// becomes an anyOf, so the entry schema can sit in a branch rather than on
// the map schema itself.
const candidates = [
filtersSchema,
...((filtersSchema.anyOf ?? filtersSchema.oneOf ?? []) as RJSFSchema[]),
];
const filterSchema = candidates.find((candidate) =>
isJsonObject(candidate?.additionalProperties as JsonValue),
)?.additionalProperties;
if (!filterSchema || typeof filterSchema !== "object") return undefined;
return collapseEmpty(
normalizeConfigValue(applySchemaDefaults(filterSchema as RJSFSchema, {})),
);
}
/**
* Complete a global baseline with the filter entries the backend only
* materializes per-camera.
*
* The effective global value for a label with no explicit global filter is the
* filter's schema defaults, not "absent": that is exactly what a camera
* inherits for it. Filling those in keeps an untouched label from reading as a
* camera override, and lets a label the camera does filter report the specific
* changed field rather than the whole filter.
*/
function withDefaultFilters(
sectionPath: string,
globalValue: JsonValue,
cameraValue: JsonValue,
schema?: RJSFSchema,
): JsonValue {
if (
!AUTO_POPULATED_FILTER_SECTIONS.includes(sectionPath) ||
!isJsonObject(globalValue) ||
!isJsonObject(cameraValue)
) {
return globalValue;
}
const cameraFilters = cameraValue.filters;
if (!isJsonObject(cameraFilters)) return globalValue;
const globalFilters = isJsonObject(globalValue.filters)
? globalValue.filters
: undefined;
const missing = Object.keys(cameraFilters).filter(
(label) => globalFilters?.[label] === undefined,
);
if (missing.length === 0) return globalValue;
const defaultFilter = getDefaultFilter(sectionPath, schema);
if (defaultFilter === undefined) return globalValue;
const filters: JsonObject = { ...globalFilters };
for (const label of missing) {
filters[label] = defaultFilter;
}
return { ...globalValue, filters };
}
/**
* Whether the given field is auto-derived for `sectionPath` and the global
* value at that path is missing in which case a per-camera value should
@@ -290,7 +375,7 @@ export function useConfigOverride({
"camera",
buildHiddenFieldContext(config, "camera", cameraName),
);
const collapsedGlobal = stripHiddenPaths(
const collapsedGlobalRaw = stripHiddenPaths(
collapseEmpty(normalizedGlobalValue),
hiddenFields,
);
@@ -300,9 +385,15 @@ export function useConfigOverride({
);
const collapsedCamera = stripAutoDerivedMissingFromGlobal(
sectionPath,
collapsedGlobal,
collapsedGlobalRaw,
collapsedCameraRaw,
);
const collapsedGlobal = withDefaultFilters(
sectionPath,
collapsedGlobalRaw,
collapsedCamera,
schema,
);
const comparisonGlobal = compareFields
? pickFields(collapsedGlobal, compareFields)
@@ -446,7 +537,7 @@ export function useAllCameraOverrides(
"camera",
buildHiddenFieldContext(config, "camera", cameraName),
);
const collapsedGlobal = stripHiddenPaths(
const collapsedGlobalRaw = stripHiddenPaths(
collapseEmpty(globalValue),
hiddenFields,
);
@@ -456,9 +547,15 @@ export function useAllCameraOverrides(
);
const collapsedCamera = stripAutoDerivedMissingFromGlobal(
key,
collapsedGlobal,
collapsedGlobalRaw,
collapsedCameraRaw,
);
const collapsedGlobal = withDefaultFilters(
key,
collapsedGlobalRaw,
collapsedCamera,
schema,
);
const comparisonGlobal = compareFields
? pickFields(collapsedGlobal, compareFields)
: collapsedGlobal;
@@ -714,9 +811,15 @@ export function useCamerasOverridingSection(
globalValue,
collapseEmpty(cameraSectionValues[idx]),
);
for (const delta of collectFieldDeltas(
const effectiveGlobalValue = withDefaultFilters(
sectionPath,
globalValue,
cameraValue,
schema,
);
for (const delta of collectFieldDeltas(
effectiveGlobalValue,
cameraValue,
compareFields,
)) {
if (isCrossCameraIgnoredPath(delta.fieldPath)) continue;
@@ -738,7 +841,7 @@ export function useCamerasOverridingSection(
if (deltasByPath.has(path)) continue;
if (isCrossCameraIgnoredPath(path)) continue;
if (!isPathAllowed(path, compareFields)) continue;
const g = get(globalValue, path);
const g = get(effectiveGlobalValue, path);
const p = get(normalizedProfile, path);
if (!isEqual(g, p)) {
deltasByPath.set(path, {
@@ -791,18 +894,24 @@ export function useCameraSectionDeltas(
const sectionMeta = OVERRIDABLE_SECTIONS.find((s) => s.key === sectionPath);
const compareFields = sectionMeta?.compareFields;
const globalValue = collapseEmpty(
const rawGlobalValue = collapseEmpty(
getEffectiveGlobalBaseline(config, sectionPath, compareFields, schema),
);
const cameraValue = stripAutoDerivedMissingFromGlobal(
sectionPath,
globalValue,
rawGlobalValue,
collapseEmpty(
normalizeConfigValue(
getBaseCameraSectionValue(config, cameraName, sectionPath),
),
),
);
const globalValue = withDefaultFilters(
sectionPath,
rawGlobalValue,
cameraValue,
schema,
);
const hiddenFields = getEffectiveHiddenFields(
sectionPath,
+1
View File
@@ -43,6 +43,7 @@ export type GenAIProviderInfo = {
models: string[];
roles: string[];
supports_toggleable_thinking: boolean;
supports_embeddings: boolean;
};
export type GenAIModelsResponse = Record<string, GenAIProviderInfo>;
+1
View File
@@ -370,6 +370,7 @@ export type CustomClassificationModelConfig = {
};
};
motion: boolean;
interval?: number;
};
};
+30
View File
@@ -1,3 +1,33 @@
// Build a config/set query fragment that removes `name` from a
// required_zones list on the given camera section (e.g. "snapshots",
// "mqtt", "objects.genai", "onvif.autotracking"), rebuilding the
// remaining entries. When removing the name empties the list, the
// required_zones key itself is deleted so the field reverts to its
// default instead of retaining the now-stale zone name. Returns an empty
// string when `name` is not present so unrelated sections are untouched.
export const removeRequiredZoneQuery = (
name: string,
camera: string,
section: string,
zones: string[],
) => {
const remaining = new Set<string>(zones || []);
if (!remaining.has(name)) {
return "";
}
remaining.delete(name);
const key = `cameras.${camera}.${section}.required_zones`;
if (remaining.size === 0) {
return `&${key}`;
}
return [...remaining].map((zone) => `&${key}=${zone}`).join("");
};
export const reviewQueries = (
name: string,
review_alerts: boolean,
@@ -3,6 +3,7 @@ import ClassificationModelWizardDialog from "@/components/classification/Classif
import ClassificationModelEditDialog from "@/components/classification/ClassificationModelEditDialog";
import ActivityIndicator from "@/components/indicators/activity-indicator";
import { ImageShadowOverlay } from "@/components/overlay/ImageShadowOverlay";
import { Badge } from "@/components/ui/badge";
import { Button, buttonVariants } from "@/components/ui/button";
import { ToggleGroup, ToggleGroupItem } from "@/components/ui/toggle-group";
import useOptimisticState from "@/hooks/use-optimistic-state";
@@ -330,7 +331,10 @@ function ModelCard({ config, onClick, onUpdate, onDelete }: ModelCardProps) {
{coverImage ? (
<>
<img
className="size-full"
className={cn(
"size-full",
!config.enabled && "opacity-50 grayscale",
)}
src={`${baseUrl}clips/${config.name}/dataset/${coverImage.name}/${coverImage.img}`}
/>
<ImageShadowOverlay lowerClassName="h-[30%] z-0" />
@@ -338,6 +342,14 @@ function ModelCard({ config, onClick, onUpdate, onDelete }: ModelCardProps) {
) : (
<Skeleton className="flex size-full items-center justify-center" />
)}
{!config.enabled && (
<Badge
variant="secondary"
className="absolute right-2 top-2 z-40 text-primary-variant"
>
{t("disabled")}
</Badge>
)}
<div className="absolute bottom-2 left-3 text-lg text-white smart-capitalize">
{config.name}
</div>
@@ -348,6 +348,19 @@ function MotionPreviewClip({
}
}, [clipStart, clipEnd, playbackRate, preview]);
useEffect(() => {
if (!videoRef.current || !preview || !videoPlaying) {
return;
}
if (isSafari || (isFirefox && isMobile)) {
// These browsers step frames manually; rebuild the interval at the new rate
resetPlayback();
} else {
videoRef.current.playbackRate = playbackRate;
}
}, [playbackRate, preview, videoPlaying, resetPlayback]);
const drawDimOverlay = useCallback(() => {
if (!dimOverlayCanvasRef.current) {
return;