mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-10-12 09:42:47 +03:00
Compare commits
17
Commits
@@ -67,6 +67,12 @@ This section can be used to set environment variables for those unable to modify
|
||||
|
||||
Variables prefixed with `FRIGATE_` can be referenced in config fields that support environment variable substitution (such as MQTT host and credentials, camera stream URLs, and ONVIF host and credentials) using the `{FRIGATE_VARIABLE_NAME}` syntax.
|
||||
|
||||
:::note
|
||||
|
||||
The `go2rtc` section is an exception. go2rtc runs as a separate process, so its stream definitions can only be substituted with variables that exist in the container's environment (set via Docker `-e`, the `environment:` section of `docker-compose.yml`, or Docker secrets). Variables defined in the `environment_vars` block above are not available to go2rtc streams. Home Assistant app users, who cannot set container environment variables, must instead put credentials directly in their go2rtc stream URLs.
|
||||
|
||||
:::
|
||||
|
||||
<ConfigTabs>
|
||||
<TabItem value="ui">
|
||||
|
||||
|
||||
+121
-65
@@ -6,6 +6,7 @@ title: Live View
|
||||
import ConfigTabs from "@site/src/components/ConfigTabs";
|
||||
import TabItem from "@theme/TabItem";
|
||||
import NavPath from "@site/src/components/NavPath";
|
||||
import FaqItem from "@site/src/components/FaqItem";
|
||||
|
||||
Frigate intelligently displays your camera streams on the Live view dashboard. By default, Frigate employs "smart streaming" where camera images update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion or active objects are detected, cameras seamlessly switch to a live stream.
|
||||
|
||||
@@ -341,100 +342,155 @@ When your browser runs into problems playing back your camera streams, it will l
|
||||
|
||||
## Live view FAQ
|
||||
|
||||
1. **Why don't I have audio in my Live view?**
|
||||
### Getting Live View Working
|
||||
|
||||
You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending PCMA/PCMU or AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
|
||||
<FaqItem id="why-dont-i-have-audio-in-my-live-view" question="Why don't I have audio in my Live view?">
|
||||
|
||||
Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio when in low bandwidth mode, even if you've set up go2rtc.
|
||||
You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending PCMA/PCMU or AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
|
||||
|
||||
2. **Frigate shows that my live stream is in "low bandwidth mode". What does this mean?**
|
||||
If the audio controls don't appear in the UI at all, verify that the Live view is actually using your go2rtc stream. If your go2rtc stream names don't match your Frigate camera name, you must map them with the `live -> streams` config (see [Setting Streams For Live UI](#setting-streams-for-live-ui) above); otherwise the UI falls back to the video-only jsmpeg player.
|
||||
|
||||
Frigate intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
|
||||
Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio when in low bandwidth mode, even if you've set up go2rtc.
|
||||
|
||||
When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI labels as "low bandwidth mode". On Live dashboards, the mode will automatically reset when smart streaming is configured and activity stops. Continuous streaming mode does not have an automatic reset mechanism, but you can use the _Reset_ option to force a reload of your stream.
|
||||
</FaqItem>
|
||||
|
||||
If you are using continuous streaming or you are loading more than a few high resolution streams at once on the dashboard, your browser may struggle to begin playback of your streams before the timeout. Frigate always prioritizes showing a live stream as quickly as possible, even if it is a lower quality jsmpeg stream. You can use the "Reset" link/button to try loading your high resolution stream again.
|
||||
<FaqItem id="i-have-unmuted-some-cameras-on-my-dashboard-but-i-do-not-hear-sound-why" question="I have unmuted some cameras on my dashboard, but I do not hear sound. Why?">
|
||||
|
||||
Errors in stream playback (e.g., connection failures, codec issues, or buffering timeouts) that cause the fallback to low bandwidth mode (jsmpeg) are logged to the browser console for easier debugging. These errors may include:
|
||||
- Network issues (e.g., MSE or WebRTC network connection problems).
|
||||
- Unsupported codecs or stream formats (e.g., H.265 in WebRTC, which is not supported in some browsers).
|
||||
- Buffering timeouts or low bandwidth conditions causing fallback to jsmpeg.
|
||||
- Browser compatibility problems (e.g., iOS Safari limitations with MSE).
|
||||
If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
|
||||
|
||||
To view browser console logs:
|
||||
1. Open the Frigate Live View in your browser.
|
||||
2. Open the browser's Developer Tools (F12 or right-click > Inspect > Console tab).
|
||||
3. Reproduce the error (e.g., load a problematic stream or simulate network issues).
|
||||
4. Look for messages prefixed with the camera name.
|
||||
</FaqItem>
|
||||
|
||||
These logs help identify if the issue is player-specific (MSE vs. WebRTC) or related to camera configuration (e.g., go2rtc streams, codecs). If you see frequent errors:
|
||||
- Verify your camera's H.264/AAC settings (see [Frigate's camera settings recommendations](#camera-settings-recommendations)).
|
||||
- Check go2rtc configuration for transcoding (e.g., audio to AAC/OPUS).
|
||||
- Test with a different stream via the UI dropdown (if `live -> streams` is configured).
|
||||
- For WebRTC-specific issues, ensure port 8555 is forwarded and candidates are set (see [WebRTC Extra Configuration](#webrtc-extra-configuration)).
|
||||
- If your cameras are streaming at a high resolution, your browser may be struggling to load all of the streams before the buffering timeout occurs. Frigate prioritizes showing a true live view as quickly as possible. If the fallback occurs often, change your live view settings to use a lower bandwidth substream.
|
||||
<FaqItem id="my-live-view-shows-a-black-screen-or-doesnt-load-but-the-debug-view-works-why" question="My live view shows a black screen or doesn't load, but the debug view works. Why?">
|
||||
|
||||
3. **It doesn't seem like my cameras are streaming on the Live dashboard. Why?**
|
||||
The debug view plays the `detect` stream processed by Frigate itself, while the Live view plays your go2rtc stream directly in the browser. If the debug view works but the Live view doesn't, your browser usually can't decode what the camera is sending, most often H.265 video or an incompatible audio track.
|
||||
|
||||
On the default Live dashboard ("All Cameras"), your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity is detected, cameras seamlessly switch to a full-resolution live stream. If you want to customize this behavior, use a camera group.
|
||||
Work through the [go2rtc troubleshooting guide](/troubleshooting/go2rtc#live-view-is-black-buffering-or-stuck-in-low-bandwidth-mode) to isolate the problem. Two fixes resolve the majority of cases:
|
||||
|
||||
4. **I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?**
|
||||
1. Restream through go2rtc's FFmpeg module by prefixing your source with `ffmpeg:`, for example `- ffmpeg:rtsp://user:password@192.168.1.5:554/stream`.
|
||||
2. If that doesn't help, transcode to compatible codecs: `- ffmpeg:rtsp://user:password@192.168.1.5:554/stream#video=h264#audio=aac#hardware`.
|
||||
|
||||
This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard (4:3, 16:9, or 32:9, etc) aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
|
||||
</FaqItem>
|
||||
|
||||
5. **How does "smart streaming" work?**
|
||||
<FaqItem id="how-do-i-get-the-best-live-view-experience-in-home-assistant" question="How do I get the best live view experience in Home Assistant?">
|
||||
|
||||
Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity (motion or object/audio detection) occurs, cameras seamlessly switch to a live stream.
|
||||
For a full-resolution, low-latency live view in Home Assistant dashboards, use the [Advanced Camera Card](https://card.camera) with the [go2rtc live provider](https://card.camera/#/configuration/cameras/live-provider?id=go2rtc), which streams directly from Frigate's bundled go2rtc. This also supports audio and [two-way talk](#two-way-talk) on capable cameras. See the [Home Assistant integration docs](/integrations/home-assistant) for setup.
|
||||
|
||||
This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second so you can see the activity until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
|
||||
</FaqItem>
|
||||
|
||||
Smart streaming depends on having your camera's motion `threshold` and `contour_area` config values dialed in. Use the Motion Tuner in Settings in the UI to tune these values in real-time.
|
||||
### Streaming Behavior
|
||||
|
||||
This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
|
||||
<FaqItem id="how-does-smart-streaming-work" question={'How does "smart streaming" work?'}>
|
||||
|
||||
6. **I have unmuted some cameras on my dashboard, but I do not hear sound. Why?**
|
||||
Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity (motion or object/audio detection) occurs, cameras seamlessly switch to a live stream.
|
||||
|
||||
If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
|
||||
This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second so you can see the activity until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
|
||||
|
||||
7. **My camera streams have lots of visual artifacts / distortion.**
|
||||
Smart streaming depends on having your camera's motion `threshold` and `contour_area` config values dialed in. Use the Motion Tuner in Settings in the UI to tune these values in real-time.
|
||||
|
||||
Some cameras don't include the hardware to support multiple connections to the high resolution stream, and this can cause unexpected behavior. In this case it is recommended to [restream](./restream.md) the high resolution stream so that it can be used for live view and recordings.
|
||||
This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
|
||||
|
||||
8. **Why does my camera stream switch aspect ratios on the Live dashboard?**
|
||||
</FaqItem>
|
||||
|
||||
Your camera may change aspect ratios on the dashboard because Frigate uses different streams for different purposes. With go2rtc and Smart Streaming, Frigate shows a static image from the `detect` stream when no activity is present, and switches to the live stream when motion is detected. The camera image will change size if your streams use different aspect ratios.
|
||||
<FaqItem id="it-doesnt-seem-like-my-cameras-are-streaming-on-the-live-dashboard-why" question="It doesn't seem like my cameras are streaming on the Live dashboard. Why?">
|
||||
|
||||
To prevent this, make the `detect` stream match the go2rtc live stream's aspect ratio (resolution does not need to match, just the aspect ratio). You can either adjust the camera's output resolution or set the `width` and `height` values in your config's `detect` section to a resolution with an aspect ratio that matches.
|
||||
On the default Live dashboard ("All Cameras"), your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity is detected, cameras seamlessly switch to a full-resolution live stream. If you want to customize this behavior, use a camera group.
|
||||
|
||||
Example: Resolutions from two streams
|
||||
- Mismatched (may cause aspect ratio switching on the dashboard):
|
||||
- Live/go2rtc stream: 1920x1080 (16:9)
|
||||
- Detect stream: 640x352 (~1.82:1, not 16:9)
|
||||
</FaqItem>
|
||||
|
||||
- Matched (prevents switching):
|
||||
- Live/go2rtc stream: 1920x1080 (16:9)
|
||||
- Detect stream: 640x360 (16:9)
|
||||
<FaqItem id="frigate-shows-that-my-live-stream-is-in-low-bandwidth-mode-what-does-this-mean" question={'Frigate shows that my live stream is in "low bandwidth mode". What does this mean?'}>
|
||||
|
||||
You can update the detect settings in your camera config to match the aspect ratio of your go2rtc live stream. For example:
|
||||
Frigate intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
|
||||
|
||||
```yaml
|
||||
cameras:
|
||||
front_door:
|
||||
detect:
|
||||
width: 640
|
||||
height: 360 # set this to 360 instead of 352
|
||||
ffmpeg:
|
||||
inputs:
|
||||
- path: rtsp://127.0.0.1:8554/front_door # main stream 1920x1080
|
||||
roles:
|
||||
- record
|
||||
- path: rtsp://127.0.0.1:8554/front_door_sub # sub stream 640x352
|
||||
roles:
|
||||
- detect
|
||||
```
|
||||
When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI labels as "low bandwidth mode". On Live dashboards, the mode will automatically reset when smart streaming is configured and activity stops. Continuous streaming mode does not have an automatic reset mechanism, but you can use the _Reset_ option to force a reload of your stream.
|
||||
|
||||
The same applies to your `record` stream: if its aspect ratio differs from your `detect` stream, your recordings will appear in a different shape than the live view. For consistent framing across live view and recordings, use the same aspect ratio for all of a camera's streams (the resolution can still differ).
|
||||
If you are using continuous streaming or you are loading more than a few high resolution streams at once on the dashboard, your browser may struggle to begin playback of your streams before the timeout. Frigate always prioritizes showing a live stream as quickly as possible, even if it is a lower quality jsmpeg stream. You can use the "Reset" link/button to try loading your high resolution stream again.
|
||||
|
||||
9. **Why does Frigate prefer MSE over WebRTC for live view?**
|
||||
Errors in stream playback (e.g., connection failures, codec issues, or buffering timeouts) that cause the fallback to low bandwidth mode (jsmpeg) are logged to the browser console for easier debugging. These errors may include:
|
||||
|
||||
Frigate prefers MSE because it delivers a better out-of-the-box experience than WebRTC on nearly every axis that matters for a security camera system. MSE is an open standard optimized and supported by all modern browsers, works without any extra configuration (WebRTC requires port forwarding and candidate setup, and lacks H.265 support in some browsers), and requires no internet access for NAT traversal. More importantly, MSE runs over TCP, so every frame arrives and is decoded in order, so nothing is ever silently skipped. WebRTC optimizes for latency over UDP by discarding late or incomplete frames, which works against you on cellular or spotty Wi-Fi: you can end up with frozen video, visual corruption, or gaps in the feed without ever knowing you missed something. Frigate's enhanced MSE player has adaptive speed playback and has been tuned for latency and connection robustness that meets or exceeds WebRTC, so you get near-real-time playback with a guarantee that when the video plays, every frame is actually there - which, for an NVR whose whole purpose is letting you see what happened, matters more than shaving fractions of a second off a latency number. That's why Frigate defaults to MSE and reserves WebRTC for cases that require it, like two-way talk.
|
||||
- Network issues (e.g., MSE or WebRTC network connection problems).
|
||||
- Unsupported codecs or stream formats (e.g., H.265 in WebRTC, which is not supported in some browsers).
|
||||
- Buffering timeouts or low bandwidth conditions causing fallback to jsmpeg.
|
||||
- Browser compatibility problems (e.g., iOS Safari limitations with MSE).
|
||||
|
||||
To view browser console logs:
|
||||
|
||||
1. Open the Frigate Live View in your browser.
|
||||
2. Open the browser's Developer Tools (F12 or right-click > Inspect > Console tab).
|
||||
3. Reproduce the error (e.g., load a problematic stream or simulate network issues).
|
||||
4. Look for messages prefixed with the camera name.
|
||||
|
||||
These logs help identify if the issue is player-specific (MSE vs. WebRTC) or related to camera configuration (e.g., go2rtc streams, codecs). If you see frequent errors:
|
||||
|
||||
- Verify your camera's H.264/AAC settings (see [Frigate's camera settings recommendations](#camera-settings-recommendations)).
|
||||
- Check go2rtc configuration for transcoding (e.g., audio to AAC/OPUS).
|
||||
- Test with a different stream via the UI dropdown (if `live -> streams` is configured).
|
||||
- For WebRTC-specific issues, ensure port 8555 is forwarded and candidates are set (see [WebRTC Extra Configuration](#webrtc-extra-configuration)).
|
||||
- If your cameras are streaming at a high resolution, your browser may be struggling to load all of the streams before the buffering timeout occurs. Frigate prioritizes showing a true live view as quickly as possible. If the fallback occurs often, change your live view settings to use a lower bandwidth substream.
|
||||
|
||||
</FaqItem>
|
||||
|
||||
<FaqItem id="why-is-my-live-view-delayed-or-lagging-behind-real-time" question="Why is my live view delayed or lagging behind real time?">
|
||||
|
||||
A delay when a stream first starts is usually caused by your camera's I-frame (keyframe) interval. Playback cannot begin until a keyframe arrives, so an interval set higher than your camera's frame rate makes the stream take longer to start. Set the I-frame interval to match the frame rate (or "1x" on Reolink) per the [camera settings recommendations](#camera-settings-recommendations).
|
||||
|
||||
A stream that starts on time but falls further behind live is buffering, which is usually the browser struggling to decode too many high-resolution streams at once. Select a lower-bandwidth substream for your dashboards (see [Setting Streams For Live UI](#setting-streams-for-live-ui)), reduce the number of streams open at once, or improve the network connection between your browser and Frigate. Frigate's player automatically speeds up playback to catch up to live after buffering, and falls back to low bandwidth mode if it stalls for too long. The _Reset_ option forces a fresh connection at the live edge.
|
||||
|
||||
</FaqItem>
|
||||
|
||||
<FaqItem id="why-does-frigate-prefer-mse-over-webrtc-for-live-view" question="Why does Frigate prefer MSE over WebRTC for live view?">
|
||||
|
||||
Frigate prefers MSE because it delivers a better out-of-the-box experience than WebRTC on nearly every axis that matters for a security camera system. MSE is an open standard optimized and supported by all modern browsers, works without any extra configuration (WebRTC requires port forwarding and candidate setup, and lacks H.265 support in some browsers), and requires no internet access for NAT traversal. More importantly, MSE runs over TCP, so every frame arrives and is decoded in order, so nothing is ever silently skipped. WebRTC optimizes for latency over UDP by discarding late or incomplete frames, which works against you on cellular or spotty Wi-Fi: you can end up with frozen video, visual corruption, or gaps in the feed without ever knowing you missed something. Frigate's enhanced MSE player has adaptive speed playback and has been tuned for latency and connection robustness that meets or exceeds WebRTC, so you get near-real-time playback with a guarantee that when the video plays, every frame is actually there - which, for an NVR whose whole purpose is letting you see what happened, matters more than shaving fractions of a second off a latency number. That's why Frigate defaults to MSE and reserves WebRTC for cases that require it, like two-way talk.
|
||||
|
||||
</FaqItem>
|
||||
|
||||
### Video Quality Issues
|
||||
|
||||
<FaqItem id="i-see-a-strange-diagonal-line-on-my-live-view-but-my-recordings-look-fine-how-can-i-fix-it" question="I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?">
|
||||
|
||||
This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard (4:3, 16:9, or 32:9, etc) aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
|
||||
|
||||
</FaqItem>
|
||||
|
||||
<FaqItem id="my-camera-streams-have-lots-of-visual-artifacts-or-distortion" question="My camera streams have lots of visual artifacts / distortion.">
|
||||
|
||||
Some cameras don't include the hardware to support multiple connections to the high resolution stream, and this can cause unexpected behavior. In this case it is recommended to [restream](./restream.md) the high resolution stream so that it can be used for live view and recordings.
|
||||
|
||||
</FaqItem>
|
||||
|
||||
<FaqItem id="why-does-my-camera-stream-switch-aspect-ratios-on-the-live-dashboard" question="Why does my camera stream switch aspect ratios on the Live dashboard?">
|
||||
|
||||
Your camera may change aspect ratios on the dashboard because Frigate uses different streams for different purposes. With go2rtc and Smart Streaming, Frigate shows a static image from the `detect` stream when no activity is present, and switches to the live stream when motion is detected. The camera image will change size if your streams use different aspect ratios.
|
||||
|
||||
To prevent this, make the `detect` stream match the go2rtc live stream's aspect ratio (resolution does not need to match, just the aspect ratio). You can either adjust the camera's output resolution or set the `width` and `height` values in your config's `detect` section to a resolution with an aspect ratio that matches.
|
||||
|
||||
Example: Resolutions from two streams
|
||||
|
||||
- Mismatched (may cause aspect ratio switching on the dashboard):
|
||||
- Live/go2rtc stream: 1920x1080 (16:9)
|
||||
- Detect stream: 640x352 (~1.82:1, not 16:9)
|
||||
|
||||
- Matched (prevents switching):
|
||||
- Live/go2rtc stream: 1920x1080 (16:9)
|
||||
- Detect stream: 640x360 (16:9)
|
||||
|
||||
You can update the detect settings in your camera config to match the aspect ratio of your go2rtc live stream. For example:
|
||||
|
||||
```yaml
|
||||
cameras:
|
||||
front_door:
|
||||
detect:
|
||||
width: 640
|
||||
height: 360 # set this to 360 instead of 352
|
||||
ffmpeg:
|
||||
inputs:
|
||||
- path: rtsp://127.0.0.1:8554/front_door # main stream 1920x1080
|
||||
roles:
|
||||
- record
|
||||
- path: rtsp://127.0.0.1:8554/front_door_sub # sub stream 640x352
|
||||
roles:
|
||||
- detect
|
||||
```
|
||||
|
||||
The same applies to your `record` stream: if its aspect ratio differs from your `detect` stream, your recordings will appear in a different shape than the live view. For consistent framing across live view and recordings, use the same aspect ratio for all of a camera's streams (the resolution can still differ).
|
||||
|
||||
</FaqItem>
|
||||
|
||||
@@ -376,20 +376,22 @@ class EmbeddingMaintainer(threading.Thread):
|
||||
logger.info(f"Disabled classification processor for model: {model_name}")
|
||||
return
|
||||
|
||||
# Check if processor already exists
|
||||
for processor in self.realtime_processors:
|
||||
if isinstance(
|
||||
processor,
|
||||
(
|
||||
CustomStateClassificationProcessor,
|
||||
CustomObjectClassificationProcessor,
|
||||
),
|
||||
if (
|
||||
isinstance(
|
||||
processor,
|
||||
(
|
||||
CustomStateClassificationProcessor,
|
||||
CustomObjectClassificationProcessor,
|
||||
),
|
||||
)
|
||||
and processor.model_config.name == model_name
|
||||
):
|
||||
if processor.model_config.name == model_name:
|
||||
logger.debug(
|
||||
f"Classification processor for model {model_name} already exists, skipping"
|
||||
)
|
||||
return
|
||||
processor.model_config = model_config
|
||||
logger.debug(
|
||||
f"Updated config for classification processor: {model_name}"
|
||||
)
|
||||
return
|
||||
|
||||
if model_config.state_config is not None:
|
||||
processor = CustomStateClassificationProcessor(
|
||||
|
||||
@@ -57,6 +57,12 @@ class BaseEmbedding(ABC):
|
||||
def _preprocess_inputs(self, raw_inputs: Any) -> Any:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _bgr_to_rgb(frame: Any) -> Any:
|
||||
if isinstance(frame, np.ndarray) and frame.ndim == 3:
|
||||
return np.ascontiguousarray(frame[:, :, ::-1])
|
||||
return frame
|
||||
|
||||
def _process_image(self, image, output: str = "RGB") -> Image.Image:
|
||||
if isinstance(image, str):
|
||||
if image.startswith("http"):
|
||||
|
||||
@@ -73,7 +73,7 @@ class FaceNetEmbedding(BaseEmbedding):
|
||||
self.tensor_output_details = self.runner.get_output_details()
|
||||
|
||||
def _preprocess_inputs(self, raw_inputs):
|
||||
pil = self._process_image(raw_inputs[0])
|
||||
pil = self._process_image(self._bgr_to_rgb(raw_inputs[0]))
|
||||
|
||||
# handle images larger than input size
|
||||
width, height = pil.size
|
||||
@@ -159,7 +159,7 @@ class ArcfaceEmbedding(BaseEmbedding):
|
||||
)
|
||||
|
||||
def _preprocess_inputs(self, raw_inputs):
|
||||
pil = self._process_image(raw_inputs[0])
|
||||
pil = self._process_image(self._bgr_to_rgb(raw_inputs[0]))
|
||||
|
||||
# handle images larger than input size
|
||||
width, height = pil.size
|
||||
|
||||
@@ -281,6 +281,11 @@ class GenAIClient:
|
||||
"""Whether the configured model exposes a per-request thinking toggle."""
|
||||
return False
|
||||
|
||||
@property
|
||||
def supports_embeddings(self) -> bool:
|
||||
"""Whether the configured model can generate embeddings via embed()."""
|
||||
return False
|
||||
|
||||
def list_models(self) -> list[str]:
|
||||
"""Return the list of model names available from this provider.
|
||||
|
||||
|
||||
@@ -121,5 +121,6 @@ class GenAIClientManager:
|
||||
"models": client.list_models(),
|
||||
"roles": [r.value for r in genai_cfg.roles],
|
||||
"supports_toggleable_thinking": client.supports_toggleable_thinking,
|
||||
"supports_embeddings": client.supports_embeddings,
|
||||
}
|
||||
return result
|
||||
|
||||
@@ -38,6 +38,37 @@ def _encode_thought_signature(signature: bytes | None) -> str | None:
|
||||
return base64.b64encode(signature).decode("ascii")
|
||||
|
||||
|
||||
def _decode_data_uri(url: str) -> tuple[str, bytes] | None:
|
||||
"""Decode a ``data:`` URI into ``(mime_type, bytes)``; None if not a data URI."""
|
||||
if not isinstance(url, str) or not url.startswith("data:"):
|
||||
return None
|
||||
try:
|
||||
header, b64 = url.split(",", 1)
|
||||
mime = header[len("data:") :].split(";")[0] or "image/jpeg"
|
||||
return mime, base64.b64decode(b64)
|
||||
except (ValueError, binascii.Error):
|
||||
return None
|
||||
|
||||
|
||||
def _parts_from_content(content: Any) -> list[types.Part]:
|
||||
"""Convert OpenAI-style message content (str or multimodal list) to Gemini parts."""
|
||||
if isinstance(content, list):
|
||||
parts: list[types.Part] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") == "text":
|
||||
parts.append(types.Part.from_text(text=item.get("text") or ""))
|
||||
elif item.get("type") == "image_url":
|
||||
decoded = _decode_data_uri((item.get("image_url") or {}).get("url", ""))
|
||||
if decoded is not None:
|
||||
mime, data = decoded
|
||||
parts.append(types.Part.from_bytes(data=data, mime_type=mime))
|
||||
# Gemini rejects empty parts; fall back to a single space.
|
||||
return parts or [types.Part.from_text(text=" ")]
|
||||
return [types.Part.from_text(text=content or "")]
|
||||
|
||||
|
||||
def _stats_from_gemini_usage(usage: Any) -> dict[str, Any] | None:
|
||||
"""Build a stats dict from a Gemini usage_metadata object."""
|
||||
prompt_tokens = getattr(usage, "prompt_token_count", None)
|
||||
@@ -227,9 +258,7 @@ class GeminiClient(GenAIClient):
|
||||
)
|
||||
else: # user
|
||||
gemini_messages.append(
|
||||
types.Content(
|
||||
role="user", parts=[types.Part.from_text(text=content)]
|
||||
)
|
||||
types.Content(role="user", parts=_parts_from_content(content))
|
||||
)
|
||||
|
||||
# Convert tools to Gemini format
|
||||
@@ -485,9 +514,7 @@ class GeminiClient(GenAIClient):
|
||||
)
|
||||
else: # user
|
||||
gemini_messages.append(
|
||||
types.Content(
|
||||
role="user", parts=[types.Part.from_text(text=content)]
|
||||
)
|
||||
types.Content(role="user", parts=_parts_from_content(content))
|
||||
)
|
||||
|
||||
# Convert tools to Gemini format
|
||||
@@ -553,7 +580,7 @@ class GeminiClient(GenAIClient):
|
||||
# Use streaming API
|
||||
content_parts: list[str] = []
|
||||
reasoning_parts: list[str] = []
|
||||
tool_calls_by_index: dict[int, dict[str, Any]] = {}
|
||||
tool_calls_accum: list[dict[str, Any]] = []
|
||||
finish_reason = "stop"
|
||||
usage_stats: dict[str, Any] | None = None
|
||||
|
||||
@@ -600,7 +627,11 @@ class GeminiClient(GenAIClient):
|
||||
content_parts.append(part.text)
|
||||
yield ("content_delta", part.text)
|
||||
elif part.function_call:
|
||||
# Handle function call
|
||||
# Gemini streams complete function calls (not partial
|
||||
# argument deltas), so each part is a distinct tool
|
||||
# call. Append rather than accumulate by name — the
|
||||
# latter concatenated parallel/repeated calls into one
|
||||
# invalid arguments string (e.g. `{...}{...}`).
|
||||
try:
|
||||
arguments = (
|
||||
dict(part.function_call.args)
|
||||
@@ -610,40 +641,16 @@ class GeminiClient(GenAIClient):
|
||||
except Exception:
|
||||
arguments = {}
|
||||
|
||||
# Store tool call
|
||||
tool_call_id = part.function_call.name or ""
|
||||
tool_call_name = part.function_call.name or ""
|
||||
|
||||
# Check if we already have this tool call
|
||||
found_index = None
|
||||
for idx, tc in tool_calls_by_index.items():
|
||||
if tc["name"] == tool_call_name:
|
||||
found_index = idx
|
||||
break
|
||||
|
||||
if found_index is None:
|
||||
found_index = len(tool_calls_by_index)
|
||||
tool_calls_by_index[found_index] = {
|
||||
"id": tool_call_id,
|
||||
"name": tool_call_name,
|
||||
"arguments": "",
|
||||
"thought_signature": None,
|
||||
tool_calls_accum.append(
|
||||
{
|
||||
"id": part.function_call.name or "",
|
||||
"name": part.function_call.name or "",
|
||||
"arguments": arguments,
|
||||
"thought_signature": getattr(
|
||||
part, "thought_signature", None
|
||||
),
|
||||
}
|
||||
|
||||
# Accumulate arguments
|
||||
if arguments:
|
||||
tool_calls_by_index[found_index]["arguments"] += (
|
||||
json.dumps(arguments)
|
||||
if isinstance(arguments, dict)
|
||||
else str(arguments)
|
||||
)
|
||||
|
||||
# Capture latest thought_signature for this call
|
||||
chunk_sig = getattr(part, "thought_signature", None)
|
||||
if chunk_sig:
|
||||
tool_calls_by_index[found_index][
|
||||
"thought_signature"
|
||||
] = chunk_sig
|
||||
)
|
||||
|
||||
# Build final message
|
||||
full_content = "".join(content_parts).strip() or None
|
||||
@@ -651,25 +658,20 @@ class GeminiClient(GenAIClient):
|
||||
|
||||
# Convert tool calls to list format
|
||||
tool_calls_list = None
|
||||
if tool_calls_by_index:
|
||||
tool_calls_list = []
|
||||
for tc in tool_calls_by_index.values():
|
||||
try:
|
||||
# Try to parse accumulated arguments as JSON
|
||||
parsed_args = json.loads(tc["arguments"])
|
||||
except (json.JSONDecodeError, Exception):
|
||||
parsed_args = tc["arguments"]
|
||||
|
||||
tool_calls_list.append(
|
||||
{
|
||||
"id": tc["id"],
|
||||
"name": tc["name"],
|
||||
"arguments": parsed_args,
|
||||
"thought_signature": _encode_thought_signature(
|
||||
tc.get("thought_signature")
|
||||
),
|
||||
}
|
||||
)
|
||||
if tool_calls_accum:
|
||||
tool_calls_list = [
|
||||
{
|
||||
"id": tc["id"],
|
||||
"name": tc["name"],
|
||||
"arguments": tc["arguments"]
|
||||
if isinstance(tc["arguments"], dict)
|
||||
else {},
|
||||
"thought_signature": _encode_thought_signature(
|
||||
tc.get("thought_signature")
|
||||
),
|
||||
}
|
||||
for tc in tool_calls_accum
|
||||
]
|
||||
finish_reason = "tool_calls"
|
||||
|
||||
if usage_stats is not None:
|
||||
|
||||
@@ -128,6 +128,11 @@ class LlamaCppClient(GenAIClient):
|
||||
_text_baseline_tokens: int | None
|
||||
_media_marker: str
|
||||
|
||||
@property
|
||||
def supports_embeddings(self) -> bool:
|
||||
"""llama.cpp exposes an /embeddings endpoint for any loaded model."""
|
||||
return True
|
||||
|
||||
def _init_provider(self) -> str | None:
|
||||
"""Initialize the client and query model metadata from the server."""
|
||||
self.provider_options = {
|
||||
|
||||
@@ -423,9 +423,18 @@ class OpenAIClient(GenAIClient):
|
||||
for tc in tool_calls_by_index.values():
|
||||
try:
|
||||
# Parse accumulated arguments as JSON
|
||||
parsed_args = json.loads(tc["arguments"])
|
||||
except (json.JSONDecodeError, Exception):
|
||||
parsed_args = tc["arguments"]
|
||||
parsed_args = json.loads(tc["arguments"] or "{}")
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
logger.warning(
|
||||
"Failed to parse streamed tool call arguments for %s",
|
||||
tc["name"],
|
||||
)
|
||||
parsed_args = {}
|
||||
|
||||
# Downstream (ToolCall model) requires a dict; never leak a
|
||||
# partial/invalid arguments string.
|
||||
if not isinstance(parsed_args, dict):
|
||||
parsed_args = {}
|
||||
|
||||
tool_calls_list.append(
|
||||
{
|
||||
|
||||
@@ -0,0 +1,496 @@
|
||||
"""Smoke tests for GenAI chat providers.
|
||||
|
||||
Each provider's ``chat_with_tools_stream`` is driven with a canned "test
|
||||
response" so the two conversion layers are exercised without any network:
|
||||
|
||||
1. Frigate (OpenAI-style) messages -> provider-native request format
|
||||
2. provider-native response -> Frigate ``("kind", value)`` stream events
|
||||
|
||||
These guard against regressions such as tool-call arguments arriving as raw
|
||||
strings instead of dicts (which crash the ``ToolCall`` model), and multimodal
|
||||
user content (a list of text/image parts, as injected by ``get_live_context``)
|
||||
crashing message conversion.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from frigate.config import GenAIConfig, GenAIProviderEnum
|
||||
from frigate.genai import PROVIDERS, load_providers
|
||||
|
||||
load_providers()
|
||||
|
||||
# A minimal but valid JPEG data URI, mirroring what get_live_context injects.
|
||||
_TINY_JPEG = base64.b64encode(b"\xff\xd8\xff\xd9").decode("ascii")
|
||||
_IMAGE_DATA_URI = f"data:image/jpeg;base64,{_TINY_JPEG}"
|
||||
|
||||
# Conversation ending in a multimodal user message (text + live image), the
|
||||
# exact shape the chat endpoint builds after a get_live_context tool result.
|
||||
MULTIMODAL_MESSAGES = [
|
||||
{"role": "system", "content": "You are a test assistant."},
|
||||
{"role": "user", "content": "what do you see on the front camera?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_live_context",
|
||||
"arguments": json.dumps({"camera": "front"}),
|
||||
},
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_1",
|
||||
"name": "get_live_context",
|
||||
"content": json.dumps({"camera": "front"}),
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Here is the current live image from camera 'front'.",
|
||||
},
|
||||
{"type": "image_url", "image_url": {"url": _IMAGE_DATA_URI}},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
SIMPLE_MESSAGES = [
|
||||
{"role": "system", "content": "You are a test assistant."},
|
||||
{"role": "user", "content": "hello"},
|
||||
]
|
||||
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search_objects",
|
||||
"description": "Search tracked objects",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"label": {"type": "string"}},
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def _make_client(provider: str, **cfg_overrides):
|
||||
"""Build a provider client offline (no model validation, no network)."""
|
||||
cfg = GenAIConfig(provider=provider, **cfg_overrides)
|
||||
cls = PROVIDERS[GenAIProviderEnum(provider)]
|
||||
return cls(cfg, timeout=5, validate_model=False)
|
||||
|
||||
|
||||
def _collect(client, messages, tools=TOOLS):
|
||||
"""Drain chat_with_tools_stream into a list of (kind, value) events."""
|
||||
|
||||
async def _run():
|
||||
events = []
|
||||
async for event in client.chat_with_tools_stream(
|
||||
messages=messages, tools=tools, tool_choice="auto"
|
||||
):
|
||||
events.append(event)
|
||||
return events
|
||||
|
||||
return asyncio.run(_run())
|
||||
|
||||
|
||||
def _final_message(events) -> dict:
|
||||
messages = [value for (kind, value) in events if kind == "message"]
|
||||
assert messages, f"stream produced no final message: {events}"
|
||||
return messages[-1]
|
||||
|
||||
|
||||
def _assert_tool_args_are_dicts(final: dict) -> None:
|
||||
"""Every returned tool call must expose arguments as a dict, never a string."""
|
||||
for tool_call in final.get("tool_calls") or []:
|
||||
assert isinstance(tool_call["arguments"], dict), (
|
||||
f"tool call arguments must be a dict, got "
|
||||
f"{type(tool_call['arguments']).__name__}: {tool_call['arguments']!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OpenAI
|
||||
# ---------------------------------------------------------------------------
|
||||
def _openai_tc(index, id=None, name=None, arguments=None):
|
||||
return SimpleNamespace(
|
||||
index=index,
|
||||
id=id,
|
||||
function=SimpleNamespace(name=name, arguments=arguments),
|
||||
)
|
||||
|
||||
|
||||
def _openai_chunk(content=None, tool_calls=None, finish_reason=None, usage=None):
|
||||
delta = SimpleNamespace(
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
reasoning_content=None,
|
||||
reasoning=None,
|
||||
)
|
||||
choice = SimpleNamespace(delta=delta, finish_reason=finish_reason)
|
||||
return SimpleNamespace(choices=[choice], usage=usage)
|
||||
|
||||
|
||||
class TestOpenAIProvider(unittest.TestCase):
|
||||
def _client(self):
|
||||
return _make_client(
|
||||
"openai", model="gpt-4o", api_key="k", base_url="http://localhost:9999/v1"
|
||||
)
|
||||
|
||||
def test_stream_tool_call_arguments_are_dict(self):
|
||||
# Arguments arrive split across chunks, as the real API streams them.
|
||||
chunks = [
|
||||
_openai_chunk(
|
||||
tool_calls=[
|
||||
_openai_tc(0, id="c1", name="search_objects", arguments='{"label":')
|
||||
]
|
||||
),
|
||||
_openai_chunk(tool_calls=[_openai_tc(0, arguments=' "person"}')]),
|
||||
_openai_chunk(finish_reason="tool_calls"),
|
||||
]
|
||||
client = self._client()
|
||||
client.provider.chat.completions.create = MagicMock(return_value=iter(chunks))
|
||||
|
||||
final = _final_message(_collect(client, SIMPLE_MESSAGES))
|
||||
self.assertEqual(final["finish_reason"], "tool_calls")
|
||||
self.assertEqual(len(final["tool_calls"]), 1)
|
||||
_assert_tool_args_are_dicts(final)
|
||||
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
|
||||
|
||||
def test_stream_content_response(self):
|
||||
chunks = [
|
||||
_openai_chunk(content="hel"),
|
||||
_openai_chunk(content="lo"),
|
||||
_openai_chunk(finish_reason="stop"),
|
||||
]
|
||||
client = self._client()
|
||||
client.provider.chat.completions.create = MagicMock(return_value=iter(chunks))
|
||||
|
||||
events = _collect(client, SIMPLE_MESSAGES)
|
||||
deltas = [v for (k, v) in events if k == "content_delta"]
|
||||
self.assertEqual("".join(deltas), "hello")
|
||||
self.assertEqual(_final_message(events)["content"], "hello")
|
||||
|
||||
def test_multimodal_message_does_not_crash(self):
|
||||
client = self._client()
|
||||
client.provider.chat.completions.create = MagicMock(
|
||||
return_value=iter([_openai_chunk(content="ok", finish_reason="stop")])
|
||||
)
|
||||
# Passing the OpenAI-native multimodal list through must not raise.
|
||||
final = _final_message(_collect(client, MULTIMODAL_MESSAGES))
|
||||
self.assertEqual(final["content"], "ok")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gemini
|
||||
# ---------------------------------------------------------------------------
|
||||
def _gemini_part(text=None, thought=False, function_call=None, thought_signature=None):
|
||||
return SimpleNamespace(
|
||||
text=text,
|
||||
thought=thought,
|
||||
function_call=function_call,
|
||||
thought_signature=thought_signature,
|
||||
)
|
||||
|
||||
|
||||
def _gemini_chunk(parts, finish_reason=None, usage_metadata=None):
|
||||
candidate = SimpleNamespace(
|
||||
content=SimpleNamespace(parts=parts), finish_reason=finish_reason
|
||||
)
|
||||
return SimpleNamespace(candidates=[candidate], usage_metadata=usage_metadata)
|
||||
|
||||
|
||||
def _gemini_stream(chunks):
|
||||
async def _agen(*args, **kwargs):
|
||||
for chunk in chunks:
|
||||
yield chunk
|
||||
|
||||
return _agen
|
||||
|
||||
|
||||
class TestGeminiProvider(unittest.TestCase):
|
||||
def _client(self):
|
||||
return _make_client("gemini", model="gemini-2.5-flash", api_key="k")
|
||||
|
||||
def _patch_stream(self, client, chunks):
|
||||
client.provider = MagicMock()
|
||||
client.provider.aio.models.generate_content_stream = AsyncMock(
|
||||
side_effect=_gemini_stream(chunks)
|
||||
)
|
||||
|
||||
def test_stream_parallel_tool_calls_stay_separate_dicts(self):
|
||||
# Regression: Gemini streams complete function calls. Two calls to the
|
||||
# same tool must NOT be merged into one concatenated arguments string.
|
||||
from google.genai.types import FinishReason
|
||||
|
||||
chunks = [
|
||||
_gemini_chunk(
|
||||
parts=[
|
||||
_gemini_part(
|
||||
function_call=SimpleNamespace(
|
||||
name="search_objects", args={"label": "person"}
|
||||
)
|
||||
),
|
||||
_gemini_part(
|
||||
function_call=SimpleNamespace(
|
||||
name="search_objects", args={"limit": 1}
|
||||
)
|
||||
),
|
||||
],
|
||||
finish_reason=FinishReason.STOP,
|
||||
),
|
||||
]
|
||||
client = self._client()
|
||||
self._patch_stream(client, chunks)
|
||||
|
||||
final = _final_message(_collect(client, SIMPLE_MESSAGES))
|
||||
self.assertEqual(final["finish_reason"], "tool_calls")
|
||||
self.assertEqual(len(final["tool_calls"]), 2)
|
||||
_assert_tool_args_are_dicts(final)
|
||||
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
|
||||
self.assertEqual(final["tool_calls"][1]["arguments"], {"limit": 1})
|
||||
|
||||
def test_stream_content_response(self):
|
||||
from google.genai.types import FinishReason
|
||||
|
||||
chunks = [
|
||||
_gemini_chunk(parts=[_gemini_part(text="hel")]),
|
||||
_gemini_chunk(
|
||||
parts=[_gemini_part(text="lo")], finish_reason=FinishReason.STOP
|
||||
),
|
||||
]
|
||||
client = self._client()
|
||||
self._patch_stream(client, chunks)
|
||||
|
||||
events = _collect(client, SIMPLE_MESSAGES)
|
||||
deltas = [v for (k, v) in events if k == "content_delta"]
|
||||
self.assertEqual("".join(deltas), "hello")
|
||||
self.assertEqual(_final_message(events)["content"], "hello")
|
||||
|
||||
def test_multimodal_message_converts_without_crash(self):
|
||||
# Regression: a user message with list content (text + image_url) used
|
||||
# to be handed to Part.from_text(text=<list>) and raise ValidationError.
|
||||
from google.genai.types import FinishReason
|
||||
|
||||
client = self._client()
|
||||
self._patch_stream(
|
||||
client,
|
||||
[
|
||||
_gemini_chunk(
|
||||
parts=[_gemini_part(text="ok")], finish_reason=FinishReason.STOP
|
||||
)
|
||||
],
|
||||
)
|
||||
final = _final_message(_collect(client, MULTIMODAL_MESSAGES))
|
||||
self.assertEqual(final["content"], "ok")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ollama
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestOllamaProvider(unittest.TestCase):
|
||||
def _client(self):
|
||||
return _make_client("ollama", model="llama3", base_url="http://localhost:9999")
|
||||
|
||||
def _run_with_response(self, client, response, messages):
|
||||
# Ollama uses a non-streaming call when tools are present, via an
|
||||
# internally-constructed async client.
|
||||
fake_async = MagicMock()
|
||||
fake_async.chat = AsyncMock(return_value=response)
|
||||
with patch(
|
||||
"frigate.genai.plugins.ollama.OllamaAsyncClient",
|
||||
return_value=fake_async,
|
||||
):
|
||||
return _collect(client, messages)
|
||||
|
||||
def test_tool_call_arguments_are_dict(self):
|
||||
response = {
|
||||
"message": {
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"function": {
|
||||
"name": "search_objects",
|
||||
"arguments": {"label": "person"},
|
||||
}
|
||||
}
|
||||
],
|
||||
},
|
||||
"done": True,
|
||||
"done_reason": "stop",
|
||||
"eval_count": 5,
|
||||
"prompt_eval_count": 3,
|
||||
"eval_duration": 1_000_000,
|
||||
}
|
||||
client = self._client()
|
||||
final = _final_message(
|
||||
self._run_with_response(client, response, SIMPLE_MESSAGES)
|
||||
)
|
||||
self.assertEqual(final["finish_reason"], "tool_calls")
|
||||
_assert_tool_args_are_dicts(final)
|
||||
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
|
||||
|
||||
def test_multimodal_message_normalizes_image(self):
|
||||
# Ollama needs content as a string with images pulled into a separate
|
||||
# field; the normalizer must extract both without crashing.
|
||||
response = {
|
||||
"message": {"content": "ok"},
|
||||
"done": True,
|
||||
"done_reason": "stop",
|
||||
}
|
||||
client = self._client()
|
||||
final = _final_message(
|
||||
self._run_with_response(client, response, MULTIMODAL_MESSAGES)
|
||||
)
|
||||
self.assertEqual(final["content"], "ok")
|
||||
|
||||
def test_normalize_multimodal_content(self):
|
||||
from frigate.genai.plugins.ollama import _normalize_multimodal_content
|
||||
|
||||
text, images = _normalize_multimodal_content(MULTIMODAL_MESSAGES[-1]["content"])
|
||||
self.assertIn("live image", text)
|
||||
self.assertEqual(len(images), 1)
|
||||
self.assertEqual(images[0], b"\xff\xd8\xff\xd9")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# llama.cpp
|
||||
# ---------------------------------------------------------------------------
|
||||
class _FakeStreamResponse:
|
||||
def __init__(self, lines):
|
||||
self._lines = lines
|
||||
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
async def aiter_lines(self):
|
||||
for line in self._lines:
|
||||
yield line
|
||||
|
||||
|
||||
class _FakeStreamCtx:
|
||||
def __init__(self, lines):
|
||||
self._resp = _FakeStreamResponse(lines)
|
||||
|
||||
async def __aenter__(self):
|
||||
return self._resp
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
class _FakeAsyncClient:
|
||||
def __init__(self, lines):
|
||||
self._lines = lines
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, json=None):
|
||||
return _FakeStreamCtx(self._lines)
|
||||
|
||||
|
||||
class TestLlamaCppProvider(unittest.TestCase):
|
||||
def _client(self):
|
||||
return _make_client("llamacpp", model="m", base_url="http://localhost:9999")
|
||||
|
||||
def _run_with_lines(self, client, lines, messages):
|
||||
with patch(
|
||||
"frigate.genai.plugins.llama_cpp.httpx.AsyncClient",
|
||||
return_value=_FakeAsyncClient(lines),
|
||||
):
|
||||
return _collect(client, messages)
|
||||
|
||||
def test_stream_tool_call_arguments_are_dict(self):
|
||||
lines = [
|
||||
"data: "
|
||||
+ json.dumps(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 0,
|
||||
"id": "c1",
|
||||
"function": {
|
||||
"name": "search_objects",
|
||||
"arguments": '{"label":',
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
"data: "
|
||||
+ json.dumps(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 0,
|
||||
"function": {"arguments": ' "person"}'},
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
"data: "
|
||||
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "tool_calls"}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
client = self._client()
|
||||
final = _final_message(self._run_with_lines(client, lines, SIMPLE_MESSAGES))
|
||||
self.assertEqual(final["finish_reason"], "tool_calls")
|
||||
_assert_tool_args_are_dicts(final)
|
||||
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
|
||||
|
||||
def test_stream_content_response(self):
|
||||
lines = [
|
||||
"data: " + json.dumps({"choices": [{"delta": {"content": "hel"}}]}),
|
||||
"data: " + json.dumps({"choices": [{"delta": {"content": "lo"}}]}),
|
||||
"data: "
|
||||
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "stop"}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
client = self._client()
|
||||
events = self._run_with_lines(client, lines, SIMPLE_MESSAGES)
|
||||
deltas = [v for (k, v) in events if k == "content_delta"]
|
||||
self.assertEqual("".join(deltas), "hello")
|
||||
self.assertEqual(_final_message(events)["content"], "hello")
|
||||
|
||||
def test_multimodal_message_does_not_crash(self):
|
||||
lines = [
|
||||
"data: " + json.dumps({"choices": [{"delta": {"content": "ok"}}]}),
|
||||
"data: "
|
||||
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "stop"}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
client = self._client()
|
||||
final = _final_message(self._run_with_lines(client, lines, MULTIMODAL_MESSAGES))
|
||||
self.assertEqual(final["content"], "ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
"documentTitle": "Classification Models - Frigate",
|
||||
"disabled": "Disabled",
|
||||
"details": {
|
||||
"scoreInfo": "Score represents the average classification confidence across all detections of this object.",
|
||||
"none": "None",
|
||||
@@ -64,7 +65,20 @@
|
||||
"title": "Edit Classification Model",
|
||||
"descriptionState": "Edit the classes for this state classification model. Changes will require retraining the model.",
|
||||
"descriptionObject": "Edit the object type and classification type for this object classification model.",
|
||||
"stateClassesInfo": "Note: Changing state classes requires retraining the model with the updated classes."
|
||||
"enabled": "Enabled",
|
||||
"enabledDesc": "Run this model. When disabled, it stops running and no longer classifies.",
|
||||
"saveAttempts": "Save Attempts",
|
||||
"saveAttemptsDesc": "Number of classification attempt images to keep for the recent classifications UI.",
|
||||
"motion": "Run on Motion",
|
||||
"motionDesc": "Run classification when motion is detected within the configured crop.",
|
||||
"interval": "Interval",
|
||||
"intervalDesc": "Seconds between periodic classification runs. Leave empty to run only on motion.",
|
||||
"intervalPlaceholder": "No interval",
|
||||
"stateClassesInfo": "Model updated. Retrain the model for the class changes to take effect.",
|
||||
"errors": {
|
||||
"saveAttemptsInvalid": "Save attempts must be a whole number of 0 or greater",
|
||||
"intervalInvalid": "Interval must be a whole number greater than 0"
|
||||
}
|
||||
},
|
||||
"deleteDatasetImages": {
|
||||
"title": "Delete Dataset Images",
|
||||
|
||||
@@ -67,7 +67,8 @@
|
||||
"error": {
|
||||
"failed": "未能加入导出队列：{{error}}",
|
||||
"endTimeMustAfterStartTime": "结束时间必须在开始时间之后",
|
||||
"noVaildTimeSelected": "未选择有效的时间范围"
|
||||
"noVaildTimeSelected": "未选择有效的时间范围",
|
||||
"noValidTimeSelected": "未选择有效的时间范围"
|
||||
},
|
||||
"view": "查看",
|
||||
"queued": "导出已加入队列。请在导出页面查看进度。",
|
||||
|
||||
@@ -1655,7 +1655,14 @@
|
||||
"keyLabel": "键",
|
||||
"valueLabel": "值",
|
||||
"keyPlaceholder": "新键名",
|
||||
"remove": "移除"
|
||||
"remove": "移除",
|
||||
"providerNameLabel": "提供商名称",
|
||||
"providerNamePlaceholder": "例如：openai、deepseek",
|
||||
"variableNameLabel": "变量名",
|
||||
"variableNamePlaceholder": "例如：MY_VARIABLE",
|
||||
"loggerNameLabel": "日志模块名称",
|
||||
"loggerNamePlaceholder": "例如：frigate.record",
|
||||
"keyPatternError": "只能使用字母、数字、连字符和下划线（不包含空格）"
|
||||
},
|
||||
"roleMap": {
|
||||
"empty": "未配置权限组映射",
|
||||
|
||||
@@ -59,7 +59,8 @@
|
||||
"error": {
|
||||
"failed": "匯出失敗：{{error}}",
|
||||
"endTimeMustAfterStartTime": "結束時間必須要在開始時間之後",
|
||||
"noVaildTimeSelected": "沒有選取有效的時間範圍"
|
||||
"noVaildTimeSelected": "沒有選取有效的時間範圍",
|
||||
"noValidTimeSelected": "未選擇有效的時間範圍"
|
||||
},
|
||||
"view": "查看",
|
||||
"queued": "匯出已加入佇列。請在匯出頁面檢視進度。",
|
||||
|
||||
@@ -497,6 +497,9 @@
|
||||
"max_concurrent": {
|
||||
"label": "最大併發匯出數",
|
||||
"description": "同時可處理的最大匯出任務數量。"
|
||||
},
|
||||
"chapters": {
|
||||
"label": "匯出錄影時要嵌入的章節資訊"
|
||||
}
|
||||
},
|
||||
"preview": {
|
||||
|
||||
@@ -1043,6 +1043,9 @@
|
||||
"max_concurrent": {
|
||||
"label": "最大併發匯出數",
|
||||
"description": "同時可處理的最大匯出任務數量。"
|
||||
},
|
||||
"chapters": {
|
||||
"label": "匯出錄影時要嵌入的章節資訊"
|
||||
}
|
||||
},
|
||||
"preview": {
|
||||
|
||||
@@ -1484,7 +1484,14 @@
|
||||
"keyLabel": "鍵",
|
||||
"valueLabel": "值",
|
||||
"keyPlaceholder": "新鍵名",
|
||||
"remove": "移除"
|
||||
"remove": "移除",
|
||||
"providerNameLabel": "服務提供者名稱",
|
||||
"providerNamePlaceholder": "例如：openai",
|
||||
"variableNameLabel": "變數名稱",
|
||||
"variableNamePlaceholder": "例如：MY_VARIABLE",
|
||||
"loggerNameLabel": "日誌名稱",
|
||||
"loggerNamePlaceholder": "例如：frigate.record",
|
||||
"keyPatternError": "僅能使用英文字母、數字、連字號（-）和底線（_），且不得包含空格"
|
||||
},
|
||||
"knownPlates": {
|
||||
"namePlaceholder": "例如：老婆的車",
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
import {
|
||||
Form,
|
||||
FormControl,
|
||||
FormDescription,
|
||||
FormField,
|
||||
FormItem,
|
||||
FormLabel,
|
||||
@@ -17,6 +18,7 @@ import {
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Label } from "@/components/ui/label";
|
||||
import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group";
|
||||
import { Switch } from "@/components/ui/switch";
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
@@ -50,14 +52,25 @@ type ClassificationModelEditDialogProps = {
|
||||
type ObjectClassificationType = "sub_label" | "attribute";
|
||||
|
||||
type ObjectFormData = {
|
||||
enabled: boolean;
|
||||
saveAttempts: number;
|
||||
objectLabel: string;
|
||||
objectType: ObjectClassificationType;
|
||||
};
|
||||
|
||||
type StateFormData = {
|
||||
enabled: boolean;
|
||||
saveAttempts: number;
|
||||
motion: boolean;
|
||||
interval?: number;
|
||||
classes: string[];
|
||||
};
|
||||
|
||||
const DEFAULT_SAVE_ATTEMPTS = {
|
||||
object: 200,
|
||||
state: 100,
|
||||
} as const;
|
||||
|
||||
export default function ClassificationModelEditDialog({
|
||||
open,
|
||||
model,
|
||||
@@ -71,6 +84,10 @@ export default function ClassificationModelEditDialog({
|
||||
const isStateModel = model.state_config !== undefined;
|
||||
const isObjectModel = model.object_config !== undefined;
|
||||
|
||||
const defaultSaveAttempts = isObjectModel
|
||||
? DEFAULT_SAVE_ATTEMPTS.object
|
||||
: DEFAULT_SAVE_ATTEMPTS.state;
|
||||
|
||||
const objectLabels = useMemo(() => {
|
||||
if (!config) return [];
|
||||
|
||||
@@ -93,8 +110,17 @@ export default function ClassificationModelEditDialog({
|
||||
|
||||
// Define form schema based on model type
|
||||
const formSchema = useMemo(() => {
|
||||
const sharedFields = {
|
||||
enabled: z.boolean(),
|
||||
saveAttempts: z.coerce
|
||||
.number({ message: t("edit.errors.saveAttemptsInvalid") })
|
||||
.int(t("edit.errors.saveAttemptsInvalid"))
|
||||
.min(0, t("edit.errors.saveAttemptsInvalid")),
|
||||
};
|
||||
|
||||
if (isObjectModel) {
|
||||
return z.object({
|
||||
...sharedFields,
|
||||
objectLabel: z
|
||||
.string()
|
||||
.min(1, t("wizard.step1.errors.objectLabelRequired")),
|
||||
@@ -103,6 +129,17 @@ export default function ClassificationModelEditDialog({
|
||||
} else {
|
||||
// State model
|
||||
return z.object({
|
||||
...sharedFields,
|
||||
motion: z.boolean(),
|
||||
interval: z.preprocess(
|
||||
(val) =>
|
||||
val === "" || val === null || val === undefined ? undefined : val,
|
||||
z.coerce
|
||||
.number({ message: t("edit.errors.intervalInvalid") })
|
||||
.int(t("edit.errors.intervalInvalid"))
|
||||
.positive(t("edit.errors.intervalInvalid"))
|
||||
.optional(),
|
||||
),
|
||||
classes: z
|
||||
.array(z.string())
|
||||
.min(1, t("wizard.step1.errors.classRequired"))
|
||||
@@ -129,12 +166,18 @@ export default function ClassificationModelEditDialog({
|
||||
resolver: zodResolver(formSchema),
|
||||
defaultValues: isObjectModel
|
||||
? ({
|
||||
enabled: model.enabled,
|
||||
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
|
||||
objectLabel: model.object_config?.objects?.[0] || "",
|
||||
objectType:
|
||||
(model.object_config
|
||||
?.classification_type as ObjectClassificationType) || "sub_label",
|
||||
} as ObjectFormData)
|
||||
: ({
|
||||
enabled: model.enabled,
|
||||
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
|
||||
motion: model.state_config?.motion ?? false,
|
||||
interval: model.state_config?.interval,
|
||||
classes: [""], // Will be populated from dataset
|
||||
} as StateFormData),
|
||||
mode: "onChange",
|
||||
@@ -151,6 +194,8 @@ export default function ClassificationModelEditDialog({
|
||||
if (open) {
|
||||
if (isObjectModel) {
|
||||
form.reset({
|
||||
enabled: model.enabled,
|
||||
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
|
||||
objectLabel: model.object_config?.objects?.[0] || "",
|
||||
objectType:
|
||||
(model.object_config
|
||||
@@ -158,6 +203,10 @@ export default function ClassificationModelEditDialog({
|
||||
} as ObjectFormData);
|
||||
} else {
|
||||
form.reset({
|
||||
enabled: model.enabled,
|
||||
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
|
||||
motion: model.state_config?.motion ?? false,
|
||||
interval: model.state_config?.interval,
|
||||
classes: [""],
|
||||
} as StateFormData);
|
||||
}
|
||||
@@ -166,7 +215,15 @@ export default function ClassificationModelEditDialog({
|
||||
mutateDataset();
|
||||
}
|
||||
}
|
||||
}, [open, isObjectModel, isStateModel, model, form, mutateDataset]);
|
||||
}, [
|
||||
open,
|
||||
isObjectModel,
|
||||
isStateModel,
|
||||
model,
|
||||
form,
|
||||
mutateDataset,
|
||||
defaultSaveAttempts,
|
||||
]);
|
||||
|
||||
// Update form with classes from dataset when loaded
|
||||
useEffect(() => {
|
||||
@@ -233,6 +290,7 @@ export default function ClassificationModelEditDialog({
|
||||
setIsSaving(true);
|
||||
try {
|
||||
if (isObjectModel) {
|
||||
// object model save
|
||||
const objectData = data as ObjectFormData;
|
||||
|
||||
// Update the config
|
||||
@@ -243,9 +301,10 @@ export default function ClassificationModelEditDialog({
|
||||
classification: {
|
||||
custom: {
|
||||
[model.name]: {
|
||||
enabled: model.enabled,
|
||||
enabled: objectData.enabled,
|
||||
name: model.name,
|
||||
threshold: model.threshold,
|
||||
save_attempts: objectData.saveAttempts,
|
||||
object_config: {
|
||||
objects: [objectData.objectLabel],
|
||||
classification_type: objectData.objectType,
|
||||
@@ -260,7 +319,34 @@ export default function ClassificationModelEditDialog({
|
||||
position: "top-center",
|
||||
});
|
||||
} else {
|
||||
// state model save
|
||||
const stateData = data as StateFormData;
|
||||
|
||||
const stateConfig: { motion: boolean; interval?: number | null } = {
|
||||
motion: stateData.motion,
|
||||
};
|
||||
if (stateData.interval != null) {
|
||||
stateConfig.interval = stateData.interval;
|
||||
} else if (model.state_config?.interval != null) {
|
||||
stateConfig.interval = null;
|
||||
}
|
||||
|
||||
await axios.put("/config/set", {
|
||||
requires_restart: 0,
|
||||
update_topic: `config/classification/custom/${model.name}`,
|
||||
config_data: {
|
||||
classification: {
|
||||
custom: {
|
||||
[model.name]: {
|
||||
enabled: stateData.enabled,
|
||||
save_attempts: stateData.saveAttempts,
|
||||
state_config: stateConfig,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const newClasses = stateData.classes.filter(
|
||||
(c) => c.trim().length > 0,
|
||||
);
|
||||
@@ -307,11 +393,11 @@ export default function ClassificationModelEditDialog({
|
||||
if (renamePromises.length > 0) {
|
||||
await Promise.all(renamePromises);
|
||||
await mutate(`classification/${model.name}/dataset`);
|
||||
toast.success(t("toast.success.updatedModel"), {
|
||||
toast.success(t("edit.stateClassesInfo"), {
|
||||
position: "top-center",
|
||||
});
|
||||
} else {
|
||||
toast.info(t("edit.stateClassesInfo"), {
|
||||
toast.success(t("toast.success.updatedModel"), {
|
||||
position: "top-center",
|
||||
});
|
||||
}
|
||||
@@ -359,6 +445,29 @@ export default function ClassificationModelEditDialog({
|
||||
<div className="space-y-6">
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="enabled"
|
||||
render={({ field }) => (
|
||||
<FormItem className="flex flex-row items-center justify-between gap-4">
|
||||
<div className="space-y-0.5">
|
||||
<FormLabel className="text-primary-variant">
|
||||
{t("edit.enabled")}
|
||||
</FormLabel>
|
||||
<FormDescription className="text-xs">
|
||||
{t("edit.enabledDesc")}
|
||||
</FormDescription>
|
||||
</div>
|
||||
<FormControl>
|
||||
<Switch
|
||||
checked={field.value}
|
||||
onCheckedChange={field.onChange}
|
||||
/>
|
||||
</FormControl>
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
{isObjectModel && (
|
||||
<>
|
||||
<FormField
|
||||
@@ -520,6 +629,77 @@ export default function ClassificationModelEditDialog({
|
||||
</div>
|
||||
)}
|
||||
|
||||
{isStateModel && (
|
||||
<>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="motion"
|
||||
render={({ field }) => (
|
||||
<FormItem className="flex flex-row items-center justify-between gap-4">
|
||||
<div className="space-y-0.5">
|
||||
<FormLabel className="text-primary-variant">
|
||||
{t("edit.motion")}
|
||||
</FormLabel>
|
||||
<FormDescription className="text-xs">
|
||||
{t("edit.motionDesc")}
|
||||
</FormDescription>
|
||||
</div>
|
||||
<FormControl>
|
||||
<Switch
|
||||
checked={field.value}
|
||||
onCheckedChange={field.onChange}
|
||||
/>
|
||||
</FormControl>
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="interval"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel className="text-primary-variant">
|
||||
{t("edit.interval")}
|
||||
</FormLabel>
|
||||
<FormControl>
|
||||
<Input
|
||||
className="h-8"
|
||||
inputMode="numeric"
|
||||
placeholder={t("edit.intervalPlaceholder")}
|
||||
{...field}
|
||||
value={field.value ?? ""}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormDescription className="text-xs">
|
||||
{t("edit.intervalDesc")}
|
||||
</FormDescription>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="saveAttempts"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel className="text-primary-variant">
|
||||
{t("edit.saveAttempts")}
|
||||
</FormLabel>
|
||||
<FormControl>
|
||||
<Input className="h-8" inputMode="numeric" {...field} />
|
||||
</FormControl>
|
||||
<FormDescription className="text-xs">
|
||||
{t("edit.saveAttemptsDesc")}
|
||||
</FormDescription>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<div className="flex flex-col gap-3 pt-3 sm:flex-row sm:justify-end sm:gap-4">
|
||||
<Button
|
||||
type="button"
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
import type { WidgetProps } from "@rjsf/utils";
|
||||
import { useMemo } from "react";
|
||||
import { useEffect, useMemo } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import useSWR from "swr";
|
||||
import { Switch } from "@/components/ui/switch";
|
||||
import type { ConfigFormContext } from "@/types/configForm";
|
||||
import type { GenAIModelsResponse } from "@/types/chat";
|
||||
|
||||
const GENAI_ROLES = ["embeddings", "descriptions", "chat"] as const;
|
||||
|
||||
@@ -37,10 +39,24 @@ export function GenAIRolesWidget(props: WidgetProps) {
|
||||
const selectedRoles = useMemo(() => normalizeValue(value), [value]);
|
||||
const providerKey = useMemo(() => getProviderKey(id), [id]);
|
||||
|
||||
// Compute occupied roles directly from formData. The computation is
|
||||
// trivially cheap (iterate providers × 3 roles max) so we skip an
|
||||
// intermediate memoization layer whose formData dependency would
|
||||
// never produce a cache hit (new object reference on every change).
|
||||
const { data: genaiInfo } = useSWR<GenAIModelsResponse>("genai/models", {
|
||||
revalidateOnFocus: false,
|
||||
});
|
||||
|
||||
const embeddingsSupported = useMemo(() => {
|
||||
if (!providerKey) return true;
|
||||
const info = genaiInfo?.[providerKey];
|
||||
return info ? info.supports_embeddings : true;
|
||||
}, [genaiInfo, providerKey]);
|
||||
|
||||
const availableRoles = useMemo(
|
||||
() =>
|
||||
embeddingsSupported
|
||||
? GENAI_ROLES
|
||||
: GENAI_ROLES.filter((role) => role !== "embeddings"),
|
||||
[embeddingsSupported],
|
||||
);
|
||||
|
||||
const occupiedRoles = useMemo(() => {
|
||||
const occupied = new Set<string>();
|
||||
const fd = formContext?.formData;
|
||||
@@ -64,6 +80,12 @@ export function GenAIRolesWidget(props: WidgetProps) {
|
||||
return occupied;
|
||||
}, [formContext?.formData, providerKey]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!embeddingsSupported && selectedRoles.includes("embeddings")) {
|
||||
onChange(selectedRoles.filter((role) => role !== "embeddings"));
|
||||
}
|
||||
}, [embeddingsSupported, selectedRoles, onChange]);
|
||||
|
||||
const toggleRole = (role: string, enabled: boolean) => {
|
||||
if (enabled) {
|
||||
if (!selectedRoles.includes(role)) {
|
||||
@@ -78,7 +100,7 @@ export function GenAIRolesWidget(props: WidgetProps) {
|
||||
return (
|
||||
<div className="rounded-lg border border-secondary-highlight bg-background_alt p-2 pr-0 md:max-w-md">
|
||||
<div className="grid gap-2">
|
||||
{GENAI_ROLES.map((role) => {
|
||||
{availableRoles.map((role) => {
|
||||
const checked = selectedRoles.includes(role);
|
||||
const roleDisabled = !checked && occupiedRoles.has(role);
|
||||
const label = t(`configForm.genaiRoles.options.${role}`, {
|
||||
|
||||
@@ -43,6 +43,7 @@ export type GenAIProviderInfo = {
|
||||
models: string[];
|
||||
roles: string[];
|
||||
supports_toggleable_thinking: boolean;
|
||||
supports_embeddings: boolean;
|
||||
};
|
||||
|
||||
export type GenAIModelsResponse = Record<string, GenAIProviderInfo>;
|
||||
|
||||
@@ -370,6 +370,7 @@ export type CustomClassificationModelConfig = {
|
||||
};
|
||||
};
|
||||
motion: boolean;
|
||||
interval?: number;
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ import ClassificationModelWizardDialog from "@/components/classification/Classif
|
||||
import ClassificationModelEditDialog from "@/components/classification/ClassificationModelEditDialog";
|
||||
import ActivityIndicator from "@/components/indicators/activity-indicator";
|
||||
import { ImageShadowOverlay } from "@/components/overlay/ImageShadowOverlay";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button, buttonVariants } from "@/components/ui/button";
|
||||
import { ToggleGroup, ToggleGroupItem } from "@/components/ui/toggle-group";
|
||||
import useOptimisticState from "@/hooks/use-optimistic-state";
|
||||
@@ -330,7 +331,10 @@ function ModelCard({ config, onClick, onUpdate, onDelete }: ModelCardProps) {
|
||||
{coverImage ? (
|
||||
<>
|
||||
<img
|
||||
className="size-full"
|
||||
className={cn(
|
||||
"size-full",
|
||||
!config.enabled && "opacity-50 grayscale",
|
||||
)}
|
||||
src={`${baseUrl}clips/${config.name}/dataset/${coverImage.name}/${coverImage.img}`}
|
||||
/>
|
||||
<ImageShadowOverlay lowerClassName="h-[30%] z-0" />
|
||||
@@ -338,6 +342,14 @@ function ModelCard({ config, onClick, onUpdate, onDelete }: ModelCardProps) {
|
||||
) : (
|
||||
<Skeleton className="flex size-full items-center justify-center" />
|
||||
)}
|
||||
{!config.enabled && (
|
||||
<Badge
|
||||
variant="secondary"
|
||||
className="absolute right-2 top-2 z-40 text-primary-variant"
|
||||
>
|
||||
{t("disabled")}
|
||||
</Badge>
|
||||
)}
|
||||
<div className="absolute bottom-2 left-3 text-lg text-white smart-capitalize">
|
||||
{config.name}
|
||||
</div>
|
||||
|
||||
@@ -348,6 +348,19 @@ function MotionPreviewClip({
|
||||
}
|
||||
}, [clipStart, clipEnd, playbackRate, preview]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!videoRef.current || !preview || !videoPlaying) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (isSafari || (isFirefox && isMobile)) {
|
||||
// These browsers step frames manually; rebuild the interval at the new rate
|
||||
resetPlayback();
|
||||
} else {
|
||||
videoRef.current.playbackRate = playbackRate;
|
||||
}
|
||||
}, [playbackRate, preview, videoPlaying, resetPlayback]);
|
||||
|
||||
const drawDimOverlay = useCallback(() => {
|
||||
if (!dimOverlayCanvasRef.current) {
|
||||
return;
|
||||
|
||||
Reference in New Issue
Block a user