Compare commits

...
8 Commits
Author SHA1 Message Date
dependabot[bot]andGitHub ea76cadc2d Merge 7962733c82 into 65af0b1351 2026-07-13 19:35:00 +00:00
Nicolas MowenandGitHub 65af0b1351 GenAI Fixes (#23708)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
* Fix Gemini tool calling

* Catch openai bug

* Implement tool calling tests for GenAI

* Expose if embeddings are supported for a given provider
2026-07-13 07:33:15 -06:00
Josh HawkinsandGitHub fcd05ec7bc UI improvements and fixes (#23690)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
* add ability to edit enabled and save_attempts for classification models in the UI

* add state motion and interval configs to edit dialog

* fix preview playback rate for motion previews

* add docs note about environment vars and go2rtc

* update live view faq
2026-07-13 06:30:15 -06:00
Josh HawkinsandGitHub 5f6043aa92 Fix enabled flag for custom classification models (#23681)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
* honor enabled flag for custom classification models

for both startup and dynamically, even though the UI doesn't currently have a way to toggle dynamically

* add test
2026-07-12 03:48:14 -08:00
Josh HawkinsandGitHub da4037eb52 UI tweaks (#23679)
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / AMD64 Build (push) Waiting to run
* lock saved GenAI provider keys and add labels/validation to config map-key fields

* fix docs
2026-07-11 16:30:40 -06:00
Martin WeineltandGitHub f3c09ae169 Typo fixes (#23669)
CI / AMD64 Build (push) Has been cancelled
CI / ARM Build (push) Has been cancelled
CI / Jetson Jetpack 6 (push) Has been cancelled
CI / AMD64 Extra Build (push) Has been cancelled
CI / ARM Extra Build (push) Has been cancelled
CI / Synaptics Build (push) Has been cancelled
CI / Assemble and push default build (push) Has been cancelled
* Fix typos

With help from https://github.com/crate-ci/typos

* Fix repeated article "the"

* Fix repeated "changed"
2026-07-10 08:10:13 -05:00
Josh HawkinsandGitHub f6596ac7b0 Miscellaneous fixes (#23661)
* update face recognition docs

* clarify

* improve faq grouping

* add faqitem component

* add enable http link for reolinks

* update plus docs

* update autotracking faq

* fix typos
2026-07-10 07:09:15 -06:00
dependabot[bot]andGitHub 7962733c82 Bump axios from 1.13.6 to 1.15.2 in /web
Bumps [axios](https://github.com/axios/axios) from 1.13.6 to 1.15.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.13.6...v1.15.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.15.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-01 13:18:00 +00:00
54 changed files with 1704 additions and 298 deletions
@@ -339,7 +339,7 @@ detect:
# especially when using separate streams for detect and record.
# Use this setting to make the timeline bounding boxes more closely align
# with the recording. The value can be positive or negative.
# TIP: Imagine there is an tracked object clip with a person walking from left to right.
# TIP: Imagine there is a tracked object clip with a person walking from left to right.
# If the tracked object lifecycle bounding box is consistently to the left of the person
# then the value should be decreased. Similarly, if a person is walking from
# left to right and the bounding box is consistently ahead of the person
+10 -4
View File
@@ -67,15 +67,21 @@ This section can be used to set environment variables for those unable to modify
Variables prefixed with `FRIGATE_` can be referenced in config fields that support environment variable substitution (such as MQTT host and credentials, camera stream URLs, and ONVIF host and credentials) using the `{FRIGATE_VARIABLE_NAME}` syntax.
:::note
The `go2rtc` section is an exception. go2rtc runs as a separate process, so its stream definitions can only be substituted with variables that exist in the container's environment (set via Docker `-e`, the `environment:` section of `docker-compose.yml`, or Docker secrets). Variables defined in the `environment_vars` block above are not available to go2rtc streams. Home Assistant app users, who cannot set container environment variables, must instead put credentials directly in their go2rtc stream URLs.
:::
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Environment variables" /> to add or edit environment variables.
| Field | Description |
| --------- | --------------------------------------------------------- |
| **Key** | The environment variable name (e.g., `FRIGATE_MQTT_USER`) |
| **Value** | The value for the variable |
| Field | Description |
| ----------------- | --------------------------------------------------------- |
| **Variable name** | The environment variable name (e.g., `FRIGATE_MQTT_USER`) |
| **Value** | The value for the variable |
Variables defined here can be referenced elsewhere in your configuration using the `{FRIGATE_VARIABLE_NAME}` syntax.
+1 -1
View File
@@ -78,7 +78,7 @@ cameras:
### Configuring Minimum Volume
The audio detector uses volume levels in the same way that motion in a camera feed is used for object detection. This means that Frigate will not run audio detection unless the audio volume is above the configured level in order to reduce resource usage. Audio levels can vary widely between camera models so it is important to run tests to see what volume levels are. The [Debug view](/usage/live#the-single-camera-view) in the Frigate UI has an Audio tab for cameras that have the `audio` role assigned where a graph and the current levels are is displayed. The `min_volume` parameter should be set to the minimum the `RMS` level required to run audio detection.
The audio detector uses volume levels in the same way that motion in a camera feed is used for object detection. This means that Frigate will not run audio detection unless the audio volume is above the configured level in order to reduce resource usage. Audio levels can vary widely between camera models so it is important to run tests to see what volume levels are. The [Debug view](/usage/live#the-single-camera-view) in the Frigate UI has an Audio tab for cameras that have the `audio` role assigned where a graph and the current levels are displayed. The `min_volume` parameter should be set to the minimum the `RMS` level required to run audio detection.
:::tip
+81 -14
View File
@@ -6,6 +6,7 @@ title: Camera Autotracking
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
import FaqItem from "@site/src/components/FaqItem";
An ONVIF-capable, PTZ (pan-tilt-zoom) camera that supports relative movement within the field of view (FOV) can be configured to automatically track moving objects and keep them in the center of the frame.
@@ -187,30 +188,96 @@ In security and surveillance, it's common to use "spotter" cameras in combinatio
## Troubleshooting and FAQ
### The autotracker loses track of my object. Why?
### Camera Compatibility
<FaqItem id="which-ptz-camera-should-i-use-for-autotracking" question="Which PTZ camera should I use for autotracking?">
See the community-maintained list of [ONVIF PTZ camera recommendations](cameras.md#onvif-ptz-camera-recommendations) for cameras and brands reported to work (and not work) with autotracking. This is not an exhaustive list that is frequently updated, so other cameras not listed may also work well. Frigate's autotracking was developed with a Dahua SD1A404XB-GNR (now sold as the EmpireTech PTZ1A4M-4X-S2), and Dahua / EmpireTech PTZs are the most consistently reported as working well.
When comparing models:
- Verify ONVIF support first. See [Checking ONVIF camera support](#checking-onvif-camera-support) above.
- Favor a camera with a fast PTZ motor. Cameras with slow motors may fail [calibration](#calibration) and will struggle to keep up with objects that move across the field of view quickly.
</FaqItem>
<FaqItem id="does-autotracking-work-with-reolink-ptz-cameras" question="Does autotracking work with Reolink PTZ cameras?">
No. Reolink cameras (including the TrackMix series) lack the ONVIF FOV RelativeMove firmware support that Frigate's autotracker requires, so autotracking will not work with any current Reolink PTZ. Their video streams and basic PTZ controls still work in Frigate. If you want object tracking on a Reolink PTZ, you will need to use the tracking feature built into the camera's firmware, which is proprietary and operates independently of Frigate.
</FaqItem>
<FaqItem id="im-seeing-an-error-in-the-logs-that-my-camera-is-still-in-onvif-moving-status-what-does-this-mean" question={"I'm seeing an error in the logs that my camera \"is still in ONVIF 'MOVING' status.\" What does this mean?"}>
There are two possible known reasons for this (and perhaps others yet unknown): a slow PTZ motor or buggy camera firmware. Frigate uses an ONVIF parameter provided by the camera, `MoveStatus`, to determine when the PTZ's motor is moving or idle. According to some users, Hikvision PTZs (even with the latest firmware), are not updating this value after PTZ movement. Unfortunately there is no workaround to this bug in Hikvision firmware, so autotracking will not function correctly and should be disabled in your config. This may also be the case with other non-Hikvision cameras utilizing Hikvision firmware, such as some Annke models. In rare cases the vendor may provide fixed firmware on request; for example, Annke has supplied firmware that resolves this for the CZ504 (see the [camera recommendations list](cameras.md#onvif-ptz-camera-recommendations)).
</FaqItem>
<FaqItem id="calibration-seems-to-have-completed-but-the-camera-is-not-actually-moving-to-track-my-object-why" question="Calibration seems to have completed, but the camera is not actually moving to track my object. Why?">
Some cameras have firmware that reports that FOV RelativeMove, the ONVIF command that Frigate uses for autotracking, is supported. However, if the camera does not pan or tilt when an object comes into the required zone, your camera's firmware does not actually support FOV RelativeMove. One such camera is the Uniview IPC672LR-AX4DUPK. It actually moves its zoom motor instead of panning and tilting and does not follow the ONVIF standard whatsoever.
</FaqItem>
### Calibration Issues
<FaqItem id="i-tried-calibrating-my-camera-but-the-logs-show-that-it-is-stuck-at-0-and-frigate-is-not-starting-up" question="I tried calibrating my camera, but the logs show that it is stuck at 0% and Frigate is not starting up.">
This is often caused by the same reason as the "MOVING" status error above - the `MoveStatus` ONVIF parameter is not changing due to a bug in your camera's firmware. Also, see the note above: Frigate's web UI and all other cameras will be unresponsive while calibration is in progress. This is expected and normal. But if you don't see log entries every few seconds for calibration progress, your camera is not compatible with autotracking.
</FaqItem>
<FaqItem id="frigate-reports-an-error-saying-that-calibration-has-failed-why" question="Frigate reports an error saying that calibration has failed. Why?">
Calibration measures the amount of time it takes for Frigate to make a series of movements with your PTZ. This error message is recorded in the log if these values are too high for Frigate to support calibrated autotracking. This is often the case when your camera's motor or network connection is too slow or your camera's firmware doesn't report the motor status in a timely manner.
Some things to try:
- If your camera's firmware has a PTZ or motor speed setting, set it to the fastest available speed and calibrate again.
- Run without calibration: remove the `movement_weights` line from your config, set `calibrate_on_startup` to `False`, and restart.
If calibration consistently fails, this often means your camera's motor is too slow and autotracking will behave unpredictably or won't be able to keep up with moving objects.
</FaqItem>
<FaqItem id="autotracking-is-erratic-or-moves-the-camera-in-the-wrong-direction" question="Autotracking is erratic, moves the camera in the wrong direction, or zooms past my object. Why?">
Frigate uses the `movement_weights` measured during calibration to predict how far the camera needs to move to keep an object centered, so inaccurate values produce movements that don't seem to make sense: overshooting, moving the opposite direction, or zooming in on an object's last known position and losing it entirely. This is almost always a calibration issue.
- Remove the `movement_weights` entry from your config and restart Frigate to run without calibration. If tracking improves, try recalibrating.
- Recalibrate several times. The `movement_weights` values should be close to each other after each run. If they vary significantly between runs, your camera may not be reporting its motor status reliably, and you may get better results without calibration.
- If you are using zooming, a high `zoom_factor` can cause the camera to zoom in too far and lose the object. Try a lower value.
Remember to recalibrate whenever you change your `return_preset`, change your camera's detect `fps`, or enable zooming after calibrating with it disabled.
</FaqItem>
### Tracking Behavior
<FaqItem id="the-autotracker-loses-track-of-my-object-why" question="The autotracker loses track of my object. Why?">
There are many reasons this could be the case. If you are using experimental zooming, your `zoom_factor` value might be too high, the object might be traveling too quickly, the scene might be too dark, there are not enough details in the scene (for example, a PTZ looking down on a driveway or other monotone background without a sufficient number of hard edges or corners), or the scene is otherwise less than optimal for Frigate to maintain tracking.
Your camera's shutter speed may also be set too low so that blurring occurs with motion. Check your camera's firmware to see if you can increase the shutter speed.
Watching Frigate's debug view can help to determine a possible cause. The autotracked object will have a thicker colored box around it.
Watching Frigate's debug view can help to determine a possible cause. The autotracked object will have a thicker colored box around it. If the camera consistently zooms in on the object and then loses it, see [Autotracking is erratic, moves the camera in the wrong direction, or zooms past my object. Why?](#autotracking-is-erratic-or-moves-the-camera-in-the-wrong-direction) above.
### I'm seeing an error in the logs that my camera "is still in ONVIF 'MOVING' status." What does this mean?
</FaqItem>
There are two possible known reasons for this (and perhaps others yet unknown): a slow PTZ motor or buggy camera firmware. Frigate uses an ONVIF parameter provided by the camera, `MoveStatus`, to determine when the PTZ's motor is moving or idle. According to some users, Hikvision PTZs (even with the latest firmware), are not updating this value after PTZ movement. Unfortunately there is no workaround to this bug in Hikvision firmware, so autotracking will not function correctly and should be disabled in your config. This may also be the case with other non-Hikvision cameras utilizing Hikvision firmware.
### I tried calibrating my camera, but the logs show that it is stuck at 0% and Frigate is not starting up.
This is often caused by the same reason as above - the `MoveStatus` ONVIF parameter is not changing due to a bug in your camera's firmware. Also, see the note above: Frigate's web UI and all other cameras will be unresponsive while calibration is in progress. This is expected and normal. But if you don't see log entries every few seconds for calibration progress, your camera is not compatible with autotracking.
### I'm seeing this error in the logs: "Autotracker: motion estimator couldn't get transformations". What does this mean?
<FaqItem id="im-seeing-this-error-in-the-logs-autotracker-motion-estimator-couldnt-get-transformations-what-does-this-mean" question={"I'm seeing this error in the logs: \"Autotracker: motion estimator couldn't get transformations\". What does this mean?"}>
To maintain object tracking during PTZ moves, Frigate tracks the motion of your camera based on the details of the frame. If you are seeing this message, it could mean that your `zoom_factor` may be set too high, the scene around your detected object does not have enough details (like hard edges or color variations), or your camera's shutter speed is too slow and motion blur is occurring. Try reducing `zoom_factor`, finding a way to alter the scene around your object, or changing your camera's shutter speed.
### Calibration seems to have completed, but the camera is not actually moving to track my object. Why?
</FaqItem>
Some cameras have firmware that reports that FOV RelativeMove, the ONVIF command that Frigate uses for autotracking, is supported. However, if the camera does not pan or tilt when an object comes into the required zone, your camera's firmware does not actually support FOV RelativeMove. One such camera is the Uniview IPC672LR-AX4DUPK. It actually moves its zoom motor instead of panning and tilting and does not follow the ONVIF standard whatsoever.
<FaqItem id="why-does-object-detection-pause-briefly-when-the-camera-moves" question="Why does object detection pause briefly when the camera moves?">
### Frigate reports an error saying that calibration has failed. Why?
When the PTZ moves, the entire frame changes at once. Frigate's motion detection treats sudden scene-wide changes (like a lightning flash, an infrared mode switch, or a camera move) specially and pauses detection momentarily until the scene stabilizes. This is expected and normal, and detection resumes shortly after the camera stops moving. If detection does not resume once the camera is stationary, use the [debug view](/usage/live#the-single-camera-view) to see what is happening.
Calibration measures the amount of time it takes for Frigate to make a series of movements with your PTZ. This error message is recorded in the log if these values are too high for Frigate to support calibrated autotracking. This is often the case when your camera's motor or network connection is too slow or your camera's firmware doesn't report the motor status in a timely manner. You can try running without calibration (just remove the `movement_weights` line from your config and restart), but if calibration fails, this often means that autotracking will behave unpredictably.
</FaqItem>
<FaqItem id="can-i-turn-autotracking-on-and-off-automatically" question="Can I turn autotracking on and off automatically?">
Yes. Autotracking can be toggled per camera at runtime over MQTT with the [`frigate/<camera_name>/ptz_autotracker/set`](../integrations/mqtt.md#frigatecamera_nameptz_autotrackerset) topic, and the [Home Assistant integration](../integrations/home-assistant.md) exposes a switch for it. This pairs well with the "spotter" camera automations described in [Usage applications](#usage-applications) above, for example only enabling autotracking at night or when nobody is home.
</FaqItem>
+5 -5
View File
@@ -156,7 +156,7 @@ Reolink has many different camera models with inconsistently supported features
| 6MP or higher | Latest (ex: Duo3, CX-8##) | http-flv with ffmpeg 8.0, or rtsp | This uses the new http-flv-enhanced over H265 which requires ffmpeg 8.0 (Frigate's default) |
| 6MP or higher | Older (ex: RLC-8##) | rtsp | |
Frigate works much better with newer reolink cameras that are setup with the below options:
Frigate works much better with newer Reolink cameras that are setup with the below options:
If available, recommended settings are:
@@ -165,7 +165,7 @@ If available, recommended settings are:
#### Setup via the Add Camera Wizard
The Add Camera Wizard is the recommended way to add a standard Reolink camera. Before starting, make sure HTTP is enabled in the camera's advanced network settings. The wizard uses the camera's HTTP API to determine its resolution and choose the recommended stream type from the table above.
The Add Camera Wizard is the recommended way to add a standard Reolink camera. Before starting, make sure [HTTP is enabled](https://support.reolink.com/articles/360003452893-How-to-Access-Reolink-Cameras-NVRs-Home-Hub-Locally-via-Web-Browsers/) in the camera's advanced network settings. The wizard uses the camera's HTTP API to determine its resolution and choose the recommended stream type from the table above.
1. Click **Add Camera** in <NavPath path="Settings > Global configuration > Camera management" />.
2. Choose **Manual selection** as the stream detection method and select **Reolink** as the camera brand.
@@ -192,7 +192,7 @@ Reolink's latest cameras support two way audio via go2rtc and other applications
NOTE: The RTSP stream can not be prefixed with `ffmpeg:`, as go2rtc needs to handle the stream to support two way audio.
Ensure HTTP is enabled in the camera's advanced network settings. To use two way talk with Frigate, see the [Live view documentation](/configuration/live#two-way-talk).
Ensure [HTTP is enabled](https://support.reolink.com/articles/360003452893-How-to-Access-Reolink-Cameras-NVRs-Home-Hub-Locally-via-Web-Browsers/) in the camera's advanced network settings. To use two way talk with Frigate, see the [Live view documentation](/configuration/live#two-way-talk).
:::
@@ -204,7 +204,7 @@ go2rtc:
- "ffmpeg:http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password#video=copy#audio=copy#audio=opus"
your_reolink_camera_sub:
- "ffmpeg:http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=username&password=password"
# example for connectin to a Reolink camera that supports two way talk
# example for connecting to a Reolink camera that supports two way talk
your_reolink_camera_twt:
- "ffmpeg:http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password#video=copy#audio=copy#audio=opus"
- "rtsp://username:password@reolink_ip/Preview_01_sub"
@@ -249,7 +249,7 @@ cameras:
:::note
Unifi G5s cameras and newer need a Unifi Protect server to enable rtsps stream, it's not posible to enable it in standalone mode.
Unifi G5s cameras and newer need a Unifi Protect server to enable rtsps stream, it's not possible to enable it in standalone mode.
:::
+97 -21
View File
@@ -6,6 +6,7 @@ title: Face Recognition
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
import FaqItem from "@site/src/components/FaqItem";
Face recognition identifies known individuals by matching detected faces with previously learned facial data. When a known `person` is recognized, their name will be added as a `sub_label`. This information is included in the UI, filters, as well as in notifications.
@@ -151,6 +152,14 @@ Follow these steps to begin:
## Creating a Robust Training Set
:::tip
**The short version:** Start with a few clear, front-facing photos of each person. As faces are detected in the Recent Recognitions tab, train clear images that scored lower, adding variety (different angles, lighting, and expressions) slowly. Diversity matters far more than volume, and low-quality images hurt recognition more than they help.
For a step-by-step narrative of these best practices (and the same principles applied to state and object classification), see the [Frigate Tips: Best Practices for Training](https://github.com/blakeblackshear/frigate/discussions/21374) discussion.
:::
The number of images needed for a sufficient training set for face recognition varies depending on several factors:
- Diversity of the dataset: A dataset with diverse images, including variations in lighting, pose, and facial expressions, will require fewer images per person than a less diverse dataset.
@@ -181,9 +190,27 @@ When choosing images to include in the face training set it is recommended to al
The Recent Recognitions tab in the face library displays recent face recognition attempts. Detected face images are grouped according to the person they were identified as potentially matching.
Each face image is labeled with a name (or `Unknown`) along with the confidence score of the recognition attempt. While each image can be used to train the system for a specific person, not all images are suitable for training.
Each face image is labeled with a name (or `Unknown`) along with the confidence score of that recognition attempt. Images are grouped by the person they were matched against, not by who they actually are, so a group labeled with a person's name can contain a crop that is really someone else but happened to score as a partial match. The name and score shown on each individual crop describe that single attempt.
Refer to the guidelines below for best practices on selecting images for training.
While each image can be used to train the system for a specific person, not all images are suitable for training. Refer to the guidelines below for best practices on selecting images for training.
### How Frigate Decides Who a Person Is
Recognition does not happen one frame at a time. While a `person` is in view, Frigate runs face recognition on many frames, not just a single frame. The final `sub_label` is decided from all of those attempts together, weighted by the area of each face (larger, closer faces count more), not from any single frame.
This has a few practical consequences:
- A handful of wrong guesses on blurry or distant frames usually do not change the result. If Frigate sees a person as "Tom, Tom, Sam, Tom, Tom," it will still conclude the person was Tom.
- The goal is not for every individual face crop to be correct. The goal is for each person to be recognized correctly overall, across all the faces captured while they were present.
- A single very high confidence match will not by itself assign a sub label. Recognition must be consistent. See [I see scores above the threshold in the Recent Recognitions tab, but a sub label wasn't assigned?](#i-see-scores-above-the-threshold-in-the-recent-recognitions-tab-but-a-sub-label-wasnt-assigned) below.
### Which Faces Are Worth Training?
Whether a face is worth training has little to do with what it was recognized as. A crop is a good training candidate when all of these are true:
- It did not already score high and correctly. Faces that are already recognized confidently add little and increase the risk of over-fitting.
- It is clear enough to be useful: not blurry, not heavily off-axis, not infrared (gray-scale). If it is hard for you to make out the face, it will not help the model.
- It adds something new: a different angle, lighting, expression, or distance than what you already have.
### Step 1 - Building a Strong Foundation
@@ -199,7 +226,9 @@ Once front-facing images are performing well, start choosing slightly off-angle
## FAQ
### How do I debug Face Recognition issues?
### Getting Recognition Working
<FaqItem id="how-do-i-debug-face-recognition-issues" question="How do I debug Face Recognition issues?">
Start with the [Usage](#usage) section and re-read the [Model Requirements](#model-requirements) above.
@@ -217,21 +246,47 @@ Start with the [Usage](#usage) section and re-read the [Model Requirements](#mod
- Make sure you have trained at least one face per the recommendations above.
- Adjust `recognition_threshold` settings per the suggestions [above](#advanced-configuration).
### Detection does not work well with blurry images?
</FaqItem>
Accuracy is definitely a going to be improved with higher quality cameras / streams. It is important to look at the DORI (Detection Observation Recognition Identification) range of your camera, if that specification is posted. This specification explains the distance from the camera that a person can be detected, observed, recognized, and identified. The identification range is the most relevant here, and the distance listed by the camera is the furthest that face recognition will realistically work.
<FaqItem id="does-face-recognition-run-on-the-recording-stream" question="Does face recognition run on the recording stream?">
Face recognition does not run on the recording stream, this would be suboptimal for many reasons:
1. The latency of accessing the recordings means the notifications would not include the names of recognized people because recognition would not complete until after.
2. The embedding models used run on a set image size, so larger images will be scaled down to match this anyway.
3. Motion clarity is much more important than extra pixels, over-compression and motion blur are much more detrimental to results than resolution.
</FaqItem>
### Improving Accuracy and Training
<FaqItem id="detection-does-not-work-well-with-blurry-images" question="Detection does not work well with blurry images?">
Accuracy is definitely going to be improved with higher quality cameras / streams. It is important to look at the DORI (Detection Observation Recognition Identification) range of your camera, if that specification is posted. This specification explains the distance from the camera that a person can be detected, observed, recognized, and identified. The identification range is the most relevant here, and the distance listed by the camera is the furthest that face recognition will realistically work.
Some users have also noted that setting the stream in camera firmware to a constant bit rate (CBR) leads to better image clarity than with a variable bit rate (VBR).
### Why can't I bulk upload photos?
</FaqItem>
<FaqItem id="can-i-train-faces-for-people-who-only-appear-at-night" question="Can I train faces for people who only appear at night?">
The embedding models are trained on color images, so gray-scale and infrared (IR) faces sit in a different feature distribution and are more easily confused with other people. Prefer color images, and avoid mixing gray-scale samples in early while you are building a foundation. If someone only ever appears at night, gray-scale training is acceptable, but keep those samples limited and as clear as possible, and add them only once color recognition is stable for your other people.
</FaqItem>
<FaqItem id="why-cant-i-bulk-upload-photos" question="Why can't I bulk upload photos?">
It is important to methodically add photos to the library, bulk importing photos (especially from a general photo library) will lead to over-fitting in that particular scenario and hurt recognition performance.
### Why can't I bulk reprocess faces?
</FaqItem>
<FaqItem id="why-cant-i-bulk-reprocess-faces" question="Why can't I bulk reprocess faces?">
Face embedding models work by breaking apart faces into different features. This means that when reprocessing an image, only images from a similar angle will have its score affected.
### Why do unknown people score similarly to known people?
</FaqItem>
<FaqItem id="why-do-unknown-people-score-similarly-to-known-people" question="Why do unknown people score similarly to known people?">
This can happen for a few different reasons, but this is usually an indicator that the training set needs to be improved. This is often related to over-fitting:
@@ -243,31 +298,52 @@ Review your face collections and remove most of the unclear or low-quality image
Avoid training on images that already score highly, as this can lead to over-fitting. Instead, focus on relatively clear images that score lower (ideally with different lighting, angles, and conditions) to help the model generalize more effectively.
### Frigate misidentified a face. Can I tell it that a face is "not" a specific person?
</FaqItem>
<FaqItem id="should-i-correct-a-face-that-was-recognized-as-the-wrong-person" question="Should I correct a face that was recognized as the wrong person?">
Only if it is a good image. Reassigning a face does add it to that person's training set, but two things are true at once:
- Reassigning a single misclassified frame has a small effect. The image is weighted against every other sample for that person, so correcting 1 frame out of 20 will not move recognition much. Occasional wrong guesses on poor frames are normal and do not need to be fixed.
- Reassigning a poor image (blurry, off-angle, low-resolution, gray-scale) can hurt more than the misidentification did, because low-quality samples degrade recognition for that whole person.
So the decision is about image quality, not about the wrong label. If the crop is clear, well-lit, and reasonably front-facing, and it scored low or was wrong, assigning it to the correct person is useful. If you can barely make out the face yourself, ignore it; do not train it just to correct the label.
If a person is repeatedly misidentified, do not keep reassigning the same frame. Instead, remove low-quality or misleading images and add a few high-quality samples to the correct person. See [Why do unknown people score similarly to known people?](#why-do-unknown-people-score-similarly-to-known-people) above.
</FaqItem>
<FaqItem id="frigate-misidentified-a-face-can-i-tell-it-that-a-face-is-not-a-specific-person" question={'Frigate misidentified a face. Can I tell it that a face is "not" a specific person?'}>
No, face recognition does not support negative training (i.e., explicitly telling it who someone is _not_). Instead, the best approach is to improve the training data by using a more diverse and representative set of images for each person.
For more guidance, refer to the section above on improving recognition accuracy.
### I see scores above the threshold in the Recent Recognitions tab, but a sub label wasn't assigned?
This also applies to a stranger who is repeatedly matched to a known person (for example, a delivery driver recognized as you). Do not create a profile for them and do not reassign their faces to yourself, as this pollutes your training set and makes recognition worse. Leave the detection as unknown and improve the known person's training set instead. Face recognition learns who someone is, not who they are not.
The Frigate considers the recognition scores across all recognition attempts for each person object. The scores are continually weighted based on the area of the face, and a sub label will only be assigned to person if a person is confidently recognized consistently. This avoids cases where a single high confidence recognition would throw off the results.
</FaqItem>
### Can I use other face recognition software like DoubleTake at the same time as the built in face recognition?
<FaqItem id="i-see-scores-above-the-threshold-in-the-recent-recognitions-tab-but-a-sub-label-wasnt-assigned" question="I see scores above the threshold in the Recent Recognitions tab, but a sub label wasn't assigned?">
Frigate considers the recognition scores across all recognition attempts for each person object. The scores are continually weighted based on the area of the face, and a sub label will only be assigned to person if a person is confidently recognized consistently. This avoids cases where a single high confidence recognition would throw off the results.
</FaqItem>
### Compatibility and Maintenance
<FaqItem id="can-i-use-other-face-recognition-software-like-doubletake-at-the-same-time-as-the-built-in-face-recognition" question="Can I use other face recognition software like DoubleTake at the same time as the built in face recognition?">
No, using another face recognition service will interfere with Frigate's built in face recognition. When using double-take the sub_label feature must be disabled if the built in face recognition is also desired.
### Does face recognition run on the recording stream?
</FaqItem>
Face recognition does not run on the recording stream, this would be suboptimal for many reasons:
1. The latency of accessing the recordings means the notifications would not include the names of recognized people because recognition would not complete until after.
2. The embedding models used run on a set image size, so larger images will be scaled down to match this anyway.
3. Motion clarity is much more important than extra pixels, over-compression and motion blur are much more detrimental to results than resolution.
### I get an unknown error when taking a photo directly with my iPhone
<FaqItem id="i-get-an-unknown-error-when-taking-a-photo-directly-with-my-iphone" question="I get an unknown error when taking a photo directly with my iPhone">
By default iOS devices will use HEIC (High Efficiency Image Container) for images, but this format is not supported for uploads. Choosing `large` as the format instead of `original` will use JPG which will work correctly.
### How can I delete the face database and start over?
</FaqItem>
<FaqItem id="how-can-i-delete-the-face-database-and-start-over" question="How can I delete the face database and start over?">
Frigate does not store anything in its database related to face recognition. You can simply delete all of your faces through the Frigate UI or remove the contents of the `/media/frigate/clips/faces` directory.
</FaqItem>
+1 -1
View File
@@ -67,4 +67,4 @@ If your stream won't play, has no audio, uses excessive CPU, or otherwise misbeh
## Homekit Configuration
To add camera streams to Homekit Frigate must be configured in docker to use `host` networking mode. Once that is done, you can use the go2rtc WebUI (accessed via port 1984, which is disabled by default) to share export a camera to Homekit. Any changes made will automatically be saved to `/config/go2rtc_homekit.yml`.
To add camera streams to Homekit Frigate must be configured in docker to use `host` networking mode. Once that is done, you can use the go2rtc WebUI (accessed via port 1984, which is disabled by default) to export a camera to Homekit. Any changes made will automatically be saved to `/config/go2rtc_homekit.yml`.
@@ -477,7 +477,7 @@ Error marking filters as finished
Restarting ffmpeg...
```
you should try to uprade to FFmpeg 7. This can be done using this config option:
you should try to upgrade to FFmpeg 7. This can be done using this config option:
```yaml
ffmpeg:
@@ -6,6 +6,7 @@ title: License Plate Recognition (LPR)
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
import FaqItem from "@site/src/components/FaqItem";
Frigate can recognize license plates on vehicles and automatically add the detected characters to the `recognized_license_plate` field or a [known](#matching) name as a `sub_label` to tracked objects of type `car` or `motorcycle`. A common use case may be to read the license plates of cars pulling into a driveway or cars passing by on a street.
@@ -591,7 +592,9 @@ By selecting the appropriate configuration, users can optimize their dedicated L
## FAQ
### Why isn't my license plate being detected and recognized?
### Detection and Recognition
<FaqItem id="why-isnt-my-license-plate-being-detected-and-recognized" question="Why isn't my license plate being detected and recognized?">
Ensure that:
@@ -606,29 +609,43 @@ Recognized plates will show as object labels in the debug view and will appear i
If you are still having issues detecting plates, start with a basic configuration and see the debugging tips below.
### Can I run LPR without detecting `car` or `motorcycle` objects?
</FaqItem>
<FaqItem id="can-i-run-lpr-without-detecting-car-or-motorcycle-objects" question={<>Can I run LPR without detecting <code>car</code> or <code>motorcycle</code> objects?</>}>
In normal LPR mode, Frigate requires a `car` or `motorcycle` to be detected first before recognizing a license plate. If you have a dedicated LPR camera, you can change the camera `type` to `"lpr"` to use the Dedicated LPR Camera algorithm. This comes with important caveats, though. See the [Dedicated LPR Cameras](#dedicated-lpr-cameras) section above.
### How can I improve detection accuracy?
</FaqItem>
<FaqItem id="how-can-i-improve-detection-accuracy" question="How can I improve detection accuracy?">
- Use high-quality cameras with good resolution.
- Adjust `detection_threshold` and `recognition_threshold` values.
- Define a `format` regex to filter out invalid detections.
### Does LPR work at night?
</FaqItem>
<FaqItem id="does-lpr-work-at-night" question="Does LPR work at night?">
Yes, but performance depends on camera quality, lighting, and infrared capabilities. Make sure your camera can capture clear images of plates at night.
### Can I limit LPR to specific zones?
</FaqItem>
<FaqItem id="can-i-limit-lpr-to-specific-zones" question="Can I limit LPR to specific zones?">
LPR, like other Frigate enrichments, runs at the camera level rather than the zone level. While you can't restrict LPR to specific zones directly, you can control when recognition runs by setting a `min_area` value to filter out smaller detections.
### How can I match known plates with minor variations?
</FaqItem>
<FaqItem id="how-can-i-match-known-plates-with-minor-variations" question="How can I match known plates with minor variations?">
Use `match_distance` to allow small character mismatches. Alternatively, define multiple variations in `known_plates`.
### How do I debug LPR issues?
</FaqItem>
### Performance and Troubleshooting
<FaqItem id="how-do-i-debug-lpr-issues" question="How do I debug LPR issues?">
Start with ["Why isn't my license plate being detected and recognized?"](#why-isnt-my-license-plate-being-detected-and-recognized). If you are still having issues, work through these steps.
@@ -685,17 +702,23 @@ lpr:
- Watch the debug view to see plates recognized in real-time. For non-dedicated LPR cameras, the `car` or `motorcycle` label will change to the recognized plate when LPR is enabled and working.
- Adjust `recognition_threshold` settings per the suggestions [above](#advanced-configuration).
### Will LPR slow down my system?
</FaqItem>
<FaqItem id="will-lpr-slow-down-my-system" question="Will LPR slow down my system?">
LPR's performance impact depends on your hardware. Ensure you have at least 4GB RAM and a capable CPU or GPU for optimal results. If you are running the Dedicated LPR Camera mode, resource usage will be higher compared to users who run a model that natively detects license plates. Tune your motion detection settings for your dedicated LPR camera so that the license plate detection model runs only when necessary.
### I am seeing a YOLOv9 plate detection metric in Enrichment Metrics, but I have a Frigate+ or custom model that detects `license_plate`. Why is the YOLOv9 model running?
</FaqItem>
<FaqItem id="i-am-seeing-a-yolov9-plate-detection-metric-in-enrichment-metrics-but-i-have-a-frigate-or-custom-model-that-detects-license_plate-why-is-the-yolov9-model-running" question={<>I am seeing a YOLOv9 plate detection metric in Enrichment Metrics, but I have a Frigate+ or custom model that detects <code>license_plate</code>. Why is the YOLOv9 model running?</>}>
The YOLOv9 license plate detector model will run (and the metric will appear) if you've enabled LPR but haven't defined `license_plate` as an object to track, either at the global or camera level.
If you are detecting `car` or `motorcycle` on cameras where you don't want to run LPR, make sure you disable LPR it at the camera level. And if you do want to run LPR on those cameras, make sure you define `license_plate` as an object to track.
### It looks like Frigate picked up my camera's timestamp or overlay text as the license plate. How can I prevent this?
</FaqItem>
<FaqItem id="it-looks-like-frigate-picked-up-my-cameras-timestamp-or-overlay-text-as-the-license-plate-how-can-i-prevent-this" question="It looks like Frigate picked up my camera's timestamp or overlay text as the license plate. How can I prevent this?">
This could happen if cars or motorcycles travel close to your camera's timestamp or overlay text. You could either move the text through your camera's firmware, or apply a mask to it in Frigate.
@@ -703,6 +726,10 @@ If you are using a model that natively detects `license_plate`, add an _object m
If you are not using a model that natively detects `license_plate` or you are using dedicated LPR camera mode, only a _motion mask_ over your text is required.
### I see "Error running ... model" in my logs, or my inference time is very high. How can I fix this?
</FaqItem>
<FaqItem id="i-see-error-running--model-in-my-logs-or-my-inference-time-is-very-high-how-can-i-fix-this" question={'I see "Error running ... model" in my logs, or my inference time is very high. How can I fix this?'}>
This usually happens when your GPU is unable to compile or use one of the LPR models. Set your `device` to `CPU` and try again. GPU acceleration only provides a slight performance increase, and the models are lightweight enough to run without issue on most CPUs.
</FaqItem>
+121 -65
View File
@@ -6,6 +6,7 @@ title: Live View
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
import FaqItem from "@site/src/components/FaqItem";
Frigate intelligently displays your camera streams on the Live view dashboard. By default, Frigate employs "smart streaming" where camera images update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion or active objects are detected, cameras seamlessly switch to a live stream.
@@ -341,100 +342,155 @@ When your browser runs into problems playing back your camera streams, it will l
## Live view FAQ
1. **Why don't I have audio in my Live view?**
### Getting Live View Working
You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending PCMA/PCMU or AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
<FaqItem id="why-dont-i-have-audio-in-my-live-view" question="Why don't I have audio in my Live view?">
Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio when in low bandwidth mode, even if you've set up go2rtc.
You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending PCMA/PCMU or AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
2. **Frigate shows that my live stream is in "low bandwidth mode". What does this mean?**
If the audio controls don't appear in the UI at all, verify that the Live view is actually using your go2rtc stream. If your go2rtc stream names don't match your Frigate camera name, you must map them with the `live -> streams` config (see [Setting Streams For Live UI](#setting-streams-for-live-ui) above); otherwise the UI falls back to the video-only jsmpeg player.
Frigate intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio when in low bandwidth mode, even if you've set up go2rtc.
When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI labels as "low bandwidth mode". On Live dashboards, the mode will automatically reset when smart streaming is configured and activity stops. Continuous streaming mode does not have an automatic reset mechanism, but you can use the _Reset_ option to force a reload of your stream.
</FaqItem>
If you are using continuous streaming or you are loading more than a few high resolution streams at once on the dashboard, your browser may struggle to begin playback of your streams before the timeout. Frigate always prioritizes showing a live stream as quickly as possible, even if it is a lower quality jsmpeg stream. You can use the "Reset" link/button to try loading your high resolution stream again.
<FaqItem id="i-have-unmuted-some-cameras-on-my-dashboard-but-i-do-not-hear-sound-why" question="I have unmuted some cameras on my dashboard, but I do not hear sound. Why?">
Errors in stream playback (e.g., connection failures, codec issues, or buffering timeouts) that cause the fallback to low bandwidth mode (jsmpeg) are logged to the browser console for easier debugging. These errors may include:
- Network issues (e.g., MSE or WebRTC network connection problems).
- Unsupported codecs or stream formats (e.g., H.265 in WebRTC, which is not supported in some browsers).
- Buffering timeouts or low bandwidth conditions causing fallback to jsmpeg.
- Browser compatibility problems (e.g., iOS Safari limitations with MSE).
If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
To view browser console logs:
1. Open the Frigate Live View in your browser.
2. Open the browser's Developer Tools (F12 or right-click > Inspect > Console tab).
3. Reproduce the error (e.g., load a problematic stream or simulate network issues).
4. Look for messages prefixed with the camera name.
</FaqItem>
These logs help identify if the issue is player-specific (MSE vs. WebRTC) or related to camera configuration (e.g., go2rtc streams, codecs). If you see frequent errors:
- Verify your camera's H.264/AAC settings (see [Frigate's camera settings recommendations](#camera-settings-recommendations)).
- Check go2rtc configuration for transcoding (e.g., audio to AAC/OPUS).
- Test with a different stream via the UI dropdown (if `live -> streams` is configured).
- For WebRTC-specific issues, ensure port 8555 is forwarded and candidates are set (see [WebRTC Extra Configuration](#webrtc-extra-configuration)).
- If your cameras are streaming at a high resolution, your browser may be struggling to load all of the streams before the buffering timeout occurs. Frigate prioritizes showing a true live view as quickly as possible. If the fallback occurs often, change your live view settings to use a lower bandwidth substream.
<FaqItem id="my-live-view-shows-a-black-screen-or-doesnt-load-but-the-debug-view-works-why" question="My live view shows a black screen or doesn't load, but the debug view works. Why?">
3. **It doesn't seem like my cameras are streaming on the Live dashboard. Why?**
The debug view plays the `detect` stream processed by Frigate itself, while the Live view plays your go2rtc stream directly in the browser. If the debug view works but the Live view doesn't, your browser usually can't decode what the camera is sending, most often H.265 video or an incompatible audio track.
On the default Live dashboard ("All Cameras"), your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity is detected, cameras seamlessly switch to a full-resolution live stream. If you want to customize this behavior, use a camera group.
Work through the [go2rtc troubleshooting guide](/troubleshooting/go2rtc#live-view-is-black-buffering-or-stuck-in-low-bandwidth-mode) to isolate the problem. Two fixes resolve the majority of cases:
4. **I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?**
1. Restream through go2rtc's FFmpeg module by prefixing your source with `ffmpeg:`, for example `- ffmpeg:rtsp://user:password@192.168.1.5:554/stream`.
2. If that doesn't help, transcode to compatible codecs: `- ffmpeg:rtsp://user:password@192.168.1.5:554/stream#video=h264#audio=aac#hardware`.
This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard (4:3, 16:9, or 32:9, etc) aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
</FaqItem>
5. **How does "smart streaming" work?**
<FaqItem id="how-do-i-get-the-best-live-view-experience-in-home-assistant" question="How do I get the best live view experience in Home Assistant?">
Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity (motion or object/audio detection) occurs, cameras seamlessly switch to a live stream.
For a full-resolution, low-latency live view in Home Assistant dashboards, use the [Advanced Camera Card](https://card.camera) with the [go2rtc live provider](https://card.camera/#/configuration/cameras/live-provider?id=go2rtc), which streams directly from Frigate's bundled go2rtc. This also supports audio and [two-way talk](#two-way-talk) on capable cameras. See the [Home Assistant integration docs](/integrations/home-assistant) for setup.
This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second so you can see the activity until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
</FaqItem>
Smart streaming depends on having your camera's motion `threshold` and `contour_area` config values dialed in. Use the Motion Tuner in Settings in the UI to tune these values in real-time.
### Streaming Behavior
This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
<FaqItem id="how-does-smart-streaming-work" question={'How does "smart streaming" work?'}>
6. **I have unmuted some cameras on my dashboard, but I do not hear sound. Why?**
Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity (motion or object/audio detection) occurs, cameras seamlessly switch to a live stream.
If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second so you can see the activity until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
7. **My camera streams have lots of visual artifacts / distortion.**
Smart streaming depends on having your camera's motion `threshold` and `contour_area` config values dialed in. Use the Motion Tuner in Settings in the UI to tune these values in real-time.
Some cameras don't include the hardware to support multiple connections to the high resolution stream, and this can cause unexpected behavior. In this case it is recommended to [restream](./restream.md) the high resolution stream so that it can be used for live view and recordings.
This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
8. **Why does my camera stream switch aspect ratios on the Live dashboard?**
</FaqItem>
Your camera may change aspect ratios on the dashboard because Frigate uses different streams for different purposes. With go2rtc and Smart Streaming, Frigate shows a static image from the `detect` stream when no activity is present, and switches to the live stream when motion is detected. The camera image will change size if your streams use different aspect ratios.
<FaqItem id="it-doesnt-seem-like-my-cameras-are-streaming-on-the-live-dashboard-why" question="It doesn't seem like my cameras are streaming on the Live dashboard. Why?">
To prevent this, make the `detect` stream match the go2rtc live stream's aspect ratio (resolution does not need to match, just the aspect ratio). You can either adjust the camera's output resolution or set the `width` and `height` values in your config's `detect` section to a resolution with an aspect ratio that matches.
On the default Live dashboard ("All Cameras"), your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any activity is detected, cameras seamlessly switch to a full-resolution live stream. If you want to customize this behavior, use a camera group.
Example: Resolutions from two streams
- Mismatched (may cause aspect ratio switching on the dashboard):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x352 (~1.82:1, not 16:9)
</FaqItem>
- Matched (prevents switching):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x360 (16:9)
<FaqItem id="frigate-shows-that-my-live-stream-is-in-low-bandwidth-mode-what-does-this-mean" question={'Frigate shows that my live stream is in "low bandwidth mode". What does this mean?'}>
You can update the detect settings in your camera config to match the aspect ratio of your go2rtc live stream. For example:
Frigate intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
```yaml
cameras:
front_door:
detect:
width: 640
height: 360 # set this to 360 instead of 352
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/front_door # main stream 1920x1080
roles:
- record
- path: rtsp://127.0.0.1:8554/front_door_sub # sub stream 640x352
roles:
- detect
```
When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI labels as "low bandwidth mode". On Live dashboards, the mode will automatically reset when smart streaming is configured and activity stops. Continuous streaming mode does not have an automatic reset mechanism, but you can use the _Reset_ option to force a reload of your stream.
The same applies to your `record` stream: if its aspect ratio differs from your `detect` stream, your recordings will appear in a different shape than the live view. For consistent framing across live view and recordings, use the same aspect ratio for all of a camera's streams (the resolution can still differ).
If you are using continuous streaming or you are loading more than a few high resolution streams at once on the dashboard, your browser may struggle to begin playback of your streams before the timeout. Frigate always prioritizes showing a live stream as quickly as possible, even if it is a lower quality jsmpeg stream. You can use the "Reset" link/button to try loading your high resolution stream again.
9. **Why does Frigate prefer MSE over WebRTC for live view?**
Errors in stream playback (e.g., connection failures, codec issues, or buffering timeouts) that cause the fallback to low bandwidth mode (jsmpeg) are logged to the browser console for easier debugging. These errors may include:
Frigate prefers MSE because it delivers a better out-of-the-box experience than WebRTC on nearly every axis that matters for a security camera system. MSE is an open standard optimized and supported by all modern browsers, works without any extra configuration (WebRTC requires port forwarding and candidate setup, and lacks H.265 support in some browsers), and requires no internet access for NAT traversal. More importantly, MSE runs over TCP, so every frame arrives and is decoded in order, so nothing is ever silently skipped. WebRTC optimizes for latency over UDP by discarding late or incomplete frames, which works against you on cellular or spotty Wi-Fi: you can end up with frozen video, visual corruption, or gaps in the feed without ever knowing you missed something. Frigate's enhanced MSE player has adaptive speed playback and has been tuned for latency and connection robustness that meets or exceeds WebRTC, so you get near-real-time playback with a guarantee that when the video plays, every frame is actually there - which, for an NVR whose whole purpose is letting you see what happened, matters more than shaving fractions of a second off a latency number. That's why Frigate defaults to MSE and reserves WebRTC for cases that require it, like two-way talk.
- Network issues (e.g., MSE or WebRTC network connection problems).
- Unsupported codecs or stream formats (e.g., H.265 in WebRTC, which is not supported in some browsers).
- Buffering timeouts or low bandwidth conditions causing fallback to jsmpeg.
- Browser compatibility problems (e.g., iOS Safari limitations with MSE).
To view browser console logs:
1. Open the Frigate Live View in your browser.
2. Open the browser's Developer Tools (F12 or right-click > Inspect > Console tab).
3. Reproduce the error (e.g., load a problematic stream or simulate network issues).
4. Look for messages prefixed with the camera name.
These logs help identify if the issue is player-specific (MSE vs. WebRTC) or related to camera configuration (e.g., go2rtc streams, codecs). If you see frequent errors:
- Verify your camera's H.264/AAC settings (see [Frigate's camera settings recommendations](#camera-settings-recommendations)).
- Check go2rtc configuration for transcoding (e.g., audio to AAC/OPUS).
- Test with a different stream via the UI dropdown (if `live -> streams` is configured).
- For WebRTC-specific issues, ensure port 8555 is forwarded and candidates are set (see [WebRTC Extra Configuration](#webrtc-extra-configuration)).
- If your cameras are streaming at a high resolution, your browser may be struggling to load all of the streams before the buffering timeout occurs. Frigate prioritizes showing a true live view as quickly as possible. If the fallback occurs often, change your live view settings to use a lower bandwidth substream.
</FaqItem>
<FaqItem id="why-is-my-live-view-delayed-or-lagging-behind-real-time" question="Why is my live view delayed or lagging behind real time?">
A delay when a stream first starts is usually caused by your camera's I-frame (keyframe) interval. Playback cannot begin until a keyframe arrives, so an interval set higher than your camera's frame rate makes the stream take longer to start. Set the I-frame interval to match the frame rate (or "1x" on Reolink) per the [camera settings recommendations](#camera-settings-recommendations).
A stream that starts on time but falls further behind live is buffering, which is usually the browser struggling to decode too many high-resolution streams at once. Select a lower-bandwidth substream for your dashboards (see [Setting Streams For Live UI](#setting-streams-for-live-ui)), reduce the number of streams open at once, or improve the network connection between your browser and Frigate. Frigate's player automatically speeds up playback to catch up to live after buffering, and falls back to low bandwidth mode if it stalls for too long. The _Reset_ option forces a fresh connection at the live edge.
</FaqItem>
<FaqItem id="why-does-frigate-prefer-mse-over-webrtc-for-live-view" question="Why does Frigate prefer MSE over WebRTC for live view?">
Frigate prefers MSE because it delivers a better out-of-the-box experience than WebRTC on nearly every axis that matters for a security camera system. MSE is an open standard optimized and supported by all modern browsers, works without any extra configuration (WebRTC requires port forwarding and candidate setup, and lacks H.265 support in some browsers), and requires no internet access for NAT traversal. More importantly, MSE runs over TCP, so every frame arrives and is decoded in order, so nothing is ever silently skipped. WebRTC optimizes for latency over UDP by discarding late or incomplete frames, which works against you on cellular or spotty Wi-Fi: you can end up with frozen video, visual corruption, or gaps in the feed without ever knowing you missed something. Frigate's enhanced MSE player has adaptive speed playback and has been tuned for latency and connection robustness that meets or exceeds WebRTC, so you get near-real-time playback with a guarantee that when the video plays, every frame is actually there - which, for an NVR whose whole purpose is letting you see what happened, matters more than shaving fractions of a second off a latency number. That's why Frigate defaults to MSE and reserves WebRTC for cases that require it, like two-way talk.
</FaqItem>
### Video Quality Issues
<FaqItem id="i-see-a-strange-diagonal-line-on-my-live-view-but-my-recordings-look-fine-how-can-i-fix-it" question="I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?">
This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard (4:3, 16:9, or 32:9, etc) aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
</FaqItem>
<FaqItem id="my-camera-streams-have-lots-of-visual-artifacts-or-distortion" question="My camera streams have lots of visual artifacts / distortion.">
Some cameras don't include the hardware to support multiple connections to the high resolution stream, and this can cause unexpected behavior. In this case it is recommended to [restream](./restream.md) the high resolution stream so that it can be used for live view and recordings.
</FaqItem>
<FaqItem id="why-does-my-camera-stream-switch-aspect-ratios-on-the-live-dashboard" question="Why does my camera stream switch aspect ratios on the Live dashboard?">
Your camera may change aspect ratios on the dashboard because Frigate uses different streams for different purposes. With go2rtc and Smart Streaming, Frigate shows a static image from the `detect` stream when no activity is present, and switches to the live stream when motion is detected. The camera image will change size if your streams use different aspect ratios.
To prevent this, make the `detect` stream match the go2rtc live stream's aspect ratio (resolution does not need to match, just the aspect ratio). You can either adjust the camera's output resolution or set the `width` and `height` values in your config's `detect` section to a resolution with an aspect ratio that matches.
Example: Resolutions from two streams
- Mismatched (may cause aspect ratio switching on the dashboard):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x352 (~1.82:1, not 16:9)
- Matched (prevents switching):
- Live/go2rtc stream: 1920x1080 (16:9)
- Detect stream: 640x360 (16:9)
You can update the detect settings in your camera config to match the aspect ratio of your go2rtc live stream. For example:
```yaml
cameras:
front_door:
detect:
width: 640
height: 360 # set this to 360 instead of 352
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/front_door # main stream 1920x1080
roles:
- record
- path: rtsp://127.0.0.1:8554/front_door_sub # sub stream 640x352
roles:
- detect
```
The same applies to your `record` stream: if its aspect ratio differs from your `detect` stream, your recordings will appear in a different shape than the live view. For consistent framing across live view and recordings, use the same aspect ratio for all of a camera's streams (the resolution can still differ).
</FaqItem>
+1 -1
View File
@@ -66,7 +66,7 @@ motion:
</TabItem>
</ConfigTabs>
Lower values mean motion detection is more sensitive to changes in color, making it more likely for example to detect motion when a brown dogs blends in with a brown fence or a person wearing a red shirt blends in with a red car. If the threshold is too low however, it may detect things like grass blowing in the wind, shadows, etc. to be detected as motion.
Lower values mean motion detection is more sensitive to changes in color, making it more likely for example to detect motion when a brown dog blends in with a brown fence or a person wearing a red shirt blends in with a red car. If the threshold is too low however, it may detect things like grass blowing in the wind, shadows, etc. to be detected as motion.
Watching the motion boxes in the debug view, increase the threshold until you only see motion that is visible to the eye. Once this is done, it is important to test and ensure that desired motion is still detected.
+3 -3
View File
@@ -725,7 +725,7 @@ The inference time was determined on a rk3588 with 3 NPU cores.
To convert a onnx model to the rknn format using the [rknn-toolkit2](https://github.com/airockchip/rknn-toolkit2/) you have to:
- Place one ore more models in onnx format in the directory `config/model_cache/rknn_cache/onnx` on your docker host (this might require `sudo` privileges).
- Place one or more models in onnx format in the directory `config/model_cache/rknn_cache/onnx` on your docker host (this might require `sudo` privileges).
- Save the configuration file under `config/conv2rknn.yaml` (see below for details).
- Run `docker exec <frigate_container_id> python3 /opt/conv2rknn.py`. If the conversion was successful, the rknn models will be placed in `config/model_cache/rknn_cache`.
@@ -743,13 +743,13 @@ config:
quant_img_RGB2BGR: true
```
Explanation of the paramters:
Explanation of the parameters:
- `soc`: A list of all SoCs you want to build the rknn model for. If you don't specify this parameter, the script tries to find out your SoC and builds the rknn model for this one.
- `quantization`: true: 8 bit integer (i8) quantization, false: 16 bit float (fp16). Default: false.
- `output_name`: The output name of the model. The following variables are available:
- `quant`: "i8" or "fp16" depending on the config
- `input_basename`: the basename of the input model (e.g. "my_model" if the input model is calles "my_model.onnx")
- `input_basename`: the basename of the input model (e.g. "my_model" if the input model is called "my_model.onnx")
- `soc`: the SoC this model was build for (e.g. "rk3588")
- `tk_version`: Version of `rknn-toolkit2` (e.g. "2.3.0")
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
+1 -1
View File
@@ -36,7 +36,7 @@ Any detection below `min_score` will be immediately thrown out and never tracked
### Threshold
`threshold` is used to determine that the object is a true positive. Once an object is detected with a score >= `threshold` object is considered a true positive. If `threshold` is too low then some higher scoring false positives may create an tracked object. If `threshold` is too high then true positive tracked objects may be missed due to the object never scoring high enough.
`threshold` is used to determine that the object is a true positive. Once an object is detected with a score >= `threshold` object is considered a true positive. If `threshold` is too low then some higher scoring false positives may create a tracked object. If `threshold` is too high then true positive tracked objects may be missed due to the object never scoring high enough.
## Configuring Object Scores
+1 -1
View File
@@ -226,7 +226,7 @@ For tips on getting the best results from Semantic Search (choosing between thum
## Triggers
Triggers utilize Semantic Search to automate actions when a tracked object matches a specified image or description. Triggers can be configured so that Frigate executes a specific actions when a tracked object's image or description matches a predefined image or text, based on a similarity threshold. Triggers are managed per camera and can be configured via the Frigate UI in the Settings page under the Triggers tab.
Triggers utilize Semantic Search to automate actions when a tracked object matches a specified image or description. Triggers can be configured so that Frigate executes specific actions when a tracked object's image or description matches a predefined image or text, based on a similarity threshold. Triggers are managed per camera and can be configured via the Frigate UI in the Settings page under the Triggers tab.
:::note
@@ -43,7 +43,7 @@ Let's look at an example use case: I want to record any cars that enter my drive
One might simply think "Why not just run object detection any time there is motion around the driveway area and notify if the bounding box is in that zone?"
With that approach, what video is related to the car that entered the driveway? Did it come from the left or right? Was it parked across the street for an hour before turning into the driveway? One approach is to just record 24/7 or for motion (on any changed changed pixels) and not attempt to do that at all. This is what most other NVRs do. Just don't even try to identify a start and end for that object since it's hard and you will be wrong some portion of the time.
With that approach, what video is related to the car that entered the driveway? Did it come from the left or right? Was it parked across the street for an hour before turning into the driveway? One approach is to just record 24/7 or for motion (on any changed pixels) and not attempt to do that at all. This is what most other NVRs do. Just don't even try to identify a start and end for that object since it's hard and you will be wrong some portion of the time.
Couldn't you just look at when motion stopped and started? Motion for a video feed is nothing more than looking for pixels that are different than they were in previous frames. If the car entered the driveway while someone was mowing the grass, how would you know which motion was for the car and which was for the person when they mow along the driveway or street? What if another car was driving the other direction on the street? Or what if its a windy day and the bush by your mailbox is blowing around?
@@ -61,4 +61,4 @@ Now you have to determine which of the bounding boxes in this frame should be ma
Now let's assume that those other 3 cars were already being tracked as stationary objects, so the car driving down the street is a new 4th car. The object tracker knows we have had 3 cars and we now have 4. As the new car approaches the parked cars, the bounding boxes for all 4 cars is predicted based on the previous frames. The predicted boxes for the parked cars is pretty much a 100% overlap with the bounding boxes in the new frame. The parked cars are slam dunk matches to the tracking ids they had before and the only one left is the remaining bounding box which gets assigned to the new car. This results in a much lower error rate. Not perfect, but better.
The most difficult scenario that causes IDs to be assigned incorrectly is when an object completely occludes another object. When a car drives in front of another car and its no longer visible, a bounding box disappeared and it's a bit of a toss up when assigning the id since it's difficult to know which one is in front of the other. This happens for cars passing in front of other cars fairly often. It's something that we want to improve in the future.
The most difficult scenario that causes IDs to be assigned incorrectly is when an object completely occludes another object. When a car drives in front of another car and it's no longer visible, a bounding box disappeared and it's a bit of a toss up when assigning the id since it's difficult to know which one is in front of the other. This happens for cars passing in front of other cars fairly often. It's something that we want to improve in the future.
+1 -1
View File
@@ -94,7 +94,7 @@ The following sections contain additional setup steps that are only required if
By default, the Raspberry Pi limits the amount of memory available to the GPU. In order to use ffmpeg hardware acceleration, you must increase the available memory by setting `gpu_mem` to the maximum recommended value in `config.txt` as described in the [official docs](https://www.raspberrypi.org/documentation/computers/config_txt.html#memory-options).
Additionally, the USB Coral draws a considerable amount of power. If using any other USB devices such as an SSD, you will experience instability due to the Pi not providing enough power to USB devices. You will need to purchase an external USB hub with it's own power supply. Some have reported success with <a href="https://amzn.to/3a2mH0P" target="_blank" rel="nofollow noopener sponsored">this</a> (affiliate link).
Additionally, the USB Coral draws a considerable amount of power. If using any other USB devices such as an SSD, you will experience instability due to the Pi not providing enough power to USB devices. You will need to purchase an external USB hub with its own power supply. Some have reported success with <a href="https://amzn.to/3a2mH0P" target="_blank" rel="nofollow noopener sponsored">this</a> (affiliate link).
### Hailo-8
+1 -1
View File
@@ -122,7 +122,7 @@ Use `http://<frigate_device_ip>:8971` as the URL for the integration so that aut
The above URL assumes you have [disabled TLS](../configuration/tls).
By default, TLS is enabled and Frigate will be using a self-signed certificate. HomeAssistant will fail to connect HTTPS to port 8971 since it fails to verify the self-signed certificate.
Either disable TLS and use HTTP from HomeAssistant, or configure Frigate to be acessible with a valid certificate.
Either disable TLS and use HTTP from HomeAssistant, or configure Frigate to be accessible with a valid certificate.
:::
+1 -1
View File
@@ -280,7 +280,7 @@ Same data available at `/api/stats` published at a configurable interval.
### `frigate/camera_activity`
Returns data about each camera, its current features, and if it is detecting motion, objects, etc. Can be triggered by publising to `frigate/onConnect`
Returns data about each camera, its current features, and if it is detecting motion, objects, etc. Can be triggered by publishing to `frigate/onConnect`
### `frigate/profile/set`
@@ -23,7 +23,7 @@ The [Advanced Camera Card](https://card.camera/#/README) is a Home Assistant das
## [Double Take](https://github.com/skrashevich/double-take)
[Double Take](https://github.com/skrashevich/double-take) provides an unified UI and API for processing and training images for facial recognition.
[Double Take](https://github.com/skrashevich/double-take) provides a unified UI and API for processing and training images for facial recognition.
It supports automatically setting the sub labels in Frigate for person objects that are detected and recognized.
This is a fork (with fixed errors and new features) of [original Double Take](https://github.com/jakowenko/double-take) project which, unfortunately, isn't being maintained by author.
@@ -53,7 +53,7 @@ This is a fork (with fixed errors and new features) of [original Double Take](ht
## [Scrypted - Frigate bridge plugin](https://github.com/apocaliss92/scrypted-frigate-bridge)
[Scrypted - Frigate bridge](https://github.com/apocaliss92/scrypted-frigate-bridge) is an plugin that allows to ingest Frigate detections, motion, videoclips on Scrypted as well as provide templates to export rebroadcast configurations on Frigate.
[Scrypted - Frigate bridge](https://github.com/apocaliss92/scrypted-frigate-bridge) is a plugin that allows you to ingest Frigate detections, motion, videoclips on Scrypted as well as provide templates to export rebroadcast configurations on Frigate.
## [Strix](https://github.com/eduard256/Strix)
+1 -1
View File
@@ -19,7 +19,7 @@ For the best results, follow these guidelines. You may also want to review the d
## AI suggested labels
If you have an active Frigate+ subscription, new uploads will be scanned for the objects configured for you camera and you will see suggested labels as light blue boxes when annotating in Frigate+. These suggestions are processed via a queue and typically complete within a minute after uploading, but processing times can be longer.
If you have an active Frigate+ subscription, new uploads will be scanned for the objects configured for your camera and you will see suggested labels as light blue boxes when annotating in Frigate+. These suggestions are processed via a queue and typically complete within a minute after uploading, but processing times can be longer.
![Suggestions](/img/plus/suggestions.webp)
+46 -7
View File
@@ -3,6 +3,10 @@ id: first_model
title: Requesting your first model
---
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
## Step 1: Upload and annotate your images
Before requesting your first model, you will need to upload and verify at least 10 images to Frigate+. The more images you upload, annotate, and verify the better your results will be. Most users start to see very good results once they have at least 100 verified images per camera. Keep in mind that varying conditions should be included. You will want images from cloudy days, sunny days, dawn, dusk, and night. Refer to the [integration docs](../integrations/plus.md#generate-an-api-key) for instructions on how to easily submit images to Frigate+ directly from Frigate.
@@ -16,13 +20,21 @@ For more detailed recommendations, you can refer to the docs on [annotating](./a
Once you have an initial set of verified images, you can request a model on the Models page. For guidance on choosing a model type, refer to [this part of the documentation](./index.md#available-model-types). If you are unsure which type to request, you can test the base model for each version from the "Base Models" tab. Each model request requires 1 of the 12 trainings that you receive with your annual subscription. This model will support all [label types available](./index.md#available-label-types) even if you do not submit any examples for those labels. Model creation can take up to 36 hours.
![Plus Models Page](/img/plus/plus-models.jpg)
## Step 3: Set your model id in the config
## Step 3: Set your model
You will receive an email notification when your Frigate+ model is ready.
![Model Ready Email](/img/plus/model-ready-email.jpg)
Models available in Frigate+ can be used with a special model path. No other information needs to be configured because it fetches the remaining config from Frigate+ automatically.
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detectors and model" />. In the **Detection Model** section, choose the **Frigate+** tab. Select your new Frigate+ model from the **Available Frigate+ models** dropdown, then click **Save**. Restart Frigate to apply the change.
</TabItem>
<TabItem value="yaml">
```yaml
detectors: ...
@@ -30,22 +42,46 @@ model:
path: plus://<your_model_id>
```
:::note
Model IDs are not secret values and can be shared freely. Access to your model is protected by your API key.
:::
:::tip
When setting the plus model id, all other fields should be removed as these are configured automatically with the Frigate+ model config
:::
</TabItem>
</ConfigTabs>
:::note
Model IDs are not secret values and can be shared freely. Access to your model is protected by your API key.
:::
## Step 4: Adjust your object filters for higher scores
Frigate+ models generally have much higher scores than the default model provided in Frigate. You will likely need to increase your `threshold` and `min_score` values. Here is an example of how these values can be refined, but you should expect these to evolve as your model improves. For more information about how `threshold` and `min_score` are related, see the docs on [object filters](../configuration/object_filters.md#object-scores).
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > Global configuration > Objects" />. Under **Object filters**, set **Min Score** and **Threshold** for each object type, then click **Save**.
| Object | Min Score | Threshold |
| ----------------- | --------- | --------- |
| **dog** | .7 | .9 |
| **cat** | .65 | .8 |
| **face** | .7 | |
| **package** | .65 | .9 |
| **license_plate** | .6 | |
| **amazon** | .75 | |
| **ups** | .75 | |
| **fedex** | .75 | |
| **person** | .65 | .85 |
| **car** | .65 | .85 |
</TabItem>
<TabItem value="yaml">
```yaml
objects:
filters:
@@ -75,3 +111,6 @@ objects:
min_score: .65
threshold: .85
```
</TabItem>
</ConfigTabs>
+5 -5
View File
@@ -12,7 +12,7 @@ There are many possible causes for a USB coral not being detected and some are O
:::tip
Using `lsusb` or checking the hardware page in HA OS will show as `1a6e:089a Global Unichip Corp.` until Frigate runs an inferance using the coral. So don't worry about the identification until after Frigate has attempted to detect the coral.
Using `lsusb` or checking the hardware page in HA OS will show as `1a6e:089a Global Unichip Corp.` until Frigate runs an inference using the coral. So don't worry about the identification until after Frigate has attempted to detect the coral.
:::
@@ -43,13 +43,13 @@ Some users have reported that this older device runs an older kernel causing iss
3. Start the docker container with Coral TPU enabled in the config
4. The TPU would be detected but a few moments later it would disconnect.
5. While leaving the TPU device plugged in, restart the NAS using the reboot command in the UI. Do NOT unplug the NAS/power it off etc.
6. Open the control panel - info scree. The coral TPU will now be recognised as a USB Device - google inc
6. Open the control panel - info screen. The coral TPU will now be recognized as a USB Device - google inc
7. Start the frigate container. Everything should work now!
### QNAP NAS
QNAP NAS devices, such as the TS-253A, may use connected Coral TPU devices if [QuMagie](https://www.qnap.com/en/software/qumagie) is installed along with its QNAP AI Core extension. If any of the features (`facial recognition`, `object recognition`, or `similar photo recognition`) are enabled, Container Station applications such as `Frigate` or `CodeProject.AI Server` will be unable to initialize the TPU device in use.
To allow the Coral TPU device to be discovered, the you must either:
To allow the Coral TPU device to be discovered, you must either:
1. [Disable the AI recognition features in QuMagie](https://docs.qnap.com/application/qumagie/2.x/en-us/configuring-qnap-ai-core-settings-FB13CE03.html),
2. Remove the QNAP AI Core extension or
@@ -76,7 +76,7 @@ This is an issue due to outdated gasket driver when being used with new linux ke
### Not detected on Raspberry Pi5
A kernel update to the RPi5 means an upate to config.txt is required, see [the raspberry pi forum for more info](https://forums.raspberrypi.com/viewtopic.php?t=363682&sid=cb59b026a412f0dc041595951273a9ca&start=25)
A kernel update to the RPi5 means an update to config.txt is required, see [the raspberry pi forum for more info](https://forums.raspberrypi.com/viewtopic.php?t=363682&sid=cb59b026a412f0dc041595951273a9ca&start=25)
Specifically, add the following to config.txt
@@ -87,7 +87,7 @@ dtoverlay=pcie-32bit-dma-pi5
## Only One PCIe Coral Is Detected With Coral Dual EdgeTPU
Coral Dual EdgeTPU is one card with two identical TPU cores. Each core has it's own PCIe interface and motherboard needs to have two PCIe busses on the m.2 slot to make them both work.
Coral Dual EdgeTPU is one card with two identical TPU cores. Each core has its own PCIe interface and motherboard needs to have two PCIe busses on the m.2 slot to make them both work.
E-key slot implemented to full m.2 electromechanical specification has two PCIe busses. Most motherboard manufacturers implement only one PCIe bus in m.2 E-key connector (this is why only one TPU is working). Some SBCs can have only USB bus on m.2 connector, ie none of TPUs will work.
+1 -1
View File
@@ -29,7 +29,7 @@ You can open `chrome://media-internals/` in another tab and then try to playback
### What do I do if my cameras sub stream is not good enough?
Frigate generally [recommends cameras with configurable sub streams](/frigate/hardware.md). However, if your camera does not have a sub stream that a suitable resolution, the main stream can be resized.
Frigate generally [recommends cameras with configurable sub streams](/frigate/hardware.md). However, if your camera does not have a sub stream that is a suitable resolution, the main stream can be resized.
To do this efficiently the following setup is required:
+1 -1
View File
@@ -91,7 +91,7 @@ To act on many objects at once, Ctrl/Cmd-click or right-click to start a selecti
1. Semantic Search is used in conjunction with the other filters available on the Explore page. Use a combination of traditional filtering and Semantic Search for the best results.
2. Use the thumbnail search type when searching for particular objects in the scene. Use the description search type when attempting to discern the intent of your object.
3. Because of how the AI models Frigate uses have been trained, the comparison between text and image embedding distances generally means that with multi-modal (`thumbnail` and `description`) searches, results matching `description` will appear first, even if a `thumbnail` embedding may be a better match. Play with the "Search Type" setting to help find what you are looking for. Note that if you are generating descriptions for specific objects or zones only, this may cause search results to prioritize the objects with descriptions even if the the ones without them are more relevant.
3. Because of how the AI models Frigate uses have been trained, the comparison between text and image embedding distances generally means that with multi-modal (`thumbnail` and `description`) searches, results matching `description` will appear first, even if a `thumbnail` embedding may be a better match. Play with the "Search Type" setting to help find what you are looking for. Note that if you are generating descriptions for specific objects or zones only, this may cause search results to prioritize the objects with descriptions even if the ones without them are more relevant.
4. Make your search language and tone closely match exactly what you're looking for. If you are using thumbnail search, **phrase your query as an image caption**. Searching for "red car" may not work as well as "red sedan driving down a residential street on a sunny day".
5. Semantic search on thumbnails tends to return better results when matching large subjects that take up most of the frame. Small things like "cat" tend to not work well.
6. Experiment! Find a tracked object you want to test and start typing keywords and phrases to see what works for you.
+1 -1
View File
@@ -60,7 +60,7 @@ You can optionally overlay live streaming statistics (stream type, bandwidth, la
Right-clicking (or long-pressing) a camera tile opens a context menu with quick controls: an **audio volume** control for streams that support audio, **Mute / Unmute all cameras**, **show or hide streaming statistics**, the **debug view**, **notification** options, and, for admins, turning the camera on or off. If the audio control doesn't appear, see [Audio Support](/configuration/live#audio-support). Audio requires go2rtc configured with a compatible codec.
A **Low-bandwidth mode** notice may also appear in the context menu with a **Reset** option appears when Frigate has fallen back to the lower-quality jsmpeg stream. See the [Live view FAQ](/configuration/live#live-view-faq) for why this happens.
A **Low-bandwidth mode** notice may also appear in the context menu with a **Reset** option when Frigate has fallen back to the lower-quality jsmpeg stream. See the [Live view FAQ](/configuration/live#live-view-faq) for why this happens.
For non-default groups, the context menu also exposes **Streaming Settings** for that camera, which let you choose:
+66
View File
@@ -0,0 +1,66 @@
import React, { useState, useEffect } from "react";
import Heading from "@theme/Heading";
import styles from "./styles.module.css";
// A single FAQ entry.
//
// The question is a real anchored heading (via @theme/Heading), so on desktop
// it gets the standard hover "#" hash link and the answer is always shown. On
// mobile the heading text is a button that toggles its answer, keeping long
// FAQ pages short. The desktop/mobile split is pure CSS (Docusaurus breakpoint:
// 996px), so there is no hydration flash. The answer is always rendered into
// the DOM, so search engines and the docs AI bot can read it regardless of
// layout or collapsed state. The heading id resolves deep links on both layouts
// and auto-expands the entry on mobile when it is the link target.
export default function FaqItem({ id, question, children }) {
const [open, setOpen] = useState(false);
useEffect(() => {
const openIfTargeted = () => {
if (window.location.hash === `#${id}`) {
setOpen(true);
}
};
openIfTargeted();
window.addEventListener("hashchange", openIfTargeted);
return () => window.removeEventListener("hashchange", openIfTargeted);
}, [id]);
const toggle = () => {
const next = !open;
setOpen(next);
// Reflect the entry in the URL like clicking the heading anchor, so an
// opened answer is shareable. Use replaceState to avoid history spam and
// an abrupt scroll. Clear it on close if it currently points here.
if (next) {
if (window.location.hash !== `#${id}`) {
window.history.replaceState(null, "", `#${id}`);
}
} else if (window.location.hash === `#${id}`) {
window.history.replaceState(
null,
"",
window.location.pathname + window.location.search,
);
}
};
return (
<div className={styles.item} data-open={open || undefined}>
<Heading as="h4" id={id} className={styles.heading}>
<button
type="button"
className={styles.toggle}
aria-expanded={open}
aria-controls={`${id}-content`}
onClick={toggle}
>
{question}
</button>
</Heading>
<div id={`${id}-content`} className={styles.content}>
{children}
</div>
</div>
);
}
@@ -0,0 +1,87 @@
/*
* FAQ entry: collapsible on mobile, static heading + expanded answer on
* desktop. The split is pure CSS (Docusaurus breakpoint: 996px) so there is
* no hydration flash. The answer is always rendered into the DOM, so search
* engines and the docs AI bot can read it regardless of layout or state.
*/
.item {
scroll-margin-top: calc(var(--ifm-navbar-height) + 1rem);
}
.heading {
margin: 0;
}
/* Mobile: the heading text is a full-width clickable toggle row. */
.toggle {
display: flex;
align-items: center;
gap: 0.6rem;
width: 100%;
padding: 0.85rem 0;
border: none;
border-bottom: 1px solid var(--ifm-color-emphasis-200);
background: none;
color: inherit;
font: inherit;
text-align: left;
cursor: pointer;
}
.toggle::before {
content: "";
flex: 0 0 auto;
width: 0.5rem;
height: 0.5rem;
border-right: 2px solid currentColor;
border-bottom: 2px solid currentColor;
transform: rotate(-45deg);
transition: transform var(--ifm-transition-fast, 200ms) ease;
}
.item[data-open] .toggle::before {
transform: rotate(45deg);
}
.content {
display: none;
padding: 0 0 0.85rem;
}
.item[data-open] .content {
display: block;
}
/* Hide the hover hash link on mobile (no hover; avoids a stray empty line). */
.heading :global(.hash-link) {
display: none;
}
/* Desktop: render as a normal expanded heading + answer. */
@media (min-width: 997px) {
.heading {
margin: 1.75rem 0 0.5rem;
}
.toggle {
display: inline;
width: auto;
padding: 0;
border: none;
cursor: default;
}
.toggle::before {
display: none;
}
.content {
display: block;
padding: 0;
}
.heading :global(.hash-link) {
display: inline;
}
}
+49 -30
View File
@@ -200,6 +200,9 @@ class EmbeddingMaintainer(threading.Thread):
)
for model_config in self.config.classification.custom.values():
if not model_config.enabled:
continue
self.realtime_processors.append(
CustomStateClassificationProcessor(
self.config, model_config, self.requestor, self.metrics
@@ -332,6 +335,25 @@ class EmbeddingMaintainer(threading.Thread):
for processor in self.post_processors:
processor.update_config(topic, payload)
def _remove_custom_classification_processor(self, model_name: str) -> None:
"""Shut down and drop any running processor for a custom model."""
remaining = []
for processor in self.realtime_processors:
if (
isinstance(
processor,
(
CustomStateClassificationProcessor,
CustomObjectClassificationProcessor,
),
)
and processor.model_config.name == model_name
):
processor.shutdown()
else:
remaining.append(processor)
self.realtime_processors = remaining
def _handle_custom_classification_update(
self, topic: str, model_config: Any
) -> None:
@@ -339,23 +361,7 @@ class EmbeddingMaintainer(threading.Thread):
model_name = topic.split("/")[-1]
if model_config is None:
remaining = []
for processor in self.realtime_processors:
if (
isinstance(
processor,
(
CustomStateClassificationProcessor,
CustomObjectClassificationProcessor,
),
)
and processor.model_config.name == model_name
):
processor.shutdown()
else:
remaining.append(processor)
self.realtime_processors = remaining
self._remove_custom_classification_processor(model_name)
logger.info(
f"Successfully removed classification processor for model: {model_name}"
)
@@ -363,20 +369,29 @@ class EmbeddingMaintainer(threading.Thread):
self.config.classification.custom[model_name] = model_config
# Check if processor already exists
# A disabled model must not run; tear down any existing processor and
# do not register a new one.
if not model_config.enabled:
self._remove_custom_classification_processor(model_name)
logger.info(f"Disabled classification processor for model: {model_name}")
return
for processor in self.realtime_processors:
if isinstance(
processor,
(
CustomStateClassificationProcessor,
CustomObjectClassificationProcessor,
),
if (
isinstance(
processor,
(
CustomStateClassificationProcessor,
CustomObjectClassificationProcessor,
),
)
and processor.model_config.name == model_name
):
if processor.model_config.name == model_name:
logger.debug(
f"Classification processor for model {model_name} already exists, skipping"
)
return
processor.model_config = model_config
logger.debug(
f"Updated config for classification processor: {model_name}"
)
return
if model_config.state_config is not None:
processor = CustomStateClassificationProcessor(
@@ -702,7 +717,11 @@ class EmbeddingMaintainer(threading.Thread):
and "license_plate" not in camera_config.objects.track
)
if not dedicated_lpr_enabled and len(self.config.classification.custom) == 0:
has_enabled_custom = any(
c.enabled for c in self.config.classification.custom.values()
)
if not dedicated_lpr_enabled and not has_enabled_custom:
# no active features that use this data
return
+5
View File
@@ -281,6 +281,11 @@ class GenAIClient:
"""Whether the configured model exposes a per-request thinking toggle."""
return False
@property
def supports_embeddings(self) -> bool:
"""Whether the configured model can generate embeddings via embed()."""
return False
def list_models(self) -> list[str]:
"""Return the list of model names available from this provider.
+1
View File
@@ -121,5 +121,6 @@ class GenAIClientManager:
"models": client.list_models(),
"roles": [r.value for r in genai_cfg.roles],
"supports_toggleable_thinking": client.supports_toggleable_thinking,
"supports_embeddings": client.supports_embeddings,
}
return result
+62 -60
View File
@@ -38,6 +38,37 @@ def _encode_thought_signature(signature: bytes | None) -> str | None:
return base64.b64encode(signature).decode("ascii")
def _decode_data_uri(url: str) -> tuple[str, bytes] | None:
"""Decode a ``data:`` URI into ``(mime_type, bytes)``; None if not a data URI."""
if not isinstance(url, str) or not url.startswith("data:"):
return None
try:
header, b64 = url.split(",", 1)
mime = header[len("data:") :].split(";")[0] or "image/jpeg"
return mime, base64.b64decode(b64)
except (ValueError, binascii.Error):
return None
def _parts_from_content(content: Any) -> list[types.Part]:
"""Convert OpenAI-style message content (str or multimodal list) to Gemini parts."""
if isinstance(content, list):
parts: list[types.Part] = []
for item in content:
if not isinstance(item, dict):
continue
if item.get("type") == "text":
parts.append(types.Part.from_text(text=item.get("text") or ""))
elif item.get("type") == "image_url":
decoded = _decode_data_uri((item.get("image_url") or {}).get("url", ""))
if decoded is not None:
mime, data = decoded
parts.append(types.Part.from_bytes(data=data, mime_type=mime))
# Gemini rejects empty parts; fall back to a single space.
return parts or [types.Part.from_text(text=" ")]
return [types.Part.from_text(text=content or "")]
def _stats_from_gemini_usage(usage: Any) -> dict[str, Any] | None:
"""Build a stats dict from a Gemini usage_metadata object."""
prompt_tokens = getattr(usage, "prompt_token_count", None)
@@ -227,9 +258,7 @@ class GeminiClient(GenAIClient):
)
else: # user
gemini_messages.append(
types.Content(
role="user", parts=[types.Part.from_text(text=content)]
)
types.Content(role="user", parts=_parts_from_content(content))
)
# Convert tools to Gemini format
@@ -485,9 +514,7 @@ class GeminiClient(GenAIClient):
)
else: # user
gemini_messages.append(
types.Content(
role="user", parts=[types.Part.from_text(text=content)]
)
types.Content(role="user", parts=_parts_from_content(content))
)
# Convert tools to Gemini format
@@ -553,7 +580,7 @@ class GeminiClient(GenAIClient):
# Use streaming API
content_parts: list[str] = []
reasoning_parts: list[str] = []
tool_calls_by_index: dict[int, dict[str, Any]] = {}
tool_calls_accum: list[dict[str, Any]] = []
finish_reason = "stop"
usage_stats: dict[str, Any] | None = None
@@ -600,7 +627,11 @@ class GeminiClient(GenAIClient):
content_parts.append(part.text)
yield ("content_delta", part.text)
elif part.function_call:
# Handle function call
# Gemini streams complete function calls (not partial
# argument deltas), so each part is a distinct tool
# call. Append rather than accumulate by name — the
# latter concatenated parallel/repeated calls into one
# invalid arguments string (e.g. `{...}{...}`).
try:
arguments = (
dict(part.function_call.args)
@@ -610,40 +641,16 @@ class GeminiClient(GenAIClient):
except Exception:
arguments = {}
# Store tool call
tool_call_id = part.function_call.name or ""
tool_call_name = part.function_call.name or ""
# Check if we already have this tool call
found_index = None
for idx, tc in tool_calls_by_index.items():
if tc["name"] == tool_call_name:
found_index = idx
break
if found_index is None:
found_index = len(tool_calls_by_index)
tool_calls_by_index[found_index] = {
"id": tool_call_id,
"name": tool_call_name,
"arguments": "",
"thought_signature": None,
tool_calls_accum.append(
{
"id": part.function_call.name or "",
"name": part.function_call.name or "",
"arguments": arguments,
"thought_signature": getattr(
part, "thought_signature", None
),
}
# Accumulate arguments
if arguments:
tool_calls_by_index[found_index]["arguments"] += (
json.dumps(arguments)
if isinstance(arguments, dict)
else str(arguments)
)
# Capture latest thought_signature for this call
chunk_sig = getattr(part, "thought_signature", None)
if chunk_sig:
tool_calls_by_index[found_index][
"thought_signature"
] = chunk_sig
)
# Build final message
full_content = "".join(content_parts).strip() or None
@@ -651,25 +658,20 @@ class GeminiClient(GenAIClient):
# Convert tool calls to list format
tool_calls_list = None
if tool_calls_by_index:
tool_calls_list = []
for tc in tool_calls_by_index.values():
try:
# Try to parse accumulated arguments as JSON
parsed_args = json.loads(tc["arguments"])
except (json.JSONDecodeError, Exception):
parsed_args = tc["arguments"]
tool_calls_list.append(
{
"id": tc["id"],
"name": tc["name"],
"arguments": parsed_args,
"thought_signature": _encode_thought_signature(
tc.get("thought_signature")
),
}
)
if tool_calls_accum:
tool_calls_list = [
{
"id": tc["id"],
"name": tc["name"],
"arguments": tc["arguments"]
if isinstance(tc["arguments"], dict)
else {},
"thought_signature": _encode_thought_signature(
tc.get("thought_signature")
),
}
for tc in tool_calls_accum
]
finish_reason = "tool_calls"
if usage_stats is not None:
+5
View File
@@ -128,6 +128,11 @@ class LlamaCppClient(GenAIClient):
_text_baseline_tokens: int | None
_media_marker: str
@property
def supports_embeddings(self) -> bool:
"""llama.cpp exposes an /embeddings endpoint for any loaded model."""
return True
def _init_provider(self) -> str | None:
"""Initialize the client and query model metadata from the server."""
self.provider_options = {
+12 -3
View File
@@ -423,9 +423,18 @@ class OpenAIClient(GenAIClient):
for tc in tool_calls_by_index.values():
try:
# Parse accumulated arguments as JSON
parsed_args = json.loads(tc["arguments"])
except (json.JSONDecodeError, Exception):
parsed_args = tc["arguments"]
parsed_args = json.loads(tc["arguments"] or "{}")
except (json.JSONDecodeError, ValueError):
logger.warning(
"Failed to parse streamed tool call arguments for %s",
tc["name"],
)
parsed_args = {}
# Downstream (ToolCall model) requires a dict; never leak a
# partial/invalid arguments string.
if not isinstance(parsed_args, dict):
parsed_args = {}
tool_calls_list.append(
{
+106
View File
@@ -0,0 +1,106 @@
"""Tests that disabled custom classification models are not registered or run."""
import sys
import unittest
from unittest.mock import MagicMock
# Mock TFLite before importing the maintainer / classification modules
_MOCK_MODULES = [
"tflite_runtime",
"tflite_runtime.interpreter",
"ai_edge_litert",
"ai_edge_litert.interpreter",
]
for mod in _MOCK_MODULES:
if mod not in sys.modules:
sys.modules[mod] = MagicMock()
from frigate.data_processing.real_time.custom_classification import ( # noqa: E402
CustomObjectClassificationProcessor,
)
from frigate.embeddings.maintainer import EmbeddingMaintainer # noqa: E402
class TestCustomClassificationEnabledGating(unittest.TestCase):
"""A model with enabled: false must not keep a processor registered."""
def _make_maintainer(self) -> EmbeddingMaintainer:
# Bypass the heavy __init__; only the attributes touched by the
# config update path are needed for these tests.
maintainer = EmbeddingMaintainer.__new__(EmbeddingMaintainer)
maintainer.realtime_processors = []
maintainer.config = MagicMock()
maintainer.config.classification.custom = {}
maintainer.requestor = MagicMock()
maintainer.metrics = MagicMock()
maintainer.event_metadata_publisher = MagicMock()
return maintainer
def _make_model_config(self, name: str, enabled: bool) -> MagicMock:
model_config = MagicMock()
model_config.name = name
model_config.enabled = enabled
model_config.state_config = None
return model_config
def _make_processor(self, name: str) -> MagicMock:
processor = MagicMock(spec=CustomObjectClassificationProcessor)
processor.model_config = MagicMock()
processor.model_config.name = name
return processor
def test_disabled_update_tears_down_existing_processor(self):
"""Toggling a running model to disabled shuts down and drops its processor."""
maintainer = self._make_maintainer()
processor = self._make_processor("atli")
maintainer.realtime_processors = [processor]
maintainer._handle_custom_classification_update(
"config/classification/custom/atli",
self._make_model_config("atli", enabled=False),
)
processor.shutdown.assert_called_once()
self.assertEqual(maintainer.realtime_processors, [])
def test_disabled_update_does_not_register_processor(self):
"""A disabled model that has no processor is never registered."""
maintainer = self._make_maintainer()
maintainer._handle_custom_classification_update(
"config/classification/custom/atli",
self._make_model_config("atli", enabled=False),
)
self.assertEqual(maintainer.realtime_processors, [])
def test_disabled_update_leaves_other_processors_untouched(self):
"""Disabling one model must not affect other running processors."""
maintainer = self._make_maintainer()
other = self._make_processor("simbi")
maintainer.realtime_processors = [other]
maintainer._handle_custom_classification_update(
"config/classification/custom/atli",
self._make_model_config("atli", enabled=False),
)
other.shutdown.assert_not_called()
self.assertEqual(maintainer.realtime_processors, [other])
def test_removed_model_tears_down_processor(self):
"""A None payload (model deleted) still shuts down its processor."""
maintainer = self._make_maintainer()
processor = self._make_processor("atli")
maintainer.realtime_processors = [processor]
maintainer._handle_custom_classification_update(
"config/classification/custom/atli", None
)
processor.shutdown.assert_called_once()
self.assertEqual(maintainer.realtime_processors, [])
if __name__ == "__main__":
unittest.main()
+496
View File
@@ -0,0 +1,496 @@
"""Smoke tests for GenAI chat providers.
Each provider's ``chat_with_tools_stream`` is driven with a canned "test
response" so the two conversion layers are exercised without any network:
1. Frigate (OpenAI-style) messages -> provider-native request format
2. provider-native response -> Frigate ``("kind", value)`` stream events
These guard against regressions such as tool-call arguments arriving as raw
strings instead of dicts (which crash the ``ToolCall`` model), and multimodal
user content (a list of text/image parts, as injected by ``get_live_context``)
crashing message conversion.
"""
import asyncio
import base64
import json
import unittest
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
from frigate.config import GenAIConfig, GenAIProviderEnum
from frigate.genai import PROVIDERS, load_providers
load_providers()
# A minimal but valid JPEG data URI, mirroring what get_live_context injects.
_TINY_JPEG = base64.b64encode(b"\xff\xd8\xff\xd9").decode("ascii")
_IMAGE_DATA_URI = f"data:image/jpeg;base64,{_TINY_JPEG}"
# Conversation ending in a multimodal user message (text + live image), the
# exact shape the chat endpoint builds after a get_live_context tool result.
MULTIMODAL_MESSAGES = [
{"role": "system", "content": "You are a test assistant."},
{"role": "user", "content": "what do you see on the front camera?"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_1",
"type": "function",
"function": {
"name": "get_live_context",
"arguments": json.dumps({"camera": "front"}),
},
}
],
},
{
"role": "tool",
"tool_call_id": "call_1",
"name": "get_live_context",
"content": json.dumps({"camera": "front"}),
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Here is the current live image from camera 'front'.",
},
{"type": "image_url", "image_url": {"url": _IMAGE_DATA_URI}},
],
},
]
SIMPLE_MESSAGES = [
{"role": "system", "content": "You are a test assistant."},
{"role": "user", "content": "hello"},
]
TOOLS = [
{
"type": "function",
"function": {
"name": "search_objects",
"description": "Search tracked objects",
"parameters": {
"type": "object",
"properties": {"label": {"type": "string"}},
},
},
}
]
def _make_client(provider: str, **cfg_overrides):
"""Build a provider client offline (no model validation, no network)."""
cfg = GenAIConfig(provider=provider, **cfg_overrides)
cls = PROVIDERS[GenAIProviderEnum(provider)]
return cls(cfg, timeout=5, validate_model=False)
def _collect(client, messages, tools=TOOLS):
"""Drain chat_with_tools_stream into a list of (kind, value) events."""
async def _run():
events = []
async for event in client.chat_with_tools_stream(
messages=messages, tools=tools, tool_choice="auto"
):
events.append(event)
return events
return asyncio.run(_run())
def _final_message(events) -> dict:
messages = [value for (kind, value) in events if kind == "message"]
assert messages, f"stream produced no final message: {events}"
return messages[-1]
def _assert_tool_args_are_dicts(final: dict) -> None:
"""Every returned tool call must expose arguments as a dict, never a string."""
for tool_call in final.get("tool_calls") or []:
assert isinstance(tool_call["arguments"], dict), (
f"tool call arguments must be a dict, got "
f"{type(tool_call['arguments']).__name__}: {tool_call['arguments']!r}"
)
# ---------------------------------------------------------------------------
# OpenAI
# ---------------------------------------------------------------------------
def _openai_tc(index, id=None, name=None, arguments=None):
return SimpleNamespace(
index=index,
id=id,
function=SimpleNamespace(name=name, arguments=arguments),
)
def _openai_chunk(content=None, tool_calls=None, finish_reason=None, usage=None):
delta = SimpleNamespace(
content=content,
tool_calls=tool_calls,
reasoning_content=None,
reasoning=None,
)
choice = SimpleNamespace(delta=delta, finish_reason=finish_reason)
return SimpleNamespace(choices=[choice], usage=usage)
class TestOpenAIProvider(unittest.TestCase):
def _client(self):
return _make_client(
"openai", model="gpt-4o", api_key="k", base_url="http://localhost:9999/v1"
)
def test_stream_tool_call_arguments_are_dict(self):
# Arguments arrive split across chunks, as the real API streams them.
chunks = [
_openai_chunk(
tool_calls=[
_openai_tc(0, id="c1", name="search_objects", arguments='{"label":')
]
),
_openai_chunk(tool_calls=[_openai_tc(0, arguments=' "person"}')]),
_openai_chunk(finish_reason="tool_calls"),
]
client = self._client()
client.provider.chat.completions.create = MagicMock(return_value=iter(chunks))
final = _final_message(_collect(client, SIMPLE_MESSAGES))
self.assertEqual(final["finish_reason"], "tool_calls")
self.assertEqual(len(final["tool_calls"]), 1)
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
def test_stream_content_response(self):
chunks = [
_openai_chunk(content="hel"),
_openai_chunk(content="lo"),
_openai_chunk(finish_reason="stop"),
]
client = self._client()
client.provider.chat.completions.create = MagicMock(return_value=iter(chunks))
events = _collect(client, SIMPLE_MESSAGES)
deltas = [v for (k, v) in events if k == "content_delta"]
self.assertEqual("".join(deltas), "hello")
self.assertEqual(_final_message(events)["content"], "hello")
def test_multimodal_message_does_not_crash(self):
client = self._client()
client.provider.chat.completions.create = MagicMock(
return_value=iter([_openai_chunk(content="ok", finish_reason="stop")])
)
# Passing the OpenAI-native multimodal list through must not raise.
final = _final_message(_collect(client, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
# ---------------------------------------------------------------------------
# Gemini
# ---------------------------------------------------------------------------
def _gemini_part(text=None, thought=False, function_call=None, thought_signature=None):
return SimpleNamespace(
text=text,
thought=thought,
function_call=function_call,
thought_signature=thought_signature,
)
def _gemini_chunk(parts, finish_reason=None, usage_metadata=None):
candidate = SimpleNamespace(
content=SimpleNamespace(parts=parts), finish_reason=finish_reason
)
return SimpleNamespace(candidates=[candidate], usage_metadata=usage_metadata)
def _gemini_stream(chunks):
async def _agen(*args, **kwargs):
for chunk in chunks:
yield chunk
return _agen
class TestGeminiProvider(unittest.TestCase):
def _client(self):
return _make_client("gemini", model="gemini-2.5-flash", api_key="k")
def _patch_stream(self, client, chunks):
client.provider = MagicMock()
client.provider.aio.models.generate_content_stream = AsyncMock(
side_effect=_gemini_stream(chunks)
)
def test_stream_parallel_tool_calls_stay_separate_dicts(self):
# Regression: Gemini streams complete function calls. Two calls to the
# same tool must NOT be merged into one concatenated arguments string.
from google.genai.types import FinishReason
chunks = [
_gemini_chunk(
parts=[
_gemini_part(
function_call=SimpleNamespace(
name="search_objects", args={"label": "person"}
)
),
_gemini_part(
function_call=SimpleNamespace(
name="search_objects", args={"limit": 1}
)
),
],
finish_reason=FinishReason.STOP,
),
]
client = self._client()
self._patch_stream(client, chunks)
final = _final_message(_collect(client, SIMPLE_MESSAGES))
self.assertEqual(final["finish_reason"], "tool_calls")
self.assertEqual(len(final["tool_calls"]), 2)
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
self.assertEqual(final["tool_calls"][1]["arguments"], {"limit": 1})
def test_stream_content_response(self):
from google.genai.types import FinishReason
chunks = [
_gemini_chunk(parts=[_gemini_part(text="hel")]),
_gemini_chunk(
parts=[_gemini_part(text="lo")], finish_reason=FinishReason.STOP
),
]
client = self._client()
self._patch_stream(client, chunks)
events = _collect(client, SIMPLE_MESSAGES)
deltas = [v for (k, v) in events if k == "content_delta"]
self.assertEqual("".join(deltas), "hello")
self.assertEqual(_final_message(events)["content"], "hello")
def test_multimodal_message_converts_without_crash(self):
# Regression: a user message with list content (text + image_url) used
# to be handed to Part.from_text(text=<list>) and raise ValidationError.
from google.genai.types import FinishReason
client = self._client()
self._patch_stream(
client,
[
_gemini_chunk(
parts=[_gemini_part(text="ok")], finish_reason=FinishReason.STOP
)
],
)
final = _final_message(_collect(client, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
# ---------------------------------------------------------------------------
# Ollama
# ---------------------------------------------------------------------------
class TestOllamaProvider(unittest.TestCase):
def _client(self):
return _make_client("ollama", model="llama3", base_url="http://localhost:9999")
def _run_with_response(self, client, response, messages):
# Ollama uses a non-streaming call when tools are present, via an
# internally-constructed async client.
fake_async = MagicMock()
fake_async.chat = AsyncMock(return_value=response)
with patch(
"frigate.genai.plugins.ollama.OllamaAsyncClient",
return_value=fake_async,
):
return _collect(client, messages)
def test_tool_call_arguments_are_dict(self):
response = {
"message": {
"content": "",
"tool_calls": [
{
"function": {
"name": "search_objects",
"arguments": {"label": "person"},
}
}
],
},
"done": True,
"done_reason": "stop",
"eval_count": 5,
"prompt_eval_count": 3,
"eval_duration": 1_000_000,
}
client = self._client()
final = _final_message(
self._run_with_response(client, response, SIMPLE_MESSAGES)
)
self.assertEqual(final["finish_reason"], "tool_calls")
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
def test_multimodal_message_normalizes_image(self):
# Ollama needs content as a string with images pulled into a separate
# field; the normalizer must extract both without crashing.
response = {
"message": {"content": "ok"},
"done": True,
"done_reason": "stop",
}
client = self._client()
final = _final_message(
self._run_with_response(client, response, MULTIMODAL_MESSAGES)
)
self.assertEqual(final["content"], "ok")
def test_normalize_multimodal_content(self):
from frigate.genai.plugins.ollama import _normalize_multimodal_content
text, images = _normalize_multimodal_content(MULTIMODAL_MESSAGES[-1]["content"])
self.assertIn("live image", text)
self.assertEqual(len(images), 1)
self.assertEqual(images[0], b"\xff\xd8\xff\xd9")
# ---------------------------------------------------------------------------
# llama.cpp
# ---------------------------------------------------------------------------
class _FakeStreamResponse:
def __init__(self, lines):
self._lines = lines
def raise_for_status(self):
return None
async def aiter_lines(self):
for line in self._lines:
yield line
class _FakeStreamCtx:
def __init__(self, lines):
self._resp = _FakeStreamResponse(lines)
async def __aenter__(self):
return self._resp
async def __aexit__(self, *exc):
return False
class _FakeAsyncClient:
def __init__(self, lines):
self._lines = lines
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return False
def stream(self, method, url, json=None):
return _FakeStreamCtx(self._lines)
class TestLlamaCppProvider(unittest.TestCase):
def _client(self):
return _make_client("llamacpp", model="m", base_url="http://localhost:9999")
def _run_with_lines(self, client, lines, messages):
with patch(
"frigate.genai.plugins.llama_cpp.httpx.AsyncClient",
return_value=_FakeAsyncClient(lines),
):
return _collect(client, messages)
def test_stream_tool_call_arguments_are_dict(self):
lines = [
"data: "
+ json.dumps(
{
"choices": [
{
"delta": {
"tool_calls": [
{
"index": 0,
"id": "c1",
"function": {
"name": "search_objects",
"arguments": '{"label":',
},
}
]
}
}
]
}
),
"data: "
+ json.dumps(
{
"choices": [
{
"delta": {
"tool_calls": [
{
"index": 0,
"function": {"arguments": ' "person"}'},
}
]
}
}
]
}
),
"data: "
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "tool_calls"}]}),
"data: [DONE]",
]
client = self._client()
final = _final_message(self._run_with_lines(client, lines, SIMPLE_MESSAGES))
self.assertEqual(final["finish_reason"], "tool_calls")
_assert_tool_args_are_dicts(final)
self.assertEqual(final["tool_calls"][0]["arguments"], {"label": "person"})
def test_stream_content_response(self):
lines = [
"data: " + json.dumps({"choices": [{"delta": {"content": "hel"}}]}),
"data: " + json.dumps({"choices": [{"delta": {"content": "lo"}}]}),
"data: "
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "stop"}]}),
"data: [DONE]",
]
client = self._client()
events = self._run_with_lines(client, lines, SIMPLE_MESSAGES)
deltas = [v for (k, v) in events if k == "content_delta"]
self.assertEqual("".join(deltas), "hello")
self.assertEqual(_final_message(events)["content"], "hello")
def test_multimodal_message_does_not_crash(self):
lines = [
"data: " + json.dumps({"choices": [{"delta": {"content": "ok"}}]}),
"data: "
+ json.dumps({"choices": [{"delta": {}, "finish_reason": "stop"}]}),
"data: [DONE]",
]
client = self._client()
final = _final_message(self._run_with_lines(client, lines, MULTIMODAL_MESSAGES))
self.assertEqual(final["content"], "ok")
if __name__ == "__main__":
unittest.main()
+12 -8
View File
@@ -39,7 +39,7 @@
"@rjsf/utils": "^6.4.1",
"@rjsf/validator-ajv8": "^6.4.1",
"apexcharts": "^3.52.0",
"axios": "^1.13.6",
"axios": "^1.15.2",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"cmdk": "^1.0.0",
@@ -6458,14 +6458,14 @@
}
},
"node_modules/axios": {
"version": "1.13.6",
"resolved": "https://registry.npmjs.org/axios/-/axios-1.13.6.tgz",
"integrity": "sha512-ChTCHMouEe2kn713WHbQGcuYrr6fXTBiu460OTwWrWob16g1bXn4vtz07Ope7ewMozJAnEquLk5lWQWtBig9DQ==",
"version": "1.15.2",
"resolved": "https://registry.npmjs.org/axios/-/axios-1.15.2.tgz",
"integrity": "sha512-wLrXxPtcrPTsNlJmKjkPnNPK2Ihe0hn0wGSaTEiHRPxwjvJwT3hKmXF4dpqxmPO9SoNb2FsYXj/xEo0gHN+D5A==",
"license": "MIT",
"dependencies": {
"follow-redirects": "^1.15.11",
"form-data": "^4.0.5",
"proxy-from-env": "^1.1.0"
"proxy-from-env": "^2.1.0"
}
},
"node_modules/bail": {
@@ -11922,9 +11922,13 @@
}
},
"node_modules/proxy-from-env": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
"integrity": "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg=="
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-2.1.0.tgz",
"integrity": "sha512-cJ+oHTW1VAEa8cJslgmUZrc+sjRKgAKl3Zyse6+PV38hZe/V6Z14TbCuXcan9F9ghlz4QrFr2c92TNF82UkYHA==",
"license": "MIT",
"engines": {
"node": ">=10"
}
},
"node_modules/psl": {
"version": "1.9.0",
+1 -1
View File
@@ -53,7 +53,7 @@
"@rjsf/utils": "^6.4.1",
"@rjsf/validator-ajv8": "^6.4.1",
"apexcharts": "^3.52.0",
"axios": "^1.13.6",
"axios": "^1.15.2",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"cmdk": "^1.0.0",
+1 -1
View File
@@ -123,7 +123,7 @@
"error": {
"failed": "Failed to queue export: {{error}}",
"endTimeMustAfterStartTime": "End time must be after start time",
"noVaildTimeSelected": "No valid time range selected"
"noValidTimeSelected": "No valid time range selected"
}
},
"fromTimeline": {
@@ -1,5 +1,6 @@
{
"documentTitle": "Classification Models - Frigate",
"disabled": "Disabled",
"details": {
"scoreInfo": "Score represents the average classification confidence across all detections of this object.",
"none": "None",
@@ -64,7 +65,20 @@
"title": "Edit Classification Model",
"descriptionState": "Edit the classes for this state classification model. Changes will require retraining the model.",
"descriptionObject": "Edit the object type and classification type for this object classification model.",
"stateClassesInfo": "Note: Changing state classes requires retraining the model with the updated classes."
"enabled": "Enabled",
"enabledDesc": "Run this model. When disabled, it stops running and no longer classifies.",
"saveAttempts": "Save Attempts",
"saveAttemptsDesc": "Number of classification attempt images to keep for the recent classifications UI.",
"motion": "Run on Motion",
"motionDesc": "Run classification when motion is detected within the configured crop.",
"interval": "Interval",
"intervalDesc": "Seconds between periodic classification runs. Leave empty to run only on motion.",
"intervalPlaceholder": "No interval",
"stateClassesInfo": "Model updated. Retrain the model for the class changes to take effect.",
"errors": {
"saveAttemptsInvalid": "Save attempts must be a whole number of 0 or greater",
"intervalInvalid": "Interval must be a whole number greater than 0"
}
},
"deleteDatasetImages": {
"title": "Delete Dataset Images",
+1 -1
View File
@@ -76,7 +76,7 @@
},
"offset": {
"label": "Annotation Offset",
"desc": "This data comes from your camera's detect feed but is overlayed on images from the the record feed. It is unlikely that the two streams are perfectly in sync. As a result, the bounding box and the footage will not line up perfectly. You can use this setting to offset the annotations forward or backward in time to better align them with the recorded footage.",
"desc": "This data comes from your camera's detect feed but is overlaid on images from the record feed. It is unlikely that the two streams are perfectly in sync. As a result, the bounding box and the footage will not line up perfectly. You can use this setting to offset the annotations forward or backward in time to better align them with the recorded footage.",
"millisecondsToOffset": "Milliseconds to offset detect annotations by. <em>Default: 0</em>",
"tips": "Lower the value if the video playback is ahead of the boxes and path points, and increase the value if the video playback is behind them. This value can be negative.",
"toast": {
+9 -2
View File
@@ -1053,7 +1053,7 @@
},
"createUser": {
"title": "Create New User",
"desc": "Add a new user account and specify an role for access to areas of the Frigate UI.",
"desc": "Add a new user account and specify a role for access to areas of the Frigate UI.",
"usernameOnlyInclude": "Username may only include letters, numbers, . or _",
"confirmPassword": "Please confirm your password"
},
@@ -1490,7 +1490,14 @@
"keyLabel": "Key",
"valueLabel": "Value",
"keyPlaceholder": "New key",
"remove": "Remove"
"remove": "Remove",
"providerNameLabel": "Provider name",
"providerNamePlaceholder": "e.g., openai",
"variableNameLabel": "Variable name",
"variableNamePlaceholder": "e.g., MY_VARIABLE",
"loggerNameLabel": "Logger name",
"loggerNamePlaceholder": "e.g., frigate.record",
"keyPatternError": "Use only letters, numbers, hyphens, and underscores (no spaces)"
},
"knownPlates": {
"namePlaceholder": "e.g., Wife's Car",
@@ -9,6 +9,7 @@ import {
import {
Form,
FormControl,
FormDescription,
FormField,
FormItem,
FormLabel,
@@ -17,6 +18,7 @@ import {
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group";
import { Switch } from "@/components/ui/switch";
import {
Select,
SelectContent,
@@ -50,14 +52,25 @@ type ClassificationModelEditDialogProps = {
type ObjectClassificationType = "sub_label" | "attribute";
type ObjectFormData = {
enabled: boolean;
saveAttempts: number;
objectLabel: string;
objectType: ObjectClassificationType;
};
type StateFormData = {
enabled: boolean;
saveAttempts: number;
motion: boolean;
interval?: number;
classes: string[];
};
const DEFAULT_SAVE_ATTEMPTS = {
object: 200,
state: 100,
} as const;
export default function ClassificationModelEditDialog({
open,
model,
@@ -71,6 +84,10 @@ export default function ClassificationModelEditDialog({
const isStateModel = model.state_config !== undefined;
const isObjectModel = model.object_config !== undefined;
const defaultSaveAttempts = isObjectModel
? DEFAULT_SAVE_ATTEMPTS.object
: DEFAULT_SAVE_ATTEMPTS.state;
const objectLabels = useMemo(() => {
if (!config) return [];
@@ -93,8 +110,17 @@ export default function ClassificationModelEditDialog({
// Define form schema based on model type
const formSchema = useMemo(() => {
const sharedFields = {
enabled: z.boolean(),
saveAttempts: z.coerce
.number({ message: t("edit.errors.saveAttemptsInvalid") })
.int(t("edit.errors.saveAttemptsInvalid"))
.min(0, t("edit.errors.saveAttemptsInvalid")),
};
if (isObjectModel) {
return z.object({
...sharedFields,
objectLabel: z
.string()
.min(1, t("wizard.step1.errors.objectLabelRequired")),
@@ -103,6 +129,17 @@ export default function ClassificationModelEditDialog({
} else {
// State model
return z.object({
...sharedFields,
motion: z.boolean(),
interval: z.preprocess(
(val) =>
val === "" || val === null || val === undefined ? undefined : val,
z.coerce
.number({ message: t("edit.errors.intervalInvalid") })
.int(t("edit.errors.intervalInvalid"))
.positive(t("edit.errors.intervalInvalid"))
.optional(),
),
classes: z
.array(z.string())
.min(1, t("wizard.step1.errors.classRequired"))
@@ -129,12 +166,18 @@ export default function ClassificationModelEditDialog({
resolver: zodResolver(formSchema),
defaultValues: isObjectModel
? ({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
objectLabel: model.object_config?.objects?.[0] || "",
objectType:
(model.object_config
?.classification_type as ObjectClassificationType) || "sub_label",
} as ObjectFormData)
: ({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
motion: model.state_config?.motion ?? false,
interval: model.state_config?.interval,
classes: [""], // Will be populated from dataset
} as StateFormData),
mode: "onChange",
@@ -151,6 +194,8 @@ export default function ClassificationModelEditDialog({
if (open) {
if (isObjectModel) {
form.reset({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
objectLabel: model.object_config?.objects?.[0] || "",
objectType:
(model.object_config
@@ -158,6 +203,10 @@ export default function ClassificationModelEditDialog({
} as ObjectFormData);
} else {
form.reset({
enabled: model.enabled,
saveAttempts: model.save_attempts ?? defaultSaveAttempts,
motion: model.state_config?.motion ?? false,
interval: model.state_config?.interval,
classes: [""],
} as StateFormData);
}
@@ -166,7 +215,15 @@ export default function ClassificationModelEditDialog({
mutateDataset();
}
}
}, [open, isObjectModel, isStateModel, model, form, mutateDataset]);
}, [
open,
isObjectModel,
isStateModel,
model,
form,
mutateDataset,
defaultSaveAttempts,
]);
// Update form with classes from dataset when loaded
useEffect(() => {
@@ -233,6 +290,7 @@ export default function ClassificationModelEditDialog({
setIsSaving(true);
try {
if (isObjectModel) {
// object model save
const objectData = data as ObjectFormData;
// Update the config
@@ -243,9 +301,10 @@ export default function ClassificationModelEditDialog({
classification: {
custom: {
[model.name]: {
enabled: model.enabled,
enabled: objectData.enabled,
name: model.name,
threshold: model.threshold,
save_attempts: objectData.saveAttempts,
object_config: {
objects: [objectData.objectLabel],
classification_type: objectData.objectType,
@@ -260,7 +319,34 @@ export default function ClassificationModelEditDialog({
position: "top-center",
});
} else {
// state model save
const stateData = data as StateFormData;
const stateConfig: { motion: boolean; interval?: number | null } = {
motion: stateData.motion,
};
if (stateData.interval != null) {
stateConfig.interval = stateData.interval;
} else if (model.state_config?.interval != null) {
stateConfig.interval = null;
}
await axios.put("/config/set", {
requires_restart: 0,
update_topic: `config/classification/custom/${model.name}`,
config_data: {
classification: {
custom: {
[model.name]: {
enabled: stateData.enabled,
save_attempts: stateData.saveAttempts,
state_config: stateConfig,
},
},
},
},
});
const newClasses = stateData.classes.filter(
(c) => c.trim().length > 0,
);
@@ -307,11 +393,11 @@ export default function ClassificationModelEditDialog({
if (renamePromises.length > 0) {
await Promise.all(renamePromises);
await mutate(`classification/${model.name}/dataset`);
toast.success(t("toast.success.updatedModel"), {
toast.success(t("edit.stateClassesInfo"), {
position: "top-center",
});
} else {
toast.info(t("edit.stateClassesInfo"), {
toast.success(t("toast.success.updatedModel"), {
position: "top-center",
});
}
@@ -359,6 +445,29 @@ export default function ClassificationModelEditDialog({
<div className="space-y-6">
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
<FormField
control={form.control}
name="enabled"
render={({ field }) => (
<FormItem className="flex flex-row items-center justify-between gap-4">
<div className="space-y-0.5">
<FormLabel className="text-primary-variant">
{t("edit.enabled")}
</FormLabel>
<FormDescription className="text-xs">
{t("edit.enabledDesc")}
</FormDescription>
</div>
<FormControl>
<Switch
checked={field.value}
onCheckedChange={field.onChange}
/>
</FormControl>
</FormItem>
)}
/>
{isObjectModel && (
<>
<FormField
@@ -520,6 +629,77 @@ export default function ClassificationModelEditDialog({
</div>
)}
{isStateModel && (
<>
<FormField
control={form.control}
name="motion"
render={({ field }) => (
<FormItem className="flex flex-row items-center justify-between gap-4">
<div className="space-y-0.5">
<FormLabel className="text-primary-variant">
{t("edit.motion")}
</FormLabel>
<FormDescription className="text-xs">
{t("edit.motionDesc")}
</FormDescription>
</div>
<FormControl>
<Switch
checked={field.value}
onCheckedChange={field.onChange}
/>
</FormControl>
</FormItem>
)}
/>
<FormField
control={form.control}
name="interval"
render={({ field }) => (
<FormItem>
<FormLabel className="text-primary-variant">
{t("edit.interval")}
</FormLabel>
<FormControl>
<Input
className="h-8"
inputMode="numeric"
placeholder={t("edit.intervalPlaceholder")}
{...field}
value={field.value ?? ""}
/>
</FormControl>
<FormDescription className="text-xs">
{t("edit.intervalDesc")}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
</>
)}
<FormField
control={form.control}
name="saveAttempts"
render={({ field }) => (
<FormItem>
<FormLabel className="text-primary-variant">
{t("edit.saveAttempts")}
</FormLabel>
<FormControl>
<Input className="h-8" inputMode="numeric" {...field} />
</FormControl>
<FormDescription className="text-xs">
{t("edit.saveAttemptsDesc")}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
<div className="flex flex-col gap-3 pt-3 sm:flex-row sm:justify-end sm:gap-4">
<Button
type="button"
@@ -7,7 +7,13 @@ const environmentVars: SectionConfigOverrides = {
advancedFields: [],
uiSchema: {
additionalProperties: {
"ui:options": { size: "lg" },
"ui:options": {
size: "lg",
additionalPropertyKeyLabel:
"configForm.additionalProperties.variableNameLabel",
additionalPropertyKeyPlaceholder:
"configForm.additionalProperties.variableNamePlaceholder",
},
},
},
},
@@ -9,7 +9,15 @@ const genai: SectionConfigOverrides = {
uiSchema: {
"ui:options": { disableNestedCard: true },
"*": {
"ui:options": { disableNestedCard: true },
"ui:options": {
disableNestedCard: true,
additionalPropertyKeyLabel:
"configForm.additionalProperties.providerNameLabel",
additionalPropertyKeyPlaceholder:
"configForm.additionalProperties.providerNamePlaceholder",
additionalPropertyKeyPattern: "^[a-zA-Z0-9_-]+$",
preventKeyRename: true,
},
"ui:order": [
"provider",
"api_key",
@@ -12,7 +12,13 @@ const logger: SectionConfigOverrides = {
},
logs: {
additionalProperties: {
"ui:options": { enumI18nPrefix: "logger.logLevel" },
"ui:options": {
enumI18nPrefix: "logger.logLevel",
additionalPropertyKeyLabel:
"configForm.additionalProperties.loggerNameLabel",
additionalPropertyKeyPlaceholder:
"configForm.additionalProperties.loggerNamePlaceholder",
},
},
},
},
@@ -6,12 +6,14 @@ import {
StrictRJSFSchema,
WrapIfAdditionalTemplateProps,
} from "@rjsf/utils";
import { useEffect, useMemo, useState, type FocusEvent } from "react";
import { Input } from "@/components/ui/input";
import { Button } from "@/components/ui/button";
import { Label } from "@/components/ui/label";
import { cn } from "@/lib/utils";
import { useTranslation } from "react-i18next";
import { LuTrash2 } from "react-icons/lu";
import type { ConfigFormContext } from "@/types/configForm";
export function WrapIfAdditionalTemplate<
T = unknown,
@@ -30,6 +32,7 @@ export function WrapIfAdditionalTemplate<
onKeyRenameBlur,
readonly,
required,
registry,
schema,
uiSchema,
} = props;
@@ -38,6 +41,55 @@ export function WrapIfAdditionalTemplate<
const additional = ADDITIONAL_PROPERTY_FLAG in schema;
const uiOptions = getUiOptions(uiSchema);
const keyIsReadonly = uiOptions.additionalPropertyKeyReadonly === true;
const keyLabelKey =
typeof uiOptions.additionalPropertyKeyLabel === "string"
? uiOptions.additionalPropertyKeyLabel
: undefined;
const keyPlaceholderKey =
typeof uiOptions.additionalPropertyKeyPlaceholder === "string"
? uiOptions.additionalPropertyKeyPlaceholder
: undefined;
const keyPattern =
typeof uiOptions.additionalPropertyKeyPattern === "string"
? uiOptions.additionalPropertyKeyPattern
: undefined;
const preventKeyRename = uiOptions.preventKeyRename === true;
const formContext = registry?.formContext as ConfigFormContext | undefined;
// optionally, lock the key once it's been saved
const baseline = formContext?.baselineFormData;
const keyLocked =
preventKeyRename &&
typeof label === "string" &&
!!baseline &&
Object.prototype.hasOwnProperty.call(baseline, label);
// controlled key value so we can validate live and block invalid renames.
const [keyValue, setKeyValue] = useState<string>(label ?? "");
useEffect(() => {
setKeyValue(label ?? "");
}, [label]);
const keyRegex = useMemo(
() => (keyPattern ? new RegExp(keyPattern) : undefined),
[keyPattern],
);
const keyError = useMemo(() => {
if (!keyRegex || keyLocked) return null;
if (!keyRegex.test(keyValue)) {
return t("configForm.additionalProperties.keyPatternError", {
ns: "views/settings",
defaultValue:
"Use only letters, numbers, hyphens, and underscores (no spaces)",
});
}
return null;
}, [keyRegex, keyLocked, keyValue, t]);
if (!additional) {
return (
<div className={classNames} style={style}>
@@ -47,20 +99,26 @@ export function WrapIfAdditionalTemplate<
}
const keyId = `${id}-key`;
const keyLabel = t("configForm.additionalProperties.keyLabel", {
ns: "views/settings",
});
const keyLabel = keyLabelKey
? t(keyLabelKey, { ns: "views/settings" })
: t("configForm.additionalProperties.keyLabel", { ns: "views/settings" });
const valueLabel = t("configForm.additionalProperties.valueLabel", {
ns: "views/settings",
});
const keyPlaceholder = t("configForm.additionalProperties.keyPlaceholder", {
ns: "views/settings",
});
const keyPlaceholder = keyPlaceholderKey
? t(keyPlaceholderKey, { ns: "views/settings" })
: t("configForm.additionalProperties.keyPlaceholder", {
ns: "views/settings",
});
const removeLabel = t("configForm.additionalProperties.remove", {
ns: "views/settings",
});
const uiOptions = getUiOptions(uiSchema);
const keyIsReadonly = uiOptions.additionalPropertyKeyReadonly === true;
const commitKeyRename = (e: FocusEvent<HTMLInputElement>) => {
if (readonly) return;
if (keyError) return;
onKeyRenameBlur?.(e);
};
return (
<div
@@ -70,23 +128,30 @@ export function WrapIfAdditionalTemplate<
{!keyIsReadonly && (
<div className="col-span-12 space-y-2 md:col-span-2">
{displayLabel && <Label htmlFor={keyId}>{keyLabel}</Label>}
{keyIsReadonly ? (
{keyLocked ? (
<div
id={keyId}
className="flex items-center text-sm text-muted-foreground"
className="flex items-center break-all text-sm text-primary-variant"
>
{label}
</div>
) : (
<Input
id={keyId}
name={keyId}
required={required}
defaultValue={label}
placeholder={keyPlaceholder}
disabled={disabled || readonly}
onBlur={!readonly ? onKeyRenameBlur : undefined}
/>
<>
<Input
id={keyId}
name={keyId}
required={required}
value={keyValue}
placeholder={keyPlaceholder}
disabled={disabled || readonly}
onChange={(e) => setKeyValue(e.target.value)}
onBlur={!readonly ? commitKeyRename : undefined}
aria-invalid={keyError ? true : undefined}
/>
{keyError && (
<p className="text-xs text-destructive">{keyError}</p>
)}
</>
)}
</div>
)}
@@ -1,8 +1,10 @@
import type { WidgetProps } from "@rjsf/utils";
import { useMemo } from "react";
import { useEffect, useMemo } from "react";
import { useTranslation } from "react-i18next";
import useSWR from "swr";
import { Switch } from "@/components/ui/switch";
import type { ConfigFormContext } from "@/types/configForm";
import type { GenAIModelsResponse } from "@/types/chat";
const GENAI_ROLES = ["embeddings", "descriptions", "chat"] as const;
@@ -37,10 +39,24 @@ export function GenAIRolesWidget(props: WidgetProps) {
const selectedRoles = useMemo(() => normalizeValue(value), [value]);
const providerKey = useMemo(() => getProviderKey(id), [id]);
// Compute occupied roles directly from formData. The computation is
// trivially cheap (iterate providers × 3 roles max) so we skip an
// intermediate memoization layer whose formData dependency would
// never produce a cache hit (new object reference on every change).
const { data: genaiInfo } = useSWR<GenAIModelsResponse>("genai/models", {
revalidateOnFocus: false,
});
const embeddingsSupported = useMemo(() => {
if (!providerKey) return true;
const info = genaiInfo?.[providerKey];
return info ? info.supports_embeddings : true;
}, [genaiInfo, providerKey]);
const availableRoles = useMemo(
() =>
embeddingsSupported
? GENAI_ROLES
: GENAI_ROLES.filter((role) => role !== "embeddings"),
[embeddingsSupported],
);
const occupiedRoles = useMemo(() => {
const occupied = new Set<string>();
const fd = formContext?.formData;
@@ -64,6 +80,12 @@ export function GenAIRolesWidget(props: WidgetProps) {
return occupied;
}, [formContext?.formData, providerKey]);
useEffect(() => {
if (!embeddingsSupported && selectedRoles.includes("embeddings")) {
onChange(selectedRoles.filter((role) => role !== "embeddings"));
}
}, [embeddingsSupported, selectedRoles, onChange]);
const toggleRole = (role: string, enabled: boolean) => {
if (enabled) {
if (!selectedRoles.includes(role)) {
@@ -78,7 +100,7 @@ export function GenAIRolesWidget(props: WidgetProps) {
return (
<div className="rounded-lg border border-secondary-highlight bg-background_alt p-2 pr-0 md:max-w-md">
<div className="grid gap-2">
{GENAI_ROLES.map((role) => {
{availableRoles.map((role) => {
const checked = selectedRoles.includes(role);
const roleDisabled = !checked && occupiedRoles.has(role);
const label = t(`configForm.genaiRoles.options.${role}`, {
+2 -2
View File
@@ -134,7 +134,7 @@ export default function ExportDialog({
}
if (!range) {
toast.error(t("export.toast.error.noVaildTimeSelected"), {
toast.error(t("export.toast.error.noValidTimeSelected"), {
position: "top-center",
});
return false;
@@ -665,7 +665,7 @@ export function ExportContent({
}
if (!range) {
toast.error(t("export.toast.error.noVaildTimeSelected"), {
toast.error(t("export.toast.error.noValidTimeSelected"), {
position: "top-center",
});
return;
@@ -149,7 +149,7 @@ export default function MobileReviewSettingsDrawer({
if (!range) {
toast.error(
t("export.toast.error.noVaildTimeSelected", {
t("export.toast.error.noValidTimeSelected", {
ns: "components/dialog",
}),
{
+1
View File
@@ -43,6 +43,7 @@ export type GenAIProviderInfo = {
models: string[];
roles: string[];
supports_toggleable_thinking: boolean;
supports_embeddings: boolean;
};
export type GenAIModelsResponse = Record<string, GenAIProviderInfo>;
+1
View File
@@ -370,6 +370,7 @@ export type CustomClassificationModelConfig = {
};
};
motion: boolean;
interval?: number;
};
};
@@ -3,6 +3,7 @@ import ClassificationModelWizardDialog from "@/components/classification/Classif
import ClassificationModelEditDialog from "@/components/classification/ClassificationModelEditDialog";
import ActivityIndicator from "@/components/indicators/activity-indicator";
import { ImageShadowOverlay } from "@/components/overlay/ImageShadowOverlay";
import { Badge } from "@/components/ui/badge";
import { Button, buttonVariants } from "@/components/ui/button";
import { ToggleGroup, ToggleGroupItem } from "@/components/ui/toggle-group";
import useOptimisticState from "@/hooks/use-optimistic-state";
@@ -330,7 +331,10 @@ function ModelCard({ config, onClick, onUpdate, onDelete }: ModelCardProps) {
{coverImage ? (
<>
<img
className="size-full"
className={cn(
"size-full",
!config.enabled && "opacity-50 grayscale",
)}
src={`${baseUrl}clips/${config.name}/dataset/${coverImage.name}/${coverImage.img}`}
/>
<ImageShadowOverlay lowerClassName="h-[30%] z-0" />
@@ -338,6 +342,14 @@ function ModelCard({ config, onClick, onUpdate, onDelete }: ModelCardProps) {
) : (
<Skeleton className="flex size-full items-center justify-center" />
)}
{!config.enabled && (
<Badge
variant="secondary"
className="absolute right-2 top-2 z-40 text-primary-variant"
>
{t("disabled")}
</Badge>
)}
<div className="absolute bottom-2 left-3 text-lg text-white smart-capitalize">
{config.name}
</div>
@@ -348,6 +348,19 @@ function MotionPreviewClip({
}
}, [clipStart, clipEnd, playbackRate, preview]);
useEffect(() => {
if (!videoRef.current || !preview || !videoPlaying) {
return;
}
if (isSafari || (isFirefox && isMobile)) {
// These browsers step frames manually; rebuild the interval at the new rate
resetPlayback();
} else {
videoRef.current.playbackRate = playbackRate;
}
}, [playbackRate, preview, videoPlaying, resetPlayback]);
const drawDimOverlay = useCallback(() => {
if (!dimOverlayCanvasRef.current) {
return;
@@ -305,7 +305,7 @@ export default function MotionSearchView({
const handleExportPreview = useCallback(() => {
if (!exportRange) {
toast.error(
t("export.toast.error.noVaildTimeSelected", {
t("export.toast.error.noValidTimeSelected", {
ns: "components/dialog",
}),
{
@@ -351,7 +351,7 @@ export default function MotionSearchView({
const handleExportSave = useCallback(() => {
if (!exportRange || !selectedCamera) {
toast.error(
t("export.toast.error.noVaildTimeSelected", {
t("export.toast.error.noValidTimeSelected", {
ns: "components/dialog",
}),
{