Compare commits

...
Author SHA1 Message Date
Josh Hawkins 332adaa717 remove all references to degirum in frigate
the company ceased operations on 1 Aug 2026
2026-08-01 07:32:45 -05:00
Josh HawkinsandGitHub b848c90f02 Fix wrong box format passed to cv2.dnn.NMSBoxes (#23876)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
2026-07-31 08:57:23 -05:00
Josh HawkinsandGitHub f1cc0e49d4 Miscellaneous fixes (0.18 beta) (#23873)
CI / AMD64 Build (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s
* improve display of gpu graphs in system metrics

* docs tweaks

* Only hide cameras with ui.dashboard disabled from the All Cameras dashboard

The settings camera selector and zone editor also filtered on ui.dashboard, so hiding a camera from the dashboard made its zones and masks uneditable in the UI (GH 23870). Drop those filters and correct the field title, help text, and reference docs to describe what the option actually does

* hide cameras with ui.review disabled from the Motion tab and the review summaries

The Motion tab built its own camera list that never checked ui.review, so a hidden camera still got a preview tile, and its motion and overlap queries fell back to every allowed camera. The review and recordings summaries had the same gap: they are aggregate day counts that can't be filtered client side, so a hidden camera kept contributing to the severity tab counts and calendar indicators while its items were absent from the list. Filter the motion camera list on ui.review and query all four endpoints with the visible camera list instead of letting the backend default to all, and skip the summary queries until the config resolves so the counts don't briefly render as zero.

* Scope every review page query to the cameras visible in review

The segments and the summary counts were derived from different camera sets: the list was fetched for all cameras and filtered client side, while the summaries were fetched for the visible cameras only when no explicit camera filter was set. A ?cameras= link can name a camera hidden from review, which left the count above zero with an empty list, pinning the new items to review popover open and making the auto refresh effect loop. Intersect an explicit camera selection with the visible list rather than trusting it, pass that to the segment and summary queries alike, and drop the now redundant client side filter, which the raw segments handed to the history view were bypassing anyway.
2026-07-30 17:20:41 -05:00
Josh HawkinsandGitHub 7ed7ed56cf Miscellaneous fixes (0.18 beta) (#23854)
CI / AMD64 Build (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s
* fix watchdog process restarts reverting to the boot config

/api/config/set parses a new FrigateConfig and swaps the API and dispatcher onto it, but FrigateApp.config was never rebound, so the watchdog factories rebuilt a crashed process from the config as of startup. Fix is to read through a ConfigHolder that the swap updates.

* fix birdseye camera overrides being clobbered by a global mode change

A global birdseye save published only the global object, leaving the output process to infer which cameras were inheriting by comparing against the previous global mode. That cannot tell an inherited value from an explicit one that happens to match, so it overwrote the override until a restart. Publish the per-camera values the config parse already resolved instead.

* Reject non-finite numbers in GenAI review descriptions

A model returning NaN for confidence or potential_threat_level slipped through the model_construct fallback, which skips validation, and was written into the review segment's JSON data. NaN is not valid JSON, so every subsequent /review request failed with "Out of range float values are not JSON compliant", blanking the review page for any time range containing the poisoned row.

* restore fused DetectionOutput in the OpenVINO SSD model conversion

* fix rgb swap issue for face dataset testing script
2026-07-29 08:39:01 -06:00
860772f9f4 Miscellaneous fixes (0.18 beta) (#23828)
CI / AMD64 Build (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s
* widen the logger name field in the per-process log level settings

* add details to timestamp error faq

* tweak genai docs

* tweak vector language

* Combine Qwen3.5 and Qwen3.6 listings

* fix openvino yolox detector crashing on every detection

The intermediate (N, 7) array in the yolox branch shadowed the pre-allocated (20, 6) detections buffer, so writing a detection into it raised "could not broadcast input array from shape (6,) into shape (7,)" on the first frame with anything above the confidence threshold. An empty frame also returned a (0, 7) array instead of the (20, 6) buffer.

Regressed in #13794, which renamed the intermediate from dets to detections as part of a cspell cleanup. Broken since 0.15.0.

---------

Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
2026-07-28 11:02:15 -06:00
40 changed files with 895 additions and 466 deletions
+83 -19
View File
@@ -1,10 +1,14 @@
"""Convert the default SSDLite MobileNet v2 model to OpenVINO IR.
Replaces the legacy openvino-dev Model Optimizer conversion. The TensorFlow
frontend converts the Object Detection API frozen graph natively; the four TF
outputs are then repacked into the single [1, 1, 100, 7] DetectionOutput-style
tensor that Frigate's OpenVINO detector expects, and the input is flipped to
BGR to match the legacy reverse_input_channels behavior.
frontend translates the Object Detection API pre and post processors literally,
producing per-class NonMaxSuppression, NonZero ops and map loops with data
dependent shapes that the GPU plugin handles very badly. Both are cut out the
way ssd_v2_support.json used to do it: the preprocessor is an identity at the
native 300x300 input, and the postprocessor becomes a single fused
DetectionOutput. The result is the [1, 1, 100, 7] tensor that Frigate's
OpenVINO detector expects, with the input flipped to BGR to match the legacy
reverse_input_channels behavior.
"""
import numpy as np
@@ -12,31 +16,91 @@ import openvino as ov
from openvino import opset8 as ops
from openvino.preprocess import PrePostProcessor
MODEL_DIR = "/models/ssdlite_mobilenet_v2_coco_2018_05_09"
OUTPUT_PATH = "/models/ssdlite_mobilenet_v2.xml"
INPUT_SHAPE = [1, 300, 300, 3]
# faster_rcnn_box_coder divides the deltas by pipeline.config's y/x/height/width
# scales of 10/10/5/5, which DetectionOutput expresses as per-prior variances.
BOX_VARIANCES = np.float32([0.1, 0.1, 0.2, 0.2])
model = ov.convert_model(
"/models/ssdlite_mobilenet_v2_coco_2018_05_09/frozen_inference_graph.pb",
input=[("image_tensor:0", [1, 300, 300, 3])],
f"{MODEL_DIR}/frozen_inference_graph.pb",
input=[("image_tensor:0", INPUT_SHAPE)],
)
# rows of (image_id, class_id, score, xmin, ymin, xmax, ymax)
boxes = model.output("detection_boxes:0").get_node().input_value(0)
classes = model.output("detection_classes:0").get_node().input_value(0)
scores = model.output("detection_scores:0").get_node().input_value(0)
nodes = {op.get_friendly_name(): op for op in model.get_ordered_ops()}
parameter = model.get_parameters()[0]
# (ymin,xmin,ymax,xmax) -> (xmin,ymin,xmax,ymax)
boxes = ops.gather(boxes, [1, 0, 3, 2], 2)
classes = ops.unsqueeze(classes, 2)
scores = ops.unsqueeze(scores, 2)
image_id = ops.multiply(scores, np.float32(0.0))
preprocessor = nodes["Preprocessor/map/TensorArrayStack/TensorArrayGatherV3"]
box_deltas = nodes["Postprocessor/Reshape_1"].output(0)
class_scores = nodes["Postprocessor/convert_scores"].output(0)
anchors_output = nodes["Postprocessor/Reshape"].output(0)
detections = ops.concat([image_id, classes, scores, boxes], 2)
detections = ops.unsqueeze(detections, 1)
# The anchors only depend on the static input shape, so fold them into a
# constant and drop the generator subgraph with the rest of the postprocessor.
probe = ov.Core().compile_model(
ov.Model([anchors_output, preprocessor.output(0)], [parameter], "probe"), "CPU"
)
probe_input = np.random.default_rng(0).integers(0, 255, INPUT_SHAPE, dtype=np.uint8)
anchors, resized = (out.copy() for out in probe([probe_input]).values())
assert np.allclose(resized, probe_input, atol=1e-3), (
"preprocessor is not an identity at 300x300, it cannot be bypassed"
)
image = ops.convert(parameter, "f32")
for consumer in list(preprocessor.output(0).get_target_inputs()):
consumer.replace_source_output(image.output(0))
# (ymin, xmin, ymax, xmax) -> (xmin, ymin, xmax, ymax)
priors = anchors[:, [1, 0, 3, 2]].astype(np.float32).reshape(-1)
variances = np.tile(BOX_VARIANCES, len(anchors))
proposals = ops.constant(np.stack([priors, variances])[np.newaxis])
# (ty, tx, th, tw) -> (dx, dy, dw, dh) for the CENTER_SIZE decode
box_logits = ops.reshape(ops.gather(box_deltas, [1, 0, 3, 2], 1), [1, -1], False)
class_preds = ops.reshape(class_scores, [1, -1], False)
detections = ops.detection_output(
box_logits,
class_preds,
proposals,
{
"background_label_id": 0,
"top_k": 100,
"keep_top_k": [100],
"nms_threshold": 0.6,
"confidence_threshold": 0.3,
"code_type": "caffe.PriorBoxParameter.CENTER_SIZE",
"share_location": True,
"variance_encoded_in_target": False,
"normalized": True,
"clip_before_nms": False,
"clip_after_nms": True,
"decrease_label_id": False,
},
)
detections.output(0).get_tensor().set_names({"detection_out"})
model = ov.Model([detections], model.get_parameters(), "ssdlite_mobilenet_v2")
model = ov.Model([detections], [parameter], "ssdlite_mobilenet_v2")
ppp = PrePostProcessor(model)
ppp.input().tensor().set_layout(ov.Layout("NHWC"))
ppp.input().preprocess().reverse_channels()
model = ppp.build()
ov.save_model(model, "/models/ssdlite_mobilenet_v2.xml", compress_to_fp16=True)
# Fail the build rather than silently ship the dynamically shaped graph again.
op_types = [op.get_type_name() for op in model.get_ordered_ops()]
assert op_types.count("DetectionOutput") == 1, "postprocessor was not fused"
for dynamic_op in ("NonMaxSuppression", "NonZero", "Loop", "TensorIterator"):
assert dynamic_op not in op_types, f"{dynamic_op} left in the graph"
output_shape = model.outputs[0].get_partial_shape()
assert output_shape.is_static and list(output_shape) == [1, 1, 100, 7], (
f"unexpected detector output shape {output_shape}"
)
ov.save_model(model, OUTPUT_PATH, compress_to_fp16=True)
-2
View File
@@ -79,7 +79,5 @@ sherpa-onnx==1.12.*
faster-whisper==1.1.*
librosa==0.11.*
soundfile==0.13.*
# DeGirum detector
degirum == 0.16.*
# Memory profiling
memray == 1.15.*
-75
View File
@@ -1269,78 +1269,3 @@ axengine:
input_dtype: int
input_pixel_format: bgr
labelmap_path: /labelmap/coco-80.txt
degirumAiServer:
title: DeGirum AI Server
models:
- key: ai-server-inference
label: AI Server Inference
recommended: true
download: |-
Launch a DeGirum AI server as a Docker container, then point the detector at it. Add this to your `docker-compose.yml`:
```yaml
degirum_detector:
container_name: degirum
image: degirum/aiserver:latest
privileged: true
ports:
- "8778:8778"
```
Set `location` to the server's service name, container name, or `host:port`.
ui: |
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
| Field | Value |
| --- | --- |
| **Location** | `degirum` |
| **Zoo** | `degirum/public` |
| **Token** | your AI Hub token (optional for the public zoo) |
yaml: |
degirum_detector:
type: degirum
location: degirum
zoo: degirum/public
token: dg_example_token
degirumLocal:
title: DeGirum Local
models:
- key: local-inference
label: Local Inference
recommended: true
download: Run hardware directly inside the Frigate container with `@local`, removing the AI server hop. The matching device runtime (e.g. the Hailo runtime) must be installed in the container; confirm it with `degirum sys-info`.
ui: |
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
| Field | Value |
| --- | --- |
| **Location** | `@local` |
| **Zoo** | `degirum/public` |
| **Token** | your AI Hub token (optional for the public zoo) |
yaml: |
degirum_detector:
type: degirum
location: @local
zoo: degirum/public
token: dg_example_token
degirumCloud:
title: DeGirum AI Hub Cloud
models:
- key: ai-hub-cloud-inference
label: AI Hub Cloud Inference
recommended: true
download: Run inferences on DeGirum's [AI Hub](https://hub.degirum.com) cloud with `@cloud`. Sign up, create an access token, and set it as `token`. Network latency may require lowering your detection fps.
ui: |
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
| Field | Value |
| --- | --- |
| **Location** | `@cloud` |
| **Zoo** | `degirum/public` |
| **Token** | your AI Hub token (optional for the public zoo) |
yaml: |
degirum_detector:
type: degirum
location: @cloud
zoo: degirum/public
token: dg_example_token
@@ -981,7 +981,9 @@ cameras:
# Optional: Adjust sort order of cameras in the UI. Larger numbers come later (default: shown below)
# By default the cameras are sorted alphabetically.
order: 0
# Optional: Whether or not to show the camera in the Frigate UI (default: shown below)
# Optional: Whether or not to show the camera on the default All Cameras live dashboard.
# The camera is still available everywhere else, including camera groups and settings
# (default: shown below)
dashboard: True
# Optional: Whether this camera is visible in review (the review page and its camera
# filter, motion review, and the history view) (default: shown below)
+103 -3
View File
@@ -6,6 +6,7 @@ title: Configuring Generative AI
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
import FaqItem from "@site/src/components/FaqItem";
## Configuration
@@ -13,6 +14,18 @@ A Generative AI provider can be configured in the global config, which will make
`genai` is a map of named providers. Each key under `genai` is a name you choose, and its value is that provider's settings:
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Enrichments > Generative AI" />.
- Click **Add** and enter a **Provider name**. Any name of letters, numbers, hyphens, and underscores is accepted, but it cannot be changed from the UI after the provider is created.
- Set **Provider** to the service you are using (e.g., `ollama`)
- Set **Base URL**, **API key**, and **Model** as required by that provider
- Set **Roles** to the roles this provider should handle.
</TabItem>
<TabItem value="yaml">
```yaml
genai:
my_provider: # any name you like
@@ -25,6 +38,9 @@ genai:
- chat
```
</TabItem>
</ConfigTabs>
The examples on this page all use `my_provider`, but the name is arbitrary and is only used to reference the provider elsewhere in the config (for example, `semantic_search.model`).
Each provider handles one or more **roles**: `chat`, `descriptions`, and `embeddings`. A provider handles all three by default, and each role may be assigned to exactly one provider. Define a single provider if you want it to do everything, or split the roles across several providers using the `roles` option.
@@ -43,15 +59,20 @@ Running Generative AI models on CPU is not recommended, as high inference times
### Recommended Local Models
You must use a vision-capable model with Frigate. The following models are recommended for local deployment:
You must use a vision-capable model with Frigate. The following models are recommended for local deployment of the `descriptions` and `chat` roles:
| Model | Notes |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `qwen3-vl` | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object. |
| `qwen3.5` | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
| `qwen3.6` | Strong situational understanding, similar to qwen3-vl |
| `qwen3.6` | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
| `gemma4` | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action. |
The `embeddings` role needs a different kind of model. Text queries are matched against the stored image embeddings, so the model must be trained to place images and text into the same vector space. A chat or description model will still return vectors when asked, but those vectors are not trained for retrieval and text searches will return poor matches with no error to indicate why.
| Model | Notes |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `qwen3-vl-embedding` | Multimodal embeddings for [Semantic Search](/configuration/semantic_search#genai-provider). Must be served by llama.cpp started with `--embeddings` and `--mmproj`. |
:::info
Each model is available in multiple parameter sizes (3b, 4b, 8b, etc.). Larger sizes are more capable of complex tasks and understanding of situations, but requires more memory and computational resources. It is recommended to try multiple models and experiment to see which performs best.
@@ -416,3 +437,82 @@ genai:
</TabItem>
</ConfigTabs>
## FAQ
<FaqItem id="how-do-i-debug-genai-issues" question="How do I debug GenAI issues?">
Frigate's Generative AI features are configured and enabled separately. [Review descriptions and summaries](/configuration/genai/genai_review) live under `review.genai`, and [object descriptions](/configuration/genai/genai_objects) live under `objects.genai`. Configuring a provider on this page does not enable either feature, and enabling one does not enable the other. Decide which of the two is not working, then work through the steps below.
1. Confirm a provider is available and holds the `descriptions` role.
- Review descriptions, review summaries, and object descriptions all use the provider that has the `descriptions` role assigned in <NavPath path="Settings > Enrichments > Generative AI > Roles" /> (`genai.<provider>.roles`).
- A provider is contacted the first time one of its roles is actually used. A provider holding the `embeddings` role for semantic search is initialized during startup, while a `descriptions` provider is not initialized until the first description is requested, which may be well after boot.
- In <NavPath path="Settings > Enrichments > Generative AI" />, use **Refresh models** next to the model field. It queries the provider for its model list and is a quick way to verify that the base URL, API key, and network path between Frigate and your provider are correct.
2. Confirm the feature you expect is actually enabled.
- Object descriptions are disabled by default. Turn on <NavPath path="Settings > Global configuration > Objects > GenAI object config > Enable GenAI" /> (`objects.genai.enabled`), either globally or per camera. This is the most common reason custom prompts appear to be ignored while review summaries are still being generated.
- Review descriptions are disabled by default. Turn on <NavPath path="Settings > Global configuration > Review > GenAI config > Enable GenAI descriptions" /> (`review.genai.enabled`). Once enabled, alerts are described by default but detections are not, so a detection-only review item will never get a summary unless **Enable GenAI for detections** (`review.genai.detections`) is also on.
3. If object descriptions are never requested, check the filters that skip generation.
- <NavPath path="Settings > Global configuration > Objects > GenAI object config > GenAI objects" /> (`objects.genai.objects`) limits generation to specific labels, and **Required zones** (`objects.genai.required_zones`) requires the object to have entered one of those zones. If either is set and does not match, Frigate skips the request silently.
- Thumbnails are only collected while an object is moving. Objects that go stationary early contribute fewer frames.
- **Use snapshots** (`objects.genai.use_snapshot`) requires snapshots to be enabled for the camera. If the snapshot cannot be read, Frigate logs `Cannot load snapshot for <id>, file not found` and no description is generated.
- **Send on end** (`objects.genai.send_triggers.tracked_object_end`) is on by default. If you have turned it off in favor of **Early GenAI trigger** (`objects.genai.send_triggers.after_significant_updates`), descriptions are only requested once that number of updates is reached.
4. Enable debug logs to see exactly what Frigate is doing. Restart Frigate after this change. The next step also requires a restart, so turn both on at the same time to avoid restarting twice.
```yaml
logger:
default: info
logs:
# highlight-start
frigate.genai: debug
frigate.data_processing.post.object_descriptions: debug
frigate.data_processing.post.review_descriptions: debug
# highlight-end
```
5. Save the exact images and prompts that were sent to your provider.
- Turn on **Save thumbnails** for the feature you are debugging (`review.genai.debug_save_thumbnails` or `objects.genai.debug_save_thumbnails`). Both features write to `/media/frigate/clips/genai-requests/`, and these files are admin-only.
- Review descriptions write `genai-requests/<review_id>/` containing the numbered frames that were sent, plus `prompt.txt` and `response.txt` with the exact prompt and the raw, unparsed model response.
- Review summary reports write `genai-requests/<start_ts>-<end_ts>/prompt.txt` and `response.txt`. No images are involved, since a report summarizes existing review descriptions.
- Object descriptions write `genai-requests/<event_id>/` containing the numbered thumbnails. The prompt for object descriptions is not written to a file, it is only visible in the debug logs from step 4.
- Look at the saved images before blaming the model. If the object is small, blurry, or out of frame, no prompt will fix the result. For object descriptions, consider turning on **Use snapshots** (`objects.genai.use_snapshot`) to send a higher quality image. For review items, consider setting **Review image source** (`review.genai.image_source`) to `recordings` for 480p frames instead of the lower resolution preview frames.
<ConfigTabs>
<TabItem value="ui">
For review descriptions, navigate to <NavPath path="Settings > Global configuration > Review" /> and set **GenAI config > Save thumbnails** to on.
For object descriptions, navigate to <NavPath path="Settings > Global configuration > Objects" />, expand **GenAI object config**, and set **Save thumbnails** to on.
</TabItem>
<TabItem value="yaml">
```yaml
review:
genai:
enabled: true
# highlight-next-line
debug_save_thumbnails: true
objects:
genai:
enabled: true
# highlight-next-line
debug_save_thumbnails: true
```
</TabItem>
</ConfigTabs>
6. Verify the prompt is what you think it is.
- Object description prompts are the ones you control directly. A camera-level <NavPath path="Settings > Camera configuration > Objects > GenAI object config > Caption prompt" /> (`objects.genai.prompt`) overrides the global one, and an entry in **Object prompts** (`objects.genai.object_prompts`) for a label overrides both for that label. Only `{label}`, `{sub_label}`, and `{camera}` are substituted.
- Review description prompts are built by Frigate and request a structured JSON response, so they are not fully replaceable. The parts you control are <NavPath path="Settings > Global configuration > Review > GenAI config > Activity context prompt" /> (`review.genai.activity_context_prompt`) and **Additional concerns** (`review.genai.additional_concerns`). Keep the activity context prompt general, since overly specific rules will sway the model's threat level scoring.
7. If descriptions are generated but the results are poor or inconsistent, look at the model and the context window.
- Empty fields, missing `shortSummary` values, or `Failed to parse review description` errors usually mean the model is not following the requested JSON schema. Smaller models struggle with structured output. Try a larger parameter size or one of the [recommended models](#recommended-local-models).
- Frigate calculates how many frames to send from the context size the provider reports. If your server reports a different value than it is actually running with, frames will be truncated or the request will fail. Pin the value by adding `context_size` under <NavPath path="Settings > Enrichments > Generative AI > Provider options" /> (`genai.<provider>.provider_options`), and for Ollama also confirm `options.num_ctx` there matches the context you have configured.
- Check **Review Description Speed** and **Object Description Speed** in <NavPath path="System metrics > Enrichments" />. If inference takes tens of seconds, requests will queue behind each other and descriptions will appear to stop. For Ollama, review `OLLAMA_NUM_PARALLEL`, `OLLAMA_MAX_QUEUE`, and `OLLAMA_MAX_LOADED_MODELS` so that concurrent requests from Frigate are handled the way you expect.
</FaqItem>
+4
View File
@@ -113,3 +113,7 @@ Many providers also have a public facing chat interface for their models. Downlo
- OpenAI - [ChatGPT](https://chatgpt.com)
- Gemini - [Google AI Studio](https://aistudio.google.com)
- Ollama - [Open WebUI](https://docs.openwebui.com/)
## Troubleshooting
If descriptions are not being generated, or the generated descriptions are not what you expect, see [How do I debug GenAI issues?](/configuration/genai/genai_config#how-do-i-debug-genai-issues).
@@ -201,3 +201,7 @@ Along with individual review item summaries, Generative AI can also produce a si
Review reports can be requested via the [API](/integrations/api/generate-review-summary-review-summarize-start-start-ts-end-end-ts-post) by sending a POST request to `/api/review/summarize/start/{start_ts}/end/{end_ts}` with Unix timestamps.
For Home Assistant users, there is a built-in service (`frigate.review_summarize`) that makes it easy to request review reports as part of automations or scripts. This allows you to automatically generate daily summaries, vacation reports, or custom time period reports based on your specific needs.
## Troubleshooting
If summaries are not being generated, or the generated summaries are not what you expect, see [How do I debug GenAI issues?](/configuration/genai/genai_config#how-do-i-debug-genai-issues).
+1 -1
View File
@@ -334,7 +334,7 @@ When your browser runs into problems playing back your camera streams, it will l
- **stalled**
- What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving).
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in the UI pane of Frigate's settings.
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in <NavPath path="Settings > UI" /> .
- Possible console messages from the player code:
- `Buffer time (10 seconds) exceeded, browser may not be playing media correctly.`
@@ -24,7 +24,6 @@ Frigate supports multiple different detectors that work on different types of ha
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
- <CommunityBadge /> [DeGirum](#degirum): Service for using hardware devices in the cloud or locally. Hardware and models provided on the cloud on [their website](https://hub.degirum.com).
**AMD**
@@ -755,87 +754,6 @@ Explanation of the parameters:
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
- `config`: Configuration passed to `rknn-toolkit2` for model conversion. For an explanation of all available parameters have a look at section "2.2. Model configuration" of [this manual](https://github.com/MarcA711/rknn-toolkit2/releases/download/v2.3.2/03_Rockchip_RKNPU_API_Reference_RKNN_Toolkit2_V2.3.2_EN.pdf).
## DeGirum
DeGirum is a detector that can use any type of hardware listed on [their website](https://hub.degirum.com). DeGirum can be used with local hardware through a DeGirum AI Server, or through the use of `@local`. You can also connect directly to DeGirum's AI Hub to run inferences. **Please Note:** This detector _cannot_ be used for commercial purposes.
### Configuration {#configuration-degirum}
#### AI Server Inference
Before starting with the config file for this section, you must first launch an AI server. DeGirum has an AI server ready to use as a docker container. Add this to your `docker-compose.yml` to get started:
```yaml
degirum_detector:
container_name: degirum
image: degirum/aiserver:latest
privileged: true
ports:
- "8778:8778"
```
All supported hardware will automatically be found on your AI server host as long as relevant runtimes and drivers are properly installed on your machine. Refer to [DeGirum's docs site](https://docs.degirum.com/pysdk/runtimes-and-drivers) if you have any trouble.
Once completed, configure the detector as follows:
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumAiServer.models} />
Setting up a model in the `config.yml` is similar to setting up an AI server.
You can set it to:
- A model listed on the [AI Hub](https://hub.degirum.com), given that the correct zoo name is listed in your detector
- If this is what you choose to do, the correct model will be downloaded onto your machine before running.
- A local directory acting as a zoo. See DeGirum's docs site [for more information](https://docs.degirum.com/pysdk/user-guide-pysdk/organizing-models#model-zoo-directory-structure).
- A path to some model.json.
```yaml
model:
path: ./mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1 # directory to model .json and file
width: 300 # width is in the model name as the first number in the "int"x"int" section
height: 300 # height is in the model name as the second number in the "int"x"int" section
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
```
#### Local Inference
It is also possible to eliminate the need for an AI server and run the hardware directly. The benefit of this approach is that you eliminate any bottlenecks that occur when transferring prediction results from the AI server docker container to the frigate one. However, the method of implementing local inference is different for every device and hardware combination, so it's usually more trouble than it's worth. A general guideline to achieve this would be:
1. Ensuring that the frigate docker container has the runtime you want to use. So for instance, running `@local` for Hailo means making sure the container you're using has the Hailo runtime installed.
2. To double check the runtime is detected by the DeGirum detector, make sure the `degirum sys-info` command properly shows whatever runtimes you mean to install.
3. Create a DeGirum detector in your configuration.
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumLocal.models} />
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
```yaml
model:
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
width: 300 # width is in the model name as the first number in the "int"x"int" section
height: 300 # height is in the model name as the second number in the "int"x"int" section
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
```
#### AI Hub Cloud Inference
If you do not possess whatever hardware you want to run, there's also the option to run cloud inferences. Do note that your detection fps might need to be lowered as network latency does significantly slow down this method of detection. For use with Frigate, we highly recommend using a local AI server as described above. To set up cloud inferences,
1. Sign up at [DeGirum's AI Hub](https://hub.degirum.com).
2. Get an access token.
3. Create a DeGirum detector in your configuration.
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumCloud.models} />
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
```yaml
model:
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
width: 300 # width is in the model name as the first number in the "int"x"int" section
height: 300 # height is in the model name as the second number in the "int"x"int" section
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
```
## AXERA
Hardware accelerated object detection is supported on the following SoCs:
+7 -3
View File
@@ -34,11 +34,15 @@ The detect FFmpeg process exited on its own. This message is only the notificati
</FaqItem>
<FaqItem id="non-monotonically-increasing-dts" question="Application provided invalid, non monotonically increasing dts to muxer">
<FaqItem id="non-monotonically-increasing-dts" question="Non-monotonic DTS / non monotonically increasing dts to muxer / Queue input is backward in time">
An FFmpeg message meaning the camera sent packets with out-of-order timestamps. Because recordings are copied without re-encoding, FFmpeg cannot fix them, and the segment muxer often splits early, producing one-second segments and a cache backlog. The usual cause is a camera "Smart Codec" / H.264+ / H.265+ mode or a camera clock that jumps.
These are FFmpeg messages indicating the camera sent packets with out-of-order timestamps, either on the video or the audio stream. Timestamp jitter like this is common with WiFi cameras and restreamed or proxied sources; other causes are a camera "Smart Codec" / H.264+ / H.265+ mode or a camera clock that jumps. A sustained flood of these messages usually precedes the stream stalling and the watchdog restarting FFmpeg.
See [Recordings: segments are only 1 second long](/troubleshooting/recordings#segments-are-only-1-second-long).
In most cases, the fix is to improve the network, reduce system resource usage, or switch to non-WiFi cameras. In general, WiFi cameras are [not recommended](https://ipcamtalk.com/threads/multiple-cameras-high-bandwidth.77100/#post-861110).
On the video stream, this can affect recordings: because they are copied without re-encoding, FFmpeg cannot fix the timestamps, and the segment muxer often splits early, producing one-second segments and a cache backlog. See [Recordings: segments are only 1 second long](/troubleshooting/recordings#segments-are-only-1-second-long).
On the audio stream, the messages can come from the output's audio encoding. If the audio stream is the problem, it may help to have go2rtc transcode it by adding `#audio=aac` to the camera's go2rtc stream to produce clean timestamps for everything consuming the restream.
</FaqItem>
+1 -1
View File
@@ -39,7 +39,7 @@ The per-clip variation is typically quite low and is mostly an artifact of keyfr
Debug Replay lets you re-run Frigate's detection pipeline against a section of recorded video without manually configuring a dummy camera. It automatically extracts the recording, creates a temporary camera with the same detection settings as the original, and loops the clip through the pipeline so you can observe detections in real time.
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame.
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame. The Debug Replay camera does not save recordings or snapshots or surface anything in Explore, but it otherwise behaves like a regular camera, including running enrichments such as Face Recognition, LPR, and custom classification.
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
+15 -1
View File
@@ -31,7 +31,10 @@ from frigate.api.auth import (
get_allowed_cameras_for_filter,
require_role,
)
from frigate.api.config_util import swap_runtime_config
from frigate.api.config_util import (
publish_camera_section_updates,
swap_runtime_config,
)
from frigate.api.defs.query.app_query_parameters import AppTimelineHourlyQueryParameters
from frigate.api.defs.request.app_body import (
AppConfigSetBody,
@@ -963,6 +966,17 @@ def config_set(request: Request, body: AppConfigSetBody):
body.update_topic, settings
)
# a config/cameras/* topic publishes camera copies, a
# global topic the global object. FrigateConfig.parse
# folds some global sections down into every camera,
# and workers read both objects, so any such section
# needs its camera copies sent alongside the global
# publish above.
if body.update_topic == "config/birdseye":
publish_camera_section_updates(
request.app, config, CameraConfigUpdateEnum.birdseye
)
return JSONResponse(
content=(
{
+28
View File
@@ -3,6 +3,30 @@
from fastapi import FastAPI
from frigate.config import FrigateConfig
from frigate.config.camera.updater import (
CameraConfigUpdateEnum,
CameraConfigUpdateTopic,
)
def publish_camera_section_updates(
app: FastAPI, config: FrigateConfig, update_type: CameraConfigUpdateEnum
) -> None:
"""Broadcast every camera's re-resolved value for a global section.
Global sections are folded into each camera at parse time and the camera
copies are what workers read, so send them rather than leave a worker to
guess which cameras were inheriting.
"""
for camera_name, camera_config in config.cameras.items():
settings = getattr(camera_config, update_type.name, None)
if settings is None:
continue
app.config_publisher.publish_update(
CameraConfigUpdateTopic(update_type, camera_name), settings
)
def swap_runtime_config(app: FastAPI, config: FrigateConfig) -> None:
@@ -16,6 +40,10 @@ def swap_runtime_config(app: FastAPI, config: FrigateConfig) -> None:
camera the user turned off would silently come back on.
"""
app.frigate_config = config
if app.config_holder is not None:
app.config_holder.set(config)
app.genai_manager.update_config(config)
if app.profile_manager is not None:
+3
View File
@@ -35,6 +35,7 @@ from frigate.comms.event_metadata_updater import (
)
from frigate.config import FrigateConfig
from frigate.config.camera.updater import CameraConfigUpdatePublisher
from frigate.config.holder import ConfigHolder
from frigate.config.profile_manager import ProfileManager
from frigate.debug_replay import DebugReplayManager, debug_replay_auto_stop_watchdog
from frigate.embeddings import EmbeddingsContext
@@ -74,6 +75,7 @@ def create_fastapi_app(
dispatcher: Dispatcher | None = None,
profile_manager: ProfileManager | None = None,
enforce_default_admin: bool = True,
config_holder: ConfigHolder | None = None,
):
logger.info("Starting FastAPI app")
app = FastAPI(
@@ -162,6 +164,7 @@ def create_fastapi_app(
app.replay_manager = replay_manager
app.dispatcher = dispatcher
app.profile_manager = profile_manager
app.config_holder = config_holder
if frigate_config.auth.enabled:
secret = get_jwt_secret()
+14 -1
View File
@@ -30,6 +30,7 @@ from frigate.comms.ws import WebSocketClient
from frigate.comms.zmq_proxy import ZmqProxy
from frigate.config.camera.updater import CameraConfigUpdatePublisher
from frigate.config.config import FrigateConfig
from frigate.config.holder import ConfigHolder
from frigate.config.profile_manager import ProfileManager
from frigate.const import (
CACHE_DIR,
@@ -122,7 +123,18 @@ class FrigateApp:
self.processes: dict[str, int] = {}
self.embeddings: EmbeddingsContext | None = None
self.profile_manager: ProfileManager | None = None
self.config = config
self.config_holder = ConfigHolder(config)
@property
def config(self) -> FrigateConfig:
"""The current config, not the one Frigate booted with.
Read through the holder so the deferred watchdog factories below build
a replacement process from the config as it is now. There is no setter
on purpose: a plain attribute would let a caller pin this back to a
single object and reintroduce the staleness.
"""
return self.config_holder.config
def ensure_dirs(self) -> None:
dirs = [
@@ -645,6 +657,7 @@ class FrigateApp:
self.replay_manager,
self.dispatcher,
self.profile_manager,
config_holder=self.config_holder,
),
host="127.0.0.1",
port=5001,
+2 -2
View File
@@ -13,8 +13,8 @@ class CameraUiConfig(FrigateBaseModel):
)
dashboard: bool = Field(
default=True,
title="Show in UI",
description="Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again.",
title="Show on Live dashboard",
description="Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings.",
)
review: bool = Field(
default=True,
+34
View File
@@ -0,0 +1,34 @@
"""Shared handle on the config object that is current for this instance."""
from .config import FrigateConfig
__all__ = ["ConfigHolder"]
class ConfigHolder:
"""Indirection for the most recently parsed config.
/api/config/set re-parses yaml into a brand new FrigateConfig instead of
mutating the old one, so any reference captured during startup goes stale
the first time a user saves. Anything that has to build something after
startup, most importantly the watchdog factories that rebuild a crashed
process, must read through a holder rather than close over a config
object, or the rebuilt process comes back with the config as it was at
boot and silently discards every change made since.
There is deliberately no setter on the read side: the swap runs in exactly
one place (frigate.api.config_util.swap_runtime_config) and everyone else
only reads.
"""
def __init__(self, config: FrigateConfig) -> None:
self._config = config
@property
def config(self) -> FrigateConfig:
"""The config as of the most recent successful save."""
return self._config
def set(self, config: FrigateConfig) -> None:
"""Install a freshly parsed config as the current one."""
self._config = config
-157
View File
@@ -1,157 +0,0 @@
import logging
import queue
from typing import Literal
import numpy as np
from pydantic import ConfigDict, Field
from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig
logger = logging.getLogger(__name__)
DETECTOR_KEY = "degirum"
### DETECTOR CONFIG ###
class DGDetectorConfig(BaseDetectorConfig):
"""DeGirum detector for running models via DeGirum cloud or local inference services."""
model_config = ConfigDict(
title="DeGirum",
)
type: Literal[DETECTOR_KEY]
location: str = Field(
default=None,
title="Inference Location",
description="Location of the DeGirim inference engine (e.g. '@cloud', '127.0.0.1').",
)
zoo: str = Field(
default=None,
title="Model Zoo",
description="Path or URL to the DeGirum model zoo.",
)
token: str = Field(
default=None,
title="DeGirum Cloud Token",
description="Token for DeGirum Cloud access.",
)
### ACTUAL DETECTOR ###
class DGDetector(DetectionApi):
type_key = DETECTOR_KEY
def __init__(self, detector_config: DGDetectorConfig):
try:
import degirum as dg
except ModuleNotFoundError:
raise ImportError("Unable to import DeGirum detector.") from None
self._queue = queue.Queue()
self._zoo = dg.connect(
detector_config.location, detector_config.zoo, detector_config.token
)
logger.debug(f"Models in zoo: {self._zoo.list_models()}")
self.dg_model = self._zoo.load_model(
detector_config.model.path,
)
# Setting input image format to raw reduces preprocessing time
self.dg_model.input_image_format = "RAW"
# Prioritize the most powerful hardware available
self.select_best_device_type()
# Frigate handles pre processing as long as these are all set
input_shape = self.dg_model.input_shape[0]
self.model_height = input_shape[1]
self.model_width = input_shape[2]
# Passing in dummy frame so initial connection latency happens in
# init function and not during actual prediction
frame = np.zeros(
(detector_config.model.width, detector_config.model.height, 3),
dtype=np.uint8,
)
# Pass in frame to overcome first frame latency
self.dg_model(frame)
self.prediction = self.prediction_generator()
def select_best_device_type(self):
"""
Helper function that selects fastest hardware available per model runtime
"""
types = self.dg_model.supported_device_types
device_map = {
"OPENVINO": ["GPU", "NPU", "CPU"],
"HAILORT": ["HAILO8L", "HAILO8"],
"N2X": ["ORCA1", "CPU"],
"ONNX": ["VITIS_NPU", "CPU"],
"RKNN": ["RK3566", "RK3568", "RK3588"],
"TENSORRT": ["DLA", "GPU", "DLA_ONLY"],
"TFLITE": ["ARMNN", "EDGETPU", "CPU"],
}
runtime = types[0].split("/")[0]
# Just create an array of format {runtime}/{hardware} for every hardware
# in the value for appropriate key in device_map
self.dg_model.device_type = [
f"{runtime}/{hardware}" for hardware in device_map[runtime]
]
def prediction_generator(self):
"""
Generator for all incoming frames. By using this generator, we don't have to keep
reconnecting our websocket on every "predict" call.
"""
logger.debug("Prediction generator was called")
with self.dg_model as model:
while 1:
logger.info(f"q size before calling get: {self._queue.qsize()}")
data = self._queue.get(block=True)
logger.info(f"q size after calling get: {self._queue.qsize()}")
logger.debug(
f"Data we're passing into model predict: {data}, shape of data: {data.shape}"
)
result = model.predict(data)
logger.debug(f"Prediction result: {result}")
yield result
def detect_raw(self, tensor_input):
# Reshaping tensor to work with pysdk
truncated_input = tensor_input.reshape(tensor_input.shape[1:])
logger.debug(f"Detect raw was called for tensor input: {tensor_input}")
# add tensor_input to input queue
self._queue.put(truncated_input)
logger.debug(f"Queue size after adding truncated input: {self._queue.qsize()}")
# define empty detection result
detections = np.zeros((20, 6), np.float32)
# grab prediction
res = next(self.prediction)
# If we have an empty prediction, return immediately
if len(res.results) == 0 or len(res.results[0]) == 0:
return detections
i = 0
for result in res.results:
if i >= 20:
break
detections[i] = [
result["category_id"],
float(result["score"]),
result["bbox"][1] / self.model_height,
result["bbox"][0] / self.model_width,
result["bbox"][3] / self.model_height,
result["bbox"][2] / self.model_width,
]
i += 1
logger.debug(f"Detections output: {detections}")
return detections
+2 -1
View File
@@ -9,6 +9,7 @@ from pydantic import ConfigDict, Field
from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
from frigate.util.model import xyxy_to_xywh_for_nms
try:
from tflite_runtime.interpreter import Interpreter, load_delegate
@@ -297,7 +298,7 @@ class EdgeTpuTfl(DetectionApi):
# until after filtering out redundant boxes
# Shift the logit scores to be non-negative (required by cv2)
indices = cv2.dnn.NMSBoxes(
bboxes=boxes_filtered_decoded,
bboxes=xyxy_to_xywh_for_nms(boxes_filtered_decoded),
scores=max_scores_filtered_shiftedpositive,
score_threshold=(
self.min_logit_value + self.logit_shift_to_positive_values
+3 -2
View File
@@ -17,6 +17,7 @@ from frigate.detectors.detector_config import (
ModelTypeEnum,
)
from frigate.util.file import FileLock
from frigate.util.model import xyxy_to_xywh_for_nms
logger = logging.getLogger(__name__)
@@ -581,7 +582,7 @@ class MemryXDetector(DetectionApi):
# Convert coordinates to integers
x_min, y_min, x_max, y_max = map(int, [x_min, y_min, x_max, y_max])
# Append valid detections [class_id, confidence, x, y, width, height]
# Append valid detections [class_id, confidence, x_min, y_min, x_max, y_max]
detections.append([class_id, confidence, x_min, y_min, x_max, y_max])
final_detections = np.zeros((20, 6), np.float32)
@@ -595,7 +596,7 @@ class MemryXDetector(DetectionApi):
detections = np.array(detections, dtype=np.float32)
# Apply Non-Maximum Suppression (NMS)
bboxes = detections[:, 2:6].tolist() # (x_min, y_min, width, height)
bboxes = xyxy_to_xywh_for_nms(detections[:, 2:6])
scores = detections[:, 1].tolist() # Confidence scores
indices = cv2.dnn.NMSBoxes(bboxes, scores, 0.45, 0.5)
+3 -3
View File
@@ -226,12 +226,12 @@ class OvDetector(DetectionApi):
conf_mask = (image_pred[:, 4] * class_conf.squeeze() >= 0.3).squeeze()
# Detections ordered as (x1, y1, x2, y2, obj_conf, class_conf, class_pred)
detections = np.concatenate(
predictions = np.concatenate(
(image_pred[:, :5], class_conf, class_pred), axis=1
)
detections = detections[conf_mask]
predictions = predictions[conf_mask]
ordered = detections[detections[:, 5].argsort()[::-1]][:20]
ordered = predictions[predictions[:, 5].argsort()[::-1]][:20]
for i, object_detected in enumerate(ordered):
detections[i] = self.process_yolo(
+2 -2
View File
@@ -12,7 +12,7 @@ from frigate.const import MODEL_CACHE_DIR, SUPPORTED_RK_SOCS
from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detection_runners import RKNNModelRunner
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
from frigate.util.model import post_process_yolo
from frigate.util.model import post_process_yolo, xyxy_to_xywh_for_nms
from frigate.util.rknn_converter import auto_convert_model
logger = logging.getLogger(__name__)
@@ -285,7 +285,7 @@ class Rknn(DetectionApi):
# run nms
indices = cv2.dnn.NMSBoxes(
bboxes=boxes,
bboxes=xyxy_to_xywh_for_nms(boxes),
scores=scores,
score_threshold=0.4,
nms_threshold=0.4,
+10
View File
@@ -23,6 +23,7 @@ from frigate.genai.prompts import (
build_review_summary_prompt,
)
from frigate.models import Event
from frigate.util.builtin import has_non_finite_number
logger = logging.getLogger(__name__)
@@ -164,6 +165,15 @@ class GenAIClient:
except json.JSONDecodeError as je:
logger.error("Failed to parse review description JSON: %s", je)
return None
# model_construct skips validation, so non-finite numbers that
# the validated path would have rejected have to be caught here
if has_non_finite_number(raw):
logger.error(
"Discarding review description containing non-finite numbers."
)
return None
# observations and confidence are required on the model; fill an empty default
# if the response omitted it so attribute access stays safe.
raw.setdefault("observations", [])
+3 -6
View File
@@ -178,13 +178,10 @@ class OutputProcess(FrigateProcess):
)
if update_topic is not None and birdseye_config is not None:
previous_global_mode = self.config.birdseye.mode
# only the global-only fields are applied here; the per-camera
# enabled and mode arrive on config/cameras/<name>/birdseye,
# already resolved against yaml by the config parse
self.config.birdseye = birdseye_config
for camera_config in self.config.cameras.values():
if camera_config.birdseye.mode == previous_global_mode:
camera_config.birdseye.mode = birdseye_config.mode
logger.debug("Applied dynamic birdseye config update")
# check if there is an updated config
@@ -13,6 +13,7 @@ from frigate.config.camera.updater import (
CameraConfigUpdatePublisher,
CameraConfigUpdateTopic,
)
from frigate.config.holder import ConfigHolder
from frigate.models import Event, Recordings, ReviewSegment
from frigate.test.http_api.base_http_test import AuthTestClient, BaseTestHttp
@@ -373,6 +374,128 @@ class TestConfigSetWildcardPropagation(BaseTestHttp):
finally:
os.unlink(config_path)
@patch("frigate.api.app.find_config_file")
def test_global_birdseye_save_fans_out_resolved_camera_configs(
self, mock_find_config
):
"""A global birdseye save must also publish the per-camera values.
Global birdseye only seeds enabled and mode; the camera copies are what
the output process actually reads. Sending just the global object makes
a worker guess which cameras were inheriting, and the only available
guess (mode still equals the previous global) wrongly claims a camera
whose explicit yaml mode happens to match.
"""
self.minimal_config["birdseye"] = {"enabled": True, "mode": "motion"}
# explicit override that matches the global value being replaced
self.minimal_config["cameras"]["front_door"]["birdseye"] = {"mode": "motion"}
config_path = self._write_config_file()
mock_find_config.return_value = config_path
try:
app, mock_publisher = self._create_app_with_publisher()
with AuthTestClient(app) as client:
resp = client.put(
"/config/set",
json={
"config_data": {"birdseye": {"mode": "continuous"}},
"update_topic": "config/birdseye",
"requires_restart": 0,
},
)
self.assertEqual(resp.status_code, 200)
# the global object still goes out on its own topic
mock_publisher.publisher.publish.assert_called_once()
topic, settings = mock_publisher.publisher.publish.call_args[0]
self.assertEqual(topic, "config/birdseye")
self.assertEqual(settings.mode.value, "continuous")
published = {
call[0][0].camera: call[0][1]
for call in mock_publisher.publish_update.call_args_list
}
self.assertEqual(set(published), {"front_door", "back_yard"})
for call in mock_publisher.publish_update.call_args_list:
self.assertEqual(
call[0][0].update_type, CameraConfigUpdateEnum.birdseye
)
# the override survives, the inheriting camera follows global
self.assertEqual(published["front_door"].mode.value, "motion")
self.assertEqual(published["back_yard"].mode.value, "continuous")
finally:
os.unlink(config_path)
@patch("frigate.api.app.find_config_file")
def test_save_updates_the_config_holder(self, mock_find_config):
"""A save must move the holder onto the freshly parsed config.
FrigateApp reads the holder when the watchdog rebuilds a crashed
process; if the save leaves it on the boot config, that process comes
back having lost every change made since Frigate started.
"""
from fastapi import Request
from frigate.api.auth import get_allowed_cameras_for_filter, get_current_user
from frigate.api.fastapi_app import create_fastapi_app
config_path = self._write_config_file()
mock_find_config.return_value = config_path
mock_publisher = Mock(spec=CameraConfigUpdatePublisher)
mock_publisher.publisher = MagicMock()
boot_config = FrigateConfig(**self.minimal_config)
holder = ConfigHolder(boot_config)
try:
app = create_fastapi_app(
boot_config,
self.db,
None,
None,
None,
None,
None,
None,
mock_publisher,
None,
enforce_default_admin=False,
config_holder=holder,
)
async def mock_get_current_user(request: Request):
return {"username": "admin", "role": "admin"}
async def mock_get_allowed_cameras_for_filter(request: Request):
return list(self.minimal_config.get("cameras", {}).keys())
app.dependency_overrides[get_current_user] = mock_get_current_user
app.dependency_overrides[get_allowed_cameras_for_filter] = (
mock_get_allowed_cameras_for_filter
)
with AuthTestClient(app) as client:
resp = client.put(
"/config/set",
json={
"config_data": {"birdseye": {"inactivity_threshold": 5}},
"update_topic": "config/birdseye",
"requires_restart": 0,
},
)
self.assertEqual(resp.status_code, 200)
self.assertIsNot(holder.config, boot_config)
self.assertIs(holder.config, app.frigate_config)
self.assertEqual(holder.config.birdseye.inactivity_threshold, 5)
finally:
os.unlink(config_path)
if __name__ == "__main__":
unittest.main()
+31
View File
@@ -4,6 +4,7 @@ import unittest
from unittest.mock import MagicMock
from frigate.api.config_util import swap_runtime_config
from frigate.config.holder import ConfigHolder
class TestSwapRuntimeConfig(unittest.TestCase):
@@ -12,6 +13,7 @@ class TestSwapRuntimeConfig(unittest.TestCase):
def _make_app(self) -> MagicMock:
app = MagicMock()
app.dispatcher.comms = [MagicMock(), MagicMock()]
app.config_holder = ConfigHolder(MagicMock(name="boot_config"))
return app
def test_rebinds_all_references(self) -> None:
@@ -37,11 +39,40 @@ class TestSwapRuntimeConfig(unittest.TestCase):
# the swap rebuilds cameras from yaml, so overrides must be re-layered
app.dispatcher.reapply_runtime_state_to_config.assert_called_once_with()
def test_updates_the_config_holder(self) -> None:
app = self._make_app()
holder = app.config_holder
config = MagicMock(name="new_config")
swap_runtime_config(app, config)
self.assertIs(holder.config, config)
def test_deferred_factory_builds_from_the_swapped_config(self) -> None:
"""A watchdog-style factory must not rebuild from the boot config.
The factories in FrigateApp are lambdas evaluated when a process is
restarted, long after a user may have saved. Reading through the
holder is what keeps a rebuilt process from reverting every change
made since Frigate started.
"""
app = self._make_app()
holder = app.config_holder
boot_config = holder.config
factory = lambda: holder.config # noqa: E731
self.assertIs(factory(), boot_config)
config = MagicMock(name="new_config")
swap_runtime_config(app, config)
self.assertIs(factory(), config)
def test_tolerates_missing_optional_collaborators(self) -> None:
app = MagicMock()
app.profile_manager = None
app.stats_emitter = None
app.dispatcher = None
app.config_holder = None
config = MagicMock(name="new_config")
# must not raise when the optional collaborators are absent
+226
View File
@@ -0,0 +1,226 @@
"""Tests for detector post-processing NMS box format handling.
cv2.dnn.NMSBoxes expects boxes as [x, y, width, height]. Passing corner
coordinates [x1, y1, x2, y2] makes OpenCV treat x2/y2 as width/height,
inflating every box toward the bottom-right by its distance from the origin.
Two well separated objects far from the origin then appear to overlap and the
lower scoring one is silently suppressed.
The regression geometry used throughout: two boxes with zero true overlap,
A = (393, 499, 484, 620) and B = (527, 499, 618, 620) in a 640x640 input
(43 px gap). Misread as [x, y, w, h] their IoU is 0.465, above the 0.4 NMS
threshold, so the buggy format drops the lower scoring box while correct
conversion keeps both.
"""
import math
import unittest
from queue import Queue
import numpy as np
from frigate.detectors.plugins.memryx import MemryXDetector
from frigate.util.model import (
post_process_dfine,
post_process_rfdetr,
post_process_yolo,
post_process_yolox,
)
WIDTH = 640
HEIGHT = 640
# box A: xyxy (393, 499, 484, 620) as center format
A_CX, A_CY, A_W, A_H = 438.5, 559.5, 91.0, 121.0
# box B: xyxy (527, 499, 618, 620) as center format
B_CX, B_CY, B_W, B_H = 572.5, 559.5, 91.0, 121.0
# expected normalized output rows: [class_id, conf, y1, x1, y2, x2]
A_ROW = [499 / 640, 393 / 640, 620 / 640, 484 / 640]
B_ROW = [499 / 640, 527 / 640, 620 / 640, 618 / 640]
def kept(detections: np.ndarray) -> np.ndarray:
"""Rows of the padded (20, 6) output that hold real detections."""
return detections[detections[:, 1] > 0]
class TestYoloNmsPostProcess(unittest.TestCase):
def _single_output(self, rows: list[list[float]]) -> list[np.ndarray]:
"""Build a single-tensor YOLO output (1, attrs, anchors) from
[cx, cy, w, h, class scores...] rows, padded with empty anchors."""
anchors = np.zeros((10, len(rows[0])), dtype=np.float32)
anchors[: len(rows)] = np.array(rows, dtype=np.float32)
return [anchors.T[np.newaxis, ...]]
def test_keeps_separated_objects_far_from_origin(self):
output = self._single_output(
[
[A_CX, A_CY, A_W, A_H, 0.90, 0.0],
[B_CX, B_CY, B_W, B_H, 0.0, 0.85],
]
)
detections = kept(post_process_yolo(output, WIDTH, HEIGHT))
self.assertEqual(len(detections), 2)
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
def test_still_suppresses_true_duplicates(self):
# same object twice, shifted 4 px: true IoU 0.92, must dedupe to one
output = self._single_output(
[
[A_CX, A_CY, A_W, A_H, 0.90, 0.0],
[A_CX + 4, A_CY, A_W, A_H, 0.85, 0.0],
]
)
detections = kept(post_process_yolo(output, WIDTH, HEIGHT))
self.assertEqual(len(detections), 1)
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
class TestMultipartYoloPostProcess(unittest.TestCase):
def _multipart_output(self) -> list[np.ndarray]:
"""Build a 3-scale anchor-based YOLO output containing boxes A and B,
both decoded through anchor 0 of the stride-32 scale."""
outputs = [
np.zeros((1, 255, 80, 80), dtype=np.float32),
np.zeros((1, 255, 40, 40), dtype=np.float32),
np.zeros((1, 255, 20, 20), dtype=np.float32),
]
stride, (anchor_w, anchor_h) = 32, (142, 110)
for cx, cy, w, h, conf, class_channel in [
(A_CX, A_CY, A_W, A_H, 0.95, 5), # class 0
(B_CX, B_CY, B_W, B_H, 0.90, 6), # class 1
]:
cell_x, cell_y = int(cx // stride), int(cy // stride)
dx = (cx / stride - cell_x + 0.5) / 2
dy = (cy / stride - cell_y + 0.5) / 2
dw = math.sqrt(w / anchor_w) / 2
dh = math.sqrt(h / anchor_h) / 2
# anchor 0 occupies channels 0-84 of the 255 channel tensor
outputs[2][0, 0:4, cell_y, cell_x] = [dx, dy, dw, dh]
outputs[2][0, 4, cell_y, cell_x] = conf
outputs[2][0, class_channel, cell_y, cell_x] = 1.0
return outputs
def test_keeps_separated_objects_far_from_origin(self):
detections = kept(post_process_yolo(self._multipart_output(), WIDTH, HEIGHT))
self.assertEqual(len(detections), 2)
np.testing.assert_allclose(detections[0], [0, 0.95, *A_ROW], atol=2e-3)
np.testing.assert_allclose(detections[1], [1, 0.90, *B_ROW], atol=2e-3)
def test_empty_output_returns_no_detections(self):
outputs = [
np.zeros((1, 255, 80, 80), dtype=np.float32),
np.zeros((1, 255, 40, 40), dtype=np.float32),
np.zeros((1, 255, 20, 20), dtype=np.float32),
]
detections = kept(post_process_yolo(outputs, WIDTH, HEIGHT))
self.assertEqual(len(detections), 0)
class TestYoloxPostProcess(unittest.TestCase):
def test_keeps_separated_objects_far_from_origin(self):
# with zero grids and unit strides the decode reduces to
# cx = raw cx and w = exp(raw w)
rows = np.zeros((10, 7), dtype=np.float32)
rows[0] = [A_CX, A_CY, math.log(A_W), math.log(A_H), 1.0, 0.90, 0.0]
rows[1] = [B_CX, B_CY, math.log(B_W), math.log(B_H), 1.0, 0.0, 0.85]
predictions = rows[np.newaxis, ...]
grids = np.zeros((1, 10, 2), dtype=np.float32)
expanded_strides = np.ones((1, 10, 1), dtype=np.float32)
detections = kept(
post_process_yolox(predictions, WIDTH, HEIGHT, grids, expanded_strides)
)
self.assertEqual(len(detections), 2)
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
class TestDfinePostProcess(unittest.TestCase):
def test_keeps_separated_objects_far_from_origin(self):
# D-FINE emits absolute pixel xyxy boxes alongside labels and scores
labels = np.zeros((1, 10), dtype=np.int64)
labels[0, 1] = 1
boxes = np.zeros((1, 10, 4), dtype=np.float32)
boxes[0, 0] = [393, 499, 484, 620]
boxes[0, 1] = [527, 499, 618, 620]
scores = np.zeros((1, 10), dtype=np.float32)
scores[0, 0] = 0.90
scores[0, 1] = 0.85
detections = kept(post_process_dfine([labels, boxes, scores], WIDTH, HEIGHT))
self.assertEqual(len(detections), 2)
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
class TestRfdetrPostProcess(unittest.TestCase):
def test_keeps_separated_objects_far_from_origin(self):
# RF-DETR emits normalized center format boxes and class logits where
# logit index 0 is the background class
boxes = np.zeros((1, 10, 4), dtype=np.float32)
boxes[0, 0] = [A_CX / WIDTH, A_CY / HEIGHT, A_W / WIDTH, A_H / HEIGHT]
boxes[0, 1] = [B_CX / WIDTH, B_CY / HEIGHT, B_W / WIDTH, B_H / HEIGHT]
# background heavy logits everywhere, then two confident objects
logits = np.tile(np.array([10.0, 0.0, 0.0], dtype=np.float32), (1, 10, 1))
logits[0, 0] = [0.0, 4.0, 0.0] # class 0 after background offset
logits[0, 1] = [0.0, 0.0, 3.5] # class 1 after background offset
detections = kept(post_process_rfdetr([boxes, logits]))
conf_a = math.exp(4.0) / (math.exp(4.0) + 2)
conf_b = math.exp(3.5) / (math.exp(3.5) + 2)
self.assertEqual(len(detections), 2)
np.testing.assert_allclose(detections[0], [0, conf_a, *A_ROW], atol=2e-3)
np.testing.assert_allclose(detections[1], [1, conf_b, *B_ROW], atol=2e-3)
class TestMemryxSsdlitePostProcess(unittest.TestCase):
def test_keeps_separated_objects_far_from_origin(self):
# the NMS math runs on the host CPU, so the real method is testable
# without MemryX hardware; it only needs the model dimensions and
# the output queue
detector = object.__new__(MemryXDetector)
detector.memx_model_width = WIDTH
detector.memx_model_height = HEIGHT
detector.output_queue = Queue()
# this path uses a 0.5 NMS threshold, so use a tighter pair: zero
# true overlap (10 px gap), IoU 0.69 when misread as [x, y, w, h]
dets = np.zeros((1, 10, 5), dtype=np.float32)
dets[0, 0] = [480, 500, 540, 620, 0.90]
dets[0, 1] = [550, 500, 610, 620, 0.85]
labels = np.zeros((1, 10), dtype=np.float32)
labels[0, 1] = 1
detector.post_process_ssdlite([dets, labels])
detections = kept(detector.output_queue.get())
self.assertEqual(len(detections), 2)
np.testing.assert_allclose(
detections[0],
[0, 0.90, 500 / 640, 480 / 640, 620 / 640, 540 / 640],
atol=2e-3,
)
np.testing.assert_allclose(
detections[1],
[1, 0.85, 500 / 640, 550 / 640, 620 / 640, 610 / 640],
atol=2e-3,
)
if __name__ == "__main__":
unittest.main()
+12
View File
@@ -472,6 +472,18 @@ def sanitize_float(value):
return value
def has_non_finite_number(value: Any) -> bool:
"""Return True if any number in a parsed JSON value is NaN or infinite."""
if isinstance(value, float):
return not math.isfinite(value)
if isinstance(value, dict):
return any(has_non_finite_number(v) for v in value.values())
if isinstance(value, list):
return any(has_non_finite_number(v) for v in value)
return False
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
return 1 - cosine_distance(a, b)
+37 -5
View File
@@ -16,6 +16,31 @@ logger = logging.getLogger(__name__)
### Post Processing
def xyxy_to_xywh_for_nms(boxes: np.ndarray | list) -> np.ndarray:
"""Convert [x1, y1, x2, y2] boxes to the [x, y, width, height] format
that cv2.dnn.NMSBoxes expects.
Passing corner coordinates directly makes OpenCV treat x2/y2 as the box
size, inflating every box toward the bottom-right by its distance from
the origin, which suppresses valid detections near other objects.
Args:
boxes: Array-like of shape (N, 4) in corner format.
Returns:
Float32 array of shape (N, 4) in top-left plus size format.
"""
boxes = np.asarray(boxes, dtype=np.float32)
if boxes.size == 0:
return np.zeros((0, 4), dtype=np.float32)
xywh = boxes.copy()
xywh[:, 2] -= xywh[:, 0]
xywh[:, 3] -= xywh[:, 1]
return xywh
def post_process_dfine(
tensor_output: np.ndarray, width: int, height: int
) -> np.ndarray:
@@ -25,7 +50,9 @@ def post_process_dfine(
input_shape = np.array([height, width, height, width])
boxes = np.divide(boxes, input_shape, dtype=np.float32)
indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.4, nms_threshold=0.4)
indices = cv2.dnn.NMSBoxes(
xyxy_to_xywh_for_nms(boxes), scores, score_threshold=0.4, nms_threshold=0.4
)
detections = np.zeros((20, 6), np.float32)
for i, (bbox, confidence, class_id) in enumerate(
@@ -78,7 +105,10 @@ def post_process_rfdetr(tensor_output: list[np.ndarray, np.ndarray]) -> np.ndarr
# apply nms
indices = cv2.dnn.NMSBoxes(
filtered_boxes, filtered_scores, score_threshold=0.4, nms_threshold=0.4
xyxy_to_xywh_for_nms(filtered_boxes),
filtered_scores,
score_threshold=0.4,
nms_threshold=0.4,
)
detections = np.zeros((20, 6), np.float32)
@@ -159,7 +189,7 @@ def __post_process_multipart_yolo(
all_class_ids.append(class_id)
indices = cv2.dnn.NMSBoxes(
bboxes=all_boxes,
bboxes=xyxy_to_xywh_for_nms(all_boxes),
scores=all_scores,
score_threshold=0.4,
nms_threshold=0.4,
@@ -206,7 +236,9 @@ def __post_process_nms_yolo(predictions: np.ndarray, width, height) -> np.ndarra
boxes = boxes_xyxy
# run NMS
indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.4, nms_threshold=0.4)
indices = cv2.dnn.NMSBoxes(
xyxy_to_xywh_for_nms(boxes), scores, score_threshold=0.4, nms_threshold=0.4
)
detections = np.zeros((20, 6), np.float32)
for i, (bbox, confidence, class_id) in enumerate(
zip(boxes[indices], scores[indices], class_ids[indices])
@@ -258,7 +290,7 @@ def post_process_yolox(
scores = scores[np.arange(len(cls_inds)), cls_inds]
indices = cv2.dnn.NMSBoxes(
boxes_xyxy, scores, score_threshold=0.4, nms_threshold=0.4
xyxy_to_xywh_for_nms(boxes_xyxy), scores, score_threshold=0.4, nms_threshold=0.4
)
detections = np.zeros((20, 6), np.float32)
+30 -38
View File
@@ -36,8 +36,8 @@ from __future__ import annotations
import argparse
import os
import sys
from collections.abc import Iterable
from dataclasses import dataclass
from typing import Iterable
import cv2
import numpy as np
@@ -53,21 +53,31 @@ ARCFACE_INPUT_SIZE = 112
# ---------------------------------------------------------------------------
def _bgr_to_rgb(frame: np.ndarray) -> np.ndarray:
"""Mirror BaseEmbedding._bgr_to_rgb."""
if isinstance(frame, np.ndarray) and frame.ndim == 3:
return np.ascontiguousarray(frame[:, :, ::-1])
return frame
def _process_image_frigate(image: np.ndarray) -> Image.Image:
"""Mirror BaseEmbedding._process_image for an ndarray input.
NOTE: Frigate passes the output of `cv2.imread` (BGR) directly in. PIL's
`Image.fromarray` does NOT reorder channels, so the embedder effectively
receives a BGR-ordered tensor. We replicate that faithfully here. (Tested
— swapping to RGB produces near-identical embeddings; this model is
robust to channel order.)
`Image.fromarray` does not reorder channels, so whatever order it is
handed is what reaches the model. Callers swap to RGB first, exactly as
ArcfaceEmbedding._preprocess_inputs does.
"""
return Image.fromarray(image)
def arcface_preprocess(image_bgr: np.ndarray) -> np.ndarray:
"""Mirror ArcfaceEmbedding._preprocess_inputs."""
pil = _process_image_frigate(image_bgr)
"""Mirror ArcfaceEmbedding._preprocess_inputs.
Face crops arrive BGR from cv2 and #23712 added the swap to RGB before
embedding, so this script has to do it too.
"""
pil = _process_image_frigate(_bgr_to_rgb(image_bgr))
width, height = pil.size
if width != ARCFACE_INPUT_SIZE or height != ARCFACE_INPUT_SIZE:
@@ -138,9 +148,7 @@ class LandmarkAligner:
M[0, 2] += tX - eyesCenter[0]
M[1, 2] += tY - eyesCenter[1]
aligned = cv2.warpAffine(
image, M, (out_w, out_h), flags=cv2.INTER_CUBIC
)
aligned = cv2.warpAffine(image, M, (out_w, out_h), flags=cv2.INTER_CUBIC)
info = dict(
angle=float(angle),
eye_dist_px=dist,
@@ -433,9 +441,7 @@ def vector_outlier_test(
if neg
else np.array([])
)
baseline_conf_neg = np.array(
[similarity_to_confidence(c) for c in baseline_neg]
)
baseline_conf_neg = np.array([similarity_to_confidence(c) for c in baseline_neg])
print(
f"\nBaseline (trim_mean only, {len(pos)} images):"
@@ -465,9 +471,7 @@ def vector_outlier_test(
mean, keep = iterative_mean(all_embs, T)
pos_sims = np.array([cosine(p.embedding, mean) for p in pos])
neg_sims = (
np.array([cosine(n.embedding, mean) for n in neg])
if neg
else np.array([])
np.array([cosine(n.embedding, mean) for n in neg]) if neg else np.array([])
)
neg_conf = np.array([similarity_to_confidence(c) for c in neg_sims])
margin = pos_sims.min() - (neg_sims.max() if len(neg_sims) else 0)
@@ -483,9 +487,7 @@ def vector_outlier_test(
# Show which images get dropped at the shipped threshold + neighbors
for T_show in (0.25, 0.30, 0.33):
_, keep = iterative_mean(all_embs, T_show)
print(
f"\nAt T={T_show}, the {int((~keep).sum())} dropped positives are:"
)
print(f"\nAt T={T_show}, the {int((~keep).sum())} dropped positives are:")
final_mean = stats.trim_mean(all_embs[keep], base_trim, axis=0)
m_n = final_mean / (np.linalg.norm(final_mean) + 1e-9)
for i, (p, k) in enumerate(zip(pos, keep)):
@@ -501,9 +503,7 @@ def vector_outlier_test(
)
def degenerate_embedding_test(
pos: list[FaceSample], neg: list[FaceSample]
) -> None:
def degenerate_embedding_test(pos: list[FaceSample], neg: list[FaceSample]) -> None:
"""Detect whether negatives and low-quality positives share a degenerate
'tiny/noisy face' region of the embedding space.
@@ -533,8 +533,7 @@ def degenerate_embedding_test(
f"(how tightly negatives cluster together)"
)
print(
f" pos<->pos mean cos : {np.nanmean(pp):.3f} "
f"(how tightly positives cluster)"
f" pos<->pos mean cos : {np.nanmean(pp):.3f} (how tightly positives cluster)"
)
print(
f" pos<->neg mean cos : {pn.mean():.3f} "
@@ -558,11 +557,7 @@ def degenerate_embedding_test(
neg_scores = np.array([cosine(n.embedding, clean_mean) for n in neg])
neg_confs = np.array([similarity_to_confidence(c) for c in neg_scores])
pos_scores = np.array(
[
cosine(pos[i].embedding, clean_mean)
for i in range(len(pos))
if keep[i]
]
[cosine(pos[i].embedding, clean_mean) for i in range(len(pos)) if keep[i]]
)
print(
f"\n mean_intra >= {thresh}: keeping {int(keep.sum())}/{len(pos)} positives"
@@ -585,9 +580,7 @@ def degenerate_embedding_test(
)
def contamination_analysis(
pos: list[FaceSample], neg: list[FaceSample]
) -> None:
def contamination_analysis(pos: list[FaceSample], neg: list[FaceSample]) -> None:
"""Check whether the positive collection contains a second identity.
Two signals:
@@ -617,10 +610,7 @@ def contamination_analysis(
"\nPositives closer to a negative than to their own class avg"
"\n(these are candidates for mislabeled images):"
)
print(
f"\n{'max_neg':>7} {'mean_neg':>8} {'mean_intra':>10} "
f"{'delta':>6} name"
)
print(f"\n{'max_neg':>7} {'mean_neg':>8} {'mean_intra':>10} {'delta':>6} name")
rows = list(zip(pos_names, max_to_neg, mean_to_neg, mean_intra))
rows.sort(key=lambda r: -(r[1] - r[3]))
for nm, mxn, mnn, mi in rows[:15]:
@@ -704,7 +694,9 @@ def main() -> int:
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
ap.add_argument("--positive", required=True, help="Training folder for one identity")
ap.add_argument(
"--positive", required=True, help="Training folder for one identity"
)
ap.add_argument(
"--negative",
default=None,
+2 -2
View File
@@ -859,8 +859,8 @@
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
},
"dashboard": {
"label": "Show in UI",
"description": "Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again."
"label": "Show on Live dashboard",
"description": "Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings."
},
"review": {
"label": "Show in review",
+2 -18
View File
@@ -357,22 +357,6 @@
"description": "Optional API key for authenticated DeepStack services."
}
},
"degirum": {
"label": "DeGirum",
"description": "DeGirum detector for running models via DeGirum cloud or local inference services.",
"location": {
"label": "Inference Location",
"description": "Location of the DeGirim inference engine (e.g. '@cloud', '127.0.0.1')."
},
"zoo": {
"label": "Model Zoo",
"description": "Path or URL to the DeGirum model zoo."
},
"token": {
"label": "DeGirum Cloud Token",
"description": "Token for DeGirum Cloud access."
}
},
"edgetpu": {
"label": "EdgeTPU",
"description": "EdgeTPU detector that runs TensorFlow Lite models compiled for Coral EdgeTPU using the EdgeTPU delegate.",
@@ -1543,8 +1527,8 @@
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
},
"dashboard": {
"label": "Show in UI",
"description": "Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again."
"label": "Show on Live dashboard",
"description": "Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings."
},
"review": {
"label": "Show in review",
+1 -1
View File
@@ -499,7 +499,7 @@
"webuiUrlHelp": "URL to visit the camera's web UI directly from the Debug view. Leave blank to disable the link.",
"webuiUrlInvalid": "Must be a valid URL (e.g., https://example.com).",
"dashboardLabel": "Show on Live dashboard",
"dashboardHelp": "Show this camera on the Live dashboard.",
"dashboardHelp": "Show this camera on the default All Cameras live dashboard. It remains available everywhere else, including camera groups.",
"reviewLabel": "Show in Review",
"reviewHelp": "Show this camera in Review, including the camera filter, motion review, and the history view."
}
@@ -14,6 +14,7 @@ const logger: SectionConfigOverrides = {
additionalProperties: {
"ui:options": {
enumI18nPrefix: "logger.logLevel",
additionalPropertyKeySize: "lg",
additionalPropertyKeyLabel:
"configForm.additionalProperties.loggerNameLabel",
additionalPropertyKeyPlaceholder:
@@ -15,6 +15,14 @@ import { useTranslation } from "react-i18next";
import { LuTrash2 } from "react-icons/lu";
import type { ConfigFormContext } from "@/types/configForm";
const KEY_SIZE_CLASSES = {
sm: { key: "md:col-span-2", value: "md:col-span-9" },
md: { key: "md:col-span-4", value: "md:col-span-7" },
lg: { key: "md:col-span-7", value: "md:col-span-4" },
} as const;
type AdditionalPropertyKeySize = keyof typeof KEY_SIZE_CLASSES;
export function WrapIfAdditionalTemplate<
T = unknown,
S extends StrictRJSFSchema = RJSFSchema,
@@ -58,6 +66,14 @@ export function WrapIfAdditionalTemplate<
: undefined;
const preventKeyRename = uiOptions.preventKeyRename === true;
const keySize =
typeof uiOptions.additionalPropertyKeySize === "string" &&
uiOptions.additionalPropertyKeySize in KEY_SIZE_CLASSES
? (uiOptions.additionalPropertyKeySize as AdditionalPropertyKeySize)
: "sm";
const keySpanClass = KEY_SIZE_CLASSES[keySize].key;
const valueSpanClass = KEY_SIZE_CLASSES[keySize].value;
const formContext = registry?.formContext as ConfigFormContext | undefined;
// optionally, lock the key once it's been saved
@@ -126,7 +142,7 @@ export function WrapIfAdditionalTemplate<
style={style}
>
{!keyIsReadonly && (
<div className="col-span-12 space-y-2 md:col-span-2">
<div className={cn("col-span-12 space-y-2", keySpanClass)}>
{displayLabel && <Label htmlFor={keyId}>{keyLabel}</Label>}
{keyLocked ? (
<div
@@ -158,7 +174,7 @@ export function WrapIfAdditionalTemplate<
<div
className={cn(
"col-span-12 space-y-2",
!keyIsReadonly && "md:col-span-9",
!keyIsReadonly && valueSpanClass,
)}
>
{!keyIsReadonly && displayLabel && (
+1 -1
View File
@@ -78,7 +78,7 @@ export default function ZoneEditPane({
}
return Object.values(config.cameras)
.filter((conf) => conf.ui.dashboard && conf.enabled_in_config)
.filter((conf) => conf.enabled_in_config)
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
}, [config]);
+42 -23
View File
@@ -198,6 +198,19 @@ export default function Events() {
return true;
});
const reviewCamerasParam = useMemo(() => {
const selected: string | undefined = reviewSearchParams["cameras"];
if (!selected) {
return reviewCameras.join(",");
}
const selectedCameras = new Set(selected.split(","));
return reviewCameras
.filter((camera) => selectedCameras.has(camera))
.join(",");
}, [reviewCameras, reviewSearchParams]);
useSearchEffect("labels", (labels: string) => {
setReviewFilter({
...reviewFilter,
@@ -330,8 +343,12 @@ export default function Events() {
}, []);
const getKey = useCallback(() => {
if (!timezone) {
return null;
}
const params = {
cameras: reviewSearchParams["cameras"],
cameras: reviewCamerasParam,
labels: reviewSearchParams["labels"],
zones: reviewSearchParams["zones"],
reviewed: null, // We want both reviewed and unreviewed items as we filter in the UI
@@ -339,7 +356,7 @@ export default function Events() {
after: reviewSearchParams["after"] || last24Hours.after,
};
return ["review", params];
}, [reviewSearchParams, last24Hours]);
}, [reviewSearchParams, reviewCamerasParam, last24Hours, timezone]);
const { data: reviews, mutate: updateSegments } = useSWR<ReviewSegment[]>(
getKey,
@@ -361,10 +378,6 @@ export default function Events() {
const motion: ReviewSegment[] = [];
reviews?.forEach((segment) => {
if (config?.cameras[segment.camera]?.ui?.review === false) {
return;
}
all.push(segment);
switch (segment.severity) {
@@ -386,7 +399,7 @@ export default function Events() {
detection: detections,
significant_motion: motion,
};
}, [reviews, config?.cameras]);
}, [reviews]);
// update review items in place when a review segment ends
const reviewUpdate = useFrigateReviews();
@@ -450,15 +463,17 @@ export default function Events() {
// review summary
const { data: reviewSummary, mutate: updateSummary } = useSWR<ReviewSummary>(
[
"review/summary",
{
timezone: timezone,
cameras: reviewSearchParams["cameras"] ?? null,
labels: reviewSearchParams["labels"] ?? null,
zones: reviewSearchParams["zones"] ?? null,
},
],
timezone
? [
"review/summary",
{
timezone: timezone,
cameras: reviewCamerasParam,
labels: reviewSearchParams["labels"] ?? null,
zones: reviewSearchParams["zones"] ?? null,
},
]
: null,
{
revalidateOnFocus: true,
refreshInterval: 30000,
@@ -473,13 +488,17 @@ export default function Events() {
// recordings summary
const { data: recordingsSummary } = useSWR<RecordingsSummary>([
"recordings/summary",
{
timezone: timezone,
cameras: reviewSearchParams["cameras"] ?? null,
},
]);
const { data: recordingsSummary } = useSWR<RecordingsSummary>(
timezone
? [
"recordings/summary",
{
timezone: timezone,
cameras: reviewCamerasParam,
},
]
: null,
);
// preview videos
const previewTimes = useMemo(() => {
+1 -6
View File
@@ -706,12 +706,7 @@ export default function Settings() {
}
return Object.values(config.cameras)
.filter(
(conf) =>
conf.ui.dashboard &&
conf.enabled_in_config &&
!isReplayCamera(conf.name),
)
.filter((conf) => conf.enabled_in_config && !isReplayCamera(conf.name))
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
}, [config]);
+10 -2
View File
@@ -1024,6 +1024,9 @@ function MotionReview({
if (!allowedCameras.includes(cam.name)) {
return false;
}
if (cam.ui?.review === false) {
return false;
}
if (selectedCams && !selectedCams.includes(cam.name)) {
return false;
}
@@ -1033,6 +1036,11 @@ function MotionReview({
return cameras.sort((a, b) => a.ui.order - b.ui.order);
}, [config, filter, allowedCameras]);
const reviewCamerasParam = useMemo(
() => reviewCameras.map((cam) => cam.name).join(","),
[reviewCameras],
);
const videoPlayersRef = useRef<{ [camera: string]: PreviewController }>({});
// motion data
@@ -1052,7 +1060,7 @@ function MotionReview({
before: alignedBefore,
after: alignedAfter,
scale: segmentDuration / 2,
cameras: filter?.cameras?.join(",") ?? null,
cameras: reviewCamerasParam,
},
]);
@@ -1061,7 +1069,7 @@ function MotionReview({
{
before: alignedBefore,
after: alignedAfter,
cameras: filter?.cameras?.join(",") ?? null,
cameras: reviewCamerasParam,
},
]);
+33 -6
View File
@@ -540,6 +540,36 @@ export default function GeneralMetrics({
return Object.keys(series).length > 0 ? Object.values(series) : undefined;
}, [statsHistory]);
// Number of cards the hardware grid renders. Which ones appear depends on
// the vendor, so the column count follows the count rather than assuming a
// fixed set is present.
const hardwareCardCount = useMemo(() => {
if (!statsHistory[0]?.gpu_usages) {
return 0;
}
const hasNpu = statsHistory[0].npu_usages != undefined;
return (
1 + // gpu usage always renders alongside gpu_usages
(gpuMemSeries ? 1 : 0) +
(gpuEncSeries?.length ? 1 : 0) +
(gpuComputeSeries?.length ? 1 : 0) +
(gpuDecSeries?.length ? 1 : 0) +
(gpuTempSeries?.length ? 1 : 0) +
(hasNpu ? 1 : 0) +
(hasNpu && npuTempSeries?.length ? 1 : 0)
);
}, [
statsHistory,
gpuMemSeries,
gpuEncSeries,
gpuComputeSeries,
gpuDecSeries,
gpuTempSeries,
npuTempSeries,
]);
// other processes stats
const hardwareType = useMemo(() => {
@@ -763,12 +793,9 @@ export default function GeneralMetrics({
<div
className={cn(
"mt-4 grid grid-cols-1 gap-2 sm:grid-cols-2",
gpuTempSeries?.length && "md:grid-cols-3",
(gpuEncSeries?.length || gpuComputeSeries?.length) &&
"xl:grid-cols-4",
(gpuEncSeries?.length || gpuComputeSeries?.length) &&
gpuTempSeries?.length &&
"3xl:grid-cols-5",
hardwareCardCount >= 3 && "lg:grid-cols-3",
hardwareCardCount >= 4 && "xl:grid-cols-4",
hardwareCardCount >= 5 && "3xl:grid-cols-5",
)}
>
{statsHistory[0]?.gpu_usages && (