mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-08-02 00:52:17 +03:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4f2a297745 | ||
|
|
b848c90f02 | ||
|
|
f1cc0e49d4 |
@@ -1,7 +1,7 @@
|
||||
default_target: local
|
||||
|
||||
COMMIT_HASH := $(shell git log -1 --pretty=format:"%h"|tail -1)
|
||||
VERSION = 0.19.0
|
||||
VERSION = 0.18.0
|
||||
IMAGE_REPO ?= ghcr.io/blakeblackshear/frigate
|
||||
GITHUB_REF_NAME ?= $(shell git rev-parse --abbrev-ref HEAD)
|
||||
BOARDS= #Initialized empty
|
||||
|
||||
@@ -79,7 +79,5 @@ sherpa-onnx==1.12.*
|
||||
faster-whisper==1.1.*
|
||||
librosa==0.11.*
|
||||
soundfile==0.13.*
|
||||
# DeGirum detector
|
||||
degirum == 0.16.*
|
||||
# Memory profiling
|
||||
memray == 1.15.*
|
||||
|
||||
@@ -1269,78 +1269,3 @@ axengine:
|
||||
input_dtype: int
|
||||
input_pixel_format: bgr
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
degirumAiServer:
|
||||
title: DeGirum AI Server
|
||||
models:
|
||||
- key: ai-server-inference
|
||||
label: AI Server Inference
|
||||
recommended: true
|
||||
download: |-
|
||||
Launch a DeGirum AI server as a Docker container, then point the detector at it. Add this to your `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
degirum_detector:
|
||||
container_name: degirum
|
||||
image: degirum/aiserver:latest
|
||||
privileged: true
|
||||
ports:
|
||||
- "8778:8778"
|
||||
```
|
||||
|
||||
Set `location` to the server's service name, container name, or `host:port`.
|
||||
ui: |
|
||||
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| **Location** | `degirum` |
|
||||
| **Zoo** | `degirum/public` |
|
||||
| **Token** | your AI Hub token (optional for the public zoo) |
|
||||
yaml: |
|
||||
degirum_detector:
|
||||
type: degirum
|
||||
location: degirum
|
||||
zoo: degirum/public
|
||||
token: dg_example_token
|
||||
degirumLocal:
|
||||
title: DeGirum Local
|
||||
models:
|
||||
- key: local-inference
|
||||
label: Local Inference
|
||||
recommended: true
|
||||
download: Run hardware directly inside the Frigate container with `@local`, removing the AI server hop. The matching device runtime (e.g. the Hailo runtime) must be installed in the container; confirm it with `degirum sys-info`.
|
||||
ui: |
|
||||
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| **Location** | `@local` |
|
||||
| **Zoo** | `degirum/public` |
|
||||
| **Token** | your AI Hub token (optional for the public zoo) |
|
||||
yaml: |
|
||||
degirum_detector:
|
||||
type: degirum
|
||||
location: @local
|
||||
zoo: degirum/public
|
||||
token: dg_example_token
|
||||
degirumCloud:
|
||||
title: DeGirum AI Hub Cloud
|
||||
models:
|
||||
- key: ai-hub-cloud-inference
|
||||
label: AI Hub Cloud Inference
|
||||
recommended: true
|
||||
download: Run inferences on DeGirum's [AI Hub](https://hub.degirum.com) cloud with `@cloud`. Sign up, create an access token, and set it as `token`. Network latency may require lowering your detection fps.
|
||||
ui: |
|
||||
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| **Location** | `@cloud` |
|
||||
| **Zoo** | `degirum/public` |
|
||||
| **Token** | your AI Hub token (optional for the public zoo) |
|
||||
yaml: |
|
||||
degirum_detector:
|
||||
type: degirum
|
||||
location: @cloud
|
||||
zoo: degirum/public
|
||||
token: dg_example_token
|
||||
|
||||
@@ -981,7 +981,9 @@ cameras:
|
||||
# Optional: Adjust sort order of cameras in the UI. Larger numbers come later (default: shown below)
|
||||
# By default the cameras are sorted alphabetically.
|
||||
order: 0
|
||||
# Optional: Whether or not to show the camera in the Frigate UI (default: shown below)
|
||||
# Optional: Whether or not to show the camera on the default All Cameras live dashboard.
|
||||
# The camera is still available everywhere else, including camera groups and settings
|
||||
# (default: shown below)
|
||||
dashboard: True
|
||||
# Optional: Whether this camera is visible in review (the review page and its camera
|
||||
# filter, motion review, and the history view) (default: shown below)
|
||||
|
||||
@@ -334,7 +334,7 @@ When your browser runs into problems playing back your camera streams, it will l
|
||||
|
||||
- **stalled**
|
||||
- What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving).
|
||||
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in the UI pane of Frigate's settings.
|
||||
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in <NavPath path="Settings > UI" /> .
|
||||
|
||||
- Possible console messages from the player code:
|
||||
- `Buffer time (10 seconds) exceeded, browser may not be playing media correctly.`
|
||||
|
||||
@@ -24,7 +24,6 @@ Frigate supports multiple different detectors that work on different types of ha
|
||||
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
|
||||
- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
|
||||
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
|
||||
- <CommunityBadge /> [DeGirum](#degirum): Service for using hardware devices in the cloud or locally. Hardware and models provided on the cloud on [their website](https://hub.degirum.com).
|
||||
|
||||
**AMD**
|
||||
|
||||
@@ -755,87 +754,6 @@ Explanation of the parameters:
|
||||
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
|
||||
- `config`: Configuration passed to `rknn-toolkit2` for model conversion. For an explanation of all available parameters have a look at section "2.2. Model configuration" of [this manual](https://github.com/MarcA711/rknn-toolkit2/releases/download/v2.3.2/03_Rockchip_RKNPU_API_Reference_RKNN_Toolkit2_V2.3.2_EN.pdf).
|
||||
|
||||
## DeGirum
|
||||
|
||||
DeGirum is a detector that can use any type of hardware listed on [their website](https://hub.degirum.com). DeGirum can be used with local hardware through a DeGirum AI Server, or through the use of `@local`. You can also connect directly to DeGirum's AI Hub to run inferences. **Please Note:** This detector _cannot_ be used for commercial purposes.
|
||||
|
||||
### Configuration {#configuration-degirum}
|
||||
|
||||
#### AI Server Inference
|
||||
|
||||
Before starting with the config file for this section, you must first launch an AI server. DeGirum has an AI server ready to use as a docker container. Add this to your `docker-compose.yml` to get started:
|
||||
|
||||
```yaml
|
||||
degirum_detector:
|
||||
container_name: degirum
|
||||
image: degirum/aiserver:latest
|
||||
privileged: true
|
||||
ports:
|
||||
- "8778:8778"
|
||||
```
|
||||
|
||||
All supported hardware will automatically be found on your AI server host as long as relevant runtimes and drivers are properly installed on your machine. Refer to [DeGirum's docs site](https://docs.degirum.com/pysdk/runtimes-and-drivers) if you have any trouble.
|
||||
|
||||
Once completed, configure the detector as follows:
|
||||
|
||||
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumAiServer.models} />
|
||||
|
||||
Setting up a model in the `config.yml` is similar to setting up an AI server.
|
||||
You can set it to:
|
||||
|
||||
- A model listed on the [AI Hub](https://hub.degirum.com), given that the correct zoo name is listed in your detector
|
||||
- If this is what you choose to do, the correct model will be downloaded onto your machine before running.
|
||||
- A local directory acting as a zoo. See DeGirum's docs site [for more information](https://docs.degirum.com/pysdk/user-guide-pysdk/organizing-models#model-zoo-directory-structure).
|
||||
- A path to some model.json.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
path: ./mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1 # directory to model .json and file
|
||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
||||
```
|
||||
|
||||
#### Local Inference
|
||||
|
||||
It is also possible to eliminate the need for an AI server and run the hardware directly. The benefit of this approach is that you eliminate any bottlenecks that occur when transferring prediction results from the AI server docker container to the frigate one. However, the method of implementing local inference is different for every device and hardware combination, so it's usually more trouble than it's worth. A general guideline to achieve this would be:
|
||||
|
||||
1. Ensuring that the frigate docker container has the runtime you want to use. So for instance, running `@local` for Hailo means making sure the container you're using has the Hailo runtime installed.
|
||||
2. To double check the runtime is detected by the DeGirum detector, make sure the `degirum sys-info` command properly shows whatever runtimes you mean to install.
|
||||
3. Create a DeGirum detector in your configuration.
|
||||
|
||||
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumLocal.models} />
|
||||
|
||||
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
|
||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
||||
```
|
||||
|
||||
#### AI Hub Cloud Inference
|
||||
|
||||
If you do not possess whatever hardware you want to run, there's also the option to run cloud inferences. Do note that your detection fps might need to be lowered as network latency does significantly slow down this method of detection. For use with Frigate, we highly recommend using a local AI server as described above. To set up cloud inferences,
|
||||
|
||||
1. Sign up at [DeGirum's AI Hub](https://hub.degirum.com).
|
||||
2. Get an access token.
|
||||
3. Create a DeGirum detector in your configuration.
|
||||
|
||||
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumCloud.models} />
|
||||
|
||||
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
|
||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
||||
```
|
||||
|
||||
## AXERA
|
||||
|
||||
Hardware accelerated object detection is supported on the following SoCs:
|
||||
|
||||
@@ -39,7 +39,7 @@ The per-clip variation is typically quite low and is mostly an artifact of keyfr
|
||||
|
||||
Debug Replay lets you re-run Frigate's detection pipeline against a section of recorded video without manually configuring a dummy camera. It automatically extracts the recording, creates a temporary camera with the same detection settings as the original, and loops the clip through the pipeline so you can observe detections in real time.
|
||||
|
||||
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame.
|
||||
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame. The Debug Replay camera does not save recordings or snapshots or surface anything in Explore, but it otherwise behaves like a regular camera, including running enrichments such as Face Recognition, LPR, and custom classification.
|
||||
|
||||
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
|
||||
|
||||
|
||||
Vendored
-38
@@ -2872,44 +2872,6 @@ paths:
|
||||
- frigateUserAuth: []
|
||||
x-required-role: any
|
||||
description: '**Access:** Any authenticated user.'
|
||||
/categorized_object_names:
|
||||
get:
|
||||
tags:
|
||||
- App
|
||||
summary: Get known object names by object type
|
||||
description: |-
|
||||
**Access:** Any authenticated user.
|
||||
|
||||
Returns the sub labels and attributes this install can attach,
|
||||
grouped by object type. Unlike /sub_labels, which reflects what has already been
|
||||
detected, this reads the config and model files, so it covers recognized face
|
||||
names, named license plates, custom object classification categories, and the
|
||||
detector attributes of tracked objects.
|
||||
operationId: categorized_object_names_categorized_object_names_get
|
||||
parameters:
|
||||
- name: object_type
|
||||
in: query
|
||||
required: false
|
||||
schema:
|
||||
anyOf:
|
||||
- type: string
|
||||
- type: 'null'
|
||||
title: Object Type
|
||||
responses:
|
||||
'200':
|
||||
description: Successful Response
|
||||
content:
|
||||
application/json:
|
||||
schema: {}
|
||||
'422':
|
||||
description: Validation Error
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/HTTPValidationError'
|
||||
security:
|
||||
- frigateUserAuth: []
|
||||
x-required-role: any
|
||||
/audio_labels:
|
||||
get:
|
||||
tags:
|
||||
|
||||
@@ -71,7 +71,6 @@ from frigate.util.config import (
|
||||
find_config_file,
|
||||
redact_credential,
|
||||
)
|
||||
from frigate.util.object_names import get_categorized_object_names
|
||||
from frigate.util.schema import get_config_schema
|
||||
from frigate.util.services import (
|
||||
get_nvidia_driver_info,
|
||||
@@ -1314,28 +1313,6 @@ def get_sub_labels(
|
||||
return JSONResponse(content=sub_labels)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/categorized_object_names",
|
||||
dependencies=[Depends(allow_any_authenticated())],
|
||||
summary="Get known object names by object type",
|
||||
description="""Returns the sub labels and attributes this install can attach,
|
||||
grouped by object type. Unlike /sub_labels, which reflects what has already been
|
||||
detected, this reads the config and model files, so it covers recognized face
|
||||
names, named license plates, custom object classification categories, and the
|
||||
detector attributes of tracked objects.""",
|
||||
)
|
||||
def categorized_object_names(
|
||||
request: Request,
|
||||
object_type: str | None = None,
|
||||
allowed_cameras: list[str] = Depends(get_allowed_cameras_for_filter),
|
||||
):
|
||||
return JSONResponse(
|
||||
content=get_categorized_object_names(
|
||||
request.app.frigate_config, allowed_cameras, object_type
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@router.get("/audio_labels", dependencies=[Depends(allow_any_authenticated())])
|
||||
def get_audio_labels():
|
||||
labels = load_labels("/audio-labelmap.txt", prefill=521)
|
||||
|
||||
@@ -83,7 +83,6 @@ def require_admin_by_default():
|
||||
"/nvinfo",
|
||||
"/labels",
|
||||
"/sub_labels",
|
||||
"/categorized_object_names",
|
||||
"/plus/models",
|
||||
"/recognized_license_plates",
|
||||
"/timeline",
|
||||
|
||||
+2
-37
@@ -50,7 +50,6 @@ from frigate.jobs.vlm_watch import (
|
||||
stop_vlm_watch_job,
|
||||
)
|
||||
from frigate.models import Event
|
||||
from frigate.util.object_names import get_categorized_object_names
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -540,13 +539,6 @@ async def execute_tool(
|
||||
if tool_name == "search_objects":
|
||||
return await _execute_search_objects(request, arguments, allowed_cameras)
|
||||
|
||||
if tool_name == "get_categorized_object_names":
|
||||
return JSONResponse(
|
||||
content=_execute_get_categorized_object_names(
|
||||
request, arguments, allowed_cameras
|
||||
)
|
||||
)
|
||||
|
||||
if tool_name == "find_similar_objects":
|
||||
result = await _execute_find_similar_objects(
|
||||
request, arguments, allowed_cameras
|
||||
@@ -725,29 +717,6 @@ async def _execute_set_camera_state(
|
||||
return {"success": True, "camera": camera, "feature": feature, "value": value}
|
||||
|
||||
|
||||
def _execute_get_categorized_object_names(
|
||||
request: Request,
|
||||
arguments: dict[str, Any],
|
||||
allowed_cameras: list[str],
|
||||
) -> dict[str, Any]:
|
||||
object_type = arguments.get("object_type") or None
|
||||
names = get_categorized_object_names(
|
||||
request.app.frigate_config, allowed_cameras, object_type
|
||||
)
|
||||
|
||||
if not names:
|
||||
return {
|
||||
"names": {},
|
||||
"message": (
|
||||
f"No names configured for '{object_type}'."
|
||||
if object_type
|
||||
else "No names configured; search by label or semantic_query."
|
||||
),
|
||||
}
|
||||
|
||||
return {"names": names}
|
||||
|
||||
|
||||
async def _execute_tool_internal(
|
||||
tool_name: str,
|
||||
arguments: dict[str, Any],
|
||||
@@ -772,10 +741,6 @@ async def _execute_tool_internal(
|
||||
except (json.JSONDecodeError, AttributeError) as e:
|
||||
logger.warning(f"Failed to extract tool result: {e}")
|
||||
return {"error": "Failed to parse tool result"}
|
||||
elif tool_name == "get_categorized_object_names":
|
||||
return _execute_get_categorized_object_names(
|
||||
request, arguments, allowed_cameras
|
||||
)
|
||||
elif tool_name == "find_similar_objects":
|
||||
return await _execute_find_similar_objects(request, arguments, allowed_cameras)
|
||||
elif tool_name == "set_camera_state":
|
||||
@@ -808,8 +773,8 @@ async def _execute_tool_internal(
|
||||
else:
|
||||
logger.error(
|
||||
"Tool call failed: unknown tool %r. Expected one of: search_objects, find_similar_objects, "
|
||||
"get_categorized_object_names, get_live_context, start_camera_watch, stop_camera_watch, "
|
||||
"get_profile_status, get_recap. Arguments received: %s",
|
||||
"get_live_context, start_camera_watch, stop_camera_watch, get_profile_status, get_recap. "
|
||||
"Arguments received: %s",
|
||||
tool_name,
|
||||
json.dumps(arguments),
|
||||
)
|
||||
|
||||
@@ -574,11 +574,11 @@ async def vod_ts(
|
||||
Recordings.start_time,
|
||||
)
|
||||
.where(
|
||||
Recordings.camera == camera_name,
|
||||
Recordings.start_time >= start_ts - MAX_SEGMENT_DURATION,
|
||||
Recordings.start_time <= end_ts,
|
||||
Recordings.end_time >= start_ts,
|
||||
Recordings.start_time.between(start_ts, end_ts)
|
||||
| Recordings.end_time.between(start_ts, end_ts)
|
||||
| ((start_ts > Recordings.start_time) & (end_ts < Recordings.end_time))
|
||||
)
|
||||
.where(Recordings.camera == camera_name)
|
||||
.order_by(Recordings.start_time.asc())
|
||||
.iterator()
|
||||
)
|
||||
|
||||
@@ -25,7 +25,7 @@ from frigate.api.defs.query.recordings_query_parameters import (
|
||||
)
|
||||
from frigate.api.defs.response.generic_response import GenericResponse
|
||||
from frigate.api.defs.tags import Tags
|
||||
from frigate.const import MAX_SEGMENT_DURATION, RECORD_DIR
|
||||
from frigate.const import RECORD_DIR
|
||||
from frigate.models import Event, Recordings
|
||||
from frigate.util.time import get_dst_transitions
|
||||
|
||||
@@ -243,7 +243,6 @@ async def recordings(
|
||||
)
|
||||
.where(
|
||||
Recordings.camera == camera_name,
|
||||
Recordings.start_time >= after - MAX_SEGMENT_DURATION,
|
||||
Recordings.end_time >= after,
|
||||
Recordings.start_time <= before,
|
||||
)
|
||||
|
||||
@@ -13,8 +13,8 @@ class CameraUiConfig(FrigateBaseModel):
|
||||
)
|
||||
dashboard: bool = Field(
|
||||
default=True,
|
||||
title="Show in UI",
|
||||
description="Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again.",
|
||||
title="Show on Live dashboard",
|
||||
description="Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings.",
|
||||
)
|
||||
review: bool = Field(
|
||||
default=True,
|
||||
|
||||
@@ -1,157 +0,0 @@
|
||||
import logging
|
||||
import queue
|
||||
from typing import Literal
|
||||
|
||||
import numpy as np
|
||||
from pydantic import ConfigDict, Field
|
||||
|
||||
from frigate.detectors.detection_api import DetectionApi
|
||||
from frigate.detectors.detector_config import BaseDetectorConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
DETECTOR_KEY = "degirum"
|
||||
|
||||
|
||||
### DETECTOR CONFIG ###
|
||||
class DGDetectorConfig(BaseDetectorConfig):
|
||||
"""DeGirum detector for running models via DeGirum cloud or local inference services."""
|
||||
|
||||
model_config = ConfigDict(
|
||||
title="DeGirum",
|
||||
)
|
||||
|
||||
type: Literal[DETECTOR_KEY]
|
||||
location: str = Field(
|
||||
default=None,
|
||||
title="Inference Location",
|
||||
description="Location of the DeGirim inference engine (e.g. '@cloud', '127.0.0.1').",
|
||||
)
|
||||
zoo: str = Field(
|
||||
default=None,
|
||||
title="Model Zoo",
|
||||
description="Path or URL to the DeGirum model zoo.",
|
||||
)
|
||||
token: str = Field(
|
||||
default=None,
|
||||
title="DeGirum Cloud Token",
|
||||
description="Token for DeGirum Cloud access.",
|
||||
)
|
||||
|
||||
|
||||
### ACTUAL DETECTOR ###
|
||||
class DGDetector(DetectionApi):
|
||||
type_key = DETECTOR_KEY
|
||||
|
||||
def __init__(self, detector_config: DGDetectorConfig):
|
||||
try:
|
||||
import degirum as dg
|
||||
except ModuleNotFoundError:
|
||||
raise ImportError("Unable to import DeGirum detector.") from None
|
||||
|
||||
self._queue = queue.Queue()
|
||||
self._zoo = dg.connect(
|
||||
detector_config.location, detector_config.zoo, detector_config.token
|
||||
)
|
||||
|
||||
logger.debug(f"Models in zoo: {self._zoo.list_models()}")
|
||||
|
||||
self.dg_model = self._zoo.load_model(
|
||||
detector_config.model.path,
|
||||
)
|
||||
|
||||
# Setting input image format to raw reduces preprocessing time
|
||||
self.dg_model.input_image_format = "RAW"
|
||||
|
||||
# Prioritize the most powerful hardware available
|
||||
self.select_best_device_type()
|
||||
# Frigate handles pre processing as long as these are all set
|
||||
input_shape = self.dg_model.input_shape[0]
|
||||
self.model_height = input_shape[1]
|
||||
self.model_width = input_shape[2]
|
||||
|
||||
# Passing in dummy frame so initial connection latency happens in
|
||||
# init function and not during actual prediction
|
||||
frame = np.zeros(
|
||||
(detector_config.model.width, detector_config.model.height, 3),
|
||||
dtype=np.uint8,
|
||||
)
|
||||
# Pass in frame to overcome first frame latency
|
||||
self.dg_model(frame)
|
||||
self.prediction = self.prediction_generator()
|
||||
|
||||
def select_best_device_type(self):
|
||||
"""
|
||||
Helper function that selects fastest hardware available per model runtime
|
||||
"""
|
||||
types = self.dg_model.supported_device_types
|
||||
|
||||
device_map = {
|
||||
"OPENVINO": ["GPU", "NPU", "CPU"],
|
||||
"HAILORT": ["HAILO8L", "HAILO8"],
|
||||
"N2X": ["ORCA1", "CPU"],
|
||||
"ONNX": ["VITIS_NPU", "CPU"],
|
||||
"RKNN": ["RK3566", "RK3568", "RK3588"],
|
||||
"TENSORRT": ["DLA", "GPU", "DLA_ONLY"],
|
||||
"TFLITE": ["ARMNN", "EDGETPU", "CPU"],
|
||||
}
|
||||
|
||||
runtime = types[0].split("/")[0]
|
||||
# Just create an array of format {runtime}/{hardware} for every hardware
|
||||
# in the value for appropriate key in device_map
|
||||
self.dg_model.device_type = [
|
||||
f"{runtime}/{hardware}" for hardware in device_map[runtime]
|
||||
]
|
||||
|
||||
def prediction_generator(self):
|
||||
"""
|
||||
Generator for all incoming frames. By using this generator, we don't have to keep
|
||||
reconnecting our websocket on every "predict" call.
|
||||
"""
|
||||
logger.debug("Prediction generator was called")
|
||||
with self.dg_model as model:
|
||||
while 1:
|
||||
logger.info(f"q size before calling get: {self._queue.qsize()}")
|
||||
data = self._queue.get(block=True)
|
||||
logger.info(f"q size after calling get: {self._queue.qsize()}")
|
||||
logger.debug(
|
||||
f"Data we're passing into model predict: {data}, shape of data: {data.shape}"
|
||||
)
|
||||
result = model.predict(data)
|
||||
logger.debug(f"Prediction result: {result}")
|
||||
yield result
|
||||
|
||||
def detect_raw(self, tensor_input):
|
||||
# Reshaping tensor to work with pysdk
|
||||
truncated_input = tensor_input.reshape(tensor_input.shape[1:])
|
||||
logger.debug(f"Detect raw was called for tensor input: {tensor_input}")
|
||||
|
||||
# add tensor_input to input queue
|
||||
self._queue.put(truncated_input)
|
||||
logger.debug(f"Queue size after adding truncated input: {self._queue.qsize()}")
|
||||
|
||||
# define empty detection result
|
||||
detections = np.zeros((20, 6), np.float32)
|
||||
# grab prediction
|
||||
res = next(self.prediction)
|
||||
|
||||
# If we have an empty prediction, return immediately
|
||||
if len(res.results) == 0 or len(res.results[0]) == 0:
|
||||
return detections
|
||||
|
||||
i = 0
|
||||
for result in res.results:
|
||||
if i >= 20:
|
||||
break
|
||||
|
||||
detections[i] = [
|
||||
result["category_id"],
|
||||
float(result["score"]),
|
||||
result["bbox"][1] / self.model_height,
|
||||
result["bbox"][0] / self.model_width,
|
||||
result["bbox"][3] / self.model_height,
|
||||
result["bbox"][2] / self.model_width,
|
||||
]
|
||||
i += 1
|
||||
|
||||
logger.debug(f"Detections output: {detections}")
|
||||
return detections
|
||||
@@ -9,6 +9,7 @@ from pydantic import ConfigDict, Field
|
||||
|
||||
from frigate.detectors.detection_api import DetectionApi
|
||||
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
||||
from frigate.util.model import xyxy_to_xywh_for_nms
|
||||
|
||||
try:
|
||||
from tflite_runtime.interpreter import Interpreter, load_delegate
|
||||
@@ -297,7 +298,7 @@ class EdgeTpuTfl(DetectionApi):
|
||||
# until after filtering out redundant boxes
|
||||
# Shift the logit scores to be non-negative (required by cv2)
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
bboxes=boxes_filtered_decoded,
|
||||
bboxes=xyxy_to_xywh_for_nms(boxes_filtered_decoded),
|
||||
scores=max_scores_filtered_shiftedpositive,
|
||||
score_threshold=(
|
||||
self.min_logit_value + self.logit_shift_to_positive_values
|
||||
|
||||
@@ -17,6 +17,7 @@ from frigate.detectors.detector_config import (
|
||||
ModelTypeEnum,
|
||||
)
|
||||
from frigate.util.file import FileLock
|
||||
from frigate.util.model import xyxy_to_xywh_for_nms
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -581,7 +582,7 @@ class MemryXDetector(DetectionApi):
|
||||
# Convert coordinates to integers
|
||||
x_min, y_min, x_max, y_max = map(int, [x_min, y_min, x_max, y_max])
|
||||
|
||||
# Append valid detections [class_id, confidence, x, y, width, height]
|
||||
# Append valid detections [class_id, confidence, x_min, y_min, x_max, y_max]
|
||||
detections.append([class_id, confidence, x_min, y_min, x_max, y_max])
|
||||
|
||||
final_detections = np.zeros((20, 6), np.float32)
|
||||
@@ -595,7 +596,7 @@ class MemryXDetector(DetectionApi):
|
||||
detections = np.array(detections, dtype=np.float32)
|
||||
|
||||
# Apply Non-Maximum Suppression (NMS)
|
||||
bboxes = detections[:, 2:6].tolist() # (x_min, y_min, width, height)
|
||||
bboxes = xyxy_to_xywh_for_nms(detections[:, 2:6])
|
||||
scores = detections[:, 1].tolist() # Confidence scores
|
||||
|
||||
indices = cv2.dnn.NMSBoxes(bboxes, scores, 0.45, 0.5)
|
||||
|
||||
@@ -12,7 +12,7 @@ from frigate.const import MODEL_CACHE_DIR, SUPPORTED_RK_SOCS
|
||||
from frigate.detectors.detection_api import DetectionApi
|
||||
from frigate.detectors.detection_runners import RKNNModelRunner
|
||||
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
||||
from frigate.util.model import post_process_yolo
|
||||
from frigate.util.model import post_process_yolo, xyxy_to_xywh_for_nms
|
||||
from frigate.util.rknn_converter import auto_convert_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -285,7 +285,7 @@ class Rknn(DetectionApi):
|
||||
|
||||
# run nms
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
bboxes=boxes,
|
||||
bboxes=xyxy_to_xywh_for_nms(boxes),
|
||||
scores=scores,
|
||||
score_threshold=0.4,
|
||||
nms_threshold=0.4,
|
||||
|
||||
+118
-66
@@ -262,10 +262,6 @@ def get_tool_definitions(
|
||||
`attribute` parameter is exposed for filtering by their labels. When the
|
||||
embeddings model only understands English (JinaV1), the `semantic_query`
|
||||
description instructs the model to write the query in English.
|
||||
|
||||
Descriptions here stay mechanical: which tool to reach for, and how the
|
||||
filters relate to each other, is stated once in the system prompt so the
|
||||
guidance is not paid for twice on every request.
|
||||
"""
|
||||
search_objects_properties: dict[str, Any] = {
|
||||
"camera": {
|
||||
@@ -274,13 +270,26 @@ def get_tool_definitions(
|
||||
},
|
||||
"label": {
|
||||
"type": "string",
|
||||
"description": "Tracked object class to filter by.",
|
||||
"description": (
|
||||
"Generic object class to filter by — one of the tracked detector "
|
||||
"labels such as 'person', 'package', 'car', 'dog', 'bird'. Use "
|
||||
"this for broad queries like 'show me all cars today'. Combine "
|
||||
"with semantic_query when the user also describes appearance or "
|
||||
"behavior (e.g. label='person', semantic_query='riding a lawn "
|
||||
"mower')."
|
||||
),
|
||||
},
|
||||
"sub_label": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Name recognized in the detection: a person, delivery company, "
|
||||
"animal species or breed, or license plate."
|
||||
"Filter by a DISCRETE NAMED entity recognized in the detection. "
|
||||
"Use this for: a known person's name ('John'), a delivery "
|
||||
"company ('Amazon', 'UPS'), a recognized animal species or "
|
||||
"breed ('blue jay', 'cardinal', 'golden retriever'), or a "
|
||||
"license plate string. When filtering by a specific name, set "
|
||||
"only sub_label and leave label unset. Do NOT use sub_label "
|
||||
"for descriptions of appearance, clothing, or actions — those "
|
||||
"belong in semantic_query."
|
||||
),
|
||||
},
|
||||
"after": {
|
||||
@@ -304,11 +313,20 @@ def get_tool_definitions(
|
||||
}
|
||||
|
||||
if attribute_classifications:
|
||||
model_outline = "; ".join(
|
||||
f"{m['name']} (applies to {', '.join(m['objects']) or 'any object'})"
|
||||
for m in attribute_classifications
|
||||
)
|
||||
search_objects_properties["attribute"] = {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Attribute label produced by a configured classification model "
|
||||
"(case-sensitive)."
|
||||
"Filter by a classification attribute label produced by a "
|
||||
"configured attribute classification model. Use this INSTEAD "
|
||||
"of semantic_query when the user's request matches one of "
|
||||
"these classifications. Configured models: "
|
||||
f"{model_outline}. "
|
||||
"Set the value to the attribute label that matches the user's "
|
||||
"phrasing (case-sensitive)."
|
||||
),
|
||||
}
|
||||
|
||||
@@ -316,12 +334,29 @@ def get_tool_definitions(
|
||||
search_objects_properties["semantic_query"] = {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Description of an appearance or activity, used to semantically "
|
||||
"narrow results."
|
||||
"Optional natural-language description of a PHYSICAL "
|
||||
"CHARACTERISTIC, APPEARANCE, or ACTIVITY the user mentioned, "
|
||||
"used to semantically narrow results. Only set this when the "
|
||||
"user describes something beyond what label and sub_label can "
|
||||
"express on their own.\n"
|
||||
"USE for descriptive phrases like: 'riding a lawn mower', "
|
||||
"'wearing a red jacket', 'carrying a package', 'walking a "
|
||||
"dog', 'on a bicycle', 'holding an umbrella'.\n"
|
||||
"DO NOT USE for:\n"
|
||||
"- specific named people, pets, or delivery companies → use sub_label\n"
|
||||
"- animal species or breed names like 'blue jay', 'cardinal', "
|
||||
"'golden retriever' → use sub_label\n"
|
||||
"- license plate strings → use sub_label\n"
|
||||
"- generic object queries like 'all cars today' or 'every "
|
||||
"person' → use label alone with no semantic_query\n"
|
||||
"When set, combine with label/time/camera/zone filters as "
|
||||
"usual (e.g. label='person', semantic_query='riding a lawn "
|
||||
"mower', after='2024-05-01T00:00:00Z')."
|
||||
+ (
|
||||
" The configured embeddings model only understands English, so "
|
||||
"always write this in English, translating the user's "
|
||||
"description if they phrased it in another language."
|
||||
" The configured embeddings model only understands "
|
||||
"English, so always write semantic_query in English, "
|
||||
"translating the user's description if they phrased it "
|
||||
"in another language."
|
||||
if embeddings_language == "english"
|
||||
else ""
|
||||
)
|
||||
@@ -329,9 +364,25 @@ def get_tool_definitions(
|
||||
}
|
||||
|
||||
search_objects_description = (
|
||||
"Search the historical record of tracked detections. Use this ONLY for "
|
||||
"questions about the PAST, e.g. 'did anyone come by today?', 'when was the "
|
||||
"last car?'. For alerting on future events use start_camera_watch instead."
|
||||
"Search the historical record of detected objects in Frigate. "
|
||||
"Use this ONLY for questions about the PAST — e.g. 'did anyone come by today?', "
|
||||
"'when was the last car?', 'show me detections from yesterday'. "
|
||||
"Do NOT use this for monitoring or alerting requests about future events — "
|
||||
"use start_camera_watch instead for those. "
|
||||
"An 'object' in Frigate represents a tracked detection (e.g., a person, package, car).\n\n"
|
||||
"Choose filters based on what the user is asking for:\n"
|
||||
"- Generic class query ('show me all cars today'): set `label` only.\n"
|
||||
"- Specific NAMED entity (known person, delivery company, animal "
|
||||
"species/breed like 'blue jay' or 'golden retriever', license "
|
||||
"plate): set `sub_label` only and leave `label` unset.\n"
|
||||
)
|
||||
if semantic_search_enabled:
|
||||
search_objects_description += (
|
||||
"- Physical CHARACTERISTIC, APPEARANCE, or ACTIVITY that is not a "
|
||||
"discrete name ('person riding a lawn mower', 'someone in a red "
|
||||
"jacket', 'person carrying a package'): set `semantic_query` with "
|
||||
"the descriptive phrase, optionally alongside `label` for the "
|
||||
"object class. Do NOT put descriptive phrases in sub_label."
|
||||
)
|
||||
|
||||
return [
|
||||
@@ -347,34 +398,20 @@ def get_tool_definitions(
|
||||
"required": [],
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_categorized_object_names",
|
||||
"description": (
|
||||
"Every name that can be attached as a sub_label, grouped by object "
|
||||
"type: recognized faces, named license plates, classification "
|
||||
"categories, and delivery logos."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"object_type": {
|
||||
"type": "string",
|
||||
"description": "Optional object label (e.g. 'person', 'car'). Omit for all.",
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "find_similar_objects",
|
||||
"description": (
|
||||
"Find tracked objects visually and semantically similar to a "
|
||||
"specific past event. Requires semantic search to be enabled."
|
||||
"Find tracked objects that are visually and semantically similar "
|
||||
"to a specific past event. Use this when the user references a "
|
||||
"particular object they have seen and wants to find other "
|
||||
"sightings of the same or similar one ('that green car', 'the "
|
||||
"person in the red jacket', 'the package that was delivered'). "
|
||||
"Prefer this over search_objects whenever the user's intent is "
|
||||
"'find more like this specific one.' Use search_objects first "
|
||||
"only if you need to locate the anchor event. Requires semantic "
|
||||
"search to be enabled."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -436,8 +473,9 @@ def get_tool_definitions(
|
||||
"function": {
|
||||
"name": "set_camera_state",
|
||||
"description": (
|
||||
"Change a camera's feature state, e.g. turn detection on or off. "
|
||||
"Only call this when the user explicitly asks to change a setting. "
|
||||
"Change a camera's feature state (e.g., turn detection on/off, enable/disable recordings). "
|
||||
"Use camera='*' to apply to all cameras at once. "
|
||||
"Only call this tool when the user explicitly asks to change a camera setting. "
|
||||
"Requires admin privileges."
|
||||
),
|
||||
"parameters": {
|
||||
@@ -479,7 +517,7 @@ def get_tool_definitions(
|
||||
},
|
||||
"value": {
|
||||
"type": "string",
|
||||
"description": "The value to set, as accepted by the chosen feature.",
|
||||
"description": "The value to set. ON or OFF for toggles, a number for thresholds, a profile name or 'none' for profile.",
|
||||
},
|
||||
},
|
||||
"required": ["camera", "feature", "value"],
|
||||
@@ -491,9 +529,11 @@ def get_tool_definitions(
|
||||
"function": {
|
||||
"name": "get_live_context",
|
||||
"description": (
|
||||
"Current live image and detections (tracked objects, zones, "
|
||||
"timestamps) for one camera. Use this for questions about what is "
|
||||
"happening right now. Call it again for each additional camera."
|
||||
"Get the current live image and detection information for a single camera: objects being tracked, "
|
||||
"zones, timestamps. Use this to understand what is visible in the live view. "
|
||||
"Call this when answering questions about what is happening right now on a specific camera. "
|
||||
"Operates on one camera at a time; call the tool again for each additional camera. "
|
||||
"Wildcards and empty values are not accepted."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -501,8 +541,8 @@ def get_tool_definitions(
|
||||
"camera": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Exact name of a single camera. Wildcards (e.g. '*', "
|
||||
"'all') and empty strings are not accepted."
|
||||
"Exact name of a single camera to get live context for. "
|
||||
"Wildcards (e.g. '*', 'all') and empty strings are not accepted."
|
||||
),
|
||||
},
|
||||
},
|
||||
@@ -515,9 +555,10 @@ def get_tool_definitions(
|
||||
"function": {
|
||||
"name": "start_camera_watch",
|
||||
"description": (
|
||||
"Start a continuous watch job that monitors a camera and notifies "
|
||||
"the user when a condition is met, e.g. 'tell me when guests "
|
||||
"arrive'. Only one watch job can run at a time. Returns a job ID."
|
||||
"Start a continuous VLM watch job that monitors a camera and sends a notification "
|
||||
"when a specified condition is met. Use this when the user wants to be alerted about "
|
||||
"a future event, e.g. 'tell me when guests arrive' or 'notify me when the package is picked up'. "
|
||||
"Only one watch job can run at a time. Returns a job ID."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -557,7 +598,10 @@ def get_tool_definitions(
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "stop_camera_watch",
|
||||
"description": "Cancel the currently running watch job.",
|
||||
"description": (
|
||||
"Cancel the currently running VLM watch job. Use this when the user wants to "
|
||||
"stop a previously started watch, e.g. 'stop watching the front door'."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
@@ -570,9 +614,11 @@ def get_tool_definitions(
|
||||
"function": {
|
||||
"name": "get_profile_status",
|
||||
"description": (
|
||||
"Get the active profile and when each profile was last activated. "
|
||||
"Call this before get_recap to derive the time window for requests "
|
||||
"like 'what happened while I was away?'."
|
||||
"Get the current profile status including the active profile and "
|
||||
"timestamps of when each profile was last activated. Use this to "
|
||||
"determine time periods for recap requests — e.g. when the user asks "
|
||||
"'what happened while I was away?', call this first to find the relevant "
|
||||
"time window based on profile activation history."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -586,9 +632,11 @@ def get_tool_definitions(
|
||||
"function": {
|
||||
"name": "get_recap",
|
||||
"description": (
|
||||
"Get all activity (alerts and detections) for a time period, as a "
|
||||
"chronological list with camera, objects, zones, and descriptions "
|
||||
"when available. Summarize the results for the user."
|
||||
"Get a recap of all activity (alerts and detections) for a given time period. "
|
||||
"Use this after calling get_profile_status to retrieve what happened during "
|
||||
"a specific window — e.g. 'what happened while I was away?'. Returns a "
|
||||
"chronological list of activity with camera, objects, zones, and GenAI-generated "
|
||||
"descriptions when available. Summarize the results for the user."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -675,13 +723,14 @@ def build_chat_system_prompt(
|
||||
)
|
||||
speed_units_section = f"\n\nReport object speeds to the user in {speed_unit}."
|
||||
|
||||
filter_routing_section = (
|
||||
semantic_search_section = ""
|
||||
if semantic_search_enabled:
|
||||
semantic_search_section = (
|
||||
"\n\nWhen routing a search_objects call, pick filters by the shape of the user's request:\n"
|
||||
"- Generic class ('show me all cars today'): set `label` only.\n"
|
||||
"- Specific named entity — a known person ('John'), delivery company ('Amazon'), animal species/breed ('blue jay', 'golden retriever'), or license plate: set `sub_label` only and leave `label` unset. Call get_categorized_object_names first and use the exact spelling it returns; a guessed spelling matches nothing. If the name is absent, say it is not configured rather than searching for it."
|
||||
"- Specific named entity — a known person ('John'), delivery company ('Amazon'), animal species/breed ('blue jay', 'cardinal', 'golden retriever'), or license plate: set `sub_label` only and leave `label` unset.\n"
|
||||
"- Physical characteristic, appearance, or activity that is NOT a discrete name ('find me people riding a lawn mower', 'someone in a red jacket', 'a person carrying a package'): set `semantic_query` with the descriptive phrase, optionally combined with `label` for the object class. Never put descriptive phrases in `sub_label`."
|
||||
)
|
||||
if semantic_search_enabled:
|
||||
filter_routing_section += "\n- Physical characteristic, appearance, or activity that is NOT a discrete name ('riding a lawn mower', 'someone in a red jacket'): set `semantic_query` with the descriptive phrase, optionally combined with `label`. Never put descriptive phrases in `sub_label`."
|
||||
|
||||
attribute_classification_section = ""
|
||||
if attribute_classifications:
|
||||
@@ -690,9 +739,9 @@ def build_chat_system_prompt(
|
||||
for m in attribute_classifications
|
||||
)
|
||||
attribute_classification_section = (
|
||||
"\n\nConfigured attribute classification models:\n"
|
||||
"\n\nAttribute classification models are configured for the following object types:\n"
|
||||
f"{model_lines}\n"
|
||||
"When the user's request matches one of these classifications, set the search_objects `attribute` field to the matching label (case-sensitive) rather than using `semantic_query`. Reserve `semantic_query` for descriptive phrases outside the configured attribute labels."
|
||||
"When the user's request matches one of these classifications, set the search_objects `attribute` field to the matching label rather than using `semantic_query`. Reserve `semantic_query` for descriptive phrases that fall outside the configured attribute labels."
|
||||
)
|
||||
|
||||
return f"""You are a helpful assistant for Frigate, a security camera NVR system. You help users answer questions about their cameras, detected objects, and events.
|
||||
@@ -701,6 +750,9 @@ Current server local date and time: {current_date_str} at {current_time_str}
|
||||
|
||||
Do not start your response with phrases like "I will check...", "Let me see...", or "Let me look...". Answer directly.
|
||||
|
||||
Always present times in the server's local timezone. When tool results include start_time_local and end_time_local, quote those strings exactly; never convert or invent timestamps, and fall back to UTC or ISO format only when a result has no local time fields. Resolve relative dates like "today" or "this week" against the current date above, and pass dates to tools in ISO 8601 (e.g. {current_date_str}T00:00:00Z for the start of today).
|
||||
Always present times to the user in the server's local timezone. When tool results include start_time_local and end_time_local, use those exact strings when listing or describing detection times—do not convert or invent timestamps. Do not use UTC or ISO format with Z for the user-facing answer unless the tool result only provides Unix timestamps without local time fields.
|
||||
When users ask about "today", "yesterday", "this week", etc., use the current date above as reference.
|
||||
When searching for objects or events, use ISO 8601 format for dates (e.g., {current_date_str}T00:00:00Z for the start of today).
|
||||
Always be accurate with time calculations based on the current date provided.
|
||||
|
||||
When the user refers to a specific object they have seen ("that green car", "the person in the red jacket", "a package left today"), prefer find_similar_objects over search_objects, using search_objects only to locate the anchor event and passing its id along. Keep search_objects for generic queries like "show me all cars today". If a user message begins with [attached_event:<id>], treat that id as the anchor for any similarity or "tell me more" request in the same message.{filter_routing_section}{attribute_classification_section}{cameras_section}{speed_units_section}"""
|
||||
When a user refers to a specific object they have seen or describe with identifying details ("that green car", "the person in the red jacket", "a package left today"), prefer the find_similar_objects tool over search_objects. Use search_objects first only to locate the anchor event, then pass its id to find_similar_objects. For generic queries like "show me all cars today", keep using search_objects. If a user message begins with [attached_event:<id>], treat that event id as the anchor for any similarity or "tell me more" request in the same message and call find_similar_objects with that id.{semantic_search_section}{attribute_classification_section}{cameras_section}{speed_units_section}"""
|
||||
|
||||
@@ -1,73 +1,15 @@
|
||||
"""Unit tests for recordings/media API endpoints."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytz
|
||||
from fastapi import Request
|
||||
|
||||
from frigate.api.auth import get_allowed_cameras_for_filter, get_current_user
|
||||
from frigate.const import MAX_SEGMENT_DURATION
|
||||
from frigate.models import Recordings
|
||||
from frigate.test.http_api.base_http_test import AuthTestClient, BaseTestHttp
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RangeCase:
|
||||
"""Expected behavior for one segment relative to the requested range.
|
||||
|
||||
Offsets are seconds from REQUEST_START; the request ends at +100 seconds.
|
||||
"""
|
||||
|
||||
name: str
|
||||
start_offset: float
|
||||
end_offset: float
|
||||
included_in_recordings: bool
|
||||
vod_clip_from_ms: int | None = None
|
||||
vod_duration_ms: int | None = None
|
||||
|
||||
|
||||
REQUEST_START = 1000
|
||||
REQUEST_END = 1100
|
||||
RANGE_CASES = (
|
||||
RangeCase("before", -MAX_SEGMENT_DURATION + 1, -1, False),
|
||||
RangeCase("meets_start", -10, 0, True),
|
||||
RangeCase(
|
||||
"overlaps_start",
|
||||
-MAX_SEGMENT_DURATION + 0.5,
|
||||
0.25,
|
||||
True,
|
||||
vod_clip_from_ms=599500,
|
||||
vod_duration_ms=250,
|
||||
),
|
||||
RangeCase("starts_at_start", 0, 10, True, vod_duration_ms=10000),
|
||||
RangeCase("inside", 20, 80, True, vod_duration_ms=60000),
|
||||
RangeCase("ends_at_end", 90, 100, True, vod_duration_ms=10000),
|
||||
RangeCase("matches_range", 0, 100, True, vod_duration_ms=100000),
|
||||
RangeCase("starts_with_range", 0, 110, True, vod_duration_ms=100000),
|
||||
RangeCase(
|
||||
"covers_range",
|
||||
-20,
|
||||
120,
|
||||
True,
|
||||
vod_clip_from_ms=20000,
|
||||
vod_duration_ms=100000,
|
||||
),
|
||||
RangeCase(
|
||||
"ends_with_range",
|
||||
-10,
|
||||
100,
|
||||
True,
|
||||
vod_clip_from_ms=10000,
|
||||
vod_duration_ms=100000,
|
||||
),
|
||||
RangeCase("overlaps_end", 95, 105, True, vod_duration_ms=5000),
|
||||
RangeCase("starts_at_end", 100, 110, True),
|
||||
RangeCase("after", 101, 110, False),
|
||||
)
|
||||
|
||||
|
||||
class TestHttpMedia(BaseTestHttp):
|
||||
"""Test media API endpoints, particularly recordings with DST handling."""
|
||||
|
||||
@@ -102,26 +44,6 @@ class TestHttpMedia(BaseTestHttp):
|
||||
self.app.dependency_overrides.clear()
|
||||
super().tearDown()
|
||||
|
||||
def _assert_vod_response(
|
||||
self,
|
||||
response,
|
||||
expected_clips: list[tuple[str, int | None, int]],
|
||||
) -> None:
|
||||
"""Assert VOD clip metadata and its derived duration fields."""
|
||||
assert response.status_code == 200
|
||||
vod = response.json()
|
||||
assert [
|
||||
(
|
||||
clip["path"],
|
||||
clip.get("clipFrom"),
|
||||
clip["keyFrameDurations"][0],
|
||||
)
|
||||
for clip in vod["sequences"][0]["clips"]
|
||||
] == expected_clips
|
||||
expected_durations = [clip[2] for clip in expected_clips]
|
||||
assert vod["durations"] == expected_durations
|
||||
assert vod["segment_duration"] == max(expected_durations)
|
||||
|
||||
def test_recordings_summary_across_dst_spring_forward(self):
|
||||
"""
|
||||
Test recordings summary across spring DST transition (spring forward).
|
||||
@@ -482,102 +404,6 @@ class TestHttpMedia(BaseTestHttp):
|
||||
assert "2024-03-10" in summary
|
||||
assert summary["2024-03-10"] is True
|
||||
|
||||
def test_recordings_handles_all_range_relations(self):
|
||||
"""Recordings return every interval relation that touches the range."""
|
||||
with AuthTestClient(self.app) as client:
|
||||
for case in RANGE_CASES:
|
||||
with self.subTest(case=case.name):
|
||||
Recordings.delete().execute()
|
||||
super().insert_mock_recording(
|
||||
case.name,
|
||||
REQUEST_START + case.start_offset,
|
||||
REQUEST_START + case.end_offset,
|
||||
)
|
||||
|
||||
response = client.get(
|
||||
"/front_door/recordings",
|
||||
params={"after": REQUEST_START, "before": REQUEST_END},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
expected_ids = [case.name] if case.included_in_recordings else []
|
||||
assert [
|
||||
recording["id"] for recording in response.json()
|
||||
] == expected_ids
|
||||
|
||||
def test_vod_handles_all_range_relations(self):
|
||||
"""VOD clips every interval relation with positive playback duration."""
|
||||
with (
|
||||
AuthTestClient(self.app) as client,
|
||||
patch(
|
||||
"frigate.api.media.get_keyframe_before",
|
||||
side_effect=lambda _path, offset: offset,
|
||||
),
|
||||
):
|
||||
for case in RANGE_CASES:
|
||||
with self.subTest(case=case.name):
|
||||
Recordings.delete().execute()
|
||||
super().insert_mock_recording(
|
||||
case.name,
|
||||
REQUEST_START + case.start_offset,
|
||||
REQUEST_START + case.end_offset,
|
||||
)
|
||||
|
||||
response = client.get(
|
||||
f"/vod/front_door/start/{REQUEST_START}/end/{REQUEST_END}"
|
||||
)
|
||||
|
||||
if case.vod_duration_ms is None:
|
||||
assert response.status_code == 404
|
||||
continue
|
||||
|
||||
self._assert_vod_response(
|
||||
response,
|
||||
[
|
||||
(
|
||||
case.name,
|
||||
case.vod_clip_from_ms,
|
||||
case.vod_duration_ms,
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
def test_vod_handles_segment_ending_at_start_with_keyframe_fallbacks(self):
|
||||
"""VOD keeps a boundary segment when keyframe lookup extends it."""
|
||||
|
||||
def keyframe_before(path: str, offset: int) -> int | None:
|
||||
return offset - 1000 if path == "previous_keyframe" else None
|
||||
|
||||
with (
|
||||
AuthTestClient(self.app) as client,
|
||||
patch(
|
||||
"frigate.api.media.get_keyframe_before",
|
||||
side_effect=keyframe_before,
|
||||
),
|
||||
):
|
||||
super().insert_mock_recording(
|
||||
"previous_keyframe",
|
||||
REQUEST_START - 10,
|
||||
REQUEST_START,
|
||||
)
|
||||
super().insert_mock_recording(
|
||||
"missing_keyframe",
|
||||
REQUEST_START - 5,
|
||||
REQUEST_START,
|
||||
)
|
||||
|
||||
response = client.get(
|
||||
f"/vod/front_door/start/{REQUEST_START}/end/{REQUEST_END}"
|
||||
)
|
||||
|
||||
self._assert_vod_response(
|
||||
response,
|
||||
[
|
||||
("previous_keyframe", 9000, 1000),
|
||||
("missing_keyframe", None, 5000),
|
||||
],
|
||||
)
|
||||
|
||||
def test_recordings_unavailable_reports_gap_between_recordings(self):
|
||||
"""A gap between two recordings is reported as an unavailable segment."""
|
||||
with AuthTestClient(self.app) as client:
|
||||
|
||||
@@ -1,12 +1,7 @@
|
||||
import unittest
|
||||
from io import StringIO
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from frigate.util.services import (
|
||||
get_amd_gpu_stats,
|
||||
get_intel_gpu_stats,
|
||||
get_openvino_npu_stats,
|
||||
)
|
||||
from frigate.util.services import get_amd_gpu_stats, get_intel_gpu_stats
|
||||
|
||||
|
||||
class TestGpuStats(unittest.TestCase):
|
||||
@@ -22,88 +17,6 @@ class TestGpuStats(unittest.TestCase):
|
||||
amd_stats = get_amd_gpu_stats()
|
||||
assert amd_stats == {"gpu": "4.17%", "mem": "60.37%"}
|
||||
|
||||
@patch("frigate.util.services.time.sleep")
|
||||
@patch("frigate.util.services.time.time", side_effect=[0.0, 1.0])
|
||||
@patch(
|
||||
"frigate.util.services.os.readlink",
|
||||
return_value="/sys/bus/pci/drivers/intel_vpu",
|
||||
)
|
||||
@patch(
|
||||
"frigate.util.services.glob.glob",
|
||||
return_value=["/sys/class/accel/accel0"],
|
||||
)
|
||||
@patch(
|
||||
"builtins.open",
|
||||
side_effect=[StringIO("1000"), StringIO("1250")],
|
||||
)
|
||||
def test_openvino_npu_stats_discovers_accel0(
|
||||
self, open_file, glob, readlink, time, sleep
|
||||
):
|
||||
assert get_openvino_npu_stats() == {"npu": "25.0", "mem": "-%"}
|
||||
|
||||
open_file.assert_any_call(
|
||||
"/sys/class/accel/accel0/device/power/runtime_active_time"
|
||||
)
|
||||
|
||||
@patch("frigate.util.services.time.sleep")
|
||||
@patch("frigate.util.services.time.time", side_effect=[0.0, 1.0])
|
||||
@patch(
|
||||
"frigate.util.services.os.readlink",
|
||||
side_effect=[
|
||||
"/sys/bus/pci/drivers/other",
|
||||
"/sys/bus/pci/drivers/intel_vpu",
|
||||
],
|
||||
)
|
||||
@patch(
|
||||
"frigate.util.services.glob.glob",
|
||||
return_value=[
|
||||
"/sys/class/accel/accel0",
|
||||
"/sys/class/accel/accel1",
|
||||
],
|
||||
)
|
||||
@patch(
|
||||
"builtins.open",
|
||||
side_effect=[StringIO("1000"), StringIO("1250")],
|
||||
)
|
||||
def test_openvino_npu_stats_skips_non_intel_accelerator(
|
||||
self, open_file, glob, readlink, time, sleep
|
||||
):
|
||||
assert get_openvino_npu_stats() == {"npu": "25.0", "mem": "-%"}
|
||||
|
||||
open_file.assert_any_call(
|
||||
"/sys/class/accel/accel1/device/power/runtime_active_time"
|
||||
)
|
||||
|
||||
@patch(
|
||||
"frigate.util.services.os.readlink",
|
||||
return_value="/sys/bus/pci/drivers/other",
|
||||
)
|
||||
@patch(
|
||||
"frigate.util.services.glob.glob",
|
||||
return_value=["/sys/class/accel/accel0"],
|
||||
)
|
||||
@patch("builtins.open")
|
||||
def test_openvino_npu_stats_no_intel_accelerator(self, open_file, glob, readlink):
|
||||
assert get_openvino_npu_stats() is None
|
||||
open_file.assert_not_called()
|
||||
|
||||
@patch(
|
||||
"frigate.util.services.os.readlink",
|
||||
return_value="/sys/bus/pci/drivers/intel_vpu",
|
||||
)
|
||||
@patch(
|
||||
"frigate.util.services.glob.glob",
|
||||
return_value=["/sys/class/accel/accel0"],
|
||||
)
|
||||
@patch("builtins.open", side_effect=FileNotFoundError)
|
||||
def test_openvino_npu_stats_runtime_counter_unavailable(
|
||||
self, open_file, glob, readlink
|
||||
):
|
||||
assert get_openvino_npu_stats() is None
|
||||
open_file.assert_called_once_with(
|
||||
"/sys/class/accel/accel0/device/power/runtime_active_time"
|
||||
)
|
||||
|
||||
@patch("frigate.stats.intel_gpu_info.intel_gpu_name_resolver.get_names")
|
||||
@patch("frigate.util.services.time.sleep")
|
||||
@patch("frigate.util.services.time.monotonic")
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
"""Tests for detector post-processing NMS box format handling.
|
||||
|
||||
cv2.dnn.NMSBoxes expects boxes as [x, y, width, height]. Passing corner
|
||||
coordinates [x1, y1, x2, y2] makes OpenCV treat x2/y2 as width/height,
|
||||
inflating every box toward the bottom-right by its distance from the origin.
|
||||
Two well separated objects far from the origin then appear to overlap and the
|
||||
lower scoring one is silently suppressed.
|
||||
|
||||
The regression geometry used throughout: two boxes with zero true overlap,
|
||||
A = (393, 499, 484, 620) and B = (527, 499, 618, 620) in a 640x640 input
|
||||
(43 px gap). Misread as [x, y, w, h] their IoU is 0.465, above the 0.4 NMS
|
||||
threshold, so the buggy format drops the lower scoring box while correct
|
||||
conversion keeps both.
|
||||
"""
|
||||
|
||||
import math
|
||||
import unittest
|
||||
from queue import Queue
|
||||
|
||||
import numpy as np
|
||||
|
||||
from frigate.detectors.plugins.memryx import MemryXDetector
|
||||
from frigate.util.model import (
|
||||
post_process_dfine,
|
||||
post_process_rfdetr,
|
||||
post_process_yolo,
|
||||
post_process_yolox,
|
||||
)
|
||||
|
||||
WIDTH = 640
|
||||
HEIGHT = 640
|
||||
|
||||
# box A: xyxy (393, 499, 484, 620) as center format
|
||||
A_CX, A_CY, A_W, A_H = 438.5, 559.5, 91.0, 121.0
|
||||
# box B: xyxy (527, 499, 618, 620) as center format
|
||||
B_CX, B_CY, B_W, B_H = 572.5, 559.5, 91.0, 121.0
|
||||
|
||||
# expected normalized output rows: [class_id, conf, y1, x1, y2, x2]
|
||||
A_ROW = [499 / 640, 393 / 640, 620 / 640, 484 / 640]
|
||||
B_ROW = [499 / 640, 527 / 640, 620 / 640, 618 / 640]
|
||||
|
||||
|
||||
def kept(detections: np.ndarray) -> np.ndarray:
|
||||
"""Rows of the padded (20, 6) output that hold real detections."""
|
||||
return detections[detections[:, 1] > 0]
|
||||
|
||||
|
||||
class TestYoloNmsPostProcess(unittest.TestCase):
|
||||
def _single_output(self, rows: list[list[float]]) -> list[np.ndarray]:
|
||||
"""Build a single-tensor YOLO output (1, attrs, anchors) from
|
||||
[cx, cy, w, h, class scores...] rows, padded with empty anchors."""
|
||||
anchors = np.zeros((10, len(rows[0])), dtype=np.float32)
|
||||
anchors[: len(rows)] = np.array(rows, dtype=np.float32)
|
||||
return [anchors.T[np.newaxis, ...]]
|
||||
|
||||
def test_keeps_separated_objects_far_from_origin(self):
|
||||
output = self._single_output(
|
||||
[
|
||||
[A_CX, A_CY, A_W, A_H, 0.90, 0.0],
|
||||
[B_CX, B_CY, B_W, B_H, 0.0, 0.85],
|
||||
]
|
||||
)
|
||||
|
||||
detections = kept(post_process_yolo(output, WIDTH, HEIGHT))
|
||||
|
||||
self.assertEqual(len(detections), 2)
|
||||
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
|
||||
|
||||
def test_still_suppresses_true_duplicates(self):
|
||||
# same object twice, shifted 4 px: true IoU 0.92, must dedupe to one
|
||||
output = self._single_output(
|
||||
[
|
||||
[A_CX, A_CY, A_W, A_H, 0.90, 0.0],
|
||||
[A_CX + 4, A_CY, A_W, A_H, 0.85, 0.0],
|
||||
]
|
||||
)
|
||||
|
||||
detections = kept(post_process_yolo(output, WIDTH, HEIGHT))
|
||||
|
||||
self.assertEqual(len(detections), 1)
|
||||
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||
|
||||
|
||||
class TestMultipartYoloPostProcess(unittest.TestCase):
|
||||
def _multipart_output(self) -> list[np.ndarray]:
|
||||
"""Build a 3-scale anchor-based YOLO output containing boxes A and B,
|
||||
both decoded through anchor 0 of the stride-32 scale."""
|
||||
outputs = [
|
||||
np.zeros((1, 255, 80, 80), dtype=np.float32),
|
||||
np.zeros((1, 255, 40, 40), dtype=np.float32),
|
||||
np.zeros((1, 255, 20, 20), dtype=np.float32),
|
||||
]
|
||||
stride, (anchor_w, anchor_h) = 32, (142, 110)
|
||||
|
||||
for cx, cy, w, h, conf, class_channel in [
|
||||
(A_CX, A_CY, A_W, A_H, 0.95, 5), # class 0
|
||||
(B_CX, B_CY, B_W, B_H, 0.90, 6), # class 1
|
||||
]:
|
||||
cell_x, cell_y = int(cx // stride), int(cy // stride)
|
||||
dx = (cx / stride - cell_x + 0.5) / 2
|
||||
dy = (cy / stride - cell_y + 0.5) / 2
|
||||
dw = math.sqrt(w / anchor_w) / 2
|
||||
dh = math.sqrt(h / anchor_h) / 2
|
||||
# anchor 0 occupies channels 0-84 of the 255 channel tensor
|
||||
outputs[2][0, 0:4, cell_y, cell_x] = [dx, dy, dw, dh]
|
||||
outputs[2][0, 4, cell_y, cell_x] = conf
|
||||
outputs[2][0, class_channel, cell_y, cell_x] = 1.0
|
||||
|
||||
return outputs
|
||||
|
||||
def test_keeps_separated_objects_far_from_origin(self):
|
||||
detections = kept(post_process_yolo(self._multipart_output(), WIDTH, HEIGHT))
|
||||
|
||||
self.assertEqual(len(detections), 2)
|
||||
np.testing.assert_allclose(detections[0], [0, 0.95, *A_ROW], atol=2e-3)
|
||||
np.testing.assert_allclose(detections[1], [1, 0.90, *B_ROW], atol=2e-3)
|
||||
|
||||
def test_empty_output_returns_no_detections(self):
|
||||
outputs = [
|
||||
np.zeros((1, 255, 80, 80), dtype=np.float32),
|
||||
np.zeros((1, 255, 40, 40), dtype=np.float32),
|
||||
np.zeros((1, 255, 20, 20), dtype=np.float32),
|
||||
]
|
||||
|
||||
detections = kept(post_process_yolo(outputs, WIDTH, HEIGHT))
|
||||
|
||||
self.assertEqual(len(detections), 0)
|
||||
|
||||
|
||||
class TestYoloxPostProcess(unittest.TestCase):
|
||||
def test_keeps_separated_objects_far_from_origin(self):
|
||||
# with zero grids and unit strides the decode reduces to
|
||||
# cx = raw cx and w = exp(raw w)
|
||||
rows = np.zeros((10, 7), dtype=np.float32)
|
||||
rows[0] = [A_CX, A_CY, math.log(A_W), math.log(A_H), 1.0, 0.90, 0.0]
|
||||
rows[1] = [B_CX, B_CY, math.log(B_W), math.log(B_H), 1.0, 0.0, 0.85]
|
||||
predictions = rows[np.newaxis, ...]
|
||||
grids = np.zeros((1, 10, 2), dtype=np.float32)
|
||||
expanded_strides = np.ones((1, 10, 1), dtype=np.float32)
|
||||
|
||||
detections = kept(
|
||||
post_process_yolox(predictions, WIDTH, HEIGHT, grids, expanded_strides)
|
||||
)
|
||||
|
||||
self.assertEqual(len(detections), 2)
|
||||
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
|
||||
|
||||
|
||||
class TestDfinePostProcess(unittest.TestCase):
|
||||
def test_keeps_separated_objects_far_from_origin(self):
|
||||
# D-FINE emits absolute pixel xyxy boxes alongside labels and scores
|
||||
labels = np.zeros((1, 10), dtype=np.int64)
|
||||
labels[0, 1] = 1
|
||||
boxes = np.zeros((1, 10, 4), dtype=np.float32)
|
||||
boxes[0, 0] = [393, 499, 484, 620]
|
||||
boxes[0, 1] = [527, 499, 618, 620]
|
||||
scores = np.zeros((1, 10), dtype=np.float32)
|
||||
scores[0, 0] = 0.90
|
||||
scores[0, 1] = 0.85
|
||||
|
||||
detections = kept(post_process_dfine([labels, boxes, scores], WIDTH, HEIGHT))
|
||||
|
||||
self.assertEqual(len(detections), 2)
|
||||
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
|
||||
|
||||
|
||||
class TestRfdetrPostProcess(unittest.TestCase):
|
||||
def test_keeps_separated_objects_far_from_origin(self):
|
||||
# RF-DETR emits normalized center format boxes and class logits where
|
||||
# logit index 0 is the background class
|
||||
boxes = np.zeros((1, 10, 4), dtype=np.float32)
|
||||
boxes[0, 0] = [A_CX / WIDTH, A_CY / HEIGHT, A_W / WIDTH, A_H / HEIGHT]
|
||||
boxes[0, 1] = [B_CX / WIDTH, B_CY / HEIGHT, B_W / WIDTH, B_H / HEIGHT]
|
||||
# background heavy logits everywhere, then two confident objects
|
||||
logits = np.tile(np.array([10.0, 0.0, 0.0], dtype=np.float32), (1, 10, 1))
|
||||
logits[0, 0] = [0.0, 4.0, 0.0] # class 0 after background offset
|
||||
logits[0, 1] = [0.0, 0.0, 3.5] # class 1 after background offset
|
||||
|
||||
detections = kept(post_process_rfdetr([boxes, logits]))
|
||||
|
||||
conf_a = math.exp(4.0) / (math.exp(4.0) + 2)
|
||||
conf_b = math.exp(3.5) / (math.exp(3.5) + 2)
|
||||
self.assertEqual(len(detections), 2)
|
||||
np.testing.assert_allclose(detections[0], [0, conf_a, *A_ROW], atol=2e-3)
|
||||
np.testing.assert_allclose(detections[1], [1, conf_b, *B_ROW], atol=2e-3)
|
||||
|
||||
|
||||
class TestMemryxSsdlitePostProcess(unittest.TestCase):
|
||||
def test_keeps_separated_objects_far_from_origin(self):
|
||||
# the NMS math runs on the host CPU, so the real method is testable
|
||||
# without MemryX hardware; it only needs the model dimensions and
|
||||
# the output queue
|
||||
detector = object.__new__(MemryXDetector)
|
||||
detector.memx_model_width = WIDTH
|
||||
detector.memx_model_height = HEIGHT
|
||||
detector.output_queue = Queue()
|
||||
|
||||
# this path uses a 0.5 NMS threshold, so use a tighter pair: zero
|
||||
# true overlap (10 px gap), IoU 0.69 when misread as [x, y, w, h]
|
||||
dets = np.zeros((1, 10, 5), dtype=np.float32)
|
||||
dets[0, 0] = [480, 500, 540, 620, 0.90]
|
||||
dets[0, 1] = [550, 500, 610, 620, 0.85]
|
||||
labels = np.zeros((1, 10), dtype=np.float32)
|
||||
labels[0, 1] = 1
|
||||
|
||||
detector.post_process_ssdlite([dets, labels])
|
||||
detections = kept(detector.output_queue.get())
|
||||
|
||||
self.assertEqual(len(detections), 2)
|
||||
np.testing.assert_allclose(
|
||||
detections[0],
|
||||
[0, 0.90, 500 / 640, 480 / 640, 620 / 640, 540 / 640],
|
||||
atol=2e-3,
|
||||
)
|
||||
np.testing.assert_allclose(
|
||||
detections[1],
|
||||
[1, 0.85, 500 / 640, 550 / 640, 620 / 640, 610 / 640],
|
||||
atol=2e-3,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+37
-5
@@ -16,6 +16,31 @@ logger = logging.getLogger(__name__)
|
||||
### Post Processing
|
||||
|
||||
|
||||
def xyxy_to_xywh_for_nms(boxes: np.ndarray | list) -> np.ndarray:
|
||||
"""Convert [x1, y1, x2, y2] boxes to the [x, y, width, height] format
|
||||
that cv2.dnn.NMSBoxes expects.
|
||||
|
||||
Passing corner coordinates directly makes OpenCV treat x2/y2 as the box
|
||||
size, inflating every box toward the bottom-right by its distance from
|
||||
the origin, which suppresses valid detections near other objects.
|
||||
|
||||
Args:
|
||||
boxes: Array-like of shape (N, 4) in corner format.
|
||||
|
||||
Returns:
|
||||
Float32 array of shape (N, 4) in top-left plus size format.
|
||||
"""
|
||||
boxes = np.asarray(boxes, dtype=np.float32)
|
||||
|
||||
if boxes.size == 0:
|
||||
return np.zeros((0, 4), dtype=np.float32)
|
||||
|
||||
xywh = boxes.copy()
|
||||
xywh[:, 2] -= xywh[:, 0]
|
||||
xywh[:, 3] -= xywh[:, 1]
|
||||
return xywh
|
||||
|
||||
|
||||
def post_process_dfine(
|
||||
tensor_output: np.ndarray, width: int, height: int
|
||||
) -> np.ndarray:
|
||||
@@ -25,7 +50,9 @@ def post_process_dfine(
|
||||
|
||||
input_shape = np.array([height, width, height, width])
|
||||
boxes = np.divide(boxes, input_shape, dtype=np.float32)
|
||||
indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.4, nms_threshold=0.4)
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
xyxy_to_xywh_for_nms(boxes), scores, score_threshold=0.4, nms_threshold=0.4
|
||||
)
|
||||
detections = np.zeros((20, 6), np.float32)
|
||||
|
||||
for i, (bbox, confidence, class_id) in enumerate(
|
||||
@@ -78,7 +105,10 @@ def post_process_rfdetr(tensor_output: list[np.ndarray, np.ndarray]) -> np.ndarr
|
||||
|
||||
# apply nms
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
filtered_boxes, filtered_scores, score_threshold=0.4, nms_threshold=0.4
|
||||
xyxy_to_xywh_for_nms(filtered_boxes),
|
||||
filtered_scores,
|
||||
score_threshold=0.4,
|
||||
nms_threshold=0.4,
|
||||
)
|
||||
detections = np.zeros((20, 6), np.float32)
|
||||
|
||||
@@ -159,7 +189,7 @@ def __post_process_multipart_yolo(
|
||||
all_class_ids.append(class_id)
|
||||
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
bboxes=all_boxes,
|
||||
bboxes=xyxy_to_xywh_for_nms(all_boxes),
|
||||
scores=all_scores,
|
||||
score_threshold=0.4,
|
||||
nms_threshold=0.4,
|
||||
@@ -206,7 +236,9 @@ def __post_process_nms_yolo(predictions: np.ndarray, width, height) -> np.ndarra
|
||||
boxes = boxes_xyxy
|
||||
|
||||
# run NMS
|
||||
indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.4, nms_threshold=0.4)
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
xyxy_to_xywh_for_nms(boxes), scores, score_threshold=0.4, nms_threshold=0.4
|
||||
)
|
||||
detections = np.zeros((20, 6), np.float32)
|
||||
for i, (bbox, confidence, class_id) in enumerate(
|
||||
zip(boxes[indices], scores[indices], class_ids[indices])
|
||||
@@ -258,7 +290,7 @@ def post_process_yolox(
|
||||
scores = scores[np.arange(len(cls_inds)), cls_inds]
|
||||
|
||||
indices = cv2.dnn.NMSBoxes(
|
||||
boxes_xyxy, scores, score_threshold=0.4, nms_threshold=0.4
|
||||
xyxy_to_xywh_for_nms(boxes_xyxy), scores, score_threshold=0.4, nms_threshold=0.4
|
||||
)
|
||||
|
||||
detections = np.zeros((20, 6), np.float32)
|
||||
|
||||
@@ -1,209 +0,0 @@
|
||||
"""Aggregation of the known sub label names an object can be tagged with."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from pathvalidate import sanitize_filename
|
||||
|
||||
from frigate.config import FrigateConfig
|
||||
from frigate.config.classification import ObjectClassificationType
|
||||
from frigate.const import CLIPS_DIR, FACE_DIR, MODEL_CACHE_DIR
|
||||
from frigate.util.builtin import load_labels
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# subdirectory of FACE_DIR holding unassigned training images, not a face name
|
||||
FACE_TRAIN_DIR = "train"
|
||||
|
||||
# category used by classification models for "no match", never attached to an object
|
||||
CLASSIFICATION_NONE_CATEGORY = "none"
|
||||
|
||||
|
||||
def get_categorized_object_names(
|
||||
config: FrigateConfig,
|
||||
allowed_cameras: list[str],
|
||||
object_type: str | None = None,
|
||||
) -> dict[str, list[str]]:
|
||||
"""Collect every sub label name this install can attach, by object type.
|
||||
|
||||
Unlike the database-backed /sub_labels endpoint, this reads the config and
|
||||
model files, so it also covers names that are configured but have not been
|
||||
detected yet. Names come from the detector's logo attributes (limited to
|
||||
objects the allowed cameras actually track), LPR known plate names,
|
||||
registered face names, and custom object classification categories.
|
||||
|
||||
Structural attributes such as `face` and `license_plate` are excluded: they
|
||||
describe a part of an object rather than naming it, and are never attached
|
||||
as a sub label.
|
||||
|
||||
Args:
|
||||
config: The running Frigate config
|
||||
allowed_cameras: Cameras the requesting user may see
|
||||
object_type: Optional object label to restrict the result to
|
||||
|
||||
Returns:
|
||||
Mapping of object label to its known sub label names, sorted and
|
||||
deduplicated. Object types with no known names are omitted.
|
||||
"""
|
||||
tracked_objects = _get_tracked_objects(config, allowed_cameras)
|
||||
names: dict[str, set[str]] = {}
|
||||
logos = set(config.model.all_attribute_logos)
|
||||
|
||||
# 1. detector logo attributes, only for objects that are actually tracked
|
||||
for label, label_attributes in config.model.attributes_map.items():
|
||||
if label not in tracked_objects:
|
||||
continue
|
||||
|
||||
label_logos = logos.intersection(label_attributes)
|
||||
|
||||
if label_logos:
|
||||
names.setdefault(label, set()).update(label_logos)
|
||||
|
||||
# 2. LPR known plate names, for objects that can carry a plate
|
||||
if config.lpr.known_plates and _lpr_enabled(config, allowed_cameras):
|
||||
known_plates = set(config.lpr.known_plates)
|
||||
|
||||
for label in _objects_with_attribute(config, tracked_objects, "license_plate"):
|
||||
names.setdefault(label, set()).update(known_plates)
|
||||
|
||||
# 3. registered face names, for objects that can carry a face
|
||||
if _face_recognition_enabled(config, allowed_cameras):
|
||||
face_names = _get_face_names()
|
||||
|
||||
if face_names:
|
||||
for label in _objects_with_attribute(config, tracked_objects, "face"):
|
||||
names.setdefault(label, set()).update(face_names)
|
||||
|
||||
# 4. custom object classification categories
|
||||
for model_key, model_config in config.classification.custom.items():
|
||||
if not model_config.enabled or model_config.object_config is None:
|
||||
continue
|
||||
|
||||
if (
|
||||
model_config.object_config.classification_type
|
||||
!= ObjectClassificationType.sub_label
|
||||
):
|
||||
continue
|
||||
|
||||
categories = _get_classification_categories(model_key)
|
||||
|
||||
if not categories:
|
||||
continue
|
||||
|
||||
for label in model_config.object_config.objects:
|
||||
names.setdefault(label, set()).update(categories)
|
||||
|
||||
return {
|
||||
label: sorted(label_names)
|
||||
for label, label_names in sorted(names.items())
|
||||
if label_names and (object_type is None or label == object_type)
|
||||
}
|
||||
|
||||
|
||||
def _get_tracked_objects(config: FrigateConfig, allowed_cameras: list[str]) -> set[str]:
|
||||
"""Get the union of objects tracked by the cameras the user can see."""
|
||||
tracked: set[str] = set()
|
||||
|
||||
for camera_name in allowed_cameras:
|
||||
camera_config = config.cameras.get(camera_name)
|
||||
|
||||
if camera_config is None:
|
||||
continue
|
||||
|
||||
tracked.update(camera_config.objects.track)
|
||||
|
||||
return tracked
|
||||
|
||||
|
||||
def _objects_with_attribute(
|
||||
config: FrigateConfig, tracked_objects: set[str], attribute: str
|
||||
) -> set[str]:
|
||||
"""Get the tracked objects that a given attribute can be recognized on.
|
||||
|
||||
The attribute may also be tracked as an object in its own right, as
|
||||
`license_plate` is on a dedicated LPR camera, in which case the name is
|
||||
attached to that object directly.
|
||||
"""
|
||||
objects = {
|
||||
label
|
||||
for label, label_attributes in config.model.attributes_map.items()
|
||||
if attribute in label_attributes and label in tracked_objects
|
||||
}
|
||||
|
||||
if attribute in tracked_objects:
|
||||
objects.add(attribute)
|
||||
|
||||
return objects
|
||||
|
||||
|
||||
def _lpr_enabled(config: FrigateConfig, allowed_cameras: list[str]) -> bool:
|
||||
return any(
|
||||
config.cameras[camera_name].lpr.enabled
|
||||
for camera_name in allowed_cameras
|
||||
if camera_name in config.cameras
|
||||
)
|
||||
|
||||
|
||||
def _face_recognition_enabled(
|
||||
config: FrigateConfig, allowed_cameras: list[str]
|
||||
) -> bool:
|
||||
return any(
|
||||
config.cameras[camera_name].face_recognition.enabled
|
||||
for camera_name in allowed_cameras
|
||||
if camera_name in config.cameras
|
||||
)
|
||||
|
||||
|
||||
def _get_face_names() -> set[str]:
|
||||
"""Get the names of every registered face collection."""
|
||||
if not os.path.exists(FACE_DIR):
|
||||
return set()
|
||||
|
||||
try:
|
||||
entries = os.listdir(FACE_DIR)
|
||||
except OSError:
|
||||
logger.debug("Failed to read face directory %s", FACE_DIR)
|
||||
return set()
|
||||
|
||||
return {
|
||||
name
|
||||
for name in entries
|
||||
if name != FACE_TRAIN_DIR and os.path.isdir(os.path.join(FACE_DIR, name))
|
||||
}
|
||||
|
||||
|
||||
def _get_classification_categories(model_key: str) -> set[str]:
|
||||
"""Get the categories a custom classification model can output.
|
||||
|
||||
The trained labelmap is authoritative, but it only exists once the model
|
||||
has been trained, so fall back to the dataset directories that will become
|
||||
the labelmap on the next training run.
|
||||
"""
|
||||
safe_key = sanitize_filename(model_key)
|
||||
categories: set[str] = set()
|
||||
labelmap_path = os.path.join(MODEL_CACHE_DIR, safe_key, "labelmap.txt")
|
||||
|
||||
if os.path.exists(labelmap_path):
|
||||
try:
|
||||
labelmap = load_labels(labelmap_path, prefill=0, indexed=False)
|
||||
except OSError:
|
||||
logger.debug("Failed to read labelmap %s", labelmap_path)
|
||||
labelmap = {}
|
||||
|
||||
categories.update(label for label in labelmap.values() if label)
|
||||
|
||||
dataset_dir = os.path.join(CLIPS_DIR, safe_key, "dataset")
|
||||
|
||||
if os.path.exists(dataset_dir):
|
||||
try:
|
||||
entries = os.listdir(dataset_dir)
|
||||
except OSError:
|
||||
logger.debug("Failed to read dataset directory %s", dataset_dir)
|
||||
entries = []
|
||||
|
||||
categories.update(
|
||||
name for name in entries if os.path.isdir(os.path.join(dataset_dir, name))
|
||||
)
|
||||
|
||||
categories.discard(CLASSIFICATION_NONE_CATEGORY)
|
||||
return categories
|
||||
@@ -1,7 +1,6 @@
|
||||
"""Utilities for services."""
|
||||
|
||||
import asyncio
|
||||
import glob
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
@@ -671,33 +670,19 @@ def get_intel_gpu_stats(
|
||||
|
||||
def get_openvino_npu_stats() -> dict[str, str] | None:
|
||||
"""Get NPU stats using openvino."""
|
||||
for accel_path in sorted(glob.glob("/sys/class/accel/accel*")):
|
||||
try:
|
||||
driver = os.path.basename(os.readlink(f"{accel_path}/device/driver"))
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
if driver != "intel_vpu":
|
||||
continue
|
||||
NPU_RUNTIME_PATH = "/sys/devices/pci0000:00/0000:00:0b.0/power/runtime_active_time"
|
||||
|
||||
try:
|
||||
runtime_path = f"{accel_path}/device/power/runtime_active_time"
|
||||
with open(runtime_path) as f:
|
||||
with open(NPU_RUNTIME_PATH) as f:
|
||||
initial_runtime = float(f.read().strip())
|
||||
break
|
||||
except (FileNotFoundError, PermissionError, ValueError):
|
||||
continue
|
||||
else:
|
||||
return None
|
||||
|
||||
try:
|
||||
initial_time = time.time()
|
||||
|
||||
# Sleep for 1 second to get an accurate reading
|
||||
time.sleep(1.0)
|
||||
|
||||
# Read runtime value again
|
||||
with open(runtime_path) as f:
|
||||
with open(NPU_RUNTIME_PATH) as f:
|
||||
current_runtime = float(f.read().strip())
|
||||
|
||||
current_time = time.time()
|
||||
|
||||
@@ -859,8 +859,8 @@
|
||||
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
|
||||
},
|
||||
"dashboard": {
|
||||
"label": "Show in UI",
|
||||
"description": "Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again."
|
||||
"label": "Show on Live dashboard",
|
||||
"description": "Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings."
|
||||
},
|
||||
"review": {
|
||||
"label": "Show in review",
|
||||
|
||||
@@ -357,22 +357,6 @@
|
||||
"description": "Optional API key for authenticated DeepStack services."
|
||||
}
|
||||
},
|
||||
"degirum": {
|
||||
"label": "DeGirum",
|
||||
"description": "DeGirum detector for running models via DeGirum cloud or local inference services.",
|
||||
"location": {
|
||||
"label": "Inference Location",
|
||||
"description": "Location of the DeGirim inference engine (e.g. '@cloud', '127.0.0.1')."
|
||||
},
|
||||
"zoo": {
|
||||
"label": "Model Zoo",
|
||||
"description": "Path or URL to the DeGirum model zoo."
|
||||
},
|
||||
"token": {
|
||||
"label": "DeGirum Cloud Token",
|
||||
"description": "Token for DeGirum Cloud access."
|
||||
}
|
||||
},
|
||||
"edgetpu": {
|
||||
"label": "EdgeTPU",
|
||||
"description": "EdgeTPU detector that runs TensorFlow Lite models compiled for Coral EdgeTPU using the EdgeTPU delegate.",
|
||||
@@ -1543,8 +1527,8 @@
|
||||
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
|
||||
},
|
||||
"dashboard": {
|
||||
"label": "Show in UI",
|
||||
"description": "Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again."
|
||||
"label": "Show on Live dashboard",
|
||||
"description": "Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings."
|
||||
},
|
||||
"review": {
|
||||
"label": "Show in review",
|
||||
|
||||
@@ -499,7 +499,7 @@
|
||||
"webuiUrlHelp": "URL to visit the camera's web UI directly from the Debug view. Leave blank to disable the link.",
|
||||
"webuiUrlInvalid": "Must be a valid URL (e.g., https://example.com).",
|
||||
"dashboardLabel": "Show on Live dashboard",
|
||||
"dashboardHelp": "Show this camera on the Live dashboard.",
|
||||
"dashboardHelp": "Show this camera on the default All Cameras live dashboard. It remains available everywhere else, including camera groups.",
|
||||
"reviewLabel": "Show in Review",
|
||||
"reviewHelp": "Show this camera in Review, including the camera filter, motion review, and the history view."
|
||||
}
|
||||
|
||||
@@ -78,7 +78,7 @@ export default function ZoneEditPane({
|
||||
}
|
||||
|
||||
return Object.values(config.cameras)
|
||||
.filter((conf) => conf.ui.dashboard && conf.enabled_in_config)
|
||||
.filter((conf) => conf.enabled_in_config)
|
||||
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
|
||||
}, [config]);
|
||||
|
||||
|
||||
+32
-13
@@ -198,6 +198,19 @@ export default function Events() {
|
||||
return true;
|
||||
});
|
||||
|
||||
const reviewCamerasParam = useMemo(() => {
|
||||
const selected: string | undefined = reviewSearchParams["cameras"];
|
||||
|
||||
if (!selected) {
|
||||
return reviewCameras.join(",");
|
||||
}
|
||||
|
||||
const selectedCameras = new Set(selected.split(","));
|
||||
return reviewCameras
|
||||
.filter((camera) => selectedCameras.has(camera))
|
||||
.join(",");
|
||||
}, [reviewCameras, reviewSearchParams]);
|
||||
|
||||
useSearchEffect("labels", (labels: string) => {
|
||||
setReviewFilter({
|
||||
...reviewFilter,
|
||||
@@ -330,8 +343,12 @@ export default function Events() {
|
||||
}, []);
|
||||
|
||||
const getKey = useCallback(() => {
|
||||
if (!timezone) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const params = {
|
||||
cameras: reviewSearchParams["cameras"],
|
||||
cameras: reviewCamerasParam,
|
||||
labels: reviewSearchParams["labels"],
|
||||
zones: reviewSearchParams["zones"],
|
||||
reviewed: null, // We want both reviewed and unreviewed items as we filter in the UI
|
||||
@@ -339,7 +356,7 @@ export default function Events() {
|
||||
after: reviewSearchParams["after"] || last24Hours.after,
|
||||
};
|
||||
return ["review", params];
|
||||
}, [reviewSearchParams, last24Hours]);
|
||||
}, [reviewSearchParams, reviewCamerasParam, last24Hours, timezone]);
|
||||
|
||||
const { data: reviews, mutate: updateSegments } = useSWR<ReviewSegment[]>(
|
||||
getKey,
|
||||
@@ -361,10 +378,6 @@ export default function Events() {
|
||||
const motion: ReviewSegment[] = [];
|
||||
|
||||
reviews?.forEach((segment) => {
|
||||
if (config?.cameras[segment.camera]?.ui?.review === false) {
|
||||
return;
|
||||
}
|
||||
|
||||
all.push(segment);
|
||||
|
||||
switch (segment.severity) {
|
||||
@@ -386,7 +399,7 @@ export default function Events() {
|
||||
detection: detections,
|
||||
significant_motion: motion,
|
||||
};
|
||||
}, [reviews, config?.cameras]);
|
||||
}, [reviews]);
|
||||
|
||||
// update review items in place when a review segment ends
|
||||
const reviewUpdate = useFrigateReviews();
|
||||
@@ -450,15 +463,17 @@ export default function Events() {
|
||||
// review summary
|
||||
|
||||
const { data: reviewSummary, mutate: updateSummary } = useSWR<ReviewSummary>(
|
||||
[
|
||||
timezone
|
||||
? [
|
||||
"review/summary",
|
||||
{
|
||||
timezone: timezone,
|
||||
cameras: reviewSearchParams["cameras"] ?? null,
|
||||
cameras: reviewCamerasParam,
|
||||
labels: reviewSearchParams["labels"] ?? null,
|
||||
zones: reviewSearchParams["zones"] ?? null,
|
||||
},
|
||||
],
|
||||
]
|
||||
: null,
|
||||
{
|
||||
revalidateOnFocus: true,
|
||||
refreshInterval: 30000,
|
||||
@@ -473,13 +488,17 @@ export default function Events() {
|
||||
|
||||
// recordings summary
|
||||
|
||||
const { data: recordingsSummary } = useSWR<RecordingsSummary>([
|
||||
const { data: recordingsSummary } = useSWR<RecordingsSummary>(
|
||||
timezone
|
||||
? [
|
||||
"recordings/summary",
|
||||
{
|
||||
timezone: timezone,
|
||||
cameras: reviewSearchParams["cameras"] ?? null,
|
||||
cameras: reviewCamerasParam,
|
||||
},
|
||||
]);
|
||||
]
|
||||
: null,
|
||||
);
|
||||
|
||||
// preview videos
|
||||
const previewTimes = useMemo(() => {
|
||||
|
||||
@@ -706,12 +706,7 @@ export default function Settings() {
|
||||
}
|
||||
|
||||
return Object.values(config.cameras)
|
||||
.filter(
|
||||
(conf) =>
|
||||
conf.ui.dashboard &&
|
||||
conf.enabled_in_config &&
|
||||
!isReplayCamera(conf.name),
|
||||
)
|
||||
.filter((conf) => conf.enabled_in_config && !isReplayCamera(conf.name))
|
||||
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
|
||||
}, [config]);
|
||||
|
||||
|
||||
@@ -1024,6 +1024,9 @@ function MotionReview({
|
||||
if (!allowedCameras.includes(cam.name)) {
|
||||
return false;
|
||||
}
|
||||
if (cam.ui?.review === false) {
|
||||
return false;
|
||||
}
|
||||
if (selectedCams && !selectedCams.includes(cam.name)) {
|
||||
return false;
|
||||
}
|
||||
@@ -1033,6 +1036,11 @@ function MotionReview({
|
||||
return cameras.sort((a, b) => a.ui.order - b.ui.order);
|
||||
}, [config, filter, allowedCameras]);
|
||||
|
||||
const reviewCamerasParam = useMemo(
|
||||
() => reviewCameras.map((cam) => cam.name).join(","),
|
||||
[reviewCameras],
|
||||
);
|
||||
|
||||
const videoPlayersRef = useRef<{ [camera: string]: PreviewController }>({});
|
||||
|
||||
// motion data
|
||||
@@ -1052,7 +1060,7 @@ function MotionReview({
|
||||
before: alignedBefore,
|
||||
after: alignedAfter,
|
||||
scale: segmentDuration / 2,
|
||||
cameras: filter?.cameras?.join(",") ?? null,
|
||||
cameras: reviewCamerasParam,
|
||||
},
|
||||
]);
|
||||
|
||||
@@ -1061,7 +1069,7 @@ function MotionReview({
|
||||
{
|
||||
before: alignedBefore,
|
||||
after: alignedAfter,
|
||||
cameras: filter?.cameras?.join(",") ?? null,
|
||||
cameras: reviewCamerasParam,
|
||||
},
|
||||
]);
|
||||
|
||||
|
||||
@@ -540,6 +540,36 @@ export default function GeneralMetrics({
|
||||
return Object.keys(series).length > 0 ? Object.values(series) : undefined;
|
||||
}, [statsHistory]);
|
||||
|
||||
// Number of cards the hardware grid renders. Which ones appear depends on
|
||||
// the vendor, so the column count follows the count rather than assuming a
|
||||
// fixed set is present.
|
||||
const hardwareCardCount = useMemo(() => {
|
||||
if (!statsHistory[0]?.gpu_usages) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
const hasNpu = statsHistory[0].npu_usages != undefined;
|
||||
|
||||
return (
|
||||
1 + // gpu usage always renders alongside gpu_usages
|
||||
(gpuMemSeries ? 1 : 0) +
|
||||
(gpuEncSeries?.length ? 1 : 0) +
|
||||
(gpuComputeSeries?.length ? 1 : 0) +
|
||||
(gpuDecSeries?.length ? 1 : 0) +
|
||||
(gpuTempSeries?.length ? 1 : 0) +
|
||||
(hasNpu ? 1 : 0) +
|
||||
(hasNpu && npuTempSeries?.length ? 1 : 0)
|
||||
);
|
||||
}, [
|
||||
statsHistory,
|
||||
gpuMemSeries,
|
||||
gpuEncSeries,
|
||||
gpuComputeSeries,
|
||||
gpuDecSeries,
|
||||
gpuTempSeries,
|
||||
npuTempSeries,
|
||||
]);
|
||||
|
||||
// other processes stats
|
||||
|
||||
const hardwareType = useMemo(() => {
|
||||
@@ -763,12 +793,9 @@ export default function GeneralMetrics({
|
||||
<div
|
||||
className={cn(
|
||||
"mt-4 grid grid-cols-1 gap-2 sm:grid-cols-2",
|
||||
gpuTempSeries?.length && "md:grid-cols-3",
|
||||
(gpuEncSeries?.length || gpuComputeSeries?.length) &&
|
||||
"xl:grid-cols-4",
|
||||
(gpuEncSeries?.length || gpuComputeSeries?.length) &&
|
||||
gpuTempSeries?.length &&
|
||||
"3xl:grid-cols-5",
|
||||
hardwareCardCount >= 3 && "lg:grid-cols-3",
|
||||
hardwareCardCount >= 4 && "xl:grid-cols-4",
|
||||
hardwareCardCount >= 5 && "3xl:grid-cols-5",
|
||||
)}
|
||||
>
|
||||
{statsHistory[0]?.gpu_usages && (
|
||||
|
||||
Reference in New Issue
Block a user