mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-08-01 16:42:18 +03:00
Compare commits
3
Commits
0.19
...
remove-degirum
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
332adaa717 | ||
|
|
b848c90f02 | ||
|
|
f1cc0e49d4 |
@@ -79,7 +79,5 @@ sherpa-onnx==1.12.*
|
|||||||
faster-whisper==1.1.*
|
faster-whisper==1.1.*
|
||||||
librosa==0.11.*
|
librosa==0.11.*
|
||||||
soundfile==0.13.*
|
soundfile==0.13.*
|
||||||
# DeGirum detector
|
|
||||||
degirum == 0.16.*
|
|
||||||
# Memory profiling
|
# Memory profiling
|
||||||
memray == 1.15.*
|
memray == 1.15.*
|
||||||
|
|||||||
@@ -1269,78 +1269,3 @@ axengine:
|
|||||||
input_dtype: int
|
input_dtype: int
|
||||||
input_pixel_format: bgr
|
input_pixel_format: bgr
|
||||||
labelmap_path: /labelmap/coco-80.txt
|
labelmap_path: /labelmap/coco-80.txt
|
||||||
degirumAiServer:
|
|
||||||
title: DeGirum AI Server
|
|
||||||
models:
|
|
||||||
- key: ai-server-inference
|
|
||||||
label: AI Server Inference
|
|
||||||
recommended: true
|
|
||||||
download: |-
|
|
||||||
Launch a DeGirum AI server as a Docker container, then point the detector at it. Add this to your `docker-compose.yml`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
degirum_detector:
|
|
||||||
container_name: degirum
|
|
||||||
image: degirum/aiserver:latest
|
|
||||||
privileged: true
|
|
||||||
ports:
|
|
||||||
- "8778:8778"
|
|
||||||
```
|
|
||||||
|
|
||||||
Set `location` to the server's service name, container name, or `host:port`.
|
|
||||||
ui: |
|
|
||||||
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
|
|
||||||
|
|
||||||
| Field | Value |
|
|
||||||
| --- | --- |
|
|
||||||
| **Location** | `degirum` |
|
|
||||||
| **Zoo** | `degirum/public` |
|
|
||||||
| **Token** | your AI Hub token (optional for the public zoo) |
|
|
||||||
yaml: |
|
|
||||||
degirum_detector:
|
|
||||||
type: degirum
|
|
||||||
location: degirum
|
|
||||||
zoo: degirum/public
|
|
||||||
token: dg_example_token
|
|
||||||
degirumLocal:
|
|
||||||
title: DeGirum Local
|
|
||||||
models:
|
|
||||||
- key: local-inference
|
|
||||||
label: Local Inference
|
|
||||||
recommended: true
|
|
||||||
download: Run hardware directly inside the Frigate container with `@local`, removing the AI server hop. The matching device runtime (e.g. the Hailo runtime) must be installed in the container; confirm it with `degirum sys-info`.
|
|
||||||
ui: |
|
|
||||||
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
|
|
||||||
|
|
||||||
| Field | Value |
|
|
||||||
| --- | --- |
|
|
||||||
| **Location** | `@local` |
|
|
||||||
| **Zoo** | `degirum/public` |
|
|
||||||
| **Token** | your AI Hub token (optional for the public zoo) |
|
|
||||||
yaml: |
|
|
||||||
degirum_detector:
|
|
||||||
type: degirum
|
|
||||||
location: @local
|
|
||||||
zoo: degirum/public
|
|
||||||
token: dg_example_token
|
|
||||||
degirumCloud:
|
|
||||||
title: DeGirum AI Hub Cloud
|
|
||||||
models:
|
|
||||||
- key: ai-hub-cloud-inference
|
|
||||||
label: AI Hub Cloud Inference
|
|
||||||
recommended: true
|
|
||||||
download: Run inferences on DeGirum's [AI Hub](https://hub.degirum.com) cloud with `@cloud`. Sign up, create an access token, and set it as `token`. Network latency may require lowering your detection fps.
|
|
||||||
ui: |
|
|
||||||
Navigate to **Settings > System > Detectors and model** and select **DeGirum** from the detector type dropdown and click **Add**.
|
|
||||||
|
|
||||||
| Field | Value |
|
|
||||||
| --- | --- |
|
|
||||||
| **Location** | `@cloud` |
|
|
||||||
| **Zoo** | `degirum/public` |
|
|
||||||
| **Token** | your AI Hub token (optional for the public zoo) |
|
|
||||||
yaml: |
|
|
||||||
degirum_detector:
|
|
||||||
type: degirum
|
|
||||||
location: @cloud
|
|
||||||
zoo: degirum/public
|
|
||||||
token: dg_example_token
|
|
||||||
|
|||||||
@@ -981,7 +981,9 @@ cameras:
|
|||||||
# Optional: Adjust sort order of cameras in the UI. Larger numbers come later (default: shown below)
|
# Optional: Adjust sort order of cameras in the UI. Larger numbers come later (default: shown below)
|
||||||
# By default the cameras are sorted alphabetically.
|
# By default the cameras are sorted alphabetically.
|
||||||
order: 0
|
order: 0
|
||||||
# Optional: Whether or not to show the camera in the Frigate UI (default: shown below)
|
# Optional: Whether or not to show the camera on the default All Cameras live dashboard.
|
||||||
|
# The camera is still available everywhere else, including camera groups and settings
|
||||||
|
# (default: shown below)
|
||||||
dashboard: True
|
dashboard: True
|
||||||
# Optional: Whether this camera is visible in review (the review page and its camera
|
# Optional: Whether this camera is visible in review (the review page and its camera
|
||||||
# filter, motion review, and the history view) (default: shown below)
|
# filter, motion review, and the history view) (default: shown below)
|
||||||
|
|||||||
@@ -334,7 +334,7 @@ When your browser runs into problems playing back your camera streams, it will l
|
|||||||
|
|
||||||
- **stalled**
|
- **stalled**
|
||||||
- What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving).
|
- What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving).
|
||||||
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in the UI pane of Frigate's settings.
|
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in <NavPath path="Settings > UI" /> .
|
||||||
|
|
||||||
- Possible console messages from the player code:
|
- Possible console messages from the player code:
|
||||||
- `Buffer time (10 seconds) exceeded, browser may not be playing media correctly.`
|
- `Buffer time (10 seconds) exceeded, browser may not be playing media correctly.`
|
||||||
|
|||||||
@@ -24,7 +24,6 @@ Frigate supports multiple different detectors that work on different types of ha
|
|||||||
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
|
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
|
||||||
- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
|
- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
|
||||||
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
|
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
|
||||||
- <CommunityBadge /> [DeGirum](#degirum): Service for using hardware devices in the cloud or locally. Hardware and models provided on the cloud on [their website](https://hub.degirum.com).
|
|
||||||
|
|
||||||
**AMD**
|
**AMD**
|
||||||
|
|
||||||
@@ -755,87 +754,6 @@ Explanation of the parameters:
|
|||||||
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
|
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
|
||||||
- `config`: Configuration passed to `rknn-toolkit2` for model conversion. For an explanation of all available parameters have a look at section "2.2. Model configuration" of [this manual](https://github.com/MarcA711/rknn-toolkit2/releases/download/v2.3.2/03_Rockchip_RKNPU_API_Reference_RKNN_Toolkit2_V2.3.2_EN.pdf).
|
- `config`: Configuration passed to `rknn-toolkit2` for model conversion. For an explanation of all available parameters have a look at section "2.2. Model configuration" of [this manual](https://github.com/MarcA711/rknn-toolkit2/releases/download/v2.3.2/03_Rockchip_RKNPU_API_Reference_RKNN_Toolkit2_V2.3.2_EN.pdf).
|
||||||
|
|
||||||
## DeGirum
|
|
||||||
|
|
||||||
DeGirum is a detector that can use any type of hardware listed on [their website](https://hub.degirum.com). DeGirum can be used with local hardware through a DeGirum AI Server, or through the use of `@local`. You can also connect directly to DeGirum's AI Hub to run inferences. **Please Note:** This detector _cannot_ be used for commercial purposes.
|
|
||||||
|
|
||||||
### Configuration {#configuration-degirum}
|
|
||||||
|
|
||||||
#### AI Server Inference
|
|
||||||
|
|
||||||
Before starting with the config file for this section, you must first launch an AI server. DeGirum has an AI server ready to use as a docker container. Add this to your `docker-compose.yml` to get started:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
degirum_detector:
|
|
||||||
container_name: degirum
|
|
||||||
image: degirum/aiserver:latest
|
|
||||||
privileged: true
|
|
||||||
ports:
|
|
||||||
- "8778:8778"
|
|
||||||
```
|
|
||||||
|
|
||||||
All supported hardware will automatically be found on your AI server host as long as relevant runtimes and drivers are properly installed on your machine. Refer to [DeGirum's docs site](https://docs.degirum.com/pysdk/runtimes-and-drivers) if you have any trouble.
|
|
||||||
|
|
||||||
Once completed, configure the detector as follows:
|
|
||||||
|
|
||||||
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumAiServer.models} />
|
|
||||||
|
|
||||||
Setting up a model in the `config.yml` is similar to setting up an AI server.
|
|
||||||
You can set it to:
|
|
||||||
|
|
||||||
- A model listed on the [AI Hub](https://hub.degirum.com), given that the correct zoo name is listed in your detector
|
|
||||||
- If this is what you choose to do, the correct model will be downloaded onto your machine before running.
|
|
||||||
- A local directory acting as a zoo. See DeGirum's docs site [for more information](https://docs.degirum.com/pysdk/user-guide-pysdk/organizing-models#model-zoo-directory-structure).
|
|
||||||
- A path to some model.json.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
model:
|
|
||||||
path: ./mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1 # directory to model .json and file
|
|
||||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
|
||||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
|
||||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Local Inference
|
|
||||||
|
|
||||||
It is also possible to eliminate the need for an AI server and run the hardware directly. The benefit of this approach is that you eliminate any bottlenecks that occur when transferring prediction results from the AI server docker container to the frigate one. However, the method of implementing local inference is different for every device and hardware combination, so it's usually more trouble than it's worth. A general guideline to achieve this would be:
|
|
||||||
|
|
||||||
1. Ensuring that the frigate docker container has the runtime you want to use. So for instance, running `@local` for Hailo means making sure the container you're using has the Hailo runtime installed.
|
|
||||||
2. To double check the runtime is detected by the DeGirum detector, make sure the `degirum sys-info` command properly shows whatever runtimes you mean to install.
|
|
||||||
3. Create a DeGirum detector in your configuration.
|
|
||||||
|
|
||||||
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumLocal.models} />
|
|
||||||
|
|
||||||
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
model:
|
|
||||||
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
|
|
||||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
|
||||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
|
||||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
|
||||||
```
|
|
||||||
|
|
||||||
#### AI Hub Cloud Inference
|
|
||||||
|
|
||||||
If you do not possess whatever hardware you want to run, there's also the option to run cloud inferences. Do note that your detection fps might need to be lowered as network latency does significantly slow down this method of detection. For use with Frigate, we highly recommend using a local AI server as described above. To set up cloud inferences,
|
|
||||||
|
|
||||||
1. Sign up at [DeGirum's AI Hub](https://hub.degirum.com).
|
|
||||||
2. Get an access token.
|
|
||||||
3. Create a DeGirum detector in your configuration.
|
|
||||||
|
|
||||||
<ModelConfigDropdown detectorTitle="DeGirum" models={objectDetectorsModels.degirumCloud.models} />
|
|
||||||
|
|
||||||
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
model:
|
|
||||||
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
|
|
||||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
|
||||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
|
||||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
|
||||||
```
|
|
||||||
|
|
||||||
## AXERA
|
## AXERA
|
||||||
|
|
||||||
Hardware accelerated object detection is supported on the following SoCs:
|
Hardware accelerated object detection is supported on the following SoCs:
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ The per-clip variation is typically quite low and is mostly an artifact of keyfr
|
|||||||
|
|
||||||
Debug Replay lets you re-run Frigate's detection pipeline against a section of recorded video without manually configuring a dummy camera. It automatically extracts the recording, creates a temporary camera with the same detection settings as the original, and loops the clip through the pipeline so you can observe detections in real time.
|
Debug Replay lets you re-run Frigate's detection pipeline against a section of recorded video without manually configuring a dummy camera. It automatically extracts the recording, creates a temporary camera with the same detection settings as the original, and loops the clip through the pipeline so you can observe detections in real time.
|
||||||
|
|
||||||
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame.
|
The replay camera behaves like a live camera feed rather than History's video player: it loops the clip continuously as Frigate analyzes it and has no playback controls, so you cannot pause, scrub, or step through it frame by frame. The Debug Replay camera does not save recordings or snapshots or surface anything in Explore, but it otherwise behaves like a regular camera, including running enrichments such as Face Recognition, LPR, and custom classification.
|
||||||
|
|
||||||
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
|
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
|
||||||
|
|
||||||
|
|||||||
@@ -13,8 +13,8 @@ class CameraUiConfig(FrigateBaseModel):
|
|||||||
)
|
)
|
||||||
dashboard: bool = Field(
|
dashboard: bool = Field(
|
||||||
default=True,
|
default=True,
|
||||||
title="Show in UI",
|
title="Show on Live dashboard",
|
||||||
description="Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again.",
|
description="Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings.",
|
||||||
)
|
)
|
||||||
review: bool = Field(
|
review: bool = Field(
|
||||||
default=True,
|
default=True,
|
||||||
|
|||||||
@@ -1,157 +0,0 @@
|
|||||||
import logging
|
|
||||||
import queue
|
|
||||||
from typing import Literal
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
from pydantic import ConfigDict, Field
|
|
||||||
|
|
||||||
from frigate.detectors.detection_api import DetectionApi
|
|
||||||
from frigate.detectors.detector_config import BaseDetectorConfig
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
DETECTOR_KEY = "degirum"
|
|
||||||
|
|
||||||
|
|
||||||
### DETECTOR CONFIG ###
|
|
||||||
class DGDetectorConfig(BaseDetectorConfig):
|
|
||||||
"""DeGirum detector for running models via DeGirum cloud or local inference services."""
|
|
||||||
|
|
||||||
model_config = ConfigDict(
|
|
||||||
title="DeGirum",
|
|
||||||
)
|
|
||||||
|
|
||||||
type: Literal[DETECTOR_KEY]
|
|
||||||
location: str = Field(
|
|
||||||
default=None,
|
|
||||||
title="Inference Location",
|
|
||||||
description="Location of the DeGirim inference engine (e.g. '@cloud', '127.0.0.1').",
|
|
||||||
)
|
|
||||||
zoo: str = Field(
|
|
||||||
default=None,
|
|
||||||
title="Model Zoo",
|
|
||||||
description="Path or URL to the DeGirum model zoo.",
|
|
||||||
)
|
|
||||||
token: str = Field(
|
|
||||||
default=None,
|
|
||||||
title="DeGirum Cloud Token",
|
|
||||||
description="Token for DeGirum Cloud access.",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
### ACTUAL DETECTOR ###
|
|
||||||
class DGDetector(DetectionApi):
|
|
||||||
type_key = DETECTOR_KEY
|
|
||||||
|
|
||||||
def __init__(self, detector_config: DGDetectorConfig):
|
|
||||||
try:
|
|
||||||
import degirum as dg
|
|
||||||
except ModuleNotFoundError:
|
|
||||||
raise ImportError("Unable to import DeGirum detector.") from None
|
|
||||||
|
|
||||||
self._queue = queue.Queue()
|
|
||||||
self._zoo = dg.connect(
|
|
||||||
detector_config.location, detector_config.zoo, detector_config.token
|
|
||||||
)
|
|
||||||
|
|
||||||
logger.debug(f"Models in zoo: {self._zoo.list_models()}")
|
|
||||||
|
|
||||||
self.dg_model = self._zoo.load_model(
|
|
||||||
detector_config.model.path,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Setting input image format to raw reduces preprocessing time
|
|
||||||
self.dg_model.input_image_format = "RAW"
|
|
||||||
|
|
||||||
# Prioritize the most powerful hardware available
|
|
||||||
self.select_best_device_type()
|
|
||||||
# Frigate handles pre processing as long as these are all set
|
|
||||||
input_shape = self.dg_model.input_shape[0]
|
|
||||||
self.model_height = input_shape[1]
|
|
||||||
self.model_width = input_shape[2]
|
|
||||||
|
|
||||||
# Passing in dummy frame so initial connection latency happens in
|
|
||||||
# init function and not during actual prediction
|
|
||||||
frame = np.zeros(
|
|
||||||
(detector_config.model.width, detector_config.model.height, 3),
|
|
||||||
dtype=np.uint8,
|
|
||||||
)
|
|
||||||
# Pass in frame to overcome first frame latency
|
|
||||||
self.dg_model(frame)
|
|
||||||
self.prediction = self.prediction_generator()
|
|
||||||
|
|
||||||
def select_best_device_type(self):
|
|
||||||
"""
|
|
||||||
Helper function that selects fastest hardware available per model runtime
|
|
||||||
"""
|
|
||||||
types = self.dg_model.supported_device_types
|
|
||||||
|
|
||||||
device_map = {
|
|
||||||
"OPENVINO": ["GPU", "NPU", "CPU"],
|
|
||||||
"HAILORT": ["HAILO8L", "HAILO8"],
|
|
||||||
"N2X": ["ORCA1", "CPU"],
|
|
||||||
"ONNX": ["VITIS_NPU", "CPU"],
|
|
||||||
"RKNN": ["RK3566", "RK3568", "RK3588"],
|
|
||||||
"TENSORRT": ["DLA", "GPU", "DLA_ONLY"],
|
|
||||||
"TFLITE": ["ARMNN", "EDGETPU", "CPU"],
|
|
||||||
}
|
|
||||||
|
|
||||||
runtime = types[0].split("/")[0]
|
|
||||||
# Just create an array of format {runtime}/{hardware} for every hardware
|
|
||||||
# in the value for appropriate key in device_map
|
|
||||||
self.dg_model.device_type = [
|
|
||||||
f"{runtime}/{hardware}" for hardware in device_map[runtime]
|
|
||||||
]
|
|
||||||
|
|
||||||
def prediction_generator(self):
|
|
||||||
"""
|
|
||||||
Generator for all incoming frames. By using this generator, we don't have to keep
|
|
||||||
reconnecting our websocket on every "predict" call.
|
|
||||||
"""
|
|
||||||
logger.debug("Prediction generator was called")
|
|
||||||
with self.dg_model as model:
|
|
||||||
while 1:
|
|
||||||
logger.info(f"q size before calling get: {self._queue.qsize()}")
|
|
||||||
data = self._queue.get(block=True)
|
|
||||||
logger.info(f"q size after calling get: {self._queue.qsize()}")
|
|
||||||
logger.debug(
|
|
||||||
f"Data we're passing into model predict: {data}, shape of data: {data.shape}"
|
|
||||||
)
|
|
||||||
result = model.predict(data)
|
|
||||||
logger.debug(f"Prediction result: {result}")
|
|
||||||
yield result
|
|
||||||
|
|
||||||
def detect_raw(self, tensor_input):
|
|
||||||
# Reshaping tensor to work with pysdk
|
|
||||||
truncated_input = tensor_input.reshape(tensor_input.shape[1:])
|
|
||||||
logger.debug(f"Detect raw was called for tensor input: {tensor_input}")
|
|
||||||
|
|
||||||
# add tensor_input to input queue
|
|
||||||
self._queue.put(truncated_input)
|
|
||||||
logger.debug(f"Queue size after adding truncated input: {self._queue.qsize()}")
|
|
||||||
|
|
||||||
# define empty detection result
|
|
||||||
detections = np.zeros((20, 6), np.float32)
|
|
||||||
# grab prediction
|
|
||||||
res = next(self.prediction)
|
|
||||||
|
|
||||||
# If we have an empty prediction, return immediately
|
|
||||||
if len(res.results) == 0 or len(res.results[0]) == 0:
|
|
||||||
return detections
|
|
||||||
|
|
||||||
i = 0
|
|
||||||
for result in res.results:
|
|
||||||
if i >= 20:
|
|
||||||
break
|
|
||||||
|
|
||||||
detections[i] = [
|
|
||||||
result["category_id"],
|
|
||||||
float(result["score"]),
|
|
||||||
result["bbox"][1] / self.model_height,
|
|
||||||
result["bbox"][0] / self.model_width,
|
|
||||||
result["bbox"][3] / self.model_height,
|
|
||||||
result["bbox"][2] / self.model_width,
|
|
||||||
]
|
|
||||||
i += 1
|
|
||||||
|
|
||||||
logger.debug(f"Detections output: {detections}")
|
|
||||||
return detections
|
|
||||||
@@ -9,6 +9,7 @@ from pydantic import ConfigDict, Field
|
|||||||
|
|
||||||
from frigate.detectors.detection_api import DetectionApi
|
from frigate.detectors.detection_api import DetectionApi
|
||||||
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
||||||
|
from frigate.util.model import xyxy_to_xywh_for_nms
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from tflite_runtime.interpreter import Interpreter, load_delegate
|
from tflite_runtime.interpreter import Interpreter, load_delegate
|
||||||
@@ -297,7 +298,7 @@ class EdgeTpuTfl(DetectionApi):
|
|||||||
# until after filtering out redundant boxes
|
# until after filtering out redundant boxes
|
||||||
# Shift the logit scores to be non-negative (required by cv2)
|
# Shift the logit scores to be non-negative (required by cv2)
|
||||||
indices = cv2.dnn.NMSBoxes(
|
indices = cv2.dnn.NMSBoxes(
|
||||||
bboxes=boxes_filtered_decoded,
|
bboxes=xyxy_to_xywh_for_nms(boxes_filtered_decoded),
|
||||||
scores=max_scores_filtered_shiftedpositive,
|
scores=max_scores_filtered_shiftedpositive,
|
||||||
score_threshold=(
|
score_threshold=(
|
||||||
self.min_logit_value + self.logit_shift_to_positive_values
|
self.min_logit_value + self.logit_shift_to_positive_values
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from frigate.detectors.detector_config import (
|
|||||||
ModelTypeEnum,
|
ModelTypeEnum,
|
||||||
)
|
)
|
||||||
from frigate.util.file import FileLock
|
from frigate.util.file import FileLock
|
||||||
|
from frigate.util.model import xyxy_to_xywh_for_nms
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -581,7 +582,7 @@ class MemryXDetector(DetectionApi):
|
|||||||
# Convert coordinates to integers
|
# Convert coordinates to integers
|
||||||
x_min, y_min, x_max, y_max = map(int, [x_min, y_min, x_max, y_max])
|
x_min, y_min, x_max, y_max = map(int, [x_min, y_min, x_max, y_max])
|
||||||
|
|
||||||
# Append valid detections [class_id, confidence, x, y, width, height]
|
# Append valid detections [class_id, confidence, x_min, y_min, x_max, y_max]
|
||||||
detections.append([class_id, confidence, x_min, y_min, x_max, y_max])
|
detections.append([class_id, confidence, x_min, y_min, x_max, y_max])
|
||||||
|
|
||||||
final_detections = np.zeros((20, 6), np.float32)
|
final_detections = np.zeros((20, 6), np.float32)
|
||||||
@@ -595,7 +596,7 @@ class MemryXDetector(DetectionApi):
|
|||||||
detections = np.array(detections, dtype=np.float32)
|
detections = np.array(detections, dtype=np.float32)
|
||||||
|
|
||||||
# Apply Non-Maximum Suppression (NMS)
|
# Apply Non-Maximum Suppression (NMS)
|
||||||
bboxes = detections[:, 2:6].tolist() # (x_min, y_min, width, height)
|
bboxes = xyxy_to_xywh_for_nms(detections[:, 2:6])
|
||||||
scores = detections[:, 1].tolist() # Confidence scores
|
scores = detections[:, 1].tolist() # Confidence scores
|
||||||
|
|
||||||
indices = cv2.dnn.NMSBoxes(bboxes, scores, 0.45, 0.5)
|
indices = cv2.dnn.NMSBoxes(bboxes, scores, 0.45, 0.5)
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ from frigate.const import MODEL_CACHE_DIR, SUPPORTED_RK_SOCS
|
|||||||
from frigate.detectors.detection_api import DetectionApi
|
from frigate.detectors.detection_api import DetectionApi
|
||||||
from frigate.detectors.detection_runners import RKNNModelRunner
|
from frigate.detectors.detection_runners import RKNNModelRunner
|
||||||
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
||||||
from frigate.util.model import post_process_yolo
|
from frigate.util.model import post_process_yolo, xyxy_to_xywh_for_nms
|
||||||
from frigate.util.rknn_converter import auto_convert_model
|
from frigate.util.rknn_converter import auto_convert_model
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -285,7 +285,7 @@ class Rknn(DetectionApi):
|
|||||||
|
|
||||||
# run nms
|
# run nms
|
||||||
indices = cv2.dnn.NMSBoxes(
|
indices = cv2.dnn.NMSBoxes(
|
||||||
bboxes=boxes,
|
bboxes=xyxy_to_xywh_for_nms(boxes),
|
||||||
scores=scores,
|
scores=scores,
|
||||||
score_threshold=0.4,
|
score_threshold=0.4,
|
||||||
nms_threshold=0.4,
|
nms_threshold=0.4,
|
||||||
|
|||||||
@@ -0,0 +1,226 @@
|
|||||||
|
"""Tests for detector post-processing NMS box format handling.
|
||||||
|
|
||||||
|
cv2.dnn.NMSBoxes expects boxes as [x, y, width, height]. Passing corner
|
||||||
|
coordinates [x1, y1, x2, y2] makes OpenCV treat x2/y2 as width/height,
|
||||||
|
inflating every box toward the bottom-right by its distance from the origin.
|
||||||
|
Two well separated objects far from the origin then appear to overlap and the
|
||||||
|
lower scoring one is silently suppressed.
|
||||||
|
|
||||||
|
The regression geometry used throughout: two boxes with zero true overlap,
|
||||||
|
A = (393, 499, 484, 620) and B = (527, 499, 618, 620) in a 640x640 input
|
||||||
|
(43 px gap). Misread as [x, y, w, h] their IoU is 0.465, above the 0.4 NMS
|
||||||
|
threshold, so the buggy format drops the lower scoring box while correct
|
||||||
|
conversion keeps both.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import math
|
||||||
|
import unittest
|
||||||
|
from queue import Queue
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from frigate.detectors.plugins.memryx import MemryXDetector
|
||||||
|
from frigate.util.model import (
|
||||||
|
post_process_dfine,
|
||||||
|
post_process_rfdetr,
|
||||||
|
post_process_yolo,
|
||||||
|
post_process_yolox,
|
||||||
|
)
|
||||||
|
|
||||||
|
WIDTH = 640
|
||||||
|
HEIGHT = 640
|
||||||
|
|
||||||
|
# box A: xyxy (393, 499, 484, 620) as center format
|
||||||
|
A_CX, A_CY, A_W, A_H = 438.5, 559.5, 91.0, 121.0
|
||||||
|
# box B: xyxy (527, 499, 618, 620) as center format
|
||||||
|
B_CX, B_CY, B_W, B_H = 572.5, 559.5, 91.0, 121.0
|
||||||
|
|
||||||
|
# expected normalized output rows: [class_id, conf, y1, x1, y2, x2]
|
||||||
|
A_ROW = [499 / 640, 393 / 640, 620 / 640, 484 / 640]
|
||||||
|
B_ROW = [499 / 640, 527 / 640, 620 / 640, 618 / 640]
|
||||||
|
|
||||||
|
|
||||||
|
def kept(detections: np.ndarray) -> np.ndarray:
|
||||||
|
"""Rows of the padded (20, 6) output that hold real detections."""
|
||||||
|
return detections[detections[:, 1] > 0]
|
||||||
|
|
||||||
|
|
||||||
|
class TestYoloNmsPostProcess(unittest.TestCase):
|
||||||
|
def _single_output(self, rows: list[list[float]]) -> list[np.ndarray]:
|
||||||
|
"""Build a single-tensor YOLO output (1, attrs, anchors) from
|
||||||
|
[cx, cy, w, h, class scores...] rows, padded with empty anchors."""
|
||||||
|
anchors = np.zeros((10, len(rows[0])), dtype=np.float32)
|
||||||
|
anchors[: len(rows)] = np.array(rows, dtype=np.float32)
|
||||||
|
return [anchors.T[np.newaxis, ...]]
|
||||||
|
|
||||||
|
def test_keeps_separated_objects_far_from_origin(self):
|
||||||
|
output = self._single_output(
|
||||||
|
[
|
||||||
|
[A_CX, A_CY, A_W, A_H, 0.90, 0.0],
|
||||||
|
[B_CX, B_CY, B_W, B_H, 0.0, 0.85],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
detections = kept(post_process_yolo(output, WIDTH, HEIGHT))
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 2)
|
||||||
|
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||||
|
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
|
||||||
|
|
||||||
|
def test_still_suppresses_true_duplicates(self):
|
||||||
|
# same object twice, shifted 4 px: true IoU 0.92, must dedupe to one
|
||||||
|
output = self._single_output(
|
||||||
|
[
|
||||||
|
[A_CX, A_CY, A_W, A_H, 0.90, 0.0],
|
||||||
|
[A_CX + 4, A_CY, A_W, A_H, 0.85, 0.0],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
detections = kept(post_process_yolo(output, WIDTH, HEIGHT))
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 1)
|
||||||
|
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||||
|
|
||||||
|
|
||||||
|
class TestMultipartYoloPostProcess(unittest.TestCase):
|
||||||
|
def _multipart_output(self) -> list[np.ndarray]:
|
||||||
|
"""Build a 3-scale anchor-based YOLO output containing boxes A and B,
|
||||||
|
both decoded through anchor 0 of the stride-32 scale."""
|
||||||
|
outputs = [
|
||||||
|
np.zeros((1, 255, 80, 80), dtype=np.float32),
|
||||||
|
np.zeros((1, 255, 40, 40), dtype=np.float32),
|
||||||
|
np.zeros((1, 255, 20, 20), dtype=np.float32),
|
||||||
|
]
|
||||||
|
stride, (anchor_w, anchor_h) = 32, (142, 110)
|
||||||
|
|
||||||
|
for cx, cy, w, h, conf, class_channel in [
|
||||||
|
(A_CX, A_CY, A_W, A_H, 0.95, 5), # class 0
|
||||||
|
(B_CX, B_CY, B_W, B_H, 0.90, 6), # class 1
|
||||||
|
]:
|
||||||
|
cell_x, cell_y = int(cx // stride), int(cy // stride)
|
||||||
|
dx = (cx / stride - cell_x + 0.5) / 2
|
||||||
|
dy = (cy / stride - cell_y + 0.5) / 2
|
||||||
|
dw = math.sqrt(w / anchor_w) / 2
|
||||||
|
dh = math.sqrt(h / anchor_h) / 2
|
||||||
|
# anchor 0 occupies channels 0-84 of the 255 channel tensor
|
||||||
|
outputs[2][0, 0:4, cell_y, cell_x] = [dx, dy, dw, dh]
|
||||||
|
outputs[2][0, 4, cell_y, cell_x] = conf
|
||||||
|
outputs[2][0, class_channel, cell_y, cell_x] = 1.0
|
||||||
|
|
||||||
|
return outputs
|
||||||
|
|
||||||
|
def test_keeps_separated_objects_far_from_origin(self):
|
||||||
|
detections = kept(post_process_yolo(self._multipart_output(), WIDTH, HEIGHT))
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 2)
|
||||||
|
np.testing.assert_allclose(detections[0], [0, 0.95, *A_ROW], atol=2e-3)
|
||||||
|
np.testing.assert_allclose(detections[1], [1, 0.90, *B_ROW], atol=2e-3)
|
||||||
|
|
||||||
|
def test_empty_output_returns_no_detections(self):
|
||||||
|
outputs = [
|
||||||
|
np.zeros((1, 255, 80, 80), dtype=np.float32),
|
||||||
|
np.zeros((1, 255, 40, 40), dtype=np.float32),
|
||||||
|
np.zeros((1, 255, 20, 20), dtype=np.float32),
|
||||||
|
]
|
||||||
|
|
||||||
|
detections = kept(post_process_yolo(outputs, WIDTH, HEIGHT))
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestYoloxPostProcess(unittest.TestCase):
|
||||||
|
def test_keeps_separated_objects_far_from_origin(self):
|
||||||
|
# with zero grids and unit strides the decode reduces to
|
||||||
|
# cx = raw cx and w = exp(raw w)
|
||||||
|
rows = np.zeros((10, 7), dtype=np.float32)
|
||||||
|
rows[0] = [A_CX, A_CY, math.log(A_W), math.log(A_H), 1.0, 0.90, 0.0]
|
||||||
|
rows[1] = [B_CX, B_CY, math.log(B_W), math.log(B_H), 1.0, 0.0, 0.85]
|
||||||
|
predictions = rows[np.newaxis, ...]
|
||||||
|
grids = np.zeros((1, 10, 2), dtype=np.float32)
|
||||||
|
expanded_strides = np.ones((1, 10, 1), dtype=np.float32)
|
||||||
|
|
||||||
|
detections = kept(
|
||||||
|
post_process_yolox(predictions, WIDTH, HEIGHT, grids, expanded_strides)
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 2)
|
||||||
|
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||||
|
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
|
||||||
|
|
||||||
|
|
||||||
|
class TestDfinePostProcess(unittest.TestCase):
|
||||||
|
def test_keeps_separated_objects_far_from_origin(self):
|
||||||
|
# D-FINE emits absolute pixel xyxy boxes alongside labels and scores
|
||||||
|
labels = np.zeros((1, 10), dtype=np.int64)
|
||||||
|
labels[0, 1] = 1
|
||||||
|
boxes = np.zeros((1, 10, 4), dtype=np.float32)
|
||||||
|
boxes[0, 0] = [393, 499, 484, 620]
|
||||||
|
boxes[0, 1] = [527, 499, 618, 620]
|
||||||
|
scores = np.zeros((1, 10), dtype=np.float32)
|
||||||
|
scores[0, 0] = 0.90
|
||||||
|
scores[0, 1] = 0.85
|
||||||
|
|
||||||
|
detections = kept(post_process_dfine([labels, boxes, scores], WIDTH, HEIGHT))
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 2)
|
||||||
|
np.testing.assert_allclose(detections[0], [0, 0.90, *A_ROW], atol=2e-3)
|
||||||
|
np.testing.assert_allclose(detections[1], [1, 0.85, *B_ROW], atol=2e-3)
|
||||||
|
|
||||||
|
|
||||||
|
class TestRfdetrPostProcess(unittest.TestCase):
|
||||||
|
def test_keeps_separated_objects_far_from_origin(self):
|
||||||
|
# RF-DETR emits normalized center format boxes and class logits where
|
||||||
|
# logit index 0 is the background class
|
||||||
|
boxes = np.zeros((1, 10, 4), dtype=np.float32)
|
||||||
|
boxes[0, 0] = [A_CX / WIDTH, A_CY / HEIGHT, A_W / WIDTH, A_H / HEIGHT]
|
||||||
|
boxes[0, 1] = [B_CX / WIDTH, B_CY / HEIGHT, B_W / WIDTH, B_H / HEIGHT]
|
||||||
|
# background heavy logits everywhere, then two confident objects
|
||||||
|
logits = np.tile(np.array([10.0, 0.0, 0.0], dtype=np.float32), (1, 10, 1))
|
||||||
|
logits[0, 0] = [0.0, 4.0, 0.0] # class 0 after background offset
|
||||||
|
logits[0, 1] = [0.0, 0.0, 3.5] # class 1 after background offset
|
||||||
|
|
||||||
|
detections = kept(post_process_rfdetr([boxes, logits]))
|
||||||
|
|
||||||
|
conf_a = math.exp(4.0) / (math.exp(4.0) + 2)
|
||||||
|
conf_b = math.exp(3.5) / (math.exp(3.5) + 2)
|
||||||
|
self.assertEqual(len(detections), 2)
|
||||||
|
np.testing.assert_allclose(detections[0], [0, conf_a, *A_ROW], atol=2e-3)
|
||||||
|
np.testing.assert_allclose(detections[1], [1, conf_b, *B_ROW], atol=2e-3)
|
||||||
|
|
||||||
|
|
||||||
|
class TestMemryxSsdlitePostProcess(unittest.TestCase):
|
||||||
|
def test_keeps_separated_objects_far_from_origin(self):
|
||||||
|
# the NMS math runs on the host CPU, so the real method is testable
|
||||||
|
# without MemryX hardware; it only needs the model dimensions and
|
||||||
|
# the output queue
|
||||||
|
detector = object.__new__(MemryXDetector)
|
||||||
|
detector.memx_model_width = WIDTH
|
||||||
|
detector.memx_model_height = HEIGHT
|
||||||
|
detector.output_queue = Queue()
|
||||||
|
|
||||||
|
# this path uses a 0.5 NMS threshold, so use a tighter pair: zero
|
||||||
|
# true overlap (10 px gap), IoU 0.69 when misread as [x, y, w, h]
|
||||||
|
dets = np.zeros((1, 10, 5), dtype=np.float32)
|
||||||
|
dets[0, 0] = [480, 500, 540, 620, 0.90]
|
||||||
|
dets[0, 1] = [550, 500, 610, 620, 0.85]
|
||||||
|
labels = np.zeros((1, 10), dtype=np.float32)
|
||||||
|
labels[0, 1] = 1
|
||||||
|
|
||||||
|
detector.post_process_ssdlite([dets, labels])
|
||||||
|
detections = kept(detector.output_queue.get())
|
||||||
|
|
||||||
|
self.assertEqual(len(detections), 2)
|
||||||
|
np.testing.assert_allclose(
|
||||||
|
detections[0],
|
||||||
|
[0, 0.90, 500 / 640, 480 / 640, 620 / 640, 540 / 640],
|
||||||
|
atol=2e-3,
|
||||||
|
)
|
||||||
|
np.testing.assert_allclose(
|
||||||
|
detections[1],
|
||||||
|
[1, 0.85, 500 / 640, 550 / 640, 620 / 640, 610 / 640],
|
||||||
|
atol=2e-3,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
+37
-5
@@ -16,6 +16,31 @@ logger = logging.getLogger(__name__)
|
|||||||
### Post Processing
|
### Post Processing
|
||||||
|
|
||||||
|
|
||||||
|
def xyxy_to_xywh_for_nms(boxes: np.ndarray | list) -> np.ndarray:
|
||||||
|
"""Convert [x1, y1, x2, y2] boxes to the [x, y, width, height] format
|
||||||
|
that cv2.dnn.NMSBoxes expects.
|
||||||
|
|
||||||
|
Passing corner coordinates directly makes OpenCV treat x2/y2 as the box
|
||||||
|
size, inflating every box toward the bottom-right by its distance from
|
||||||
|
the origin, which suppresses valid detections near other objects.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
boxes: Array-like of shape (N, 4) in corner format.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Float32 array of shape (N, 4) in top-left plus size format.
|
||||||
|
"""
|
||||||
|
boxes = np.asarray(boxes, dtype=np.float32)
|
||||||
|
|
||||||
|
if boxes.size == 0:
|
||||||
|
return np.zeros((0, 4), dtype=np.float32)
|
||||||
|
|
||||||
|
xywh = boxes.copy()
|
||||||
|
xywh[:, 2] -= xywh[:, 0]
|
||||||
|
xywh[:, 3] -= xywh[:, 1]
|
||||||
|
return xywh
|
||||||
|
|
||||||
|
|
||||||
def post_process_dfine(
|
def post_process_dfine(
|
||||||
tensor_output: np.ndarray, width: int, height: int
|
tensor_output: np.ndarray, width: int, height: int
|
||||||
) -> np.ndarray:
|
) -> np.ndarray:
|
||||||
@@ -25,7 +50,9 @@ def post_process_dfine(
|
|||||||
|
|
||||||
input_shape = np.array([height, width, height, width])
|
input_shape = np.array([height, width, height, width])
|
||||||
boxes = np.divide(boxes, input_shape, dtype=np.float32)
|
boxes = np.divide(boxes, input_shape, dtype=np.float32)
|
||||||
indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.4, nms_threshold=0.4)
|
indices = cv2.dnn.NMSBoxes(
|
||||||
|
xyxy_to_xywh_for_nms(boxes), scores, score_threshold=0.4, nms_threshold=0.4
|
||||||
|
)
|
||||||
detections = np.zeros((20, 6), np.float32)
|
detections = np.zeros((20, 6), np.float32)
|
||||||
|
|
||||||
for i, (bbox, confidence, class_id) in enumerate(
|
for i, (bbox, confidence, class_id) in enumerate(
|
||||||
@@ -78,7 +105,10 @@ def post_process_rfdetr(tensor_output: list[np.ndarray, np.ndarray]) -> np.ndarr
|
|||||||
|
|
||||||
# apply nms
|
# apply nms
|
||||||
indices = cv2.dnn.NMSBoxes(
|
indices = cv2.dnn.NMSBoxes(
|
||||||
filtered_boxes, filtered_scores, score_threshold=0.4, nms_threshold=0.4
|
xyxy_to_xywh_for_nms(filtered_boxes),
|
||||||
|
filtered_scores,
|
||||||
|
score_threshold=0.4,
|
||||||
|
nms_threshold=0.4,
|
||||||
)
|
)
|
||||||
detections = np.zeros((20, 6), np.float32)
|
detections = np.zeros((20, 6), np.float32)
|
||||||
|
|
||||||
@@ -159,7 +189,7 @@ def __post_process_multipart_yolo(
|
|||||||
all_class_ids.append(class_id)
|
all_class_ids.append(class_id)
|
||||||
|
|
||||||
indices = cv2.dnn.NMSBoxes(
|
indices = cv2.dnn.NMSBoxes(
|
||||||
bboxes=all_boxes,
|
bboxes=xyxy_to_xywh_for_nms(all_boxes),
|
||||||
scores=all_scores,
|
scores=all_scores,
|
||||||
score_threshold=0.4,
|
score_threshold=0.4,
|
||||||
nms_threshold=0.4,
|
nms_threshold=0.4,
|
||||||
@@ -206,7 +236,9 @@ def __post_process_nms_yolo(predictions: np.ndarray, width, height) -> np.ndarra
|
|||||||
boxes = boxes_xyxy
|
boxes = boxes_xyxy
|
||||||
|
|
||||||
# run NMS
|
# run NMS
|
||||||
indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.4, nms_threshold=0.4)
|
indices = cv2.dnn.NMSBoxes(
|
||||||
|
xyxy_to_xywh_for_nms(boxes), scores, score_threshold=0.4, nms_threshold=0.4
|
||||||
|
)
|
||||||
detections = np.zeros((20, 6), np.float32)
|
detections = np.zeros((20, 6), np.float32)
|
||||||
for i, (bbox, confidence, class_id) in enumerate(
|
for i, (bbox, confidence, class_id) in enumerate(
|
||||||
zip(boxes[indices], scores[indices], class_ids[indices])
|
zip(boxes[indices], scores[indices], class_ids[indices])
|
||||||
@@ -258,7 +290,7 @@ def post_process_yolox(
|
|||||||
scores = scores[np.arange(len(cls_inds)), cls_inds]
|
scores = scores[np.arange(len(cls_inds)), cls_inds]
|
||||||
|
|
||||||
indices = cv2.dnn.NMSBoxes(
|
indices = cv2.dnn.NMSBoxes(
|
||||||
boxes_xyxy, scores, score_threshold=0.4, nms_threshold=0.4
|
xyxy_to_xywh_for_nms(boxes_xyxy), scores, score_threshold=0.4, nms_threshold=0.4
|
||||||
)
|
)
|
||||||
|
|
||||||
detections = np.zeros((20, 6), np.float32)
|
detections = np.zeros((20, 6), np.float32)
|
||||||
|
|||||||
@@ -859,8 +859,8 @@
|
|||||||
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
|
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
|
||||||
},
|
},
|
||||||
"dashboard": {
|
"dashboard": {
|
||||||
"label": "Show in UI",
|
"label": "Show on Live dashboard",
|
||||||
"description": "Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again."
|
"description": "Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings."
|
||||||
},
|
},
|
||||||
"review": {
|
"review": {
|
||||||
"label": "Show in review",
|
"label": "Show in review",
|
||||||
|
|||||||
@@ -357,22 +357,6 @@
|
|||||||
"description": "Optional API key for authenticated DeepStack services."
|
"description": "Optional API key for authenticated DeepStack services."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"degirum": {
|
|
||||||
"label": "DeGirum",
|
|
||||||
"description": "DeGirum detector for running models via DeGirum cloud or local inference services.",
|
|
||||||
"location": {
|
|
||||||
"label": "Inference Location",
|
|
||||||
"description": "Location of the DeGirim inference engine (e.g. '@cloud', '127.0.0.1')."
|
|
||||||
},
|
|
||||||
"zoo": {
|
|
||||||
"label": "Model Zoo",
|
|
||||||
"description": "Path or URL to the DeGirum model zoo."
|
|
||||||
},
|
|
||||||
"token": {
|
|
||||||
"label": "DeGirum Cloud Token",
|
|
||||||
"description": "Token for DeGirum Cloud access."
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"edgetpu": {
|
"edgetpu": {
|
||||||
"label": "EdgeTPU",
|
"label": "EdgeTPU",
|
||||||
"description": "EdgeTPU detector that runs TensorFlow Lite models compiled for Coral EdgeTPU using the EdgeTPU delegate.",
|
"description": "EdgeTPU detector that runs TensorFlow Lite models compiled for Coral EdgeTPU using the EdgeTPU delegate.",
|
||||||
@@ -1543,8 +1527,8 @@
|
|||||||
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
|
"description": "Numeric order used to sort the camera in the UI (default dashboard and lists); larger numbers appear later."
|
||||||
},
|
},
|
||||||
"dashboard": {
|
"dashboard": {
|
||||||
"label": "Show in UI",
|
"label": "Show on Live dashboard",
|
||||||
"description": "Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again."
|
"description": "Toggle whether this camera is visible on the default All Cameras live dashboard. The camera remains available everywhere else in the UI, including camera groups and settings."
|
||||||
},
|
},
|
||||||
"review": {
|
"review": {
|
||||||
"label": "Show in review",
|
"label": "Show in review",
|
||||||
|
|||||||
@@ -499,7 +499,7 @@
|
|||||||
"webuiUrlHelp": "URL to visit the camera's web UI directly from the Debug view. Leave blank to disable the link.",
|
"webuiUrlHelp": "URL to visit the camera's web UI directly from the Debug view. Leave blank to disable the link.",
|
||||||
"webuiUrlInvalid": "Must be a valid URL (e.g., https://example.com).",
|
"webuiUrlInvalid": "Must be a valid URL (e.g., https://example.com).",
|
||||||
"dashboardLabel": "Show on Live dashboard",
|
"dashboardLabel": "Show on Live dashboard",
|
||||||
"dashboardHelp": "Show this camera on the Live dashboard.",
|
"dashboardHelp": "Show this camera on the default All Cameras live dashboard. It remains available everywhere else, including camera groups.",
|
||||||
"reviewLabel": "Show in Review",
|
"reviewLabel": "Show in Review",
|
||||||
"reviewHelp": "Show this camera in Review, including the camera filter, motion review, and the history view."
|
"reviewHelp": "Show this camera in Review, including the camera filter, motion review, and the history view."
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -78,7 +78,7 @@ export default function ZoneEditPane({
|
|||||||
}
|
}
|
||||||
|
|
||||||
return Object.values(config.cameras)
|
return Object.values(config.cameras)
|
||||||
.filter((conf) => conf.ui.dashboard && conf.enabled_in_config)
|
.filter((conf) => conf.enabled_in_config)
|
||||||
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
|
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
|
||||||
}, [config]);
|
}, [config]);
|
||||||
|
|
||||||
|
|||||||
+42
-23
@@ -198,6 +198,19 @@ export default function Events() {
|
|||||||
return true;
|
return true;
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const reviewCamerasParam = useMemo(() => {
|
||||||
|
const selected: string | undefined = reviewSearchParams["cameras"];
|
||||||
|
|
||||||
|
if (!selected) {
|
||||||
|
return reviewCameras.join(",");
|
||||||
|
}
|
||||||
|
|
||||||
|
const selectedCameras = new Set(selected.split(","));
|
||||||
|
return reviewCameras
|
||||||
|
.filter((camera) => selectedCameras.has(camera))
|
||||||
|
.join(",");
|
||||||
|
}, [reviewCameras, reviewSearchParams]);
|
||||||
|
|
||||||
useSearchEffect("labels", (labels: string) => {
|
useSearchEffect("labels", (labels: string) => {
|
||||||
setReviewFilter({
|
setReviewFilter({
|
||||||
...reviewFilter,
|
...reviewFilter,
|
||||||
@@ -330,8 +343,12 @@ export default function Events() {
|
|||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
const getKey = useCallback(() => {
|
const getKey = useCallback(() => {
|
||||||
|
if (!timezone) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
const params = {
|
const params = {
|
||||||
cameras: reviewSearchParams["cameras"],
|
cameras: reviewCamerasParam,
|
||||||
labels: reviewSearchParams["labels"],
|
labels: reviewSearchParams["labels"],
|
||||||
zones: reviewSearchParams["zones"],
|
zones: reviewSearchParams["zones"],
|
||||||
reviewed: null, // We want both reviewed and unreviewed items as we filter in the UI
|
reviewed: null, // We want both reviewed and unreviewed items as we filter in the UI
|
||||||
@@ -339,7 +356,7 @@ export default function Events() {
|
|||||||
after: reviewSearchParams["after"] || last24Hours.after,
|
after: reviewSearchParams["after"] || last24Hours.after,
|
||||||
};
|
};
|
||||||
return ["review", params];
|
return ["review", params];
|
||||||
}, [reviewSearchParams, last24Hours]);
|
}, [reviewSearchParams, reviewCamerasParam, last24Hours, timezone]);
|
||||||
|
|
||||||
const { data: reviews, mutate: updateSegments } = useSWR<ReviewSegment[]>(
|
const { data: reviews, mutate: updateSegments } = useSWR<ReviewSegment[]>(
|
||||||
getKey,
|
getKey,
|
||||||
@@ -361,10 +378,6 @@ export default function Events() {
|
|||||||
const motion: ReviewSegment[] = [];
|
const motion: ReviewSegment[] = [];
|
||||||
|
|
||||||
reviews?.forEach((segment) => {
|
reviews?.forEach((segment) => {
|
||||||
if (config?.cameras[segment.camera]?.ui?.review === false) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
all.push(segment);
|
all.push(segment);
|
||||||
|
|
||||||
switch (segment.severity) {
|
switch (segment.severity) {
|
||||||
@@ -386,7 +399,7 @@ export default function Events() {
|
|||||||
detection: detections,
|
detection: detections,
|
||||||
significant_motion: motion,
|
significant_motion: motion,
|
||||||
};
|
};
|
||||||
}, [reviews, config?.cameras]);
|
}, [reviews]);
|
||||||
|
|
||||||
// update review items in place when a review segment ends
|
// update review items in place when a review segment ends
|
||||||
const reviewUpdate = useFrigateReviews();
|
const reviewUpdate = useFrigateReviews();
|
||||||
@@ -450,15 +463,17 @@ export default function Events() {
|
|||||||
// review summary
|
// review summary
|
||||||
|
|
||||||
const { data: reviewSummary, mutate: updateSummary } = useSWR<ReviewSummary>(
|
const { data: reviewSummary, mutate: updateSummary } = useSWR<ReviewSummary>(
|
||||||
[
|
timezone
|
||||||
"review/summary",
|
? [
|
||||||
{
|
"review/summary",
|
||||||
timezone: timezone,
|
{
|
||||||
cameras: reviewSearchParams["cameras"] ?? null,
|
timezone: timezone,
|
||||||
labels: reviewSearchParams["labels"] ?? null,
|
cameras: reviewCamerasParam,
|
||||||
zones: reviewSearchParams["zones"] ?? null,
|
labels: reviewSearchParams["labels"] ?? null,
|
||||||
},
|
zones: reviewSearchParams["zones"] ?? null,
|
||||||
],
|
},
|
||||||
|
]
|
||||||
|
: null,
|
||||||
{
|
{
|
||||||
revalidateOnFocus: true,
|
revalidateOnFocus: true,
|
||||||
refreshInterval: 30000,
|
refreshInterval: 30000,
|
||||||
@@ -473,13 +488,17 @@ export default function Events() {
|
|||||||
|
|
||||||
// recordings summary
|
// recordings summary
|
||||||
|
|
||||||
const { data: recordingsSummary } = useSWR<RecordingsSummary>([
|
const { data: recordingsSummary } = useSWR<RecordingsSummary>(
|
||||||
"recordings/summary",
|
timezone
|
||||||
{
|
? [
|
||||||
timezone: timezone,
|
"recordings/summary",
|
||||||
cameras: reviewSearchParams["cameras"] ?? null,
|
{
|
||||||
},
|
timezone: timezone,
|
||||||
]);
|
cameras: reviewCamerasParam,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
: null,
|
||||||
|
);
|
||||||
|
|
||||||
// preview videos
|
// preview videos
|
||||||
const previewTimes = useMemo(() => {
|
const previewTimes = useMemo(() => {
|
||||||
|
|||||||
@@ -706,12 +706,7 @@ export default function Settings() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return Object.values(config.cameras)
|
return Object.values(config.cameras)
|
||||||
.filter(
|
.filter((conf) => conf.enabled_in_config && !isReplayCamera(conf.name))
|
||||||
(conf) =>
|
|
||||||
conf.ui.dashboard &&
|
|
||||||
conf.enabled_in_config &&
|
|
||||||
!isReplayCamera(conf.name),
|
|
||||||
)
|
|
||||||
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
|
.sort((aConf, bConf) => aConf.ui.order - bConf.ui.order);
|
||||||
}, [config]);
|
}, [config]);
|
||||||
|
|
||||||
|
|||||||
@@ -1024,6 +1024,9 @@ function MotionReview({
|
|||||||
if (!allowedCameras.includes(cam.name)) {
|
if (!allowedCameras.includes(cam.name)) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
if (cam.ui?.review === false) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
if (selectedCams && !selectedCams.includes(cam.name)) {
|
if (selectedCams && !selectedCams.includes(cam.name)) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -1033,6 +1036,11 @@ function MotionReview({
|
|||||||
return cameras.sort((a, b) => a.ui.order - b.ui.order);
|
return cameras.sort((a, b) => a.ui.order - b.ui.order);
|
||||||
}, [config, filter, allowedCameras]);
|
}, [config, filter, allowedCameras]);
|
||||||
|
|
||||||
|
const reviewCamerasParam = useMemo(
|
||||||
|
() => reviewCameras.map((cam) => cam.name).join(","),
|
||||||
|
[reviewCameras],
|
||||||
|
);
|
||||||
|
|
||||||
const videoPlayersRef = useRef<{ [camera: string]: PreviewController }>({});
|
const videoPlayersRef = useRef<{ [camera: string]: PreviewController }>({});
|
||||||
|
|
||||||
// motion data
|
// motion data
|
||||||
@@ -1052,7 +1060,7 @@ function MotionReview({
|
|||||||
before: alignedBefore,
|
before: alignedBefore,
|
||||||
after: alignedAfter,
|
after: alignedAfter,
|
||||||
scale: segmentDuration / 2,
|
scale: segmentDuration / 2,
|
||||||
cameras: filter?.cameras?.join(",") ?? null,
|
cameras: reviewCamerasParam,
|
||||||
},
|
},
|
||||||
]);
|
]);
|
||||||
|
|
||||||
@@ -1061,7 +1069,7 @@ function MotionReview({
|
|||||||
{
|
{
|
||||||
before: alignedBefore,
|
before: alignedBefore,
|
||||||
after: alignedAfter,
|
after: alignedAfter,
|
||||||
cameras: filter?.cameras?.join(",") ?? null,
|
cameras: reviewCamerasParam,
|
||||||
},
|
},
|
||||||
]);
|
]);
|
||||||
|
|
||||||
|
|||||||
@@ -540,6 +540,36 @@ export default function GeneralMetrics({
|
|||||||
return Object.keys(series).length > 0 ? Object.values(series) : undefined;
|
return Object.keys(series).length > 0 ? Object.values(series) : undefined;
|
||||||
}, [statsHistory]);
|
}, [statsHistory]);
|
||||||
|
|
||||||
|
// Number of cards the hardware grid renders. Which ones appear depends on
|
||||||
|
// the vendor, so the column count follows the count rather than assuming a
|
||||||
|
// fixed set is present.
|
||||||
|
const hardwareCardCount = useMemo(() => {
|
||||||
|
if (!statsHistory[0]?.gpu_usages) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const hasNpu = statsHistory[0].npu_usages != undefined;
|
||||||
|
|
||||||
|
return (
|
||||||
|
1 + // gpu usage always renders alongside gpu_usages
|
||||||
|
(gpuMemSeries ? 1 : 0) +
|
||||||
|
(gpuEncSeries?.length ? 1 : 0) +
|
||||||
|
(gpuComputeSeries?.length ? 1 : 0) +
|
||||||
|
(gpuDecSeries?.length ? 1 : 0) +
|
||||||
|
(gpuTempSeries?.length ? 1 : 0) +
|
||||||
|
(hasNpu ? 1 : 0) +
|
||||||
|
(hasNpu && npuTempSeries?.length ? 1 : 0)
|
||||||
|
);
|
||||||
|
}, [
|
||||||
|
statsHistory,
|
||||||
|
gpuMemSeries,
|
||||||
|
gpuEncSeries,
|
||||||
|
gpuComputeSeries,
|
||||||
|
gpuDecSeries,
|
||||||
|
gpuTempSeries,
|
||||||
|
npuTempSeries,
|
||||||
|
]);
|
||||||
|
|
||||||
// other processes stats
|
// other processes stats
|
||||||
|
|
||||||
const hardwareType = useMemo(() => {
|
const hardwareType = useMemo(() => {
|
||||||
@@ -763,12 +793,9 @@ export default function GeneralMetrics({
|
|||||||
<div
|
<div
|
||||||
className={cn(
|
className={cn(
|
||||||
"mt-4 grid grid-cols-1 gap-2 sm:grid-cols-2",
|
"mt-4 grid grid-cols-1 gap-2 sm:grid-cols-2",
|
||||||
gpuTempSeries?.length && "md:grid-cols-3",
|
hardwareCardCount >= 3 && "lg:grid-cols-3",
|
||||||
(gpuEncSeries?.length || gpuComputeSeries?.length) &&
|
hardwareCardCount >= 4 && "xl:grid-cols-4",
|
||||||
"xl:grid-cols-4",
|
hardwareCardCount >= 5 && "3xl:grid-cols-5",
|
||||||
(gpuEncSeries?.length || gpuComputeSeries?.length) &&
|
|
||||||
gpuTempSeries?.length &&
|
|
||||||
"3xl:grid-cols-5",
|
|
||||||
)}
|
)}
|
||||||
>
|
>
|
||||||
{statsHistory[0]?.gpu_usages && (
|
{statsHistory[0]?.gpu_usages && (
|
||||||
|
|||||||
Reference in New Issue
Block a user