administrator/frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2026-06-25 13:51:52 +03:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Merge 87f55be805 into efe585a920

Browse Source
This commit is contained in:
knoffelcut 2026-06-11 14:43:32 +02:00 committed by GitHub
commit 1eed000c4d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 254 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

112
docs/docs/configuration/object_detectors.md
View File

@ -495,6 +495,7 @@ detectors:
| [MobileNet v2](#ssdlite-mobilenet-v2) | ✅ | ✅ | Fast and lightweight model, less accurate than larger models |
| [YOLOX](#yolox) | ✅ | ? | |
| [D-FINE / DEIMv2](#d-fine--deimv2) | ❌ | ❌ | |
| [NanoDet-Plus](#nanodet-plus) | ? | ? | |
#### SSDLite MobileNet v2
@ -791,6 +792,44 @@ Note that the labelmap uses a subset of the complete COCO label set that has onl
</details>
#### NanoDet-Plus
[NanoDet-Plus](https://github.com/RangiLyu/nanodet) is a lightweight object detection model that achieves
good accuracy on CPUs given its small footprint.
Script to export an ONNX model for use in Frigate is provided in [the models section](#downloading-nanodet-plus-models).
:::warning
NanoDet-Plus has not been tested in GPU nor NPU modes.
:::
<details>
<summary>NanoDet-Plus Setup & Config</summary>
After placing the exported onnx model in your config/model_cache folder, you can use the following configuration:
```yaml
detectors:
ov:
type: openvino
device: CPU
model:
model_type: nanodet_plus
width: 320
height: 320
input_tensor: nchw
input_dtype: float
input_pixel_format: bgr
path: /config/model_cache/nanodet_plus.onnx
labelmap_path: /labelmap/coco-80.txt
```
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
</details>
## Apple Silicon detector
The NPU in Apple Silicon can't be accessed from within a container, so the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) must first be setup. It is recommended to use the Frigate docker image with `-standard-arm64` suffix, for example `ghcr.io/blakeblackshear/frigate:stable-standard-arm64`.
@ -1029,6 +1068,7 @@ detectors:
| [YOLO-NAS](#yolo-nas-1) | ⚠️ | ⚠️ | Not supported by CUDA Graphs |
| [YOLOX](#yolox-1) | ✅ | ✅ | Supports CUDA Graphs for optimal Nvidia performance |
| [D-FINE / DEIMv2](#d-fine--deimv2-1) | ⚠️ | ❌ | Not supported by CUDA Graphs |
| [NanoDet-Plus](#nanodet-plus-1) | ✅ | ? | Supports CUDA Graphs for optimal Nvidia performance |
There is no default model provided, the following formats are supported:
@ -1311,6 +1351,42 @@ model:
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
#### NanoDet-Plus
[NanoDet-Plus](https://github.com/RangiLyu/nanodet) is a lightweight object detection model that achieves
good accuracy on CPUs given its small footprint.
Script to export an ONNX model for use in Frigate is provided in [the models section](#downloading-nanodet-plus-models).
:::warning
NanoDet-Plus has not been tested on AMD GPUs.
:::
<details>
<summary>NanoDet-Plus Setup & Config</summary>
After placing the exported onnx model in your config/model_cache folder, you can use the following configuration:
```yaml
detectors:
onnx:
type: onnx
model:
model_type: nanodet_plus
width: 320
height: 320
input_tensor: nchw
input_dtype: float
input_pixel_format: bgr
path: /config/model_cache/nanodet_plus.onnx
labelmap_path: /labelmap/coco-80.txt
```
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
</details>
## CPU Detector (not recommended)
The CPU detector type runs a TensorFlow Lite model utilizing the CPU without hardware acceleration. It is recommended to use a hardware accelerated detector type instead for better performance. To configure a CPU based detector, set the `"type"` attribute to `"cpu"`.
@ -2460,3 +2536,39 @@ ARG IMG_SIZE
COPY --from=build /yolov9/yolov9-${MODEL_SIZE}.onnx /yolov9-${MODEL_SIZE}-${IMG_SIZE}.onnx
EOF
```
### Downloading NanoDet-Plus models
NanoDet-Plus can be downloaded using the command below. Copy and paste the complete command to your terminal to export
the model as `nanodet_plus.onnx` in the current working directory. The command builds the NanoDet-Plus environment,
downloads the specified model and converts it to ONNX.
The below command is configured to use the smallest model provided by the authors, NanoDet-Plus-m-320. Other models
can be specified by changing the `URL_WEIGHTS` link to the appropriate pretrained weights URL from
[NanoDet-Plus Model Zoo](https://github.com/RangiLyu/nanodet#model-zoo). Remember to change the `IMG_HEIGHT`,
`IMG_WIDTH` and `CFG_PATH` ([configuration files](https://github.com/RangiLyu/nanodet/tree/main/config)) parameters
accordingly.
Compatible with the `labelmap/coco-80.txt` labelmap
```sh
docker build . --build-arg URL_WEIGHTS=https://drive.google.com/file/d/1Dq0cTIdJDUhQxJe45z6rWncbZmOyh1Tv/view?usp=sharing --build-arg IMG_HEIGHT=320 --build-arg IMG_WIDTH=320 --build-arg CFG_PATH=config/nanodet-plus-m_320.yml --output . -f- <<'EOF'
FROM python:3.9 AS build
RUN apt-get update && apt-get install --no-install-recommends -y cmake libgl1 && rm -rf /var/lib/apt/lists/*
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
WORKDIR /nanodet_plus
ADD https://github.com/RangiLyu/nanodet.git .
RUN uv pip install --system onnx==1.18.0 onnxruntime onnx-simplifier==0.4.* onnxscript
RUN uv pip install --system "numpy<2"
RUN uv pip install --system -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
RUN uv pip install --system -e .
ARG URL_WEIGHTS
RUN uv pip install --system gdown
RUN gdown --fuzzy ${URL_WEIGHTS} -O nanodet_plus.pth
ARG IMG_HEIGHT
ARG IMG_WIDTH
ARG CFG_PATH
RUN python tools/export_onnx.py --cfg_path=${CFG_PATH} --model_path=nanodet_plus.pth --input_shape=${IMG_HEIGHT},${IMG_WIDTH} --out_path=nanodet_plus.onnx
FROM scratch
COPY --from=build /nanodet_plus/nanodet_plus.onnx nanodet_plus.onnx
EOF
```

1
frigate/detectors/detector_config.py
View File

@ -42,6 +42,7 @@ class ModelTypeEnum(str, Enum):
yolox = "yolox"
yolonas = "yolonas"
yologeneric = "yolo-generic"
nanodet_plus = "nanodet_plus"
class ModelConfig(BaseModel):

7
frigate/detectors/plugins/onnx.py
View File

@ -14,6 +14,7 @@ from frigate.detectors.detector_config import (
)
from frigate.util.model import (
post_process_dfine,
post_process_nanodet_plus,
post_process_rfdetr,
post_process_yolo,
post_process_yolox,
@ -137,6 +138,12 @@ class ONNXDetector(DetectionApi):
self.grids,
self.expanded_strides,
)
elif self.onnx_model_type == ModelTypeEnum.nanodet_plus:
return post_process_nanodet_plus(
tensor_output[0],
self.width,
self.height,
)
else:
raise Exception(
f"{self.onnx_model_type} is currently not supported for onnx. See the docs for more info on supported models."

8
frigate/detectors/plugins/openvino.py
View File

@ -10,6 +10,7 @@ from frigate.detectors.detection_runners import OpenVINOModelRunner
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
from frigate.util.model import (
post_process_dfine,
post_process_nanodet_plus,
post_process_rfdetr,
post_process_yolo,
)
@ -43,6 +44,7 @@ class OvDetector(DetectionApi):
ModelTypeEnum.yolonas,
ModelTypeEnum.yologeneric,
ModelTypeEnum.yolox,
ModelTypeEnum.nanodet_plus,
]
def __init__(self, detector_config: OvDetectorConfig):
@ -238,3 +240,9 @@ class OvDetector(DetectionApi):
object_detected[6], object_detected[5], object_detected[:4]
)
return detections
elif self.ov_model_type == ModelTypeEnum.nanodet_plus:
return post_process_nanodet_plus(
outputs[0],
self.width,
self.height,
)

126
frigate/util/model.py
View File

@ -7,6 +7,7 @@ from typing import Any
import cv2
import numpy as np
import onnxruntime as ort
import scipy.special
from frigate.const import MODEL_CACHE_DIR
@ -16,6 +17,57 @@ logger = logging.getLogger(__name__)
### Post Processing
def calculate_nanodet_center_priors(
input_height: int, input_width: int, strides: tuple, dtype: type
):
"""
Adapted from https://github.com/RangiLyu/nanodet/blob/be9b4a9001d7f9b6fc89c2df31ae8d428e35b4f0/nanodet/model/head/nanodet_plus_head.py
"""
def get_single_level_center_priors(featmap_size, stride, dtype):
"""Generate centers of a single stage feature map.
Args:
batch_size (int): Number of images in one batch.
featmap_size (tuple[int]): height and width of the feature map
stride (int): down sample stride of the feature map
dtype (obj:`torch.dtype`): data type of the tensors
device (obj:`torch.device`): device of the tensors
Return:
priors (Tensor): center priors of a single level feature map.
"""
h, w = featmap_size
x_range = (np.arange(w, dtype=dtype)) * stride
y_range = (np.arange(h, dtype=dtype)) * stride
y, x = np.meshgrid(y_range, x_range, indexing="ij")
y = y.flatten()
x = x.flatten()
strides = np.full(x.shape[0], stride)
priors = np.stack([x, y, strides, strides], axis=-1)
return priors
featmap_sizes = [
(
int(np.ceil(input_height / stride)),
int(np.ceil(input_width) / stride),
)
for stride in strides
]
mlvl_center_priors = [
get_single_level_center_priors(
featmap_sizes[i],
stride,
dtype,
)
for i, stride in enumerate(strides)
]
center_priors = np.concatenate(mlvl_center_priors, axis=0)
return center_priors
nanodet_center_priors: dict[(int, int, tuple, type), np.ndarray] = {}
def post_process_dfine(
tensor_output: np.ndarray, width: int, height: int
) -> np.ndarray:
@ -280,6 +332,80 @@ def post_process_yolox(
return detections
def post_process_nanodet_plus(
predictions: np.ndarray,
width: int,
height: int,
):
"""
Adapted from https://github.com/RangiLyu/nanodet/blob/be9b4a9001d7f9b6fc89c2df31ae8d428e35b4f0/nanodet/model/head/nanodet_plus_head.py
"""
def distance2bbox(points, distance, max_shape=None):
"""Decode distance prediction to bounding box.
Args:
points (Tensor): Shape (n, 2), [x, y].
distance (Tensor): Distance from the given point to 4
boundaries (left, top, right, bottom).
max_shape (tuple): Shape of the image.
Returns:
Tensor: Decoded bboxes.
"""
x1 = points[..., 0] - distance[..., 0]
y1 = points[..., 1] - distance[..., 1]
x2 = points[..., 0] + distance[..., 2]
y2 = points[..., 1] + distance[..., 3]
if max_shape is not None:
x1 = np.clip(x1, 0, max_shape[1])
y1 = np.clip(y1, 0, max_shape[0])
x2 = np.clip(x2, 0, max_shape[1])
y2 = np.clip(y2, 0, max_shape[0])
return np.stack([x1, y1, x2, y2], -1)
predictions = predictions[0]
# Below two parameters are consistent with all nanodet **plus** models
reg_max = 7
strides = (8, 16, 32, 64)
num_classes = predictions.shape[-1] - 4 * (reg_max + 1)
cls_scores, bbox_preds = predictions[:, :num_classes], predictions[:, num_classes:]
try:
center_priors = nanodet_center_priors[
(height, width, strides, predictions[0].dtype)
]
except KeyError:
center_priors = calculate_nanodet_center_priors(
height, width, strides, predictions[0].dtype
)
nanodet_center_priors[(height, width, strides, predictions[0].dtype)] = (
center_priors
)
x = bbox_preds.reshape(bbox_preds.shape[0], 4, reg_max + 1)
x = scipy.special.softmax(x, axis=-1)
x = np.dot(x, np.linspace(0, reg_max, reg_max + 1))
dis_preds = x * center_priors[..., 2, None]
bboxes = distance2bbox(center_priors[..., :2], dis_preds, max_shape=(height, width))
class_ids = np.argmax(cls_scores, axis=1)
scores = np.max(cls_scores, axis=1)
detections = np.zeros((20, 6), dtype=np.float32)
for i, j in enumerate(np.argsort(scores)[::-1][:20]):
detections[i, 0] = class_ids[j]
detections[i, 1] = scores[j]
detections[i, 2] = bboxes[j, 1] / height
detections[i, 3] = bboxes[j, 0] / width
detections[i, 4] = bboxes[j, 3] / height
detections[i, 5] = bboxes[j, 2] / width
return detections
### ONNX Utilities