mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-08-01 08:32:18 +03:00
Merge pull request #19787 from blakeblackshear/dev
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
CI / AMD64 Extra Build (push) Blocked by required conditions
0.17 Release
This commit is contained in:
@@ -3,6 +3,8 @@ id: object_detectors
|
||||
title: Object Detectors
|
||||
---
|
||||
|
||||
import CommunityBadge from '@site/src/components/CommunityBadge';
|
||||
|
||||
# Supported Hardware
|
||||
|
||||
:::info
|
||||
@@ -11,14 +13,20 @@ Frigate supports multiple different detectors that work on different types of ha
|
||||
|
||||
**Most Hardware**
|
||||
|
||||
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB and m.2 format allowing for a wide range of compatibility with devices.
|
||||
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
|
||||
- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
|
||||
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
|
||||
- <CommunityBadge /> [DeGirum](#degirum): Service for using hardware devices in the cloud or locally. Hardware and models provided on the cloud on [their website](https://hub.degirum.com).
|
||||
|
||||
**AMD**
|
||||
|
||||
- [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection.
|
||||
- [ONNX](#onnx): ROCm will automatically be detected and used as a detector in the `-rocm` Frigate image when a supported ONNX model is configured.
|
||||
|
||||
**Apple Silicon**
|
||||
|
||||
- [Apple Silicon](#apple-silicon-detector): Apple Silicon can run on M1 and newer Apple Silicon devices.
|
||||
|
||||
**Intel**
|
||||
|
||||
- [OpenVino](#openvino-detector): OpenVino can run on Intel Arc GPUs, Intel integrated GPUs, and Intel CPUs to provide efficient object detection.
|
||||
@@ -26,17 +34,21 @@ Frigate supports multiple different detectors that work on different types of ha
|
||||
|
||||
**Nvidia GPU**
|
||||
|
||||
- [ONNX](#onnx): TensorRT will automatically be detected and used as a detector in the `-tensorrt` Frigate image when a supported ONNX model is configured.
|
||||
- [ONNX](#onnx): Nvidia GPUs will automatically be detected and used as a detector in the `-tensorrt` Frigate image when a supported ONNX model is configured.
|
||||
|
||||
**Nvidia Jetson**
|
||||
**Nvidia Jetson** <CommunityBadge />
|
||||
|
||||
- [TensortRT](#nvidia-tensorrt-detector): TensorRT can run on Jetson devices, using one of many default models.
|
||||
- [ONNX](#onnx): TensorRT will automatically be detected and used as a detector in the `-tensorrt-jp6` Frigate image when a supported ONNX model is configured.
|
||||
|
||||
**Rockchip**
|
||||
**Rockchip** <CommunityBadge />
|
||||
|
||||
- [RKNN](#rockchip-platform): RKNN models can run on Rockchip devices with included NPUs.
|
||||
|
||||
**Synaptics** <CommunityBadge />
|
||||
|
||||
- [Synaptics](#synaptics): synap models can run on Synaptics devices(e.g astra machina) with included NPUs.
|
||||
|
||||
**For Testing**
|
||||
|
||||
- [CPU Detector (not recommended for actual use](#cpu-detector-not-recommended): Use a CPU to run tflite model, this is not recommended and in most cases OpenVINO can be used in CPU mode with better results.
|
||||
@@ -53,16 +65,14 @@ This does not affect using hardware for accelerating other tasks such as [semant
|
||||
|
||||
# Officially Supported Detectors
|
||||
|
||||
Frigate provides the following builtin detector types: `cpu`, `edgetpu`, `hailo8l`, `onnx`, `openvino`, `rknn`, and `tensorrt`. By default, Frigate will use a single CPU detector. Other detectors may require additional configuration as described below. When using multiple detectors they will run in dedicated processes, but pull from a common queue of detection requests from across all cameras.
|
||||
Frigate provides a number of builtin detector types. By default, Frigate will use a single CPU detector. Other detectors may require additional configuration as described below. When using multiple detectors they will run in dedicated processes, but pull from a common queue of detection requests from across all cameras.
|
||||
|
||||
## Edge TPU Detector
|
||||
|
||||
The Edge TPU detector type runs a TensorFlow Lite model utilizing the Google Coral delegate for hardware acceleration. To configure an Edge TPU detector, set the `"type"` attribute to `"edgetpu"`.
|
||||
The Edge TPU detector type runs TensorFlow Lite models utilizing the Google Coral delegate for hardware acceleration. To configure an Edge TPU detector, set the `"type"` attribute to `"edgetpu"`.
|
||||
|
||||
The Edge TPU device can be specified using the `"device"` attribute according to the [Documentation for the TensorFlow Lite Python API](https://coral.ai/docs/edgetpu/multiple-edgetpu/#using-the-tensorflow-lite-python-api). If not set, the delegate will use the first device it finds.
|
||||
|
||||
A TensorFlow Lite model is provided in the container at `/edgetpu_model.tflite` and is used by this detector type by default. To provide your own model, bind mount the file into the container and provide the path with `model.path`.
|
||||
|
||||
:::tip
|
||||
|
||||
See [common Edge TPU troubleshooting steps](/troubleshooting/edgetpu) if the Edge TPU is not detected.
|
||||
@@ -134,6 +144,50 @@ detectors:
|
||||
device: pci
|
||||
```
|
||||
|
||||
### EdgeTPU Supported Models
|
||||
|
||||
| Model | Notes |
|
||||
| ----------------------- | ------------------------------------------- |
|
||||
| [Mobiledet](#mobiledet) | Default model |
|
||||
| [YOLOv9](#yolov9) | More accurate but slower than default model |
|
||||
|
||||
#### Mobiledet
|
||||
|
||||
A TensorFlow Lite model is provided in the container at `/edgetpu_model.tflite` and is used by this detector type by default. To provide your own model, bind mount the file into the container and provide the path with `model.path`.
|
||||
|
||||
#### YOLOv9
|
||||
|
||||
YOLOv9 models that are compiled for TensorFlow Lite and properly quantized are supported, but not included by default. [Instructions](#yolov9-for-google-coral-support) for downloading a model with support for the Google Coral.
|
||||
|
||||
:::tip
|
||||
|
||||
**Frigate+ Users:** Follow the [instructions](../integrations/plus#use-models) to set a model ID in your config file.
|
||||
|
||||
:::
|
||||
|
||||
<details>
|
||||
<summary>YOLOv9 Setup & Config</summary>
|
||||
|
||||
After placing the downloaded files for the tflite model and labels in your config folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
coral:
|
||||
type: edgetpu
|
||||
device: usb
|
||||
|
||||
model:
|
||||
model_type: yolo-generic
|
||||
width: 320 # <--- should match the imgsize of the model, typically 320
|
||||
height: 320 # <--- should match the imgsize of the model, typically 320
|
||||
path: /config/model_cache/yolov9-s-relu6-best_320_int8_edgetpu.tflite
|
||||
labelmap_path: /config/labels-coco17.txt
|
||||
```
|
||||
|
||||
Note that due to hardware limitations of the Coral, the labelmap is a subset of the COCO labels and includes only 17 object classes.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Hailo-8
|
||||
@@ -243,41 +297,55 @@ Hailo8 supports all models in the Hailo Model Zoo that include HailoRT post-proc
|
||||
|
||||
## OpenVINO Detector
|
||||
|
||||
The OpenVINO detector type runs an OpenVINO IR model on AMD and Intel CPUs, Intel GPUs and Intel VPU hardware. To configure an OpenVINO detector, set the `"type"` attribute to `"openvino"`.
|
||||
The OpenVINO detector type runs an OpenVINO IR model on AMD and Intel CPUs, Intel GPUs and Intel NPUs. To configure an OpenVINO detector, set the `"type"` attribute to `"openvino"`.
|
||||
|
||||
The OpenVINO device to be used is specified using the `"device"` attribute according to the naming conventions in the [Device Documentation](https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes.html). The most common devices are `CPU` and `GPU`. Currently, there is a known issue with using `AUTO`. For backwards compatibility, Frigate will attempt to use `GPU` if `AUTO` is set in your configuration.
|
||||
The OpenVINO device to be used is specified using the `"device"` attribute according to the naming conventions in the [Device Documentation](https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes.html). The most common devices are `CPU`, `GPU`, or `NPU`.
|
||||
|
||||
OpenVINO is supported on 6th Gen Intel platforms (Skylake) and newer. It will also run on AMD CPUs despite having no official support for it. A supported Intel platform is required to use the `GPU` device with OpenVINO. For detailed system requirements, see [OpenVINO System Requirements](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html)
|
||||
OpenVINO is supported on 6th Gen Intel platforms (Skylake) and newer. It will also run on AMD CPUs despite having no official support for it. A supported Intel platform is required to use the `GPU` or `NPU` device with OpenVINO. For detailed system requirements, see [OpenVINO System Requirements](https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html)
|
||||
|
||||
:::tip
|
||||
|
||||
**NPU + GPU Systems:** If you have both NPU and GPU available (Intel Core Ultra processors), use NPU for object detection and GPU for enrichments (semantic search, face recognition, etc.) for best performance and compatibility.
|
||||
|
||||
When using many cameras one detector may not be enough to keep up. Multiple detectors can be defined assuming GPU resources are available. An example configuration would be:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
ov_0:
|
||||
type: openvino
|
||||
device: GPU
|
||||
device: GPU # or NPU
|
||||
ov_1:
|
||||
type: openvino
|
||||
device: GPU
|
||||
device: GPU # or NPU
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
### Supported Models
|
||||
### OpenVINO Supported Models
|
||||
|
||||
| Model | GPU | NPU | Notes |
|
||||
| ------------------------------------- | --- | --- | ------------------------------------------------------------ |
|
||||
| [YOLOv9](#yolo-v3-v4-v7-v9) | ✅ | ✅ | Recommended for GPU & NPU |
|
||||
| [RF-DETR](#rf-detr) | ✅ | ✅ | Requires XE iGPU or Arc |
|
||||
| [YOLO-NAS](#yolo-nas) | ✅ | ✅ | |
|
||||
| [MobileNet v2](#ssdlite-mobilenet-v2) | ✅ | ✅ | Fast and lightweight model, less accurate than larger models |
|
||||
| [YOLOX](#yolox) | ✅ | ? | |
|
||||
| [D-FINE](#d-fine) | ❌ | ❌ | |
|
||||
|
||||
#### SSDLite MobileNet v2
|
||||
|
||||
An OpenVINO model is provided in the container at `/openvino-model/ssdlite_mobilenet_v2.xml` and is used by this detector type by default. The model comes from Intel's Open Model Zoo [SSDLite MobileNet V2](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssdlite_mobilenet_v2) and is converted to an FP16 precision IR model.
|
||||
|
||||
<details>
|
||||
<summary>MobileNet v2 Config</summary>
|
||||
|
||||
Use the model configuration shown below when using the OpenVINO detector with the default OpenVINO model:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
ov:
|
||||
type: openvino
|
||||
device: GPU
|
||||
device: GPU # Or NPU
|
||||
|
||||
model:
|
||||
width: 300
|
||||
@@ -288,6 +356,8 @@ model:
|
||||
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
#### YOLOX
|
||||
|
||||
This detector also supports YOLOX. Frigate does not come with any YOLOX models preloaded, so you will need to supply your own models.
|
||||
@@ -296,6 +366,9 @@ This detector also supports YOLOX. Frigate does not come with any YOLOX models p
|
||||
|
||||
[YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md) models are supported, but not included by default. See [the models section](#downloading-yolo-nas-model) for more information on downloading the YOLO-NAS model for use in Frigate.
|
||||
|
||||
<details>
|
||||
<summary>YOLO-NAS Setup & Config</summary>
|
||||
|
||||
After placing the downloaded onnx model in your config folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
@@ -316,6 +389,8 @@ model:
|
||||
|
||||
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
|
||||
|
||||
</details>
|
||||
|
||||
#### YOLO (v3, v4, v7, v9)
|
||||
|
||||
YOLOv3, YOLOv4, YOLOv7, and [YOLOv9](https://github.com/WongKinYiu/yolov9) models are supported, but not included by default.
|
||||
@@ -326,9 +401,12 @@ The YOLO detector has been designed to support YOLOv3, YOLOv4, YOLOv7, and YOLOv
|
||||
|
||||
:::
|
||||
|
||||
<details>
|
||||
<summary>YOLOv Setup & Config</summary>
|
||||
|
||||
:::warning
|
||||
|
||||
If you are using a Frigate+ YOLOv9 model, you should not define any of the below `model` parameters in your config except for `path`. See [the Frigate+ model docs](/plus/first_model#step-3-set-your-model-id-in-the-config) for more information on setting up your model.
|
||||
If you are using a Frigate+ model, you should not define any of the below `model` parameters in your config except for `path`. See [the Frigate+ model docs](/plus/first_model#step-3-set-your-model-id-in-the-config) for more information on setting up your model.
|
||||
|
||||
:::
|
||||
|
||||
@@ -338,7 +416,7 @@ After placing the downloaded onnx model in your config folder, you can use the f
|
||||
detectors:
|
||||
ov:
|
||||
type: openvino
|
||||
device: GPU
|
||||
device: GPU # or NPU
|
||||
|
||||
model:
|
||||
model_type: yolo-generic
|
||||
@@ -352,6 +430,8 @@ model:
|
||||
|
||||
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
|
||||
|
||||
</details>
|
||||
|
||||
#### RF-DETR
|
||||
|
||||
[RF-DETR](https://github.com/roboflow/rf-detr) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-rf-detr-model) for more informatoin on downloading the RF-DETR model for use in Frigate.
|
||||
@@ -362,6 +442,9 @@ Due to the size and complexity of the RF-DETR model, it is only recommended to b
|
||||
|
||||
:::
|
||||
|
||||
<details>
|
||||
<summary>RF-DETR Setup & Config</summary>
|
||||
|
||||
After placing the downloaded onnx model in your `config/model_cache` folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
@@ -379,6 +462,8 @@ model:
|
||||
path: /config/model_cache/rfdetr.onnx
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
#### D-FINE
|
||||
|
||||
[D-FINE](https://github.com/Peterande/D-FINE) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-d-fine-model) for more information on downloading the D-FINE model for use in Frigate.
|
||||
@@ -389,6 +474,9 @@ Currently D-FINE models only run on OpenVINO in CPU mode, GPUs currently fail to
|
||||
|
||||
:::
|
||||
|
||||
<details>
|
||||
<summary>D-FINE Setup & Config</summary>
|
||||
|
||||
After placing the downloaded onnx model in your config/model_cache folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
@@ -403,7 +491,63 @@ model:
|
||||
height: 640
|
||||
input_tensor: nchw
|
||||
input_dtype: float
|
||||
path: /config/model_cache/dfine_s_obj2coco.onnx
|
||||
path: /config/model_cache/dfine-s.onnx
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
```
|
||||
|
||||
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
|
||||
|
||||
</details>
|
||||
|
||||
## Apple Silicon detector
|
||||
|
||||
The NPU in Apple Silicon can't be accessed from within a container, so the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) must first be setup. It is recommended to use the Frigate docker image with `-standard-arm64` suffix, for example `ghcr.io/blakeblackshear/frigate:stable-standard-arm64`.
|
||||
|
||||
### Setup
|
||||
|
||||
1. Setup the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) and run the client
|
||||
2. Configure the detector in Frigate and startup Frigate
|
||||
|
||||
### Configuration
|
||||
|
||||
Using the detector config below will connect to the client:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
apple-silicon:
|
||||
type: zmq
|
||||
endpoint: tcp://host.docker.internal:5555
|
||||
```
|
||||
|
||||
### Apple Silicon Supported Models
|
||||
|
||||
There is no default model provided, the following formats are supported:
|
||||
|
||||
#### YOLO (v3, v4, v7, v9)
|
||||
|
||||
YOLOv3, YOLOv4, YOLOv7, and [YOLOv9](https://github.com/WongKinYiu/yolov9) models are supported, but not included by default.
|
||||
|
||||
:::tip
|
||||
|
||||
The YOLO detector has been designed to support YOLOv3, YOLOv4, YOLOv7, and YOLOv9 models, but may support other YOLO model architectures as well. See [the models section](#downloading-yolo-models) for more information on downloading YOLO models for use in Frigate.
|
||||
|
||||
:::
|
||||
|
||||
When Frigate is started with the following config it will connect to the detector client and transfer the model automatically:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
apple-silicon:
|
||||
type: zmq
|
||||
endpoint: tcp://host.docker.internal:5555
|
||||
|
||||
model:
|
||||
model_type: yolo-generic
|
||||
width: 320 # <--- should match the imgsize set during model export
|
||||
height: 320 # <--- should match the imgsize set during model export
|
||||
input_tensor: nchw
|
||||
input_dtype: float
|
||||
path: /config/model_cache/yolo.onnx
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
```
|
||||
|
||||
@@ -489,7 +633,18 @@ We unset the `HSA_OVERRIDE_GFX_VERSION` to prevent an existing override from mes
|
||||
$ docker exec -it frigate /bin/bash -c '(unset HSA_OVERRIDE_GFX_VERSION && /opt/rocm/bin/rocminfo |grep gfx)'
|
||||
```
|
||||
|
||||
### Supported Models
|
||||
### ROCm Supported Models
|
||||
|
||||
:::tip
|
||||
|
||||
The AMD GPU kernel is known problematic especially when converting models to mxr format. The recommended approach is:
|
||||
|
||||
1. Disable object detection in the config.
|
||||
2. Startup Frigate with the onnx detector configured, the main object detection model will be converted to mxr format and cached in the config directory.
|
||||
3. Once this is finished as indicated by the logs, enable object detection in the UI and confirm that it is working correctly.
|
||||
4. Re-enable object detection in the config.
|
||||
|
||||
:::
|
||||
|
||||
See [ONNX supported models](#supported-models) for supported models, there are some caveats:
|
||||
|
||||
@@ -505,11 +660,9 @@ ONNX is an open format for building machine learning models, Frigate supports ru
|
||||
If the correct build is used for your GPU then the GPU will be detected and used automatically.
|
||||
|
||||
- **AMD**
|
||||
|
||||
- ROCm will automatically be detected and used with the ONNX detector in the `-rocm` Frigate image.
|
||||
|
||||
- **Intel**
|
||||
|
||||
- OpenVINO will automatically be detected and used with the ONNX detector in the default Frigate image.
|
||||
|
||||
- **Nvidia**
|
||||
@@ -532,7 +685,15 @@ detectors:
|
||||
|
||||
:::
|
||||
|
||||
### Supported Models
|
||||
### ONNX Supported Models
|
||||
|
||||
| Model | Nvidia GPU | AMD GPU | Notes |
|
||||
| ----------------------------- | ---------- | ------- | --------------------------------------------------- |
|
||||
| [YOLOv9](#yolo-v3-v4-v7-v9-2) | ✅ | ✅ | Supports CUDA Graphs for optimal Nvidia performance |
|
||||
| [RF-DETR](#rf-detr) | ✅ | ❌ | Supports CUDA Graphs for optimal Nvidia performance |
|
||||
| [YOLO-NAS](#yolo-nas-1) | ⚠️ | ⚠️ | Not supported by CUDA Graphs |
|
||||
| [YOLOX](#yolox-1) | ✅ | ✅ | Supports CUDA Graphs for optimal Nvidia performance |
|
||||
| [D-FINE](#d-fine) | ⚠️ | ❌ | Not supported by CUDA Graphs |
|
||||
|
||||
There is no default model provided, the following formats are supported:
|
||||
|
||||
@@ -540,6 +701,9 @@ There is no default model provided, the following formats are supported:
|
||||
|
||||
[YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md) models are supported, but not included by default. See [the models section](#downloading-yolo-nas-model) for more information on downloading the YOLO-NAS model for use in Frigate.
|
||||
|
||||
<details>
|
||||
<summary>YOLO-NAS Setup & Config</summary>
|
||||
|
||||
:::warning
|
||||
|
||||
If you are using a Frigate+ YOLO-NAS model, you should not define any of the below `model` parameters in your config except for `path`. See [the Frigate+ model docs](/plus/first_model#step-3-set-your-model-id-in-the-config) for more information on setting up your model.
|
||||
@@ -563,6 +727,8 @@ model:
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
#### YOLO (v3, v4, v7, v9)
|
||||
|
||||
YOLOv3, YOLOv4, YOLOv7, and [YOLOv9](https://github.com/WongKinYiu/yolov9) models are supported, but not included by default.
|
||||
@@ -573,9 +739,12 @@ The YOLO detector has been designed to support YOLOv3, YOLOv4, YOLOv7, and YOLOv
|
||||
|
||||
:::
|
||||
|
||||
<details>
|
||||
<summary>YOLOv Setup & Config</summary>
|
||||
|
||||
:::warning
|
||||
|
||||
If you are using a Frigate+ YOLOv9 model, you should not define any of the below `model` parameters in your config except for `path`. See [the Frigate+ model docs](/plus/first_model#step-3-set-your-model-id-in-the-config) for more information on setting up your model.
|
||||
If you are using a Frigate+ model, you should not define any of the below `model` parameters in your config except for `path`. See [the Frigate+ model docs](/plus/first_model#step-3-set-your-model-id-in-the-config) for more information on setting up your model.
|
||||
|
||||
:::
|
||||
|
||||
@@ -596,12 +765,17 @@ model:
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
|
||||
|
||||
#### YOLOx
|
||||
|
||||
[YOLOx](https://github.com/Megvii-BaseDetection/YOLOX) models are supported, but not included by default. See [the models section](#downloading-yolo-models) for more information on downloading the YOLOx model for use in Frigate.
|
||||
|
||||
<details>
|
||||
<summary>YOLOx Setup & Config</summary>
|
||||
|
||||
After placing the downloaded onnx model in your config folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
@@ -621,10 +795,15 @@ model:
|
||||
|
||||
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
|
||||
|
||||
</details>
|
||||
|
||||
#### RF-DETR
|
||||
|
||||
[RF-DETR](https://github.com/roboflow/rf-detr) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-rf-detr-model) for more information on downloading the RF-DETR model for use in Frigate.
|
||||
|
||||
<details>
|
||||
<summary>RF-DETR Setup & Config</summary>
|
||||
|
||||
After placing the downloaded onnx model in your `config/model_cache` folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
@@ -641,10 +820,15 @@ model:
|
||||
path: /config/model_cache/rfdetr.onnx
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
#### D-FINE
|
||||
|
||||
[D-FINE](https://github.com/Peterande/D-FINE) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-d-fine-model) for more information on downloading the D-FINE model for use in Frigate.
|
||||
|
||||
<details>
|
||||
<summary>D-FINE Setup & Config</summary>
|
||||
|
||||
After placing the downloaded onnx model in your `config/model_cache` folder, you can use the following configuration:
|
||||
|
||||
```yaml
|
||||
@@ -662,6 +846,8 @@ model:
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
|
||||
|
||||
## CPU Detector (not recommended)
|
||||
@@ -717,6 +903,196 @@ To verify that the integration is working correctly, start Frigate and observe t
|
||||
|
||||
# Community Supported Detectors
|
||||
|
||||
## MemryX MX3
|
||||
|
||||
This detector is available for use with the MemryX MX3 accelerator M.2 module. Frigate supports the MX3 on compatible hardware platforms, providing efficient and high-performance object detection.
|
||||
|
||||
See the [installation docs](../frigate/installation.md#memryx-mx3) for information on configuring the MemryX hardware.
|
||||
|
||||
To configure a MemryX detector, simply set the `type` attribute to `memryx` and follow the configuration guide below.
|
||||
|
||||
### Configuration
|
||||
|
||||
To configure the MemryX detector, use the following example configuration:
|
||||
|
||||
#### Single PCIe MemryX MX3
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
memx0:
|
||||
type: memryx
|
||||
device: PCIe:0
|
||||
```
|
||||
|
||||
#### Multiple PCIe MemryX MX3 Modules
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
memx0:
|
||||
type: memryx
|
||||
device: PCIe:0
|
||||
|
||||
memx1:
|
||||
type: memryx
|
||||
device: PCIe:1
|
||||
|
||||
memx2:
|
||||
type: memryx
|
||||
device: PCIe:2
|
||||
```
|
||||
|
||||
### Supported Models
|
||||
|
||||
MemryX `.dfp` models are automatically downloaded at runtime, if enabled, to the container at `/memryx_models/model_folder/`.
|
||||
|
||||
#### YOLO-NAS
|
||||
|
||||
The [YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md) model included in this detector is downloaded from the [Models Section](#downloading-yolo-nas-model) and compiled to DFP with [mx_nc](https://developer.memryx.com/tools/neural_compiler.html#usage).
|
||||
|
||||
**Note:** The default model for the MemryX detector is YOLO-NAS 320x320.
|
||||
|
||||
The input size for **YOLO-NAS** can be set to either **320x320** (default) or **640x640**.
|
||||
|
||||
- The default size of **320x320** is optimized for lower CPU usage and faster inference times.
|
||||
|
||||
##### Configuration
|
||||
|
||||
Below is the recommended configuration for using the **YOLO-NAS** (small) model with the MemryX detector:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
memx0:
|
||||
type: memryx
|
||||
device: PCIe:0
|
||||
|
||||
model:
|
||||
model_type: yolonas
|
||||
width: 320 # (Can be set to 640 for higher resolution)
|
||||
height: 320 # (Can be set to 640 for higher resolution)
|
||||
input_tensor: nchw
|
||||
input_dtype: float
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
||||
# path: /config/yolonas.zip
|
||||
# The .zip file must contain:
|
||||
# ├── yolonas.dfp (a file ending with .dfp)
|
||||
# └── yolonas_post.onnx (optional; only if the model includes a cropped post-processing network)
|
||||
```
|
||||
|
||||
#### YOLOv9
|
||||
|
||||
The YOLOv9s model included in this detector is downloaded from [the original GitHub](https://github.com/WongKinYiu/yolov9) like in the [Models Section](#yolov9-1) and compiled to DFP with [mx_nc](https://developer.memryx.com/tools/neural_compiler.html#usage).
|
||||
|
||||
##### Configuration
|
||||
|
||||
Below is the recommended configuration for using the **YOLOv9** (small) model with the MemryX detector:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
memx0:
|
||||
type: memryx
|
||||
device: PCIe:0
|
||||
|
||||
model:
|
||||
model_type: yolo-generic
|
||||
width: 320 # (Can be set to 640 for higher resolution)
|
||||
height: 320 # (Can be set to 640 for higher resolution)
|
||||
input_tensor: nchw
|
||||
input_dtype: float
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
||||
# path: /config/yolov9.zip
|
||||
# The .zip file must contain:
|
||||
# ├── yolov9.dfp (a file ending with .dfp)
|
||||
```
|
||||
|
||||
#### YOLOX
|
||||
|
||||
The model is sourced from the [OpenCV Model Zoo](https://github.com/opencv/opencv_zoo) and precompiled to DFP.
|
||||
|
||||
##### Configuration
|
||||
|
||||
Below is the recommended configuration for using the **YOLOX** (small) model with the MemryX detector:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
memx0:
|
||||
type: memryx
|
||||
device: PCIe:0
|
||||
|
||||
model:
|
||||
model_type: yolox
|
||||
width: 640
|
||||
height: 640
|
||||
input_tensor: nchw
|
||||
input_dtype: float_denorm
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
||||
# path: /config/yolox.zip
|
||||
# The .zip file must contain:
|
||||
# ├── yolox.dfp (a file ending with .dfp)
|
||||
```
|
||||
|
||||
#### SSDLite MobileNet v2
|
||||
|
||||
The model is sourced from the [OpenMMLab Model Zoo](https://mmdeploy-oss.openmmlab.com/model/mmdet-det/ssdlite-e8679f.onnx) and has been converted to DFP.
|
||||
|
||||
##### Configuration
|
||||
|
||||
Below is the recommended configuration for using the **SSDLite MobileNet v2** model with the MemryX detector:
|
||||
|
||||
```yaml
|
||||
detectors:
|
||||
memx0:
|
||||
type: memryx
|
||||
device: PCIe:0
|
||||
|
||||
model:
|
||||
model_type: ssd
|
||||
width: 320
|
||||
height: 320
|
||||
input_tensor: nchw
|
||||
input_dtype: float
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
||||
# path: /config/ssdlite_mobilenet.zip
|
||||
# The .zip file must contain:
|
||||
# ├── ssdlite_mobilenet.dfp (a file ending with .dfp)
|
||||
# └── ssdlite_mobilenet_post.onnx (optional; only if the model includes a cropped post-processing network)
|
||||
```
|
||||
|
||||
#### Using a Custom Model
|
||||
|
||||
To use your own model:
|
||||
|
||||
1. Package your compiled model into a `.zip` file.
|
||||
|
||||
2. The `.zip` must contain the compiled `.dfp` file.
|
||||
|
||||
3. Depending on the model, the compiler may also generate a cropped post-processing network. If present, it will be named with the suffix `_post.onnx`.
|
||||
|
||||
4. Bind-mount the `.zip` file into the container and specify its path using `model.path` in your config.
|
||||
|
||||
5. Update the `labelmap_path` to match your custom model's labels.
|
||||
|
||||
For detailed instructions on compiling models, refer to the [MemryX Compiler](https://developer.memryx.com/tools/neural_compiler.html#usage) docs and [Tutorials](https://developer.memryx.com/tutorials/tutorials.html).
|
||||
|
||||
```yaml
|
||||
# The detector automatically selects the default model if nothing is provided in the config.
|
||||
#
|
||||
# Optionally, you can specify a local model path as a .zip file to override the default.
|
||||
# If a local path is provided and the file exists, it will be used instead of downloading.
|
||||
#
|
||||
# Example:
|
||||
# path: /config/yolonas.zip
|
||||
#
|
||||
# The .zip file must contain:
|
||||
# ├── yolonas.dfp (a file ending with .dfp)
|
||||
# └── yolonas_post.onnx (optional; only if the model includes a cropped post-processing network)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## NVidia TensorRT Detector
|
||||
|
||||
Nvidia Jetson devices may be used for object detection using the TensorRT libraries. Due to the size of the additional libraries, this detector is only provided in images with the `-tensorrt-jp6` tag suffix, e.g. `ghcr.io/blakeblackshear/frigate:stable-tensorrt-jp6`. This detector is designed to work with Yolo models for object detection.
|
||||
@@ -799,6 +1175,41 @@ model:
|
||||
height: 320 # MUST match the chosen model i.e yolov7-320 -> 320 yolov4-416 -> 416
|
||||
```
|
||||
|
||||
## Synaptics
|
||||
|
||||
Hardware accelerated object detection is supported on the following SoCs:
|
||||
|
||||
- SL1680
|
||||
|
||||
This implementation uses the [Synaptics model conversion](https://synaptics-synap.github.io/doc/v/latest/docs/manual/introduction.html#offline-model-conversion), version v3.1.0.
|
||||
|
||||
This implementation is based on sdk `v1.5.0`.
|
||||
|
||||
See the [installation docs](../frigate/installation.md#synaptics) for information on configuring the SL-series NPU hardware.
|
||||
|
||||
### Configuration
|
||||
|
||||
When configuring the Synap detector, you have to specify the model: a local **path**.
|
||||
|
||||
#### SSD Mobilenet
|
||||
|
||||
A synap model is provided in the container at /mobilenet.synap and is used by this detector type by default. The model comes from [Synap-release Github](https://github.com/synaptics-astra/synap-release/tree/v1.5.0/models/dolphin/object_detection/coco/model/mobilenet224_full80).
|
||||
|
||||
Use the model configuration shown below when using the synaptics detector with the default synap model:
|
||||
|
||||
```yaml
|
||||
detectors: # required
|
||||
synap_npu: # required
|
||||
type: synaptics # required
|
||||
|
||||
model: # required
|
||||
path: /synaptics/mobilenet.synap # required
|
||||
width: 224 # required
|
||||
height: 224 # required
|
||||
tensor_format: nhwc # default value (optional. If you change the model, it is required)
|
||||
labelmap_path: /labelmap/coco-80.txt # required
|
||||
```
|
||||
|
||||
## Rockchip platform
|
||||
|
||||
Hardware accelerated object detection is supported on the following SoCs:
|
||||
@@ -842,7 +1253,7 @@ $ cat /sys/kernel/debug/rknpu/load
|
||||
|
||||
:::
|
||||
|
||||
### Supported Models
|
||||
### RockChip Supported Models
|
||||
|
||||
This `config.yml` shows all relevant options to configure the detector and explains them. All values shown are the default values (except for two). Lines that are required at least to use the detector are labeled as required, all other lines are optional.
|
||||
|
||||
@@ -968,6 +1379,105 @@ Explanation of the paramters:
|
||||
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
|
||||
- `config`: Configuration passed to `rknn-toolkit2` for model conversion. For an explanation of all available parameters have a look at section "2.2. Model configuration" of [this manual](https://github.com/MarcA711/rknn-toolkit2/releases/download/v2.3.2/03_Rockchip_RKNPU_API_Reference_RKNN_Toolkit2_V2.3.2_EN.pdf).
|
||||
|
||||
## DeGirum
|
||||
|
||||
DeGirum is a detector that can use any type of hardware listed on [their website](https://hub.degirum.com). DeGirum can be used with local hardware through a DeGirum AI Server, or through the use of `@local`. You can also connect directly to DeGirum's AI Hub to run inferences. **Please Note:** This detector _cannot_ be used for commercial purposes.
|
||||
|
||||
### Configuration
|
||||
|
||||
#### AI Server Inference
|
||||
|
||||
Before starting with the config file for this section, you must first launch an AI server. DeGirum has an AI server ready to use as a docker container. Add this to your `docker-compose.yml` to get started:
|
||||
|
||||
```yaml
|
||||
degirum_detector:
|
||||
container_name: degirum
|
||||
image: degirum/aiserver:latest
|
||||
privileged: true
|
||||
ports:
|
||||
- "8778:8778"
|
||||
```
|
||||
|
||||
All supported hardware will automatically be found on your AI server host as long as relevant runtimes and drivers are properly installed on your machine. Refer to [DeGirum's docs site](https://docs.degirum.com/pysdk/runtimes-and-drivers) if you have any trouble.
|
||||
|
||||
Once completed, changing the `config.yml` file is simple.
|
||||
|
||||
```yaml
|
||||
degirum_detector:
|
||||
type: degirum
|
||||
location: degirum # Set to service name (degirum_detector), container_name (degirum), or a host:port (192.168.29.4:8778)
|
||||
zoo: degirum/public # DeGirum's public model zoo. Zoo name should be in format "workspace/zoo_name". degirum/public is available to everyone, so feel free to use it if you don't know where to start. If you aren't pulling a model from the AI Hub, leave this and 'token' blank.
|
||||
token: dg_example_token # For authentication with the AI Hub. Get this token through the "tokens" section on the main page of the [AI Hub](https://hub.degirum.com). This can be left blank if you're pulling a model from the public zoo and running inferences on your local hardware using @local or a local DeGirum AI Server
|
||||
```
|
||||
|
||||
Setting up a model in the `config.yml` is similar to setting up an AI server.
|
||||
You can set it to:
|
||||
|
||||
- A model listed on the [AI Hub](https://hub.degirum.com), given that the correct zoo name is listed in your detector
|
||||
- If this is what you choose to do, the correct model will be downloaded onto your machine before running.
|
||||
- A local directory acting as a zoo. See DeGirum's docs site [for more information](https://docs.degirum.com/pysdk/user-guide-pysdk/organizing-models#model-zoo-directory-structure).
|
||||
- A path to some model.json.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
path: ./mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1 # directory to model .json and file
|
||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
||||
```
|
||||
|
||||
#### Local Inference
|
||||
|
||||
It is also possible to eliminate the need for an AI server and run the hardware directly. The benefit of this approach is that you eliminate any bottlenecks that occur when transferring prediction results from the AI server docker container to the frigate one. However, the method of implementing local inference is different for every device and hardware combination, so it's usually more trouble than it's worth. A general guideline to achieve this would be:
|
||||
|
||||
1. Ensuring that the frigate docker container has the runtime you want to use. So for instance, running `@local` for Hailo means making sure the container you're using has the Hailo runtime installed.
|
||||
2. To double check the runtime is detected by the DeGirum detector, make sure the `degirum sys-info` command properly shows whatever runtimes you mean to install.
|
||||
3. Create a DeGirum detector in your `config.yml` file.
|
||||
|
||||
```yaml
|
||||
degirum_detector:
|
||||
type: degirum
|
||||
location: "@local" # For accessing AI Hub devices and models
|
||||
zoo: degirum/public # DeGirum's public model zoo. Zoo name should be in format "workspace/zoo_name". degirum/public is available to everyone, so feel free to use it if you don't know where to start.
|
||||
token: dg_example_token # For authentication with the AI Hub. Get this token through the "tokens" section on the main page of the [AI Hub](https://hub.degirum.com). This can be left blank if you're pulling a model from the public zoo and running inferences on your local hardware using @local or a local DeGirum AI Server
|
||||
```
|
||||
|
||||
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
|
||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
||||
```
|
||||
|
||||
#### AI Hub Cloud Inference
|
||||
|
||||
If you do not possess whatever hardware you want to run, there's also the option to run cloud inferences. Do note that your detection fps might need to be lowered as network latency does significantly slow down this method of detection. For use with Frigate, we highly recommend using a local AI server as described above. To set up cloud inferences,
|
||||
|
||||
1. Sign up at [DeGirum's AI Hub](https://hub.degirum.com).
|
||||
2. Get an access token.
|
||||
3. Create a DeGirum detector in your `config.yml` file.
|
||||
|
||||
```yaml
|
||||
degirum_detector:
|
||||
type: degirum
|
||||
location: "@cloud" # For accessing AI Hub devices and models
|
||||
zoo: degirum/public # DeGirum's public model zoo. Zoo name should be in format "workspace/zoo_name". degirum/public is available to everyone, so feel free to use it if you don't know where to start.
|
||||
token: dg_example_token # For authentication with the AI Hub. Get this token through the "tokens" section on the main page of the (AI Hub)[https://hub.degirum.com).
|
||||
```
|
||||
|
||||
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
|
||||
width: 300 # width is in the model name as the first number in the "int"x"int" section
|
||||
height: 300 # height is in the model name as the second number in the "int"x"int" section
|
||||
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
|
||||
```
|
||||
|
||||
# Models
|
||||
|
||||
Some model types are not included in Frigate by default.
|
||||
@@ -1008,11 +1518,11 @@ RF-DETR can be exported as ONNX by running the command below. You can copy and p
|
||||
|
||||
```sh
|
||||
docker build . --build-arg MODEL_SIZE=Nano --rm --output . -f- <<'EOF'
|
||||
FROM python:3.11 AS build
|
||||
FROM python:3.12 AS build
|
||||
RUN apt-get update && apt-get install --no-install-recommends -y libgl1 && rm -rf /var/lib/apt/lists/*
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.8.0 /uv /bin/
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
|
||||
WORKDIR /rfdetr
|
||||
RUN uv pip install --system rfdetr[onnxexport] torch==2.8.0 onnx==1.19.1 onnxscript
|
||||
RUN uv pip install --system rfdetr[onnxexport] torch==2.8.0 onnx==1.19.1 transformers==4.57.6 onnxscript
|
||||
ARG MODEL_SIZE
|
||||
RUN python3 -c "from rfdetr import RFDETR${MODEL_SIZE}; x = RFDETR${MODEL_SIZE}(resolution=320); x.export(simplify=True)"
|
||||
FROM scratch
|
||||
@@ -1050,7 +1560,11 @@ cd tensorrt_demos/yolo
|
||||
python3 yolo_to_onnx.py -m yolov7-320
|
||||
```
|
||||
|
||||
#### YOLOv9
|
||||
#### YOLOv9 for Google Coral Support
|
||||
|
||||
[Download the model](https://github.com/dbro/frigate-detector-edgetpu-yolo9/releases/download/v1.0/yolov9-s-relu6-best_320_int8_edgetpu.tflite), bind mount the file into the container, and provide the path with `model.path`. Note that the linked model requires a 17-label [labelmap file](https://raw.githubusercontent.com/dbro/frigate-detector-edgetpu-yolo9/refs/heads/main/labels-coco17.txt) that includes only 17 COCO classes.
|
||||
|
||||
#### YOLOv9 for other detectors
|
||||
|
||||
YOLOv9 model can be exported as ONNX using the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=t` and `IMG_SIZE=320` in the first line to the [model size](https://github.com/WongKinYiu/yolov9#performance) you would like to convert (available model sizes are `t`, `s`, `m`, `c`, and `e`, common image sizes are `320` and `640`).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user