mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-09-24 18:26:51 +03:00
Feat/deepx npu detector (#24336)
CI / AMD64 Build (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Smoke Test (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s
CI / AMD64 Build (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Smoke Test (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s
* feat(deepx): add DEEPX NPU detector and runtime integration. * feat(deepx): enforce model_format requirement when ppu is enabled and add integrity checks for driver installation * Update frigate/detectors/plugins/deepx.py Public method lacks docstring Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Refactor DEEPX detector tests, support SSD and DAMO-YOLO * feat(deepx): add anchor-free output decoding and corresponding tests * Add tests and updates for DEEPX detector and refactor DEEPX accelerator code structure. * fix: enhance model type validation and update documentation for DEEPX detector * fix: add support for customizable score and NMS thresholds * refactor: infer YOLO layout from the model, drop per-detector options and the dxrtd placeholder * fix: keep only the anchor-free PPU verdict, re-read anchor-based each frame * Update latency data for DEEPX NPU * Expanding PPU support for DEEPX and set yolo-generic as default. * enhance scale count resolution logic * Extend PPU layout handling and YOLOX support to DEEPX detector * fix: assume the largest PPU anchor table when the .dxnn has no layout * Improve PPU decoding and introduce strides handling * Improve PPU scope and fix box format mismatch * Fix unnamed node issue that breaks traversal * fix: update object detection model type description to remove outdated architecture --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
This commit is contained in:
A. Ahmet
co-authored by
greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
parent
52f50a7396
commit
af0ba19196
@@ -967,6 +967,65 @@ memryx:
|
||||
# The .zip file must contain:
|
||||
# ├── ssdlite_mobilenet.dfp (a file ending with .dfp)
|
||||
# └── ssdlite_mobilenet_post.onnx (optional; only if the model includes a cropped post-processing network)
|
||||
deepx:
|
||||
title: DEEPX NPU
|
||||
models:
|
||||
- key: yolo
|
||||
label: YOLO
|
||||
recommended: true
|
||||
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The recommended model is `yolox-s_640x640_ppu.dxnn`. Its Post-Processing Unit (PPU) compile moves candidate selection onto the NPU, which makes it the fastest ModelZoo model measured through Frigate (about 13 ms on a DX-M1). The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration; prefer a `PPU` variant whenever the ModelZoo offers one. PPU models must be compiled with DX-COM 2.4.0 or later, which writes the head layout Frigate reads into the file.
|
||||
ui: |-
|
||||
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
||||
|
||||
| Field | Value |
|
||||
| ---------------------------------------- | ---------------------------------------------------- |
|
||||
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640_ppu.dxnn` |
|
||||
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
||||
| **Object detection model input width** | `640` |
|
||||
| **Object detection model input height** | `640` |
|
||||
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
||||
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
||||
| **Model Input D Type** | `int` (Frigate's default value) |
|
||||
| **Object Detection Model Type** | `yolo-generic` |
|
||||
|
||||
Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
|
||||
yaml: |-
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
model_type: yolo-generic
|
||||
width: 640
|
||||
height: 640
|
||||
- key: yolox
|
||||
label: YOLOX
|
||||
recommended: false
|
||||
download: No model is bundled with Frigate. Download a pre-compiled YOLOX `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), then bind-mount it into the container and point the model's `path` at it. The `_ppu` variant is faster and also works with the `yolo-generic` model type; the plain export needs `yolox` so its raw head is decoded.
|
||||
ui: |-
|
||||
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
||||
|
||||
| Field | Value |
|
||||
| ---------------------------------------- | ---------------------------------------- |
|
||||
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640.dxnn`|
|
||||
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
||||
| **Object detection model input width** | `640` |
|
||||
| **Object detection model input height** | `640` |
|
||||
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
||||
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
||||
| **Model Input D Type** | `int` (Frigate's default value) |
|
||||
| **Object Detection Model Type** | `yolox` |
|
||||
|
||||
The width and height must match the resolution the `.dxnn` file was compiled for.
|
||||
yaml: |-
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
path: /config/model_cache/deepx/yolox-s_640x640.dxnn
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
model_type: yolox
|
||||
width: 640
|
||||
height: 640
|
||||
tensorrt:
|
||||
title: TensorRT
|
||||
models:
|
||||
|
||||
@@ -24,6 +24,7 @@ Frigate supports multiple different detectors that work on different types of ha
|
||||
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
|
||||
- [Hailo](#hailo): The Hailo-8, Hailo-8L and Hailo-8R AI Acceleration modules are available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
|
||||
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
|
||||
- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, offering broad compatibility across various platforms.
|
||||
|
||||
**AMD**
|
||||
|
||||
@@ -612,6 +613,63 @@ For detailed instructions on compiling models, refer to the [MemryX Compiler](ht
|
||||
|
||||
---
|
||||
|
||||
## DEEPX NPU
|
||||
|
||||
This detector is available for use with the DEEPX NPU, both the DX-M1 M.2 module and the DX-M1M on the Sixfab AI HAT+ for the Raspberry Pi 5. The configuration below applies unchanged to either form factor. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
|
||||
|
||||
See the [installation docs](../frigate/installation.md#deepx-npu) for information on installing the DEEPX kernel driver and runtime on the host and passing the NPU through to the container.
|
||||
|
||||
To run a model on a DEEPX NPU, list a `deepx` device on that model.
|
||||
|
||||
:::info
|
||||
|
||||
The DX-RT Python bindings are not part of the Frigate image. They are downloaded and installed into `/config/.local` the first time a DEEPX device is configured, verified against pinned checksums, and updated automatically when a Frigate release pins a new version. If the container has no internet access, see [Detector runtimes](/frigate/network_requirements#detector-runtimes) for how to provide the files yourself.
|
||||
|
||||
:::
|
||||
|
||||
### Configuration {#configuration-deepx}
|
||||
|
||||
<ModelConfigDropdown detectorTitle="DEEPX" models={objectDetectorsModels.deepx.models} />
|
||||
|
||||
Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Two model types are supported:
|
||||
|
||||
- `yolo-generic` for YOLO object detection models, the recommended default. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do models compiled with DEEPX's Post-Processing Unit (PPU) support.
|
||||
- `yolox` for YOLOX models compiled without PPU support, whose raw head needs Frigate's YOLOX decoder. A YOLOX model compiled with PPU support works under either `yolox` or `yolo-generic`.
|
||||
|
||||
The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler. The recommended starting point is `yolox-s_640x640_ppu.dxnn`, the fastest ModelZoo model measured through Frigate:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
model_type: yolo-generic
|
||||
width: 640
|
||||
height: 640
|
||||
```
|
||||
|
||||
For PPU models, use a `.dxnn` compiled with DX-COM 2.4.0 or later. Frigate reads the PPU head layout the compiler writes into the file and refuses to load a PPU model without it.
|
||||
|
||||
`model_type` must be set to `yolo-generic` or `yolox` to match the model; `yolo-generic` is the recommended default unless the model is a raw YOLOX export. Frigate defaults it to `ssd`, which this detector does not support, so the detector refuses to start on a model that leaves it unset.
|
||||
|
||||
`width` and `height` must match the resolution the model was compiled for. Quantization parameters are baked into the `.dxnn` file at compile time, so no normalization is applied on the host and Frigate's default `input_tensor`, `input_pixel_format`, and `input_dtype` values do not need to be overridden.
|
||||
|
||||
A DEEPX device is `PCIe:<index>`, as reported on the detector settings page. The NPU daemon multiplexes across processes, so the same device may be listed more than once to run additional inference processes against it:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
- deepx:PCIe:0
|
||||
```
|
||||
|
||||
#### Label maps
|
||||
|
||||
The object detection models in the DEEPX ModelZoo are trained on the standard 80-class COCO label set, so `labelmap_path` must be set to `/labelmap/coco-80.txt`. Frigate's default label map uses an extended 91-class COCO scheme, and leaving it in place will cause detections to be reported as the wrong object type. For `yolo-generic` models the label map is also what the detector uses to tell the output layout, so a label map with the wrong number of classes is reported as an error at startup.
|
||||
|
||||
---
|
||||
|
||||
## NVidia TensorRT Detector
|
||||
|
||||
Nvidia Jetson devices may be used for object detection using the TensorRT libraries. Due to the size of the additional libraries, this detector is only provided in images with the `-tensorrt-jp6` tag suffix, e.g. `ghcr.io/blakeblackshear/frigate:stable-tensorrt-jp6`. This detector is designed to work with Yolo models for object detection.
|
||||
|
||||
@@ -65,6 +65,11 @@ Frigate supports multiple different detectors that work on different types of ha
|
||||
- [Supports many model architectures](../../configuration/object_detectors#memryx-mx3)
|
||||
- Runs best with tiny, small, or medium-size models
|
||||
|
||||
- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, allowing for a wide range of compatibility with devices.
|
||||
- [Supports YOLO model architectures](../../configuration/object_detectors#deepx-npu)
|
||||
- Runs best with tiny or small size models
|
||||
- Runs efficiently on low power hardware
|
||||
|
||||
**AMD**
|
||||
|
||||
- [ROCm](#rocm---amd-gpu): ROCm can run on AMD Discrete GPUs to provide efficient object detection
|
||||
@@ -257,6 +262,32 @@ The MX3 is a pipelined architecture, where the maximum frames per second support
|
||||
|
||||
Inference speeds may vary depending on the host platform. The above data was measured on an **Intel 13700 CPU**. Platforms like Raspberry Pi, Orange Pi, and other ARM-based SBCs have different levels of processing capability, which may limit total FPS.
|
||||
|
||||
### DEEPX NPU
|
||||
|
||||
Frigate supports the DEEPX NPU in both of its form factors: the **DX-M1** M.2 module, which works on x86 (Intel/AMD) and ARM-based SBCs such as the Raspberry Pi 5, and the **DX-M1M** on the [Sixfab AI HAT+](https://docs.sixfab.com/docs/ai-hat-plus-raspberry-pi-5-quickstart) for the Raspberry Pi 5. Both use the same driver and runtime, so the configuration is identical for either one. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
|
||||
|
||||
The DEEPX driver and runtime run on the Docker host rather than inside the Frigate container and must be installed before the NPU can be used. See the [installation docs](installation.md#deepx-npu) for the setup steps and [the detector docs](/configuration/object_detectors#deepx-npu) for the configuration.
|
||||
|
||||
Frigate does not bundle a model for this detector. Models use DEEPX's `.dxnn` format, and pre-compiled YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Prefer a model with a `_ppu` suffix whenever one is available for the architecture you want: these run part of the post-processing on the NPU itself and are considerably faster, roughly 2.5x for the same architecture and input size. **YOLOX-S with PPU is the recommended starting point.**
|
||||
|
||||
Inference times for a few recommended models, measured through Frigate's own stats on a DX-M1:
|
||||
|
||||
| Model | Input Size | DX-M1 Inference Time |
|
||||
| ----------------- | ---------- | -------------------- |
|
||||
| YOLOX-S (PPU) | 640 | ~ 13 ms |
|
||||
| YOLOv9-t (PPU) | 640 | ~ 18 ms |
|
||||
| YOLOv4 (PPU) | 512 | ~ 20 ms |
|
||||
| YOLOX-S | 640 | ~ 34 ms |
|
||||
| YOLOv9-s | 640 | ~ 39 ms |
|
||||
|
||||
Other ModelZoo YOLO variants are also supported but have not been measured. Inference speeds vary with the host platform, so a slower host such as a Raspberry Pi 5 will report higher times than those above.
|
||||
|
||||
:::note
|
||||
|
||||
A few ModelZoo models can not be used with Frigate: SSD models (they are trained on Pascal VOC, so their labels do not match Frigate's), DAMO-YOLO models, face and pose models, and the PPU builds of YOLOv7.
|
||||
|
||||
:::
|
||||
|
||||
### Nvidia Jetson
|
||||
|
||||
Jetson devices are supported via the TensorRT or ONNX detectors when running Jetpack 6. It will [make use of the Jetson's hardware media engine](/configuration/hardware_acceleration_video#nvidia-jetson) when configured with the [appropriate presets](/configuration/ffmpeg_presets#hwaccel-presets), and will make use of the Jetson's GPU and DLA for object detection when configured with the [TensorRT detector](/configuration/object_detectors#nvidia-tensorrt-detector).
|
||||
|
||||
@@ -381,6 +381,99 @@ If you can't use Docker Compose, you can run the container with something simila
|
||||
|
||||
Finally, configure [hardware object detection](/configuration/object_detectors#memryx-mx3) to complete the setup.
|
||||
|
||||
### DEEPX NPU
|
||||
|
||||
The DEEPX NPU is available in two form factors, and Frigate supports both:
|
||||
|
||||
- **DX-M1** in the M.2 2280 form factor (like an NVMe SSD), for x86 (Intel/AMD) PCs, the Raspberry Pi 5, and other ARM SBCs with an exposed PCIe M.2 slot.
|
||||
- **DX-M1M** on the [Sixfab AI HAT+](https://docs.sixfab.com/docs/ai-hat-plus-raspberry-pi-5-quickstart), a HAT+ board that connects to the Raspberry Pi 5 over PCIe Gen 3 x1.
|
||||
|
||||
Both present the NPU through the same PCIe driver and DX-RT runtime, so the setup below and the detector configuration are identical for either one. Nothing needs to change when moving between them.
|
||||
|
||||
DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
|
||||
|
||||
#### Versions
|
||||
|
||||
A DEEPX install has several separately versioned pieces, and they all have to agree. The driver, the runtime, and the daemon live on the Docker host; Frigate itself carries only the Python bindings, which it downloads on first start:
|
||||
|
||||
| Component | Version | Installed on | Installed by |
|
||||
| -------------- | -------- | ------------ | ------------------------- |
|
||||
| Kernel driver | `v2.6.0` | Host | `user_installation.sh` |
|
||||
| DX-RT runtime | `v3.4.0` | Host | `user_installation.sh` |
|
||||
| NPU firmware | `v2.7.4` | The module | Flashed from the host |
|
||||
| DX-RT bindings | `v3.4.0` | Frigate | Downloaded at first start |
|
||||
|
||||
:::warning
|
||||
|
||||
A version mismatch does not produce a startup error. It typically shows up as inference requests that are accepted but never return a result, so detections simply stop appearing while Frigate looks healthy. If that happens after a Frigate upgrade, check every version in the table before anything else.
|
||||
|
||||
:::
|
||||
|
||||
The installation script installs the DX-RT runtime on the host and enables `dxrt.service`, so the daemon starts at boot and any other program on the host can share the NPU with Frigate. Check the firmware version with `dxrt-cli --status` and update the module if it does not match the table above.
|
||||
|
||||
#### Installation
|
||||
|
||||
The DEEPX kernel driver must be installed on the host rather than in the container, because containers share the host kernel and cannot load kernel modules. Installing it creates the `/dev/dxrt*` device nodes that are passed through to Frigate. The same script installs the DX-RT runtime and enables `dxrt.service`, the daemon that owns the NPU and hands work to it on behalf of Frigate and anything else on the host.
|
||||
|
||||
1. Copy or download [this script](https://github.com/blakeblackshear/frigate/blob/dev/docker/deepx/user_installation.sh).
|
||||
2. Ensure it has execution permissions with `sudo chmod +x user_installation.sh`
|
||||
3. Run the script with `./user_installation.sh`
|
||||
4. **Restart your computer** to complete driver installation.
|
||||
|
||||
Confirm the NPU is visible before continuing:
|
||||
|
||||
```bash
|
||||
ls /dev/dxrt*
|
||||
```
|
||||
|
||||
Then confirm the daemon is running and listening in `/run/dxrt`:
|
||||
|
||||
```bash
|
||||
systemctl is-active dxrt.service
|
||||
ls /run/dxrt/
|
||||
```
|
||||
|
||||
#### Setup
|
||||
|
||||
To set up Frigate, follow the default installation instructions, for example: `ghcr.io/blakeblackshear/frigate:stable`
|
||||
|
||||
#### Docker configuration
|
||||
|
||||
Frigate needs the NPU device node and the directory holding the daemon's socket:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
frigate:
|
||||
devices:
|
||||
- /dev/dxrt0:/dev/dxrt0
|
||||
volumes:
|
||||
- /run/dxrt:/run/dxrt
|
||||
```
|
||||
|
||||
If you can't use Docker Compose, add `--device /dev/dxrt0:/dev/dxrt0 -v /run/dxrt:/run/dxrt` to your `docker run` command.
|
||||
|
||||
Add one `--device` per NPU, contiguously from `/dev/dxrt0`, since the client stops enumerating at the first gap.
|
||||
|
||||
The installation script configures `dxrt.service` to place its socket in `/run/dxrt` through a systemd drop-in. Mounting the directory rather than the socket file means the container sees the new socket after `dxrt.service` is restarted, rather than holding on to a deleted one.
|
||||
|
||||
`dxrtd` listens on an abstract socket as well, but that one does not cross into a container, so Frigate names the filesystem socket through `DXRT_DYNAMIC_IPC_ENDPOINT` on your behalf. Set that variable on the container yourself only if the daemon listens somewhere else, which means you also set it for `dxrtd` through its own systemd drop-in. The script writes `/etc/systemd/system/dxrt.service.d/frigate.conf` for exactly that, and has `dxrt.service` link the socket to `/tmp/dxrt_dynamic_ipc.sock` when it starts, so the host's own `dxrt-cli` and `dxtop` keep finding it at the default path they fall back to.
|
||||
|
||||
:::note
|
||||
|
||||
The DX-RT client exits when `dxrt.service` stops, so restart the Frigate container after restarting `dxrt.service`.
|
||||
|
||||
:::
|
||||
|
||||
The device node is needed as well as the socket, because the client opens the NPU directly even though the daemon arbitrates access. Without it, inference fails with `Device not found`.
|
||||
|
||||
`/dev/shm` does not need sharing.
|
||||
|
||||
The DX-RT python bindings are not shipped in the Frigate image. Frigate downloads them on first start when a DEEPX detector is configured, and caches them under `/config`.
|
||||
|
||||
#### Configuration
|
||||
|
||||
Finally, configure [hardware object detection](/configuration/object_detectors#deepx-npu) to complete the setup.
|
||||
|
||||
### Rockchip platform
|
||||
|
||||
Make sure that you use a linux distribution that comes with the rockchip BSP kernel 5.10 or 6.1 and necessary drivers (especially rkvdec2 and rknpu). To check, enter the following commands:
|
||||
|
||||
Reference in New Issue
Block a user