Feat/deepx npu detector (#24336)
CI / AMD64 Build (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Smoke Test (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s

* feat(deepx): add DEEPX NPU detector and runtime integration.

* feat(deepx): enforce model_format requirement when ppu is enabled and add integrity checks for driver installation

* Update frigate/detectors/plugins/deepx.py

Public method lacks docstring

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Refactor DEEPX detector tests, support SSD and DAMO-YOLO

* feat(deepx): add anchor-free output decoding and corresponding tests

* Add tests and updates for DEEPX detector and refactor DEEPX accelerator code structure.

* fix: enhance model type validation and update documentation for DEEPX detector

* fix: add support for customizable score and NMS thresholds

* refactor: infer YOLO layout from the model, drop per-detector options and the dxrtd placeholder

* fix: keep only the anchor-free PPU verdict, re-read anchor-based each frame

* Update latency data for DEEPX NPU

* Expanding PPU support for DEEPX and set yolo-generic as default.

* enhance scale count resolution logic

* Extend PPU layout handling and YOLOX support to DEEPX detector

* fix: assume the largest PPU anchor table when the .dxnn has no layout

* Improve PPU decoding and introduce strides handling

* Improve PPU scope and fix box format mismatch

* Fix unnamed node issue that breaks traversal

* fix: update object detection model type description to remove outdated architecture

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
This commit is contained in:
A. Ahmet
2026-09-21 07:50:44 -05:00
committed by GitHub
co-authored by greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
parent 52f50a7396
commit af0ba19196
11 changed files with 2467 additions and 0 deletions
@@ -24,6 +24,7 @@ Frigate supports multiple different detectors that work on different types of ha
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
- [Hailo](#hailo): The Hailo-8, Hailo-8L and Hailo-8R AI Acceleration modules are available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, offering broad compatibility across various platforms.
**AMD**
@@ -612,6 +613,63 @@ For detailed instructions on compiling models, refer to the [MemryX Compiler](ht
---
## DEEPX NPU
This detector is available for use with the DEEPX NPU, both the DX-M1 M.2 module and the DX-M1M on the Sixfab AI HAT+ for the Raspberry Pi 5. The configuration below applies unchanged to either form factor. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
See the [installation docs](../frigate/installation.md#deepx-npu) for information on installing the DEEPX kernel driver and runtime on the host and passing the NPU through to the container.
To run a model on a DEEPX NPU, list a `deepx` device on that model.
:::info
The DX-RT Python bindings are not part of the Frigate image. They are downloaded and installed into `/config/.local` the first time a DEEPX device is configured, verified against pinned checksums, and updated automatically when a Frigate release pins a new version. If the container has no internet access, see [Detector runtimes](/frigate/network_requirements#detector-runtimes) for how to provide the files yourself.
:::
### Configuration {#configuration-deepx}
<ModelConfigDropdown detectorTitle="DEEPX" models={objectDetectorsModels.deepx.models} />
Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Two model types are supported:
- `yolo-generic` for YOLO object detection models, the recommended default. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do models compiled with DEEPX's Post-Processing Unit (PPU) support.
- `yolox` for YOLOX models compiled without PPU support, whose raw head needs Frigate's YOLOX decoder. A YOLOX model compiled with PPU support works under either `yolox` or `yolo-generic`.
The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler. The recommended starting point is `yolox-s_640x640_ppu.dxnn`, the fastest ModelZoo model measured through Frigate:
```yaml
models:
- devices:
- deepx:PCIe:0
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: yolo-generic
width: 640
height: 640
```
For PPU models, use a `.dxnn` compiled with DX-COM 2.4.0 or later. Frigate reads the PPU head layout the compiler writes into the file and refuses to load a PPU model without it.
`model_type` must be set to `yolo-generic` or `yolox` to match the model; `yolo-generic` is the recommended default unless the model is a raw YOLOX export. Frigate defaults it to `ssd`, which this detector does not support, so the detector refuses to start on a model that leaves it unset.
`width` and `height` must match the resolution the model was compiled for. Quantization parameters are baked into the `.dxnn` file at compile time, so no normalization is applied on the host and Frigate's default `input_tensor`, `input_pixel_format`, and `input_dtype` values do not need to be overridden.
A DEEPX device is `PCIe:<index>`, as reported on the detector settings page. The NPU daemon multiplexes across processes, so the same device may be listed more than once to run additional inference processes against it:
```yaml
models:
- devices:
- deepx:PCIe:0
- deepx:PCIe:0
```
#### Label maps
The object detection models in the DEEPX ModelZoo are trained on the standard 80-class COCO label set, so `labelmap_path` must be set to `/labelmap/coco-80.txt`. Frigate's default label map uses an extended 91-class COCO scheme, and leaving it in place will cause detections to be reported as the wrong object type. For `yolo-generic` models the label map is also what the detector uses to tell the output layout, so a label map with the wrong number of classes is reported as an error at startup.
---
## NVidia TensorRT Detector
Nvidia Jetson devices may be used for object detection using the TensorRT libraries. Due to the size of the additional libraries, this detector is only provided in images with the `-tensorrt-jp6` tag suffix, e.g. `ghcr.io/blakeblackshear/frigate:stable-tensorrt-jp6`. This detector is designed to work with Yolo models for object detection.