diff --git a/docs/data/object_detectors_models.yaml b/docs/data/object_detectors_models.yaml index 9d9aa40a1d..44d7471389 100644 --- a/docs/data/object_detectors_models.yaml +++ b/docs/data/object_detectors_models.yaml @@ -894,6 +894,41 @@ deepstack: api_url: http://:/v1/vision/detection type: deepstack api_timeout: 0.1 # seconds +xdna2: + title: AMD XDNA2 + models: + - key: yolov9 + label: YOLOv9 + recommended: true + download: |- + Prepare the model using the frigate-xdna setup instructions linked above. For local YOLO models, Frigate must have access to the same ONNX file bytes as the sidecar. The example below uses YOLOv9-C at 320x320. Frigate+ models may instead use the same `plus://MODEL_ID` in Frigate and the sidecar. + ui: |- + Navigate to **Settings > System > Detectors and model** and select **ZMQ IPC** from the detector type dropdown and click **Add**, then set the endpoint to `tcp://xdna:5555`. Then on the same page, in the **Custom Model** tab, configure: + + | Field | Value | + | ---------------------------------------- | ------------------------------------------ | + | **Custom object detector model path** | `/config/models/yolov9-c-320.onnx` | + | **Label map for custom object detector** | `/labelmap/coco-80.txt` | + | **Object detection model input width** | `320` | + | **Object detection model input height** | `320` | + | **Model Input Pixel Color Format** | `rgb` (Frigate's default value) | + | **Model Input Tensor Shape** | `nchw` | + | **Model Input D Type** | `float` | + | **Object Detection Model Type** | `yolo-generic` | + yaml: |- + detectors: + xdna: + type: zmq + endpoint: tcp://xdna:5555 + + model: + model_type: yolo-generic + width: 320 + height: 320 + input_tensor: nchw + input_dtype: float + path: /config/models/yolov9-c-320.onnx + labelmap_path: /labelmap/coco-80.txt memryx: title: MemryX models: diff --git a/docs/docs/configuration/object_detectors.md b/docs/docs/configuration/object_detectors.md index 602ebfcb85..c200488aba 100644 --- a/docs/docs/configuration/object_detectors.md +++ b/docs/docs/configuration/object_detectors.md @@ -29,6 +29,7 @@ Frigate supports multiple different detectors that work on different types of ha - [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection. - [ONNX](#onnx): ROCm will automatically be detected and used as a detector in the `-rocm` Frigate image when a supported ONNX model is configured. +- [XDNA2](#amd-xdna2): AMD Ryzen AI / XDNA2 NPUs can run object detection through the community-maintained `frigate-xdna` ZMQ sidecar. **Apple Silicon** @@ -508,6 +509,28 @@ To verify that the integration is working correctly, start Frigate and observe t # Community Supported Detectors +## AMD XDNA2 + +AMD Ryzen AI / XDNA2 NPUs can be used through the community-maintained +[frigate-xdna](https://github.com/mitchins/frigate-xdna) detector sidecar. +The sidecar runs separately from Frigate and connects using Frigate's ZMQ +detector interface. + +Currently qualified on **Ryzen AI Max 300 / Strix Halo**. Other XDNA2 devices +are not yet qualified; XDNA1 is unsupported. + +Follow the frigate-xdna setup instructions to prepare and start the sidecar +before starting Frigate. + +### Configuration {#configuration-xdna2} + +Using the detector config below will connect Frigate to the sidecar: + + + +The example assumes Frigate and the sidecar share a Docker network where the +sidecar is named `xdna`. + ## MemryX MX3 This detector is available for use with the MemryX MX3 accelerator M.2 module. Frigate supports the MX3 on compatible hardware platforms, providing efficient and high-performance object detection. diff --git a/docs/docs/frigate/hardware.md b/docs/docs/frigate/hardware.md index 1df3bb8368..d38d065443 100644 --- a/docs/docs/frigate/hardware.md +++ b/docs/docs/frigate/hardware.md @@ -70,6 +70,9 @@ Frigate supports multiple different detectors that work on different types of ha - [ROCm](#rocm---amd-gpu): ROCm can run on AMD Discrete GPUs to provide efficient object detection - [Supports limited model architectures](../../configuration/object_detectors#amdrocm-gpu-detector) - Runs best on discrete AMD GPUs +- [XDNA2 (Ryzen AI)](#amd-xdna2): AMD XDNA2 NPU (sub-watt power AI/ML processor separate to the GPU) inside Strix and other "AI" branded AMD platforms + - Has only been tested with YOLOv9, in theory other graphs may be compiled too. + - Runs via ZMQ proxy which adds some latency, only recommended for local connection **Apple Silicon** @@ -296,6 +299,32 @@ The inference time of a rk3588 with all 3 cores enabled is typically 25-30 ms fo | ---------------- | ----------------------------------- | | yolov9-tiny | ~ 4 ms | +### AMD Ryzen AI / XDNA2 + +Frigate supports AMD XDNA2 NPUs through the community-maintained +frigate-xdna ZMQ sidecar. It works with stock Frigate and supports +Frigate+ models or compatible local YOLO ONNX models. Models are compiled +once on the target system and cached for subsequent use. + +Currently qualified on **Ryzen AI Max 300 / Strix Halo**. Other XDNA2 +devices are not yet qualified; XDNA1 is unsupported. + +Measured YOLOv9 detector latency on Strix Halo: + +| Model | 320 | 640 | +| ----- | ---: | ---: | +| YOLOv9-T | ~7.4 ms | unsupported | +| YOLOv9-S | ~9.0 ms | ~20.0 ms | +| YOLOv9-M | ~13.1 ms | ~34.4 ms | +| YOLOv9-C | ~14.1 ms | ~35.2 ms | +| YOLOv9-E | ~69.4 ms | ~224.8 ms | + +**YOLOv9-C at 320 is the recommended quality/performance balance.** +C at 640 is also usable where the lower throughput is acceptable. + +Setup, model preparation, and compatibility details are available +[in the frigate-xdna documentation](https://github.com/mitchins/frigate-xdna). + ## What does Frigate use the CPU for and what does it use a detector for? (ELI5 Version) This is taken from a [user question on reddit](https://www.reddit.com/r/homeassistant/comments/q8mgau/comment/hgqbxh5/?utm_source=share&utm_medium=web2x&context=3). Modified slightly for clarity.