administrator/frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2026-10-23 07:02:48 +03:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

feat(detectors): add Qualcomm Hexagon NPU support (community board)

Browse Source 
Add a community-supported board build for Qualcomm SoCs with a Hexagon
NPU, accelerating TFLite object detection through the QNN TFLite delegate
on the Hexagon Tensor Processor (HTP).

Supported boards:
- IQ9100 (IQ-9075 EVK)
- QCS6490 (RB3 Gen 2 Vision Kit / Rubik Pi 3)

- detector: new `qualcomm_tfl` plugin loading libQnnTFLiteDelegate.so on
  the HTP backend, reusing the shared TFLite delegate helpers
- docker: `docker/qualcomm` board build (Dockerfile, qualcomm.hcl,
  qualcomm.mk) producing the arm64 `-qualcomm` image
- cdi: per-board Container Device Interface descriptors and an install
  helper exposing the NPU device nodes and QNN libraries to the container
- ci/codeowners: register the qualcomm build target and code owner
- docs: installation, hardware, and object detector documentation, using
  the shared model config dropdown component
- i18n: add the Qualcomm detector label and description to the generated
  en config locale

Signed-off-by: Rami Mouro <rmouro@qti.qualcomm.com>
This commit is contained in:
Rami Mouro
2026-06-29 19:44:04 -06:00
parent ea131e1663
commit e769ba7a8c
14 changed files with 3661 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/data/object_detectors_models.json
+13
View File
@@ -265,6 +265,19 @@
}
]
},
"qualcomm": {
"title": "Qualcomm",
"models": [
{
"key": "ssd_mobiledet",
"label": "SSD MobileDet",
"recommended": true,
"download": "The default SSD MobileDet TFLite model (`/cpu_model.tflite`) bundled with the Frigate container is used automatically. This model is INT8 quantized and compatible with the QNN HTP backend.\n\nOnly `.tflite` models are supported with this detector. A custom model must be INT8 quantized for the Hexagon NPU.",
"ui": "Navigate to **Settings > System > Detectors and model** and select **Qualcomm** from the detector type dropdown and click **Add**. The bundled SSD MobileDet model is used automatically.\n\nTo use a custom model, configure in the **Custom Model** tab:\n\n| Field | Value |\n| ---------------------------------------- | ---------------------------------- |\n| **Custom object detector model path** | `/config/your_custom_model.tflite` |\n| **Object detection model input width** | `320` |\n| **Object detection model input height** | `320` |\n| **Label map for custom object detector** | `/labelmap/coco-80.txt` |",
"yaml": "detectors:\n qualcomm_npu:\n type: qualcomm_tfl"
}
]
},
"rknn": {
"title": "RKNN",
"models": [
docs/docs/configuration/object_detectors.md
+19
View File
@@ -53,6 +53,10 @@ Frigate supports multiple different detectors that work on different types of ha
- [RKNN](#rockchip-platform): RKNN models can run on Rockchip devices with included NPUs.
**Qualcomm** <CommunityBadge />
- [Qualcomm](#qualcomm): TFLite models can run on Qualcomm SoCs with a Hexagon NPU (e.g. IQ9100, QCS6490) via the QNN TFLite delegate.
**Synaptics** <CommunityBadge />
- [Synaptics](#synaptics): synap models can run on Synaptics devices(e.g astra machina) with included NPUs.
@@ -652,6 +656,21 @@ When configuring the Synap detector, you have to specify the model: a local **pa
<ModelConfigDropdown detectorTitle="Synaptics" models={objectDetectorsModels.synaptics.models} />
## Qualcomm
Hardware accelerated object detection is supported on the following Qualcomm SoCs with a Hexagon NPU:
- IQ9100 / IQ-9075 EVK
- QCS6490 / RB3 Gen 2 Vision Kit / Rubik Pi 3
This implementation uses the QNN TFLite delegate (`libQnnTFLiteDelegate.so`) to accelerate TFLite model inference on the Qualcomm Hexagon Tensor Processor (HTP / NPU). The delegate library and device drivers are provided by the host operating system and made available to the container via [CDI (Container Device Interface)](https://docs.docker.com/build/building/cdi/).
See the [installation docs](../frigate/installation.md#qualcomm) for information on setting up CDI and configuring the hardware.
### Configuration
<ModelConfigDropdown detectorTitle="Qualcomm" models={objectDetectorsModels.qualcomm.models} />
## Rockchip platform
Hardware accelerated object detection is supported on the following SoCs:
docs/docs/frigate/hardware.md
+15
View File
@@ -107,6 +107,10 @@ Frigate supports multiple different detectors that work on different types of ha
- [AXEngine](#axera): axera models can run on AXERA NPUs via AXEngine, delivering highly efficient object detection.
**Qualcomm** <CommunityBadge />
- [Qualcomm](#qualcomm): TFLite models can run on Qualcomm SoCs with a Hexagon NPU (e.g. IQ9100, QCS6490) via the QNN TFLite delegate.
:::
### Hailo-8
@@ -295,6 +299,17 @@ The inference time of a rk3588 with all 3 cores enabled is typically 25-30 ms fo
| Name | AXERA AX650N/AX8850N Inference Time |
| ---------------- | ----------------------------------- |
| yolov9-tiny | ~ 4 ms |
### Qualcomm
Frigate supports hardware object detection on Qualcomm SoCs with a Hexagon NPU, including the IQ9100 and QCS6490 (RB3 Gen 2). The QNN TFLite delegate is used to accelerate inference on the HTP (Hexagon Tensor Processor).
A single detector is typically sufficient for multiple camera streams. The default model is **SSD MobileDet** (INT8 quantized).
| Name | IQ9100 Inference Time | QCS6490 Inference Time |
| ------------- | --------------------- | ---------------------- |
| ssd_mobiledet | ~ 0.8 ms | ~ 5.55 ms |
Detailed information is available [in the detector docs](/configuration/object_detectors#qualcomm).
## What does Frigate use the CPU for and what does it use a detector for? (ELI5 Version)
docs/docs/frigate/installation.md
+112
View File
@@ -472,6 +472,117 @@ If you are using `docker run`, add this option to your command `--device /dev/ax
#### Configuration
Finally, configure [hardware object detection](/configuration/object_detectors#axera) to complete the setup.
### Qualcomm
Hardware accelerated object detection is supported on the following Qualcomm SoCs:
| Board | SoC | CDI Config File |
| ----- | --- | --------------- |
| IQ-9075 EVK | IQ9100 | `cdi-hw-acc-9100.json` |
| RB3 Gen 2 Vision Kit / Rubik Pi 3 | QCS6490 | `cdi-hw-acc-6490.json` |
The Qualcomm integration uses [CDI (Container Device Interface)](https://docs.docker.com/build/building/cdi/) to provide the container with access to the NPU device nodes and the QNN delegate libraries from the host.
#### Prerequisites
- **Qualcomm Linux BSP**: Your board must be running the official Qualcomm Linux Board Support Package image. The QNN runtime libraries (including `libQnnTFLiteDelegate.so` and `libQnnHtp.so`) and NPU device nodes (`/dev/fastrpc-cdsp`) are provided by the BSP and are bind-mounted into the container via CDI.
- **Docker 25.0+**: CDI device support requires Docker 25.0 or later. Check with `docker --version`.
- **arm64 architecture**: The Qualcomm Docker image is built for `linux/arm64` only. If building locally, you must run the build on the target board itself.
#### CDI Installation
1. Download or copy the CDI setup files from the [Frigate repository](https://github.com/blakeblackshear/frigate/tree/dev/docker/qualcomm/cdi).
2. Install the CDI configuration for your board:
```bash
# IQ9100 / IQ-9075 EVK
sudo python3 install_cdi.py --file cdi-hw-acc-9100.json
# QCS6490 / RB3 Gen 2 Vision Kit / Rubik Pi 3
sudo python3 install_cdi.py --file cdi-hw-acc-6490.json
```
This writes a CDI descriptor to `/etc/cdi/cdi-hw-acc.json`. The script will automatically skip any host paths that do not exist on your system.
3. Verify CDI is set up by checking the file exists:
```bash
ls -l /etc/cdi/cdi-hw-acc.json
```
#### Setup
Follow Frigate's default installation instructions, but use a docker image with `-qualcomm` suffix, for example `ghcr.io/blakeblackshear/frigate:stable-qualcomm`.
:::note
The pre-built `stable-qualcomm` image is not yet published. Until it is available, you must build the image locally:
```bash
git clone https://github.com/blakeblackshear/frigate.git
cd frigate
make local-qualcomm
```
This builds `frigate:latest-qualcomm` on your device. Use `frigate:latest-qualcomm` as the image name in the examples below instead of the `ghcr.io` URL.
:::
Grant Docker access to your Qualcomm hardware by passing the CDI device. In your `docker-compose.yml`:
```yaml
services:
frigate:
container_name: frigate
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:stable-qualcomm
devices:
- qualcomm.com/device=cdi-hw-acc
volumes:
- /etc/localtime:/etc/localtime:ro
- /path/to/your/config:/config
- /path/to/your/storage:/media/frigate
- type: tmpfs
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "8971:8971"
- "8554:8554"
- "8555:8555/tcp"
- "8555:8555/udp"
```
If using `docker run`, pass the CDI device with:
```bash
docker run -d \
--name frigate \
--restart=unless-stopped \
--device qualcomm.com/device=cdi-hw-acc \
--mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \
--shm-size=256m \
-v /path/to/your/storage:/media/frigate \
-v /path/to/your/config:/config \
-v /etc/localtime:/etc/localtime:ro \
-e FRIGATE_RTSP_PASSWORD='password' \
-p 8971:8971 \
-p 8554:8554 \
-p 8555:8555/tcp \
-p 8555:8555/udp \
ghcr.io/blakeblackshear/frigate:stable-qualcomm
```
:::tip
Unlike other hardware integrations that pass individual `/dev/` device nodes, the Qualcomm CDI approach bundles all required device nodes, library bind mounts, and environment variables into a single `--device qualcomm.com/device=cdi-hw-acc` flag.
:::
#### Configuration
Next, you should configure [hardware object detection](/configuration/object_detectors#qualcomm).
## Docker
@@ -558,6 +669,7 @@ The community supported docker image tags for the current stable version are:
- `stable-tensorrt-jp6` - Frigate build optimized for Nvidia Jetson devices running Jetpack 6
- `stable-rk` - Frigate build for SBCs with Rockchip SoC
- `stable-qualcomm` - Frigate build for Qualcomm SoCs with Hexagon NPU (IQ9100, QCS6490)
## Home Assistant App