mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-10-21 06:02:49 +03:00
Patryk OrwatGitHubPatryk Watkowskigreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / AMD64 Smoke Test (push) Blocked by required conditions
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions
* Add Axelera Metis NPU object detector plugin Runs Frigate inference on the Axelera Metis NPU through the vendor's low-level axelera.runtime API with pip-installed, sha256-pinned wheels from Axelera's public artifactory (self-contained: auditwheel-repaired libs + bundled firmware). - async send_input/receive_output contract (FIFO pairing, overload emits zero rows instead of raising, frames are copied out of camera shm) - generic YOLO-family decoder driven by the compiled model's manifest.json (padded input geometry, dequant params, baked postprocess graph via onnx) - labels come from the compiled model's model_info.json; empty labelmaps fail loudly at init * object_detection: async runner path for the axelera detector * detectors: detect_axelera probe (/dev/metis*) + device node init * stats: Axelera Metis board temperature poller The card exposes no utilization counter usable next to a live detector, so the poller reports the board-controller temperature read via axcmd (cached 5s; the read must not race detector init). * docs: Axelera Metis detector (install, config, model catalog, downloads) * axelera: expose Metis colon-name device node and image deps for runtime enumeration The Axelera runtime enumerates Metis cards via the colon-separated node name created by the vendor udev rule on the host (e.g. /dev/metis-0:1:0) and probes kernel state with lsmod/lspci. Containers only receive the raw /dev/metis-0-1-0 node, so the detector never found the card. - init-devices: create /dev/metis-BB:DD:F aliases for each Metis node - install_deps: add kmod and pciutils so the runtime's driver check works * axelera: review round 2 (downloader, stop release, pixel format, docs size, per-class NMS) - download model zips through ModelDownloader (atomic .part rename; a truncated download no longer poisons the cache) - stop(): release the Context (releases connection/model/instance); the previous instance.free() call never existed in the runtime API and was silently swallowed - honor input_pixel_format: flip to RGB planes only when Frigate delivers BGR (input_pixel_format: bgr); with rgb the frame passes through as-is - docs/catalog: width/height = compiled 640x640 input (square regions no longer get squashed twice), document input_pixel_format: bgr - NMS via xyxy_to_xywh_for_nms + per-class cv2.dnn.NMSBoxes, matching the other detector plugins instead of a hand-rolled class-agnostic pass * axelera: report Metis temperature only on the NPU card The same board readout was surfaced twice on the System page: once as the detector temperature (get_hardware_temperatures) and once on the NPU card (npu_usages). Keep the NPU card, where the value belongs. * axelera: report AIPU utilization as the detector run duty cycle The runtime exposes no device-side usage counter (mvm_utilisation is a configuration limit, not a readout), so the NPU card rendered empty. ModelInstance.run blocks until the AIPU completes: the detector process now times each run and publishes busy/wall over a 2s window through a small file under the tmp cache dir; the stats poller reads it and sets the npu key. A stale file (detector idle/restarting) reports nothing rather than a stale number. * axelera: cut hot-path allocations in preprocess and dequantise Preprocess reused three fresh arrays per frame (resize output, quantised copy, strided channel flip at write time) and refilled the whole padded input buffer. Now resize writes into a scratch with cv2 dst=, the BGR->RGB swap is a cvtColor into scratch (no strided copy), the int8 quantise runs with out=, and only the border strips the content rect does not cover get memset. Measured 39% faster on the 704x576 -> 640x640 path. Dequantise allocated two fp32 temporaries per head (astype then the dequant expression) before the transpose copy. copyto now casts int8->fp32 while transposing straight into a reused fp32 feed buffer, and the zp/scale math runs in place. Measured 78% faster for the YOLOX-S head set, outputs bit-identical (int8->fp32 cast is lossless). Argmax+gather replaces max+argmax (one class-axis reduction instead of two), and the postprocess session pins intra_op_num_threads=1: one frame is in flight through the decoder, extra ORT threads only add context switches. * axelera: publish the run duty cycle through a shared Value Replace the tmp-file duty-cycle handoff (and the FRIGATE_TMP env var nothing else reads) with a third multiprocessing.Value alongside avg_inference_speed and detection_start: ObjectDetectProcess creates it, AsyncDetectorRunner passes it to the detector, and the plugin publishes busy/wall of the blocking ModelInstance.run over a 2s window, mirroring how memryx receives its shared state. No filesystem path, no staleness window, no atomic rename, and stats/hardware.py no longer imports the plugin module to find the file. The producer also ticks the window while the queue is empty, so the duty cycle falls to 0 when the card idles instead of sticking at its last value. Name it what it is: host-side occupancy of the run call, including DMA and driver time, not an AIPU core counter (the runtime exposes no device-side one). The NPU entry now reads temperature from the board controller plus the detector-reported run duty cycle folded in where the detector processes are in reach. * axelera: enumerate devices explicitly and fail with an actionable error Context.list_devices() keys off the lspci BDF and needs the colon-form node to be present, so surface an empty enumeration at init with the fix-it hint instead of a bare device_connect(None) ConnectionError several frames deeper. Uses the DeviceInfo the runtime enumerated rather than relying on the pick-first implicit path. * axelera: deliver every accepted result under backpressure, reset duty cycle on restart Greptile P1: _push_result dropped a completion when the result queue was full, stranding the runner's send_times entry and the input SHM mapping (1 submission : 1 result contract). Replace the drop with a stop-aware blocking put so an accepted submission always produces exactly one delivered completion. Greptile P2: start_or_restart did not clear run_duty_cycle when starting a replacement process, so stats could show the previous process's duty cycle while the new detector was still initializing. Reset it to the -1.0 sentinel alongside detection_start, inside the async branch. Greptile P2 (offline docs): list axelera among the detector types whose pre-seeded runtime files live under /config/model_cache/runtimes/<type>. * axelera: reviewer round 3 (plugin-owned enumeration tools, generic DEVICE_ALIASES, generic duty-cycle fold) install_deps.sh: drop kmod/pciutils from the base image. The runtime's list_devices() only greps popen'd lspci/lsmod output over /sys and /proc, so the plugin now installs shell stand-ins in its own /config/model_cache/axelera/bin and prepends it to PATH when the real tools are missing (hardware-verified: the stand-in lspci prints the Device 1f9d:<id> line the runtime matches, the lsmod shape matches /proc/modules). init-devices: replace the Metis-keyed alias block with a generic DEVICE_ALIASES env var (comma separated alias=target pairs, next to DEVICE_ACL_PATHS). Docker rejects colon-named nodes in --device AND in bind mounts (probed: 'too many colons' on both the long volume form and the mount form), so the in-container symlink stays necessary; docs carry the pair users must set. stats/util.py: fold any detector's published run_duty_cycle into the matching npu_usages entry generically; _axelera_run_duty_cycle and the per-vendor branch are gone. plugin: the no-device RuntimeError now names the raw dash-form node (--device rejects colons, users can only pass the raw node) and points at DEVICE_ALIASES for the colon alias. * axelera: reviewer round 4 (aggregate shared-type duty cycles, american spellings) stats/util.py: several detector processes can share one hardware type (e.g. two axelera cards on one host). Iterating and assigning made the displayed NPU usage depend on dict order, last writer wins. Collect the published windows per type and write the busiest one, matching what the removed per-vendor helper did with max(). plugin: the remaining British spellings (dequantise family) corrected to the American forms the repo requires. plugin: model path resolution follows hailo8l's single-string pattern (URL or local path or preset name in one value), the download uses urllib directly like memryx, and _find_model_json now raises on an ambiguous multi-model directory instead of returning whichever sorts first. plugin: AxeleraDetector exposes shutdown() (the async contract hook base.py calls) instead of close(); __del__ guards a partially initialised instance. * axelera: download URL models through the atomic ModelDownloader * axelera: bring the detector unit tests in tree * init-devices: create DEVICE_ALIASES before the ACL-mode exits An alias is not an ACL grant: users running with FRIGATE_RUN_AS_ROOT or FRIGATE_DEVICE_ACLS=false still need the colon-form node the runtime enumerates, so create the aliases right after the euid check. Document DEVICE_ALIASES next to DEVICE_ACL_PATHS and note it cannot work under docker's user: mode (the container cannot write to /dev). * detectors: cover the Axelera Metis probe in test_detector_hardware * Update docs/docs/configuration/non_root.md Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * axelera: address the 2026-10-12 review round - hash directory models deterministically in compute_model_hash instead of raising IsADirectoryError at startup - keep canonical model labels addressable after user renames in the labelmap lookup (two classes can share one renamed name) - key the URL model cache by source and archive content, extract replaced archives into fresh directories, prune superseded variants conservatively - keep publishing the run duty cycle when the hardware poll has no entry (duty-only npu_usages entry instead of a hidden detector) - document the user: mode device reach instead of host-only alias creation * axelera: prune only generated cache entries A fresh download matched user-provided layouts (the docs-recommended <name>/ directory, <name>.zip archives) and look-alike model names (yolox.zip vs yolox-s-*) by prefix. Pruning is now gated on the exact generated identity pattern <name>-<source>-<32 hex digits>, so downloads never touch user-provided directories, other models, or other sources. * axelera: narrow the model path design to memryx-style zips - path is always a .zip of the compiled output (local or http(s) URL), extracted fresh on every start under a lock, like the memryx custom-model flow - _decode returns the raw class index and Frigate's labelmap names it; _build_label_lookup and _read_model_labels are gone, and the YOLOX objectness check keys off model_type - scheme-less URL support and the quantized/manifest.json fallback are dropped - width/height must equal the compiled input size; init raises otherwise instead of letterboxing - compute_model_hash is back to the unchanged file-only hashing; the .extracted marker, source/content cache-name hashes and _prune_model_cache are removed * Update frigate/test/test_axelera_detector.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * axelera: drop the run duty cycle, match the hailo temperature flow - no other accelerator reports NPU usage and the published value was time blocked in run(), not device utilization from the vendor API; removing it leaves the runner selection as the only core change - the board temperature moves to a get_hardware_temperatures branch like hailo, so it renders on the detector card; _poll_axelera, its POLLERS entry and the _scan_detectors line are gone - axelera_runtime2 is not installed: axelera.runtime imports it only inside a try for its separate new-runtime API, and the low-level API the plugin uses (Context, load_model, device_connect, load_model_instance) lives in axelera.runtime.objects * axelera: follow-ups from the 2026-10-13 review round - the offline-setup table no longer lists the axelera_runtime2 wheels, which are not installed - a failed board temperature read starts a 60s cooldown so an unresponsive axcmd does not delay every stats update * axelera: drop the model-hash test, fix the model.json error messages * axelera: drop DEVICE_ALIASES, document the CDI device spec * axelera: use ln -sf in the CDI hook example --------- Co-authored-by: Patryk Watkowski <patrykorwat@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
1312 lines
83 KiB
YAML
1312 lines
83 KiB
YAML
edgeTPU:
|
|
title: EdgeTPU
|
|
models:
|
|
- key: mobiledet
|
|
label: Mobiledet
|
|
recommended: true
|
|
download: A TensorFlow Lite model is provided in the container at `/edgetpu_model.tflite` and is used by this detector type by default. To provide your own model, bind mount the file into the container and provide the path with a model's `path`.
|
|
ui: Navigate to **Settings > System > Detection models** and select **Coral EdgeTPU (USB)** from the **Hardware** dropdown.
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- edgetpu:usb
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: false
|
|
download: "[Download the model](https://github.com/dbro/frigate-detector-edgetpu-yolo9/releases/download/v1.0/yolov9-s-relu6-best_320_int8_edgetpu.tflite), bind mount the file into the container, and provide the path with a model's `path`. Note that the linked model requires a 17-label [labelmap file](https://raw.githubusercontent.com/dbro/frigate-detector-edgetpu-yolo9/refs/heads/main/labels-coco17.txt) that includes only 17 COCO classes."
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Coral EdgeTPU (USB)** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure the model settings:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ----------------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolov9-s-relu6-best_320_int8_edgetpu.tflite` |
|
|
| **Label map for custom object detector** | `/config/labels-coco17.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize of the model) |
|
|
| **Object detection model input height** | `320` (should match the imgsize of the model) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- edgetpu:usb
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize of the model, typically 320
|
|
height: 320 # <--- should match the imgsize of the model, typically 320
|
|
path: /config/model_cache/yolov9-s-relu6-best_320_int8_edgetpu.tflite
|
|
labelmap_path: /config/labels-coco17.txt
|
|
hailo:
|
|
title: Hailo
|
|
models:
|
|
- key: yolo
|
|
label: YOLO
|
|
recommended: true
|
|
download: If no custom model path or URL is provided, the Hailo detector automatically downloads the default model (YOLOv6n) from the Hailo Model Zoo on first startup, choosing the build that matches the attached device. Once cached under `/config/model_cache`, the model works fully offline.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Hailo** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure the model settings:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ----------------------- |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `rgb` |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
|
|
The detector automatically selects the default model based on your hardware. Optionally, specify a local model path or URL to override.
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- hailo:PCIe
|
|
width: 320
|
|
height: 320
|
|
input_tensor: nhwc
|
|
input_pixel_format: rgb
|
|
input_dtype: int
|
|
model_type: yolo-generic
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
|
|
# The detector automatically selects the default model based on your hardware:
|
|
# - For Hailo-8 hardware: YOLOv6n (default: yolov6n.hef)
|
|
# - For Hailo-8L hardware: YOLOv6n (default: yolov6n.hef)
|
|
#
|
|
# Optionally, you can specify a local model path to override the default.
|
|
# If a local path is provided and the file exists, it will be used instead of downloading.
|
|
# Example:
|
|
# path: /config/model_cache/hailo/yolov6n.hef
|
|
#
|
|
# You can also override using a custom URL:
|
|
# path: https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.14.0/hailo8/yolov6n.hef
|
|
# just make sure to give it the write configuration based on the model
|
|
- key: ssd
|
|
label: SSD MobileNet v1
|
|
recommended: false
|
|
download: For SSD-based models, provide either a model path or URL to your compiled SSD model. The integration will first check the local path before downloading if necessary. The model file is cached under `/config/model_cache/hailo`.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Hailo** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure the model settings:
|
|
|
|
| Field | Value |
|
|
| --------------------------------------- | ------ |
|
|
| **Object detection model input width** | `300` |
|
|
| **Object detection model input height** | `300` |
|
|
| **Model Input Pixel Color Format** | `rgb` |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `ssd` |
|
|
|
|
Specify the local model path or URL for SSD MobileNet v1.
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- hailo:PCIe
|
|
width: 300
|
|
height: 300
|
|
input_tensor: nhwc
|
|
input_pixel_format: rgb
|
|
model_type: ssd
|
|
# Specify the local model path (if available) or URL for SSD MobileNet v1.
|
|
# Example with a local path:
|
|
# path: /config/model_cache/h8l_cache/ssd_mobilenet_v1.hef
|
|
#
|
|
# Or override using a custom URL:
|
|
# path: https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.14.0/hailo8l/ssd_mobilenet_v1.hef
|
|
openvino:
|
|
title: OpenVINO
|
|
models:
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: true
|
|
download: |-
|
|
YOLOv9 model can be exported as ONNX using the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=t` and `IMG_SIZE=320` in the first line to the [model size](https://github.com/WongKinYiu/yolov9#performance) you would like to convert (available model sizes are `t`, `s`, `m`, `c`, and `e`, common image sizes are `320` and `640`).
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=t --build-arg IMG_SIZE=320 --output . -f- <<'EOF'
|
|
FROM python:3.11 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y cmake libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
|
|
WORKDIR /yolov9
|
|
ADD https://github.com/WongKinYiu/yolov9.git .
|
|
RUN uv pip install --system -r requirements.txt
|
|
RUN uv pip install --system onnx==1.18.0 onnxruntime onnx-simplifier==0.4.* onnxscript
|
|
ARG MODEL_SIZE
|
|
ARG IMG_SIZE
|
|
ADD https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-${MODEL_SIZE}-converted.pt yolov9-${MODEL_SIZE}.pt
|
|
RUN sed -i "s/ckpt = torch.load(attempt_download(w), map_location='cpu')/ckpt = torch.load(attempt_download(w), map_location='cpu', weights_only=False)/g" models/experimental.py
|
|
RUN python3 export.py --weights ./yolov9-${MODEL_SIZE}.pt --imgsz ${IMG_SIZE} --simplify --include onnx
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
ARG IMG_SIZE
|
|
COPY --from=build /yolov9/yolov9-${MODEL_SIZE}.onnx /yolov9-${MODEL_SIZE}-${IMG_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Intel GPU** (or **Intel NPU**) from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolo.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `320` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:GPU
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize set during model export
|
|
height: 320 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/yolo.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: ssd
|
|
label: SSDLite MobileNet v2
|
|
recommended: false
|
|
download: An OpenVINO model is provided in the container at `/openvino-model/ssdlite_mobilenet_v2.xml` and is used by this detector type by default. The model comes from Intel's Open Model Zoo [SSDLite MobileNet V2](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssdlite_mobilenet_v2) and is converted to an FP16 precision IR model.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Intel GPU** (or **Intel NPU**) from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------ |
|
|
| **Custom object detector model path** | `/openvino-model/ssdlite_mobilenet_v2.xml` |
|
|
| **Label map for custom object detector** | `/openvino-model/coco_91cl_bkgr.txt` |
|
|
| **Object detection model input width** | `300` |
|
|
| **Object detection model input height** | `300` |
|
|
| **Model Input Pixel Color Format** | `bgr` |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `ssd` (Frigate's default value) |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:GPU
|
|
width: 300
|
|
height: 300
|
|
input_tensor: nhwc
|
|
input_pixel_format: bgr
|
|
path: /openvino-model/ssdlite_mobilenet_v2.xml
|
|
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
|
|
- key: yolo-legacy
|
|
label: YOLO (v3, v4, v7)
|
|
recommended: false
|
|
download: |-
|
|
To export as ONNX:
|
|
|
|
```sh
|
|
git clone https://github.com/NateMeyer/tensorrt_demos
|
|
cd tensorrt_demos/yolo
|
|
./download_yolo.sh
|
|
python3 yolo_to_onnx.py -m yolov7-320
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Intel GPU** (or **Intel NPU**) from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolo.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `320` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:GPU
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize set during model export
|
|
height: 320 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/yolo.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolonas
|
|
label: YOLO-NAS
|
|
recommended: false
|
|
download: |-
|
|
You can build and download a compatible model with pre-trained weights using [this notebook](https://github.com/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb) [](https://colab.research.google.com/github/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb) which can be run directly in [Google Colab](https://colab.research.google.com/github/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb).
|
|
|
|
:::warning
|
|
|
|
The pre-trained YOLO-NAS weights from DeciAI are subject to their license and can't be used commercially. For more information, see: https://docs.deci.ai/super-gradients/latest/LICENSE.YOLONAS.html
|
|
|
|
:::
|
|
|
|
The input image size in this notebook is set to 320x320. This results in lower CPU usage and faster inference times without impacting performance in most cases due to the way Frigate crops video frames to areas of interest before running detection. The notebook and config can be updated to 640x640 if desired.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Intel GPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/yolo_nas_s.onnx` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match whatever was set in notebook) |
|
|
| **Object detection model input height** | `320` (should match whatever was set in notebook) |
|
|
| **Model Input Pixel Color Format** | `bgr` |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolonas` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:GPU
|
|
model_type: yolonas
|
|
width: 320 # <--- should match whatever was set in notebook
|
|
height: 320 # <--- should match whatever was set in notebook
|
|
input_tensor: nchw
|
|
input_pixel_format: bgr
|
|
path: /config/yolo_nas_s.onnx
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolox
|
|
label: YOLOX
|
|
recommended: false
|
|
download: YOLOx models can be downloaded [from the YOLOx repo](https://github.com/Megvii-BaseDetection/YOLOX/tree/main/demo/ONNXRuntime).
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Intel GPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ------------------------------------- | -------------------------------- |
|
|
| **Custom object detector model path** | `/config/yolox.onnx` (use the filename you generated above) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolox` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:GPU
|
|
model_type: yolox
|
|
path: /config/model_cache/yolox.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: rfdetr
|
|
label: RF-DETR
|
|
recommended: false
|
|
download: |-
|
|
RF-DETR can be exported as ONNX by running the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=Nano` in the first line to `Nano`, `Small`, or `Medium` size.
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=Nano --rm --output . -f- <<'EOF'
|
|
FROM python:3.12 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
|
|
WORKDIR /rfdetr
|
|
RUN uv pip install --system rfdetr[onnxexport] torch==2.8.0 onnx==1.19.1 transformers==4.57.6 onnxscript
|
|
ARG MODEL_SIZE
|
|
RUN python3 -c "from rfdetr import RFDETR${MODEL_SIZE}; x = RFDETR${MODEL_SIZE}(resolution=320); x.export(simplify=True)"
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
COPY --from=build /rfdetr/output/inference_model.onnx /rfdetr-${MODEL_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Intel GPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| --------------------------------------- | --------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/rfdetr.onnx` (use the filename you generated above) |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `rfdetr` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:GPU
|
|
model_type: rfdetr
|
|
width: 320
|
|
height: 320
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/rfdetr.onnx # use the filename you generated above
|
|
- key: dfine
|
|
label: D-FINE / DEIMv2
|
|
recommended: false
|
|
download: |-
|
|
#### D-FINE
|
|
|
|
D-FINE can be exported as ONNX by running the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=s` in the first line to `s`, `m`, or `l` size.
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=s --output . -f- <<'EOF'
|
|
FROM python:3.11 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.8.0 /uv /bin/
|
|
WORKDIR /dfine
|
|
RUN git clone https://github.com/Peterande/D-FINE.git .
|
|
RUN uv pip install --system -r requirements.txt
|
|
RUN uv pip install --system onnx onnxruntime onnxsim onnxscript
|
|
# Create output directory and download checkpoint
|
|
RUN mkdir -p output
|
|
ARG MODEL_SIZE
|
|
RUN wget https://github.com/Peterande/storage/releases/download/dfinev1.0/dfine_${MODEL_SIZE}_obj2coco.pth -O output/dfine_${MODEL_SIZE}_obj2coco.pth
|
|
# Modify line 58 of export_onnx.py to change batch size to 1
|
|
RUN sed -i '58s/data = torch.rand(.*)/data = torch.rand(1, 3, 640, 640)/' tools/deployment/export_onnx.py
|
|
RUN python3 tools/deployment/export_onnx.py -c configs/dfine/objects365/dfine_hgnetv2_${MODEL_SIZE}_obj2coco.yml -r output/dfine_${MODEL_SIZE}_obj2coco.pth
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
COPY --from=build /dfine/output/dfine_${MODEL_SIZE}_obj2coco.onnx /dfine-${MODEL_SIZE}.onnx
|
|
EOF
|
|
```
|
|
|
|
#### DEIMv2
|
|
|
|
[DEIMv2](https://github.com/Intellindust-AI-Lab/DEIMv2) can be exported as ONNX by running the command below. Pretrained weights are available on Hugging Face for two backbone families:
|
|
|
|
- **HGNetv2** (smaller/faster): `atto`, `femto`, `pico`, `n`
|
|
- **DINOv3** (larger/more accurate): `s`, `m`, `l`, `x`
|
|
|
|
Set `BACKBONE` and `MODEL_SIZE` in the first line to match your desired variant. Hugging Face model names use uppercase (e.g. `HGNetv2_N`, `DINOv3_S`), while config files use lowercase (e.g. `hgnetv2_n`, `dinov3_s`).
|
|
|
|
```sh
|
|
docker build . --rm --build-arg BACKBONE=hgnetv2 --build-arg MODEL_SIZE=n --output . -f- <<'EOF'
|
|
FROM python:3.11-slim AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y git libgl1 libglib2.0-0 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.8.0 /uv /bin/
|
|
WORKDIR /deimv2
|
|
RUN git clone https://github.com/Intellindust-AI-Lab/DEIMv2.git .
|
|
# Install CPU-only PyTorch first to avoid pulling CUDA variant
|
|
RUN uv pip install --no-cache --system torch torchvision --index-url https://download.pytorch.org/whl/cpu
|
|
RUN uv pip install --no-cache --system -r requirements.txt
|
|
RUN uv pip install --no-cache --system onnx safetensors huggingface_hub
|
|
RUN mkdir -p output
|
|
ARG BACKBONE
|
|
ARG MODEL_SIZE
|
|
# Download from Hugging Face and convert safetensors to pth
|
|
RUN python3 -c "\
|
|
from huggingface_hub import hf_hub_download; \
|
|
from safetensors.torch import load_file; \
|
|
import torch; \
|
|
backbone = '${BACKBONE}'.replace('hgnetv2','HGNetv2').replace('dinov3','DINOv3'); \
|
|
size = '${MODEL_SIZE}'.upper(); \
|
|
st = load_file(hf_hub_download('Intellindust/DEIMv2_' + backbone + '_' + size + '_COCO', 'model.safetensors')); \
|
|
torch.save({'model': st}, 'output/deimv2.pth')"
|
|
RUN sed -i "s/data = torch.rand(2/data = torch.rand(1/" tools/deployment/export_onnx.py
|
|
# HuggingFace safetensors omits frozen constants that the model constructor initializes
|
|
RUN sed -i "s/cfg.model.load_state_dict(state)/cfg.model.load_state_dict(state, strict=False)/" tools/deployment/export_onnx.py
|
|
RUN python3 tools/deployment/export_onnx.py -c configs/deimv2/deimv2_${BACKBONE}_${MODEL_SIZE}_coco.yml -r output/deimv2.pth
|
|
FROM scratch
|
|
ARG BACKBONE
|
|
ARG MODEL_SIZE
|
|
COPY --from=build /deimv2/output/deimv2.onnx /deimv2_${BACKBONE}_${MODEL_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **CPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ---------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/dfine-s.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `640` |
|
|
| **Object detection model input height** | `640` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `dfine` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- openvino:CPU
|
|
model_type: dfine
|
|
width: 640
|
|
height: 640
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/dfine-s.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
appleSilicon:
|
|
title: Apple Silicon
|
|
models:
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: true
|
|
download: |-
|
|
YOLOv9 model can be exported as ONNX using the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=t` and `IMG_SIZE=320` in the first line to the [model size](https://github.com/WongKinYiu/yolov9#performance) you would like to convert (available model sizes are `t`, `s`, `m`, `c`, and `e`, common image sizes are `320` and `640`).
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=t --build-arg IMG_SIZE=320 --output . -f- <<'EOF'
|
|
FROM python:3.11 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y cmake libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
|
|
WORKDIR /yolov9
|
|
ADD https://github.com/WongKinYiu/yolov9.git .
|
|
RUN uv pip install --system -r requirements.txt
|
|
RUN uv pip install --system onnx==1.18.0 onnxruntime onnx-simplifier==0.4.* onnxscript
|
|
ARG MODEL_SIZE
|
|
ARG IMG_SIZE
|
|
ADD https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-${MODEL_SIZE}-converted.pt yolov9-${MODEL_SIZE}.pt
|
|
RUN sed -i "s/ckpt = torch.load(attempt_download(w), map_location='cpu')/ckpt = torch.load(attempt_download(w), map_location='cpu', weights_only=False)/g" models/experimental.py
|
|
RUN python3 export.py --weights ./yolov9-${MODEL_SIZE}.pt --imgsz ${IMG_SIZE} --simplify --include onnx
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
ARG IMG_SIZE
|
|
COPY --from=build /yolov9/yolov9-${MODEL_SIZE}.onnx /yolov9-${MODEL_SIZE}-${IMG_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and add a model. The ZMQ endpoint is not reported by the hardware probe, so set `devices` to `zmq:tcp://host.docker.internal:5555` in YAML. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolo.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `320` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- zmq:tcp://host.docker.internal:5555
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize set during model export
|
|
height: 320 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/yolo.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolo-legacy
|
|
label: YOLO (v3, v4, v7)
|
|
recommended: false
|
|
download: |-
|
|
To export as ONNX:
|
|
|
|
```sh
|
|
git clone https://github.com/NateMeyer/tensorrt_demos
|
|
cd tensorrt_demos/yolo
|
|
./download_yolo.sh
|
|
python3 yolo_to_onnx.py -m yolov7-320
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and add a model. The ZMQ endpoint is not reported by the hardware probe, so set `devices` to `zmq:tcp://host.docker.internal:5555` in YAML. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolo.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `320` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- zmq:tcp://host.docker.internal:5555
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize set during model export
|
|
height: 320 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/yolo.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
onnx:
|
|
title: ONNX
|
|
models:
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: true
|
|
download: |-
|
|
YOLOv9 model can be exported as ONNX using the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=t` and `IMG_SIZE=320` in the first line to the [model size](https://github.com/WongKinYiu/yolov9#performance) you would like to convert (available model sizes are `t`, `s`, `m`, `c`, and `e`, common image sizes are `320` and `640`).
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=t --build-arg IMG_SIZE=320 --output . -f- <<'EOF'
|
|
FROM python:3.11 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y cmake libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
|
|
WORKDIR /yolov9
|
|
ADD https://github.com/WongKinYiu/yolov9.git .
|
|
RUN uv pip install --system -r requirements.txt
|
|
RUN uv pip install --system onnx==1.18.0 onnxruntime onnx-simplifier==0.4.* onnxscript
|
|
ARG MODEL_SIZE
|
|
ARG IMG_SIZE
|
|
ADD https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-${MODEL_SIZE}-converted.pt yolov9-${MODEL_SIZE}.pt
|
|
RUN sed -i "s/ckpt = torch.load(attempt_download(w), map_location='cpu')/ckpt = torch.load(attempt_download(w), map_location='cpu', weights_only=False)/g" models/experimental.py
|
|
RUN python3 export.py --weights ./yolov9-${MODEL_SIZE}.pt --imgsz ${IMG_SIZE} --simplify --include onnx
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
ARG IMG_SIZE
|
|
COPY --from=build /yolov9/yolov9-${MODEL_SIZE}.onnx /yolov9-${MODEL_SIZE}-${IMG_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select your GPU from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolo.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `320` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- onnx
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize set during model export
|
|
height: 320 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/yolo.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: rfdetr
|
|
label: RF-DETR
|
|
recommended: false
|
|
download: |-
|
|
RF-DETR can be exported as ONNX by running the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=Nano` in the first line to `Nano`, `Small`, or `Medium` size.
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=Nano --rm --output . -f- <<'EOF'
|
|
FROM python:3.12 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
|
|
WORKDIR /rfdetr
|
|
RUN uv pip install --system rfdetr[onnxexport] torch==2.8.0 onnx==1.19.1 transformers==4.57.6 onnxscript
|
|
ARG MODEL_SIZE
|
|
RUN python3 -c "from rfdetr import RFDETR${MODEL_SIZE}; x = RFDETR${MODEL_SIZE}(resolution=320); x.export(simplify=True)"
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
COPY --from=build /rfdetr/output/inference_model.onnx /rfdetr-${MODEL_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select your GPU from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| --------------------------------------- | --------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/rfdetr.onnx` (use the filename you generated above) |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `rfdetr` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- onnx
|
|
model_type: rfdetr
|
|
width: 320
|
|
height: 320
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/rfdetr.onnx # use the filename you generated above
|
|
- key: yolonas
|
|
label: YOLO-NAS
|
|
recommended: false
|
|
download: |-
|
|
You can build and download a compatible model with pre-trained weights using [this notebook](https://github.com/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb) [](https://colab.research.google.com/github/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb) which can be run directly in [Google Colab](https://colab.research.google.com/github/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb).
|
|
|
|
:::warning
|
|
|
|
The pre-trained YOLO-NAS weights from DeciAI are subject to their license and can't be used commercially. For more information, see: https://docs.deci.ai/super-gradients/latest/LICENSE.YOLONAS.html
|
|
|
|
:::
|
|
|
|
The input image size in this notebook is set to 320x320. This results in lower CPU usage and faster inference times without impacting performance in most cases due to the way Frigate crops video frames to areas of interest before running detection. The notebook and config can be updated to 640x640 if desired.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select your GPU from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/yolo_nas_s.onnx` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match whatever was set in notebook) |
|
|
| **Object detection model input height** | `320` (should match whatever was set in notebook) |
|
|
| **Model Input Pixel Color Format** | `bgr` |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolonas` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- onnx
|
|
model_type: yolonas
|
|
width: 320 # <--- should match whatever was set in notebook
|
|
height: 320 # <--- should match whatever was set in notebook
|
|
input_pixel_format: bgr
|
|
input_tensor: nchw
|
|
path: /config/yolo_nas_s.onnx
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolox
|
|
label: YOLOX
|
|
recommended: false
|
|
download: YOLOx models can be downloaded [from the YOLOx repo](https://github.com/Megvii-BaseDetection/YOLOX/tree/main/demo/ONNXRuntime).
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select your GPU from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolox_tiny.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `416` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `416` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float_denorm` |
|
|
| **Object Detection Model Type** | `yolox` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- onnx
|
|
model_type: yolox
|
|
width: 416 # <--- should match the imgsize set during model export
|
|
height: 416 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float_denorm
|
|
path: /config/model_cache/yolox_tiny.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: dfine
|
|
label: D-FINE / DEIMv2
|
|
recommended: false
|
|
download: |-
|
|
#### Downloading D-FINE Model
|
|
|
|
D-FINE can be exported as ONNX by running the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=s` in the first line to `s`, `m`, or `l` size.
|
|
|
|
```sh
|
|
docker build . --build-arg MODEL_SIZE=s --output . -f- <<'EOF'
|
|
FROM python:3.11 AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y libgl1 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.8.0 /uv /bin/
|
|
WORKDIR /dfine
|
|
RUN git clone https://github.com/Peterande/D-FINE.git .
|
|
RUN uv pip install --system -r requirements.txt
|
|
RUN uv pip install --system onnx onnxruntime onnxsim onnxscript
|
|
# Create output directory and download checkpoint
|
|
RUN mkdir -p output
|
|
ARG MODEL_SIZE
|
|
RUN wget https://github.com/Peterande/storage/releases/download/dfinev1.0/dfine_${MODEL_SIZE}_obj2coco.pth -O output/dfine_${MODEL_SIZE}_obj2coco.pth
|
|
# Modify line 58 of export_onnx.py to change batch size to 1
|
|
RUN sed -i '58s/data = torch.rand(.*)/data = torch.rand(1, 3, 640, 640)/' tools/deployment/export_onnx.py
|
|
RUN python3 tools/deployment/export_onnx.py -c configs/dfine/objects365/dfine_hgnetv2_${MODEL_SIZE}_obj2coco.yml -r output/dfine_${MODEL_SIZE}_obj2coco.pth
|
|
FROM scratch
|
|
ARG MODEL_SIZE
|
|
COPY --from=build /dfine/output/dfine_${MODEL_SIZE}_obj2coco.onnx /dfine-${MODEL_SIZE}.onnx
|
|
EOF
|
|
```
|
|
|
|
#### Downloading DEIMv2 Model
|
|
|
|
[DEIMv2](https://github.com/Intellindust-AI-Lab/DEIMv2) can be exported as ONNX by running the command below. Pretrained weights are available on Hugging Face for two backbone families:
|
|
|
|
- **HGNetv2** (smaller/faster): `atto`, `femto`, `pico`, `n`
|
|
- **DINOv3** (larger/more accurate): `s`, `m`, `l`, `x`
|
|
|
|
Set `BACKBONE` and `MODEL_SIZE` in the first line to match your desired variant. Hugging Face model names use uppercase (e.g. `HGNetv2_N`, `DINOv3_S`), while config files use lowercase (e.g. `hgnetv2_n`, `dinov3_s`).
|
|
|
|
```sh
|
|
docker build . --rm --build-arg BACKBONE=hgnetv2 --build-arg MODEL_SIZE=n --output . -f- <<'EOF'
|
|
FROM python:3.11-slim AS build
|
|
RUN apt-get update && apt-get install --no-install-recommends -y git libgl1 libglib2.0-0 && rm -rf /var/lib/apt/lists/*
|
|
COPY --from=ghcr.io/astral-sh/uv:0.8.0 /uv /bin/
|
|
WORKDIR /deimv2
|
|
RUN git clone https://github.com/Intellindust-AI-Lab/DEIMv2.git .
|
|
# Install CPU-only PyTorch first to avoid pulling CUDA variant
|
|
RUN uv pip install --no-cache --system torch torchvision --index-url https://download.pytorch.org/whl/cpu
|
|
RUN uv pip install --no-cache --system -r requirements.txt
|
|
RUN uv pip install --no-cache --system onnx safetensors huggingface_hub
|
|
RUN mkdir -p output
|
|
ARG BACKBONE
|
|
ARG MODEL_SIZE
|
|
# Download from Hugging Face and convert safetensors to pth
|
|
RUN python3 -c "\
|
|
from huggingface_hub import hf_hub_download; \
|
|
from safetensors.torch import load_file; \
|
|
import torch; \
|
|
backbone = '${BACKBONE}'.replace('hgnetv2','HGNetv2').replace('dinov3','DINOv3'); \
|
|
size = '${MODEL_SIZE}'.upper(); \
|
|
st = load_file(hf_hub_download('Intellindust/DEIMv2_' + backbone + '_' + size + '_COCO', 'model.safetensors')); \
|
|
torch.save({'model': st}, 'output/deimv2.pth')"
|
|
RUN sed -i "s/data = torch.rand(2/data = torch.rand(1/" tools/deployment/export_onnx.py
|
|
# HuggingFace safetensors omits frozen constants that the model constructor initializes
|
|
RUN sed -i "s/cfg.model.load_state_dict(state)/cfg.model.load_state_dict(state, strict=False)/" tools/deployment/export_onnx.py
|
|
RUN python3 tools/deployment/export_onnx.py -c configs/deimv2/deimv2_${BACKBONE}_${MODEL_SIZE}_coco.yml -r output/deimv2.pth
|
|
FROM scratch
|
|
ARG BACKBONE
|
|
ARG MODEL_SIZE
|
|
COPY --from=build /deimv2/output/deimv2.onnx /deimv2_${BACKBONE}_${MODEL_SIZE}.onnx
|
|
EOF
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select your GPU from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/dfine_m_obj2coco.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `640` |
|
|
| **Object detection model input height** | `640` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `dfine` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- onnx
|
|
model_type: dfine
|
|
width: 640
|
|
height: 640
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/dfine_m_obj2coco.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolo-legacy
|
|
label: YOLO (v3, v4, v7)
|
|
recommended: false
|
|
download: |-
|
|
To export as ONNX:
|
|
|
|
```sh
|
|
git clone https://github.com/NateMeyer/tensorrt_demos
|
|
cd tensorrt_demos/yolo
|
|
./download_yolo.sh
|
|
python3 yolo_to_onnx.py -m yolov7-320
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select your GPU from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/yolo.onnx` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (should match the imgsize set during model export) |
|
|
| **Object detection model input height** | `320` (should match the imgsize set during model export) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- onnx
|
|
model_type: yolo-generic
|
|
width: 320 # <--- should match the imgsize set during model export
|
|
height: 320 # <--- should match the imgsize set during model export
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/model_cache/yolo.onnx # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
cpu:
|
|
title: CPU
|
|
models:
|
|
- key: ssd
|
|
label: MobileNet v2
|
|
recommended: true
|
|
download: A TensorFlow Lite model is provided in the container at `/cpu_model.tflite` and is used by this detector type by default. To provide your own model, bind mount the file into the container and provide the path with a model's `path`.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **CPU** from the **Hardware** dropdown and set **Detectors** to the number of detection processes to run (one per camera is recommended).
|
|
|
|
| Field | Value |
|
|
| ----------------- | ----- |
|
|
| **Detector type** | `cpu` |
|
|
| **Num threads** | `3` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- cpu:3
|
|
xdna2:
|
|
title: AMD XDNA2
|
|
models:
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: true
|
|
download: |-
|
|
Prepare the model using the frigate-xdna setup instructions linked above. For local YOLO models, Frigate must have access to the same ONNX file bytes as the sidecar. The example below uses YOLOv9-C at 320x320. Frigate+ models may instead use the same `plus://MODEL_ID` in Frigate and the sidecar.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and add a model. The ZMQ endpoint is not reported by the hardware probe, so set `devices` to `zmq:tcp://xdna:5555` in YAML. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------ |
|
|
| **Custom object detector model path** | `/config/models/yolov9-c-320.onnx` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- zmq:tcp://xdna:5555
|
|
model_type: yolo-generic
|
|
width: 320
|
|
height: 320
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
path: /config/models/yolov9-c-320.onnx
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
memryx:
|
|
title: MemryX
|
|
models:
|
|
- key: yolonas
|
|
label: YOLO-NAS
|
|
recommended: true
|
|
download: |-
|
|
The [YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md) model included in this detector is downloaded automatically and compiled to DFP with [mx_nc](https://developer.memryx.com/2p1/tools/neural_compiler.html#usage).
|
|
|
|
**Note:** The default model for the MemryX detector is YOLO-NAS 320x320.
|
|
|
|
The input size for **YOLO-NAS** can be set to either **320x320** (default) or **640x640**.
|
|
|
|
- The default size of **320x320** is optimized for lower CPU usage and faster inference times.
|
|
|
|
MemryX `.dfp` models are automatically downloaded at runtime, if enabled, to the container at `/memryx_models/model_folder/`.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **MemryX MX3** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------------- |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (can be set to `640` for higher resolution) |
|
|
| **Object detection model input height** | `320` (can be set to `640` for higher resolution) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolonas` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- memryx:PCIe:0
|
|
model_type: yolonas
|
|
width: 320 # (Can be set to 640 for higher resolution)
|
|
height: 320 # (Can be set to 640 for higher resolution)
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
|
# path: /config/yolonas.zip
|
|
# The .zip file must contain:
|
|
# ├── yolonas.dfp (a file ending with .dfp)
|
|
# └── yolonas_post.onnx (optional; only if the model includes a cropped post-processing network)
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: false
|
|
download: |-
|
|
The YOLOv9s model included in this detector is downloaded from [the original GitHub](https://github.com/WongKinYiu/yolov9) and compiled to DFP with [mx_nc](https://developer.memryx.com/2p1/tools/neural_compiler.html#usage).
|
|
|
|
MemryX `.dfp` models are automatically downloaded at runtime, if enabled, to the container at `/memryx_models/model_folder/`.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **MemryX MX3** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------------- |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (can be set to `640` for higher resolution) |
|
|
| **Object detection model input height** | `320` (can be set to `640` for higher resolution) |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- memryx:PCIe:0
|
|
model_type: yolo-generic
|
|
width: 320 # (Can be set to 640 for higher resolution)
|
|
height: 320 # (Can be set to 640 for higher resolution)
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
|
# path: /config/yolov9.zip
|
|
# The .zip file must contain:
|
|
# ├── yolov9.dfp (a file ending with .dfp)
|
|
- key: yolox
|
|
label: YOLOX
|
|
recommended: false
|
|
download: |-
|
|
The model is sourced from the [OpenCV Model Zoo](https://github.com/opencv/opencv_zoo) and precompiled to DFP.
|
|
|
|
MemryX `.dfp` models are automatically downloaded at runtime, if enabled, to the container at `/memryx_models/model_folder/`.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **MemryX MX3** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ----------------------- |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `640` |
|
|
| **Object detection model input height** | `640` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float_denorm` |
|
|
| **Object Detection Model Type** | `yolox` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- memryx:PCIe:0
|
|
model_type: yolox
|
|
width: 640
|
|
height: 640
|
|
input_tensor: nchw
|
|
input_dtype: float_denorm
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
|
# path: /config/yolox.zip
|
|
# The .zip file must contain:
|
|
# ├── yolox.dfp (a file ending with .dfp)
|
|
- key: ssd
|
|
label: SSDLite MobileNet v2
|
|
recommended: false
|
|
download: |-
|
|
The model is sourced from the [OpenMMLab Model Zoo](https://mmdeploy-oss.openmmlab.com/model/mmdet-det/ssdlite-e8679f.onnx) and has been converted to DFP.
|
|
|
|
MemryX `.dfp` models are automatically downloaded at runtime, if enabled, to the container at `/memryx_models/model_folder/`.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **MemryX MX3** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ----------------------- |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `float` |
|
|
| **Object Detection Model Type** | `ssd` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- memryx:PCIe:0
|
|
model_type: ssd
|
|
width: 320
|
|
height: 320
|
|
input_tensor: nchw
|
|
input_dtype: float
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
# Optional: The model is normally fetched through the runtime, so 'path' can be omitted unless you want to use a custom or local model.
|
|
# path: /config/ssdlite_mobilenet.zip
|
|
# The .zip file must contain:
|
|
# ├── ssdlite_mobilenet.dfp (a file ending with .dfp)
|
|
# └── ssdlite_mobilenet_post.onnx (optional; only if the model includes a cropped post-processing network)
|
|
deepx:
|
|
title: DEEPX NPU
|
|
models:
|
|
- key: yolo
|
|
label: YOLO
|
|
recommended: true
|
|
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The recommended model is `yolox-s_640x640_ppu.dxnn`. Its Post-Processing Unit (PPU) compile moves candidate selection onto the NPU, which makes it the fastest ModelZoo model measured through Frigate (about 13 ms on a DX-M1). The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration; prefer a `PPU` variant whenever the ModelZoo offers one. PPU models must be compiled with DX-COM 2.4.0 or later, which writes the head layout Frigate reads into the file.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ---------------------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640_ppu.dxnn` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `640` |
|
|
| **Object detection model input height** | `640` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
|
|
Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- deepx:PCIe:0
|
|
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
model_type: yolo-generic
|
|
width: 640
|
|
height: 640
|
|
- key: yolox
|
|
label: YOLOX
|
|
recommended: false
|
|
download: No model is bundled with Frigate. Download a pre-compiled YOLOX `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), then bind-mount it into the container and point the model's `path` at it. The `_ppu` variant is faster and also works with the `yolo-generic` model type; the plain export needs `yolox` so its raw head is decoded.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ---------------------------------------- |
|
|
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640.dxnn`|
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `640` |
|
|
| **Object detection model input height** | `640` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolox` |
|
|
|
|
The width and height must match the resolution the `.dxnn` file was compiled for.
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- deepx:PCIe:0
|
|
path: /config/model_cache/deepx/yolox-s_640x640.dxnn
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
model_type: yolox
|
|
width: 640
|
|
height: 640
|
|
tensorrt:
|
|
title: TensorRT
|
|
models:
|
|
- key: yolo-legacy
|
|
label: YOLO (v3, v4, v7)
|
|
recommended: true
|
|
download: |-
|
|
The model used for TensorRT must be preprocessed on the same hardware platform that it will run on, so Frigate generates the `.trt` model file on-device at startup. Processed models are stored in the `/config/model_cache` folder.
|
|
|
|
By default no models are generated. Set the `YOLO_MODELS` environment variable in Docker to one or more comma-separated model names (from the available `yolov3`/`yolov4`/`yolov7` models) and each one will be generated on startup if the corresponding `{model}.trt` file is not already present in `model_cache` (delete it to force regeneration). On Jetson devices with DLAs (Xavier or Orin), append `-dla` to a model name to generate a DLA model. If your GPU does not support FP16 operations, pass `USE_FP16=False` to disable it.
|
|
|
|
An example `docker-compose.yml` fragment that converts the `yolov7-320` and `yolov7x-640` models:
|
|
|
|
```yml
|
|
frigate:
|
|
environment:
|
|
- YOLO_MODELS=yolov7-320,yolov7x-640
|
|
- USE_FP16=false
|
|
```
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **NVIDIA Jetson** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------------------------ |
|
|
| **Custom object detector model path** | `/config/model_cache/tensorrt/yolov7-320.trt` (use the filename you generated above) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` (MUST match the chosen model, e.g., yolov7-320 -> 320) |
|
|
| **Object detection model input height** | `320` (MUST match the chosen model, e.g., yolov7-320 -> 320) |
|
|
| **Model Input Pixel Color Format** | `rgb` |
|
|
| **Model Input Tensor Shape** | `nchw` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `ssd` (Frigate's default value) |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- tensorrt:0
|
|
path: /config/model_cache/tensorrt/yolov7-320.trt # use the filename you generated above
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
input_tensor: nchw
|
|
input_pixel_format: rgb
|
|
width: 320 # MUST match the chosen model i.e yolov7-320 -> 320, yolov4-416 -> 416
|
|
height: 320 # MUST match the chosen model i.e yolov7-320 -> 320 yolov4-416 -> 416
|
|
synaptics:
|
|
title: Synaptics
|
|
models:
|
|
- key: ssd
|
|
label: SSD MobileNet
|
|
recommended: true
|
|
download: A synap model is provided in the container at `/synaptics/mobilenet.synap` and is used by this detector type by default. The model comes from the [Synap-release Github](https://github.com/synaptics-astra/synap-release/tree/v1.5.0/models/dolphin/object_detection/coco/model/mobilenet224_full80).
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Synaptics NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ---------------------------- |
|
|
| **Custom object detector model path** | `/synaptics/mobilenet.synap` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `224` |
|
|
| **Object detection model input height** | `224` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `ssd` (Frigate's default value) |
|
|
yaml: |-
|
|
models:
|
|
- # required
|
|
devices:
|
|
- synaptics
|
|
path: /synaptics/mobilenet.synap # required
|
|
width: 224 # required
|
|
height: 224 # required
|
|
input_tensor: nhwc # default value (optional. If you change the model, it is required)
|
|
labelmap_path: /labelmap/coco-80.txt # required
|
|
rknn:
|
|
title: RKNN
|
|
models:
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: true
|
|
download: |-
|
|
If no custom model is provided, the RKNN detector downloads a default model from GitHub on first startup. Once cached, the model works fully offline. All models are automatically downloaded and stored in the folder `config/model_cache/rknn_cache`. After upgrading Frigate, you should remove older models to free up space.
|
|
|
|
You can also provide your own `.rknn` model. You should not save your own models in the `rknn_cache` folder, store them directly in the `model_cache` folder or another subfolder. To convert a model to `.rknn` format see the `rknn-toolkit2` (requires a x86 machine). Note, that there is only post-processing for the supported models.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** select **Rockchip NPU** from the **Hardware** dropdown, then open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | -------------------------------------------------- |
|
|
| **Custom object detector model path** | `frigate-fp16-yolov9-t` (or other yolov9 variants) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- rknn
|
|
# name of model (will be automatically downloaded) or path to your own .rknn model file
|
|
# possible values are:
|
|
# - frigate-fp16-yolov9-t
|
|
# - frigate-fp16-yolov9-s
|
|
# - frigate-fp16-yolov9-m
|
|
# - frigate-fp16-yolov9-c
|
|
# - frigate-fp16-yolov9-e
|
|
# your yolo_model.rknn
|
|
path: frigate-fp16-yolov9-t
|
|
model_type: yolo-generic
|
|
width: 320
|
|
height: 320
|
|
input_tensor: nhwc
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolonas
|
|
label: YOLO-NAS
|
|
recommended: false
|
|
download: |-
|
|
If no custom model is provided, the RKNN detector downloads a default model from GitHub on first startup. Once cached, the model works fully offline. All models are automatically downloaded and stored in the folder `config/model_cache/rknn_cache`. After upgrading Frigate, you should remove older models to free up space.
|
|
|
|
You can also provide your own `.rknn` model. You should not save your own models in the `rknn_cache` folder, store them directly in the `model_cache` folder or another subfolder. To convert a model to `.rknn` format see the `rknn-toolkit2` (requires a x86 machine). Note, that there is only post-processing for the supported models.
|
|
|
|
**Note:** The pre-trained YOLO-NAS weights from DeciAI are subject to their license and can't be used commercially. For more information, see: https://docs.deci.ai/super-gradients/latest/LICENSE.YOLONAS.html
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** select **Rockchip NPU** from the **Hardware** dropdown, then open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ----------------------------------------------------------------------- |
|
|
| **Custom object detector model path** | `deci-fp16-yolonas_s` (or `deci-fp16-yolonas_m`, `deci-fp16-yolonas_l`) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `bgr` |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolonas` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- rknn
|
|
# name of model (will be automatically downloaded) or path to your own .rknn model file
|
|
# possible values are:
|
|
# - deci-fp16-yolonas_s
|
|
# - deci-fp16-yolonas_m
|
|
# - deci-fp16-yolonas_l
|
|
# your yolonas_model.rknn
|
|
path: deci-fp16-yolonas_s
|
|
model_type: yolonas
|
|
width: 320
|
|
height: 320
|
|
input_pixel_format: bgr
|
|
input_tensor: nhwc
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
- key: yolox
|
|
label: YOLOx
|
|
recommended: false
|
|
download: |-
|
|
If no custom model is provided, the RKNN detector downloads a default model from GitHub on first startup. Once cached, the model works fully offline. All models are automatically downloaded and stored in the folder `config/model_cache/rknn_cache`. After upgrading Frigate, you should remove older models to free up space.
|
|
|
|
You can also provide your own `.rknn` model. You should not save your own models in the `rknn_cache` folder, store them directly in the `model_cache` folder or another subfolder. To convert a model to `.rknn` format see the `rknn-toolkit2` (requires a x86 machine). Note, that there is only post-processing for the supported models.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** select **Rockchip NPU** from the **Hardware** dropdown, then open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ---------------------------------------------- |
|
|
| **Custom object detector model path** | `rock-i8-yolox_nano` (or other yolox variants) |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `416` |
|
|
| **Object detection model input height** | `416` |
|
|
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
|
| **Model Input Tensor Shape** | `nhwc` |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolox` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- rknn
|
|
# name of model (will be automatically downloaded) or path to your own .rknn model file
|
|
# possible values are:
|
|
# - rock-i8-yolox_nano
|
|
# - rock-i8-yolox_tiny
|
|
# - rock-fp16-yolox_nano
|
|
# - rock-fp16-yolox_tiny
|
|
# your yolox_model.rknn
|
|
path: rock-i8-yolox_nano
|
|
model_type: yolox
|
|
width: 416
|
|
height: 416
|
|
input_tensor: nhwc
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
axengine:
|
|
title: AXEngine
|
|
models:
|
|
- key: yolov9
|
|
label: YOLOv9
|
|
recommended: true
|
|
download: A yolov9 axmodel is provided in the container at `/axmodels` and is used by this detector type by default. The AXEngine detector downloads its default model from HuggingFace on first startup; once cached, the model works fully offline.
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **AXERA NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ----------------------- |
|
|
| **Custom object detector model path** | `frigate-yolov9-tiny` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `320` |
|
|
| **Object detection model input height** | `320` |
|
|
| **Model Input Pixel Color Format** | `bgr` |
|
|
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
|
| **Model Input D Type** | `int` |
|
|
| **Object Detection Model Type** | `yolo-generic` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- axengine
|
|
path: frigate-yolov9-tiny
|
|
model_type: yolo-generic
|
|
width: 320
|
|
height: 320
|
|
input_dtype: int
|
|
input_pixel_format: bgr
|
|
labelmap_path: /labelmap/coco-80.txt
|
|
axelera:
|
|
title: Axelera Metis
|
|
models:
|
|
- key: yolox
|
|
label: YOLOX
|
|
recommended: true
|
|
download: |-
|
|
Axelera models must be compiled for the Metis NPU with the Voyager SDK before Frigate can run them (compilation is not supported inside the container). Compile a model with the SDK, zip the compiled output, and point the model's `path` at the zip. The default recommendation is the Apache-2.0 licensed YOLOX (Megvii YOLOX, e.g. the `yolox-s-coco-onnx` model-zoo network).
|
|
ui: |-
|
|
Navigate to **Settings > System > Detection models** and select **Axelera Metis** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
|
|
|
| Field | Value |
|
|
| ---------------------------------------- | ------------------------------------------------------------ |
|
|
| **Custom object detector model path** | `/config/model_cache/axelera/yolox-s-coco-onnx.zip` |
|
|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
|
| **Object detection model input width** | `640` (must equal the compiled input size) |
|
|
| **Object detection model input height** | `640` |
|
|
| **Model Input Pixel Color Format** | `bgr` (Frigate delivers BGR; the plugin hands RGB planes to the compiled input) |
|
|
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
|
| **Model Input D Type** | `int` (Frigate's default value) |
|
|
| **Object Detection Model Type** | `yolox` |
|
|
yaml: |-
|
|
models:
|
|
- devices:
|
|
- axelera:PCIe
|
|
path: /config/model_cache/axelera/yolox-s-coco-onnx.zip
|
|
model_type: yolox
|
|
width: 640
|
|
height: 640
|
|
input_pixel_format: bgr
|
|
labelmap_path: /labelmap/coco-80.txt
|