Merge remote-tracking branch 'origin/master' into dev

2026-08-02 09:02:15 +03:00 · 2026-05-02 10:08:36 -05:00
parent 45213d0420 2cfb530dbf
commit 6a2b914b10
8 changed files with 498 additions and 118 deletions
@@ -494,7 +494,7 @@ detectors:
 | [YOLO-NAS](#yolo-nas)                 | ✅  | ✅  |                                                              |
 | [MobileNet v2](#ssdlite-mobilenet-v2) | ✅  | ✅  | Fast and lightweight model, less accurate than larger models |
 | [YOLOX](#yolox)                       | ✅  | ?   |                                                              |
-| [D-FINE](#d-fine)                     | ❌  | ❌  |                                                              |
+| [D-FINE / DEIMv2](#d-fine--deimv2)    | ❌  | ❌  |                                                              |

 #### SSDLite MobileNet v2

@@ -710,13 +710,13 @@ model:

 </details>

-#### D-FINE
+#### D-FINE / DEIMv2

-[D-FINE](https://github.com/Peterande/D-FINE) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-d-fine-model) for more information on downloading the D-FINE model for use in Frigate.
+[D-FINE](https://github.com/Peterande/D-FINE) and [DEIMv2](https://github.com/Intellindust-AI-Lab/DEIMv2) are DETR based models that share the same ONNX input/output format. The ONNX exported models are supported, but not included by default. See the models section for downloading [D-FINE](#downloading-d-fine-model) or [DEIMv2](#downloading-deimv2-model) for use in Frigate.

 :::warning

-Currently D-FINE models only run on OpenVINO in CPU mode, GPUs currently fail to compile the model
+Currently D-FINE / DEIMv2 models only run on OpenVINO in CPU mode, GPUs currently fail to compile the model

 :::

@@ -766,6 +766,31 @@ Note that the labelmap uses a subset of the complete COCO label set that has onl

 </details>

+<details>
+  <summary>DEIMv2 Setup & Config</summary>
+
+After placing the downloaded onnx model in your `config/model_cache` folder, you can use the following configuration:
+
+```yaml
+detectors:
+  ov:
+    type: openvino
+    device: CPU
+
+model:
+  model_type: dfine
+  width: 640
+  height: 640
+  input_tensor: nchw
+  input_dtype: float
+  path: /config/model_cache/deimv2_hgnetv2_n.onnx
+  labelmap_path: /labelmap/coco-80.txt
+```
+
+Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
+
+</details>
+
 ## Apple Silicon detector

 The NPU in Apple Silicon can't be accessed from within a container, so the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) must first be setup. It is recommended to use the Frigate docker image with `-standard-arm64` suffix, for example `ghcr.io/blakeblackshear/frigate:stable-standard-arm64`.
@@ -947,7 +972,7 @@ The AMD GPU kernel is known problematic especially when converting models to mxr

 See [ONNX supported models](#supported-models) for supported models, there are some caveats:

- D-FINE models are not supported
+- D-FINE / DEIMv2 models are not supported
 - YOLO-NAS models are known to not run well on integrated GPUs

 ## ONNX
@@ -1003,7 +1028,7 @@ detectors:
 | [RF-DETR](#rf-detr)           | ✅         | ❌      | Supports CUDA Graphs for optimal Nvidia performance |
 | [YOLO-NAS](#yolo-nas-1)       | ⚠️         | ⚠️      | Not supported by CUDA Graphs                        |
 | [YOLOX](#yolox-1)             | ✅         | ✅      | Supports CUDA Graphs for optimal Nvidia performance |
-| [D-FINE](#d-fine)             | ⚠️         | ❌      | Not supported by CUDA Graphs                        |
+| [D-FINE / DEIMv2](#d-fine--deimv2-1) | ⚠️         | ❌      | Not supported by CUDA Graphs                        |

 There is no default model provided, the following formats are supported:

@@ -1215,9 +1240,9 @@ model:

 </details>

-#### D-FINE
+#### D-FINE / DEIMv2

-[D-FINE](https://github.com/Peterande/D-FINE) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-d-fine-model) for more information on downloading the D-FINE model for use in Frigate.
+[D-FINE](https://github.com/Peterande/D-FINE) and [DEIMv2](https://github.com/Intellindust-AI-Lab/DEIMv2) are DETR based models that share the same ONNX input/output format. The ONNX exported models are supported, but not included by default. See the models section for downloading [D-FINE](#downloading-d-fine-model) or [DEIMv2](#downloading-deimv2-model) for use in Frigate.

 <details>
  <summary>D-FINE Setup & Config</summary>
@@ -1262,6 +1287,28 @@ model:

 </details>

+<details>
+  <summary>DEIMv2 Setup & Config</summary>
+
+After placing the downloaded onnx model in your `config/model_cache` folder, you can use the following configuration:
+
+```yaml
+detectors:
+  onnx:
+    type: onnx
+
+model:
+  model_type: dfine
+  width: 640
+  height: 640
+  input_tensor: nchw
+  input_dtype: float
+  path: /config/model_cache/deimv2_hgnetv2_n.onnx
+  labelmap_path: /labelmap/coco-80.txt
+```
+
+</details>
+
 Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.

 ## CPU Detector (not recommended)
@@ -1405,7 +1452,7 @@ MemryX `.dfp` models are automatically downloaded at runtime, if enabled, to the

 #### YOLO-NAS

-The [YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md) model included in this detector is downloaded from the [Models Section](#downloading-yolo-nas-model) and compiled to DFP with [mx_nc](https://developer.memryx.com/tools/neural_compiler.html#usage).
+The [YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md) model included in this detector is downloaded from the [Models Section](#downloading-yolo-nas-model) and compiled to DFP with [mx_nc](https://developer.memryx.com/2p1/tools/neural_compiler.html#usage).

 **Note:** The default model for the MemryX detector is YOLO-NAS 320x320.

@@ -1459,7 +1506,7 @@ model:

 #### YOLOv9

-The YOLOv9s model included in this detector is downloaded from [the original GitHub](https://github.com/WongKinYiu/yolov9) like in the [Models Section](#yolov9-1) and compiled to DFP with [mx_nc](https://developer.memryx.com/tools/neural_compiler.html#usage).
+The YOLOv9s model included in this detector is downloaded from [the original GitHub](https://github.com/WongKinYiu/yolov9) like in the [Models Section](#yolov9-1) and compiled to DFP with [mx_nc](https://developer.memryx.com/2p1/tools/neural_compiler.html#usage).

 ##### Configuration

@@ -1601,19 +1648,39 @@ model:

 #### Using a Custom Model

-To use your own model:
+To use your own custom model, first compile it into a [.dfp](https://developer.memryx.com/2p1/specs/files.html#dataflow-program) file, which is the format used by MemryX.

-1.  Package your compiled model into a `.zip` file.
+#### Compile the Model

-2.  The `.zip` must contain the compiled `.dfp` file.
+Custom models must be compiled using **MemryX SDK 2.1**.

-3.  Depending on the model, the compiler may also generate a cropped post-processing network. If present, it will be named with the suffix `_post.onnx`.
+Before compiling your model, install the MemryX Neural Compiler tools from the
+[Install Tools](https://developer.memryx.com/2p1/get_started/install_tools.html) page on the **host**.

-4.  Bind-mount the `.zip` file into the container and specify its path using `model.path` in your config.
+> **Note:** It is recommended to compile the model on the host machine, or on another separate machine, rather than inside the Frigate Docker container. Installing the compiler inside Docker may conflict with container packages. It is recommended to create a Python virtual environment and install the compiler there.

-5.  Update the `labelmap_path` to match your custom model's labels.
+Once the SDK 2.1 environment is set up, follow the
+[MemryX Compiler](https://developer.memryx.com/2p1/tools/neural_compiler.html#usage) documentation to compile your model.

-For detailed instructions on compiling models, refer to the [MemryX Compiler](https://developer.memryx.com/tools/neural_compiler.html#usage) docs and [Tutorials](https://developer.memryx.com/tutorials/tutorials.html).
+Example:
+
+```bash
+mx_nc -m yolonas.onnx -c 4 --autocrop -v --dfp_fname yolonas.dfp
+```
+
+For detailed instructions on compiling models, refer to the [MemryX Compiler](https://developer.memryx.com/2p1/tools/neural_compiler.html#usage) docs and [Tutorials](https://developer.memryx.com/2p1/tutorials/tutorials.html).
+
+#### Package the Compiled Model
+
+1. Package your compiled model into a `.zip` file.
+
+2. The `.zip` file must contain the compiled `.dfp` file.
+
+3. Depending on the model, the compiler may also generate a cropped post-processing network. If present, it will be named with the suffix `_post.onnx`.
+
+4. Bind-mount the `.zip` file into the container and specify its path using `model.path` in your config.
+
+5. Update `labelmap_path` to match your custom model's labels.

 ```yaml
 # The detector automatically selects the default model if nothing is provided in the config.
@@ -2274,6 +2341,49 @@ COPY --from=build /dfine/output/dfine_${MODEL_SIZE}_obj2coco.onnx /dfine-${MODEL
 EOF
 ```

+### Downloading DEIMv2 Model
+
+[DEIMv2](https://github.com/Intellindust-AI-Lab/DEIMv2) can be exported as ONNX by running the command below. Pretrained weights are available on Hugging Face for two backbone families:
+
+- **HGNetv2** (smaller/faster): `atto`, `femto`, `pico`, `n`
+- **DINOv3** (larger/more accurate): `s`, `m`, `l`, `x`
+
+Set `BACKBONE` and `MODEL_SIZE` in the first line to match your desired variant. Hugging Face model names use uppercase (e.g. `HGNetv2_N`, `DINOv3_S`), while config files use lowercase (e.g. `hgnetv2_n`, `dinov3_s`).
+
+```sh
+docker build . --rm --build-arg BACKBONE=hgnetv2 --build-arg MODEL_SIZE=n --output . -f- <<'EOF'
+FROM python:3.11-slim AS build
+RUN apt-get update && apt-get install --no-install-recommends -y git libgl1 libglib2.0-0 && rm -rf /var/lib/apt/lists/*
+COPY --from=ghcr.io/astral-sh/uv:0.8.0 /uv /bin/
+WORKDIR /deimv2
+RUN git clone https://github.com/Intellindust-AI-Lab/DEIMv2.git .
+# Install CPU-only PyTorch first to avoid pulling CUDA variant
+RUN uv pip install --no-cache --system torch torchvision --index-url https://download.pytorch.org/whl/cpu
+RUN uv pip install --no-cache --system -r requirements.txt
+RUN uv pip install --no-cache --system onnx safetensors huggingface_hub
+RUN mkdir -p output
+ARG BACKBONE
+ARG MODEL_SIZE
+# Download from Hugging Face and convert safetensors to pth
+RUN python3 -c "\
+from huggingface_hub import hf_hub_download; \
+from safetensors.torch import load_file; \
+import torch; \
+backbone = '${BACKBONE}'.replace('hgnetv2','HGNetv2').replace('dinov3','DINOv3'); \
+size = '${MODEL_SIZE}'.upper(); \
+st = load_file(hf_hub_download('Intellindust/DEIMv2_' + backbone + '_' + size + '_COCO', 'model.safetensors')); \
+torch.save({'model': st}, 'output/deimv2.pth')"
+RUN sed -i "s/data = torch.rand(2/data = torch.rand(1/" tools/deployment/export_onnx.py
+# HuggingFace safetensors omits frozen constants that the model constructor initializes
+RUN sed -i "s/cfg.model.load_state_dict(state)/cfg.model.load_state_dict(state, strict=False)/" tools/deployment/export_onnx.py
+RUN python3 tools/deployment/export_onnx.py -c configs/deimv2/deimv2_${BACKBONE}_${MODEL_SIZE}_coco.yml -r output/deimv2.pth
+FROM scratch
+ARG BACKBONE
+ARG MODEL_SIZE
+COPY --from=build /deimv2/output/deimv2.onnx /deimv2_${BACKBONE}_${MODEL_SIZE}.onnx
+EOF
+```
+
 ### Downloading RF-DETR Model

 RF-DETR can be exported as ONNX by running the command below. You can copy and paste the whole thing to your terminal and execute, altering `MODEL_SIZE=Nano` in the first line to `Nano`, `Small`, or `Medium` size.