Apple Silicon / ZMQ Detector (#19592)

* Add zmq detector * Cleanup * Logging * Cleanup * Cleanup * Add to hardware docs * Add apple silicon to docs * Formatting
2026-08-03 01:22:17 +03:00 · 2025-08-18 09:51:12 -05:00
parent 5a49d1f73c
commit 152d9ed4a0
3 changed files with 238 additions and 9 deletions
@@ -19,6 +19,10 @@ Frigate supports multiple different detectors that work on different types of ha
 - [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection.
 - [ONNX](#onnx): ROCm will automatically be detected and used as a detector in the `-rocm` Frigate image when a supported ONNX model is configured.

+**Apple Silicon**
+
+- [Apple Silicon](#apple-silicon-detector): Apple Silicon can run on M1 and newer Apple Silicon devices.
+
 **Intel**

 - [OpenVino](#openvino-detector): OpenVino can run on Intel Arc GPUs, Intel integrated GPUs, and Intel CPUs to provide efficient object detection.
@@ -264,7 +268,7 @@ detectors:

 :::

-### Supported Models
+### OpenVINO Supported Models

 #### SSDLite MobileNet v2

@@ -402,6 +406,59 @@ model:

 Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.

+## Apple Silicon detector
+
+The NPU in Apple Silicon can't be accessed from within a container, so the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) must fist be setup. It is recommended to use the Frigate docker image with `-standard-arm64` suffix, for example  `ghcr.io/blakeblackshear/frigate:stable-arm64-standard`.
+
+### Setup
+
+1. Setup the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) and run the client
+2. Configure the detector in Frigate and startup Frigate
+
+### Configuration
+
+Using the detector config below will connect to the client:
+
+```yaml
+detectors:
+  apple-silicon:
+    type: zmq
+    endpoint: tcp://host.docker.internal:5555
+```
+
+### Apple Silicon Supported Models
+
+There is no default model provided, the following formats are supported:
+
+#### YOLO (v3, v4, v7, v9)
+
+YOLOv3, YOLOv4, YOLOv7, and [YOLOv9](https://github.com/WongKinYiu/yolov9) models are supported, but not included by default.
+
+:::tip
+
+The YOLO detector has been designed to support YOLOv3, YOLOv4, YOLOv7, and YOLOv9 models, but may support other YOLO model architectures as well. See [the models section](#downloading-yolo-models) for more information on downloading YOLO models for use in Frigate.
+
+:::
+
+After placing the downloaded onnx model in your config folder, you can use the following configuration:
+
+```yaml
+detectors:
+  onnx:
+    type: onnx
+
+model:
+  model_type: yolo-generic
+  width: 320 # <--- should match the imgsize set during model export
+  height: 320 # <--- should match the imgsize set during model export
+  input_tensor: nchw
+  input_dtype: float
+  path: /config/model_cache/yolo.onnx
+  labelmap_path: /labelmap/coco-80.txt
+```
+
+Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
+
 ## AMD/ROCm GPU detector

 ### Setup
@@ -483,7 +540,7 @@ We unset the `HSA_OVERRIDE_GFX_VERSION` to prevent an existing override from mes
 $ docker exec -it frigate /bin/bash -c '(unset HSA_OVERRIDE_GFX_VERSION && /opt/rocm/bin/rocminfo |grep gfx)'
 ```

-### Supported Models
+### ROCm Supported Models

 See [ONNX supported models](#supported-models) for supported models, there are some caveats:

@@ -526,7 +583,7 @@ detectors:

 :::

-### Supported Models
+### ONNX Supported Models

 There is no default model provided, the following formats are supported:

@@ -824,7 +881,7 @@ $ cat /sys/kernel/debug/rknpu/load

 :::

-### Supported Models
+### RockChip Supported Models

 This `config.yml` shows all relevant options to configure the detector and explains them. All values shown are the default values (except for two). Lines that are required at least to use the detector are labeled as required, all other lines are optional.

@@ -61,19 +61,26 @@ Frigate supports multiple different detectors that work on different types of ha
 **AMD**

 - [ROCm](#rocm---amd-gpu): ROCm can run on AMD Discrete GPUs to provide efficient object detection
-  - [Supports limited model architectures](../../configuration/object_detectors#supported-models-1)
+  - [Supports limited model architectures](../../configuration/object_detectors#rocm-supported-models)
  - Runs best on discrete AMD GPUs

+**Apple Silicon**
+
+- [Apple Silicon](#apple-silicon): Apple Silicon is usable on all M1 and newer Apple Silicon devices to provide efficient and fast object detection
+  - [Supports primarily ssdlite and mobilenet model architectures](../../configuration/object_detectors#apple-silicon-supported-models)
+  - Runs well with any size models including large
+  - Runs via ZMQ proxy which adds some latency, only recommended for local connection
+
 **Intel**

 - [OpenVino](#openvino---intel): OpenVino can run on Intel Arc GPUs, Intel integrated GPUs, and Intel CPUs to provide efficient object detection.
-  - [Supports majority of model architectures](../../configuration/object_detectors#supported-models)
+  - [Supports majority of model architectures](../../configuration/object_detectors#openvino-supported-models)
  - Runs best with tiny, small, or medium models

 **Nvidia**

 - [TensortRT](#tensorrt---nvidia-gpu): TensorRT can run on Nvidia GPUs and Jetson devices.
-  - [Supports majority of model architectures via ONNX](../../configuration/object_detectors#supported-models-2)
+  - [Supports majority of model architectures via ONNX](../../configuration/object_detectors#onnx-supported-models)
  - Runs well with any size models including large

 **Rockchip**
@@ -173,14 +180,28 @@ Inference speeds will vary greatly depending on the GPU and the model used.
 | RTX A4000       |                       | 320: ~ 15 ms              |                        |
 | Tesla P40       |                       | 320: ~ 105 ms             |                        |

+### Apple Silicon
+
+With the [Apple Silicon](../configuration/object_detectors.md#apple-silicon-detector) detector Frigate can take advantage of the NPU in M1 and newer Apple Silicon.
+
+:::warning
+
+Apple Silicon can not run within a container, so a ZMQ proxy is utilized to communicate with [the Apple Silicon Frigate detector](https://github.com/frigate-nvr/apple-silicon-detector) which runs on the host. This should add minimal latency when run on the same device.
+
+:::
+
+| Name      | YOLOv9 Inference Time  |
+| --------- | ---------------------- |
+| M3 Pro    | t-320: 6 ms s-320: 8ms |
+| M1        | s-320: 9ms             |
+
 ### ROCm - AMD GPU

-With the [rocm](../configuration/object_detectors.md#amdrocm-gpu-detector) detector Frigate can take advantage of many discrete AMD GPUs.
+With the [ROCm](../configuration/object_detectors.md#amdrocm-gpu-detector) detector Frigate can take advantage of many discrete AMD GPUs.

 | Name      | YOLOv9 Inference Time | YOLO-NAS Inference Time   |
 | --------- | --------------------- | ------------------------- |
 | AMD 780M  | ~ 14 ms               | 320: ~ 25 ms 640: ~ 50 ms |
-| AMD 8700G |                       | 320: ~ 20 ms 640: ~ 40 ms |

 ## Community Supported Detectors