frigate/docs/docs/configuration/detectors.md

id	title
detectors	Detectors

Frigate provides the following builtin detector types: cpu, edgetpu, and openvino. By default, Frigate will use a single CPU detector. Other detectors may require additional configuration as described below. When using multiple detectors they will run in dedicated processes, but pull from a common queue of detection requests from across all cameras.

Note: There is not yet support for Nvidia GPUs to perform object detection with tensorflow. It can be used for ffmpeg decoding, but not object detection.

CPU Detector (not recommended)

The CPU detector type runs a TensorFlow Lite model utilizing the CPU without hardware acceleration. It is recommended to use a hardware accelerated detector type instead for better performance. To configure a CPU based detector, set the "type" attribute to "cpu".

The number of threads used by the interpreter can be specified using the "num_threads" attribute, and defaults to 3.

A TensorFlow Lite model is provided in the container at /cpu_model.tflite and is used by this detector type by default. To provide your own model, bind mount the file into the container and provide the path with model.path.

detectors:
  cpu1:
    type: cpu
    num_threads: 3
    model:
      path: "/custom_model.tflite"
  cpu2:
    type: cpu
    num_threads: 3

When using CPU detectors, you can add one CPU detector per camera. Adding more detectors than the number of cameras should not improve performance.

Edge-TPU Detector

The EdgeTPU detector type runs a TensorFlow Lite model utilizing the Google Coral delegate for hardware acceleration. To configure an EdgeTPU detector, set the "type" attribute to "edgetpu".

The EdgeTPU device can specified using the "device" attribute according to the Documentation for the TensorFlow Lite Python API. If not set, the delegate will use the first device it finds.

A TensorFlow Lite model is provided in the container at /edgetpu_model.tflite and is used by this detector type by default. To provide your own model, bind mount the file into the container and provide the path with model.path.

Single USB Coral

detectors:
  coral:
    type: edgetpu
    device: usb
    model:
      path: "/custom_model.tflite"

Multiple USB Corals

detectors:
  coral1:
    type: edgetpu
    device: usb:0
  coral2:
    type: edgetpu
    device: usb:1

Native Coral (Dev Board)

warning: may have compatibility issues after v0.9.x

detectors:
  coral:
    type: edgetpu
    device: ""

Multiple PCIE/M.2 Corals

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1

Mixing Corals

detectors:
  coral_usb:
    type: edgetpu
    device: usb
  coral_pci:
    type: edgetpu
    device: pci

OpenVINO Detector

The OpenVINO detector type runs an OpenVINO IR model on Intel CPU, GPU and VPU hardware. To configure an OpenVINO detector, set the "type" attribute to "openvino".

The OpenVINO device plugin is specified using the "device" attribute according to naming conventions in the Device Documentation. Other supported devices could be AUTO, CPU, GPU, MYRIAD, etc. If not specified, the default OpenVINO device will be selected by the AUTO plugin.

OpenVINO is supported on 6th Gen Intel platforms (Skylake) and newer. A supported Intel platform is required to use the GPU device with OpenVINO. The MYRIAD device may be run on any platform, including Arm devices. For detailed system requirements, see OpenVINO System Requirements

An OpenVINO model is provided in the container at /openvino-model/ssdlite_mobilenet_v2.xml and is used by this detector type by default. The model comes from Intel's Open Model Zoo SSDLite MobileNet V2 and is converted to an FP16 precision IR model. Use the model configuration shown below when using the OpenVINO detector.

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

Intel NCS2 VPU and Myriad X Setup

Intel produces a neural net inference accelleration chip called Myriad X. This chip was sold in their Neural Compute Stick 2 (NCS2) which has been discontinued. If intending to use the MYRIAD device for accelleration, additional setup is required to pass through the USB device. The host needs a udev rule installed to handle the NCS2 device.

sudo usermod -a -G users "$(whoami)"
cat <<EOF > 97-myriad-usbboot.rules
SUBSYSTEM=="usb", ATTRS{idProduct}=="2485", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0666", ENV{ID_MM_DEVICE_IGNORE}="1"
SUBSYSTEM=="usb", ATTRS{idProduct}=="f63b", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0666", ENV{ID_MM_DEVICE_IGNORE}="1"
EOF
sudo cp 97-myriad-usbboot.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules
sudo udevadm trigger

Additionally, the Frigate docker container needs to run with the following configuration:

--device-cgroup-rule='c 189:\* rmw' -v /dev/bus/usb:/dev/bus/usb

or in your compose file:

device_cgroup_rules:
  - 'c 189:* rmw'
volumes:
  - /dev/bus/usb:/dev/bus/usb

Custom Detectors

Custom python detector modules can be added at /opt/frigate/frigate/detectors/plugins in the container. See the below for an example implementation.

from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig
from typing import Literal
from pydantic import Extra, Field

DETECTOR_KEY = "<detector_key>"

# A pydantic model inheriting from `BaseDetectorConfig`
# Must implement the `type` attribute as below
# Can add any number of fields to be passed from 
# configuration to the constructor of your detector.
class CustomDetectorConfig(BaseDetectorConfig):
    type: Literal[DETECTOR_KEY]
    custom_field: str = Field(default="value", title="Custom field description")


# The custom detector class must inherit from `DetectionApi`
# and must implement the `type_key`, `__init__`, 
# and `detect_raw` methods as below
class CustomDetector(DetectionApi):
    type_key = DETECTOR_KEY

    def __init__(self, detector_config: CustomDetectorConfig):
      ...

    def detect_raw(self, tensor_input):
      return <list of detections>