* Add Axelera Metis NPU object detector plugin

Runs Frigate inference on the Axelera Metis NPU through the vendor's
low-level axelera.runtime API with pip-installed, sha256-pinned wheels
from Axelera's public artifactory (self-contained: auditwheel-repaired
libs + bundled firmware).

- async send_input/receive_output contract (FIFO pairing, overload emits
  zero rows instead of raising, frames are copied out of camera shm)
- generic YOLO-family decoder driven by the compiled model's manifest.json
  (padded input geometry, dequant params, baked postprocess graph via onnx)
- labels come from the compiled model's model_info.json; empty labelmaps
  fail loudly at init

* object_detection: async runner path for the axelera detector

* detectors: detect_axelera probe (/dev/metis*) + device node init

* stats: Axelera Metis board temperature poller

The card exposes no utilization counter usable next to a live detector, so
the poller reports the board-controller temperature read via axcmd (cached
5s; the read must not race detector init).

* docs: Axelera Metis detector (install, config, model catalog, downloads)

* axelera: expose Metis colon-name device node and image deps for runtime enumeration

The Axelera runtime enumerates Metis cards via the colon-separated node
name created by the vendor udev rule on the host (e.g. /dev/metis-0:1:0)
and probes kernel state with lsmod/lspci. Containers only receive the
raw /dev/metis-0-1-0 node, so the detector never found the card.

- init-devices: create /dev/metis-BB:DD:F aliases for each Metis node
- install_deps: add kmod and pciutils so the runtime's driver check works

* axelera: review round 2 (downloader, stop release, pixel format, docs size, per-class NMS)

- download model zips through ModelDownloader (atomic .part rename; a
  truncated download no longer poisons the cache)
- stop(): release the Context (releases connection/model/instance); the
  previous instance.free() call never existed in the runtime API and was
  silently swallowed
- honor input_pixel_format: flip to RGB planes only when Frigate delivers
  BGR (input_pixel_format: bgr); with rgb the frame passes through as-is
- docs/catalog: width/height = compiled 640x640 input (square regions no
  longer get squashed twice), document input_pixel_format: bgr
- NMS via xyxy_to_xywh_for_nms + per-class cv2.dnn.NMSBoxes, matching the
  other detector plugins instead of a hand-rolled class-agnostic pass

* axelera: report Metis temperature only on the NPU card

The same board readout was surfaced twice on the System page: once as
the detector temperature (get_hardware_temperatures) and once on the
NPU card (npu_usages). Keep the NPU card, where the value belongs.

* axelera: report AIPU utilization as the detector run duty cycle

The runtime exposes no device-side usage counter (mvm_utilisation is a
configuration limit, not a readout), so the NPU card rendered empty.
ModelInstance.run blocks until the AIPU completes: the detector process
now times each run and publishes busy/wall over a 2s window through a
small file under the tmp cache dir; the stats poller reads it and sets
the npu key. A stale file (detector idle/restarting) reports nothing
rather than a stale number.

* axelera: cut hot-path allocations in preprocess and dequantise

Preprocess reused three fresh arrays per frame (resize output, quantised
copy, strided channel flip at write time) and refilled the whole padded
input buffer. Now resize writes into a scratch with cv2 dst=, the BGR->RGB
swap is a cvtColor into scratch (no strided copy), the int8 quantise runs
with out=, and only the border strips the content rect does not cover get
memset. Measured 39% faster on the 704x576 -> 640x640 path.

Dequantise allocated two fp32 temporaries per head (astype then the
dequant expression) before the transpose copy. copyto now casts int8->fp32
while transposing straight into a reused fp32 feed buffer, and the zp/scale
math runs in place. Measured 78% faster for the YOLOX-S head set, outputs
bit-identical (int8->fp32 cast is lossless).

Argmax+gather replaces max+argmax (one class-axis reduction instead of
two), and the postprocess session pins intra_op_num_threads=1: one frame
is in flight through the decoder, extra ORT threads only add context
switches.

* axelera: publish the run duty cycle through a shared Value

Replace the tmp-file duty-cycle handoff (and the FRIGATE_TMP env var
nothing else reads) with a third multiprocessing.Value alongside
avg_inference_speed and detection_start: ObjectDetectProcess creates it,
AsyncDetectorRunner passes it to the detector, and the plugin publishes
busy/wall of the blocking ModelInstance.run over a 2s window, mirroring
how memryx receives its shared state. No filesystem path, no staleness
window, no atomic rename, and stats/hardware.py no longer imports the
plugin module to find the file.

The producer also ticks the window while the queue is empty, so the duty
cycle falls to 0 when the card idles instead of sticking at its last
value.

Name it what it is: host-side occupancy of the run call, including DMA
and driver time, not an AIPU core counter (the runtime exposes no
device-side one). The NPU entry now reads temperature from the board
controller plus the detector-reported run duty cycle folded in where
the detector processes are in reach.

* axelera: enumerate devices explicitly and fail with an actionable error

Context.list_devices() keys off the lspci BDF and needs the colon-form
node to be present, so surface an empty enumeration at init with the
fix-it hint instead of a bare device_connect(None) ConnectionError
several frames deeper. Uses the DeviceInfo the runtime enumerated
rather than relying on the pick-first implicit path.

* axelera: deliver every accepted result under backpressure, reset duty cycle on restart

Greptile P1: _push_result dropped a completion when the result queue was
full, stranding the runner's send_times entry and the input SHM mapping
(1 submission : 1 result contract). Replace the drop with a stop-aware
blocking put so an accepted submission always produces exactly one
delivered completion.

Greptile P2: start_or_restart did not clear run_duty_cycle when starting
a replacement process, so stats could show the previous process's duty
cycle while the new detector was still initializing. Reset it to the
-1.0 sentinel alongside detection_start, inside the async branch.

Greptile P2 (offline docs): list axelera among the detector types whose
pre-seeded runtime files live under /config/model_cache/runtimes/<type>.

* axelera: reviewer round 3 (plugin-owned enumeration tools, generic DEVICE_ALIASES, generic duty-cycle fold)

install_deps.sh: drop kmod/pciutils from the base image. The runtime's
list_devices() only greps popen'd lspci/lsmod output over /sys and /proc,
so the plugin now installs shell stand-ins in its own
/config/model_cache/axelera/bin and prepends it to PATH when the real
tools are missing (hardware-verified: the stand-in lspci prints the
Device 1f9d:<id> line the runtime matches, the lsmod shape matches
/proc/modules).

init-devices: replace the Metis-keyed alias block with a generic
DEVICE_ALIASES env var (comma separated alias=target pairs, next to
DEVICE_ACL_PATHS). Docker rejects colon-named nodes in --device AND in
bind mounts (probed: 'too many colons' on both the long volume form and
the mount form), so the in-container symlink stays necessary; docs carry
the pair users must set.

stats/util.py: fold any detector's published run_duty_cycle into the
matching npu_usages entry generically; _axelera_run_duty_cycle and the
per-vendor branch are gone.

plugin: the no-device RuntimeError now names the raw dash-form node
(--device rejects colons, users can only pass the raw node) and points
at DEVICE_ALIASES for the colon alias.

* axelera: reviewer round 4 (aggregate shared-type duty cycles, american spellings)

stats/util.py: several detector processes can share one hardware type (e.g.
two axelera cards on one host). Iterating and assigning made the displayed
NPU usage depend on dict order, last writer wins. Collect the published
windows per type and write the busiest one, matching what the removed
per-vendor helper did with max().

plugin: the remaining British spellings (dequantise family) corrected to
the American forms the repo requires.

plugin: model path resolution follows hailo8l's single-string pattern (URL
or local path or preset name in one value), the download uses urllib
directly like memryx, and _find_model_json now raises on an ambiguous
multi-model directory instead of returning whichever sorts first.

plugin: AxeleraDetector exposes shutdown() (the async contract hook base.py
calls) instead of close(); __del__ guards a partially initialised instance.

* axelera: download URL models through the atomic ModelDownloader

* axelera: bring the detector unit tests in tree

* init-devices: create DEVICE_ALIASES before the ACL-mode exits

An alias is not an ACL grant: users running with FRIGATE_RUN_AS_ROOT or
FRIGATE_DEVICE_ACLS=false still need the colon-form node the runtime
enumerates, so create the aliases right after the euid check. Document
DEVICE_ALIASES next to DEVICE_ACL_PATHS and note it cannot work under
docker's user: mode (the container cannot write to /dev).

* detectors: cover the Axelera Metis probe in test_detector_hardware

* Update docs/docs/configuration/non_root.md

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* axelera: address the 2026-10-12 review round

- hash directory models deterministically in compute_model_hash instead of
  raising IsADirectoryError at startup
- keep canonical model labels addressable after user renames in the
  labelmap lookup (two classes can share one renamed name)
- key the URL model cache by source and archive content, extract replaced
  archives into fresh directories, prune superseded variants conservatively
- keep publishing the run duty cycle when the hardware poll has no entry
  (duty-only npu_usages entry instead of a hidden detector)
- document the user: mode device reach instead of host-only alias creation

* axelera: prune only generated cache entries

A fresh download matched user-provided layouts (the docs-recommended
<name>/ directory, <name>.zip archives) and look-alike model names
(yolox.zip vs yolox-s-*) by prefix. Pruning is now gated on the exact
generated identity pattern <name>-<source>-<32 hex digits>, so downloads
never touch user-provided directories, other models, or other sources.

* axelera: narrow the model path design to memryx-style zips

- path is always a .zip of the compiled output (local or http(s) URL),
  extracted fresh on every start under a lock, like the memryx
  custom-model flow
- _decode returns the raw class index and Frigate's labelmap names it;
  _build_label_lookup and _read_model_labels are gone, and the YOLOX
  objectness check keys off model_type
- scheme-less URL support and the quantized/manifest.json fallback are
  dropped
- width/height must equal the compiled input size; init raises otherwise
  instead of letterboxing
- compute_model_hash is back to the unchanged file-only hashing; the
  .extracted marker, source/content cache-name hashes and
  _prune_model_cache are removed

* Update frigate/test/test_axelera_detector.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* axelera: drop the run duty cycle, match the hailo temperature flow

- no other accelerator reports NPU usage and the published value was time
  blocked in run(), not device utilization from the vendor API; removing
  it leaves the runner selection as the only core change
- the board temperature moves to a get_hardware_temperatures branch like
  hailo, so it renders on the detector card; _poll_axelera, its POLLERS
  entry and the _scan_detectors line are gone
- axelera_runtime2 is not installed: axelera.runtime imports it only
  inside a try for its separate new-runtime API, and the low-level API
  the plugin uses (Context, load_model, device_connect,
  load_model_instance) lives in axelera.runtime.objects

* axelera: follow-ups from the 2026-10-13 review round

- the offline-setup table no longer lists the axelera_runtime2 wheels,
  which are not installed
- a failed board temperature read starts a 60s cooldown so an
  unresponsive axcmd does not delay every stats update

* axelera: drop the model-hash test, fix the model.json error messages

* axelera: drop DEVICE_ALIASES, document the CDI device spec

* axelera: use ln -sf in the CDI hook example

---------

Co-authored-by: Patryk Watkowski <patrykorwat@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

2026-10-20 09:53:28 -05:00

14 KiB

Raw Blame History

id, title

id	title
network_requirements	Network Requirements

Network Requirements

Frigate is designed to run locally and does not require a persistent internet connection for core functionality. However, certain features need internet access for initial setup or ongoing operation. This page describes what connects to the internet, when, and how to control it.

How Frigate Uses the Internet

Frigate's internet usage falls into three categories:

One-time model downloads: ML models are downloaded the first time a feature is enabled, then cached locally. No internet is needed on subsequent startups.
Optional cloud services: Features like Frigate+ and Generative AI connect to external APIs only when explicitly configured.
Build-time dependencies: Components bundled into the Docker image during the build process. These require no internet at runtime.

:::tip

After initial setup, Frigate can run fully offline as long as all required models have been downloaded and no cloud-dependent features are enabled.

:::

One-Time Model Downloads

The following models are downloaded automatically the first time their associated feature is enabled. Once cached in /config/model_cache/, they do not require internet again.

Feature	Models Downloaded	Source
Semantic search	Jina CLIP v1 or v2 (ONNX) + tokenizer	HuggingFace
Face recognition	FaceNet, ArcFace, face detection model	GitHub
License plate recognition	PaddleOCR (detection, classification, recognition) + YOLOv9 plate detector	GitHub
Bird classification	MobileNetV2 bird model + label map	GitHub
Custom classification (training)	MobileNetV2 ImageNet base weights (via Keras)	Google storage
Audio transcription	Whisper or Sherpa-ONNX streaming model	HuggingFace / OpenAI

:::note

The MobileNetV2 base weights are the one exception to the /config/model_cache/ rule. They are also the only entry that is not downloaded when the feature is enabled: Frigate fetches them when a training run actually starts.

:::

Hardware-Specific Detector Models

If you are using one of the following hardware detectors and have not provided your own model file, a default model will be downloaded on first startup:

Detector	Model Downloaded	Source
Rockchip RKNN	RKNN detection model	GitHub
Hailo 8 / 8L / 8R	YOLOv6n (.hef)	Hailo Model Zoo (AWS S3)
AXERA AXEngine	Detection model	HuggingFace

:::note

The default CPU, EdgeTPU, and OpenVINO object detection models are bundled into the Docker image and do not require any download at runtime.

:::

Detector Runtimes

The SDKs for a few hardware detectors are not shipped in the Frigate image. They are downloaded the first time that detector is configured, verified against checksums pinned in the Frigate release, and installed into the Frigate user's home directory (/config/.local by default). Once installed they are not downloaded again until a Frigate release pins a new version.

Detector	Version	Files	Source
Hailo 8 / 8L / 8R	4.21.0	`hailort-debian12-amd64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_x86_64.whl` on x86, `hailort-debian12-arm64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_aarch64.whl` on arm64	GitHub release
MemryX MX3	2.1.0	`mx_accl_frigate-2.1.0.zip` (the release source archive, renamed)	GitHub archive
AXERA AXEngine	0.1.3	`axengine-0.1.3-py3-none-any.whl`	GitHub release
Axelera Metis	1.8.0	`axelera_runtime-1.8.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl` on x86, `axelera_runtime-1.8.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl` on arm64, plus `axelera_firmware-1.8.0-py3-none-any.whl` on both	Axelera Artifactory PyPI

If the container cannot reach GitHub, provide the files yourself:

Download the files for your architecture on a machine with internet access.
Place them, with exactly the file names listed above, in /config/model_cache/runtimes/<detector>/, where <detector> is the detector named in your config's devices (hailo, memryx, axengine, or axelera).
Start Frigate. Files whose checksum matches are installed without any download; a file with the wrong checksum is discarded and downloaded again, so a failed startup log names the file to replace.

The GITHUB_ENDPOINT mirror variable below applies to these downloads as well.

Preventing Model Downloads

If you have already downloaded all required models and want to prevent Frigate from attempting any outbound connections to HuggingFace or the Transformers library, set the following environment variables on your Frigate container:

environment:
  HF_HUB_OFFLINE: "1"
  TRANSFORMERS_OFFLINE: "1"

:::warning

Setting these variables without having the correct model files already cached in /config/model_cache/ will cause failures. Only use these after a successful initial setup with internet access.

:::

Mirror Support

If your Frigate instance has restricted internet access, you can point model downloads at internal mirrors using environment variables:

Environment Variable	Default	Used By
`HF_ENDPOINT`	`https://huggingface.co`	Semantic search, Sherpa-ONNX, AXEngine models
`GITHUB_ENDPOINT`	`https://github.com`	Face recognition, LPR, RKNN models, detector runtimes
`GITHUB_RAW_ENDPOINT`	`https://raw.githubusercontent.com`	Bird classification
`TF_KERAS_MOBILENET_V2_WEIGHTS_URL`	Unset (Keras uses its own default)	Custom classification training

Optional Cloud Services

These features connect to external services during normal operation and require internet whenever they are active.

Frigate+

When a Frigate+ API key is configured, Frigate communicates with https://api.frigate.video to download models, upload snapshots for training, submit annotations, and report false positives. Remove the API key to disable all Frigate+ network activity.

See Frigate+ for details.

Generative AI

When a Generative AI provider is configured, Frigate sends images and prompts to the configured provider for event descriptions, chat, and camera monitoring. Available providers:

Provider	Internet Required
OpenAI	Yes, connects to OpenAI API (or custom base URL)
Google Gemini	Yes, connects to Google Generative AI API
Azure OpenAI	Yes, connects to your Azure endpoint
Ollama	Depends: typically local (`localhost:11434`), but can be remote
llama.cpp	No, runs entirely locally

Disable Generative AI by removing the genai configuration from your cameras. See Generative AI for details.

Version Check

Frigate checks GitHub for the latest release version on startup by querying https://api.github.com. This can be disabled:

telemetry:
  version_check: false

Push Notifications

When notifications are enabled and users have registered for push notifications in the web UI, Frigate sends push messages through the browser vendor's push service (e.g., Google FCM, Mozilla autopush). This requires internet access from the Frigate server to these push endpoints.

MQTT

If an MQTT broker is configured, Frigate maintains a connection to the broker's host and port. This is typically a local network connection, but will require internet if you use a cloud-hosted MQTT broker.

WebRTC (STUN)

For WebRTC live streaming, Frigate uses STUN for NAT traversal:

go2rtc defaults to a local STUN listener (stun:8555), no internet required.
The web UI uses the servers in go2rtc.webrtc.ice_servers for its WebRTC player and for the WebRTC connectivity check it runs when the Live view loads. If none are set, it uses Google's public STUN server (stun:stun.l.google.com:19302), which requires internet access from the browser. Set ice_servers to a STUN or TURN server on your network to avoid this.

Home Assistant Supervisor

When running as a Home Assistant App, the go2rtc startup script queries the local Supervisor API (http://supervisor/) to discover the host IP address and WebRTC port. This is a local network call to the Home Assistant host, not an internet connection.

What Does NOT Require Internet

Object detection: CPU, EdgeTPU, OpenVINO, and other bundled detector models are included in the Docker image.
Recording and playback: All video is stored and served locally.
Live streaming: Camera streams are pulled over your local network. MSE and HLS streaming work without any external connections.
The web interface: Fully self-contained with no external fonts, scripts, analytics, or CDN dependencies. All translations are bundled locally.
Custom classification inference: After training, custom models run entirely locally.
Audio detection: The YAMNet audio classification model is bundled in the Docker image.

Running Frigate Offline

To run Frigate in an air-gapped or offline environment:

Pre-download models: Start Frigate with internet access once with all desired features enabled. Models will be cached in /config/model_cache/.
Pre-download the training base weights: If you plan to train custom classification models, set TF_KERAS_MOBILENET_V2_WEIGHTS_URL before training, then run one training job while online. Without this variable the base weights are cached outside /config/ and are lost whenever the container is recreated, so a later training run will fail offline. If the machine never has internet access, copy the weights in manually as described below.
Disable version check: Set telemetry.version_check: false in your configuration.
Block outbound model requests: Set the HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 environment variables to prevent HuggingFace and Transformers from attempting any network requests.
Avoid cloud features: Do not configure Frigate+, Generative AI providers that require internet, or cloud MQTT brokers.
Use local model mirrors: If limited internet is available, set the HF_ENDPOINT, GITHUB_ENDPOINT, GITHUB_RAW_ENDPOINT, and TF_KERAS_MOBILENET_V2_WEIGHTS_URL environment variables to point to local mirrors.

After these steps, Frigate will operate with no outbound internet connections.

Manually Copying the Training Base Weights

On a machine with internet access, download the weights:

curl -L -o mobilenet_v2_weights.h5 \
  "https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_0.35_224_no_top.h5"

Copy the file into your Frigate config volume as /config/model_cache/MobileNet/mobilenet_v2_weights.h5, keeping that exact filename, then set the environment variable TF_KERAS_MOBILENET_V2_WEIGHTS_URL in your Docker compose file to the URL above and restart Frigate.

The variable must be set even though the URL is never contacted. If it is unset, Frigate ignores the copied file and asks Keras to download the weights instead.

14 KiB Raw Blame History