* Add Axelera Metis NPU object detector plugin
Runs Frigate inference on the Axelera Metis NPU through the vendor's
low-level axelera.runtime API with pip-installed, sha256-pinned wheels
from Axelera's public artifactory (self-contained: auditwheel-repaired
libs + bundled firmware).
- async send_input/receive_output contract (FIFO pairing, overload emits
zero rows instead of raising, frames are copied out of camera shm)
- generic YOLO-family decoder driven by the compiled model's manifest.json
(padded input geometry, dequant params, baked postprocess graph via onnx)
- labels come from the compiled model's model_info.json; empty labelmaps
fail loudly at init
* object_detection: async runner path for the axelera detector
* detectors: detect_axelera probe (/dev/metis*) + device node init
* stats: Axelera Metis board temperature poller
The card exposes no utilization counter usable next to a live detector, so
the poller reports the board-controller temperature read via axcmd (cached
5s; the read must not race detector init).
* docs: Axelera Metis detector (install, config, model catalog, downloads)
* axelera: expose Metis colon-name device node and image deps for runtime enumeration
The Axelera runtime enumerates Metis cards via the colon-separated node
name created by the vendor udev rule on the host (e.g. /dev/metis-0:1:0)
and probes kernel state with lsmod/lspci. Containers only receive the
raw /dev/metis-0-1-0 node, so the detector never found the card.
- init-devices: create /dev/metis-BB:DD:F aliases for each Metis node
- install_deps: add kmod and pciutils so the runtime's driver check works
* axelera: review round 2 (downloader, stop release, pixel format, docs size, per-class NMS)
- download model zips through ModelDownloader (atomic .part rename; a
truncated download no longer poisons the cache)
- stop(): release the Context (releases connection/model/instance); the
previous instance.free() call never existed in the runtime API and was
silently swallowed
- honor input_pixel_format: flip to RGB planes only when Frigate delivers
BGR (input_pixel_format: bgr); with rgb the frame passes through as-is
- docs/catalog: width/height = compiled 640x640 input (square regions no
longer get squashed twice), document input_pixel_format: bgr
- NMS via xyxy_to_xywh_for_nms + per-class cv2.dnn.NMSBoxes, matching the
other detector plugins instead of a hand-rolled class-agnostic pass
* axelera: report Metis temperature only on the NPU card
The same board readout was surfaced twice on the System page: once as
the detector temperature (get_hardware_temperatures) and once on the
NPU card (npu_usages). Keep the NPU card, where the value belongs.
* axelera: report AIPU utilization as the detector run duty cycle
The runtime exposes no device-side usage counter (mvm_utilisation is a
configuration limit, not a readout), so the NPU card rendered empty.
ModelInstance.run blocks until the AIPU completes: the detector process
now times each run and publishes busy/wall over a 2s window through a
small file under the tmp cache dir; the stats poller reads it and sets
the npu key. A stale file (detector idle/restarting) reports nothing
rather than a stale number.
* axelera: cut hot-path allocations in preprocess and dequantise
Preprocess reused three fresh arrays per frame (resize output, quantised
copy, strided channel flip at write time) and refilled the whole padded
input buffer. Now resize writes into a scratch with cv2 dst=, the BGR->RGB
swap is a cvtColor into scratch (no strided copy), the int8 quantise runs
with out=, and only the border strips the content rect does not cover get
memset. Measured 39% faster on the 704x576 -> 640x640 path.
Dequantise allocated two fp32 temporaries per head (astype then the
dequant expression) before the transpose copy. copyto now casts int8->fp32
while transposing straight into a reused fp32 feed buffer, and the zp/scale
math runs in place. Measured 78% faster for the YOLOX-S head set, outputs
bit-identical (int8->fp32 cast is lossless).
Argmax+gather replaces max+argmax (one class-axis reduction instead of
two), and the postprocess session pins intra_op_num_threads=1: one frame
is in flight through the decoder, extra ORT threads only add context
switches.
* axelera: publish the run duty cycle through a shared Value
Replace the tmp-file duty-cycle handoff (and the FRIGATE_TMP env var
nothing else reads) with a third multiprocessing.Value alongside
avg_inference_speed and detection_start: ObjectDetectProcess creates it,
AsyncDetectorRunner passes it to the detector, and the plugin publishes
busy/wall of the blocking ModelInstance.run over a 2s window, mirroring
how memryx receives its shared state. No filesystem path, no staleness
window, no atomic rename, and stats/hardware.py no longer imports the
plugin module to find the file.
The producer also ticks the window while the queue is empty, so the duty
cycle falls to 0 when the card idles instead of sticking at its last
value.
Name it what it is: host-side occupancy of the run call, including DMA
and driver time, not an AIPU core counter (the runtime exposes no
device-side one). The NPU entry now reads temperature from the board
controller plus the detector-reported run duty cycle folded in where
the detector processes are in reach.
* axelera: enumerate devices explicitly and fail with an actionable error
Context.list_devices() keys off the lspci BDF and needs the colon-form
node to be present, so surface an empty enumeration at init with the
fix-it hint instead of a bare device_connect(None) ConnectionError
several frames deeper. Uses the DeviceInfo the runtime enumerated
rather than relying on the pick-first implicit path.
* axelera: deliver every accepted result under backpressure, reset duty cycle on restart
Greptile P1: _push_result dropped a completion when the result queue was
full, stranding the runner's send_times entry and the input SHM mapping
(1 submission : 1 result contract). Replace the drop with a stop-aware
blocking put so an accepted submission always produces exactly one
delivered completion.
Greptile P2: start_or_restart did not clear run_duty_cycle when starting
a replacement process, so stats could show the previous process's duty
cycle while the new detector was still initializing. Reset it to the
-1.0 sentinel alongside detection_start, inside the async branch.
Greptile P2 (offline docs): list axelera among the detector types whose
pre-seeded runtime files live under /config/model_cache/runtimes/<type>.
* axelera: reviewer round 3 (plugin-owned enumeration tools, generic DEVICE_ALIASES, generic duty-cycle fold)
install_deps.sh: drop kmod/pciutils from the base image. The runtime's
list_devices() only greps popen'd lspci/lsmod output over /sys and /proc,
so the plugin now installs shell stand-ins in its own
/config/model_cache/axelera/bin and prepends it to PATH when the real
tools are missing (hardware-verified: the stand-in lspci prints the
Device 1f9d:<id> line the runtime matches, the lsmod shape matches
/proc/modules).
init-devices: replace the Metis-keyed alias block with a generic
DEVICE_ALIASES env var (comma separated alias=target pairs, next to
DEVICE_ACL_PATHS). Docker rejects colon-named nodes in --device AND in
bind mounts (probed: 'too many colons' on both the long volume form and
the mount form), so the in-container symlink stays necessary; docs carry
the pair users must set.
stats/util.py: fold any detector's published run_duty_cycle into the
matching npu_usages entry generically; _axelera_run_duty_cycle and the
per-vendor branch are gone.
plugin: the no-device RuntimeError now names the raw dash-form node
(--device rejects colons, users can only pass the raw node) and points
at DEVICE_ALIASES for the colon alias.
* axelera: reviewer round 4 (aggregate shared-type duty cycles, american spellings)
stats/util.py: several detector processes can share one hardware type (e.g.
two axelera cards on one host). Iterating and assigning made the displayed
NPU usage depend on dict order, last writer wins. Collect the published
windows per type and write the busiest one, matching what the removed
per-vendor helper did with max().
plugin: the remaining British spellings (dequantise family) corrected to
the American forms the repo requires.
plugin: model path resolution follows hailo8l's single-string pattern (URL
or local path or preset name in one value), the download uses urllib
directly like memryx, and _find_model_json now raises on an ambiguous
multi-model directory instead of returning whichever sorts first.
plugin: AxeleraDetector exposes shutdown() (the async contract hook base.py
calls) instead of close(); __del__ guards a partially initialised instance.
* axelera: download URL models through the atomic ModelDownloader
* axelera: bring the detector unit tests in tree
* init-devices: create DEVICE_ALIASES before the ACL-mode exits
An alias is not an ACL grant: users running with FRIGATE_RUN_AS_ROOT or
FRIGATE_DEVICE_ACLS=false still need the colon-form node the runtime
enumerates, so create the aliases right after the euid check. Document
DEVICE_ALIASES next to DEVICE_ACL_PATHS and note it cannot work under
docker's user: mode (the container cannot write to /dev).
* detectors: cover the Axelera Metis probe in test_detector_hardware
* Update docs/docs/configuration/non_root.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* axelera: address the 2026-10-12 review round
- hash directory models deterministically in compute_model_hash instead of
raising IsADirectoryError at startup
- keep canonical model labels addressable after user renames in the
labelmap lookup (two classes can share one renamed name)
- key the URL model cache by source and archive content, extract replaced
archives into fresh directories, prune superseded variants conservatively
- keep publishing the run duty cycle when the hardware poll has no entry
(duty-only npu_usages entry instead of a hidden detector)
- document the user: mode device reach instead of host-only alias creation
* axelera: prune only generated cache entries
A fresh download matched user-provided layouts (the docs-recommended
<name>/ directory, <name>.zip archives) and look-alike model names
(yolox.zip vs yolox-s-*) by prefix. Pruning is now gated on the exact
generated identity pattern <name>-<source>-<32 hex digits>, so downloads
never touch user-provided directories, other models, or other sources.
* axelera: narrow the model path design to memryx-style zips
- path is always a .zip of the compiled output (local or http(s) URL),
extracted fresh on every start under a lock, like the memryx
custom-model flow
- _decode returns the raw class index and Frigate's labelmap names it;
_build_label_lookup and _read_model_labels are gone, and the YOLOX
objectness check keys off model_type
- scheme-less URL support and the quantized/manifest.json fallback are
dropped
- width/height must equal the compiled input size; init raises otherwise
instead of letterboxing
- compute_model_hash is back to the unchanged file-only hashing; the
.extracted marker, source/content cache-name hashes and
_prune_model_cache are removed
* Update frigate/test/test_axelera_detector.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* axelera: drop the run duty cycle, match the hailo temperature flow
- no other accelerator reports NPU usage and the published value was time
blocked in run(), not device utilization from the vendor API; removing
it leaves the runner selection as the only core change
- the board temperature moves to a get_hardware_temperatures branch like
hailo, so it renders on the detector card; _poll_axelera, its POLLERS
entry and the _scan_detectors line are gone
- axelera_runtime2 is not installed: axelera.runtime imports it only
inside a try for its separate new-runtime API, and the low-level API
the plugin uses (Context, load_model, device_connect,
load_model_instance) lives in axelera.runtime.objects
* axelera: follow-ups from the 2026-10-13 review round
- the offline-setup table no longer lists the axelera_runtime2 wheels,
which are not installed
- a failed board temperature read starts a 60s cooldown so an
unresponsive axcmd does not delay every stats update
* axelera: drop the model-hash test, fix the model.json error messages
* axelera: drop DEVICE_ALIASES, document the CDI device spec
* axelera: use ln -sf in the CDI hook example
---------
Co-authored-by: Patryk Watkowski <patrykorwat@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Run ONNX models on a Mac's Neural Engine through lighter's plugin provider
lighter's lighter.sh/ane device places an ONNX Runtime plugin execution
provider in the container. When it is present, the ONNX session setup
registers it once and opens sessions on its Neural Engine device, the same
place CUDA, ROCm and OpenVINO are chosen, so the onnx detector (and any
model that is not pinned to the CPU) runs there with no configuration. The
hardware probe reports it as an onnx unit.
* docs: hardware decode on an Apple Silicon Mac under lighter
A community section on the video decoding page: lighter's lighter.sh/video
device, hwaccel_args -c:v h264_v4l2m2m, and why the Raspberry Pi presets
decode a single-stream camera in software. The detector docs link to it.
* docs: set the lighter decoder per camera when codecs are mixed
* docs: the ONNX detector on a Mac's Neural Engine under lighter
* Format the Neural Engine provider setup
* Fall back to the default providers when the Neural Engine cannot load a model
* Apple Silicon ffmpeg presets for lighter's media engine, recommended when it is present
2026-09-24 15:37:32 -06:00
A. AhmetGitHubgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(deepx): add DEEPX NPU detector and runtime integration.
* feat(deepx): enforce model_format requirement when ppu is enabled and add integrity checks for driver installation
* Update frigate/detectors/plugins/deepx.py
Public method lacks docstring
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Refactor DEEPX detector tests, support SSD and DAMO-YOLO
* feat(deepx): add anchor-free output decoding and corresponding tests
* Add tests and updates for DEEPX detector and refactor DEEPX accelerator code structure.
* fix: enhance model type validation and update documentation for DEEPX detector
* fix: add support for customizable score and NMS thresholds
* refactor: infer YOLO layout from the model, drop per-detector options and the dxrtd placeholder
* fix: keep only the anchor-free PPU verdict, re-read anchor-based each frame
* Update latency data for DEEPX NPU
* Expanding PPU support for DEEPX and set yolo-generic as default.
* enhance scale count resolution logic
* Extend PPU layout handling and YOLOX support to DEEPX detector
* fix: assume the largest PPU anchor table when the .dxnn has no layout
* Improve PPU decoding and introduce strides handling
* Improve PPU scope and fix box format mismatch
* Fix unnamed node issue that breaks traversal
* fix: update object detection model type description to remove outdated architecture
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* add a notice registry and System Health tab
Problems Frigate detects on its own (ffmpeg crash loops, stuck detectors, failed model downloads, recordings deleted before their retention period) only ever existed as log lines. This adds a `NoticeRegistry` in the main process backed by two tables, an `update_notice` IPC topic so producers in other processes can reach it through the dispatcher, an admin-only API and websocket topic, and a Health tab that lists them. Kinds declare their own mode, severity, and category in one place: state notices are resolved by their producer, event notices are dismissed by the user.
* treat a prerelease as behind its final release
* fix notices clearing early
* rename menu items and update docs
* don't resolve the update notice on a failed version lookup
* add host npu requirements to docs
* allow toggling live audio transcription via mqtt
* improve spacing consistency on mobile drawers
* fix clearing the region grid not surviving a restart
* sort preview cameras in history by ui order
* sort cameras by UI order in various components for consistent display
* add no recordings faq
* fix link
* recording cache faq
* add link
* improve anchor naming in object detector docs
rather than #configuration-1, #configuration-2, etc
* use yaml instead of json for object detector docs
* fix anchor
* add embedded mode to BaseSection so parents can host the save action
* add optional action slot to current Frigate+ model summary
* add w-full to action slot flex wrapper for explicit width contract
* i18n
* merged detectors and model settings view
* fix document title
* Embed detector form in merged settings view
* add detection model card with tabs and custom model embed
* add Frigate+ model selector with filter popover to merged page
* Add mismatch banner and gate save on detector and model compatibility
* Wire atomic save, restart toast, and undo on detectors and model page
* Clear child pending data on undo
* route merged detectors and model view in settings
* trim Frigate+ page to account-only and remove old detection model view
* basic e2e
* Fix unsaved-changes guard, custom path leak, and post-failure cache resync
* Rename to Detectors and model, float Modified badge, use ConfigMessageBanner for mismatch
* Hide Plus/Custom tabs when Frigate+ is not enabled
* Detect active Plus model via model.plus.id instead of path prefix
* Sync state back to snapshot when child form un-modifies and remount on undo
* Always require restart on save since model changes also need one
* Wrap Frigate+ model selector in SplitCardRow with label and description
* rename tab
* update docs
* sync top-level model with default detector's resolved model
when the user doesn't define a top-level `model:` block, `FrigateConfig.model` stayed at pydantic field defaults (320×320, /labelmap.txt) while the per-detector model picked up `DEFAULT_MODEL` for openvino on cpu (300×300, coco_91cl_bkgr.txt introduced in #23127), causing `RemoteObjectDetector` to fail with "buffer is too small for requested array" because the SHM was sized from the per-detector model but mapped using the top-level one. After the detector loop, copy the first detector's resolved model up to `self.model` so both sides agree on dimensions and labelmap
* revert to cpu detector by default
use openvino cpu for new configs only
* add defaults
* hide camera overrides badge from system sections
* show empty card on camera metrics page when no cameras are defined
* fix enabled camera state switch after adding via wizard
Cameras added mid-session have no WS state until the dispatcher publishes camera_activity (which only happens on a fresh onConnect). Fall back to the config's enabled value so the switch reflects reality immediately after the wizard closes.
* guard camera enabled access
console would throw errors after adding via camera wizard
* fix useOptimisticState dropping debounced setState under StrictMode
* use openvino on cpu as default model
- faster than tflite on cpu
- add to default generated config
* use an enum for model_size
the frontend will then render this as a select dropdown because of the changes in the json schema
* i18n
* sync object filter entries with tracked labels in camera config form
Filter sub-collapsibles in the camera Objects section are driven by `filters` dict keys, but profile merges and live track-switch edits don't add matching entries, so newly tracked labels (like from a profile override) had no collapsible. Synthesize default filter entries from `track` in the form data so every tracked label renders a collapsible; baseline data also gets the synthesized entries, so save payloads are unchanged.
* revalidate raw paths cache after config save so CameraPathWidget shows fresh credentials
* fix test
* restore masked ffmpeg credentials when persisting camera config
* formatting
* rebuild ffmpeg commands when enabling recording for the first time
Toggling record.enabled from the config UI updated the in-memory config but left ffmpeg running with its original command, so the record output args were never wired in and nothing landed in the cache for the maintainer to move. The record config update now rebuilds ffmpeg_cmds when enabled_in_config transitions, and the camera watchdog restarts ffmpeg on a false to true transition so the record output gets wired in. MQTT toggles, which only flip record.enabled at runtime, are unaffected and continue to work via the maintainer's drop/keep gate.
* keep record toggle switch in single camera view disabled until enabled in config
* fix override detection for sections unset in the global config
Override badges and the blue dot now compare against schema defaults for sections like motion that the API serializes as null when omitted from the global YAML, instead of treating any populated camera config as an override
* add support for config-aware patterns in section hiddenFields
Section configs can now declare dynamic hidden-field entries as functions of the loaded config; objects.ts uses this to hide auto-populated attribute filters (DHL, face, license_plate, etc.) from the form, save flow, and override popover when those labels aren't user-settable
* siimplify object filters handling
live updating was getting very messy. users will just need to save once they enable a new object in order to see filters for that object
* tweaks
* update docs for new detector default
* make genai provider required and add special case for UI
prevent validation errors from appearing on initial creation of genai provider by setting the first option in the select dropdown as default
* Update to ROCm 7.2.3
* Add inference time for 9060XT
* Update times
* Update hardware info for latest ROCm
* Add env vars to save kernels and miopen database
* re-enable face recognition for ROCm
* Update
* Save LLVM cache
* set edgetpu for multi-instance
* improve error messages when mixing/matching detectors
* allow custom add button text via uiSchema
* clarify language in docs for configuring detectors via the UI
* add generation script
a script to read yaml code blocks from docs markdown files and generate corresponding "Frigate UI" tab instructions based on the json schema, i18n, section configs (hidden fields), and nav mappings
* first pass
* components
* add to gitignore
* second pass
* fix broken anchors
* fixes
* clean up tabs
* version bump
* tweaks
* remove role mapping config from ui
* fix config examples
* remove reference to trt model generation script
* tweak tmpfs comment
* update old version
* tweak tmpfs comment
* clean up and clarify tensorrt
* re-add size
* Update docs/docs/configuration/hardware_acceleration_enrichments.md
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
---------
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
* Docs: fix missing dependency in YOLOv9 build script
I had this command fail because it didn't have cmake available.
This change fixes that problem.
* Docs: avoid failure in YOLOv9 build script
Pinning to 0.4.36 avoids this error:
```
10.58 Downloading onnx
12.87 Building onnxsim==0.5.0
1029.4 × Failed to download and build `onnxsim==0.5.0`
1029.4 ╰─▶ Package metadata version `0.4.36` does not match given version `0.5.0`
1029.4 help: `onnxsim` (v0.5.0) was included because `onnx-simplifier` (v0.5.0)
1029.4 depends on `onnxsim`
```
* Update Dockerfile instructions for object detectors
---------
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
* disable modal on dropdown menu in explore
* add another example case for when classification overrides a sub label
* update ollama docs link
* Improve handling of automatic playback for recordings
* Improve ollama documentation
* Don't fall out when all recording segments exist
* clarify coral docs
* improve initial scroll to active item in detail stream
* i18n fixes
* remove console warning
* detail stream scrolling fixes for HA/iOS
* Improve usability of GenAI summary dialog and make clicking on the description directly open it
* Review card too
* Use empty card with dynamic text for review based on the user's config
---------
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>