Compare commits

..
9 Commits
Author SHA1 Message Date
Josh Hawkins e0d48c98d9 show runtime overrides in the settings form
The settings form read a camera section's saved config value, but its dependent warnings (audio transcription requiring audio detection, snapshots requiring detect, etc.) read the live config instead. A runtime toggle from the live view, MQTT, or an active profile can turn a section off without touching yaml, and that override persists across restarts, so the Enable switch showed on while the warning said the feature wasn't enabled. This adds an "Overridden (Live)" badge to any field whose live value differs from what's saved, and swaps the affected warnings to runtime-specific wording when a runtime override is the actual cause instead of the config.
2026-09-17 20:42:41 -05:00
Josh Hawkins 1ba10bce2a fix restart failing under non-root
restart_frigate() called psutil.Process(1).terminate() to signal s6-svscan, but s6-svscan runs as root while frigate runs as uid 1000, so the call raised AccessDenied. That exception escaped every caller: the UI restart button dropped its websocket client, MQTT restart and Save & Restart just logged and did nothing, and the watchdog crashed its own monitoring thread on a dead detector. This catches AccessDenied and falls through to the existing SIGINT branch, which exits the process for s6 to restart it.
2026-09-17 20:42:15 -05:00
Josh Hawkins 8120a4b117 don't edit a chat message while a reply streams
The edit button stayed active while a reply streamed. `submitConversation` returns early while loading, but the message bubble still closed its editor, so the edit was silently lost. The edit button is hidden while a reply streams, and an editor that's already open keeps its draft with send disabled until the reply ends.
2026-09-17 16:55:39 -05:00
Josh Hawkins 03e6f54670 fix train image filtering for a class with a dash
The backend writes a class with a `-` as `_` in train file names, since it splits those names on `-`, while a dataset folder keeps the dash. Filtering the Train grid by `half-open` compared it with `half_open` and hid every attempt. Both sides are normalized the same way now.
2026-09-17 16:55:39 -05:00
Josh Hawkins 13a3af1957 don't run page shortcuts for keys a dialog already handled
Radix dismisses a dialog on Escape from a capture-phase keydown listener and calls `preventDefault()` without stopping propagation, so `useKeyboardListener` still ran the page's Escape shortcut: cancelling the delete dialog in the face library or a classification model also cleared the whole selection. Keys another shortcut hook handled still get through, since their listener order changes with every render.
2026-09-17 16:55:31 -05:00
Josh Hawkins 8aa0ddf740 find DST transitions to the second
`get_dst_transitions` probed the offset once every 24 hours from the start time and reported a change at the first probe after it, up to a day late, so events, review items and recordings near a transition were grouped into days with the old offset. A transition after the last daily probe wasn't found at all. The end of the range is probed too now, and a probe that sees the offset change bisects the interval to the second of the transition.
2026-09-17 16:55:25 -05:00
Josh Hawkins 5fc8d4f1c6 return 403 for a snapshot or thumbnail on another camera
The broad `except Exception` handlers in `event_snapshot` and `event_thumbnail` caught the `HTTPException` from `require_camera_access`, so a restricted user asking for another camera's snapshot got a 404 instead of a 403, and for an object still being tracked the snapshot was rendered before the check ran. Both endpoints now look up the event and check access in their own block, the way the other endpoints do, so a denial propagates.
2026-09-17 16:55:19 -05:00
Josh Hawkins a8c838aee2 fix the has_clip self-heal for events with no recordings
`vod_event` looked for a `(body, 404)` tuple, but `vod_ts` returns a `JSONResponse`, so the check never matched and an old event whose recordings are gone kept offering a clip that can't play. It now checks the response status code.
2026-09-17 16:55:12 -05:00
Josh Hawkins 4080c46428 check for a valid frame before using its shape
With the camera offline, no preview frame, and `camera-error.jpg` missing, `latest_frame` read `frame.shape` before its `frame is None` check, so it raised `AttributeError` and answered 500 with a traceback instead of the intended "Unable to get valid frame". The check now runs first.
2026-09-17 16:55:06 -05:00
597 changed files with 7645 additions and 37779 deletions
+4 -9
View File
@@ -17,14 +17,9 @@ runs:
shell: bash shell: bash
# This creates a virtual volume at /var/lib/docker to maximize the size # This creates a virtual volume at /var/lib/docker to maximize the size
# As of 2/14/2024, this results in 97G for docker images # As of 2/14/2024, this results in 97G for docker images
# Runners no longer have a separate /mnt disk, so the temp PV is also carved
# from root and temp-reserve-mb is what actually stays free on root. Keep 4G
# there for setup-qemu/buildx caches in ~/.docker and the tool cache
- name: Maximize build space - name: Maximize build space
uses: easimon/maximize-build-space@master uses: easimon/maximize-build-space@master
with: with:
root-reserve-mb: 8192
temp-reserve-mb: 4096
remove-dotnet: 'true' remove-dotnet: 'true'
remove-android: 'true' remove-android: 'true'
remove-haskell: 'true' remove-haskell: 'true'
@@ -34,15 +29,15 @@ runs:
run: sudo systemctl start docker run: sudo systemctl start docker
shell: bash shell: bash
- id: lowercaseRepo - id: lowercaseRepo
uses: ASzc/change-string-case-action@v8 uses: ASzc/change-string-case-action@v5
with: with:
string: ${{ github.repository }} string: ${{ github.repository }}
- name: Set up QEMU - name: Set up QEMU
uses: docker/setup-qemu-action@v4 uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 uses: docker/setup-buildx-action@v3
- name: Log in to the Container registry - name: Log in to the Container registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f uses: docker/login-action@465a07811f14bebb1938fbed4728c6a1ff8901fc
with: with:
registry: ghcr.io registry: ghcr.io
username: ${{ github.actor }} username: ${{ github.actor }}
+22 -139
View File
@@ -1,157 +1,40 @@
version: 2 version: 2
updates: updates:
- package-ecosystem: "github-actions" - package-ecosystem: "github-actions"
# Local composite actions are only scanned when their directory is listed directory: "/"
directories:
- "/"
- "/.github/actions/setup"
schedule: schedule:
interval: weekly interval: daily
cooldown: open-pull-requests-limit: 10
default-days: 3 target-branch: dev
open-pull-requests-limit: 5 - package-ecosystem: "docker"
groups: directory: "/docker"
actions: schedule:
patterns: ["*"] interval: daily
actions-security: open-pull-requests-limit: 10
applies-to: security-updates target-branch: dev
patterns: ["*"]
- package-ecosystem: "pip" - package-ecosystem: "pip"
directory: "/docker/main" directory: "/docker/main"
schedule: schedule:
interval: weekly interval: daily
cooldown: open-pull-requests-limit: 10
default-days: 3 target-branch: dev
open-pull-requests-limit: 5
groups:
# Sets that fail pip resolution unless they move together
peewee:
patterns: ["peewee", "peewee-migrate"]
crypto:
patterns: ["cryptography", "joserfc", "pywebpush", "py-vapid"]
pydantic-genai:
patterns: ["pydantic", "google-genai"]
pip-main-minor-patch:
patterns: ["*"]
update-types: ["minor", "patch"]
pip-main-security:
applies-to: security-updates
patterns: ["*"]
ignore:
# numpy 2.5 needs Python 3.12; norfair, tflite_runtime and rknn-toolkit2
# cap numpy below 2
- dependency-name: "numpy"
versions: [">=2"]
# Every OpenCV wheel from 4.12 requires numpy 2
- dependency-name: "opencv-python-headless"
versions: [">=4.12"]
- dependency-name: "opencv-contrib-python"
versions: [">=4.12"]
# aarch64 only. Rockchip rknn-toolkit2 caps protobuf at 4.25.4 and
# TensorFlow 2.21 needs protobuf 6.31 or newer.
- dependency-name: "tensorflow"
versions: [">=2.20"]
# Needs Python 3.12 and numpy 2
- dependency-name: "librosa"
versions: [">=1.0"]
# Needs Python 3.12
- dependency-name: "scipy"
versions: [">=1.18"]
# Newer stubs report false errors against peewee 4
- dependency-name: "types-peewee"
versions: [">=4.1"]
# sdist only and imported by HailoRT; needs a Hailo device to test
- dependency-name: "netifaces"
versions: [">=0.11"]
- package-ecosystem: "pip" - package-ecosystem: "pip"
directory: "/docker/tensorrt" directory: "/docker/tensorrt"
schedule: schedule:
interval: weekly interval: daily
# No pull request check builds these images, so version updates are off. open-pull-requests-limit: 10
# Security updates still open. target-branch: dev
open-pull-requests-limit: 0
groups:
pip-trt-security:
applies-to: security-updates
patterns: ["*"]
ignore:
# onnx 1.17 and newer write a newer IR and opset than TensorRT 8.6 on
# JetPack 6 is known to parse
- dependency-name: "onnx"
versions: [">=1.17"]
# 1.27 moved to CUDA 13, and the image ships the CUDA 12.8 wheel set
- dependency-name: "onnxruntime-gpu"
versions: [">=1.27"]
# Hand-matched CUDA 12.8 set (2c9a25e67); move as a block
- dependency-name: "nvidia-*"
# Must match the JetPack CUDA runtime
- dependency-name: "cuda-python"
versions: [">=12.9"]
- package-ecosystem: "npm" - package-ecosystem: "npm"
directory: "/web" directory: "/web"
schedule: schedule:
interval: weekly interval: daily
cooldown: open-pull-requests-limit: 10
default-days: 3 target-branch: dev
open-pull-requests-limit: 5
groups:
radix:
patterns: ["@radix-ui/*"]
rjsf:
patterns: ["@rjsf/*"]
web-minor-patch:
patterns: ["*"]
update-types: ["minor", "patch"]
web-security:
applies-to: security-updates
patterns: ["*"]
ignore:
# typescript-eslint peers typescript below 6.1
- dependency-name: "typescript"
versions: [">=6.1"]
# Tailwind 4 is a config rewrite; scrollbar 4 and merge 3 depend on it
- dependency-name: "tailwindcss"
update-types: ["version-update:semver-major"]
- dependency-name: "tailwind-scrollbar"
update-types: ["version-update:semver-major"]
- dependency-name: "tailwind-merge"
update-types: ["version-update:semver-major"]
# Intermittent failure in the UI settings import spec, cause not found
- dependency-name: "zod"
update-types: ["version-update:semver-major"]
# Needs an on-device pinch test before moving
- dependency-name: "react-zoom-pan-pinch"
update-types: ["version-update:semver-major"]
# Build fails until monaco-worker-manager releases its PR 4
- dependency-name: "monaco-editor"
versions: [">=0.55.0"]
# Later versions need Node 22 and extract false keys
- dependency-name: "i18next-cli"
versions: [">1.5.11"]
# 6.11.0 refills the saved model in the GenAI settings form
- dependency-name: "@rjsf/*"
versions: ["6.11.0"]
- package-ecosystem: "npm" - package-ecosystem: "npm"
directory: "/docs" directory: "/docs"
schedule: schedule:
interval: weekly interval: daily
cooldown:
default-days: 3
open-pull-requests-limit: 5
allow: allow:
- dependency-name: "@docusaurus/*" - dependency-name: "@docusaurus/*"
groups: open-pull-requests-limit: 10
docusaurus: target-branch: dev
patterns: ["@docusaurus/*"]
docs-security:
applies-to: security-updates
patterns: ["*"]
ignore:
# A Docusaurus major needs the openapi plugin and theme moved in the
# same commit
- dependency-name: "@docusaurus/*"
update-types: ["version-update:semver-major"]
+1 -1
View File
@@ -586,7 +586,7 @@ jobs:
- arm64_build - arm64_build
steps: steps:
- id: lowercaseRepo - id: lowercaseRepo
uses: ASzc/change-string-case-action@v8 uses: ASzc/change-string-case-action@v6
with: with:
string: ${{ github.repository }} string: ${{ github.repository }}
- name: Log in to the Container registry - name: Log in to the Container registry
+1 -1
View File
@@ -14,7 +14,7 @@ jobs:
with: with:
persist-credentials: false persist-credentials: false
- id: lowercaseRepo - id: lowercaseRepo
uses: ASzc/change-string-case-action@v8 uses: ASzc/change-string-case-action@v6
with: with:
string: ${{ github.repository }} string: ${{ github.repository }}
- name: Log in to the Container registry - name: Log in to the Container registry
-1
View File
@@ -5,4 +5,3 @@
/docker/rockchip/ @MarcA711 /docker/rockchip/ @MarcA711
/docker/rocm/ @harakas /docker/rocm/ @harakas
/docker/hailo8l/ @spanner3003 /docker/hailo8l/ @spanner3003
/docker/deepx/ @sixfab
-131
View File
@@ -1,131 +0,0 @@
#!/bin/bash
# Installs the DEEPX NPU kernel driver and the DX-RT runtime on the Docker host,
# then enables the vendor's dxrt.service. A container cannot load kernel
# modules, so this runs outside the image; the driver creates the /dev/dxrt*
# nodes and the daemon multiplexes the NPU across host and container.
#
# Driver, runtime and firmware versions must agree or inference hangs instead
# of failing at startup. The set this script installs is pinned in
# driver_version, runtime_version and firmware_version below; move them
# together, never one at a time.
#
# DEEPX NPU support in Frigate is maintained by Sixfab (https://sixfab.com).
set -euo pipefail
driver_version="v2.6.0"
# the commit the tag resolves to, since DEEPX signs neither tags nor releases
# and this is compiled and installed as root. Update both together
driver_commit="7074748e7104f470b02f517583abba652b3f05fa"
firmware_version="v2.7.4"
sudo apt-get update
sudo apt-get install -y git build-essential "linux-headers-$(uname -r)" pciutils wget
if ! lspci -d 1ff4: | grep -q .; then
echo "No DEEPX device found on the PCIe bus (lspci -d 1ff4:)."
echo "Check that the module is seated correctly before continuing."
exit 1
fi
# fetch the pinned commit rather than cloning the tag, so a retag cannot swap
# in different source. The build directory is reused so a second run after a
# failure does not stop on the directory already being there
mkdir -p dx_rt_npu_linux_driver
cd dx_rt_npu_linux_driver
git init -q
git remote get-url origin > /dev/null 2>&1 ||
git remote add origin https://github.com/DEEPX-AI/dx_rt_npu_linux_driver.git
git fetch --depth 1 origin "${driver_commit}"
git checkout -q FETCH_HEAD
fetched_commit=$(git rev-parse HEAD)
if [[ "${fetched_commit}" != "${driver_commit}" ]]; then
echo "Fetched commit ${fetched_commit} does not match pinned driver_commit ${driver_commit}."
echo "Refusing to build unverified driver source."
exit 1
fi
cd modules
sudo ./build.sh -c install --reload
sudo depmod -A
# dx_dma is the PCIe transport, dxrt_driver the NPU driver on top of it
for module in dx_dma dxrt_driver; do
if ! sudo modprobe "${module}"; then
echo "Unable to load the ${module} kernel module, common reasons are:"
echo "- Secure Boot is enabled and is rejecting the unsigned module."
echo "- The running kernel does not match the installed linux-headers."
exit 1
fi
done
if ! compgen -G "/dev/dxrt*" > /dev/null; then
echo "Modules loaded but no /dev/dxrt* device node appeared."
echo "Run ./sanity_check.sh from the driver repo to diagnose."
exit 1
fi
runtime_version="v3.4.0"
declare -A runtime_sha256=(
[amd64]="736cfef009ce9e974ab1ab610d867239d19d72a426a53e367ddcbd53297b6e20"
[arm64]="eb6107f5f02f2ad76ae89f414e8b5f346f34fbc6f0888236136853a26be6f6a0"
)
runtime_release="${runtime_version#v}"
deb_arch=$(dpkg --print-architecture)
deb_file="/tmp/libdxrt-bin_${runtime_release}_${deb_arch}.deb"
wget -qO "${deb_file}" \
"https://raw.githubusercontent.com/DEEPX-AI/dx_rt/${runtime_version}/release/${runtime_release}/libdxrt-bin_${runtime_release}_${deb_arch}.deb"
expected_sha256="${runtime_sha256[${deb_arch}]:-}"
if [[ -z "${expected_sha256}" ]]; then
echo "No pinned SHA-256 for architecture ${deb_arch}; refusing to install."
exit 1
fi
if [[ "$(sha256sum "${deb_file}" | cut -d' ' -f1)" != "${expected_sha256}" ]]; then
echo "SHA-256 mismatch for ${deb_file}; refusing to install."
exit 1
fi
sudo dpkg -i "${deb_file}"
sudo ldconfig
rm -f "${deb_file}"
sudo cp /usr/share/libdxrt-bin/service/dxrt.service /etc/systemd/system/
# With an endpoint set, dxrtd binds that path only, so the socket goes in a
# directory Frigate can mount (kept across restarts so the mount stays valid)
# and a symlink at the default /tmp path keeps host tools that do not set the
# variable working through their own fallback.
sudo mkdir -p /etc/systemd/system/dxrt.service.d
sudo tee /etc/systemd/system/dxrt.service.d/frigate.conf > /dev/null <<'UNIT'
[Service]
RuntimeDirectory=dxrt
RuntimeDirectoryMode=0755
RuntimeDirectoryPreserve=yes
Environment=DXRT_DYNAMIC_IPC_ENDPOINT=/run/dxrt/dxrt_dynamic_ipc.sock
ExecStartPost=/bin/ln -sfn /run/dxrt/dxrt_dynamic_ipc.sock /tmp/dxrt_dynamic_ipc.sock
UNIT
sudo systemctl daemon-reload
sudo systemctl enable dxrt.service
sudo systemctl restart dxrt.service
if ! sudo systemctl is-active --quiet dxrt.service; then
echo "dxrt.service did not start. Check: sudo journalctl -u dxrt.service"
exit 1
fi
echo "DEEPX driver and runtime installation complete."
echo "Driver version: $(modinfo -F version dxrt_driver) (expected ${driver_version#v})"
echo "Runtime version: ${runtime_release}"
echo "Device node(s): $(echo /dev/dxrt*)"
echo
echo "This driver expects NPU firmware ${firmware_version}. Check it with:"
echo " dxrt-cli --status"
echo "Update the module if it does not match before starting Frigate."
+1 -1
View File
@@ -1,4 +1,4 @@
ruff == 0.16.10 ruff == 0.15.20
# types # types
types-peewee == 4.0.* types-peewee == 4.0.*
+34 -34
View File
@@ -1,66 +1,66 @@
aiofiles == 25.1.* aiofiles == 25.1.*
click == 8.5.* click == 8.5.*
# FastAPI # FastAPI
aiohttp == 3.14.* aiohttp == 3.12.*
starlette == 1.7.* starlette == 0.47.*
starlette-context == 0.5.* starlette-context == 0.5.*
fastapi[standard-no-fastapi-cloud-cli] == 0.142.* fastapi[standard-no-fastapi-cloud-cli] == 0.116.*
uvicorn == 0.54.* uvicorn == 0.52.*
slowapi == 0.1.* slowapi == 0.1.*
joserfc == 1.7.* joserfc == 1.6.*
cryptography == 50.0.* cryptography == 46.0.*
pathvalidate == 3.3.* pathvalidate == 3.3.*
markupsafe == 3.0.* markupsafe == 3.0.*
python-multipart == 0.0.32 python-multipart == 0.0.31
# Classification Model Training # Classification Model Training
tensorflow == 2.19.* ; platform_machine == 'aarch64' tensorflow == 2.19.* ; platform_machine == 'aarch64'
tensorflow-cpu == 2.21.* ; platform_machine == 'x86_64' tensorflow-cpu == 2.19.* ; platform_machine == 'x86_64'
# General # General
mypy == 2.4.0 mypy == 1.6.1
onvif-zeep-async == 4.3.* onvif-zeep-async == 4.0.*
paho-mqtt == 2.1.* paho-mqtt == 2.1.*
pandas == 3.0.* pandas == 2.2.*
peewee == 4.5.* peewee == 3.17.*
peewee_migrate == 2.3.* peewee_migrate == 1.14.*
psutil == 7.2.* psutil == 7.1.*
pydantic == 2.13.* pydantic == 2.10.*
git+https://github.com/fbcotter/py3nvml#egg=py3nvml git+https://github.com/fbcotter/py3nvml#egg=py3nvml
tzdata == 2026.5 pytz == 2025.*
pyzmq == 27.2.* pyzmq == 27.1.*
ruamel.yaml == 0.19.* ruamel.yaml == 0.18.*
tzlocal == 5.4.4 tzlocal == 5.2
requests == 2.34.* requests == 2.33.*
types-requests == 2.33.* types-requests == 2.32.*
norfair == 2.3.* norfair == 2.3.*
setproctitle == 1.3.* setproctitle == 1.3.*
ws4py == 0.6.* ws4py == 0.5.*
unidecode == 1.4.* unidecode == 1.4.*
titlecase == 2.4.* titlecase == 2.4.*
# Image Manipulation # Image Manipulation
numpy == 1.26.* numpy == 1.26.*
opencv-python-headless == 4.11.0.* opencv-python-headless == 4.11.0.*
opencv-contrib-python == 4.11.0.* opencv-contrib-python == 4.11.0.*
scipy == 1.17.* scipy == 1.16.*
# OpenVino & ONNX # OpenVino & ONNX
openvino == 2025.4.* openvino == 2025.4.*
onnxruntime == 1.30.* onnxruntime == 1.22.*
# Embeddings # Embeddings
transformers == 5.19.* transformers == 4.45.*
# Generative AI # Generative AI
google-genai == 2.29.* google-genai == 1.58.*
ollama == 0.6.* ollama == 0.6.*
openai == 3.26.* openai == 1.65.*
# push notifications # push notifications
py-vapid == 1.9.4 py-vapid == 1.9.4
pywebpush == 2.5.* pywebpush == 2.0.*
# alpr # alpr
pyclipper == 1.4.* pyclipper == 1.4.*
shapely == 2.2.* shapely == 2.0.*
rapidfuzz==3.14.* rapidfuzz==3.12.*
# HailoRT # HailoRT
argcomplete==3.7.* argcomplete==2.0.*
contextlib2==21.6.* contextlib2==0.6.*
future==1.0.* future==0.18.*
netaddr==1.3.* netaddr==1.3.*
netifaces==0.10.* netifaces==0.10.*
prometheus-client == 0.26.* prometheus-client == 0.26.*
@@ -71,6 +71,6 @@ tflite_runtime @ https://github.com/feranick/TFlite-builds/releases/download/v2.
sherpa-onnx==1.13.* sherpa-onnx==1.13.*
faster-whisper==1.2.* faster-whisper==1.2.*
librosa==0.11.* librosa==0.11.*
soundfile==0.14.* soundfile==0.13.*
# Memory profiling # Memory profiling
memray == 1.20.* memray == 1.20.*
@@ -37,7 +37,6 @@ device_globs=(
"/dev/nvmap" "/dev/nvmap"
"/dev/nvidia*" "/dev/nvidia*"
"/dev/memx*" "/dev/memx*"
"/dev/dxrt*"
) )
IFS=',' read -ra extra_globs <<< "${DEVICE_ACL_PATHS:-}" IFS=',' read -ra extra_globs <<< "${DEVICE_ACL_PATHS:-}"
@@ -8,14 +8,13 @@ from typing import Any
from ruamel.yaml import YAML from ruamel.yaml import YAML
sys.path.insert(0, "/opt/frigate") sys.path.insert(0, "/opt/frigate")
from frigate.config.env import apply_config_env_vars, substitute_frigate_vars
from frigate.const import ( from frigate.const import (
BIRDSEYE_PIPE, BIRDSEYE_PIPE,
LIBAVFORMAT_VERSION_MAJOR, LIBAVFORMAT_VERSION_MAJOR,
) )
from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode
from frigate.util.config import find_config_file, resolve_ffmpeg_path from frigate.util.config import find_config_file, resolve_ffmpeg_path
from frigate.util.env import apply_config_env_vars, substitute_frigate_vars
from frigate.util.live_streams import raw_transcode_streams
from frigate.util.services import ( from frigate.util.services import (
is_go2rtc_arbitrary_exec_allowed, is_go2rtc_arbitrary_exec_allowed,
is_restricted_go2rtc_source, is_restricted_go2rtc_source,
@@ -175,17 +174,6 @@ for name in list(go2rtc_config.get("streams", {})):
del go2rtc_config["streams"][name] del go2rtc_config["streams"][name]
continue continue
# add transcoded live streams; a user stream with the same name wins here and
# fails Frigate's config validation
transcoded_streams = raw_transcode_streams(config)
if transcoded_streams:
if go2rtc_config.get("streams") is None:
go2rtc_config["streams"] = {}
for name, source in transcoded_streams.items():
go2rtc_config["streams"].setdefault(name, source)
# add birdseye restream stream if enabled # add birdseye restream stream if enabled
if config.get("birdseye", {}).get("restream", False): if config.get("birdseye", {}).get("restream", False):
birdseye: dict[str, Any] = config.get("birdseye") birdseye: dict[str, Any] = config.get("birdseye")
+1 -1
View File
@@ -14,4 +14,4 @@ nvidia-nccl-cu12==2.26.2.post1; platform_machine == 'x86_64'
nvidia-nvjitlink-cu12==12.8.93; platform_machine == 'x86_64' nvidia-nvjitlink-cu12==12.8.93; platform_machine == 'x86_64'
onnx==1.16.*; platform_machine == 'x86_64' onnx==1.16.*; platform_machine == 'x86_64'
onnxruntime-gpu==1.24.*; platform_machine == 'x86_64' onnxruntime-gpu==1.24.*; platform_machine == 'x86_64'
protobuf==7.36.*; platform_machine == 'x86_64' protobuf==3.20.3; platform_machine == 'x86_64'
-91
View File
@@ -824,38 +824,6 @@ cpu:
models: models:
- devices: - devices:
- cpu:3 - cpu:3
xdna2:
title: AMD XDNA2
models:
- key: yolov9
label: YOLOv9
recommended: true
download: |-
Prepare the model using the frigate-xdna setup instructions linked above. For local YOLO models, Frigate must have access to the same ONNX file bytes as the sidecar. The example below uses YOLOv9-C at 320x320. Frigate+ models may instead use the same `plus://MODEL_ID` in Frigate and the sidecar.
ui: |-
Navigate to **Settings > System > Detection models** and add a model. The ZMQ endpoint is not reported by the hardware probe, so set `devices` to `zmq:tcp://xdna:5555` in YAML. Then, on the same model, open the **Custom Model** tab and configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------ |
| **Custom object detector model path** | `/config/models/yolov9-c-320.onnx` |
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
| **Object detection model input width** | `320` |
| **Object detection model input height** | `320` |
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
| **Model Input Tensor Shape** | `nchw` |
| **Model Input D Type** | `float` |
| **Object Detection Model Type** | `yolo-generic` |
yaml: |-
models:
- devices:
- zmq:tcp://xdna:5555
model_type: yolo-generic
width: 320
height: 320
input_tensor: nchw
input_dtype: float
path: /config/models/yolov9-c-320.onnx
labelmap_path: /labelmap/coco-80.txt
memryx: memryx:
title: MemryX title: MemryX
models: models:
@@ -999,65 +967,6 @@ memryx:
# The .zip file must contain: # The .zip file must contain:
# ├── ssdlite_mobilenet.dfp (a file ending with .dfp) # ├── ssdlite_mobilenet.dfp (a file ending with .dfp)
# └── ssdlite_mobilenet_post.onnx (optional; only if the model includes a cropped post-processing network) # └── ssdlite_mobilenet_post.onnx (optional; only if the model includes a cropped post-processing network)
deepx:
title: DEEPX NPU
models:
- key: yolo
label: YOLO
recommended: true
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The recommended model is `yolox-s_640x640_ppu.dxnn`. Its Post-Processing Unit (PPU) compile moves candidate selection onto the NPU, which makes it the fastest ModelZoo model measured through Frigate (about 13 ms on a DX-M1). The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration; prefer a `PPU` variant whenever the ModelZoo offers one. PPU models must be compiled with DX-COM 2.4.0 or later, which writes the head layout Frigate reads into the file.
ui: |-
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
| Field | Value |
| ---------------------------------------- | ---------------------------------------------------- |
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640_ppu.dxnn` |
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
| **Object detection model input width** | `640` |
| **Object detection model input height** | `640` |
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
| **Model Input D Type** | `int` (Frigate's default value) |
| **Object Detection Model Type** | `yolo-generic` |
Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
yaml: |-
models:
- devices:
- deepx:PCIe:0
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: yolo-generic
width: 640
height: 640
- key: yolox
label: YOLOX
recommended: false
download: No model is bundled with Frigate. Download a pre-compiled YOLOX `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), then bind-mount it into the container and point the model's `path` at it. The `_ppu` variant is faster and also works with the `yolo-generic` model type; the plain export needs `yolox` so its raw head is decoded.
ui: |-
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
| Field | Value |
| ---------------------------------------- | ---------------------------------------- |
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640.dxnn`|
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
| **Object detection model input width** | `640` |
| **Object detection model input height** | `640` |
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
| **Model Input D Type** | `int` (Frigate's default value) |
| **Object Detection Model Type** | `yolox` |
The width and height must match the resolution the `.dxnn` file was compiled for.
yaml: |-
models:
- devices:
- deepx:PCIe:0
path: /config/model_cache/deepx/yolox-s_640x640.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: yolox
width: 640
height: 640
tensorrt: tensorrt:
title: TensorRT title: TensorRT
models: models:
+12 -38
View File
@@ -152,10 +152,9 @@ auth:
models: models:
# Optional: the camera environment this model is for (default: shown below) # Optional: the camera environment this model is for (default: shown below)
# Cameras select a model by setting detect -> scene to a matching value, and # Cameras select a model by setting detect -> scene to a matching value, and
# the model with a scene of default is used by any camera that does not set one. # a model with a scene of all is used by any camera that does not set one.
# Any name made up of letters, numbers, _ and - is valid, such as thermal. # Valid values are all, indoor, outdoor, indoor_thermal, outdoor_thermal
# Models that use the same model file are combined into one model. - scene: all
- scene: default
# Required: hardware this model runs on, as <detector> or <detector>:<device> # Required: hardware this model runs on, as <detector> or <detector>:<device>
# See https://docs.frigate.video/configuration/object_detectors for the # See https://docs.frigate.video/configuration/object_detectors for the
# detectors available and the devices each one accepts. All of a model's # detectors available and the devices each one accepts. All of a model's
@@ -294,9 +293,9 @@ ffmpeg:
# Optional: output args for detect streams (default: shown below) # Optional: output args for detect streams (default: shown below)
detect: -threads 2 -f rawvideo -pix_fmt yuv420p detect: -threads 2 -f rawvideo -pix_fmt yuv420p
# Optional: output args for record streams (default: shown below) # Optional: output args for record streams (default: shown below)
record: preset-record-generic-audio-aac record: preset-record-generic
# Optional: output args for sub stream record streams (default: the record output args above) # Optional: output args for sub stream record streams (default: the record output args above)
# record_sub: preset-record-generic-audio-aac # record_sub: preset-record-generic
# Optional: Time in seconds to wait before ffmpeg retries connecting to the camera. (default: shown below) # Optional: Time in seconds to wait before ffmpeg retries connecting to the camera. (default: shown below)
# If set too low, frigate will retry a connection to the camera's stream too frequently, using up the limited streams some cameras can allow at once # If set too low, frigate will retry a connection to the camera's stream too frequently, using up the limited streams some cameras can allow at once
# If set too high, then if a ffmpeg crash or camera stream timeout occurs, you could potentially lose up to a maximum of retry_interval second(s) of footage # If set too high, then if a ffmpeg crash or camera stream timeout occurs, you could potentially lose up to a maximum of retry_interval second(s) of footage
@@ -317,9 +316,9 @@ detect:
# Optional: height of the frame for the input with the detect role (default: use native stream resolution) # Optional: height of the frame for the input with the detect role (default: use native stream resolution)
height: 720 height: 720
# Optional: the environment this camera looks at, which picks the model it runs on # Optional: the environment this camera looks at, which picks the model it runs on
# (default: the model with a scene of default) # (default: the model with a scene of all)
# Must match the scene of a configured model # Valid values are all, indoor, outdoor, indoor_thermal, outdoor_thermal
scene: thermal scene: outdoor
# Optional: desired fps for your camera for the input with the detect role (default: shown below) # Optional: desired fps for your camera for the input with the detect role (default: shown below)
# NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera. # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
fps: 5 fps: 5
@@ -571,8 +570,6 @@ notifications:
enabled: False enabled: False
# Optional: Email for push service to reach out to # Optional: Email for push service to reach out to
# NOTE: This is required to use notifications # NOTE: This is required to use notifications
# NOTE: Email can be specified with an environment variable or docker secrets that must begin with 'FRIGATE_'.
# e.g. email: '{FRIGATE_NOTIFICATION_EMAIL}'
email: "admin@example.com" email: "admin@example.com"
# Optional: Cooldown time for notifications in seconds (default: shown below) # Optional: Cooldown time for notifications in seconds (default: shown below)
cooldown: 0 cooldown: 0
@@ -803,7 +800,7 @@ lpr:
# to Google or OpenAI's LLMs to generate descriptions. GenAI features can be configured at # to Google or OpenAI's LLMs to generate descriptions. GenAI features can be configured at
# the camera level to enhance privacy for indoor cameras. # the camera level to enhance privacy for indoor cameras.
# NOTE: genai is a map of named providers. Each key is a name you choose for the provider, # NOTE: genai is a map of named providers. Each key is a name you choose for the provider,
# and each role (chat, descriptions, embeddings, transcribe) may be assigned to exactly one provider. # and each role (chat, descriptions, embeddings) may be assigned to exactly one provider.
genai: genai:
# Required: name of the provider (chosen by you, used to reference it elsewhere) # Required: name of the provider (chosen by you, used to reference it elsewhere)
my_provider: my_provider:
@@ -816,13 +813,11 @@ genai:
# Required: The model to use with the provider. # Required: The model to use with the provider.
model: gemini-1.5-flash model: gemini-1.5-flash
# Optional: Roles this provider handles (default: shown below) # Optional: Roles this provider handles (default: shown below)
# Each role (chat, descriptions, embeddings, transcribe) must be assigned to exactly # Each role (chat, descriptions, embeddings) must be assigned to exactly one provider.
# one provider.
roles: roles:
- chat - chat
- descriptions - descriptions
- embeddings - embeddings
- transcribe
# Optional additional args to pass to the GenAI Provider (default: None) # Optional additional args to pass to the GenAI Provider (default: None)
provider_options: provider_options:
keep_alive: -1 keep_alive: -1
@@ -835,19 +830,13 @@ genai:
audio_transcription: audio_transcription:
# Optional: Enable live and speech event audio transcription (default: shown below) # Optional: Enable live and speech event audio transcription (default: shown below)
enabled: False enabled: False
# Optional: The transcription backend (default: shown below)
# Either 'whisper' for Frigate's built-in local models, or the name of a genai
# provider that has 'transcribe' in its roles. device and model_size are ignored
# when a genai provider is named.
model: whisper
# Optional: The device to run the models on for live transcription. (default: shown below) # Optional: The device to run the models on for live transcription. (default: shown below)
device: CPU device: CPU
# Optional: Set the model size used for live transcription. (default: shown below) # Optional: Set the model size used for live transcription. (default: shown below)
model_size: small model_size: small
# Optional: Set the language used for transcription translation. (default: shown below) # Optional: Set the language used for transcription translation. (default: shown below)
# Use 'auto' to let the model detect the language, or a language code from # List of language codes: https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10
# https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10 language: en
language: auto
# Optional: Configuration for classification models # Optional: Configuration for classification models
classification: classification:
@@ -903,21 +892,6 @@ live:
streams: streams:
main_stream: main_stream_name main_stream: main_stream_name
sub_stream: sub_stream_name sub_stream: sub_stream_name
# Optional: Lower-quality live streams transcoded by go2rtc while someone is watching.
# NOTE: Set at the camera level only.
transcode:
# Optional: Enable transcoded streams (default: shown below)
enabled: False
# Optional: go2rtc stream to transcode (default: the first live stream)
source: main_stream_name
# Optional: One transcoded stream per quality (default: shown below)
qualities:
- height: 720
bitrate: 1200
- height: 480
bitrate: 500
- height: 360
bitrate: 250
# Optional: Set the height of the jsmpeg stream. (default: 720) # Optional: Set the height of the jsmpeg stream. (default: 720)
# This must be less than or equal to the height of the detect stream. Lower resolutions # This must be less than or equal to the height of the detect stream. Lower resolutions
# reduce bandwidth required for viewing the jsmpeg stream. Width is computed to match known aspect ratio. # reduce bandwidth required for viewing the jsmpeg stream. Width is computed to match known aspect ratio.
+7 -79
View File
@@ -204,7 +204,7 @@ Frequently-heard labels like `speech` can generate a lot of events, and each eve
### Audio Transcription ### Audio Transcription
Frigate supports fully local audio transcription using either `sherpa-onnx` or OpenAI's open-source Whisper models via `faster-whisper`, and can alternatively offload transcription to a [GenAI provider](#genai-provider). The goal of this feature is to support Semantic Search for `speech` audio events. Frigate is not intended to act as a continuous, fully-automatic speech transcription service. Automatically transcribing all speech (or queuing many audio events for transcription) requires substantial CPU (or GPU) resources and is impractical on most systems. For this reason, transcriptions for events are initiated manually from the UI or the API rather than being run continuously in the background. Frigate supports fully local audio transcription using either `sherpa-onnx` or OpenAI's open-source Whisper models via `faster-whisper`. The goal of this feature is to support Semantic Search for `speech` audio events. Frigate is not intended to act as a continuous, fully-automatic speech transcription service. Automatically transcribing all speech (or queuing many audio events for transcription) requires substantial CPU (or GPU) resources and is impractical on most systems. For this reason, transcriptions for events are initiated manually from the UI or the API rather than being run continuously in the background.
:::info :::info
@@ -224,7 +224,6 @@ To enable transcription, configure it globally and optionally disable for specif
**Global:** Navigate to <NavPath path="Settings > Enrichments > Audio transcription" />. **Global:** Navigate to <NavPath path="Settings > Enrichments > Audio transcription" />.
- Set **Enable audio transcription** to on - Set **Enable audio transcription** to on
- Set **Audio transcription model or GenAI provider name** to `whisper` for Frigate's built-in local models, or to the name of a GenAI provider
- Set **Transcription device** to the desired device - Set **Transcription device** to the desired device
- Set **Model size** to the desired size - Set **Model size** to the desired size
@@ -236,7 +235,6 @@ To enable transcription, configure it globally and optionally disable for specif
```yaml ```yaml
audio_transcription: audio_transcription:
enabled: True enabled: True
model: whisper
device: ... device: ...
model_size: ... model_size: ...
``` ```
@@ -265,88 +263,20 @@ The optional config parameters that can be set at the global level include:
- **`enabled`**: Enable or disable the audio transcription feature. - **`enabled`**: Enable or disable the audio transcription feature.
- Default: `False` - Default: `False`
- It is recommended to only configure the features at the global level, and enable it at the individual camera level. - It is recommended to only configure the features at the global level, and enable it at the individual camera level.
- **`model`**: The transcription backend.
- Default: `whisper`
- `whisper` uses Frigate's built-in local models, described by `device` and `model_size` below.
- Any other value must name a key in your `genai` config whose entry has `transcribe` in its `roles`. See [GenAI Provider](#genai-provider).
- **`device`**: Device to use to run transcription and translation models. - **`device`**: Device to use to run transcription and translation models.
- Default: `CPU` - Default: `CPU`
- This can be `CPU` or `GPU`. The `sherpa-onnx` models are lightweight and run on the CPU only. The `whisper` models can run on GPU but are only supported on CUDA hardware. - This can be `CPU` or `GPU`. The `sherpa-onnx` models are lightweight and run on the CPU only. The `whisper` models can run on GPU but are only supported on CUDA hardware.
- Ignored when `model` names a GenAI provider.
- **`model_size`**: The size of the model used for live transcription. - **`model_size`**: The size of the model used for live transcription.
- Default: `small` - Default: `small`
- This can be `small` or `large`. The `small` setting uses `sherpa-onnx` models that are fast, lightweight, and always run on the CPU but are not as accurate as the `whisper` model. - This can be `small` or `large`. The `small` setting uses `sherpa-onnx` models that are fast, lightweight, and always run on the CPU but are not as accurate as the `whisper` model.
- This config option applies to **live transcription only**. With `model: whisper`, recorded `speech` events always use a different `whisper` model (and can be accelerated for CUDA hardware if available with `device: GPU`). - This config option applies to **live transcription only**. Recorded `speech` events will always use a different `whisper` model (and can be accelerated for CUDA hardware if available with `device: GPU`).
- Ignored when `model` names a GenAI provider. - **`language`**: Defines the language used by `whisper` to translate `speech` audio events (and live audio only if using the `large` model).
- **`language`**: Defines the language used to transcribe and translate `speech` audio events (and live audio only if using the `large` model or a GenAI provider). - Default: `en`
- Default: `auto` - You must use a valid [language code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10).
- `auto` lets the model detect the language itself, which most models do well. Set an explicit language only if detection is picking the wrong one.
- Otherwise you must use a valid [language code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10).
- Transcriptions for `speech` events are translated. - Transcriptions for `speech` events are translated.
- Live audio is translated only if you are using the `large` model. The `small` `sherpa-onnx` model is English-only. - Live audio is translated only if you are using the `large` model. The `small` `sherpa-onnx` model is English-only.
The only field that is valid at the camera level is `enabled`. In particular `model` is global only: the transcription backend is a process-wide resource shared by every camera. The only field that is valid at the camera level is `enabled`.
#### GenAI Provider
Frigate can send audio to a GenAI provider for transcription when that provider has the `transcribe` role. This is useful if you already run a GenAI provider, or if you do not have the CPU/GPU headroom for a local whisper model. See [Provider support](/configuration/genai/genai_config#provider-support) for which providers can serve this role. The model must accept audio: either a dedicated ASR model such as Qwen3-ASR, or a general multimodal model that accepts audio.
To use a GenAI provider for audio transcription:
1. Configure a GenAI provider with `transcribe` in its `roles`.
2. Set the audio transcription model to that GenAI config key (e.g. `whisper_cloud`).
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > Enrichments > Audio transcription" />.
| Field | Description |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Audio transcription model or GenAI provider name** | Set to the GenAI config key (e.g. `whisper_cloud`) to use a configured GenAI provider for transcription |
The GenAI provider must also be configured with the `transcribe` role under <NavPath path="Settings > Enrichments > Generative AI" />.
</TabItem>
<TabItem value="yaml">
```yaml
genai:
whisper_cloud:
provider: openai
api_key: your-api-key
model: gpt-transcribe
roles:
- transcribe
audio_transcription:
enabled: True
model: whisper_cloud
language: en
```
</TabItem>
</ConfigTabs>
:::warning
**Give `transcribe` its own `genai` entry.** A `genai` entry has a single `model` string that is shared by every role it holds, so `roles: [descriptions, transcribe]` would send the same model name to both the chat endpoint and the transcription endpoint. Transcription models and chat models are almost never the same model, so define a dedicated entry as shown above.
:::
:::warning
**Live transcription against a metered provider is billed continuously.** In live mode Frigate uploads an overlapping ~2 second window of audio roughly once per second, per camera, for as long as audio stays above that camera's `audio.min_volume`. Windows below that threshold are never uploaded, which is what keeps a quiet camera near zero requests, but a camera pointed at a busy street will keep sending.
Three things keep this opt-in: `transcribe` is not one of the default roles, live transcription is off by default, and the volume gate suppresses silence. Transcription of recorded `speech` events is unaffected - it remains a manual, one-request-per-event action.
:::
`device` and `model_size` have no effect on this path and no local model is ever downloaded.
`language` defaults to `auto`, which sends no language hint and lets the model detect it. Most audio models detect language well, so leave it on `auto` unless detection is picking the wrong one.
When set explicitly, it is sent as the transcription endpoint's native `language` parameter for OpenAI, Azure, and llama.cpp, and as part of the prompt for Gemini. This matters for dedicated ASR models such as Qwen3-ASR: they read the prompt as contextual biasing rather than as an instruction, so a language named in the prompt is ignored, while the endpoint parameter is honored.
#### Live transcription #### Live transcription
@@ -362,8 +292,6 @@ Results can be error-prone due to a number of factors, including:
For speech sources close to the camera with minimal background noise, use the `small` model. For speech sources close to the camera with minimal background noise, use the `small` model.
A [GenAI provider](#genai-provider) is generally the most accurate option for live transcription, at the cost of a network round trip per window. That round trip has to stay under about a second to keep up with the audio; if it does not, Frigate drops the oldest buffered audio rather than letting the backlog grow.
If you have CUDA hardware, you can experiment with the `large` `whisper` model on GPU. Performance is not quite as fast as the `sherpa-onnx` `small` model, but live transcription is far more accurate. Using the `large` model with CPU will likely be too slow for real-time transcription. If you have CUDA hardware, you can experiment with the `large` `whisper` model on GPU. Performance is not quite as fast as the `sherpa-onnx` `small` model, but live transcription is far more accurate. Using the `large` model with CPU will likely be too slow for real-time transcription.
#### Transcription and translation of `speech` audio events #### Transcription and translation of `speech` audio events
@@ -380,7 +308,7 @@ Only one `speech` event may be transcribed at a time. Frigate does not automatic
::: :::
With `model: whisper`, recorded `speech` events always use a `whisper` model, regardless of the `model_size` config setting. Without a supported Nvidia GPU, generating transcriptions for longer `speech` events may take a fair amount of time, so be patient. With a [GenAI provider](#genai-provider), the recorded clip is sent to the provider instead and no local model is used. Recorded `speech` events will always use a `whisper` model, regardless of the `model_size` config setting. Without a supported Nvidia GPU, generating transcriptions for longer `speech` events may take a fair amount of time, so be patient.
#### FAQ #### FAQ
+3 -3
View File
@@ -169,10 +169,10 @@ The [Add Camera Wizard](cameras.md#adding-a-camera-with-the-add-camera-wizard) i
1. Click **Add Camera** in <NavPath path="Settings > Global configuration > Camera management" />. 1. Click **Add Camera** in <NavPath path="Settings > Global configuration > Camera management" />.
2. Choose **Manual selection** as the stream detection method and select **Reolink** as the camera brand. 2. Choose **Manual selection** as the stream detection method and select **Reolink** as the camera brand.
3. The wizard queries the camera and automatically uses an http-flv stream for cameras 5MP and lower. For higher resolution cameras, it tries the http-flv stream first and falls back to RTSP when the camera does not support it. 3. The wizard queries the camera and automatically uses an http-flv stream for cameras 5MP and lower, or an RTSP stream for higher resolution cameras.
4. The wizard turns on **Use stream compatibility mode** for the http-flv stream it selects. Enable it for any other http-flv stream you add, such as the sub stream. 4. In the validation step, enable **Use stream compatibility mode** for http-flv streams when the wizard recommends it.
If you use the **Probe camera** method instead, the discovered stream URLs will be RTSP. For Reolink cameras 5MP and lower, where http-flv is recommended, the wizard will show a warning in the validation step. If you use the **Probe camera** method instead, the discovered stream URLs will be RTSP. For Reolink cameras where http-flv is recommended, the wizard will show a warning in the validation step.
The wizard covers standard single-camera setups. For two way talk, cameras connected through a Reolink NVR, or audio transcoding for WebRTC live view, configure the camera manually as shown below. The wizard covers standard single-camera setups. For two way talk, cameras connected through a Reolink NVR, or audio transcoding for WebRTC live view, configure the camera manually as shown below.
@@ -85,14 +85,6 @@ An optional config, `save_attempts`, can be set as a key under the model name. T
</TabItem> </TabItem>
</ConfigTabs> </ConfigTabs>
## Review items
When a model's state changes while its camera has an active review item, the change is recorded on that review item. This includes changes in the few seconds before the item starts, such as a garage door opening just before the car is detected. State changes never create or extend review items on their own, and the first state reported after Frigate starts is not recorded as a change.
Recorded changes appear in the review item's data as `classification_state_changes` (see the [`frigate/reviews`](/integrations/mqtt#frigatereviews) MQTT topic) and are passed to [GenAI review summaries](/configuration/genai/genai_review) as facts, so a description can note that a gate was opened during the activity.
Change times are most accurate with `motion: true`. A model that only runs on an `interval` notices a change at its next run, so the change may be recorded late or attached to a later review item.
## Training the model ## Training the model
Creating and training the model is done within the Frigate UI using the `Classification` page. The process consists of three steps: Creating and training the model is done within the Frigate UI using the `Classification` page. The process consists of three steps:
+11 -13
View File
@@ -17,19 +17,17 @@ Hardware acceleration arguments tell FFmpeg to decode your camera's video stream
See [the hardware acceleration docs](/configuration/hardware_acceleration_video.md) for details on setting up hardware acceleration for your GPU / iGPU, then select the preset that matches your hardware. See [the hardware acceleration docs](/configuration/hardware_acceleration_video.md) for details on setting up hardware acceleration for your GPU / iGPU, then select the preset that matches your hardware.
| Preset (YAML config) | UI Label | Usage | Notes | | Preset (YAML config) | UI Label | Usage | Notes |
| ------------------------- | ----------------------- | --------------------------------------------- | --------------------------------------------------------------- | | --------------------- | ----------------------- | --------------------------------- | --------------------------------------------------------------- |
| preset-rpi-64-h264 | Raspberry Pi (H.264) | 64-bit Raspberry Pi, H.264 stream | | | preset-rpi-64-h264 | Raspberry Pi (H.264) | 64-bit Raspberry Pi, H.264 stream | |
| preset-rpi-64-h265 | Raspberry Pi (H.265) | 64-bit Raspberry Pi, H.265 stream | | | preset-rpi-64-h265 | Raspberry Pi (H.265) | 64-bit Raspberry Pi, H.265 stream | |
| preset-apple-silicon-h264 | Apple Silicon (H.264) | Apple Silicon Mac under lighter, H.264 stream | Needs the `lighter.sh/video` device | | preset-vaapi | VAAPI (Intel/AMD GPU) | Intel or AMD GPU via VAAPI | Check the hwaccel docs to ensure the correct driver is selected |
| preset-apple-silicon-h265 | Apple Silicon (H.265) | Apple Silicon Mac under lighter, H.265 stream | Needs the `lighter.sh/video` device | | preset-intel-qsv-h264 | Intel QuickSync (H.264) | Intel QuickSync, H.264 stream | If you have issues, use the VAAPI preset instead |
| preset-vaapi | VAAPI (Intel/AMD GPU) | Intel or AMD GPU via VAAPI | Check the hwaccel docs to ensure the correct driver is selected | | preset-intel-qsv-h265 | Intel QuickSync (H.265) | Intel QuickSync, H.265 stream | If you have issues, use the VAAPI preset instead |
| preset-intel-qsv-h264 | Intel QuickSync (H.264) | Intel QuickSync, H.264 stream | If you have issues, use the VAAPI preset instead | | preset-nvidia | NVIDIA GPU | NVIDIA GPU | |
| preset-intel-qsv-h265 | Intel QuickSync (H.265) | Intel QuickSync, H.265 stream | If you have issues, use the VAAPI preset instead | | preset-jetson-h264 | NVIDIA Jetson (H.264) | NVIDIA Jetson, H.264 stream | |
| preset-nvidia | NVIDIA GPU | NVIDIA GPU | | | preset-jetson-h265 | NVIDIA Jetson (H.265) | NVIDIA Jetson, H.265 stream | |
| preset-jetson-h264 | NVIDIA Jetson (H.264) | NVIDIA Jetson, H.264 stream | | | preset-rkmpp | Rockchip RKMPP | Rockchip MPP | Use an image with the `-rk` suffix and run in privileged mode |
| preset-jetson-h265 | NVIDIA Jetson (H.265) | NVIDIA Jetson, H.265 stream | |
| preset-rkmpp | Rockchip RKMPP | Rockchip MPP | Use an image with the `-rk` suffix and run in privileged mode |
<ConfigTabs> <ConfigTabs>
<TabItem value="ui"> <TabItem value="ui">
+9 -41
View File
@@ -43,19 +43,7 @@ genai:
The examples on this page all use `my_provider`, but the name is arbitrary and is only used to reference the provider elsewhere in the config (for example, `semantic_search.model`). The examples on this page all use `my_provider`, but the name is arbitrary and is only used to reference the provider elsewhere in the config (for example, `semantic_search.model`).
Each provider handles one or more **roles**: `chat`, `descriptions`, `embeddings`, and `transcribe`. A provider handles the first three by default; `transcribe` must always be listed explicitly. Each role may be assigned to exactly one provider. Define a single provider if you want it to do everything, or split the roles across several providers using the `roles` option. Not every provider supports every role; see [Provider support](#provider-support). Each provider handles one or more **roles**: `chat`, `descriptions`, and `embeddings`. A provider handles all three by default, and each role may be assigned to exactly one provider. Define a single provider if you want it to do everything, or split the roles across several providers using the `roles` option.
### Provider support
| Provider | Descriptions | Chat | Embeddings | Transcription |
| ----------------------------- | :----------: | :--: | :--------: | :-----------: |
| llama.cpp (`llamacpp`) | ✅ | ✅ | ✅ | ✅ |
| Ollama (`ollama`) | ✅ | ✅ | ✅ | ❌ |
| OpenAI (`openai`) | ✅ | ✅ | ❌ | ✅ |
| Azure OpenAI (`azure_openai`) | ✅ | ✅ | ❌ | ✅ |
| Google Gemini (`gemini`) | ✅ | ✅ | ❌ | ✅ |
A ✅ means Frigate can use the provider for that feature. The configured model must also support it: a vision model for descriptions and chat, a multimodal embedding model for embeddings (see [Embedding models](#embedding-models)), and an audio-capable model for transcription. Some features also need extra provider setup, covered in each provider's section below. OpenAI-compatible servers use the `openai` provider, so they follow the OpenAI row.
If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`. If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
@@ -75,31 +63,19 @@ Running Generative AI models on CPU is not recommended, as high inference times
You must use a vision-capable model with Frigate. The following models are recommended for local deployment of the `descriptions` and `chat` roles: You must use a vision-capable model with Frigate. The following models are recommended for local deployment of the `descriptions` and `chat` roles:
| Model | Review [frame mode](/configuration/genai/genai_review#frame-mode) | Notes | | Model | Notes |
| ------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `qwen3-vl` | `frames` | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object. Follows a sequence of frames on its own. | | `qwen3-vl` | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object. |
| `qwen3.6`/`qwen3.8` | `frames` | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. | | `qwen3.6`/`qwen3.8` | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
| `gemma4` | `annotated_frames` | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action. Loses track of activity that repeats or reverses, so it benefits from annotated frames. | | `gemma4` | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action. |
#### Embedding models #### Embedding models
The `embeddings` role needs a different kind of model. Text queries are matched against the stored image embeddings, so the model must be trained to place images and text into the same vector space. A chat or description model will still return vectors when asked, but those vectors are not trained for retrieval and text searches will return poor matches with no error to indicate why. The `embeddings` role needs a different kind of model. Text queries are matched against the stored image embeddings, so the model must be trained to place images and text into the same vector space. A chat or description model will still return vectors when asked, but those vectors are not trained for retrieval and text searches will return poor matches with no error to indicate why.
| Model | Notes | | Model | Notes |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `embeddinggemma-2` | Multimodal embeddings for [Semantic Search](/configuration/semantic_search#genai-provider). Strong semantic search accuracy with efficient inference on a small model. | | `qwen3-vl-embedding` | Multimodal embeddings for [Semantic Search](/configuration/semantic_search#genai-provider). Must be served by llama.cpp started with `--embeddings` and `--mmproj`. |
| `qwen3-vl-embedding` | Multimodal embeddings for [Semantic Search](/configuration/semantic_search#genai-provider). Good performance, large model that requires strong hardware for inference. |
#### Transcription models
The `transcribe` role needs a model that accepts audio input. A text-only or vision-only model cannot serve this role. The following are recommended for local deployment of the `transcribe` role:
| Model | Notes |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `qwen3-asr` | Dedicated speech recognition model covering 30 languages, and the better choice for transcription quality. It only transcribes, so it cannot be shared with the `descriptions` or `chat` roles. |
| `gemma4` | General multimodal model that accepts audio as well as images, so one served model can cover `transcribe` alongside the other roles. Transcript quality is below `qwen3-asr`, particularly on noisy audio. |
Both must be served by llama.cpp started with the matching audio `--mmproj`. llama.cpp only reports audio support when an audio projector is loaded. Without it Frigate sees the model as text-only and the `transcribe` role is unavailable in the UI. Frigate transcribes through the server's `/v1/audio/transcriptions` route, which llama.cpp serves for any audio-capable model.
:::info :::info
@@ -165,10 +141,6 @@ genai:
Frigate queries the llama.cpp server for the model's context size at startup and logs it along with the other detected capabilities. If `context_size` is set in `provider_options`, that value is always used instead, even when the server reports its own. Frigate queries the llama.cpp server for the model's context size at startup and logs it along with the other detected capabilities. If `context_size` is set in `provider_options`, that value is always used instead, even when the server reports its own.
#### Embeddings
To serve the `embeddings` role for [Semantic Search](/configuration/semantic_search#genai-provider), start the llama.cpp server with `--embeddings`, plus `--mmproj` for image support. See the [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) for details.
### Ollama ### Ollama
[Ollama](https://ollama.com/) allows you to self-host large language models and keep everything running locally. It is highly recommended to host this server on a machine with an Nvidia graphics card, or on a Apple silicon Mac for best performance. [Ollama](https://ollama.com/) allows you to self-host large language models and keep everything running locally. It is highly recommended to host this server on a machine with an Nvidia graphics card, or on a Apple silicon Mac for best performance.
@@ -214,10 +186,6 @@ genai:
</TabItem> </TabItem>
</ConfigTabs> </ConfigTabs>
#### Embeddings
Ollama can serve the `embeddings` role for [Semantic Search](/configuration/semantic_search#genai-provider). Embedding images requires Ollama 0.40.1 or newer and an embedding model with a vision encoder, such as `embeddinggemma-2:440m`. For a saved provider, the UI hides the role unless Ollama reports its model as an embedding model. Use a separate provider entry for the embedding model rather than adding the role to a vision chat model.
### OpenAI-Compatible ### OpenAI-Compatible
Frigate supports any provider that implements the OpenAI API standard. This includes self-hosted solutions like [vLLM](https://docs.vllm.ai/), [LocalAI](https://localai.io/), and other OpenAI-compatible servers. Frigate supports any provider that implements the OpenAI API standard. This includes self-hosted solutions like [vLLM](https://docs.vllm.ai/), [LocalAI](https://localai.io/), and other OpenAI-compatible servers.
@@ -192,45 +192,6 @@ review:
</TabItem> </TabItem>
</ConfigTabs> </ConfigTabs>
### Frame Mode
Review items are sent to the model as a sequence of still frames. Some models follow that sequence well on their own; others lose track of activity that repeats or reverses, and describe a single trip when the subject actually made several. The `frame_mode` option controls how those frames are presented.
- `frames` (default): the prompt followed by the frames, exactly as earlier versions of Frigate sent them.
- `annotated_frames`: each frame is preceded by its frame number and elapsed time, along with notes describing what the object tracker recorded at that moment, such as an object being first detected, starting to move, turning around, stopping, or no longer being detected.
The notes come from tracking data rather than from the images, so they describe activity the model may not have picked up on its own. In testing with a person carrying three waste bins to the curb one at a time, `gemma4` described a single trip on every attempt with `frames`, and consistently described multiple trips with `annotated_frames`. Models that already handle these sequences well, such as the `qwen3-vl` family, gain little and should stay on `frames`.
Changes reported by [state classification](/configuration/custom_classification/state_classification#review-items) models during the review item are listed in the prompt in both modes. `annotated_frames` also notes each change before the frame where it happened.
Annotated mode also caps the number of frames, since the notes already establish the order of events and extra near-duplicate frames tend to crowd out the middle of a clip. Longer review items are sampled more sparsely as a result, and typically use fewer tokens than `frames` mode for the same item.
:::note
Annotated mode needs tracking data for the review item. If none is available, Frigate falls back to sending plain frames for that item.
:::
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > Global configuration > Review" />.
- Set **GenAI config > Frame mode** to the desired mode (e.g., `annotated_frames`)
</TabItem>
<TabItem value="yaml">
```yaml {4}
review:
genai:
enabled: true
frame_mode: annotated_frames
```
</TabItem>
</ConfigTabs>
### Response Style ### Response Style
Different models respond to the built-in prompt with very different writing styles: some produce natural narration while others sound short and mechanical. The `response_style` option selects a writing style preset that rewords the prompt's instructions for the user-facing fields (the title, short summary, and scene description). Presets replace those instructions rather than adding extra ones, so the model never receives competing style directions. Different models respond to the built-in prompt with very different writing styles: some produce natural narration while others sound short and mechanical. The `response_style` option selects a writing style preset that rewords the prompt's instructions for the user-facing fields (the title, short summary, and scene description). Presets replace those instructions rather than adding extra ones, so the model never receives competing style directions.
@@ -43,10 +43,6 @@ Frigate supports presets for optimal hardware accelerated video decoding:
- [RKNN](#rockchip-platform): Frigate can utilize the media engine in RockChip SOCs to accelerate video decoding. - [RKNN](#rockchip-platform): Frigate can utilize the media engine in RockChip SOCs to accelerate video decoding.
**Apple Silicon Mac** <CommunityBadge />
- [lighter](#apple-silicon-mac-lighter): Frigate can utilize the media engine in Apple Silicon Macs to accelerate video decoding, when running under the lighter container runtime.
**Other Hardware** **Other Hardware**
Depending on your system, these presets may not be compatible, and you may need to use manual hwaccel args to take advantage of your hardware. More information on hardware accelerated decoding for ffmpeg can be found here: https://trac.ffmpeg.org/wiki/HWAccelIntro Depending on your system, these presets may not be compatible, and you may need to use manual hwaccel args to take advantage of your hardware. More information on hardware accelerated decoding for ffmpeg can be found here: https://trac.ffmpeg.org/wiki/HWAccelIntro
@@ -537,35 +533,3 @@ output_args:
Make sure that your SoC supports hardware acceleration for your input stream and your input stream is h264 encoding. For example, if your camera streams with h264 encoding, your SoC must be able to de- and encode with it. If you are unsure whether your SoC meets the requirements, take a look at the datasheet. Make sure that your SoC supports hardware acceleration for your input stream and your input stream is h264 encoding. For example, if your camera streams with h264 encoding, your SoC must be able to de- and encode with it. If you are unsure whether your SoC meets the requirements, take a look at the datasheet.
::: :::
## Apple Silicon Mac (lighter)
[lighter](https://github.com/fieldwork-ai/lighter) is an open-source container runtime for macOS. It gives a container the Mac's media engine as a standard V4L2 decoder, backed by VideoToolbox, so Frigate decodes H.264 and H.265 streams in hardware with the ffmpeg it already ships. It works on M1 and newer Macs with lighter 0.9.2 or newer.
Give the container the video device. With Docker Compose:
```yaml {4-5}
services:
frigate:
...
devices:
- lighter.sh/video=all
```
Or with `docker run`, add `--device lighter.sh/video=all`.
Then set the preset for the codec your cameras stream. The decoder is specific to the codec, so if your cameras mix H.264 and H.265, set the preset for the most common codec globally and override it on the other cameras:
```yaml
ffmpeg:
hwaccel_args: preset-apple-silicon-h264
cameras:
garage: # an H.265 camera
ffmpeg:
hwaccel_args: preset-apple-silicon-h265
```
The presets decode on the media engine and encode the Birdseye restream and timelapses there too. Scaling to the detect resolution runs on the CPU, as ffmpeg's V4L2 decoders cannot scale.
lighter can also run object detection on the Mac's Neural Engine; see [Apple Neural Engine (lighter)](object_detectors.md#apple-neural-engine-lighter).
@@ -378,10 +378,10 @@ Navigate to <NavPath path="Settings > Camera configuration > Object detection" /
Navigate to <NavPath path="Settings > Camera configuration > Objects" />. Navigate to <NavPath path="Settings > Camera configuration > Objects" />.
| Field | Description | | Field | Description |
| --------------------------------------------------------- | ------------------- | | ---------------------------------------------- | ------------------- |
| **Objects to track** | Add `license_plate` | | **Objects to track** | Add `license_plate` |
| **Object filters > License Plate > Confidence threshold** | Set to `0.7` | | **Object filters > License Plate > Threshold** | Set to `0.7` |
Navigate to <NavPath path="Settings > Camera configuration > Motion detection" />. Navigate to <NavPath path="Settings > Camera configuration > Motion detection" />.
+1 -28
View File
@@ -92,7 +92,7 @@ go2rtc:
### Setting Streams For Live UI ### Setting Streams For Live UI
You can configure Frigate to allow manual selection of the stream you want to view in the Live UI. For example, you may want to view your camera's substream on mobile devices, but the full resolution stream on desktop devices. Setting the streams list will populate a dropdown in the UI's Live view that allows you to choose between the streams. This stream setting is _per device_ and is saved in your browser's local storage. When a camera has more than one stream, the dropdown also offers **Auto**, which is used until you pick a specific stream. Auto starts on the first stream, steps down the list when your connection can't keep up, and steps back up when it recovers. To retry the top stream right away, select **Try highest quality** under the stream picker. List streams from highest to lowest quality, and avoid names that are plain numbers (such as `720`), which the browser sorts ahead of the others. In the UI, drag streams to reorder them, or use **Auto order** to sort them by measured bitrate. You can configure Frigate to allow manual selection of the stream you want to view in the Live UI. For example, you may want to view your camera's substream on mobile devices, but the full resolution stream on desktop devices. Setting the streams list will populate a dropdown in the UI's Live view that allows you to choose between the streams. This stream setting is _per device_ and is saved in your browser's local storage.
Additionally, when creating and editing camera groups in the UI, you can choose the stream you want to use for your camera group's Live dashboard. Additionally, when creating and editing camera groups in the UI, you can choose the stream you want to use for your camera group's Live dashboard.
@@ -158,26 +158,6 @@ cameras:
</TabItem> </TabItem>
</ConfigTabs> </ConfigTabs>
### Transcoded streams
When a camera has no suitable sub stream, Frigate can add lower-quality streams that go2rtc transcodes to H.264 while someone is watching. They appear in the stream list like any other stream, so Auto mode can step down to them. Enable them under <NavPath path="Settings > Camera configuration > Live playback" />, or in YAML:
```yaml
cameras:
test_cam:
live:
transcode:
enabled: true
source: test_cam # optional, defaults to the first live stream
qualities:
- height: 720
bitrate: 1200 # kbps
- height: 480
bitrate: 500
```
Each quality becomes a go2rtc stream named `<camera>_transcode_<height>p`. go2rtc picks a hardware encoder automatically and falls back to the CPU, which costs CPU for each transcode while it is being watched. Check go2rtc's `api/ffmpeg/hardware` page to see which encoder it found. Using a sub stream as the `source` lowers the cost.
### WebRTC extra configuration: ### WebRTC extra configuration:
WebRTC works by creating a TCP or UDP connection on port `8555`. However, it requires additional configuration: WebRTC works by creating a TCP or UDP connection on port `8555`. However, it requires additional configuration:
@@ -383,13 +363,6 @@ When your browser runs into problems playing back your camera streams, it will l
- `Safari reported InvalidStateError.` - `Safari reported InvalidStateError.`
- `Safari reported decoding errors.` - `Safari reported decoding errors.`
- **mse-codec**
- What it means: go2rtc has no codec for this stream that the browser can play.
- What to try: Pick a stream with a codec the browser supports (H.264 is the most compatible), or use a browser that supports the stream's codec. In Auto, Frigate skips this stream for the rest of the session.
- Possible console messages from the player code:
- `mse: streams: codecs not matched: ...`
- **stalled** - **stalled**
- What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving). - What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving).
- What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in <NavPath path="Settings > UI" /> . - What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in <NavPath path="Settings > UI" /> .
+12 -117
View File
@@ -24,18 +24,15 @@ Frigate supports multiple different detectors that work on different types of ha
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices. - [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
- [Hailo](#hailo): The Hailo-8, Hailo-8L and Hailo-8R AI Acceleration modules are available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices. - [Hailo](#hailo): The Hailo-8, Hailo-8L and Hailo-8R AI Acceleration modules are available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
- <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms. - <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, offering broad compatibility across various platforms.
**AMD** **AMD**
- [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection. - [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection.
- [ONNX](#onnx): ROCm will automatically be detected and used as a detector in the `-rocm` Frigate image when a supported ONNX model is configured. - [ONNX](#onnx): ROCm will automatically be detected and used as a detector in the `-rocm` Frigate image when a supported ONNX model is configured.
- <CommunityBadge /> [XDNA2](#amd-xdna2): AMD Ryzen AI / XDNA2 NPUs can run object detection through the community-maintained `frigate-xdna` ZMQ sidecar.
**Apple Silicon** **Apple Silicon**
- [Apple Silicon](#apple-silicon-detector): Apple Silicon can run on M1 and newer Apple Silicon devices. - [Apple Silicon](#apple-silicon-detector): Apple Silicon can run on M1 and newer Apple Silicon devices.
- <CommunityBadge /> [ONNX](#apple-neural-engine-lighter): the ONNX detector runs on the Neural Engine of M1 and newer Macs when Frigate runs under the lighter container runtime.
**Intel** **Intel**
@@ -104,36 +101,32 @@ Coral EdgeTPU and MemryX accelerators can only be opened by one process, so thos
### Running more than one model ### Running more than one model
Cameras can be split across models by scene, which is useful when some cameras benefit from a differently trained model, such as thermal cameras. Each model declares the `scene` it is for, and each camera picks one with `detect -> scene`: Cameras can be split across models by scene, which is useful when indoor and outdoor cameras benefit from differently trained models. Each model declares the `scene` it is for, and each camera picks one with `detect -> scene`:
```yaml ```yaml
models: models:
- scene: default - scene: outdoor
path: plus://your-model path: plus://your-outdoor-model
devices: devices:
- edgetpu:pci:0 - edgetpu:pci:0
- scene: thermal - scene: indoor
path: /config/model_cache/thermal.onnx path: /config/model_cache/indoor.onnx
model_type: yolo-generic model_type: yolo-generic
devices: devices:
- openvino:GPU - openvino:GPU
cameras: cameras:
driveway: driveway:
...
backyard_thermal:
detect: detect:
scene: thermal scene: outdoor
...
hallway:
detect:
scene: indoor
... ...
``` ```
A scene is any name made up of letters, numbers, `_`, and `-`. The model with a scene of `default` is used by every camera that does not set one (or sets a scene that no model is configured for), and `default` is used when a model does not declare a scene. Changing a camera's scene requires a restart. Available scenes are `all`, `indoor`, `outdoor`, `indoor_thermal`, and `outdoor_thermal`. A model with a scene of `all` is used by every camera that does not set one, and `all` is the default when a model does not declare a scene. Changing a camera's scene requires a restart.
:::warning
Scenes are for running **different** models. Do not configure the same model under several scenes to dedicate a detector to specific cameras: every detector of a model already serves every camera using it, and splitting them only leaves some detectors idle while others fall behind. Frigate detects models that use the same model file, even under a different path or file name, combines them into one model with all of their hardware, and logs a warning.
:::
### Choosing a model size ### Choosing a model size
@@ -490,7 +483,7 @@ See [ONNX supported models](#onnx) for supported models, there are some caveats:
## ONNX ## ONNX
ONNX is an open format for building machine learning models, Frigate supports running ONNX models on CPU, OpenVINO, ROCm, TensorRT, and a Mac's Neural Engine. On startup Frigate will automatically try to use a GPU if one is available. ONNX is an open format for building machine learning models, Frigate supports running ONNX models on CPU, OpenVINO, ROCm, and TensorRT. On startup Frigate will automatically try to use a GPU if one is available.
:::info :::info
@@ -506,9 +499,6 @@ If the correct build is used for your GPU then the GPU will be detected and used
- Nvidia GPUs will automatically be detected and used with the ONNX detector in the `-tensorrt` Frigate image. - Nvidia GPUs will automatically be detected and used with the ONNX detector in the `-tensorrt` Frigate image.
- Jetson devices will automatically be detected and used with the ONNX detector in the `-tensorrt-jp6` Frigate image. - Jetson devices will automatically be detected and used with the ONNX detector in the `-tensorrt-jp6` Frigate image.
- **Apple Silicon Mac** <CommunityBadge />
- The Neural Engine will automatically be detected and used with the ONNX detector when Frigate runs under lighter with its Neural Engine device. See [Apple Neural Engine (lighter)](#apple-neural-engine-lighter).
::: :::
:::tip :::tip
@@ -524,22 +514,6 @@ models:
::: :::
### Apple Neural Engine (lighter) {#apple-neural-engine-lighter}
[lighter](https://github.com/fieldwork-ai/lighter) is an open-source container runtime for macOS. A container started with its `lighter.sh/ane` device gets an ONNX Runtime execution provider that runs models on the Mac's Neural Engine, and the ONNX detector uses it automatically, with the same models and configuration as on any other hardware. It works on M1 and newer Macs with lighter 0.9.2 or newer.
Give the Frigate container the Neural Engine device. With Docker Compose:
```yaml
services:
frigate:
image: ghcr.io/blakeblackshear/frigate:stable-standard-arm64
devices:
- lighter.sh/ane=all
```
Or with `docker run`, add `--device lighter.sh/ane=all`. Frigate then reports the Neural Engine under **Settings > System > Detection models**, and the ONNX detector's model loads on it. lighter can also decode camera streams on the Mac's media engine; see [Video Decoding](hardware_acceleration_video.md#apple-silicon-mac-lighter).
### Configuration {#configuration-onnx} ### Configuration {#configuration-onnx}
<ModelConfigDropdown detectorTitle="ONNX" models={objectDetectorsModels.onnx.models} /> <ModelConfigDropdown detectorTitle="ONNX" models={objectDetectorsModels.onnx.models} />
@@ -568,28 +542,6 @@ When using CPU detectors, you can add one CPU detector per camera. Adding more d
# Community Supported Detectors # Community Supported Detectors
## AMD XDNA2
AMD Ryzen AI / XDNA2 NPUs can be used through the community-maintained
[frigate-xdna](https://github.com/mitchins/frigate-xdna) detector sidecar.
The sidecar runs separately from Frigate and connects using Frigate's ZMQ
detector interface.
Currently qualified on **Ryzen AI Max 300 / Strix Halo**. Other XDNA2 devices
are not yet qualified; XDNA1 is unsupported.
Follow the frigate-xdna setup instructions to prepare and start the sidecar
before starting Frigate.
### Configuration {#configuration-xdna2}
Using the detector config below will connect Frigate to the sidecar:
<ModelConfigDropdown detectorTitle="AMD XDNA2" models={objectDetectorsModels.xdna2.models} />
The example assumes Frigate and the sidecar share a Docker network where the
sidecar is named `xdna`.
## MemryX MX3 ## MemryX MX3
This detector is available for use with the MemryX MX3 accelerator M.2 module. Frigate supports the MX3 on compatible hardware platforms, providing efficient and high-performance object detection. This detector is available for use with the MemryX MX3 accelerator M.2 module. Frigate supports the MX3 on compatible hardware platforms, providing efficient and high-performance object detection.
@@ -660,63 +612,6 @@ For detailed instructions on compiling models, refer to the [MemryX Compiler](ht
--- ---
## DEEPX NPU
This detector is available for use with the DEEPX NPU, both the DX-M1 M.2 module and the DX-M1M on the Sixfab AI HAT+ for the Raspberry Pi 5. The configuration below applies unchanged to either form factor. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
See the [installation docs](../frigate/installation.md#deepx-npu) for information on installing the DEEPX kernel driver and runtime on the host and passing the NPU through to the container.
To run a model on a DEEPX NPU, list a `deepx` device on that model.
:::info
The DX-RT Python bindings are not part of the Frigate image. They are downloaded and installed into `/config/.local` the first time a DEEPX device is configured, verified against pinned checksums, and updated automatically when a Frigate release pins a new version. If the container has no internet access, see [Detector runtimes](/frigate/network_requirements#detector-runtimes) for how to provide the files yourself.
:::
### Configuration {#configuration-deepx}
<ModelConfigDropdown detectorTitle="DEEPX" models={objectDetectorsModels.deepx.models} />
Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Two model types are supported:
- `yolo-generic` for YOLO object detection models, the recommended default. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do models compiled with DEEPX's Post-Processing Unit (PPU) support.
- `yolox` for YOLOX models compiled without PPU support, whose raw head needs Frigate's YOLOX decoder. A YOLOX model compiled with PPU support works under either `yolox` or `yolo-generic`.
The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler. The recommended starting point is `yolox-s_640x640_ppu.dxnn`, the fastest ModelZoo model measured through Frigate:
```yaml
models:
- devices:
- deepx:PCIe:0
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: yolo-generic
width: 640
height: 640
```
For PPU models, use a `.dxnn` compiled with DX-COM 2.4.0 or later. Frigate reads the PPU head layout the compiler writes into the file and refuses to load a PPU model without it.
`model_type` must be set to `yolo-generic` or `yolox` to match the model; `yolo-generic` is the recommended default unless the model is a raw YOLOX export. Frigate defaults it to `ssd`, which this detector does not support, so the detector refuses to start on a model that leaves it unset.
`width` and `height` must match the resolution the model was compiled for. Quantization parameters are baked into the `.dxnn` file at compile time, so no normalization is applied on the host and Frigate's default `input_tensor`, `input_pixel_format`, and `input_dtype` values do not need to be overridden.
A DEEPX device is `PCIe:<index>`, as reported on the detector settings page. The NPU daemon multiplexes across processes, so the same device may be listed more than once to run additional inference processes against it:
```yaml
models:
- devices:
- deepx:PCIe:0
- deepx:PCIe:0
```
#### Label maps
The object detection models in the DEEPX ModelZoo are trained on the standard 80-class COCO label set, so `labelmap_path` must be set to `/labelmap/coco-80.txt`. Frigate's default label map uses an extended 91-class COCO scheme, and leaving it in place will cause detections to be reported as the wrong object type. For `yolo-generic` models the label map is also what the detector uses to tell the output layout, so a label map with the wrong number of classes is reported as an error at startup.
---
## NVidia TensorRT Detector ## NVidia TensorRT Detector
Nvidia Jetson devices may be used for object detection using the TensorRT libraries. Due to the size of the additional libraries, this detector is only provided in images with the `-tensorrt-jp6` tag suffix, e.g. `ghcr.io/blakeblackshear/frigate:stable-tensorrt-jp6`. This detector is designed to work with Yolo models for object detection. Nvidia Jetson devices may be used for object detection using the TensorRT libraries. Due to the size of the additional libraries, this detector is only provided in images with the `-tensorrt-jp6` tag suffix, e.g. `ghcr.io/blakeblackshear/frigate:stable-tensorrt-jp6`. This detector is designed to work with Yolo models for object detection.
+10 -10
View File
@@ -45,10 +45,10 @@ Any detection below `min_score` will be immediately thrown out and never tracked
Navigate to <NavPath path="Settings > Global configuration > Objects" /> to set score filters globally. Navigate to <NavPath path="Settings > Global configuration > Objects" /> to set score filters globally.
| Field | Description | | Field | Description |
| -------------------------------------------------- | ---------------------------------------------------------------- | | --------------------------------------- | ---------------------------------------------------------------- |
| **Object filters > Person > Minimum confidence** | Minimum score for a single detection to initiate tracking | | **Object filters > Person > Min Score** | Minimum score for a single detection to initiate tracking |
| **Object filters > Person > Confidence threshold** | Minimum computed (median) score to be considered a true positive | | **Object filters > Person > Threshold** | Minimum computed (median) score to be considered a true positive |
To override score filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" /> and select the camera. To override score filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" /> and select the camera.
@@ -103,12 +103,12 @@ Conceptually, a ratio of 1 is a square, 0.5 is a "tall skinny" box, and 2 is a "
Navigate to <NavPath path="Settings > Global configuration > Objects" /> to set shape filters globally. Navigate to <NavPath path="Settings > Global configuration > Objects" /> to set shape filters globally.
| Field | Description | | Field | Description |
| -------------------------------------------------- | ------------------------------------------------------------------------ | | --------------------------------------- | ------------------------------------------------------------------------ |
| **Object filters > Person > Minimum object area** | Minimum bounding box area in pixels (or decimal for percentage of frame) | | **Object filters > Person > Min Area** | Minimum bounding box area in pixels (or decimal for percentage of frame) |
| **Object filters > Person > Maximum object area** | Maximum bounding box area in pixels (or decimal for percentage of frame) | | **Object filters > Person > Max Area** | Maximum bounding box area in pixels (or decimal for percentage of frame) |
| **Object filters > Person > Minimum aspect ratio** | Minimum width/height ratio of the bounding box | | **Object filters > Person > Min Ratio** | Minimum width/height ratio of the bounding box |
| **Object filters > Person > Maximum aspect ratio** | Maximum width/height ratio of the bounding box | | **Object filters > Person > Max Ratio** | Maximum width/height ratio of the bounding box |
To override shape filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" /> and select the camera. To override shape filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" /> and select the camera.
+8 -8
View File
@@ -70,14 +70,14 @@ Object filters help reduce false positives by constraining the size, shape, and
Navigate to <NavPath path="Settings > Global configuration > Objects" />. Navigate to <NavPath path="Settings > Global configuration > Objects" />.
| Field | Description | | Field | Description |
| -------------------------------------------------- | ------------------------------------------------------------------------ | | --------------------------------------- | ------------------------------------------------------------------------ |
| **Object filters > Person > Minimum object area** | Minimum bounding box area in pixels (or decimal for percentage of frame) | | **Object filters > Person > Min Area** | Minimum bounding box area in pixels (or decimal for percentage of frame) |
| **Object filters > Person > Maximum object area** | Maximum bounding box area in pixels (or decimal for percentage of frame) | | **Object filters > Person > Max Area** | Maximum bounding box area in pixels (or decimal for percentage of frame) |
| **Object filters > Person > Minimum aspect ratio** | Minimum width/height ratio of the bounding box | | **Object filters > Person > Min Ratio** | Minimum width/height ratio of the bounding box |
| **Object filters > Person > Maximum aspect ratio** | Maximum width/height ratio of the bounding box | | **Object filters > Person > Max Ratio** | Maximum width/height ratio of the bounding box |
| **Object filters > Person > Minimum confidence** | Minimum score for the object to initiate tracking | | **Object filters > Person > Min Score** | Minimum score for the object to initiate tracking |
| **Object filters > Person > Confidence threshold** | Minimum computed score to be considered a true positive | | **Object filters > Person > Threshold** | Minimum computed score to be considered a true positive |
To override filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" />. To override filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" />.
+3 -7
View File
@@ -191,12 +191,14 @@ cameras:
detect: detect:
enabled: false enabled: false
record: record:
enabled: true enabled: false
profiles: profiles:
away: away:
enabled: true enabled: true
detect: detect:
enabled: true enabled: true
record:
enabled: true
home: home:
enabled: false enabled: false
``` ```
@@ -249,12 +251,6 @@ Leaving the `objects` section empty (or omitting `track`) does not clear the lis
Fields that require a Frigate restart to take effect cannot be overridden by profiles, since profiles are applied at runtime without restarting. Those fields are hidden when editing a profile override and can only be changed on the base configuration. Fields that require a Frigate restart to take effect cannot be overridden by profiles, since profiles are applied at runtime without restarting. Those fields are hidden when editing a profile override and can only be changed on the base configuration.
### Why can't a profile enable recording when it's disabled in the base config?
Frigate only sets up a camera's recording stream at startup when recording is enabled in the base config, so enabling it later from a profile has no effect. The same applies to turning recording on from the UI or MQTT.
To keep recording off by default, leave `record.enabled: true` in the base config and create a profile that sets `record.enabled: false`. Activate that profile and it will be restored automatically when Frigate starts.
### Can I schedule profiles to be enabled or disabled at certain times? ### Can I schedule profiles to be enabled or disabled at certain times?
Not within Frigate itself. Frigate is an NVR, not an automation platform, so it intentionally does not include a scheduler for activating profiles. Instead, activate profiles from an automation platform that already handles time- and event-based triggers well, such as [Home Assistant](https://www.home-assistant.io/) or [Node-RED](https://nodered.org/). These integrate with Frigate and give you far more robust and flexible scheduling than a built-in scheduler could. Not within Frigate itself. Frigate is an NVR, not an automation platform, so it intentionally does not include a scheduler for activating profiles. Instead, activate profiles from an automation platform that already handles time- and event-based triggers well, such as [Home Assistant](https://www.home-assistant.io/) or [Node-RED](https://nodered.org/). These integrate with Frigate and give you far more robust and flexible scheduling than a built-in scheduler could.
+8 -5
View File
@@ -133,12 +133,13 @@ Switching between V1 and V2 requires reindexing your embeddings. The embeddings
### GenAI Provider ### GenAI Provider
Frigate can use a GenAI provider for semantic search embeddings when that provider has the `embeddings` role. See [Provider support](/configuration/genai/genai_config#provider-support) for which providers can serve this role. Frigate can use a GenAI provider for semantic search embeddings when that provider has the `embeddings` role. Currently, only **llama.cpp** supports multimodal embeddings (both text and images).
To use a GenAI provider for semantic search: To use llama.cpp for semantic search:
1. Configure a GenAI provider with `embeddings` in its `roles`, using a multimodal embedding model (both text and images). See [Embedding models](/configuration/genai/genai_config#embedding-models) for recommendations, and your provider's section of the [GenAI docs](/configuration/genai/genai_config) for any extra setup it needs. 1. Configure a GenAI provider with `embeddings` in its `roles`.
2. Set the semantic search model to the GenAI config key (e.g. `default`). 2. Set the semantic search model to the GenAI config key (e.g. `default`).
3. Start the llama.cpp server with `--embeddings` and `--mmproj` for image support.
<ConfigTabs> <ConfigTabs>
<TabItem value="ui"> <TabItem value="ui">
@@ -173,6 +174,8 @@ semantic_search:
</TabItem> </TabItem>
</ConfigTabs> </ConfigTabs>
The llama.cpp server must be started with `--embeddings` for the embeddings API, and a multi-modal embeddings model. See the [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) for details.
:::note :::note
Switching between Jina models and a GenAI provider requires reindexing. Embeddings from different backends are incompatible. Switching between Jina models and a GenAI provider requires reindexing. Embeddings from different backends are incompatible.
@@ -242,8 +245,8 @@ Triggers are best configured through the Frigate UI.
1. Navigate to <NavPath path="Settings > Enrichments > Triggers" /> and select a camera from the dropdown menu. 1. Navigate to <NavPath path="Settings > Enrichments > Triggers" /> and select a camera from the dropdown menu.
2. Click **Add Trigger** to create a new trigger or use the pencil icon to edit an existing one. 2. Click **Add Trigger** to create a new trigger or use the pencil icon to edit an existing one.
3. In the **Create Trigger** wizard: 3. In the **Create Trigger** wizard:
- Enter a **Name** for the trigger (e.g., "Red Car Alert"). Frigate derives the trigger's - Enter a **Name** for the trigger (e.g., "Red Car Alert").
internal **ID** from this name, which can be revealed and edited with the show/hide toggle. - Enter a descriptive **Friendly Name** for the trigger (e.g., "Red car on the driveway camera").
- Select the **Type** (`Thumbnail` or `Description`). - Select the **Type** (`Thumbnail` or `Description`).
- For `Thumbnail`, select an image to trigger this action when a similar thumbnail image is detected, based on the threshold. - For `Thumbnail`, select an image to trigger this action when a similar thumbnail image is detected, based on the threshold.
- For `Description`, enter text to trigger this action when a similar tracked object description is detected. - For `Description`, enter text to trigger this action when a similar tracked object description is detected.
+4 -4
View File
@@ -28,7 +28,7 @@ During testing, enable the Zones option for the [Debug view](/usage/live#the-sin
1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera. 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
2. Under the **Zones** section, click the plus icon to add a new zone. 2. Under the **Zones** section, click the plus icon to add a new zone.
3. Click on the camera's latest image to create the points for the zone boundary. Click the first point again to close the polygon. 3. Click on the camera's latest image to create the points for the zone boundary. Click the first point again to close the polygon.
4. Configure zone options such as **Name**, **Objects**, **Loitering Time**, and **Inertia** in the zone editor. 4. Configure zone options such as **Friendly name**, **Objects**, **Loitering time**, and **Inertia** in the zone editor.
5. Press **Save** when finished. 5. Press **Save** when finished.
</TabItem> </TabItem>
@@ -200,7 +200,7 @@ When using loitering zones, a review item will behave in the following way:
1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera. 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
2. Edit or create the zone (e.g., `sidewalk`). 2. Edit or create the zone (e.g., `sidewalk`).
- Set **Loitering Time** to the desired number of seconds (e.g., `4`) - Set **Loitering time** to the desired number of seconds (e.g., `4`)
- Under **Objects**, add the relevant object types (e.g., `person`) - Under **Objects**, add the relevant object types (e.g., `person`)
</TabItem> </TabItem>
@@ -291,7 +291,7 @@ Accurate real-world distance measurements are required to estimate speeds. These
1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera. 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
2. Create or edit a zone with exactly 4 points aligned to the ground plane. 2. Create or edit a zone with exactly 4 points aligned to the ground plane.
3. In the zone editor, enable **Speed Estimation** and enter the real-world **Line A distance**, **Line B distance**, **Line C distance**, and **Line D distance** between each pair of consecutive points. 3. In the zone editor, enter the real-world **Distances** between each pair of consecutive points.
- For example, if the distance between the first and second points is 10 meters, between the second and third is 12 meters, etc. - For example, if the distance between the first and second points is 10 meters, between the second and third is 12 meters, etc.
4. Distances are measured in meters (metric) or feet (imperial), depending on the **Unit system** setting. 4. Distances are measured in meters (metric) or feet (imperial), depending on the **Unit system** setting.
@@ -358,7 +358,7 @@ Zones can be configured with a minimum speed requirement, meaning an object must
1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera. 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
2. Edit or create the zone with distances configured. 2. Edit or create the zone with distances configured.
- Set **Speed Threshold** to the desired minimum speed (e.g., `20`) - Set **Speed threshold** to the desired minimum speed (e.g., `20`)
- The unit is kph or mph, depending on the **Unit system** setting - The unit is kph or mph, depending on the **Unit system** setting
</TabItem> </TabItem>
+2 -2
View File
@@ -54,7 +54,7 @@ An object filter mask drops any [bounding box](#bounding-box) whose bottom cente
## Min Score ## Min Score
The lowest score a detected object can have to be kept during tracking. Anything scoring below the minimum is assumed to be a [false positive](#false-positive) and discarded. Set with `min_score` in the config, shown as **Minimum confidence** in the settings UI. The lowest score a detected object can have to be kept during tracking. Anything scoring below the minimum is assumed to be a [false positive](#false-positive) and discarded.
## Model ## Model
@@ -86,7 +86,7 @@ A more specific identity assigned to a [tracked object](#tracked-object-event-in
## Threshold ## Threshold
The median score an object must reach to be considered a true positive. Set with `threshold` in the config, shown as **Confidence threshold** in the settings UI. The median score an object must reach to be considered a true positive.
## Top Score ## Top Score
+2 -72
View File
@@ -65,26 +65,14 @@ Frigate supports multiple different detectors that work on different types of ha
- [Supports many model architectures](../../configuration/object_detectors#memryx-mx3) - [Supports many model architectures](../../configuration/object_detectors#memryx-mx3)
- Runs best with tiny, small, or medium-size models - Runs best with tiny, small, or medium-size models
- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, allowing for a wide range of compatibility with devices.
- [Supports YOLO model architectures](../../configuration/object_detectors#deepx-npu)
- Runs best with tiny or small size models
- Runs efficiently on low power hardware
**AMD** **AMD**
- [ROCm](#rocm---amd-gpu): ROCm can run on AMD Discrete GPUs to provide efficient object detection - [ROCm](#rocm---amd-gpu): ROCm can run on AMD Discrete GPUs to provide efficient object detection
- [Supports limited model architectures](../../configuration/object_detectors#amdrocm-gpu-detector) - [Supports limited model architectures](../../configuration/object_detectors#amdrocm-gpu-detector)
- Runs best on discrete AMD GPUs - Runs best on discrete AMD GPUs
- <CommunityBadge /> [XDNA2 (Ryzen AI)](#amd-xdna2): AMD XDNA2 NPU (sub-watt power AI/ML processor separate to the GPU) inside Strix and other "AI" branded AMD platforms
- Has only been tested with YOLOv9, in theory other graphs may be compiled too.
- Runs via ZMQ proxy which adds some latency, only recommended for local connection
**Apple Silicon** **Apple Silicon**
- [ONNX via lighter](#apple-silicon): The ONNX detector runs on the Neural Engine of M1 and newer Macs when Frigate runs in the lighter container runtime
- [Supports the same model architectures as the ONNX detector](../../configuration/object_detectors#apple-neural-engine-lighter)
- Runs inside the Frigate container, with no separate detector process to set up
- The recommended way to run Frigate on a Mac
- [Apple Silicon](#apple-silicon): Apple Silicon is usable on all M1 and newer Apple Silicon devices to provide efficient and fast object detection - [Apple Silicon](#apple-silicon): Apple Silicon is usable on all M1 and newer Apple Silicon devices to provide efficient and fast object detection
- [Supports primarily ssdlite and mobilenet model architectures](../../configuration/object_detectors#apple-silicon-detector) - [Supports primarily ssdlite and mobilenet model architectures](../../configuration/object_detectors#apple-silicon-detector)
- Runs well with any size models including large - Runs well with any size models including large
@@ -189,7 +177,7 @@ Frigate is able to utilize an Nvidia GPU which supports the 12.x series of CUDA
#### Minimum Hardware Support #### Minimum Hardware Support
12.x series of CUDA libraries are used which have minor version compatibility. The minimum driver version on the host system must be `>=545`. The oldest supported GPU generation for object detection is Pascal (Compute Capability 6.0 or greater). Older generations such as Maxwell are not supported but can still be used by following a [community workaround](https://github.com/blakeblackshear/frigate/discussions/20088). For other GPUs, check the NVIDIA GPU Compute Capability table linked below. 12.x series of CUDA libraries are used which have minor version compatibility. The minimum driver version on the host system must be `>=545`. Also the GPU must support a Compute Capability of `5.0` or greater. This generally correlates to a Maxwell-era GPU or newer, check the NVIDIA GPU Compute Capability table linked below.
Make sure your host system has the [nvidia-container-runtime](https://docs.docker.com/config/containers/resource_constraints/#access-an-nvidia-gpu) installed to pass through the GPU to the container and the host system has a compatible driver installed for your GPU. Make sure your host system has the [nvidia-container-runtime](https://docs.docker.com/config/containers/resource_constraints/#access-an-nvidia-gpu) installed to pass through the GPU to the container and the host system has a compatible driver installed for your GPU.
@@ -218,13 +206,7 @@ Inference is done with the `onnx` detector type. Speeds will vary greatly depend
### Apple Silicon ### Apple Silicon
Frigate on a Mac is best run in the [lighter](https://github.com/fieldwork-ai/lighter) container runtime, where the [ONNX detector](../configuration/object_detectors.md#apple-neural-engine-lighter) runs on the Neural Engine of M1 and newer Macs from inside the Frigate container. There is no separate detector process to install or keep running, and the same container can decode video on the Mac's media engine. With the [Apple Silicon](../configuration/object_detectors.md#apple-silicon-detector) detector Frigate can take advantage of the NPU in M1 and newer Apple Silicon.
| Name | YOLOv9 Inference Time | YOLO-NAS Inference Time | RF-DETR Inference Time |
| ---- | -------------------------------------- | ----------------------- | ---------------------- |
| M1 | t-320: 3.3 ms s-320: 7 ms s-640: 13 ms | 320: 6.6 ms | Nano-320: 38 ms |
Alternatively, with the [Apple Silicon](../configuration/object_detectors.md#apple-silicon-detector) detector Frigate can take advantage of the NPU in M1 and newer Apple Silicon.
:::warning :::warning
@@ -275,32 +257,6 @@ The MX3 is a pipelined architecture, where the maximum frames per second support
Inference speeds may vary depending on the host platform. The above data was measured on an **Intel 13700 CPU**. Platforms like Raspberry Pi, Orange Pi, and other ARM-based SBCs have different levels of processing capability, which may limit total FPS. Inference speeds may vary depending on the host platform. The above data was measured on an **Intel 13700 CPU**. Platforms like Raspberry Pi, Orange Pi, and other ARM-based SBCs have different levels of processing capability, which may limit total FPS.
### DEEPX NPU
Frigate supports the DEEPX NPU in both of its form factors: the **DX-M1** M.2 module, which works on x86 (Intel/AMD) and ARM-based SBCs such as the Raspberry Pi 5, and the **DX-M1M** on the [Sixfab AI HAT+](https://docs.sixfab.com/docs/ai-hat-plus-raspberry-pi-5-quickstart) for the Raspberry Pi 5. Both use the same driver and runtime, so the configuration is identical for either one. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
The DEEPX driver and runtime run on the Docker host rather than inside the Frigate container and must be installed before the NPU can be used. See the [installation docs](installation.md#deepx-npu) for the setup steps and [the detector docs](/configuration/object_detectors#deepx-npu) for the configuration.
Frigate does not bundle a model for this detector. Models use DEEPX's `.dxnn` format, and pre-compiled YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Prefer a model with a `_ppu` suffix whenever one is available for the architecture you want: these run part of the post-processing on the NPU itself and are considerably faster, roughly 2.5x for the same architecture and input size. **YOLOX-S with PPU is the recommended starting point.**
Inference times for a few recommended models, measured through Frigate's own stats on a DX-M1:
| Model | Input Size | DX-M1 Inference Time |
| ----------------- | ---------- | -------------------- |
| YOLOX-S (PPU) | 640 | ~ 13 ms |
| YOLOv9-t (PPU) | 640 | ~ 18 ms |
| YOLOv4 (PPU) | 512 | ~ 20 ms |
| YOLOX-S | 640 | ~ 34 ms |
| YOLOv9-s | 640 | ~ 39 ms |
Other ModelZoo YOLO variants are also supported but have not been measured. Inference speeds vary with the host platform, so a slower host such as a Raspberry Pi 5 will report higher times than those above.
:::note
A few ModelZoo models can not be used with Frigate: SSD models (they are trained on Pascal VOC, so their labels do not match Frigate's), DAMO-YOLO models, face and pose models, and the PPU builds of YOLOv7.
:::
### Nvidia Jetson ### Nvidia Jetson
Jetson devices are supported via the TensorRT or ONNX detectors when running Jetpack 6. It will [make use of the Jetson's hardware media engine](/configuration/hardware_acceleration_video#nvidia-jetson) when configured with the [appropriate presets](/configuration/ffmpeg_presets#hwaccel-presets), and will make use of the Jetson's GPU and DLA for object detection when configured with the [TensorRT detector](/configuration/object_detectors#nvidia-tensorrt-detector). Jetson devices are supported via the TensorRT or ONNX detectors when running Jetpack 6. It will [make use of the Jetson's hardware media engine](/configuration/hardware_acceleration_video#nvidia-jetson) when configured with the [appropriate presets](/configuration/ffmpeg_presets#hwaccel-presets), and will make use of the Jetson's GPU and DLA for object detection when configured with the [TensorRT detector](/configuration/object_detectors#nvidia-tensorrt-detector).
@@ -341,32 +297,6 @@ The inference time of a rk3588 with all 3 cores enabled is typically 25-30 ms fo
| ---------------- | ----------------------------------- | | ---------------- | ----------------------------------- |
| yolov9-tiny | ~ 4 ms | | yolov9-tiny | ~ 4 ms |
### AMD Ryzen AI / XDNA2
Frigate supports AMD XDNA2 NPUs through the community-maintained
frigate-xdna ZMQ sidecar. It works with stock Frigate and supports
Frigate+ models or compatible local YOLO ONNX models. Models are compiled
once on the target system and cached for subsequent use.
Currently qualified on **Ryzen AI Max 300 / Strix Halo**. Other XDNA2
devices are not yet qualified; XDNA1 is unsupported.
Measured YOLOv9 detector latency on Strix Halo:
| Model | 320 | 640 |
| ----- | ---: | ---: |
| YOLOv9-T | ~7.4 ms | unsupported |
| YOLOv9-S | ~9.0 ms | ~20.0 ms |
| YOLOv9-M | ~13.1 ms | ~34.4 ms |
| YOLOv9-C | ~14.1 ms | ~35.2 ms |
| YOLOv9-E | ~69.4 ms | ~224.8 ms |
**YOLOv9-C at 320 is the recommended quality/performance balance.**
C at 640 is also usable where the lower throughput is acceptable.
Setup, model preparation, and compatibility details are available
[in the frigate-xdna documentation](https://github.com/mitchins/frigate-xdna).
## What does Frigate use the CPU for and what does it use a detector for? (ELI5 Version) ## What does Frigate use the CPU for and what does it use a detector for? (ELI5 Version)
This is taken from a [user question on reddit](https://www.reddit.com/r/homeassistant/comments/q8mgau/comment/hgqbxh5/?utm_source=share&utm_medium=web2x&context=3). Modified slightly for clarity. This is taken from a [user question on reddit](https://www.reddit.com/r/homeassistant/comments/q8mgau/comment/hgqbxh5/?utm_source=share&utm_medium=web2x&context=3). Modified slightly for clarity.
-93
View File
@@ -381,99 +381,6 @@ If you can't use Docker Compose, you can run the container with something simila
Finally, configure [hardware object detection](/configuration/object_detectors#memryx-mx3) to complete the setup. Finally, configure [hardware object detection](/configuration/object_detectors#memryx-mx3) to complete the setup.
### DEEPX NPU
The DEEPX NPU is available in two form factors, and Frigate supports both:
- **DX-M1** in the M.2 2280 form factor (like an NVMe SSD), for x86 (Intel/AMD) PCs, the Raspberry Pi 5, and other ARM SBCs with an exposed PCIe M.2 slot.
- **DX-M1M** on the [Sixfab AI HAT+](https://docs.sixfab.com/docs/ai-hat-plus-raspberry-pi-5-quickstart), a HAT+ board that connects to the Raspberry Pi 5 over PCIe Gen 3 x1.
Both present the NPU through the same PCIe driver and DX-RT runtime, so the setup below and the detector configuration are identical for either one. Nothing needs to change when moving between them.
DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
#### Versions
A DEEPX install has several separately versioned pieces, and they all have to agree. The driver, the runtime, and the daemon live on the Docker host; Frigate itself carries only the Python bindings, which it downloads on first start:
| Component | Version | Installed on | Installed by |
| -------------- | -------- | ------------ | ------------------------- |
| Kernel driver | `v2.6.0` | Host | `user_installation.sh` |
| DX-RT runtime | `v3.4.0` | Host | `user_installation.sh` |
| NPU firmware | `v2.7.4` | The module | Flashed from the host |
| DX-RT bindings | `v3.4.0` | Frigate | Downloaded at first start |
:::warning
A version mismatch does not produce a startup error. It typically shows up as inference requests that are accepted but never return a result, so detections simply stop appearing while Frigate looks healthy. If that happens after a Frigate upgrade, check every version in the table before anything else.
:::
The installation script installs the DX-RT runtime on the host and enables `dxrt.service`, so the daemon starts at boot and any other program on the host can share the NPU with Frigate. Check the firmware version with `dxrt-cli --status` and update the module if it does not match the table above.
#### Installation
The DEEPX kernel driver must be installed on the host rather than in the container, because containers share the host kernel and cannot load kernel modules. Installing it creates the `/dev/dxrt*` device nodes that are passed through to Frigate. The same script installs the DX-RT runtime and enables `dxrt.service`, the daemon that owns the NPU and hands work to it on behalf of Frigate and anything else on the host.
1. Copy or download [this script](https://github.com/blakeblackshear/frigate/blob/dev/docker/deepx/user_installation.sh).
2. Ensure it has execution permissions with `sudo chmod +x user_installation.sh`
3. Run the script with `./user_installation.sh`
4. **Restart your computer** to complete driver installation.
Confirm the NPU is visible before continuing:
```bash
ls /dev/dxrt*
```
Then confirm the daemon is running and listening in `/run/dxrt`:
```bash
systemctl is-active dxrt.service
ls /run/dxrt/
```
#### Setup
To set up Frigate, follow the default installation instructions, for example: `ghcr.io/blakeblackshear/frigate:stable`
#### Docker configuration
Frigate needs the NPU device node and the directory holding the daemon's socket:
```yaml
services:
frigate:
devices:
- /dev/dxrt0:/dev/dxrt0
volumes:
- /run/dxrt:/run/dxrt
```
If you can't use Docker Compose, add `--device /dev/dxrt0:/dev/dxrt0 -v /run/dxrt:/run/dxrt` to your `docker run` command.
Add one `--device` per NPU, contiguously from `/dev/dxrt0`, since the client stops enumerating at the first gap.
The installation script configures `dxrt.service` to place its socket in `/run/dxrt` through a systemd drop-in. Mounting the directory rather than the socket file means the container sees the new socket after `dxrt.service` is restarted, rather than holding on to a deleted one.
`dxrtd` listens on an abstract socket as well, but that one does not cross into a container, so Frigate names the filesystem socket through `DXRT_DYNAMIC_IPC_ENDPOINT` on your behalf. Set that variable on the container yourself only if the daemon listens somewhere else, which means you also set it for `dxrtd` through its own systemd drop-in. The script writes `/etc/systemd/system/dxrt.service.d/frigate.conf` for exactly that, and has `dxrt.service` link the socket to `/tmp/dxrt_dynamic_ipc.sock` when it starts, so the host's own `dxrt-cli` and `dxtop` keep finding it at the default path they fall back to.
:::note
The DX-RT client exits when `dxrt.service` stops, so restart the Frigate container after restarting `dxrt.service`.
:::
The device node is needed as well as the socket, because the client opens the NPU directly even though the daemon arbitrates access. Without it, inference fails with `Device not found`.
`/dev/shm` does not need sharing.
The DX-RT python bindings are not shipped in the Frigate image. Frigate downloads them on first start when a DEEPX detector is configured, and caches them under `/config`.
#### Configuration
Finally, configure [hardware object detection](/configuration/object_detectors#deepx-npu) to complete the setup.
### Rockchip platform ### Rockchip platform
Make sure that you use a linux distribution that comes with the rockchip BSP kernel 5.10 or 6.1 and necessary drivers (especially rkvdec2 and rknpu). To check, enter the following commands: Make sure that you use a linux distribution that comes with the rockchip BSP kernel 5.10 or 6.1 and necessary drivers (especially rkvdec2 and rknpu). To check, enter the following commands:
+2 -19
View File
@@ -11,12 +11,6 @@ MQTT requires a network connection to your broker. This is typically local, but
::: :::
:::note
Wherever a topic below includes a camera, mask, or zone name, use its `ID` from the config, not its `friendly_name`. For example, a camera with `friendly_name: "Back Yard"` and ID `back_yard` publishes to `frigate/back_yard/...`, not `frigate/Back Yard/...`.
:::
## General Frigate Topics ## General Frigate Topics
### `frigate/available` ### `frigate/available`
@@ -218,7 +212,6 @@ An `update` with the same ID will be published when:
- The severity changes from `detection` to `alert` - The severity changes from `detection` to `alert`
- Additional objects are detected - Additional objects are detected
- An object is recognized via face, lpr, etc. - An object is recognized via face, lpr, etc.
- A [state classification](/configuration/custom_classification/state_classification#review-items) model changes state
When the review activity has ended a final `end` message is published. When the review activity has ended a final `end` message is published.
@@ -242,8 +235,7 @@ When the review activity has ended a final `end` message is published.
"objects": ["person", "car"], "objects": ["person", "car"],
"sub_labels": [], "sub_labels": [],
"zones": [], "zones": [],
"audio": [], "audio": []
"classification_state_changes": []
} }
}, },
"after": { "after": {
@@ -262,16 +254,7 @@ When the review activity has ended a final `end` message is published.
"objects": ["person", "car"], "objects": ["person", "car"],
"sub_labels": ["Bob"], "sub_labels": ["Bob"],
"zones": ["front_yard"], "zones": ["front_yard"],
"audio": [], "audio": []
"classification_state_changes": [
// verified changes of state classification models on this camera
{
"model": "front_gate",
"from": "closed",
"to": "open",
"timestamp": 1718987131.52
}
]
} }
} }
} }
@@ -27,10 +27,6 @@ The [Advanced Camera Card](https://card.camera/#/README) is a Home Assistant das
It supports automatically setting the sub labels in Frigate for person objects that are detected and recognized. It supports automatically setting the sub labels in Frigate for person objects that are detected and recognized.
This is a fork (with fixed errors and new features) of [original Double Take](https://github.com/jakowenko/double-take) project which, unfortunately, isn't being maintained by author. This is a fork (with fixed errors and new features) of [original Double Take](https://github.com/jakowenko/double-take) project which, unfortunately, isn't being maintained by author.
## [frigate-abr](https://github.com/007hacky007/frigate-abr)
[frigate-abr](https://github.com/007hacky007/frigate-abr) is a drop-in Docker image of Frigate that adds adaptive bitrate (ABR) playback for recordings: a sidecar transcodes footage to lower quality tiers on demand, for reviewing over slow remote connections. Segments are transcoded when played and cached, so no additional stream is recorded. Frigate itself is not modified.
## [Frigate Notify](https://github.com/0x2142/frigate-notify) ## [Frigate Notify](https://github.com/0x2142/frigate-notify)
[Frigate Notify](https://github.com/0x2142/frigate-notify) is a simple app designed to send notifications from Frigate to your favorite platforms. Intended to be used with standalone Frigate installations - Home Assistant not required, MQTT is optional but recommended. [Frigate Notify](https://github.com/0x2142/frigate-notify) is a simple app designed to send notifications from Frigate to your favorite platforms. Intended to be used with standalone Frigate installations - Home Assistant not required, MQTT is optional but recommended.
+1 -7
View File
@@ -21,13 +21,7 @@ Yes. Models and metadata are stored in the `model_cache` directory within the co
### Can I keep using my Frigate+ models even if I do not renew my subscription? ### Can I keep using my Frigate+ models even if I do not renew my subscription?
Yes. Subscriptions to Frigate+ provide access to the infrastructure used to train the models. Models you train during an active subscription remain licensed for your continued use even after your subscription ends — models already in your model cache will keep working indefinitely. An active subscription is required to train new models and download new versions. Yes. Subscriptions to Frigate+ provide access to the infrastructure used to train the models. Models trained with your subscription are yours to keep and use forever. However, do note that the terms and conditions prohibit you from sharing, reselling, or creating derivative products from the models.
### Can I use Frigate+ models commercially?
A standard subscription covers use on camera systems you own or operate, including for your business. A shop, restaurant, warehouse, or office running Frigate+ at its own locations (including multiple locations) is exactly the kind of use the subscription is for.
What the standard subscription does not cover is using Frigate+ models to provide a product or service to others. If you're deploying models at your customers' sites, bundling them with hardware you sell, or running them as part of a hosted or managed service, even if your customers never receive the model files themselves, you'll need a commercial license.
Note that professional installers are fine under standard subscriptions when each customer holds their own Frigate+ subscription. The commercial license is for cases where your license powers your customers' sites.
### Why can't I submit images to Frigate+? ### Why can't I submit images to Frigate+?
+13 -13
View File
@@ -63,20 +63,20 @@ Frigate+ models generally have much higher scores than the default model provide
<ConfigTabs> <ConfigTabs>
<TabItem value="ui"> <TabItem value="ui">
Navigate to <NavPath path="Settings > Global configuration > Objects" />. Under **Object filters**, set **Minimum confidence** and **Confidence threshold** for each object type, then click **Save**. Navigate to <NavPath path="Settings > Global configuration > Objects" />. Under **Object filters**, set **Min Score** and **Threshold** for each object type, then click **Save**.
| Object | Minimum confidence | Confidence threshold | | Object | Min Score | Threshold |
| ----------------- | ------------------ | -------------------- | | ----------------- | --------- | --------- |
| **dog** | .7 | .9 | | **dog** | .7 | .9 |
| **cat** | .65 | .8 | | **cat** | .65 | .8 |
| **face** | .7 | | | **face** | .7 | |
| **package** | .65 | .9 | | **package** | .65 | .9 |
| **license_plate** | .6 | | | **license_plate** | .6 | |
| **amazon** | .75 | | | **amazon** | .75 | |
| **ups** | .75 | | | **ups** | .75 | |
| **fedex** | .75 | | | **fedex** | .75 | |
| **person** | .65 | .85 | | **person** | .65 | .85 |
| **car** | .65 | .85 | | **car** | .65 | .85 |
</TabItem> </TabItem>
<TabItem value="yaml"> <TabItem value="yaml">
+6 -9
View File
@@ -65,11 +65,11 @@ Some users may find that Frigate+ models result in more false positives initiall
Frigate+ models support a more relevant set of objects for security cameras. The labels for annotation in Frigate+ are configurable by editing the camera in the Cameras section of Frigate+. Currently, the following objects are supported: Frigate+ models support a more relevant set of objects for security cameras. The labels for annotation in Frigate+ are configurable by editing the camera in the Cameras section of Frigate+. Currently, the following objects are supported:
- **People**: `person`, `face`, `baby` - **People**: `person`, `face`
- **Vehicles**: `car`, `motorcycle`, `bicycle`, `boat`, `school_bus`, `garbage truck`, `license_plate` - **Vehicles**: `car`, `motorcycle`, `bicycle`, `boat`, `school_bus`, `license_plate`
- **Delivery Logos**: `amazon`, `usps`, `ups`, `fedex`, `dhl`, `an_post`, `purolator`, `postnl`, `nzpost`, `postnord`, `gls`, `dpd`, `canada_post`, `royal_mail` - **Delivery Logos**: `amazon`, `usps`, `ups`, `fedex`, `dhl`, `an_post`, `purolator`, `postnl`, `nzpost`, `postnord`, `gls`, `dpd`, `canada_post`, `royal_mail`
- **Animals**: `dog`, `cat`, `deer`, `horse`, `bird`, `raccoon`, `fox`, `bear`, `cow`, `squirrel`, `goat`, `rabbit`, `skunk`, `kangaroo`, `possum`, `rodent` - **Animals**: `dog`, `cat`, `deer`, `horse`, `bird`, `raccoon`, `fox`, `bear`, `cow`, `squirrel`, `goat`, `rabbit`, `skunk`, `kangaroo`
- **Other**: `package`, `waste_bin`, `bbq_grill`, `robot_lawnmower`, `umbrella`, `baby_stroller` - **Other**: `package`, `waste_bin`, `bbq_grill`, `robot_lawnmower`, `umbrella`
Other object types available in the default Frigate model are not available. Additional object types will be added in future releases. Other object types available in the default Frigate model are not available. Additional object types will be added in future releases.
@@ -77,12 +77,9 @@ Other object types available in the default Frigate model are not available. Add
Candidate labels are also available for annotation. These labels don't have enough data to be included in the model yet, but using them will help add support sooner. You can enable these labels by editing the camera settings. Candidate labels are also available for annotation. These labels don't have enough data to be included in the model yet, but using them will help add support sooner. You can enable these labels by editing the camera settings.
Where possible, these labels are mapped to existing labels during training. For example, any `duck` labels are mapped to `bird` until support for new labels is added. Where possible, these labels are mapped to existing labels during training. For example, any `baby` labels are mapped to `person` until support for new labels is added.
- **Vehicles**: `tractor`, `golf_cart`, `bus`, `airplane`, `helicopter`, `rickshaw`, `scooter` The candidate labels are: `baby`, `bpost`, `badger`, `possum`, `rodent`, `chicken`, `groundhog`, `boar`, `hedgehog`, `tractor`, `golf cart`, `garbage truck`, `bus`, `sports ball`, `la_poste`, `lawnmower`, `heron`, `rickshaw`, `wombat`, `auspost`, `aramex`, `bobcat`, `mustelid`, `transoflex`, `airplane`, `drone`, `mountain_lion`, `crocodile`, `turkey`, `baby_stroller`, `monkey`, `coyote`, `porcupine`, `parcelforce`, `sheep`, `snake`, `helicopter`, `lizard`, `duck`, `hermes`, `cargus`, `fan_courier`, `sameday`
- **Delivery Logos**: `bpost`, `auspost`, `aramex`, `transoflex`, `parcelforce`, `hermes`, `cargus`, `fan_courier`, `sameday`, `la_poste`
- **Animals**: `badger`, `chicken`, `duck`, `turkey`, `groundhog`, `boar`, `hedgehog`, `wombat`, `bobcat`, `mustelid`, `mountain_lion`, `crocodile`, `monkey`, `coyote`, `porcupine`, `sheep`, `snake`, `lizard`, `heron`, `elk`, `moose`, `pig`, `donkey`, `civet`
- **Other**: `sports_ball`, `drone`, `lawnmower`
Candidate labels are not available for automatic suggestions. Candidate labels are not available for automatic suggestions.
+1 -9
View File
@@ -53,14 +53,6 @@ go2rtc:
Point the camera's inputs at the restream as described in the [restream docs](/configuration/restream.md), and swap `detect -> width` and `detect -> height` to match the rotated resolution. Point the camera's inputs at the restream as described in the [restream docs](/configuration/restream.md), and swap `detect -> width` and `detect -> height` to match the rotated resolution.
### Can I add a privacy mask to hide part of my camera's view?
Frigate does not have privacy masks. [Motion masks and object filter masks](../configuration/masks.md) only affect detection, they don't hide anything in live view, recordings, or snapshots.
Privacy masks are best configured in the camera's firmware settings so the area is blacked out before the video ever leaves the camera and no extra processing is needed. Check there first.
If your camera does not support privacy masks, there is no efficient alternative. Frigate copies the camera's video into recordings and live view without decoding it, so part of the image can't be hidden without transcoding (re-encoding) the stream. This can be done with a custom ffmpeg filter in go2rtc, but it is not recommended. Every masked camera needs a continuous re-encode, which significantly increases CPU usage, especially for high resolution streams.
### My mjpeg stream or snapshots look green and crazy ### My mjpeg stream or snapshots look green and crazy
This almost always means that the width/height defined for your camera are not correct. Double check the resolution with VLC or another player. Also make sure you don't have the width and height values backwards. This almost always means that the width/height defined for your camera are not correct. Double check the resolution with VLC or another player. Also make sure you don't have the width and height values backwards.
@@ -192,6 +184,6 @@ Filters and masks only hide the incorrect result - they don't teach Frigate what
### Where do I see problems Frigate has detected? ### Where do I see problems Frigate has detected?
Open System > Health. The Notices list keeps a record of problems Frigate has found. Acknowledge an entry to hide it until the problem happens again, or mute it to hide it for good. Hidden entries stay listed under Show hidden in the filter. Ongoing conditions, such as an offline camera or recordings deleted before their retention period, appear in the status bar for admins until they clear, and the status bar links to the Notices list while it has entries showing. On mobile, tap the warning icon in the bottom navigation bar to see them. Open System > Health. The Notices list keeps a record of problems Frigate has found, and you can dismiss any entry to acknowledge it. Ongoing conditions, such as an offline camera or recordings deleted before their retention period, appear in the status bar for admins until they clear, and the status bar links to the Notices list while it has undismissed entries. On mobile, tap the warning icon in the bottom navigation bar to see them.
The Hardware section below the notices shows whether the detection hardware, hardware acceleration, and enrichment devices in your config were found and are being used, so a GPU that silently fell back to the CPU shows up as a warning. Run stream checks to probe every camera's streams for the same problems the camera wizard reports. The Hardware section below the notices shows whether the detection hardware, hardware acceleration, and enrichment devices in your config were found and are being used, so a GPU that silently fell back to the CPU shows up as a warning. Run stream checks to probe every camera's streams for the same problems the camera wizard reports.
+10 -2
View File
@@ -397,11 +397,19 @@ dmesg | grep -i -E "gpu|drm|reset|hang"
Messages like `trying reset from guc_exec_queue_timedout_job` or similar GPU reset/hang messages indicate a driver or hardware issue. Ensure your kernel and GPU drivers (especially Intel) are up to date. Messages like `trying reset from guc_exec_queue_timedout_job` or similar GPU reset/hang messages indicate a driver or hardware issue. Ensure your kernel and GPU drivers (especially Intel) are up to date.
#### Step 6: Verify go2rtc stream configuration #### Step 6: Verify hardware acceleration configuration
An incorrect `hwaccel_args` preset can cause ffmpeg to fail silently or consume excessive CPU, starving the detector of resources.
- After upgrading Frigate, verify your preset matches your hardware (e.g., `preset-intel-qsv-h264` instead of the deprecated `preset-vaapi`).
- For h265 cameras, use the corresponding h265 preset (e.g., `preset-intel-qsv-h265`).
- Note that `hwaccel_args` are only relevant for the detect stream. Frigate does not decode the record stream.
#### Step 7: Verify go2rtc stream configuration
Ensure that the ffmpeg source names in your go2rtc configuration match the correct camera stream. A misconfigured stream name (e.g., copying a config from one camera to another without updating the stream reference) will cause the wrong stream to be used or the stream to fail entirely. Ensure that the ffmpeg source names in your go2rtc configuration match the correct camera stream. A misconfigured stream name (e.g., copying a config from one camera to another without updating the stream reference) will cause the wrong stream to be used or the stream to fail entirely.
#### Step 7: Check system resources #### Step 8: Check system resources
If none of the above apply, the issue may be a general resource constraint. Monitor the following on your host: If none of the above apply, the issue may be a general resource constraint. Monitor the following on your host:
+22 -30
View File
@@ -3,9 +3,6 @@ import * as path from "node:path";
import type { Config, PluginConfig } from "@docusaurus/types"; import type { Config, PluginConfig } from "@docusaurus/types";
import type * as OpenApiPlugin from "docusaurus-plugin-openapi-docs"; import type * as OpenApiPlugin from "docusaurus-plugin-openapi-docs";
// Bump when a new stable release ships
const STABLE_VERSION = "0.18";
const config: Config = { const config: Config = {
title: "Frigate", title: "Frigate",
tagline: "NVR With Realtime Object Detection for IP Cameras", tagline: "NVR With Realtime Object Detection for IP Cameras",
@@ -26,17 +23,17 @@ const config: Config = {
mermaid: true, mermaid: true,
}, },
i18n: { i18n: {
defaultLocale: "en", defaultLocale: 'en',
locales: ["en"], locales: ['en'],
localeConfigs: { localeConfigs: {
en: { en: {
label: "English", label: 'English',
}, }
}, },
}, },
themeConfig: { themeConfig: {
announcementBar: { announcementBar: {
id: "frigate_plus", id: 'frigate_plus',
content: ` content: `
<span style="margin-right: 8px; display: inline-block; animation: pulse 2s infinite;">🚀</span> <span style="margin-right: 8px; display: inline-block; animation: pulse 2s infinite;">🚀</span>
Get more relevant and accurate detections with Frigate+ models. Get more relevant and accurate detections with Frigate+ models.
@@ -48,8 +45,8 @@ const config: Config = {
50% { transform: scale(1.1); } 50% { transform: scale(1.1); }
} }
</style>`, </style>`,
backgroundColor: "#005f73", backgroundColor: '#005f73',
textColor: "#e0fbfc", textColor: '#e0fbfc',
isCloseable: false, isCloseable: false,
}, },
docs: { docs: {
@@ -86,15 +83,15 @@ const config: Config = {
}, },
}, },
prism: { prism: {
magicComments: [ magicComments:[
{ {
className: "theme-code-block-highlighted-line", className: 'theme-code-block-highlighted-line',
line: "highlight-next-line", line: 'highlight-next-line',
block: { start: "highlight-start", end: "highlight-end" }, block: {start: 'highlight-start', end: 'highlight-end'},
}, },
{ {
className: "code-block-error-line", className: 'code-block-error-line',
line: "highlight-error-line", line: 'highlight-error-line',
}, },
], ],
additionalLanguages: ["bash", "json"], additionalLanguages: ["bash", "json"],
@@ -134,11 +131,6 @@ const config: Config = {
srcDark: "img/branding/logo-dark.svg", srcDark: "img/branding/logo-dark.svg",
}, },
items: [ items: [
{
href: "https://github.com/blakeblackshear/frigate/releases",
label: `${STABLE_VERSION}`,
position: "left",
},
{ {
to: "/", to: "/",
activeBasePath: "docs", activeBasePath: "docs",
@@ -156,19 +148,19 @@ const config: Config = {
position: "right", position: "right",
}, },
{ {
type: "localeDropdown", type: 'localeDropdown',
position: "right", position: 'right',
dropdownItemsAfter: [ dropdownItemsAfter: [
{ {
label: "简体中文(社区翻译)", label: '简体中文(社区翻译)',
href: "https://docs.frigate-cn.video", href: 'https://docs.frigate-cn.video',
}, }
], ]
}, },
{ {
href: "https://github.com/blakeblackshear/frigate", href: 'https://github.com/blakeblackshear/frigate',
label: "GitHub", label: 'GitHub',
position: "right", position: 'right',
}, },
], ],
}, },
+25 -34
View File
@@ -7030,9 +7030,9 @@
} }
}, },
"node_modules/brace-expansion": { "node_modules/brace-expansion": {
"version": "1.1.21", "version": "1.1.18",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz",
"integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==", "integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"balanced-match": "^1.0.0", "balanced-match": "^1.0.0",
@@ -7698,15 +7698,14 @@
} }
}, },
"node_modules/compression": { "node_modules/compression": {
"version": "1.8.2", "version": "1.8.1",
"resolved": "https://registry.npmjs.org/compression/-/compression-1.8.2.tgz", "resolved": "https://registry.npmjs.org/compression/-/compression-1.8.1.tgz",
"integrity": "sha512-o8vI5RE5A6EVVOd9o41jKp41aJom+QTEO/Bx8MYNjexMo/Bv2WOjUfZr+aL0WnYSgymUy6zeguqLTsIhV0gMvQ==", "integrity": "sha512-9mAqGPHLakhCLeNyxPkK4xVo746zQ/czLH1Ky+vkitMnWfWZps8r0qXuwhwizagCRttsL4lfG4pIOvaWLpAP0w==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"bytes": "3.1.2", "bytes": "3.1.2",
"compressible": "~2.0.18", "compressible": "~2.0.18",
"debug": "2.6.9", "debug": "2.6.9",
"destroy": "1.2.0",
"negotiator": "~0.6.4", "negotiator": "~0.6.4",
"on-headers": "~1.1.0", "on-headers": "~1.1.0",
"safe-buffer": "5.2.1", "safe-buffer": "5.2.1",
@@ -7714,10 +7713,6 @@
}, },
"engines": { "engines": {
"node": ">= 0.8.0" "node": ">= 0.8.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/express"
} }
}, },
"node_modules/compression/node_modules/bytes": { "node_modules/compression/node_modules/bytes": {
@@ -9465,9 +9460,9 @@
} }
}, },
"node_modules/dompurify": { "node_modules/dompurify": {
"version": "3.4.16", "version": "3.4.13",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.16.tgz", "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.13.tgz",
"integrity": "sha512-sqo+pNp3qRhCIpbgRi1y8Tgk27Bo2Ry7w0dC1NBeNTdZChWjz9Xb/KOoZbRP/R6pQZ80Qw8YhXw13hWWBbMRnQ==", "integrity": "sha512-2vmYIoqjze2d+kakP8S/nS5shfsl587kzwEjcGlTdiksUVgFHnFCsLYDVj/JNqJVOQZGSYBTmuycv0PodwmnMQ==",
"license": "(MPL-2.0 OR Apache-2.0)", "license": "(MPL-2.0 OR Apache-2.0)",
"optionalDependencies": { "optionalDependencies": {
"@types/trusted-types": "^2.0.7" "@types/trusted-types": "^2.0.7"
@@ -10130,9 +10125,9 @@
"license": "MIT" "license": "MIT"
}, },
"node_modules/fast-uri": { "node_modules/fast-uri": {
"version": "3.1.8", "version": "3.1.7",
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz", "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.7.tgz",
"integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==", "integrity": "sha512-dOvZVzjdZdz7phd9v6jCbwxrBW3fK6n8Rc0CtdmM4bumzMnxywBYhuph6J819RRw/ku+rLbelwfMunktuzVVHg==",
"funding": [ "funding": [
{ {
"type": "github", "type": "github",
@@ -11380,15 +11375,15 @@
} }
}, },
"node_modules/image-size": { "node_modules/image-size": {
"version": "2.0.4", "version": "2.0.2",
"resolved": "https://registry.npmjs.org/image-size/-/image-size-2.0.4.tgz", "resolved": "https://registry.npmjs.org/image-size/-/image-size-2.0.2.tgz",
"integrity": "sha512-QRUkFFsRV/6fuESxb9Vkq+a0LkSrgKXuc2NEqfikiXxxN/G3tjWt5EVUlMaImRBZRZK/jRBEbYvpPYZL8t08Zw==", "integrity": "sha512-IRqXKlaXwgSMAMtpNzZa1ZAe8m+Sa1770Dhk8VkSsP9LS+iHD62Zd8FQKs8fbPiagBE7BzoFX23cxFnwshpV6w==",
"license": "MIT", "license": "MIT",
"bin": { "bin": {
"image-size": "bin/image-size.js" "image-size": "bin/image-size.js"
}, },
"engines": { "engines": {
"node": ">=18" "node": ">=16.x"
} }
}, },
"node_modules/immer": { "node_modules/immer": {
@@ -17453,9 +17448,9 @@
"license": "ISC" "license": "ISC"
}, },
"node_modules/proxy-addr": { "node_modules/proxy-addr": {
"version": "2.0.8", "version": "2.0.7",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz", "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
"integrity": "sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==", "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"forwarded": "0.2.0", "forwarded": "0.2.0",
@@ -17463,10 +17458,6 @@
}, },
"engines": { "engines": {
"node": ">= 0.10" "node": ">= 0.10"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/express"
} }
}, },
"node_modules/proxy-addr/node_modules/ipaddr.js": { "node_modules/proxy-addr/node_modules/ipaddr.js": {
@@ -19238,9 +19229,9 @@
} }
}, },
"node_modules/shell-quote": { "node_modules/shell-quote": {
"version": "1.12.0", "version": "1.10.0",
"resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.12.0.tgz", "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.10.0.tgz",
"integrity": "sha512-PcByqNyT/38F2kDNi006HAMRJaULuBzq/FOsw3qdZvX/GA9W/jamDaRskgHjubHiftXK5sIFxLNkvrXUwcof6Q==", "integrity": "sha512-w1aiOKwKuRgtwAReIIj89puqg+I7GvX4IbLrvmhXbzQsj1+Zwi4VO3+fa6ZF91TWSjIxoEkKnMeHcLEODK5ZXA==",
"license": "MIT", "license": "MIT",
"engines": { "engines": {
"node": ">= 0.4" "node": ">= 0.4"
@@ -19522,9 +19513,9 @@
} }
}, },
"node_modules/source-map-js": { "node_modules/source-map-js": {
"version": "1.2.2", "version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz", "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
"integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==", "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
"license": "BSD-3-Clause", "license": "BSD-3-Clause",
"engines": { "engines": {
"node": ">=0.10.0" "node": ">=0.10.0"
+28 -178
View File
@@ -412,39 +412,6 @@ paths:
security: security:
- frigateAdminAuth: [] - frigateAdminAuth: []
x-required-role: admin x-required-role: admin
/go2rtc/streams/{stream_name}/bitrate:
get:
tags:
- Camera
summary: Go2Rtc Stream Bitrate
description: |-
**Access:** Admin role required.
Measure a go2rtc stream's bitrate over a few seconds.
operationId:
go2rtc_stream_bitrate_go2rtc_streams__stream_name__bitrate_get
parameters:
- name: stream_name
in: path
required: true
schema:
type: string
title: Stream Name
responses:
'200':
description: Successful Response
content:
application/json:
schema: {}
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- frigateAdminAuth: []
x-required-role: admin
/ffprobe: /ffprobe:
get: get:
tags: tags:
@@ -1088,7 +1055,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1132,7 +1098,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1287,7 +1252,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1647,7 +1611,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1695,7 +1658,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1743,7 +1705,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1785,7 +1746,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -1869,7 +1829,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -2131,7 +2090,7 @@ paths:
required: false required: false
schema: schema:
type: string type: string
default: UTC default: utc
title: Timezone title: Timezone
responses: responses:
'200': '200':
@@ -3266,7 +3225,7 @@ paths:
anyOf: anyOf:
- type: string - type: string
- type: 'null' - type: 'null'
default: UTC default: utc
title: Timezone title: Timezone
responses: responses:
'200': '200':
@@ -3491,7 +3450,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Body title: Body
responses: responses:
'200': '200':
@@ -4216,20 +4174,19 @@ paths:
Get notices, most severe first. Get notices, most severe first.
Args: Args:
include_hidden: Also return acknowledged and muted notices, for the include_dismissed: Also return dismissed notices, for the history view
hidden list
Returns: Returns:
The notices The notices
operationId: get_notices_notices_get operationId: get_notices_notices_get
parameters: parameters:
- name: include_hidden - name: include_dismissed
in: query in: query
required: false required: false
schema: schema:
type: boolean type: boolean
default: false default: false
title: Include Hidden title: Include Dismissed
responses: responses:
'200': '200':
description: Successful Response description: Successful Response
@@ -4264,16 +4221,16 @@ paths:
security: security:
- frigateAdminAuth: [] - frigateAdminAuth: []
x-required-role: admin x-required-role: admin
/notices/muted_checks: /notices/dismissed_checks:
get: get:
tags: tags:
- Notices - Notices
summary: Get Muted Checks summary: Get Dismissed Checks
description: |- description: |-
**Access:** Admin role required. **Access:** Admin role required.
Get the muted config and stream check rows, newest first. Get the dismissed config and stream check rows, newest first.
operationId: get_muted_checks_notices_muted_checks_get operationId: get_dismissed_checks_notices_dismissed_checks_get
responses: responses:
'200': '200':
description: Successful Response description: Successful Response
@@ -4283,16 +4240,16 @@ paths:
security: security:
- frigateAdminAuth: [] - frigateAdminAuth: []
x-required-role: admin x-required-role: admin
/notices/hidden: /notices/dismissed:
delete: delete:
tags: tags:
- Notices - Notices
summary: Unhide All Notices summary: Purge Dismissed
description: |- description: |-
**Access:** Admin role required. **Access:** Admin role required.
Show every acknowledged and muted notice and check row again. Delete every dismissed notice and check row so each can show again.
operationId: unhide_all_notices_notices_hidden_delete operationId: purge_dismissed_notices_dismissed_delete
responses: responses:
'200': '200':
description: Successful Response description: Successful Response
@@ -4302,83 +4259,18 @@ paths:
security: security:
- frigateAdminAuth: [] - frigateAdminAuth: []
x-required-role: admin x-required-role: admin
/notices/{notice_id}/acknowledge: /notices/{notice_id}/dismiss:
post: post:
tags: tags:
- Notices - Notices
summary: Acknowledge Notice summary: Dismiss Notice
description: |- description: |-
**Access:** Admin role required. **Access:** Admin role required.
Hide a notice until it happens again. Hide a notice or a config or stream check row.
Config and stream check rows and the update notice never repeat, so they It stays hidden if the same problem happens again.
can only be muted. operationId: dismiss_notice_notices__notice_id__dismiss_post
operationId: acknowledge_notice_notices__notice_id__acknowledge_post
parameters:
- name: notice_id
in: path
required: true
schema:
type: string
title: Notice Id
responses:
'200':
description: Successful Response
content:
application/json:
schema: {}
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- frigateAdminAuth: []
x-required-role: admin
/notices/{notice_id}/mute:
post:
tags:
- Notices
summary: Mute Notice
description: |-
**Access:** Admin role required.
Hide a notice or a config or stream check row for good.
operationId: mute_notice_notices__notice_id__mute_post
parameters:
- name: notice_id
in: path
required: true
schema:
type: string
title: Notice Id
responses:
'200':
description: Successful Response
content:
application/json:
schema: {}
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- frigateAdminAuth: []
x-required-role: admin
/notices/{notice_id}/hidden:
delete:
tags:
- Notices
summary: Unhide Notice
description: |-
**Access:** Admin role required.
Show an acknowledged or muted notice or check row again.
operationId: unhide_notice_notices__notice_id__hidden_delete
parameters: parameters:
- name: notice_id - name: notice_id
in: path in: path
@@ -4502,16 +4394,6 @@ paths:
- type: 'null' - type: 'null'
default: 100 default: 100
title: Limit title: Limit
- name: offset
in: query
required: false
schema:
anyOf:
- type: integer
minimum: 0
- type: 'null'
default: 0
title: Offset
- name: after - name: after
in: query in: query
required: false required: false
@@ -4666,7 +4548,7 @@ paths:
anyOf: anyOf:
- type: string - type: string
- type: 'null' - type: 'null'
default: UTC default: utc
title: Timezone title: Timezone
responses: responses:
'200': '200':
@@ -4817,16 +4699,6 @@ paths:
- type: 'null' - type: 'null'
default: 50 default: 50
title: Limit title: Limit
- name: offset
in: query
required: false
schema:
anyOf:
- type: integer
minimum: 0
- type: 'null'
default: 0
title: Offset
- name: cameras - name: cameras
in: query in: query
required: false required: false
@@ -4928,7 +4800,7 @@ paths:
anyOf: anyOf:
- type: string - type: string
- type: 'null' - type: 'null'
default: UTC default: utc
title: Timezone title: Timezone
- name: min_score - name: min_score
in: query in: query
@@ -5008,7 +4880,7 @@ paths:
anyOf: anyOf:
- type: string - type: string
- type: 'null' - type: 'null'
default: UTC default: utc
title: Timezone title: Timezone
- name: has_clip - name: has_clip
in: query in: query
@@ -5661,7 +5533,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Response Create Trigger Embedding Trigger Embedding Post title: Response Create Trigger Embedding Trigger Embedding Post
'422': '422':
description: Validation Error description: Validation Error
@@ -5710,7 +5581,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Response Update Trigger Embedding Trigger Embedding title: Response Update Trigger Embedding Trigger Embedding
Camera Name Name Put Camera Name Name Put
'422': '422':
@@ -5753,7 +5623,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Response Delete Trigger Embedding Trigger Embedding title: Response Delete Trigger Embedding Trigger Embedding
Camera Name Name Delete Camera Name Name Delete
'422': '422':
@@ -5790,7 +5659,6 @@ paths:
application/json: application/json:
schema: schema:
type: object type: object
additionalProperties: true
title: Response Get Triggers Status Triggers Status Camera Name title: Response Get Triggers Status Triggers Status Camera Name
Get Get
'422': '422':
@@ -7367,7 +7235,7 @@ paths:
required: false required: false
schema: schema:
type: string type: string
default: UTC default: utc
title: Timezone title: Timezone
- name: cameras - name: cameras
in: query in: query
@@ -7417,7 +7285,7 @@ paths:
required: false required: false
schema: schema:
type: string type: string
default: UTC default: utc
title: Timezone title: Timezone
responses: responses:
'200': '200':
@@ -7587,7 +7455,6 @@ paths:
type: array type: array
items: items:
type: object type: object
additionalProperties: true
title: Response No Recordings Recordings Unavailable Get title: Response No Recordings Recordings Unavailable Get
'422': '422':
description: Validation Error description: Validation Error
@@ -7791,19 +7658,13 @@ components:
title: Update Topic title: Update Topic
config_data: config_data:
anyOf: anyOf:
- additionalProperties: true - type: object
type: object
- type: 'null' - type: 'null'
title: Config Data title: Config Data
skip_save: skip_save:
type: boolean type: boolean
title: Skip Save title: Skip Save
default: false default: false
replace_paths:
items:
type: string
type: array
title: Replace Paths
type: object type: object
title: AppConfigSetBody title: AppConfigSetBody
AppPostLoginBody: AppPostLoginBody:
@@ -8032,7 +7893,7 @@ components:
properties: properties:
file: file:
type: string type: string
contentMediaType: application/octet-stream format: binary
title: File title: File
type: object type: object
required: required:
@@ -8042,7 +7903,7 @@ components:
properties: properties:
file: file:
type: string type: string
contentMediaType: application/octet-stream format: binary
title: File title: File
type: object type: object
required: required:
@@ -8140,7 +8001,6 @@ components:
tool_calls: tool_calls:
anyOf: anyOf:
- items: - items:
additionalProperties: true
type: object type: object
type: array type: array
- type: 'null' - type: 'null'
@@ -8436,7 +8296,6 @@ components:
- type: 'null' - type: 'null'
title: Model Type title: Model Type
data: data:
additionalProperties: true
type: object type: object
title: Data title: Data
type: object type: object
@@ -8508,8 +8367,7 @@ components:
default: true default: true
draw: draw:
anyOf: anyOf:
- additionalProperties: true - type: object
type: object
- type: 'null' - type: 'null'
title: Draw title: Draw
default: {} default: {}
@@ -8752,8 +8610,7 @@ components:
description: Error message for failed jobs description: Error message for failed jobs
results: results:
anyOf: anyOf:
- additionalProperties: true - type: object
type: object
- type: 'null' - type: 'null'
title: Results title: Results
description: Result metadata for completed jobs description: Result metadata for completed jobs
@@ -9003,7 +8860,6 @@ components:
- type: 'null' - type: 'null'
title: Base Url title: Base Url
provider_options: provider_options:
additionalProperties: true
type: object type: object
title: Provider Options title: Provider Options
type: object type: object
@@ -9544,7 +9400,6 @@ components:
type: string type: string
title: Tool Name title: Tool Name
arguments: arguments:
additionalProperties: true
type: object type: object
title: Arguments title: Arguments
type: object type: object
@@ -9622,11 +9477,6 @@ components:
type: type:
type: string type: string
title: Error Type title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object type: object
required: required:
- loc - loc
-4
View File
@@ -99,10 +99,6 @@ def main() -> None:
print("*** End Config Validation Errors ***") print("*** End Config Validation Errors ***")
print("*************************************************************") print("*************************************************************")
# force a non-zero exit code for config failures
if args.validate_config:
sys.exit(1)
# attempt to start Frigate in recovery mode # attempt to start Frigate in recovery mode
try: try:
config = FrigateConfig.load(install=True, safe_load=True) config = FrigateConfig.load(install=True, safe_load=True)
+3 -57
View File
@@ -63,7 +63,6 @@ from frigate.util.builtin import (
flatten_config_data, flatten_config_data,
load_labels, load_labels,
process_config_query_string, process_config_query_string,
split_config_key_path,
update_yaml_file_bulk, update_yaml_file_bulk,
) )
from frigate.util.config import ( from frigate.util.config import (
@@ -71,10 +70,6 @@ from frigate.util.config import (
find_config_file, find_config_file,
redact_credential, redact_credential,
) )
from frigate.util.live_streams import (
generated_transcode_streams,
sync_transcode_streams,
)
from frigate.util.object_names import get_categorized_object_names from frigate.util.object_names import get_categorized_object_names
from frigate.util.schema import get_config_schema from frigate.util.schema import get_config_schema
from frigate.util.services import ( from frigate.util.services import (
@@ -311,12 +306,9 @@ def config(request: Request):
mode="json", warnings="none", exclude_none=True mode="json", warnings="none", exclude_none=True
) )
is_admin = request.headers.get("remote-role") == "admin" # remove environment_vars for non-admin users
if request.headers.get("remote-role") != "admin":
# hide environment_vars and the notification email from non-admin users
if not is_admin:
config.pop("environment_vars", None) config.pop("environment_vars", None)
redact_credential(config["notifications"], "email")
# redact mqtt credentials # redact mqtt credentials
redact_credential(config["mqtt"], "password") redact_credential(config["mqtt"], "password")
@@ -373,15 +365,7 @@ def config(request: Request):
camera_name camera_name
) )
if base_sections: if base_sections:
# copy so redaction below can't alter the profile manager's cache camera_dict["base_config"] = base_sections
camera_dict["base_config"] = copy.deepcopy(base_sections)
# cameras inherit the global notification email
if not is_admin:
redact_credential(camera_dict["notifications"], "email")
redact_credential(
camera_dict.get("base_config", {}).get("notifications", {}), "email"
)
# remove go2rtc stream passwords # remove go2rtc stream passwords
go2rtc: dict[str, Any] = config_obj.go2rtc.model_dump( go2rtc: dict[str, Any] = config_obj.go2rtc.model_dump(
@@ -409,11 +393,6 @@ def config(request: Request):
model_dict["non_logo_attributes"] = model.non_logo_attributes model_dict["non_logo_attributes"] = model.non_logo_attributes
model_dict["labelmap"] = model.merged_labelmap model_dict["labelmap"] = model.merged_labelmap
# report the configured reference rather than the resolved cache path,
# so saving the config back doesn't lose the Frigate+ model
if model.plus_id:
model_dict["path"] = f"plus://{model.plus_id}"
if not config["plus"]["enabled"]: if not config["plus"]["enabled"]:
continue continue
@@ -450,8 +429,6 @@ def ffmpeg_presets():
hwaccel_presets = [ hwaccel_presets = [
"preset-rpi-64-h264", "preset-rpi-64-h264",
"preset-rpi-64-h265", "preset-rpi-64-h265",
"preset-apple-silicon-h264",
"preset-apple-silicon-h265",
"preset-jetson-h264", "preset-jetson-h264",
"preset-jetson-h265", "preset-jetson-h265",
"preset-rkmpp", "preset-rkmpp",
@@ -831,17 +808,6 @@ def _config_set_in_memory(request: Request, body: AppConfigSetBody) -> JSONRespo
) )
def _config_path_exists(data: Any, key_path: str) -> bool:
"""Return whether a dotted config path is present in parsed yaml."""
for key in split_config_key_path(key_path):
if not isinstance(data, dict) or key not in data:
return False
data = data[key]
return True
@router.put("/config/set", dependencies=[Depends(require_role(["admin"]))]) @router.put("/config/set", dependencies=[Depends(require_role(["admin"]))])
def config_set(request: Request, body: AppConfigSetBody): def config_set(request: Request, body: AppConfigSetBody):
config_file = find_config_file() config_file = find_config_file()
@@ -896,19 +862,6 @@ def config_set(request: Request, body: AppConfigSetBody):
status_code=400, status_code=400,
) )
# delete replaced paths first so their maps are rewritten in
# the order sent; update_yaml would otherwise keep old order
if body.replace_paths:
old_yaml = ruamel.yaml.YAML(typ="safe").load(old_raw_config) or {}
updates = {
**{
path: ""
for path in body.replace_paths
if _config_path_exists(old_yaml, path)
},
**updates,
}
# apply all updates in a single operation # apply all updates in a single operation
update_yaml_file_bulk(config_file, updates) update_yaml_file_bulk(config_file, updates)
@@ -972,15 +925,9 @@ def config_set(request: Request, body: AppConfigSetBody):
if request.app.dispatcher is not None: if request.app.dispatcher is not None:
request.app.dispatcher.clear_runtime_state_for_yaml_keys(updates.keys()) request.app.dispatcher.clear_runtime_state_for_yaml_keys(updates.keys())
go2rtc_synced = True
if body.requires_restart == 0 or body.update_topic: if body.requires_restart == 0 or body.update_topic:
old_config: FrigateConfig = request.app.frigate_config old_config: FrigateConfig = request.app.frigate_config
swap_runtime_config(request.app, config) swap_runtime_config(request.app, config)
go2rtc_synced = sync_transcode_streams(
generated_transcode_streams(old_config),
generated_transcode_streams(config),
)
if body.update_topic: if body.update_topic:
if body.update_topic.startswith("config/cameras/"): if body.update_topic.startswith("config/cameras/"):
@@ -1040,7 +987,6 @@ def config_set(request: Request, body: AppConfigSetBody):
if body.requires_restart == 0 if body.requires_restart == 0
else "Config successfully updated, restart to apply" else "Config successfully updated, restart to apply"
), ),
"go2rtc_synced": go2rtc_synced,
} }
), ),
status_code=200, status_code=200,
+24 -24
View File
@@ -130,18 +130,23 @@ def require_admin_by_default():
if path.startswith(EXEMPT_PREFIXES): if path.startswith(EXEMPT_PREFIXES):
return return
# Camera routes enforce per-camera access via route-level dependencies # Dynamic camera path exemption:
# (e.g. require_camera_access). Match on the route template, not the raw # Any path whose first segment matches a configured camera name should
# path, so a camera named like another namespace (e.g. "faces") can't # bypass the global admin requirement. These endpoints enforce access
# waive the admin check for that namespace's routes. # via route-level dependencies (e.g. require_camera_access) to ensure
route = request.scope.get("route") # per-camera authorization. This allows non-admin authenticated users
if ( # (e.g. viewer role) to access camera-specific resources without
route is not None # needing admin privileges.
and route.path.startswith("/{camera_name}") try:
and request.path_params.get("camera_name") if path.startswith("/"):
in request.app.frigate_config.cameras first_segment = path.split("/", 2)[1]
): if (
return first_segment
and first_segment in request.app.frigate_config.cameras
):
return
except Exception:
pass
# For all other paths, require admin role # For all other paths, require admin role
# Internal port requests have admin role set automatically # Internal port requests have admin role set automatically
@@ -319,17 +324,11 @@ def get_remote_addr(request: Request):
network = ipaddress.ip_network(proxy) network = ipaddress.ip_network(proxy)
except ValueError: except ValueError:
logger.warning(f"Unable to parse trusted network: {proxy}") logger.warning(f"Unable to parse trusted network: {proxy}")
continue
trusted_proxies.append(network) trusted_proxies.append(network)
# return the first remote address that is not trusted # return the first remote address that is not trusted
for addr in route: for addr in route:
try: ip = ipaddress.ip_address(addr.strip())
ip = ipaddress.ip_address(addr.strip())
except ValueError:
logger.debug("Invalid address in X-Forwarded-For header")
return direct_addr or "127.0.0.1"
logger.debug(f"Checking {ip} (v{ip.version})") logger.debug(f"Checking {ip} (v{ip.version})")
trusted = False trusted = False
for trusted_proxy in trusted_proxies: for trusted_proxy in trusted_proxies:
@@ -474,11 +473,12 @@ def create_encoded_jwt(user, role, expiration, secret):
def set_jwt_cookie(response: Response, cookie_name, encoded_jwt, max_age, secure): def set_jwt_cookie(response: Response, cookie_name, encoded_jwt, max_age, secure):
# TODO: ideally this would set secure as well, but that requires TLS # TODO: ideally this would set secure as well, but that requires TLS
# Starlette sets SameSite=Lax by default. The cookie is still sent to # SameSite is intentionally left unset (browsers default to Lax). Setting
# same-site iframes (e.g. Home Assistant on the same host or domain), but # SameSite=Lax/Strict would stop the cookie from being sent in cross-origin
# not to cross-site ones. CSRF is also mitigated by requiring a custom # iframes, breaking embedded views such as the Home Assistant Frigate card.
# X-CSRF-TOKEN header, which cross-origin pages cannot set without a CORS # CSRF is instead mitigated by requiring a custom X-CSRF-TOKEN header, which
# preflight that Frigate never grants (see check_csrf in api/fastapi_app.py). # cross-origin pages cannot set without a CORS preflight that Frigate never
# grants (see check_csrf in api/fastapi_app.py).
response.set_cookie( response.set_cookie(
key=cookie_name, key=cookie_name,
value=encoded_jwt, value=encoded_jwt,
+1 -40
View File
@@ -33,17 +33,12 @@ from frigate.config.camera.updater import (
CameraConfigUpdateEnum, CameraConfigUpdateEnum,
CameraConfigUpdateTopic, CameraConfigUpdateTopic,
) )
from frigate.config.env import UnknownVariableError, substitute_frigate_vars
from frigate.models import User from frigate.models import User
from frigate.util.builtin import clean_camera_user_pass, get_record_segment_time from frigate.util.builtin import clean_camera_user_pass, get_record_segment_time
from frigate.util.camera_cleanup import cleanup_camera_db, cleanup_camera_files from frigate.util.camera_cleanup import cleanup_camera_db, cleanup_camera_files
from frigate.util.config import find_config_file from frigate.util.config import find_config_file
from frigate.util.env import UnknownVariableError, substitute_frigate_vars
from frigate.util.image import run_ffmpeg_snapshot from frigate.util.image import run_ffmpeg_snapshot
from frigate.util.live_streams import (
generated_transcode_streams,
measure_stream_bitrate,
sync_transcode_streams,
)
from frigate.util.services import ( from frigate.util.services import (
analyze_record_keyframes, analyze_record_keyframes,
ffprobe_stream, ffprobe_stream,
@@ -253,34 +248,6 @@ def go2rtc_delete_stream(stream_name: str):
) )
@router.get(
"/go2rtc/streams/{stream_name}/bitrate",
dependencies=[Depends(require_role(["admin"]))],
)
async def go2rtc_stream_bitrate(request: Request, stream_name: str):
"""Measure a go2rtc stream's bitrate over a few seconds."""
config: FrigateConfig = request.app.frigate_config
known = set(config.go2rtc.model_dump().get("streams") or {}) | set(
generated_transcode_streams(config)
)
if stream_name not in known:
return JSONResponse(
content={"success": False, "message": "Unknown stream"},
status_code=404,
)
kbps = await asyncio.to_thread(measure_stream_bitrate, stream_name)
if kbps is None:
return JSONResponse(
content={"success": False, "message": "Stream sent no data"},
status_code=502,
)
return JSONResponse(content={"success": True, "kbps": round(kbps)})
@router.get("/ffprobe", dependencies=[Depends(require_role(["admin"]))]) @router.get("/ffprobe", dependencies=[Depends(require_role(["admin"]))])
def ffprobe(request: Request, paths: str = "", detailed: bool = False): def ffprobe(request: Request, paths: str = "", detailed: bool = False):
path_param = paths path_param = paths
@@ -1375,12 +1342,6 @@ async def delete_camera(
except Exception: except Exception:
logger.debug("Failed to remove go2rtc stream for %s", camera_name) logger.debug("Failed to remove go2rtc stream for %s", camera_name)
await asyncio.to_thread(
sync_transcode_streams,
generated_transcode_streams(frigate_config),
generated_transcode_streams(request.app.frigate_config),
)
return JSONResponse( return JSONResponse(
content={ content={
"success": True, "success": True,
+1 -8
View File
@@ -1,6 +1,5 @@
"""Object classification APIs.""" """Object classification APIs."""
import asyncio
import datetime import datetime
import logging import logging
import os import os
@@ -49,9 +48,6 @@ logger = logging.getLogger(__name__)
router = APIRouter(tags=[Tags.classification]) router = APIRouter(tags=[Tags.classification])
# concurrent collections for one model share its temp and train directories
example_collection_lock = asyncio.Lock()
def invalid_name_response(value: str) -> JSONResponse: def invalid_name_response(value: str) -> JSONResponse:
"""Response for a name that cannot be used as a path component.""" """Response for a name that cannot be used as a path component."""
@@ -1294,10 +1290,7 @@ async def generate_object_examples(request: Request, body: GenerateObjectExample
if model_name is None: if model_name is None:
return invalid_name_response(body.model_name) return invalid_name_response(body.model_name)
async with example_collection_lock: collect_object_classification_examples(model_name, body.label)
await asyncio.to_thread(
collect_object_classification_examples, model_name, body.label
)
return JSONResponse( return JSONResponse(
content={"success": True, "message": "Example generation completed"}, content={"success": True, "message": "Example generation completed"},
-1
View File
@@ -51,7 +51,6 @@ def swap_runtime_config(app: FastAPI, config: FrigateConfig) -> None:
if app.stats_emitter is not None: if app.stats_emitter is not None:
app.stats_emitter.config = config app.stats_emitter.config = config
app.stats_emitter.hardware_stats.set_config(config)
if app.dispatcher is not None: if app.dispatcher is not None:
app.dispatcher.config = config app.dispatcher.config = config
@@ -7,4 +7,4 @@ class AppTimelineHourlyQueryParameters(BaseModel):
after: float | None = None after: float | None = None
before: float | None = None before: float | None = None
limit: int | None = 200 limit: int | None = 200
timezone: str | None = "UTC" timezone: str | None = "utc"
@@ -14,7 +14,6 @@ class EventsQueryParams(BaseModel):
zone: str | None = "all" zone: str | None = "all"
zones: str | None = "all" zones: str | None = "all"
limit: int | None = 100 limit: int | None = 100
offset: int | None = Field(0, ge=0)
after: float | None = None after: float | None = None
before: float | None = None before: float | None = None
time_range: str | None = DEFAULT_TIME_RANGE time_range: str | None = DEFAULT_TIME_RANGE
@@ -40,7 +39,7 @@ class EventsQueryParams(BaseModel):
max_length: float | None = None max_length: float | None = None
event_id: str | None = None event_id: str | None = None
sort: str | None = None sort: str | None = None
timezone: str | None = "UTC" timezone: str | None = "utc"
class EventsSearchQueryParams(BaseModel): class EventsSearchQueryParams(BaseModel):
@@ -56,7 +55,6 @@ class EventsSearchQueryParams(BaseModel):
deprecated=True, deprecated=True,
) )
limit: int | None = 50 limit: int | None = 50
offset: int | None = Field(0, ge=0)
cameras: str | None = "all" cameras: str | None = "all"
labels: str | None = "all" labels: str | None = "all"
sub_labels: str | None = "all" sub_labels: str | None = "all"
@@ -68,7 +66,7 @@ class EventsSearchQueryParams(BaseModel):
has_clip: bool | None = None has_clip: bool | None = None
has_snapshot: bool | None = None has_snapshot: bool | None = None
is_submitted: bool | None = None is_submitted: bool | None = None
timezone: str | None = "UTC" timezone: str | None = "utc"
min_score: float | None = None min_score: float | None = None
max_score: float | None = None max_score: float | None = None
min_speed: float | None = None min_speed: float | None = None
@@ -78,6 +76,6 @@ class EventsSearchQueryParams(BaseModel):
class EventsSummaryQueryParams(BaseModel): class EventsSummaryQueryParams(BaseModel):
timezone: str | None = "UTC" timezone: str | None = "utc"
has_clip: int | None = None has_clip: int | None = None
has_snapshot: int | None = None has_snapshot: int | None = None
@@ -3,7 +3,7 @@ from pydantic.json_schema import SkipJsonSchema
class MediaRecordingsSummaryQueryParams(BaseModel): class MediaRecordingsSummaryQueryParams(BaseModel):
timezone: str = "UTC" timezone: str = "utc"
cameras: str | None = "all" cameras: str | None = "all"
@@ -19,7 +19,7 @@ class ReviewSummaryQueryParams(BaseModel):
cameras: str = "all" cameras: str = "all"
labels: str = "all" labels: str = "all"
zones: str = "all" zones: str = "all"
timezone: str = "UTC" timezone: str = "utc"
class ReviewActivityMotionQueryParams(BaseModel): class ReviewActivityMotionQueryParams(BaseModel):
-2
View File
@@ -10,8 +10,6 @@ class AppConfigSetBody(BaseModel):
update_topic: str | None = None update_topic: str | None = None
config_data: dict[str, Any] | None = None config_data: dict[str, Any] | None = None
skip_save: bool = False skip_save: bool = False
# paths rewritten whole, so a map saves in the order sent
replace_paths: list[str] = Field(default_factory=list)
class GenAIProbeBody(BaseModel): class GenAIProbeBody(BaseModel):
+2 -11
View File
@@ -129,7 +129,6 @@ def events(
zones = zone zones = zone
limit = params.limit limit = params.limit
offset = params.offset
after = params.after after = params.after
before = params.before before = params.before
time_range = params.time_range time_range = params.time_range
@@ -362,15 +361,11 @@ def events(
else: else:
order_by = Event.start_time.desc() order_by = Event.start_time.desc()
# offset paging needs a stable order when scores or speeds tie
tiebreaker = [Event.id] if sort and sort.startswith(("score", "speed")) else []
events = ( events = (
Event.select(*selected_columns) Event.select(*selected_columns)
.where(reduce(operator.and_, clauses)) .where(reduce(operator.and_, clauses))
.order_by(order_by, *tiebreaker) .order_by(order_by)
.limit(limit) .limit(limit)
.offset(offset)
.dicts() .dicts()
.iterator() .iterator()
) )
@@ -539,7 +534,6 @@ def events_search(
search_type = params.search_type search_type = params.search_type
include_thumbnails = params.include_thumbnails include_thumbnails = params.include_thumbnails
limit = params.limit limit = params.limit
offset = params.offset
sort = params.sort sort = params.sort
# Filters # Filters
@@ -846,9 +840,6 @@ def events_search(
if search_results: if search_results:
events_query = events_query.where(Event.id << list(search_results.keys())) events_query = events_query.where(Event.id << list(search_results.keys()))
# sorts below are stable, so this orders ties for offset paging
events_query = events_query.order_by(Event.id)
# Fetch events and process them in a single pass # Fetch events and process them in a single pass
processed_events = [] processed_events = []
for event in events_query.dicts(): for event in events_query.dicts():
@@ -906,7 +897,7 @@ def events_search(
processed_events.sort(key=lambda x: x["start_time"], reverse=True) processed_events.sort(key=lambda x: x["start_time"], reverse=True)
# Limit the number of events returned # Limit the number of events returned
processed_events = processed_events[offset:][:limit] processed_events = processed_events[:limit]
return JSONResponse(content=processed_events) return JSONResponse(content=processed_events)
+9 -12
View File
@@ -1,7 +1,6 @@
import asyncio import asyncio
import logging import logging
import re import re
from contextlib import asynccontextmanager
from fastapi import Depends, FastAPI, Request from fastapi import Depends, FastAPI, Request
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
@@ -82,20 +81,9 @@ def create_fastapi_app(
config_holder: ConfigHolder | None = None, config_holder: ConfigHolder | None = None,
notice_registry: NoticeRegistry | None = None, notice_registry: NoticeRegistry | None = None,
): ):
@asynccontextmanager
async def lifespan(app: FastAPI):
logger.info("FastAPI started")
asyncio.create_task(
debug_replay_auto_stop_watchdog(
replay_manager, frigate_config, config_publisher
)
)
yield
logger.info("Starting FastAPI app") logger.info("Starting FastAPI app")
app = FastAPI( app = FastAPI(
debug=False, debug=False,
lifespan=lifespan,
swagger_ui_parameters={"apisSorter": "alpha", "operationsSorter": "alpha"}, swagger_ui_parameters={"apisSorter": "alpha", "operationsSorter": "alpha"},
dependencies=[Depends(require_admin_by_default())] dependencies=[Depends(require_admin_by_default())]
if enforce_default_admin if enforce_default_admin
@@ -131,6 +119,15 @@ def create_fastapi_app(
database.close() database.close()
return response return response
@app.on_event("startup")
async def startup():
logger.info("FastAPI started")
asyncio.create_task(
debug_replay_auto_stop_watchdog(
replay_manager, frigate_config, config_publisher
)
)
# Rate limiter (used for login endpoint) # Rate limiter (used for login endpoint)
if frigate_config.auth.failed_login_rate_limit is None: if frigate_config.auth.failed_login_rate_limit is None:
limiter.enabled = False limiter.enabled = False
+9 -6
View File
@@ -14,10 +14,10 @@ from enum import Enum
from pathlib import Path as FilePath from pathlib import Path as FilePath
from typing import IO, Any from typing import IO, Any
from urllib.parse import unquote from urllib.parse import unquote
from zoneinfo import ZoneInfo
import cv2 import cv2
import numpy as np import numpy as np
import pytz
from fastapi import APIRouter, Depends, Path, Query, Request, Response from fastapi import APIRouter, Depends, Path, Query, Request, Response
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
from pathvalidate import sanitize_filename from pathvalidate import sanitize_filename
@@ -63,8 +63,8 @@ from frigate.util.recording_coverage import (
null_audio_glitches, null_audio_glitches,
plan_clip, plan_clip,
resolve_coverage, resolve_coverage,
stream_has_audio,
) )
from frigate.util.time import get_normalized_tz_name
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -681,10 +681,15 @@ async def _vod_response(
end_ts, end_ts,
force_discontinuity, force_discontinuity,
) )
intervals = resolve_coverage(camera_name, start_ts, end_ts)
# rows contradicting their stream's audio composition are # rows contradicting their stream's audio composition are
# truncated-shutdown glitches # truncated-shutdown glitches
main_audio = stream_has_audio(intervals, main=True)
sub_audio = stream_has_audio(intervals, main=False)
spans = build_spans( spans = build_spans(
null_audio_glitches(resolve_coverage(camera_name, start_ts, end_ts)), null_audio_glitches(intervals, main_audio, sub_audio),
stream_preference, stream_preference,
) )
@@ -838,9 +843,7 @@ async def vod_hour(
parts = year_month.split("-") parts = year_month.split("-")
start_date = ( start_date = (
datetime(int(parts[0]), int(parts[1]), day, hour, tzinfo=UTC) datetime(int(parts[0]), int(parts[1]), day, hour, tzinfo=UTC)
- datetime.now( - datetime.now(pytz.timezone(tz_name.replace(",", "/"))).utcoffset()
ZoneInfo(get_normalized_tz_name(tz_name.replace(",", "/")))
).utcoffset()
) )
end_date = start_date + timedelta(hours=1) - timedelta(milliseconds=1) end_date = start_date + timedelta(hours=1) - timedelta(milliseconds=1)
start_ts = start_date.timestamp() start_ts = start_date.timestamp()
+22 -50
View File
@@ -14,18 +14,17 @@ router = APIRouter(tags=[Tags.notices])
@router.get("/notices", dependencies=[Depends(require_role(["admin"]))]) @router.get("/notices", dependencies=[Depends(require_role(["admin"]))])
def get_notices(request: Request, include_hidden: bool = False) -> JSONResponse: def get_notices(request: Request, include_dismissed: bool = False) -> JSONResponse:
"""Get notices, most severe first. """Get notices, most severe first.
Args: Args:
include_hidden: Also return acknowledged and muted notices, for the include_dismissed: Also return dismissed notices, for the history view
hidden list
Returns: Returns:
The notices The notices
""" """
return JSONResponse( return JSONResponse(
content=request.app.notice_registry.active(include_hidden=include_hidden) content=request.app.notice_registry.active(include_dismissed=include_dismissed)
) )
@@ -35,64 +34,37 @@ def get_notice_stats(request: Request) -> JSONResponse:
return JSONResponse(content=request.app.notice_registry.stats()) return JSONResponse(content=request.app.notice_registry.stats())
@router.get("/notices/muted_checks", dependencies=[Depends(require_role(["admin"]))]) @router.get(
def get_muted_checks(request: Request) -> JSONResponse: "/notices/dismissed_checks", dependencies=[Depends(require_role(["admin"]))]
"""Get the muted config and stream check rows, newest first.""" )
return JSONResponse(content=request.app.notice_registry.muted_checks()) def get_dismissed_checks(request: Request) -> JSONResponse:
"""Get the dismissed config and stream check rows, newest first."""
return JSONResponse(content=request.app.notice_registry.dismissed_checks())
@router.delete("/notices/hidden", dependencies=[Depends(require_role(["admin"]))]) @router.delete("/notices/dismissed", dependencies=[Depends(require_role(["admin"]))])
def unhide_all_notices(request: Request) -> JSONResponse: def purge_dismissed(request: Request) -> JSONResponse:
"""Show every acknowledged and muted notice and check row again.""" """Delete every dismissed notice and check row so each can show again."""
request.app.notice_registry.unhide_all() request.app.notice_registry.purge_dismissed()
return JSONResponse(content={"success": True, "message": "Notices shown again"}) return JSONResponse(
content={"success": True, "message": "Dismissed notices cleared"}
)
# model notice ids contain a slash, so the id is a path parameter # model notice ids contain a slash, so the id is a path parameter
@router.post( @router.post(
"/notices/{notice_id:path}/acknowledge", "/notices/{notice_id:path}/dismiss",
dependencies=[Depends(require_role(["admin"]))], dependencies=[Depends(require_role(["admin"]))],
) )
def acknowledge_notice(request: Request, notice_id: str) -> JSONResponse: def dismiss_notice(request: Request, notice_id: str) -> JSONResponse:
"""Hide a notice until it happens again. """Hide a notice or a config or stream check row.
Config and stream check rows and the update notice never repeat, so they It stays hidden if the same problem happens again.
can only be muted.
""" """
if not request.app.notice_registry.acknowledge(notice_id): if not request.app.notice_registry.dismiss(notice_id):
return JSONResponse( return JSONResponse(
content={"success": False, "message": "Notice not found"}, content={"success": False, "message": "Notice not found"},
status_code=404, status_code=404,
) )
return JSONResponse(content={"success": True, "message": "Notice acknowledged"}) return JSONResponse(content={"success": True, "message": "Notice dismissed"})
@router.post(
"/notices/{notice_id:path}/mute",
dependencies=[Depends(require_role(["admin"]))],
)
def mute_notice(request: Request, notice_id: str) -> JSONResponse:
"""Hide a notice or a config or stream check row for good."""
if not request.app.notice_registry.mute(notice_id):
return JSONResponse(
content={"success": False, "message": "Notice not found"},
status_code=404,
)
return JSONResponse(content={"success": True, "message": "Notice muted"})
@router.delete(
"/notices/{notice_id:path}/hidden",
dependencies=[Depends(require_role(["admin"]))],
)
def unhide_notice(request: Request, notice_id: str) -> JSONResponse:
"""Show an acknowledged or muted notice or check row again."""
if not request.app.notice_registry.unhide(notice_id):
return JSONResponse(
content={"success": False, "message": "Notice not found"},
status_code=404,
)
return JSONResponse(content={"success": True, "message": "Notice shown again"})
+2 -5
View File
@@ -5,8 +5,8 @@ import logging
import os import os
import threading import threading
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from zoneinfo import ZoneInfo
import pytz
from fastapi import APIRouter, Depends, HTTPException from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
@@ -22,7 +22,6 @@ from frigate.api.defs.response.preview_response import (
from frigate.api.defs.tags import Tags from frigate.api.defs.tags import Tags
from frigate.const import BASE_DIR, CACHE_DIR, PREVIEW_FRAME_TYPE from frigate.const import BASE_DIR, CACHE_DIR, PREVIEW_FRAME_TYPE
from frigate.models import Previews from frigate.models import Previews
from frigate.util.time import get_normalized_tz_name
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -127,9 +126,7 @@ def preview_hour(
parts = year_month.split("-") parts = year_month.split("-")
start_date = ( start_date = (
datetime(int(parts[0]), int(parts[1]), int(day), int(hour), tzinfo=UTC) datetime(int(parts[0]), int(parts[1]), int(day), int(hour), tzinfo=UTC)
- datetime.now( - datetime.now(pytz.timezone(tz_name.replace(",", "/"))).utcoffset()
ZoneInfo(get_normalized_tz_name(tz_name.replace(",", "/")))
).utcoffset()
) )
end_date = start_date + timedelta(hours=1) - timedelta(milliseconds=1) end_date = start_date + timedelta(hours=1) - timedelta(milliseconds=1)
start_ts = start_date.timestamp() start_ts = start_date.timestamp()
+1 -1
View File
@@ -175,7 +175,7 @@ async def all_recordings_summary(
@router.get( @router.get(
"/{camera_name}/recordings/summary", dependencies=[Depends(require_camera_access)] "/{camera_name}/recordings/summary", dependencies=[Depends(require_camera_access)]
) )
async def recordings_summary(camera_name: str, timezone: str = "UTC"): async def recordings_summary(camera_name: str, timezone: str = "utc"):
"""Returns hourly summary for recordings of given camera""" """Returns hourly summary for recordings of given camera"""
time_range_query = ( time_range_query = (
+12 -21
View File
@@ -44,22 +44,6 @@ logger = logging.getLogger(__name__)
router = APIRouter(tags=[Tags.review]) router = APIRouter(tags=[Tags.review])
def get_label_clause(label: str, include_audio: bool = True):
"""Build a clause matching a label within a review segment's data.
Verified objects are stored with a `-verified` suffix (eg. `person-verified`)
so that variant is matched as well.
"""
clause = (ReviewSegment.data["objects"].cast("text") % f'*"{label}"*') | (
ReviewSegment.data["objects"].cast("text") % f'*"{label}-verified"*'
)
if include_audio:
clause |= ReviewSegment.data["audio"].cast("text") % f'*"{label}"*'
return clause
@router.get( @router.get(
"/review", "/review",
response_model=list[ReviewSegmentResponse], response_model=list[ReviewSegmentResponse],
@@ -109,7 +93,10 @@ async def review(
filtered_labels = labels.split(",") filtered_labels = labels.split(",")
for label in filtered_labels: for label in filtered_labels:
label_clauses.append(get_label_clause(label)) label_clauses.append(
(ReviewSegment.data["objects"].cast("text") % f'*"{label}"*')
| (ReviewSegment.data["audio"].cast("text") % f'*"{label}"*')
)
clauses.append(reduce(operator.or_, label_clauses)) clauses.append(reduce(operator.or_, label_clauses))
if zones != "all": if zones != "all":
@@ -252,7 +239,10 @@ async def review_summary(
filtered_labels = labels.split(",") filtered_labels = labels.split(",")
for label in filtered_labels: for label in filtered_labels:
label_clauses.append(get_label_clause(label)) label_clauses.append(
(ReviewSegment.data["objects"].cast("text") % f'*"{label}"*')
| (ReviewSegment.data["audio"].cast("text") % f'*"{label}"*')
)
clauses.append(reduce(operator.or_, label_clauses)) clauses.append(reduce(operator.or_, label_clauses))
if zones != "all": if zones != "all":
# use matching so segments with multiple zones # use matching so segments with multiple zones
@@ -350,8 +340,9 @@ async def review_summary(
filtered_labels = labels.split(",") filtered_labels = labels.split(",")
for label in filtered_labels: for label in filtered_labels:
label_clauses.append(get_label_clause(label, include_audio=False)) label_clauses.append(
ReviewSegment.data["objects"].cast("text") % f'*"{label}"*'
)
clauses.append(reduce(operator.or_, label_clauses)) clauses.append(reduce(operator.or_, label_clauses))
# Find the time range of available data # Find the time range of available data
@@ -702,7 +693,7 @@ def motion_activity(
df = df[df["camera"] != ""] df = df[df["camera"] != ""]
# change types for output # change types for output
df.index = df.index.as_unit("s").astype(int) df.index = df.index.astype(int) // (10**9)
normalized = df.reset_index().to_dict("records") normalized = df.reset_index().to_dict("records")
return JSONResponse(content=normalized) return JSONResponse(content=normalized)
+15 -20
View File
@@ -12,8 +12,8 @@ from pathlib import Path
import psutil import psutil
import uvicorn import uvicorn
from peewee import SqliteDatabase
from peewee_migrate import Router from peewee_migrate import Router
from playhouse.sqlite_ext import SqliteExtDatabase
from frigate.api.auth import hash_password from frigate.api.auth import hash_password
from frigate.api.fastapi_app import create_fastapi_app from frigate.api.fastapi_app import create_fastapi_app
@@ -49,6 +49,7 @@ from frigate.debug_replay import (
DebugReplayManager, DebugReplayManager,
cleanup_replay_cameras, cleanup_replay_cameras,
) )
from frigate.detectors.detector_config import SceneEnum
from frigate.detectors.detector_types import api_types from frigate.detectors.detector_types import api_types
from frigate.detectors.device import build_detector_config, runner_names from frigate.detectors.device import build_detector_config, runner_names
from frigate.embeddings import EmbeddingProcess, EmbeddingsContext from frigate.embeddings import EmbeddingProcess, EmbeddingsContext
@@ -77,7 +78,7 @@ from frigate.notices.registry import NoticeRegistry
from frigate.object_detection.base import ObjectDetectProcess from frigate.object_detection.base import ObjectDetectProcess
from frigate.object_detection.util import detection_frame_size from frigate.object_detection.util import detection_frame_size
from frigate.output.output import OutputProcess from frigate.output.output import OutputProcess
from frigate.ptz.autotrack import PtzAutoTracker from frigate.ptz.autotrack import PtzAutoTrackerThread
from frigate.ptz.onvif import OnvifController from frigate.ptz.onvif import OnvifController
from frigate.record.cleanup import RecordingCleanup from frigate.record.cleanup import RecordingCleanup
from frigate.record.export import migrate_exports from frigate.record.export import migrate_exports
@@ -107,7 +108,7 @@ class FrigateApp:
self.metrics_manager = manager self.metrics_manager = manager
self.audio_process: mp.Process | None = None self.audio_process: mp.Process | None = None
self.stop_event = stop_event self.stop_event = stop_event
self.detection_queues: dict[str, Queue] = { self.detection_queues: dict[SceneEnum, Queue] = {
model.scene: mp.Queue() for model in config.models model.scene: mp.Queue() for model in config.models
} }
self.detectors: dict[str, ObjectDetectProcess] = {} self.detectors: dict[str, ObjectDetectProcess] = {}
@@ -166,7 +167,11 @@ class FrigateApp:
# create camera_metrics # create camera_metrics
for camera_name in self.config.cameras.keys(): for camera_name in self.config.cameras.keys():
self.camera_metrics[camera_name] = CameraMetrics(self.metrics_manager) self.camera_metrics[camera_name] = CameraMetrics(self.metrics_manager)
self.ptz_metrics[camera_name] = PTZMetrics() self.ptz_metrics[camera_name] = PTZMetrics(
autotracker_enabled=self.config.cameras[
camera_name
].onvif.autotracking.enabled
)
def init_queues(self) -> None: def init_queues(self) -> None:
# Queue for cameras to push tracked objects to # Queue for cameras to push tracked objects to
@@ -186,7 +191,7 @@ class FrigateApp:
self.timeline_queue: Queue = mp.Queue() self.timeline_queue: Queue = mp.Queue()
def init_database(self) -> None: def init_database(self) -> None:
def vacuum_db(db: SqliteDatabase) -> None: def vacuum_db(db: SqliteExtDatabase) -> None:
logger.info("Running database vacuum") logger.info("Running database vacuum")
db.execute_sql("VACUUM;") db.execute_sql("VACUUM;")
@@ -197,7 +202,7 @@ class FrigateApp:
logger.error("Unable to write to /config to save DB state") logger.error("Unable to write to /config to save DB state")
# Migrate DB schema # Migrate DB schema
migrate_db = SqliteDatabase(self.config.database.path) migrate_db = SqliteExtDatabase(self.config.database.path)
# Run migrations # Run migrations
del logging.getLogger("peewee_migrate").handlers[:] del logging.getLogger("peewee_migrate").handlers[:]
@@ -390,7 +395,7 @@ class FrigateApp:
logger.error("Unable to prepare the %s runtime: %s", detector_type, err) logger.error("Unable to prepare the %s runtime: %s", detector_type, err)
def start_detectors(self) -> None: def start_detectors(self) -> None:
model_cameras: dict[str, list[str]] = { model_cameras: dict[SceneEnum, list[str]] = {
model.scene: [] for model in self.config.models model.scene: [] for model in self.config.models
} }
@@ -439,7 +444,7 @@ class FrigateApp:
) )
def start_ptz_autotracker(self) -> None: def start_ptz_autotracker(self) -> None:
self.ptz_autotracker_thread = PtzAutoTracker( self.ptz_autotracker_thread = PtzAutoTrackerThread(
self.config, self.config,
self.onvif_controller, self.onvif_controller,
self.ptz_metrics, self.ptz_metrics,
@@ -478,7 +483,7 @@ class FrigateApp:
def start_audio_processor(self) -> None: def start_audio_processor(self) -> None:
self.audio_process = AudioProcessor( self.audio_process = AudioProcessor(
self.config, self.camera_metrics, self.embeddings_metrics, self.stop_event self.config, self.camera_metrics, self.stop_event
) )
self.audio_process.start() self.audio_process.start()
self.processes["audio_detector"] = self.audio_process.pid or 0 self.processes["audio_detector"] = self.audio_process.pid or 0
@@ -551,20 +556,10 @@ class FrigateApp:
"output", "output",
lambda: OutputProcess(self.config, self.stop_event), lambda: OutputProcess(self.config, self.stop_event),
), ),
(
"audio_process",
"audio_detector",
lambda: AudioProcessor(
self.config,
self.camera_metrics,
self.embeddings_metrics,
self.stop_event,
),
),
] ]
for attr, key, factory in specs: for attr, key, factory in specs:
if getattr(self, attr, None) is None: if not hasattr(self, attr):
continue continue
def on_restart( def on_restart(
+13 -7
View File
@@ -43,6 +43,8 @@ class CameraMetrics:
class PTZMetrics: class PTZMetrics:
autotracker_enabled: Synchronized
start_time: Synchronized start_time: Synchronized
stop_time: Synchronized stop_time: Synchronized
frame_time: Synchronized frame_time: Synchronized
@@ -50,17 +52,21 @@ class PTZMetrics:
max_zoom: Synchronized max_zoom: Synchronized
min_zoom: Synchronized min_zoom: Synchronized
tracking_active: Event
motor_stopped: Event motor_stopped: Event
reset: Event reset: Event
def __init__(self) -> None: def __init__(self, *, autotracker_enabled: bool):
self.start_time = mp.Value("d", 0) self.autotracker_enabled = mp.Value("i", autotracker_enabled) # type: ignore[assignment]
self.stop_time = mp.Value("d", 0)
self.frame_time = mp.Value("d", 0)
self.zoom_level = mp.Value("d", 0)
self.max_zoom = mp.Value("d", 0)
self.min_zoom = mp.Value("d", 0)
self.start_time = mp.Value("d", 0) # type: ignore[assignment]
self.stop_time = mp.Value("d", 0) # type: ignore[assignment]
self.frame_time = mp.Value("d", 0) # type: ignore[assignment]
self.zoom_level = mp.Value("d", 0) # type: ignore[assignment]
self.max_zoom = mp.Value("d", 0) # type: ignore[assignment]
self.min_zoom = mp.Value("d", 0) # type: ignore[assignment]
self.tracking_active = mp.Event()
self.motor_stopped = mp.Event() self.motor_stopped = mp.Event()
self.reset = mp.Event() self.reset = mp.Event()
+4 -6
View File
@@ -104,13 +104,12 @@ class CameraActivityManager:
all_objects: list[dict[str, Any]] = [] all_objects: list[dict[str, Any]] = []
for camera in new_activity.keys(): for camera in new_activity.keys():
camera_config = self.config.cameras.get(camera) if camera not in self.config.cameras:
if camera_config is None:
continue continue
# handle cameras that were added dynamically # handle cameras that were added dynamically
if camera not in self.camera_all_object_counts: if camera not in self.camera_all_object_counts:
self.__init_camera(camera_config) self.__init_camera(self.config.cameras[camera])
new_objects = new_activity[camera].get("objects", []) new_objects = new_activity[camera].get("objects", [])
all_objects.extend(new_objects) all_objects.extend(new_objects)
@@ -235,13 +234,12 @@ class AudioActivityManager:
now = datetime.datetime.now().timestamp() now = datetime.datetime.now().timestamp()
for camera in new_activity.keys(): for camera in new_activity.keys():
camera_config = self.config.cameras.get(camera) if camera not in self.config.cameras:
if camera_config is None:
continue continue
# handle cameras that were added dynamically # handle cameras that were added dynamically
if camera not in self.current_audio_detections: if camera not in self.current_audio_detections:
self.__init_camera(camera_config) self.__init_camera(self.config.cameras[camera])
new_detections = new_activity[camera].get("detections", []) new_detections = new_activity[camera].get("detections", [])
if self.compare_audio_activity(camera, new_detections, now): if self.compare_audio_activity(camera, new_detections, now):
+5 -6
View File
@@ -15,6 +15,7 @@ from frigate.config.camera.updater import (
CameraConfigUpdateSubscriber, CameraConfigUpdateSubscriber,
) )
from frigate.const import REPLAY_CAMERA_PREFIX from frigate.const import REPLAY_CAMERA_PREFIX
from frigate.detectors.detector_config import SceneEnum
from frigate.models import Regions from frigate.models import Regions
from frigate.object_detection.util import detection_frame_size from frigate.object_detection.util import detection_frame_size
from frigate.util.builtin import empty_and_close_queue from frigate.util.builtin import empty_and_close_queue
@@ -30,7 +31,7 @@ class CameraMaintainer(threading.Thread):
def __init__( def __init__(
self, self,
config: FrigateConfig, config: FrigateConfig,
detection_queues: dict[str, Queue], detection_queues: dict[SceneEnum, Queue],
detected_frames_queue: Queue, detected_frames_queue: Queue,
camera_metrics: DictProxy, camera_metrics: DictProxy,
ptz_metrics: dict[str, PTZMetrics], ptz_metrics: dict[str, PTZMetrics],
@@ -120,7 +121,9 @@ class CameraMaintainer(threading.Thread):
if runtime: if runtime:
self.camera_metrics[name] = CameraMetrics(self.metrics_manager) self.camera_metrics[name] = CameraMetrics(self.metrics_manager)
self.ptz_metrics[name] = PTZMetrics() self.ptz_metrics[name] = PTZMetrics(
autotracker_enabled=config.onvif.autotracking.enabled
)
self.region_grids[name] = get_camera_regions_grid( self.region_grids[name] = get_camera_regions_grid(
name, name,
config.detect, config.detect,
@@ -169,10 +172,6 @@ class CameraMaintainer(threading.Thread):
frame_size = config.frame_shape_yuv[0] * config.frame_shape_yuv[1] frame_size = config.frame_shape_yuv[0] * config.frame_shape_yuv[1]
self.frame_manager.create(f"{config.name}_frame{i}", frame_size) self.frame_manager.create(f"{config.name}_frame{i}", frame_size)
# a leftover ledger from an unclean exit may be the wrong length
self.frame_manager.delete(f"{config.name}_frame_times")
self.frame_manager.create(f"{config.name}_frame_times", max(count, 1) * 8)
capture_process = CameraCapture( capture_process = CameraCapture(
config, config,
count, count,
+21 -14
View File
@@ -16,7 +16,7 @@ from frigate.config import (
ZoomingModeEnum, ZoomingModeEnum,
) )
from frigate.const import CLIPS_DIR, THUMB_DIR from frigate.const import CLIPS_DIR, THUMB_DIR
from frigate.ptz.autotrack import PtzAutoTracker, calculate_max_target_box from frigate.ptz.autotrack import PtzAutoTrackerThread
from frigate.track.tracked_object import TrackedObject from frigate.track.tracked_object import TrackedObject
from frigate.util.image import ( from frigate.util.image import (
SharedMemoryFrameManager, SharedMemoryFrameManager,
@@ -35,7 +35,7 @@ class CameraState:
name: str, name: str,
config: FrigateConfig, config: FrigateConfig,
frame_manager: SharedMemoryFrameManager, frame_manager: SharedMemoryFrameManager,
ptz_autotracker_thread: PtzAutoTracker, ptz_autotracker_thread: PtzAutoTrackerThread,
) -> None: ) -> None:
self.name = name self.name = name
self.config = config self.config = config
@@ -115,13 +115,17 @@ class CameraState:
# draw thicker box around ptz autotracked object # draw thicker box around ptz autotracked object
if ( if (
self.camera_config.onvif.autotracking.enabled self.camera_config.onvif.autotracking.enabled
and self.ptz_autotracker_thread.autotracker_init.get(self.name) and self.ptz_autotracker_thread.ptz_autotracker.autotracker_init.get(
and self.ptz_autotracker_thread.tracked_object[self.name] self.name
)
and self.ptz_autotracker_thread.ptz_autotracker.tracked_object[
self.name
]
is not None is not None
and obj["id"] and obj["id"]
== self.ptz_autotracker_thread.tracked_object[ # type: ignore[union-attr] == self.ptz_autotracker_thread.ptz_autotracker.tracked_object[
self.name self.name
].obj_data["id"] ].obj_data["id"] # type: ignore[attr-defined]
and obj["frame_time"] == frame_time and obj["frame_time"] == frame_time
): ):
thickness = 5 thickness = 5
@@ -134,9 +138,9 @@ class CameraState:
and self.camera_config.detect.width is not None and self.camera_config.detect.width is not None
and self.camera_config.detect.height is not None and self.camera_config.detect.height is not None
): ):
max_target_box = calculate_max_target_box( max_target_box = self.ptz_autotracker_thread.ptz_autotracker.tracked_object_metrics[
self.camera_config.onvif.autotracking.zoom_factor self.name
) ]["max_target_box"] # type: ignore[index]
side_length = max_target_box * ( side_length = max_target_box * (
max( max(
self.camera_config.detect.width, self.camera_config.detect.width,
@@ -349,7 +353,7 @@ class CameraState:
return return
current_frame = self.frame_manager.get( current_frame = self.frame_manager.get(
frame_name, self.camera_config.frame_shape_yuv, frame_time frame_name, self.camera_config.frame_shape_yuv
) )
tracked_objects = self.tracked_objects.copy() tracked_objects = self.tracked_objects.copy()
@@ -404,7 +408,7 @@ class CameraState:
self.send_mqtt_snapshot(new_obj, object_type) self.send_mqtt_snapshot(new_obj, object_type)
for c in self.callbacks["start"]: for c in self.callbacks["start"]:
c(self.name, new_obj, frame_name, frame_time) c(self.name, new_obj, frame_name)
for id in updated_ids: for id in updated_ids:
updated_obj = tracked_objects[id] updated_obj = tracked_objects[id]
@@ -416,7 +420,7 @@ class CameraState:
if autotracker_update or significant_update: if autotracker_update or significant_update:
for c in self.callbacks["autotrack"]: for c in self.callbacks["autotrack"]:
c(self.name, updated_obj, frame_name, frame_time) c(self.name, updated_obj, frame_name)
if thumb_update and current_frame is not None: if thumb_update and current_frame is not None:
# ensure this frame is stored in the cache # ensure this frame is stored in the cache
@@ -466,7 +470,7 @@ class CameraState:
): ):
# call event handlers # call event handlers
for c in self.callbacks["update"]: for c in self.callbacks["update"]:
c(self.name, updated_obj, frame_name, frame_time) c(self.name, updated_obj, frame_name)
updated_obj.last_published = frame_time updated_obj.last_published = frame_time
# send MQTT snapshot when object first enters a required zone, # send MQTT snapshot when object first enters a required zone,
@@ -488,7 +492,7 @@ class CameraState:
removed_obj.obj_data["end_time"] = frame_time removed_obj.obj_data["end_time"] = frame_time
logger.debug(f"{self.name}: end callback for object {id}") logger.debug(f"{self.name}: end callback for object {id}")
for c in self.callbacks["end"]: for c in self.callbacks["end"]:
c(self.name, removed_obj, frame_name, frame_time) c(self.name, removed_obj, frame_name)
# TODO: can i switch to looking this up and only changing when an event ends? # TODO: can i switch to looking this up and only changing when an event ends?
# maintain best objects # maintain best objects
@@ -603,6 +607,9 @@ class CameraState:
self.current_frame_time = frame_time self.current_frame_time = frame_time
self._current_frame = np.copy(current_frame) self._current_frame = np.copy(current_frame)
if self.previous_frame_id is not None:
self.frame_manager.close(self.previous_frame_id)
self.previous_frame_id = frame_name self.previous_frame_id = frame_name
def send_mqtt_snapshot(self, new_obj: TrackedObject, object_type: str) -> None: def send_mqtt_snapshot(self, new_obj: TrackedObject, object_type: str) -> None:
-1
View File
@@ -12,7 +12,6 @@ class DetectionTypeEnum(str, Enum):
video = "video" video = "video"
audio = "audio" audio = "audio"
lpr = "lpr" lpr = "lpr"
classification_state = "classification_state"
class DetectionPublisher(Publisher): class DetectionPublisher(Publisher):
+7 -11
View File
@@ -759,13 +759,15 @@ class Dispatcher:
"Autotracking must be enabled in the config to be turned on via MQTT." "Autotracking must be enabled in the config to be turned on via MQTT."
) )
return return
if not ptz_autotracker_settings.enabled: if not self.ptz_metrics[camera_name].autotracker_enabled.value:
logger.info(f"Turning on ptz autotracker for {camera_name}") logger.info(f"Turning on ptz autotracker for {camera_name}")
self.ptz_metrics[camera_name].autotracker_enabled.value = True
self.ptz_metrics[camera_name].start_time.value = 0 self.ptz_metrics[camera_name].start_time.value = 0
ptz_autotracker_settings.enabled = True ptz_autotracker_settings.enabled = True
elif payload == "OFF": elif payload == "OFF":
if ptz_autotracker_settings.enabled: if self.ptz_metrics[camera_name].autotracker_enabled.value:
logger.info(f"Turning off ptz autotracker for {camera_name}") logger.info(f"Turning off ptz autotracker for {camera_name}")
self.ptz_metrics[camera_name].autotracker_enabled.value = False
self.ptz_metrics[camera_name].start_time.value = 0 self.ptz_metrics[camera_name].start_time.value = 0
ptz_autotracker_settings.enabled = False ptz_autotracker_settings.enabled = False
@@ -780,9 +782,7 @@ class Dispatcher:
try: try:
payload = int(payload) payload = int(payload)
except ValueError: except ValueError:
logger.warning( f"Received unsupported value for motion contour area: {payload}"
f"Received unsupported value for motion contour area: {payload}"
)
return return
motion_settings = self.config.cameras[camera_name].motion motion_settings = self.config.cameras[camera_name].motion
@@ -799,9 +799,7 @@ class Dispatcher:
try: try:
payload = int(payload) payload = int(payload)
except ValueError: except ValueError:
logger.warning( f"Received unsupported value for motion threshold: {payload}"
f"Received unsupported value for motion threshold: {payload}"
)
return return
motion_settings = self.config.cameras[camera_name].motion motion_settings = self.config.cameras[camera_name].motion
@@ -816,9 +814,7 @@ class Dispatcher:
def _on_global_notification_command(self, payload: str) -> None: def _on_global_notification_command(self, payload: str) -> None:
"""Callback for global notification topic.""" """Callback for global notification topic."""
if payload != "ON" and payload != "OFF": if payload != "ON" and payload != "OFF":
logger.warning( f"Received unsupported value for all notification: {payload}"
f"Received unsupported value for all notification: {payload}"
)
return return
notification_settings = self.config.notifications notification_settings = self.config.notifications
+2 -18
View File
@@ -8,16 +8,7 @@ from .zmq_proxy import Publisher, Subscriber
class EventUpdatePublisher( class EventUpdatePublisher(
Publisher[ Publisher[tuple[EventTypeEnum, EventStateEnum, str | None, str, dict[str, Any]]]
tuple[
EventTypeEnum,
EventStateEnum,
str | None,
str,
dict[str, Any],
float | None,
]
]
): ):
"""Publishes events (objects, audio, manual).""" """Publishes events (objects, audio, manual)."""
@@ -28,14 +19,7 @@ class EventUpdatePublisher(
def publish( def publish(
self, self,
payload: tuple[ payload: tuple[EventTypeEnum, EventStateEnum, str | None, str, dict[str, Any]],
EventTypeEnum,
EventStateEnum,
str | None,
str,
dict[str, Any],
float | None,
],
sub_topic: str = "", sub_topic: str = "",
) -> None: ) -> None:
super().publish(payload, sub_topic) super().publish(payload, sub_topic)
+3 -58
View File
@@ -2,8 +2,6 @@ from __future__ import annotations
import logging import logging
import queue import queue
import selectors
import socket
import threading import threading
import time import time
from collections.abc import Callable from collections.abc import Callable
@@ -58,11 +56,6 @@ class MqttClient(Communicator):
self._next_connect_time = 0.0 self._next_connect_time = 0.0
self._last_on_connect_dispatch = 0.0 self._last_on_connect_dispatch = 0.0
# lets other threads interrupt the worker's socket wait
self._wake_recv, self._wake_send = socket.socketpair()
self._wake_recv.setblocking(False)
self._wake_send.setblocking(False)
def subscribe(self, receiver: Callable) -> None: def subscribe(self, receiver: Callable) -> None:
"""Wrapper for allowing dispatcher to subscribe.""" """Wrapper for allowing dispatcher to subscribe."""
self._dispatcher = receiver self._dispatcher = receiver
@@ -92,7 +85,6 @@ class MqttClient(Communicator):
return return
self._publish_queue.put(QueuedPublish(full_topic, payload, retain)) self._publish_queue.put(QueuedPublish(full_topic, payload, retain))
self._wake_worker()
def stop(self) -> None: def stop(self) -> None:
if self._worker is None: if self._worker is None:
@@ -108,11 +100,9 @@ class MqttClient(Communicator):
publish_done, publish_done,
) )
) )
self._wake_worker()
publish_done.wait(MQTT_SHUTDOWN_FLUSH_TIMEOUT) publish_done.wait(MQTT_SHUTDOWN_FLUSH_TIMEOUT)
self._stop_event.set() self._stop_event.set()
self._wake_worker()
if self.client is not None: if self.client is not None:
try: try:
@@ -368,7 +358,7 @@ class MqttClient(Communicator):
deadline = time.monotonic() + MQTT_SHUTDOWN_FLUSH_TIMEOUT deadline = time.monotonic() + MQTT_SHUTDOWN_FLUSH_TIMEOUT
while not message_info.is_published() and time.monotonic() < deadline: while not message_info.is_published() and time.monotonic() < deadline:
if ( if (
self._loop_client(timeout=MQTT_PUBLISH_WAIT_INTERVAL) self.client.loop(timeout=MQTT_PUBLISH_WAIT_INTERVAL)
!= mqtt.MQTT_ERR_SUCCESS != mqtt.MQTT_ERR_SUCCESS
): ):
break break
@@ -378,51 +368,6 @@ class MqttClient(Communicator):
exc_info=True, exc_info=True,
) )
def _wake_worker(self) -> None:
try:
self._wake_send.send(b"\0")
except BlockingIOError:
# the buffer is full, so a wake is already pending
pass
def _loop_client(self, timeout: float) -> int:
"""Drive Paho without select()'s limit on socket file descriptors."""
assert self.client is not None
client = self.client
sock = client.socket()
if sock is None:
return mqtt.MQTT_ERR_NO_CONN
events = selectors.EVENT_READ
if client.want_write():
events |= selectors.EVENT_WRITE
# TLS can have decrypted bytes buffered even when the socket is not ready.
pending = hasattr(sock, "pending") and sock.pending() > 0
with selectors.DefaultSelector() as selector:
selector.register(sock, events)
selector.register(self._wake_recv, selectors.EVENT_READ)
ready = {
key.fileobj: mask
for key, mask in selector.select(0.0 if pending else timeout)
}
if self._wake_recv in ready:
try:
self._wake_recv.recv(4096)
except BlockingIOError:
pass
ready_events = ready.get(sock, 0)
if pending or ready_events & selectors.EVENT_READ:
result = client.loop_read()
if result != mqtt.MQTT_ERR_SUCCESS or client.socket() is None:
return result
if ready_events & selectors.EVENT_WRITE:
result = client.loop_write()
if result != mqtt.MQTT_ERR_SUCCESS or client.socket() is None:
return result
return client.loop_misc()
def _mqtt_loop_worker(self) -> None: def _mqtt_loop_worker(self) -> None:
# The worker owns all socket I/O so reconnect, subscribe, and publish # The worker owns all socket I/O so reconnect, subscribe, and publish
# ordering stays serialized in one place. # ordering stays serialized in one place.
@@ -439,7 +384,7 @@ class MqttClient(Communicator):
assert self.client is not None assert self.client is not None
try: try:
result = self._loop_client(timeout=MQTT_LOOP_TIMEOUT) result = self.client.loop(timeout=MQTT_LOOP_TIMEOUT)
except (OSError, mqtt.WebsocketConnectionError) as err: except (OSError, mqtt.WebsocketConnectionError) as err:
logger.warning("MQTT loop error: %s", err) logger.warning("MQTT loop error: %s", err)
self._schedule_reconnect() self._schedule_reconnect()
@@ -672,7 +617,7 @@ class MqttClient(Communicator):
return return
try: try:
result = self._loop_client(timeout=MQTT_PUBLISH_WAIT_INTERVAL) result = self.client.loop(timeout=MQTT_PUBLISH_WAIT_INTERVAL)
except (OSError, mqtt.WebsocketConnectionError) as err: except (OSError, mqtt.WebsocketConnectionError) as err:
logger.warning("MQTT publish wait failed: %s", err) logger.warning("MQTT publish wait failed: %s", err)
self._schedule_reconnect() self._schedule_reconnect()
+2 -2
View File
@@ -6,7 +6,7 @@ import logging
import os import os
import queue import queue
import threading import threading
from collections.abc import Callable, Iterator from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from multiprocessing.synchronize import Event as MpEvent from multiprocessing.synchronize import Event as MpEvent
from typing import Any from typing import Any
@@ -72,7 +72,7 @@ class WebPushClient(Communicator):
# Pull keys from PEM or generate if they do not exist # Pull keys from PEM or generate if they do not exist
self.vapid = Vapid01.from_file(os.path.join(CONFIG_DIR, "notifications.pem")) self.vapid = Vapid01.from_file(os.path.join(CONFIG_DIR, "notifications.pem"))
users: Iterator[dict[str, Any]] = ( users: list[dict[str, Any]] = (
User.select(User.username, User.notification_tokens).dicts().iterator() User.select(User.username, User.notification_tokens).dicts().iterator()
) )
for user in users: for user in users:
+4 -82
View File
@@ -3,9 +3,6 @@
import errno import errno
import json import json
import logging import logging
import queue
import socket
import struct
import threading import threading
from collections.abc import Callable from collections.abc import Callable
from typing import Any from typing import Any
@@ -79,12 +76,6 @@ _WS_VIEWER_TOPICS = frozenset(
# Camera-scoped command topics a camera-authorized (non-admin) user may send. # Camera-scoped command topics a camera-authorized (non-admin) user may send.
_WS_CAMERA_COMMAND_TOPICS = frozenset({"ptz"}) _WS_CAMERA_COMMAND_TOPICS = frozenset({"ptz"})
# Max outbound messages waiting on a client's writer thread.
WS_MAX_PENDING_MESSAGES = 256
# Seconds a single send() may wait for a client to free socket buffer space.
WS_SEND_TIMEOUT = 10
def _check_ws_authorization( def _check_ws_authorization(
topic: str, topic: str,
@@ -473,78 +464,6 @@ def _materialize_for_ws(
class WebSocket(WebSocket_): # type: ignore[misc] class WebSocket(WebSocket_): # type: ignore[misc]
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, **kwargs)
self._send_queue: queue.Queue[tuple[Any, bool] | None] = queue.Queue(
maxsize=WS_MAX_PENDING_MESSAGES
)
self._writer: threading.Thread | None = None
self._aborted = False
def opened(self) -> None:
"""Set a send timeout on the socket and start this client's writer thread."""
# ws4py writes close and pong frames on the shared manager thread, so a
# stalled client must not block sendall forever. SO_SNDTIMEO leaves recv
# (and idle clients) untouched, unlike settimeout().
self.sock.setsockopt(
socket.SOL_SOCKET, socket.SO_SNDTIMEO, struct.pack("ll", WS_SEND_TIMEOUT, 0)
)
# every client gets its own writer so a client that stops reading only
# blocks itself, never the thread that called publish()
self._writer = threading.Thread(
target=self._drain_send_queue, name="ws_writer", daemon=True
)
self._writer.start()
def send(self, payload: Any, binary: bool = False) -> None:
"""Queue a message for this client's writer thread.
Returns without waiting for delivery. Disconnects the client if its queue
is full.
"""
try:
self._send_queue.put_nowait((payload, binary))
except queue.Full:
self._abort("Websocket client is not keeping up, disconnecting it")
def _drain_send_queue(self) -> None:
while True:
item = self._send_queue.get()
if item is None or self.terminated or self.sock is None:
return
try:
super().send(*item)
except Exception:
self._abort()
return
def _abort(self, reason: str | None = None) -> None:
# publish() keeps hitting a full queue until the manager thread removes
# the connection, so only act (and log) the first time
if self._aborted:
return
self._aborted = True
if reason:
logger.warning(reason)
# shutdown rather than close so the ws4py manager thread sees EOF and
# runs its normal unregister/terminate; this also unblocks a stuck sendall
sock = self.sock
if sock is not None:
try:
sock.shutdown(socket.SHUT_RDWR)
except OSError:
pass
def closed(self, code: int, reason: str | None = None) -> None:
"""Signal the writer thread to exit."""
try:
self._send_queue.put_nowait(None)
except queue.Full:
pass
def unhandled_error(self, error: Any) -> None: def unhandled_error(self, error: Any) -> None:
""" """
Handles the unfriendly socket closures on the server side Handles the unfriendly socket closures on the server side
@@ -678,7 +597,10 @@ class WebSocketClient(Communicator):
) )
if message is None: if message is None:
continue continue
ws.send(message) try:
ws.send(message)
except (ConnectionResetError, BrokenPipeError, ValueError):
pass
def stop(self) -> None: def stop(self) -> None:
if self.websocket_server is not None: if self.websocket_server is not None:
+4 -5
View File
@@ -1,6 +1,6 @@
from pydantic import Field, model_validator from pydantic import Field, model_validator
from frigate.detectors.detector_config import DEFAULT_SCENE, SCENE_PATTERN from frigate.detectors.detector_config import SceneEnum
from ..base import FrigateBaseModel from ..base import FrigateBaseModel
@@ -62,11 +62,10 @@ class DetectConfig(FrigateBaseModel):
title="Detect width", title="Detect width",
description="Width (pixels) of frames used for the detect stream; leave empty to use the native stream resolution.", description="Width (pixels) of frames used for the detect stream; leave empty to use the native stream resolution.",
) )
scene: str = Field( scene: SceneEnum = Field(
default=DEFAULT_SCENE, default=SceneEnum.all,
pattern=SCENE_PATTERN,
title="Detect scene", title="Detect scene",
description="The environment this camera looks at, used to pick which of the configured models runs on it. Cameras left on 'default' run the model configured with a scene of 'default'.", description="The environment this camera looks at, used to pick which of the configured models runs on it. Cameras left on 'all' run the model configured with a scene of 'all'.",
) )
fps: int = Field( fps: int = Field(
default=5, default=5,
+1 -1
View File
@@ -3,9 +3,9 @@ from enum import Enum
from pydantic import Field, field_validator from pydantic import Field, field_validator
from frigate.util.config import resolve_ffmpeg_path from frigate.util.config import resolve_ffmpeg_path
from frigate.util.env import EnvString
from ..base import FrigateBaseModel from ..base import FrigateBaseModel
from ..env import EnvString
__all__ = [ __all__ = [
"CameraFfmpegConfig", "CameraFfmpegConfig",
+4 -30
View File
@@ -1,11 +1,10 @@
from enum import Enum from enum import Enum
from typing import Any, Self from typing import Any
from pydantic import Field, model_validator from pydantic import Field
from frigate.util.env import EnvString
from ..base import FrigateBaseModel from ..base import FrigateBaseModel
from ..env import EnvString
__all__ = ["GenAIConfig", "GenAIProviderEnum", "GenAIRoleEnum"] __all__ = ["GenAIConfig", "GenAIProviderEnum", "GenAIRoleEnum"]
@@ -22,17 +21,6 @@ class GenAIRoleEnum(str, Enum):
chat = "chat" chat = "chat"
descriptions = "descriptions" descriptions = "descriptions"
embeddings = "embeddings" embeddings = "embeddings"
transcribe = "transcribe"
# Providers that can accept audio input for the transcribe role. Ollama has no
# audio input support, so claiming the role there would fail at request time.
TRANSCRIBE_CAPABLE_PROVIDERS = {
GenAIProviderEnum.openai,
GenAIProviderEnum.azure_openai,
GenAIProviderEnum.gemini,
GenAIProviderEnum.llamacpp,
}
class GenAIConfig(FrigateBaseModel): class GenAIConfig(FrigateBaseModel):
@@ -64,7 +52,7 @@ class GenAIConfig(FrigateBaseModel):
GenAIRoleEnum.chat, GenAIRoleEnum.chat,
], ],
title="Roles", title="Roles",
description="GenAI roles (chat, descriptions, embeddings, transcribe); one provider per role. Only chat, descriptions, and embeddings are granted by default; transcribe must be listed explicitly.", description="GenAI roles (chat, descriptions, embeddings); one provider per role.",
) )
provider_options: dict[str, Any] = Field( provider_options: dict[str, Any] = Field(
default={}, default={},
@@ -78,17 +66,3 @@ class GenAIConfig(FrigateBaseModel):
description="Runtime options passed to the provider for each inference call.", description="Runtime options passed to the provider for each inference call.",
json_schema_extra={"additionalProperties": {}}, json_schema_extra={"additionalProperties": {}},
) )
@model_validator(mode="after")
def validate_transcribe_provider(self) -> Self:
"""Reject the transcribe role on providers that cannot accept audio input."""
if (
GenAIRoleEnum.transcribe in self.roles
and self.provider not in TRANSCRIBE_CAPABLE_PROVIDERS
):
raise ValueError(
f"GenAI provider '{self.provider.value}' does not support audio input "
"and cannot be given the 'transcribe' role."
)
return self
+2 -56
View File
@@ -1,57 +1,8 @@
from pydantic import Field, field_validator from pydantic import Field
from frigate.util.live_streams import DEFAULT_TRANSCODE_QUALITIES
from ..base import FrigateBaseModel from ..base import FrigateBaseModel
__all__ = ["CameraLiveConfig", "LiveTranscodeConfig", "LiveTranscodeQualityConfig"] __all__ = ["CameraLiveConfig"]
class LiveTranscodeQualityConfig(FrigateBaseModel):
height: int = Field(
ge=144,
le=2160,
title="Height",
description="Output height in pixels; width follows the source aspect ratio.",
)
bitrate: int = Field(
ge=64,
title="Bitrate",
description="Target and maximum video bitrate in kbps.",
)
class LiveTranscodeConfig(FrigateBaseModel):
enabled: bool = Field(
default=False,
title="Enable transcoded streams",
description="Add lower-quality live streams that go2rtc transcodes in real time while someone is watching.",
)
source: str | None = Field(
default=None,
title="Source stream",
description="go2rtc stream to transcode. Defaults to the first live stream.",
)
qualities: list[LiveTranscodeQualityConfig] = Field(
default_factory=lambda: [
LiveTranscodeQualityConfig(**quality)
for quality in DEFAULT_TRANSCODE_QUALITIES
],
title="Qualities",
description="One transcoded stream is added per quality.",
)
@field_validator("qualities")
@classmethod
def validate_unique_heights(
cls, qualities: list[LiveTranscodeQualityConfig]
) -> list[LiveTranscodeQualityConfig]:
heights = [quality.height for quality in qualities]
if len(heights) != len(set(heights)):
raise ValueError("Transcoded stream heights must be unique.")
return qualities
class CameraLiveConfig(FrigateBaseModel): class CameraLiveConfig(FrigateBaseModel):
@@ -60,11 +11,6 @@ class CameraLiveConfig(FrigateBaseModel):
title="Live stream names", title="Live stream names",
description="Mapping of configured stream names to restream/go2rtc names used for live playback.", description="Mapping of configured stream names to restream/go2rtc names used for live playback.",
) )
transcode: LiveTranscodeConfig = Field(
default_factory=LiveTranscodeConfig,
title="Transcoded streams",
description="Lower-quality live streams transcoded on demand by go2rtc.",
)
height: int = Field( height: int = Field(
default=720, default=720,
title="Live height", title="Live height",
+1 -3
View File
@@ -1,7 +1,5 @@
from pydantic import Field from pydantic import Field
from frigate.util.env import EnvString
from ..base import FrigateBaseModel from ..base import FrigateBaseModel
__all__ = ["NotificationConfig"] __all__ = ["NotificationConfig"]
@@ -13,7 +11,7 @@ class NotificationConfig(FrigateBaseModel):
title="Enable notifications", title="Enable notifications",
description="Enable or disable notifications for all cameras; can be overridden per-camera.", description="Enable or disable notifications for all cameras; can be overridden per-camera.",
) )
email: EnvString | None = Field( email: str | None = Field(
default=None, default=None,
title="Notification email", title="Notification email",
description="Email address used for push notifications or required by certain notification providers.", description="Email address used for push notifications or required by certain notification providers.",
+1 -2
View File
@@ -2,9 +2,8 @@ from enum import Enum
from pydantic import Field, field_validator from pydantic import Field, field_validator
from frigate.util.env import EnvString
from ..base import FrigateBaseModel from ..base import FrigateBaseModel
from ..env import EnvString
from .objects import DEFAULT_TRACKED_OBJECTS from .objects import DEFAULT_TRACKED_OBJECTS
__all__ = ["OnvifConfig", "PtzAutotrackConfig", "ZoomingModeEnum"] __all__ = ["OnvifConfig", "PtzAutotrackConfig", "ZoomingModeEnum"]
-13
View File
@@ -9,7 +9,6 @@ __all__ = [
"DetectionsConfig", "DetectionsConfig",
"AlertsConfig", "AlertsConfig",
"ImageSourceEnum", "ImageSourceEnum",
"ReviewFrameModeEnum",
"ReviewResponseStyleEnum", "ReviewResponseStyleEnum",
] ]
@@ -21,13 +20,6 @@ class ImageSourceEnum(str, Enum):
recordings = "recordings" recordings = "recordings"
class ReviewFrameModeEnum(str, Enum):
"""How review frames are presented to the GenAI provider."""
frames = "frames"
annotated_frames = "annotated_frames"
class ReviewResponseStyleEnum(str, Enum): class ReviewResponseStyleEnum(str, Enum):
"""Writing style presets for GenAI review descriptions.""" """Writing style presets for GenAI review descriptions."""
@@ -161,11 +153,6 @@ class GenAIReviewConfig(FrigateBaseModel):
description="Preferred language to request from the GenAI provider for generated responses.", description="Preferred language to request from the GenAI provider for generated responses.",
default=None, default=None,
) )
frame_mode: ReviewFrameModeEnum = Field(
default=ReviewFrameModeEnum.frames,
title="Frame mode",
description="How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses.",
)
response_style: ReviewResponseStyleEnum = Field( response_style: ReviewResponseStyleEnum = Field(
default=ReviewResponseStyleEnum.default, default=ReviewResponseStyleEnum.default,
title="Response style", title="Response style",
+2 -32
View File
@@ -5,7 +5,6 @@ from pydantic import ConfigDict, Field, field_validator
from .base import FrigateBaseModel from .base import FrigateBaseModel
__all__ = [ __all__ = [
"AudioTranscriptionModelEnum",
"CameraFaceRecognitionConfig", "CameraFaceRecognitionConfig",
"CameraLicensePlateRecognitionConfig", "CameraLicensePlateRecognitionConfig",
"CameraAudioTranscriptionConfig", "CameraAudioTranscriptionConfig",
@@ -21,10 +20,6 @@ class SemanticSearchModelEnum(str, Enum):
jinav2 = "jinav2" jinav2 = "jinav2"
class AudioTranscriptionModelEnum(str, Enum):
whisper = "whisper"
class EnrichmentsDeviceEnum(str, Enum): class EnrichmentsDeviceEnum(str, Enum):
GPU = "GPU" GPU = "GPU"
CPU = "CPU" CPU = "CPU"
@@ -58,35 +53,10 @@ class AudioTranscriptionConfig(FrigateBaseModel):
description="Enable or disable automatic audio transcription for all cameras; can be overridden per-camera.", description="Enable or disable automatic audio transcription for all cameras; can be overridden per-camera.",
) )
language: str = Field( language: str = Field(
default="auto", default="en",
title="Transcription language", title="Transcription language",
description="Language code used for transcription/translation (for example 'en' for English), or 'auto' to let the model detect it. See https://whisper-api.com/docs/languages/ for supported language codes.", description="Language code used for transcription/translation (for example 'en' for English). See https://whisper-api.com/docs/languages/ for supported language codes.",
) )
model: AudioTranscriptionModelEnum | str | None = Field(
default=AudioTranscriptionModelEnum.whisper,
title="Audio transcription model or GenAI provider name",
description="The transcription backend: 'whisper' for Frigate's built-in local models, or the name of a GenAI provider with the transcribe role.",
)
@field_validator("model", mode="before")
@classmethod
def coerce_model_enum(cls, v):
# An absent value ("model:" with nothing after it, or an explicit null)
# means unspecified, so fall back to the built-in backend. Left as None
# it would pass the GenAI-provider validation, which only inspects
# strings, and then be treated as a provider name that resolves to no
# client, turning transcription into a silent no-op.
if v is None or (isinstance(v, str) and not v.strip()):
return AudioTranscriptionModelEnum.whisper
if isinstance(v, str):
try:
return AudioTranscriptionModelEnum(v)
except ValueError:
return v
return v
device: EnrichmentsDeviceEnum = Field( device: EnrichmentsDeviceEnum = Field(
default=EnrichmentsDeviceEnum.CPU, default=EnrichmentsDeviceEnum.CPU,
title="Transcription device", title="Transcription device",
+24 -212
View File
@@ -19,7 +19,7 @@ from ruamel.yaml import YAML
from frigate.const import REGEX_JSON from frigate.const import REGEX_JSON
from frigate.detectors import ModelConfig from frigate.detectors import ModelConfig
from frigate.detectors.detector_config import DEFAULT_SCENE from frigate.detectors.detector_config import SceneEnum
from frigate.detectors.device import DeviceParseError, DeviceSpec, parse_device from frigate.detectors.device import DeviceParseError, DeviceSpec, parse_device
from frigate.plus import PlusApi from frigate.plus import PlusApi
from frigate.util.builtin import ( from frigate.util.builtin import (
@@ -34,15 +34,7 @@ from frigate.util.config import (
get_relative_coordinates, get_relative_coordinates,
migrate_frigate_config, migrate_frigate_config,
) )
from frigate.util.env import EnvVars, reload_sources
from frigate.util.image import create_mask from frigate.util.image import create_mask
from frigate.util.live_streams import (
default_transcode_source,
is_transcode_stream_name,
transcode_stream_name,
transcode_streams,
)
from frigate.util.runtime_deps import sha256_of
from frigate.util.services import auto_detect_hwaccel from frigate.util.services import auto_detect_hwaccel
from .auth import AuthConfig from .auth import AuthConfig
@@ -64,7 +56,6 @@ from .camera.timestamp import TimestampStyleConfig
from .camera_group import CameraGroupConfig from .camera_group import CameraGroupConfig
from .classification import ( from .classification import (
AudioTranscriptionConfig, AudioTranscriptionConfig,
AudioTranscriptionModelEnum,
ClassificationConfig, ClassificationConfig,
FaceRecognitionConfig, FaceRecognitionConfig,
LicensePlateRecognitionConfig, LicensePlateRecognitionConfig,
@@ -72,6 +63,7 @@ from .classification import (
SemanticSearchModelEnum, SemanticSearchModelEnum,
) )
from .database import DatabaseConfig from .database import DatabaseConfig
from .env import EnvVars, reload_sources
from .logger import LoggerConfig from .logger import LoggerConfig
from .mqtt import MqttConfig from .mqtt import MqttConfig
from .network import NetworkingConfig from .network import NetworkingConfig
@@ -289,78 +281,11 @@ def verify_config_roles(camera_config: CameraConfig) -> None:
) )
def apply_live_transcode_streams(
frigate_config: FrigateConfig, camera_config: CameraConfig
) -> None:
"""Fold a camera's transcoded streams into its live stream list.
Enabled qualities missing from live.streams are appended, and entries the
user placed keep their position. Transcoded names that are no longer
generated are dropped unless they name a real go2rtc stream.
"""
live = camera_config.live
transcode = live.transcode
go2rtc_streams = frigate_config.go2rtc.model_dump().get("streams") or {}
generated: dict[str, str] = {}
if transcode.enabled:
if transcode.source is None:
transcode.source = default_transcode_source(
camera_config.name, live.streams
)
if transcode.source not in go2rtc_streams:
raise ValueError(
f"Camera {camera_config.name} has transcoded streams enabled, but its source {transcode.source} is not a go2rtc stream."
)
generated = transcode_streams(
camera_config.name,
transcode.source,
[quality.model_dump() for quality in transcode.qualities],
)
for name in generated:
if name in go2rtc_streams:
raise ValueError(
f"Camera {camera_config.name} generates transcoded stream {name}, which collides with a go2rtc stream of the same name."
)
streams = {
label: name
for label, name in live.streams.items()
if name in generated
or name in go2rtc_streams
or not is_transcode_stream_name(camera_config.name, name)
}
placed = set(streams.values())
for quality in transcode.qualities if transcode.enabled else []:
name = transcode_stream_name(camera_config.name, quality.height)
if name in placed:
continue
label = f"{quality.height}p"
if label in streams:
raise ValueError(
f"Camera {camera_config.name} already has a live stream named {label}; rename it or place the transcoded stream under another name."
)
streams[label] = name
live.streams = streams
def verify_valid_live_stream_names( def verify_valid_live_stream_names(
frigate_config: FrigateConfig, camera_config: CameraConfig frigate_config: FrigateConfig, camera_config: CameraConfig
) -> ValueError | None: ) -> ValueError | None:
"""Verify that a restream exists to use for live view.""" """Verify that a restream exists to use for live view."""
for _, stream_name in camera_config.live.streams.items(): for _, stream_name in camera_config.live.streams.items():
if is_transcode_stream_name(camera_config.name, stream_name):
continue
if ( if (
stream_name stream_name
not in frigate_config.go2rtc.model_dump().get("streams", {}).keys() not in frigate_config.go2rtc.model_dump().get("streams", {}).keys()
@@ -610,7 +535,7 @@ class FrigateConfig(FrigateBaseModel):
models: list[ModelConfig] = Field( models: list[ModelConfig] = Field(
default_factory=_default_models, default_factory=_default_models,
title="Detection models", title="Detection models",
description="Object detection models and the hardware each one runs on. Cameras pick a model by matching their detect.scene against a model's scene, falling back to the 'default' model.", description="Object detection models and the hardware each one runs on. Cameras pick a model by matching their detect.scene against a model's scene.",
) )
# GenAI config (named provider configs: name -> GenAIConfig) # GenAI config (named provider configs: name -> GenAIConfig)
@@ -725,9 +650,7 @@ class FrigateConfig(FrigateBaseModel):
) )
_plus_api: PlusApi _plus_api: PlusApi
_model_devices: dict[str, list[DeviceSpec]] _model_devices: dict[SceneEnum, list[DeviceSpec]]
# scene -> model, including the scenes of duplicate models folded into another
_scene_models: dict[str, ModelConfig]
_camera_models: dict[str, ModelConfig] _camera_models: dict[str, ModelConfig]
_all_attributes: list[str] _all_attributes: list[str]
_all_attribute_logos: list[str] _all_attribute_logos: list[str]
@@ -762,7 +685,7 @@ class FrigateConfig(FrigateBaseModel):
def primary_model(self) -> ModelConfig: def primary_model(self) -> ModelConfig:
"""The model used when no specific camera is in play.""" """The model used when no specific camera is in play."""
for model in self.models: for model in self.models:
if model.scene == DEFAULT_SCENE: if model.scene == SceneEnum.all:
return model return model
return self.models[0] return self.models[0]
@@ -784,7 +707,7 @@ class FrigateConfig(FrigateBaseModel):
if model is None: if model is None:
camera = self.cameras.get(camera_name) camera = self.cameras.get(camera_name)
scene = camera.detect.scene if camera is not None else DEFAULT_SCENE scene = camera.detect.scene if camera is not None else SceneEnum.all
model = self._resolve_camera_model(camera_name, scene) model = self._resolve_camera_model(camera_name, scene)
self._camera_models[camera_name] = model self._camera_models[camera_name] = model
@@ -844,14 +767,14 @@ class FrigateConfig(FrigateBaseModel):
if not self.models: if not self.models:
raise ValueError("At least one model must be configured under models") raise ValueError("At least one model must be configured under models")
model_devices: dict[str, list[DeviceSpec]] = {} model_devices: dict[SceneEnum, list[DeviceSpec]] = {}
# device string -> the scene of the model that already claimed it # device string -> the scene of the model that already claimed it
claimed_devices: dict[str, str] = {} claimed_devices: dict[str, SceneEnum] = {}
for index, model in enumerate(self.models): for index, model in enumerate(self.models):
scene = model.scene scene = model.scene.value
if scene in model_devices: if model.scene in model_devices:
raise ValueError( raise ValueError(
f"Multiple models are configured with a scene of '{scene}'. Each model must use a different scene." f"Multiple models are configured with a scene of '{scene}'. Each model must use a different scene."
) )
@@ -880,20 +803,17 @@ class FrigateConfig(FrigateBaseModel):
other = claimed_devices[device.raw] other = claimed_devices[device.raw]
where = ( where = (
f"twice by model '{scene}'" f"twice by model '{scene}'"
if other == scene if other == model.scene
else f"by both the '{other}' and '{scene}' models" else f"by both the '{other.value}' and '{scene}' models"
) )
raise ValueError( raise ValueError(
f"Device '{device.raw}' is used {where}, but it can only run one detection process." f"Device '{device.raw}' is used {where}, but it can only run one detection process."
) )
claimed_devices[device.raw] = scene claimed_devices[device.raw] = model.scene
self.models[index] = self._load_model(model, devices[0].detector) self.models[index] = self._load_model(model, devices[0].detector)
model_devices[scene] = devices model_devices[model.scene] = devices
self._scene_models = {model.scene: model for model in self.models}
self._consolidate_duplicate_models(model_devices)
attributes: set[str] = set() attributes: set[str] = set()
attribute_logos: set[str] = set() attribute_logos: set[str] = set()
@@ -917,107 +837,36 @@ class FrigateConfig(FrigateBaseModel):
} }
self._all_labels = labels self._all_labels = labels
def _consolidate_duplicate_models( def _resolve_camera_model(self, name: str, scene: SceneEnum) -> ModelConfig:
self, model_devices: dict[str, list[DeviceSpec]]
) -> None:
"""Fold models that load the same model file into a single model.
Separate scenes for one model only split the same work across separate
detection queues, so each device serves fewer cameras and is slower
overall than one shared model. The duplicate's devices are moved to the
model it duplicates and its scene resolves to that model.
Args:
model_devices: Scene to parsed devices, updated in place
"""
kept: list[ModelConfig] = []
hashes: dict[str, str | None] = {}
def file_hash(path: str) -> str | None:
if path not in hashes:
hashes[path] = sha256_of(path) if os.path.isfile(path) else None
return hashes[path]
def same_model(a: ModelConfig, b: ModelConfig) -> bool:
# a model's devices all share a detector, so folding across
# detectors would produce an invalid model
if model_devices[a.scene][0].detector != model_devices[b.scene][0].detector:
return False
if not a.path or not b.path:
return False
if os.path.realpath(a.path) == os.path.realpath(b.path):
return True
a_hash = file_hash(a.path)
return a_hash is not None and a_hash == file_hash(b.path)
for model in self.models:
original = next((other for other in kept if same_model(other, model)), None)
if original is None:
kept.append(model)
continue
# keep the default model so cameras without a scene still find it
if model.scene == DEFAULT_SCENE:
kept[kept.index(original)] = model
original, model = model, original
logger.warning(
"Models '%s' and '%s' use the same model file, so they have been combined into the '%s' model. Defining one model under several scenes to assign detectors to specific cameras is slower and less efficient than letting every detector serve every camera. Remove the '%s' model and list its devices under the '%s' model instead",
original.scene,
model.scene,
original.scene,
model.scene,
original.scene,
)
original.devices = [*original.devices, *model.devices]
model_devices[original.scene] = [
*model_devices[original.scene],
*model_devices.pop(model.scene),
]
self._scene_models[model.scene] = original
# anything already folded into the duplicate follows it
for scene, target in self._scene_models.items():
if target is model:
self._scene_models[scene] = original
self.models = kept
def _resolve_camera_model(self, name: str, scene: str) -> ModelConfig:
"""Resolve which model a camera runs on. """Resolve which model a camera runs on.
A camera may name a scene no model is configured for, which is valid as A camera may name a scene no model is configured for, which is valid as
long as a 'default' model is there to fall back to. long as an 'all' model is there to fall back to.
Args: Args:
name: Name of the camera name: Name of the camera
scene: The camera's detect scene, which defaults to 'default' scene: The camera's detect scene, which defaults to 'all'
Returns: Returns:
The model the camera runs on The model the camera runs on
""" """
model = self._scene_models.get(scene) by_scene = {model.scene: model for model in self.models}
model = by_scene.get(scene)
if model is not None: if model is not None:
return model return model
default = self._scene_models.get(DEFAULT_SCENE) default = by_scene.get(SceneEnum.all)
if default is None: if default is None:
raise ValueError( raise ValueError(
f"Camera '{name}' has a detect scene of '{scene}', but no model is configured for that scene or for '{DEFAULT_SCENE}'." f"Camera '{name}' has a detect scene of '{scene.value}', but no model is configured for that scene or for 'all'."
) )
logger.warning( logger.warning(
"Camera '%s' has a detect scene of '%s', but no model is configured for that scene, so the '%s' model is used", "Camera '%s' has a detect scene of '%s', but no model is configured for that scene, so the 'all' model is used",
name, name,
scene, scene.value,
DEFAULT_SCENE,
) )
return default return default
@@ -1035,7 +884,7 @@ class FrigateConfig(FrigateBaseModel):
# set notifications state # set notifications state
self.notifications.enabled_in_config = self.notifications.enabled self.notifications.enabled_in_config = self.notifications.enabled
# validate genai: each role (chat, descriptions, embeddings, transcribe) at most once # validate genai: each role (chat, descriptions, embeddings) at most once
role_to_name: dict[GenAIRoleEnum, str] = {} role_to_name: dict[GenAIRoleEnum, str] = {}
for name, genai_cfg in self.genai.items(): for name, genai_cfg in self.genai.items():
for role in genai_cfg.roles: for role in genai_cfg.roles:
@@ -1128,8 +977,6 @@ class FrigateConfig(FrigateBaseModel):
"face_recognition": ["enabled", "min_area"], "face_recognition": ["enabled", "min_area"],
"lpr": ["enabled", "expire_time", "min_area", "enhancement"], "lpr": ["enabled", "expire_time", "min_area", "enhancement"],
"audio_transcription": ["enabled", "live_enabled"], "audio_transcription": ["enabled", "live_enabled"],
# transcode is camera-level only
"live": ["streams", "height", "quality"],
} }
for section in allowed_fields_map: for section in allowed_fields_map:
@@ -1150,10 +997,6 @@ class FrigateConfig(FrigateBaseModel):
camera_model = self._resolve_camera_model(name, camera_config.detect.scene) camera_model = self._resolve_camera_model(name, camera_config.detect.scene)
self._camera_models[name] = camera_model self._camera_models[name] = camera_model
# point cameras at the model their duplicate scene was folded into
if camera_config.detect.scene in self._scene_models:
camera_config.detect.scene = camera_model.scene
if camera_config.ffmpeg.hwaccel_args == "auto": if camera_config.ffmpeg.hwaccel_args == "auto":
camera_config.ffmpeg.hwaccel_args = self.ffmpeg.hwaccel_args camera_config.ffmpeg.hwaccel_args = self.ffmpeg.hwaccel_args
@@ -1362,8 +1205,6 @@ class FrigateConfig(FrigateBaseModel):
if not camera_config.live.streams: if not camera_config.live.streams:
camera_config.live.streams = {name: name} camera_config.live.streams = {name: name}
apply_live_transcode_streams(self, camera_config)
# generate the ffmpeg commands # generate the ffmpeg commands
camera_config.create_ffmpeg_cmds() camera_config.create_ffmpeg_cmds()
self.cameras[name] = camera_config self.cameras[name] = camera_config
@@ -1404,35 +1245,6 @@ class FrigateConfig(FrigateBaseModel):
for model in self.models: for model in self.models:
model.create_colormap(colored_labels) model.create_colormap(colored_labels)
# validate audio_transcription.model when it is a GenAI provider name.
# this runs here rather than beside the semantic_search check because the
# global->camera merge above is what resolves camera-level enablement.
transcription_active = self.audio_transcription.enabled or any(
camera.audio_transcription.enabled for camera in self.cameras.values()
)
if (
transcription_active
and isinstance(self.audio_transcription.model, str)
and not isinstance(
self.audio_transcription.model, AudioTranscriptionModelEnum
)
):
if self.audio_transcription.model not in self.genai:
raise ValueError(
f"audio_transcription.model '{self.audio_transcription.model}' is not a "
"valid GenAI config key. Must match a key in genai config."
)
if (
GenAIRoleEnum.transcribe
not in self.genai[self.audio_transcription.model].roles
):
raise ValueError(
f"GenAI provider '{self.audio_transcription.model}' must have "
"'transcribe' in its roles for audio transcription."
)
# Check audio transcription and audio detection requirements # Check audio transcription and audio detection requirements
if self.audio_transcription.enabled: if self.audio_transcription.enabled:
# If audio transcription is enabled globally, at least one camera must have audio detection enabled # If audio transcription is enabled globally, at least one camera must have audio detection enabled
+1 -1
View File
@@ -3,9 +3,9 @@ from typing import Self
from pydantic import Field, ValidationInfo, model_validator from pydantic import Field, ValidationInfo, model_validator
from frigate.const import FREQUENCY_STATS_POINTS from frigate.const import FREQUENCY_STATS_POINTS
from frigate.util.env import EnvString
from .base import FrigateBaseModel from .base import FrigateBaseModel
from .env import EnvString
__all__ = ["MqttConfig"] __all__ = ["MqttConfig"]
+1 -2
View File
@@ -1,8 +1,7 @@
from pydantic import Field, field_validator from pydantic import Field, field_validator
from frigate.util.env import EnvString
from .base import FrigateBaseModel from .base import FrigateBaseModel
from .env import EnvString
__all__ = ["ProxyConfig", "HeaderMappingConfig"] __all__ = ["ProxyConfig", "HeaderMappingConfig"]
-1
View File
@@ -8,7 +8,6 @@ MODEL_CACHE_DIR = f"{CONFIG_DIR}/model_cache"
BASE_DIR = "/media/frigate" BASE_DIR = "/media/frigate"
CLIPS_DIR = f"{BASE_DIR}/clips" CLIPS_DIR = f"{BASE_DIR}/clips"
EXPORT_DIR = f"{BASE_DIR}/exports" EXPORT_DIR = f"{BASE_DIR}/exports"
EXPORT_STAGING_DIR = f"{EXPORT_DIR}/staging"
FACE_DIR = f"{CLIPS_DIR}/faces" FACE_DIR = f"{CLIPS_DIR}/faces"
THUMB_DIR = f"{CLIPS_DIR}/thumbs" THUMB_DIR = f"{CLIPS_DIR}/thumbs"
RECORD_DIR = f"{BASE_DIR}/recordings" RECORD_DIR = f"{BASE_DIR}/recordings"
@@ -176,7 +176,6 @@ class LicensePlateProcessingMixin:
""" """
input_shape = [3, 48, 320] input_shape = [3, 48, 320]
num_images = len(images) num_images = len(images)
outputs: list[np.ndarray] = []
for index in range(0, num_images, self.batch_size): for index in range(0, num_images, self.batch_size):
input_h, input_w = input_shape[1], input_shape[2] input_h, input_w = input_shape[1], input_shape[2]
@@ -196,11 +195,11 @@ class LicensePlateProcessingMixin:
norm_image = norm_image[np.newaxis, :] norm_image = norm_image[np.newaxis, :]
norm_images.append(norm_image) norm_images.append(norm_image)
try: try:
outputs.extend(self.model_runner.recognition_model(norm_images)) # type: ignore[arg-type] outputs = self.model_runner.recognition_model(norm_images) # type: ignore[arg-type]
except Exception as e: except Exception as e:
logger.warning(f"Error running LPR recognition model: {e}") logger.warning(f"Error running LPR recognition model: {e}")
return [], [] return [], []
return self.ctc_decoder(outputs) return self.ctc_decoder(outputs)
@@ -580,8 +579,8 @@ class LicensePlateProcessingMixin:
boxes = [] boxes = []
scores = [] scores = []
for index in range(len(contours)): for index in range(len(contours)): # type: ignore[arg-type]
contour = contours[index] contour = contours[index] # type: ignore[index]
# get minimum bounding box (rotated rectangle) around the contour and the smallest side length. # get minimum bounding box (rotated rectangle) around the contour and the smallest side length.
points, sside = self._get_min_boxes(contour) points, sside = self._get_min_boxes(contour)
@@ -1222,7 +1221,7 @@ class LicensePlateProcessingMixin:
"""Look for license plates in image.""" """Look for license plates in image."""
self.metrics.alpr_pps.value = self.plates_rec_second.eps() self.metrics.alpr_pps.value = self.plates_rec_second.eps()
self.metrics.yolov9_lpr_pps.value = self.plates_det_second.eps() self.metrics.yolov9_lpr_pps.value = self.plates_det_second.eps()
camera: str = obj_data if dedicated_lpr else obj_data["camera"] # type: ignore[assignment] camera = obj_data if dedicated_lpr else obj_data["camera"]
current_time = int(datetime.datetime.now().timestamp()) current_time = int(datetime.datetime.now().timestamp())
debug_frame_id = int(datetime.datetime.now().timestamp() * 1000) debug_frame_id = int(datetime.datetime.now().timestamp() * 1000)
@@ -10,7 +10,6 @@ from peewee import DoesNotExist
from frigate.comms.inter_process import InterProcessRequestor from frigate.comms.inter_process import InterProcessRequestor
from frigate.config import FrigateConfig from frigate.config import FrigateConfig
from frigate.config.classification import AudioTranscriptionModelEnum
from frigate.const import ( from frigate.const import (
CACHE_DIR, CACHE_DIR,
MODEL_CACHE_DIR, MODEL_CACHE_DIR,
@@ -19,13 +18,8 @@ from frigate.const import (
) )
from frigate.data_processing.types import PostProcessDataEnum from frigate.data_processing.types import PostProcessDataEnum
from frigate.embeddings.embeddings import Embeddings from frigate.embeddings.embeddings import Embeddings
from frigate.genai.manager import GenAIClientManager
from frigate.types import TrackedObjectUpdateTypesEnum from frigate.types import TrackedObjectUpdateTypesEnum
from frigate.util.audio import ( from frigate.util.audio import get_audio_from_recording
clean_transcript,
get_audio_from_recording,
resolve_language,
)
from ..types import DataProcessorMetrics from ..types import DataProcessorMetrics
from .api import PostProcessorApi from .api import PostProcessorApi
@@ -40,25 +34,15 @@ class AudioTranscriptionPostProcessor(PostProcessorApi):
requestor: InterProcessRequestor, requestor: InterProcessRequestor,
embeddings: Embeddings, embeddings: Embeddings,
metrics: DataProcessorMetrics, metrics: DataProcessorMetrics,
genai_manager: GenAIClientManager | None = None,
): ):
super().__init__(config, metrics, None) super().__init__(config, metrics, None)
self.config = config self.config = config
self.requestor = requestor self.requestor = requestor
self.embeddings = embeddings self.embeddings = embeddings
self.genai_manager = genai_manager
self.recognizer = None self.recognizer = None
self.transcription_lock = threading.Lock() self.transcription_lock = threading.Lock()
self.transcription_thread: threading.Thread | None = None self.transcription_thread: threading.Thread | None = None
self.transcription_running = False self.transcription_running = False
self._use_genai = not isinstance(
config.audio_transcription.model, AudioTranscriptionModelEnum
)
if self._use_genai:
# never build the local recognizer on the GenAI path; WhisperModel
# downloads several hundred MB on first use
return
# faster-whisper handles model downloading automatically # faster-whisper handles model downloading automatically
self.model_path = os.path.join(MODEL_CACHE_DIR, "whisper") self.model_path = os.path.join(MODEL_CACHE_DIR, "whisper")
@@ -163,31 +147,6 @@ class AudioTranscriptionPostProcessor(PostProcessorApi):
logger.error(f"Error in audio transcription post-processing: {e}") logger.error(f"Error in audio transcription post-processing: {e}")
def __transcribe_audio(self, audio_data: bytes) -> str | None: def __transcribe_audio(self, audio_data: bytes) -> str | None:
"""Transcribe WAV audio data with the configured backend."""
if self._use_genai:
return self.__transcribe_audio_genai(audio_data)
return self.__transcribe_audio_whisper(audio_data)
def __transcribe_audio_genai(self, audio_data: bytes) -> str | None:
"""Hand the WAV bytes to the GenAI provider holding the transcribe role."""
client = self.genai_manager.transcribe_client if self.genai_manager else None
if not client:
logger.error(
"audio_transcription.model is '%s' (GenAI provider) but no transcribe "
"client is configured. Ensure the GenAI provider has 'transcribe' in its roles",
self.config.audio_transcription.model,
)
return None
text = client.transcribe(
audio_data,
language=resolve_language(self.config.audio_transcription.language),
)
return clean_transcript(text) or None
def __transcribe_audio_whisper(self, audio_data: bytes) -> str | None:
"""Transcribe WAV audio data using faster-whisper.""" """Transcribe WAV audio data using faster-whisper."""
if not self.recognizer: if not self.recognizer:
logger.debug("Recognizer not initialized") logger.debug("Recognizer not initialized")
@@ -201,7 +160,7 @@ class AudioTranscriptionPostProcessor(PostProcessorApi):
segments, info = self.recognizer.transcribe( segments, info = self.recognizer.transcribe(
temp_wav, temp_wav,
language=resolve_language(self.config.audio_transcription.language), language=self.config.audio_transcription.language,
beam_size=5, beam_size=5,
) )
@@ -1,416 +0,0 @@
"""Frame annotations derived from object tracking data.
Builds short notes describing what changed during a review item, keyed to the
frames sampled from it. Everything here comes from data already recorded
(each event's `path_data` trajectory, the timeline's stationary/active
changes, and the review item's state classification changes), so the notes
can be stated to the model as fact rather than as something it must perceive.
"""
import logging
import math
from collections.abc import Sequence
from typing import Any
from frigate.models import Event, Timeline
logger = logging.getLogger(__name__)
# Movement smaller than this (normalized frame units) between two path points
# is treated as the object holding still rather than travelling.
STILL_THRESHOLD = 0.02
# A heading change beyond this (dot product against the leg's own heading)
# counts as the object turning back rather than curving.
REVERSAL_DOT = -0.3
# A run of travel shorter than this (normalized frame units) is treated as
# milling about rather than going somewhere. Without it, a subject pacing in
# one spot produces a burst of contradictory "turns around" notes on a single
# frame.
MIN_LEG_DISTANCE = 0.08
# Movement that begins within this many seconds of detection is folded into
# the detection note, so each arrival reads as one event instead of several.
DETECT_MOVE_MERGE_SECONDS = 2.0
STATE_CHANGE_PHRASES = {
"stationary": "has stopped moving",
"active": "starts moving again",
}
Point = tuple[float, float, float]
Leg = tuple[int, int]
def describe_position(x: float, y: float) -> str:
"""Name a normalized frame position in plain terms."""
horizontal = "left" if x < 0.34 else ("right" if x > 0.66 else "center")
vertical = "top" if y < 0.34 else ("bottom" if y > 0.66 else "middle")
if horizontal == "center" and vertical == "middle":
return "the middle of the frame"
if horizontal == "center":
return f"the {vertical} of the frame"
if vertical == "middle":
return f"the {horizontal} of the frame"
return f"the {vertical} {horizontal} of the frame"
def describe_heading(dx: float, dy: float) -> str:
"""Name a direction of travel in frame terms.
y grows downward in normalized coordinates, so a falling y reads as moving
toward the top of the frame.
"""
parts = []
if abs(dy) > abs(dx) * 0.4:
parts.append("down" if dy > 0 else "up")
if abs(dx) > abs(dy) * 0.4:
parts.append("right" if dx > 0 else "left")
return " and ".join(parts) if parts else "in place"
def event_name(event: dict[str, Any]) -> str:
"""Name an object for the notes, e.g. 'a person' or 'waste bin "Compost"'.
Objects are never numbered or given track identifiers. Frigate opens a new
tracked object whenever a subject is re-detected, so the tracking data
cannot say whether two entries are the same subject, and the notes stay
ambiguous rather than implying either answer.
"""
label = str(event["label"]).replace("_", " ").replace("-verified", "")
sub_label = event.get("sub_label")
if sub_label:
return f'{label} "{sub_label}"'
article = "an" if label[:1].lower() in "aeiou" else "a"
return f"{article} {label}"
def describe_classification_change(change: dict[str, Any]) -> str:
"""Phrase a state classification change, e.g. 'front gate changed from
closed to open'."""
model = str(change["model"]).replace("_", " ")
before = str(change["from"]).replace("_", " ")
after = str(change["to"]).replace("_", " ")
return f"{model} changed from {before} to {after}"
def path_legs(points: list[Point]) -> list[Leg]:
"""Split a trajectory into runs of travel in a consistent direction.
A leg ends when the subject starts moving back against the direction that
leg established, and only once the leg has covered MIN_LEG_DISTANCE, so
jitter around a standing subject does not register as a turn.
Returns (start, end) index pairs into `points`.
"""
legs: list[Leg] = []
start = 0
for i in range(1, len(points)):
lx = points[i][0] - points[start][0]
ly = points[i][1] - points[start][1]
leg_distance = (lx * lx + ly * ly) ** 0.5
if leg_distance < MIN_LEG_DISTANCE:
continue
sx = points[i][0] - points[i - 1][0]
sy = points[i][1] - points[i - 1][1]
step = (sx * sx + sy * sy) ** 0.5
if step < STILL_THRESHOLD:
continue
dot = (lx / leg_distance) * (sx / step) + (ly / leg_distance) * (sy / step)
if dot < REVERSAL_DOT:
legs.append((start, i - 1))
start = i - 1
if start < len(points) - 1:
legs.append((start, len(points) - 1))
return [
(a, b)
for a, b in legs
if ((points[b][0] - points[a][0]) ** 2 + (points[b][1] - points[a][1]) ** 2)
** 0.5
>= MIN_LEG_DISTANCE
]
def path_points(path_data: list[Any]) -> list[Point]:
"""Flatten path_data into (x, y, timestamp) tuples, or [] if malformed."""
try:
return [(p[0][0], p[0][1], p[1]) for p in path_data or []]
except (IndexError, TypeError):
logger.debug("Malformed path_data, skipping trajectory notes")
return []
def leg_start_time(points: list[Point], leg: Leg) -> float:
"""When a leg's movement actually began.
path_data always keeps an object's first two samples, so a leg can open
with points recorded long before the object moved. The first sample that
has left the leg's origin is the earliest evidence of movement.
"""
a, b = leg
x0, y0, t0 = points[a]
for x, y, t in points[a + 1 : b + 1]:
if math.hypot(x - x0, y - y0) >= STILL_THRESHOLD:
return t
return t0
def leg_heading(points: list[Point], leg: Leg) -> str:
a, b = leg
return describe_heading(points[b][0] - points[a][0], points[b][1] - points[a][1])
def leg_phrase(points: list[Point], leg: Leg, first: bool) -> str:
"""Describe the start of a leg, e.g. 'turns around at ... and heads left'."""
heading = leg_heading(points, leg)
place = describe_position(points[leg[0]][0], points[leg[0]][1])
if first:
return f"starts moving {heading} from {place}"
return f"turns around at {place} and heads {heading}"
def path_moments(path_data: list[Any]) -> list[tuple[float, str]]:
"""Key moments in one trajectory as (timestamp, phrase).
Emits one note per leg of travel. Where the last leg ends is left out:
path_data only records significant movement, so its final point cannot
distinguish an object coming to rest from one leaving the frame.
"""
points = path_points(path_data)
if len(points) < 2:
return []
return [
(leg_start_time(points, leg), leg_phrase(points, leg, index == 0))
for index, leg in enumerate(path_legs(points))
]
def build_timeline(
events: list[dict[str, Any]],
span_end: float,
state_changes: Sequence[dict[str, Any]] = (),
) -> list[tuple[float, str]]:
"""All annotated moments across every event, in time order.
Only changes are noted, since those are what sparse frames miss; an
object's state at the end of the clip is visible in the last frame.
`span_end` is the timestamp of the last sampled frame, and moments past it
describe nothing the model can see. A track ending means the object
stopped being detected, which may or may not mean it left the frame.
`state_changes` are timeline rows (timestamp, source_id, class_type); the
stationary and active ones become "has stopped moving" / "starts moving
again". Frigate only marks an object stationary after it has been still
for a while, which the past-tense wording reflects.
Each object keeps its own notes. Folding an object into the note of the
person moving it ("alongside ...") was tried and made models lose track of
where the object went.
"""
changes_by_event: dict[str, list[tuple[float, str]]] = {}
for change in state_changes:
phrase = STATE_CHANGE_PHRASES.get(change["class_type"])
if phrase:
changes_by_event.setdefault(change["source_id"], []).append(
(change["timestamp"], phrase)
)
timeline: list[tuple[float, str]] = []
for event in sorted(events, key=lambda e: e["start_time"]):
# Frame extraction can come up short at the end of a clip, leaving
# objects that only appear after the last frame we actually have.
if event["start_time"] > span_end:
continue
points = path_points(event.get("path_data") or [])
legs = path_legs(points) if len(points) >= 2 else []
name = event_name(event)
detected_at = event["start_time"]
where = describe_position(points[0][0], points[0][1]) if points else "the frame"
merges = bool(legs) and leg_start_time(points, legs[0]) - detected_at <= (
DETECT_MOVE_MERGE_SECONDS
)
remaining = list(enumerate(legs))
moments: list[tuple[float, str]] = []
if merges:
heading = leg_heading(points, legs[0])
timeline.append(
(detected_at, f"{name} first detected at {where}, moving {heading}")
)
remaining = remaining[1:]
else:
timeline.append((detected_at, f"{name} first detected at {where}"))
for index, leg in remaining:
moments.append(
(
leg_start_time(points, leg),
leg_phrase(points, leg, index == 0),
)
)
moments.extend(
(timestamp, phrase)
for timestamp, phrase in changes_by_event.get(event["id"], [])
if timestamp >= detected_at
)
for timestamp, phrase in moments:
if timestamp <= span_end:
timeline.append((timestamp, f"{name} {phrase}"))
if event["end_time"] and event["end_time"] <= span_end:
timeline.append((event["end_time"], f"{name} is no longer detected"))
return sorted(timeline, key=lambda m: m[0])
def annotations_by_frame(
timeline: list[tuple[float, str]], frame_times: list[float]
) -> dict[int, list[str]]:
"""Bucket timeline moments onto the frame that follows each one.
A moment is attached to the first frame at or after it happened, so the
note always precedes the image in which the change becomes visible.
"""
buckets: dict[int, list[str]] = {}
if not frame_times:
return buckets
for timestamp, phrase in timeline:
index = next(
(i for i, ft in enumerate(frame_times) if ft >= timestamp),
len(frame_times) - 1,
)
buckets.setdefault(index, []).append(phrase)
return buckets
def get_tracked_events(detection_ids: list[str]) -> list[dict[str, Any]]:
"""Load the tracked objects behind a review item's detections."""
if not detection_ids:
return []
rows = list(
Event.select(
Event.id,
Event.label,
Event.sub_label,
Event.start_time,
Event.end_time,
Event.data,
)
.where(Event.id << detection_ids)
.dicts()
.iterator()
)
return [
{
"id": row["id"],
"label": row["label"],
"sub_label": row["sub_label"],
"start_time": row["start_time"],
"end_time": row["end_time"],
"path_data": (row["data"] or {}).get("path_data") or [],
}
for row in rows
if row["start_time"] is not None
]
def get_state_changes(detection_ids: list[str]) -> list[dict[str, Any]]:
"""Stationary/active changes the timeline recorded for these objects."""
if not detection_ids:
return []
return list(
Timeline.select(Timeline.timestamp, Timeline.source_id, Timeline.class_type)
.where(
(Timeline.source_id << detection_ids)
& (Timeline.class_type << list(STATE_CHANGE_PHRASES))
)
.dicts()
.iterator()
)
def build_frame_captions(
detection_ids: list[str],
frame_times: list[float],
classification_changes: Sequence[dict[str, Any]] = (),
) -> list[str]:
"""A caption for each sampled frame, in frame order.
Every frame gets its index and elapsed time so the model can tell them
apart; frames where something changed also carry the tracker and state
classification notes for that moment. Returns an empty list when there is
nothing to say, which callers treat as a reason to fall back to sending
plain frames.
"""
if not frame_times:
return []
span_end = frame_times[-1]
timeline: list[tuple[float, str]] = []
events = get_tracked_events(detection_ids)
# audio and manual review items can have state changes but no tracked objects
if events:
timeline.extend(
(timestamp, f"[tracker] {note}")
for timestamp, note in build_timeline(
events, span_end, get_state_changes(detection_ids)
)
)
timeline.extend(
(change["timestamp"], f"[state] {describe_classification_change(change)}")
for change in classification_changes
if change["timestamp"] <= span_end
)
buckets = annotations_by_frame(sorted(timeline, key=lambda m: m[0]), frame_times)
if not buckets:
return []
total = len(frame_times)
origin = frame_times[0]
captions: list[str] = []
for index, timestamp in enumerate(frame_times):
lines = [f"Frame {index + 1} of {total} (+{timestamp - origin:.1f}s):"]
lines.extend(buckets.get(index, []))
captions.append("\n".join(lines))
return captions
@@ -19,11 +19,7 @@ from frigate.comms.embeddings_updater import EmbeddingsRequestEnum
from frigate.comms.inter_process import InterProcessRequestor from frigate.comms.inter_process import InterProcessRequestor
from frigate.config import FrigateConfig from frigate.config import FrigateConfig
from frigate.config.camera import CameraConfig from frigate.config.camera import CameraConfig
from frigate.config.camera.review import ( from frigate.config.camera.review import GenAIReviewConfig, ImageSourceEnum
GenAIReviewConfig,
ImageSourceEnum,
ReviewFrameModeEnum,
)
from frigate.const import ( from frigate.const import (
ATTRIBUTE_LABEL_DISPLAY_MAP, ATTRIBUTE_LABEL_DISPLAY_MAP,
CACHE_DIR, CACHE_DIR,
@@ -40,7 +36,6 @@ from frigate.util.image import get_image_from_recording
from ..post.api import PostProcessorApi from ..post.api import PostProcessorApi
from ..types import DataProcessorMetrics from ..types import DataProcessorMetrics
from .review_annotations import build_frame_captions, describe_classification_change
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -48,7 +43,6 @@ RECORDING_BUFFER_EXTENSION_PERCENT = 0.10
MIN_RECORDING_DURATION = 10 MIN_RECORDING_DURATION = 10
MAX_IMAGE_TOKENS = 24000 MAX_IMAGE_TOKENS = 24000
MAX_FRAMES_PER_SECOND = 1 MAX_FRAMES_PER_SECOND = 1
MAX_ANNOTATED_FRAMES = 28
class ReviewDescriptionProcessor(PostProcessorApi): class ReviewDescriptionProcessor(PostProcessorApi):
@@ -73,7 +67,6 @@ class ReviewDescriptionProcessor(PostProcessorApi):
duration: float, duration: float,
image_source: ImageSourceEnum = ImageSourceEnum.preview, image_source: ImageSourceEnum = ImageSourceEnum.preview,
height: int = 480, height: int = 480,
frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
) -> int: ) -> int:
"""Calculate optimal number of frames based on event duration, context size, """Calculate optimal number of frames based on event duration, context size,
image source, and resolution. image source, and resolution.
@@ -87,8 +80,6 @@ class ReviewDescriptionProcessor(PostProcessorApi):
- MAX_FRAMES_PER_SECOND x duration, to avoid drowning short events in - MAX_FRAMES_PER_SECOND x duration, to avoid drowning short events in
near-duplicate frames where the model latches onto the redundant middle near-duplicate frames where the model latches onto the redundant middle
and skips the start/end action and skips the start/end action
- MAX_ANNOTATED_FRAMES in annotated mode, where the tracking notes
already carry the sequence
""" """
client = self.genai_manager.description_client client = self.genai_manager.description_client
@@ -134,10 +125,6 @@ class ReviewDescriptionProcessor(PostProcessorApi):
max_frames_by_tokens = int(image_token_budget / tokens_per_image) max_frames_by_tokens = int(image_token_budget / tokens_per_image)
max_frames_by_duration = int(duration * MAX_FRAMES_PER_SECOND) max_frames_by_duration = int(duration * MAX_FRAMES_PER_SECOND)
max_frames = min(max_frames_by_tokens, max_frames_by_duration) max_frames = min(max_frames_by_tokens, max_frames_by_duration)
if frame_mode == ReviewFrameModeEnum.annotated_frames:
max_frames = min(max_frames, MAX_ANNOTATED_FRAMES)
return max(max_frames, 3) return max(max_frames, 3)
def process_data( def process_data(
@@ -179,7 +166,6 @@ class ReviewDescriptionProcessor(PostProcessorApi):
return return
image_source = camera_config.review.genai.image_source image_source = camera_config.review.genai.image_source
frame_mode = camera_config.review.genai.frame_mode
if image_source == ImageSourceEnum.recordings: if image_source == ImageSourceEnum.recordings:
buffer_extension = get_recording_buffer_extension( buffer_extension = get_recording_buffer_extension(
@@ -188,43 +174,40 @@ class ReviewDescriptionProcessor(PostProcessorApi):
final_data["start_time"] -= buffer_extension final_data["start_time"] -= buffer_extension
final_data["end_time"] += buffer_extension final_data["end_time"] += buffer_extension
frames = self.get_recording_frames( thumbs = self.get_recording_frames(
camera, camera,
final_data["start_time"], final_data["start_time"],
final_data["end_time"], final_data["end_time"],
height=480, # Use 480p for good balance between quality and token usage height=480, # Use 480p for good balance between quality and token usage
frame_mode=frame_mode,
) )
if not frames: if not thumbs:
# Fallback to preview frames if no recordings available # Fallback to preview frames if no recordings available
logger.warning( logger.warning(
f"No recording frames found for {camera}, falling back to preview frames" f"No recording frames found for {camera}, falling back to preview frames"
) )
frames = self.get_preview_frames_as_bytes( thumbs = self.get_preview_frames_as_bytes(
camera, camera,
final_data["start_time"], final_data["start_time"],
final_data["end_time"], final_data["end_time"],
final_data["thumb_path"], final_data["thumb_path"],
id, id,
camera_config.review.genai.debug_save_thumbnails, camera_config.review.genai.debug_save_thumbnails,
frame_mode,
) )
elif camera_config.review.genai.debug_save_thumbnails: elif camera_config.review.genai.debug_save_thumbnails:
self.save_debug_recording_frames(id, frames) self.save_debug_recording_frames(id, thumbs)
else: else:
# Use preview frames # Use preview frames
frames = self.get_preview_frames_as_bytes( thumbs = self.get_preview_frames_as_bytes(
camera, camera,
final_data["start_time"], final_data["start_time"],
final_data["end_time"], final_data["end_time"],
final_data["thumb_path"], final_data["thumb_path"],
id, id,
camera_config.review.genai.debug_save_thumbnails, camera_config.review.genai.debug_save_thumbnails,
frame_mode,
) )
self.start_analysis(camera_config, final_data, frames) self.start_analysis(camera_config, final_data, thumbs)
def handle_request(self, topic: str, request_data: dict[str, Any]) -> str | None: def handle_request(self, topic: str, request_data: dict[str, Any]) -> str | None:
if topic == EmbeddingsRequestEnum.regenerate_review_description.value: if topic == EmbeddingsRequestEnum.regenerate_review_description.value:
@@ -254,10 +237,6 @@ class ReviewDescriptionProcessor(PostProcessorApi):
"start_time": r["start_time"], "start_time": r["start_time"],
"end_time": r["end_time"], "end_time": r["end_time"],
"metadata": r["data"]["metadata"], "metadata": r["data"]["metadata"],
"state_changes": [
describe_classification_change(change)
for change in sorted_classification_state_changes(r["data"])
],
} }
for r in ( for r in (
ReviewSegment.select( ReviewSegment.select(
@@ -302,9 +281,6 @@ class ReviewDescriptionProcessor(PostProcessorApi):
primary_item["start_time"] = primary_seg["start_time"] primary_item["start_time"] = primary_seg["start_time"]
primary_item["end_time"] = primary_seg["end_time"] primary_item["end_time"] = primary_seg["end_time"]
if primary_seg["state_changes"]:
primary_item["state_changes"] = primary_seg["state_changes"]
# Find overlapping contextual items from other cameras # Find overlapping contextual items from other cameras
primary_start = primary_seg["start_time"] primary_start = primary_seg["start_time"]
primary_end = primary_seg["end_time"] primary_end = primary_seg["end_time"]
@@ -325,25 +301,14 @@ class ReviewDescriptionProcessor(PostProcessorApi):
seg_end = seg["end_time"] seg_end = seg["end_time"]
if seg_start < primary_end and primary_start < seg_end: if seg_start < primary_end and primary_start < seg_end:
# Avoid duplicates if same camera has multiple overlapping # Avoid duplicates if same camera has multiple overlapping segments
# segments. One with state changes is kept as its own item if seg_camera not in seen_contextual_cameras:
# so each change stays within its item's time range. contextual_item = copy.deepcopy(seg["metadata"])
if ( contextual_item["camera"] = seg_camera
seg_camera in seen_contextual_cameras contextual_item["start_time"] = seg_start
and not seg["state_changes"] contextual_item["end_time"] = seg_end
): contextual_items.append(contextual_item)
continue seen_contextual_cameras.add(seg_camera)
contextual_item = copy.deepcopy(seg["metadata"])
contextual_item["camera"] = seg_camera
contextual_item["start_time"] = seg_start
contextual_item["end_time"] = seg_end
if seg["state_changes"]:
contextual_item["state_changes"] = seg["state_changes"]
contextual_items.append(contextual_item)
seen_contextual_cameras.add(seg_camera)
# Add context array to primary item # Add context array to primary item
primary_item["context"] = contextual_items primary_item["context"] = contextual_items
@@ -421,51 +386,31 @@ class ReviewDescriptionProcessor(PostProcessorApi):
buffer_extension = get_recording_buffer_extension( buffer_extension = get_recording_buffer_extension(
final_data["end_time"] - final_data["start_time"] final_data["end_time"] - final_data["start_time"]
) )
frames = self.get_recording_frames( thumbs = self.get_recording_frames(
str(review.camera), str(review.camera),
final_data["start_time"] - buffer_extension, final_data["start_time"] - buffer_extension,
final_data["end_time"] + buffer_extension, final_data["end_time"] + buffer_extension,
height=480, height=480,
frame_mode=camera_config.review.genai.frame_mode,
) )
if not frames: if not thumbs:
logger.error( logger.error(
"No recording frames are available for review item %s", review_id "No recording frames are available for review item %s", review_id
) )
return return
if camera_config.review.genai.debug_save_thumbnails: if camera_config.review.genai.debug_save_thumbnails:
self.save_debug_recording_frames(review_id, frames) self.save_debug_recording_frames(review_id, thumbs)
self.start_analysis(camera_config, final_data, frames) self.start_analysis(camera_config, final_data, thumbs)
def start_analysis( def start_analysis(
self, self,
camera_config: CameraConfig, camera_config: CameraConfig,
final_data: dict[str, Any], final_data: dict[str, Any],
frames: list[tuple[bytes, float]], thumbs: list[bytes],
) -> None: ) -> None:
"""Kick off description generation for a review item in the background.""" """Kick off description generation for a review item in the background."""
thumbs = [frame for frame, _ in frames]
captions: list[str] = []
if (
camera_config.review.genai.frame_mode
== ReviewFrameModeEnum.annotated_frames
):
captions = build_frame_captions(
final_data["data"].get("detections") or [],
[timestamp for _, timestamp in frames],
sorted_classification_state_changes(final_data["data"]),
)
if not captions:
logger.debug(
"No tracking annotations for review item %s, sending plain frames",
final_data["id"],
)
self.review_desc_dps.update() self.review_desc_dps.update()
threading.Thread( threading.Thread(
target=run_analysis, target=run_analysis,
@@ -476,22 +421,19 @@ class ReviewDescriptionProcessor(PostProcessorApi):
camera_config, camera_config,
final_data, final_data,
thumbs, thumbs,
captions,
camera_config.review.genai, camera_config.review.genai,
sorted(self.config.all_labels), sorted(self.config.all_labels),
self.config.all_attributes, self.config.all_attributes,
), ),
).start() ).start()
def save_debug_recording_frames( def save_debug_recording_frames(self, review_id: str, thumbs: list[bytes]) -> None:
self, review_id: str, frames: list[tuple[bytes, float]]
) -> None:
"""Write the recording frames sent to the provider out for debugging.""" """Write the recording frames sent to the provider out for debugging."""
Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir( Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir(
parents=True, exist_ok=True parents=True, exist_ok=True
) )
for idx, (frame_bytes, _) in enumerate(frames): for idx, frame_bytes in enumerate(thumbs):
with open( with open(
os.path.join(CLIPS_DIR, f"genai-requests/{review_id}/{idx}.jpg"), os.path.join(CLIPS_DIR, f"genai-requests/{review_id}/{idx}.jpg"),
"wb", "wb",
@@ -503,9 +445,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
camera: str, camera: str,
start_time: float, start_time: float,
end_time: float, end_time: float,
frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames, ) -> list[str]:
) -> list[tuple[str, float]]:
"""Preview frame paths paired with the time each one was captured."""
preview_dir = os.path.join(CACHE_DIR, "preview_frames") preview_dir = os.path.join(CACHE_DIR, "preview_frames")
file_start = f"preview_{camera}-" file_start = f"preview_{camera}-"
start_file = f"{file_start}{start_time}.webp" start_file = f"{file_start}{start_time}.webp"
@@ -537,29 +477,18 @@ class ReviewDescriptionProcessor(PostProcessorApi):
frame_count = len(all_frames) frame_count = len(all_frames)
desired_frame_count = self.calculate_frame_count( desired_frame_count = self.calculate_frame_count(
camera, camera, duration=end_time - start_time
duration=end_time - start_time,
frame_mode=frame_mode,
) )
def with_timestamp(path: str) -> tuple[str, float]:
# Preview frames are named preview_<camera>-<timestamp>.webp
stem = os.path.basename(path).removesuffix(".webp")
try:
return (path, float(stem.removeprefix(file_start)))
except ValueError:
return (path, start_time)
if frame_count <= desired_frame_count: if frame_count <= desired_frame_count:
return [with_timestamp(f) for f in all_frames] return all_frames
selected_frames = [] selected_frames = []
step_size = (frame_count - 1) / (desired_frame_count - 1) step_size = (frame_count - 1) / (desired_frame_count - 1)
for i in range(desired_frame_count): for i in range(desired_frame_count):
index = round(i * step_size) index = round(i * step_size)
selected_frames.append(with_timestamp(all_frames[index])) selected_frames.append(all_frames[index])
return selected_frames return selected_frames
@@ -569,12 +498,11 @@ class ReviewDescriptionProcessor(PostProcessorApi):
start_time: float, start_time: float,
end_time: float, end_time: float,
height: int = 480, height: int = 480,
frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames, ) -> list[bytes]:
) -> list[tuple[bytes, float]]: """Get frames from recordings at specified timestamps."""
"""Get frames from recordings paired with the time each was captured."""
duration = end_time - start_time duration = end_time - start_time
desired_frame_count = self.calculate_frame_count( desired_frame_count = self.calculate_frame_count(
camera, duration, ImageSourceEnum.recordings, height, frame_mode camera, duration, ImageSourceEnum.recordings, height
) )
# Calculate evenly spaced timestamps throughout the duration # Calculate evenly spaced timestamps throughout the duration
@@ -612,7 +540,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
except DoesNotExist: except DoesNotExist:
return None return None
frames: list[tuple[bytes, float]] = [] frames = []
for timestamp in timestamps: for timestamp in timestamps:
try: try:
@@ -625,7 +553,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
image_data = extract_frame_from_recording(rounded_timestamp) image_data = extract_frame_from_recording(rounded_timestamp)
if image_data: if image_data:
frames.append((image_data, timestamp)) frames.append(image_data)
else: else:
logger.warning( logger.warning(
f"No recording found for {camera} at timestamp {timestamp}" f"No recording found for {camera} at timestamp {timestamp}"
@@ -646,8 +574,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
thumb_path_fallback: str, thumb_path_fallback: str,
review_id: str, review_id: str,
save_debug: bool, save_debug: bool,
frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames, ) -> list[bytes]:
) -> list[tuple[bytes, float]]:
"""Get preview frames and convert them to JPEG bytes. """Get preview frames and convert them to JPEG bytes.
Args: Args:
@@ -659,14 +586,14 @@ class ReviewDescriptionProcessor(PostProcessorApi):
save_debug: Whether to save debug thumbnails save_debug: Whether to save debug thumbnails
Returns: Returns:
List of (JPEG image bytes, capture timestamp) pairs List of JPEG image bytes
""" """
frame_paths = self.get_cache_frames(camera, start_time, end_time, frame_mode) frame_paths = self.get_cache_frames(camera, start_time, end_time)
if not frame_paths: if not frame_paths:
frame_paths = [(thumb_path_fallback, start_time)] frame_paths = [thumb_path_fallback]
thumbs: list[tuple[bytes, float]] = [] thumbs = []
for idx, (thumb_path, timestamp) in enumerate(frame_paths): for idx, thumb_path in enumerate(frame_paths):
thumb_data = cv2.imread(thumb_path) thumb_data = cv2.imread(thumb_path)
if thumb_data is None: if thumb_data is None:
@@ -679,7 +606,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
".jpg", thumb_data, [int(cv2.IMWRITE_JPEG_QUALITY), 100] ".jpg", thumb_data, [int(cv2.IMWRITE_JPEG_QUALITY), 100]
) )
if ret: if ret:
thumbs.append((jpg.tobytes(), timestamp)) thumbs.append(jpg.tobytes())
if save_debug: if save_debug:
Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir( Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir(
@@ -707,39 +634,6 @@ def get_recording_buffer_extension(duration: float) -> float:
return buffer_extension return buffer_extension
def sorted_classification_state_changes(
review_data: dict[str, Any],
) -> list[dict[str, Any]]:
"""A review item's state classification changes in time order."""
return sorted(
review_data.get("classification_state_changes") or [],
key=lambda change: change["timestamp"],
)
def format_classification_state_changes(
changes: list[dict[str, Any]], start_time: float, end_time: float
) -> list[str]:
"""Phrase state classification changes with their timing in the activity.
Changes are attached while the review item is active, which runs past its
end_time by the review cutoff, and a few seconds before its start.
"""
lines = []
for change in changes:
if change["timestamp"] < start_time:
when = "just before the activity started"
elif change["timestamp"] > end_time:
when = "after the activity ended"
else:
when = f"{round(change['timestamp'] - start_time)}s into the activity"
lines.append(f"{describe_classification_change(change)}, {when}")
return lines
def run_analysis( def run_analysis(
requestor: InterProcessRequestor, requestor: InterProcessRequestor,
genai_client: GenAIClient, genai_client: GenAIClient,
@@ -747,7 +641,6 @@ def run_analysis(
camera_config: CameraConfig, camera_config: CameraConfig,
final_data: dict[str, Any], final_data: dict[str, Any],
thumbs: list[bytes], thumbs: list[bytes],
frame_captions: list[str],
genai_config: GenAIReviewConfig, genai_config: GenAIReviewConfig,
labelmap_objects: list[str], labelmap_objects: list[str],
attribute_labels: list[str], attribute_labels: list[str],
@@ -795,13 +688,6 @@ def run_analysis(
unified_objects.append(object_type) unified_objects.append(object_type)
analytics_data["unified_objects"] = unified_objects analytics_data["unified_objects"] = unified_objects
analytics_data["classification_state_changes"] = (
format_classification_state_changes(
sorted_classification_state_changes(final_data["data"]),
final_data["start_time"],
final_data["end_time"],
)
)
metadata = genai_client.generate_review_description( metadata = genai_client.generate_review_description(
analytics_data, analytics_data,
@@ -811,7 +697,6 @@ def run_analysis(
genai_config.debug_save_thumbnails, genai_config.debug_save_thumbnails,
genai_config.activity_context_prompt, genai_config.activity_context_prompt,
genai_config.response_style, genai_config.response_style,
frame_captions,
) )
review_inference_speed.update(datetime.datetime.now().timestamp() - start) review_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -1,19 +1,16 @@
"""Handle processing audio for speech transcription using sherpa-onnx with FFmpeg pipe.""" """Handle processing audio for speech transcription using sherpa-onnx with FFmpeg pipe."""
import collections
import logging import logging
import os import os
import queue import queue
import threading import threading
import time from typing import Any
from typing import TYPE_CHECKING, Any
import numpy as np import numpy as np
from frigate.comms.inter_process import InterProcessRequestor from frigate.comms.inter_process import InterProcessRequestor
from frigate.config import CameraConfig, FrigateConfig from frigate.config import CameraConfig, FrigateConfig
from frigate.config.classification import AudioTranscriptionModelEnum from frigate.const import MODEL_CACHE_DIR
from frigate.const import AUDIO_DURATION, MODEL_CACHE_DIR
from frigate.data_processing.common.audio_transcription.model import ( from frigate.data_processing.common.audio_transcription.model import (
AudioTranscriptionModelRunner, AudioTranscriptionModelRunner,
) )
@@ -21,38 +18,12 @@ from frigate.data_processing.real_time.whisper_online import (
FasterWhisperASR, FasterWhisperASR,
OnlineASRProcessor, OnlineASRProcessor,
) )
from frigate.util.audio import (
clean_transcript,
pcm16_to_wav,
resolve_language,
stitch_transcripts,
)
from ..types import DataProcessorMetrics from ..types import DataProcessorMetrics
from .api import RealTimeProcessorApi from .api import RealTimeProcessorApi
if TYPE_CHECKING:
# importing frigate.genai eagerly would pull the provider SDKs into the
# audio process even when transcription runs on a local model
from frigate.genai.manager import GenAIClientManager
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# Number of ~0.975s audio detector chunks per GenAI request. The window advances
# one chunk at a time, so two chunks means a 50% overlap: every word lands whole
# in at least one window, which whisper-family models need to avoid hallucinating
# on a clipped clip. The cadence is fixed by the audio detector's frame size, so
# this is a constant rather than a config knob.
GENAI_WINDOW_CHUNKS = 2
# Bound the queue at ~30s of audio so a slow or hung provider cannot grow it
# without limit. The producer is the ffmpeg read thread and must never block.
AUDIO_QUEUE_MAXSIZE = int(30 / AUDIO_DURATION)
# A backed-up queue drops a chunk per cycle, so warning on each one would spam
# the log once a second per camera for as long as the provider stays slow.
AUDIO_DROP_WARN_INTERVAL = 10.0
class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi): class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
def __init__( def __init__(
@@ -60,10 +31,9 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
config: FrigateConfig, config: FrigateConfig,
camera_config: CameraConfig, camera_config: CameraConfig,
requestor: InterProcessRequestor, requestor: InterProcessRequestor,
model_runner: AudioTranscriptionModelRunner | None, model_runner: AudioTranscriptionModelRunner,
metrics: DataProcessorMetrics, metrics: DataProcessorMetrics,
stop_event: threading.Event, stop_event: threading.Event,
genai_manager: "GenAIClientManager | None" = None,
): ):
super().__init__(config, metrics) super().__init__(config, metrics)
self.config = config self.config = config
@@ -72,31 +42,11 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
self.stream: Any = None self.stream: Any = None
self.whisper_model: FasterWhisperASR | None = None self.whisper_model: FasterWhisperASR | None = None
self.model_runner = model_runner self.model_runner = model_runner
self.genai_manager = genai_manager
self.transcription_segments: list[str] = [] self.transcription_segments: list[str] = []
self.audio_queue: queue.Queue[tuple[dict[str, Any], np.ndarray]] = queue.Queue( self.audio_queue: queue.Queue[tuple[dict[str, Any], np.ndarray]] = queue.Queue()
maxsize=AUDIO_QUEUE_MAXSIZE
)
self.stop_event = stop_event self.stop_event = stop_event
self._use_genai = not isinstance(
config.audio_transcription.model, AudioTranscriptionModelEnum
)
# sliding window of raw int16 chunks; the deque's maxlen is what evicts
# the oldest chunk and so produces the overlap
self._genai_window: collections.deque[np.ndarray] = collections.deque(
maxlen=GENAI_WINDOW_CHUNKS
)
self._genai_committed = ""
# set by the producer when it discards a chunk, so the consumer knows the
# audio it is about to receive is not contiguous with what it buffered
self._audio_dropped = threading.Event()
self._last_drop_warning = 0.0
def __build_recognizer(self) -> None: def __build_recognizer(self) -> None:
if self._use_genai:
# nothing local to load; never import sherpa or FasterWhisperASR
return
try: try:
if self.config.audio_transcription.model_size == "large": if self.config.audio_transcription.model_size == "large":
# Whisper models need to be per-process and can only run one stream at a time # Whisper models need to be per-process and can only run one stream at a time
@@ -114,7 +64,7 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
self.stream = OnlineASRProcessor( self.stream = OnlineASRProcessor(
asr=self.whisper_model, asr=self.whisper_model,
) )
elif self.model_runner is not None: else:
logger.debug(f"Loading sherpa stream for {self.camera_config.name}") logger.debug(f"Loading sherpa stream for {self.camera_config.name}")
self.stream = self.model_runner.model.create_stream() self.stream = self.model_runner.model.create_stream()
logger.debug( logger.debug(
@@ -126,15 +76,6 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
) )
def __process_audio_stream(self, audio_data: np.ndarray) -> tuple[str, bool] | None: def __process_audio_stream(self, audio_data: np.ndarray) -> tuple[str, bool] | None:
# must precede both the model_runner guard (model_runner is None on this
# path) and the float32 normalization below (GenAI wants untouched int16)
if self._use_genai:
return self.__process_audio_genai(audio_data)
if self.model_runner is None:
logger.debug("Audio transcription (live) model runner not initialized")
return None
if ( if (
self.model_runner.model is None self.model_runner.model is None
and self.config.audio_transcription.model_size == "small" and self.config.audio_transcription.model_size == "small"
@@ -199,82 +140,6 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
logger.error(f"Error processing audio stream: {e}") logger.error(f"Error processing audio stream: {e}")
return None return None
def __process_audio_genai(self, audio_data: np.ndarray) -> tuple[str, bool] | None:
"""Transcribe a sliding overlapped window through the GenAI provider."""
client = self.genai_manager.transcribe_client if self.genai_manager else None
if not client:
logger.error(
"audio_transcription.model is '%s' (GenAI provider) but no transcribe "
"client is configured. Ensure the GenAI provider has 'transcribe' in its roles",
self.config.audio_transcription.model,
)
return None
if self._audio_dropped.is_set():
self._audio_dropped.clear()
# Chunks were discarded between what is buffered and this one, so
# concatenating them would splice non-adjacent audio into one window
# and destroy the overlap the stitcher depends on.
self._genai_window.clear()
if self._genai_committed:
# the transcript has a gap in it; close the utterance out rather
# than stitching across missing speech
return self.__end_genai_utterance()
self._genai_window.append(audio_data)
if len(self._genai_window) < GENAI_WINDOW_CHUNKS:
# wait for a full window so the first request is never a clipped clip
return None
window = np.concatenate(list(self._genai_window))
# Silence gate, using the same threshold audio detection uses. Gate the
# whole window rather than individual chunks; this is the primary cost
# and privacy brake and is what keeps a quiet camera near zero requests.
window_as_float = window.astype(np.float32)
rms = float(np.sqrt(np.mean(np.absolute(np.square(window_as_float)))))
if rms < self.camera_config.audio.min_volume:
logger.debug(
f"Window RMS {rms:.1f} below min_volume, skipping transcription"
)
return self.__end_genai_utterance()
text = client.transcribe(
pcm16_to_wav(window),
language=resolve_language(self.config.audio_transcription.language),
)
# cleaning has to come first: a silent window often comes back as the
# model's preamble alone, which is silence, not a word to commit
cleaned = clean_transcript(text)
if not cleaned:
return self.__end_genai_utterance()
self._genai_committed = stitch_transcripts(self._genai_committed, cleaned)
# no VAD on this path, so mirror the whisper branch's heuristic endpoint
is_endpoint = (
self._genai_committed.endswith((".", "!", "?"))
and len(self._genai_committed) > 300
)
logger.debug(f"GenAI transcription: '{self._genai_committed}'")
return self._genai_committed, is_endpoint
def __end_genai_utterance(self) -> tuple[str, bool] | None:
"""Close out the current utterance when a window carries no speech."""
if not self._genai_committed:
return None
return self._genai_committed, True
def process_frame(self, obj_data: dict[str, Any], frame: np.ndarray) -> None: def process_frame(self, obj_data: dict[str, Any], frame: np.ndarray) -> None:
pass pass
@@ -283,38 +148,8 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
logger.debug("No audio data provided for transcription") logger.debug("No audio data provided for transcription")
return None return None
# enqueue audio data for processing in the thread. never block: the # enqueue audio data for processing in the thread
# producer is the ffmpeg read thread that audio detection depends on, self.audio_queue.put((obj_data, audio))
# so on a backlog drop the oldest chunk instead.
try:
self.audio_queue.put_nowait((obj_data, audio))
except queue.Full:
try:
self.audio_queue.get_nowait()
self.audio_queue.task_done()
except queue.Empty:
pass
# the stream now has a hole in it, which the consumer has to know
# about before it splices the next chunk onto what it already holds
self._audio_dropped.set()
now = time.monotonic()
if now - self._last_drop_warning >= AUDIO_DROP_WARN_INTERVAL:
self._last_drop_warning = now
logger.warning(
"Audio transcription queue for %s is full, dropping audio. The "
"provider is not keeping up with the %.2fs chunk rate",
self.camera_config.name,
AUDIO_DURATION,
)
try:
self.audio_queue.put_nowait((obj_data, audio))
except queue.Full:
pass
return None return None
def run(self) -> None: def run(self) -> None:
@@ -370,14 +205,6 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
break break
def reset(self) -> None: def reset(self) -> None:
if self._use_genai:
self._genai_committed = ""
# stale audio carried across an utterance boundary would be
# re-transcribed into the next one
self._genai_window.clear()
logger.debug("Stream reset")
return
if self.config.audio_transcription.model_size == "large": if self.config.audio_transcription.model_size == "large":
# get final output from whisper # get final output from whisper
output = self.stream.finish() output = self.stream.finish()
@@ -391,7 +218,7 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
# reset whisper # reset whisper
self.stream.init() self.stream.init()
self.transcription_segments = [] self.transcription_segments = []
elif self.model_runner is not None: else:
# reset sherpa # reset sherpa
self.model_runner.model.reset(self.stream) self.model_runner.model.reset(self.stream)
@@ -399,24 +226,6 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
def check_unload_model(self) -> None: def check_unload_model(self) -> None:
# regularly called in the loop in audio maintainer # regularly called in the loop in audio maintainer
if self._use_genai:
# no model to unload, but this is the hook that fires when
# live_enabled flips off. guard on emptiness: called ~1x/s per camera.
if self._genai_committed or self._genai_window:
logger.debug(
f"Clearing GenAI transcription state for {self.camera_config.name}"
)
self.clear_audio_queue()
self._genai_committed = ""
self._genai_window.clear()
self.requestor.send_data(
f"{self.camera_config.name}/audio/transcription",
"",
)
return
if ( if (
self.config.audio_transcription.model_size == "large" self.config.audio_transcription.model_size == "large"
and self.whisper_model is not None and self.whisper_model is not None
@@ -461,10 +270,6 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
self, topic: str, request_data: dict[str, Any] self, topic: str, request_data: dict[str, Any]
) -> dict[str, Any] | None: ) -> dict[str, Any] | None:
if topic == "clear_audio_recognizer": if topic == "clear_audio_recognizer":
if self._use_genai:
self.reset()
return {"message": "Audio transcription state cleared", "success": True}
self.stream = None self.stream = None
self.__build_recognizer() self.__build_recognizer()
return {"message": "Audio recognizer cleared and rebuilt", "success": True} return {"message": "Audio recognizer cleared and rebuilt", "success": True}
@@ -91,23 +91,8 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
self.tensor_input_details = self.interpreter.get_input_details() self.tensor_input_details = self.interpreter.get_input_details()
self.tensor_output_details = self.interpreter.get_output_details() self.tensor_output_details = self.interpreter.get_output_details()
self.labelmap = load_labels(labelmap_path, prefill=0, indexed=False) self.labelmap = load_labels(labelmap_path, prefill=0, indexed=False)
self._forget_unknown_states()
self.classifications_per_second.start() self.classifications_per_second.start()
def _forget_unknown_states(self) -> None:
"""Drop verified states that are not labels of the loaded model.
A retrained model can rename or remove labels. Keeping a state it can
no longer produce would report its first verified state as a change
from that obsolete label.
"""
labels = set(self.labelmap.values())
self.state_history = {
camera: history
for camera, history in self.state_history.items()
if history["current_state"] in labels
}
def __update_metrics(self, duration: float) -> None: def __update_metrics(self, duration: float) -> None:
self.classifications_per_second.update() self.classifications_per_second.update()
if self.inference_speed: if self.inference_speed:
@@ -149,20 +134,15 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
# Don't save if state is stable (detected_state == current_state) AND score is 100% # Don't save if state is stable (detected_state == current_state) AND score is 100%
return False return False
def verify_state_change( def verify_state_change(self, camera: str, detected_state: str) -> str | None:
self, camera: str, detected_state: str, timestamp: float
) -> tuple[str | None, float] | None:
""" """
Verify state change requires 3 consecutive identical states before publishing. Verify state change requires 3 consecutive identical states before publishing.
Returns (previous state, time the new state was first seen) once verified, Returns state to publish or None if verification not complete.
or None if verification not complete. The previous state is None for the
first state verified on a camera.
""" """
if camera not in self.state_history: if camera not in self.state_history:
self.state_history[camera] = { self.state_history[camera] = {
"current_state": None, "current_state": None,
"pending_state": None, "pending_state": None,
"pending_since": 0.0,
"consecutive_count": 0, "consecutive_count": 0,
} }
@@ -177,14 +157,12 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
verification["consecutive_count"] += 1 verification["consecutive_count"] += 1
if verification["consecutive_count"] >= 3: if verification["consecutive_count"] >= 3:
previous_state = verification["current_state"]
verification["current_state"] = detected_state verification["current_state"] = detected_state
verification["pending_state"] = None verification["pending_state"] = None
verification["consecutive_count"] = 0 verification["consecutive_count"] = 0
return previous_state, verification["pending_since"] return detected_state
else: else:
verification["pending_state"] = detected_state verification["pending_state"] = detected_state
verification["pending_since"] = timestamp
verification["consecutive_count"] = 1 verification["consecutive_count"] = 1
logger.debug( logger.debug(
f"New state '{detected_state}' detected for {camera}, need {3 - verification['consecutive_count']} more consecutive detections" f"New state '{detected_state}' detected for {camera}, need {3 - verification['consecutive_count']} more consecutive detections"
@@ -362,19 +340,16 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
) )
return return
verified = self.verify_state_change(camera, detected_state, timestamp) verified_state = self.verify_state_change(camera, detected_state)
if verified is not None: if verified_state is not None:
previous_state, changed_at = verified
self._emit_result( self._emit_result(
{ {
"type": "classification", "type": "classification",
"processor": "state", "processor": "state",
"model_name": self.model_config.name, "model_name": self.model_config.name,
"camera": camera, "camera": camera,
"state": detected_state, "state": verified_state,
"previous_state": previous_state,
"timestamp": changed_at,
} }
) )
+1 -1
View File
@@ -365,7 +365,7 @@ class FaceRealTimeProcessor(RealTimeProcessorApi):
img = cv2.imread(current_file) img = cv2.imread(current_file)
if img is None: if img is None:
return { return { # type: ignore[unreachable]
"message": "Invalid image file.", "message": "Invalid image file.",
"success": False, "success": False,
} }
+15 -76
View File
@@ -1,10 +1,8 @@
import logging import logging
import sqlite3 import sqlite3
import threading
from typing import Any from typing import Any
import regex import regex
from peewee import DatabaseError
from playhouse.sqliteq import SqliteQueueDatabase from playhouse.sqliteq import SqliteQueueDatabase
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -19,7 +17,6 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
self.load_vec_extension: bool = load_vec_extension self.load_vec_extension: bool = load_vec_extension
# no extension necessary, sqlite will load correctly for each platform # no extension necessary, sqlite will load correctly for each platform
self.sqlite_vec_path = "/usr/local/lib/vec0" self.sqlite_vec_path = "/usr/local/lib/vec0"
self.upsert_lock = threading.Lock()
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
def _connect(self, *args: Any, **kwargs: Any) -> sqlite3.Connection: def _connect(self, *args: Any, **kwargs: Any) -> sqlite3.Connection:
@@ -56,22 +53,6 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
conn.create_function("REGEXP", 2, regexp) conn.create_function("REGEXP", 2, regexp)
def execute_write(self, sql: str, params: Any = None) -> None:
"""Run a write and wait for it, so that failures are raised here.
SqliteQueueDatabase hands non-SELECT statements to a writer thread and
stores any exception on the cursor it returns, so callers that ignore
that cursor never learn the write failed.
"""
self.execute_sql(sql, params).fetchall()
def _table_exists(self, table: str) -> bool:
cursor = self.execute_sql(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?",
(table,),
)
return cursor.fetchone() is not None
def _delete_embeddings(self, table: str, event_ids: list[str]) -> None: def _delete_embeddings(self, table: str, event_ids: list[str]) -> None:
"""Delete embeddings for the given events, if the table exists. """Delete embeddings for the given events, if the table exists.
@@ -82,17 +63,17 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
return return
# the embeddings tables are only created once semantic search has run # the embeddings tables are only created once semantic search has run
if not self._table_exists(table): cursor = self.execute_sql(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?",
(table,),
)
if cursor.fetchone() is None:
logger.debug("Skipping %s cleanup, table does not exist", table) logger.debug("Skipping %s cleanup, table does not exist", table)
return return
ids = ",".join(["?" for _ in event_ids]) ids = ",".join(["?" for _ in event_ids])
self.execute_sql(f"DELETE FROM {table} WHERE id IN ({ids})", event_ids)
# callers treat cleanup as best effort, so log rather than propagate
try:
self.execute_write(f"DELETE FROM {table} WHERE id IN ({ids})", event_ids)
except DatabaseError:
logger.exception("Failed to delete embeddings from %s", table)
def delete_embeddings_thumbnail(self, event_ids: list[str]) -> None: def delete_embeddings_thumbnail(self, event_ids: list[str]) -> None:
self._delete_embeddings("vec_thumbnails", event_ids) self._delete_embeddings("vec_thumbnails", event_ids)
@@ -100,67 +81,25 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
def delete_embeddings_description(self, event_ids: list[str]) -> None: def delete_embeddings_description(self, event_ids: list[str]) -> None:
self._delete_embeddings("vec_descriptions", event_ids) self._delete_embeddings("vec_descriptions", event_ids)
def _restore_vec_info_table(self, table: str) -> None:
"""Recreate the _info shadow table a legacy vec0 table is missing.
sqlite-vec added _info in 0.1.6 and drops it unconditionally when a
table is destroyed, so tables written by Frigate 0.17 and earlier fail
to drop. An empty stub is enough, and leaving it unseeded keeps the
table reading as pre-0.1.10 if the drop does not follow.
"""
if not self._table_exists(table) or self._table_exists(f"{table}_info"):
return
logger.debug("Restoring the %s_info shadow table before dropping", table)
self.execute_write(
f'CREATE TABLE "{table}_info" (key TEXT PRIMARY KEY, value ANY)'
)
def drop_embeddings_tables(self) -> None: def drop_embeddings_tables(self) -> None:
for table in ("vec_descriptions", "vec_thumbnails"): self.execute_sql("""
self._restore_vec_info_table(table) DROP TABLE vec_descriptions;
self.execute_write(f"DROP TABLE IF EXISTS {table}") """)
self.execute_sql("""
DROP TABLE vec_thumbnails;
""")
def create_embeddings_tables(self) -> None: def create_embeddings_tables(self) -> None:
"""Create vec0 virtual table for embeddings""" """Create vec0 virtual table for embeddings"""
self.execute_write(""" self.execute_sql("""
CREATE VIRTUAL TABLE IF NOT EXISTS vec_thumbnails USING vec0( CREATE VIRTUAL TABLE IF NOT EXISTS vec_thumbnails USING vec0(
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
thumbnail_embedding FLOAT[768] distance_metric=cosine thumbnail_embedding FLOAT[768] distance_metric=cosine
); );
""") """)
self.execute_write(""" self.execute_sql("""
CREATE VIRTUAL TABLE IF NOT EXISTS vec_descriptions USING vec0( CREATE VIRTUAL TABLE IF NOT EXISTS vec_descriptions USING vec0(
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
description_embedding FLOAT[768] distance_metric=cosine description_embedding FLOAT[768] distance_metric=cosine
); );
""") """)
def upsert_embeddings(
self, table: str, column: str, embeddings: dict[str, bytes]
) -> None:
"""Write embeddings for the given event ids, replacing any that exist.
vec0 implements neither REPLACE nor UPSERT, so rows that are already
there have to be deleted first.
"""
if not embeddings:
return
event_ids = list(embeddings.keys())
ids = ",".join(["?" for _ in event_ids])
params: list[Any] = []
for event_id in event_ids:
params.extend((event_id, embeddings[event_id]))
values = ", ".join(["(?, ?)"] * len(event_ids))
# reindexing and live embedding run on separate threads, and each write
# is queued separately, so the delete and the insert have to be held
# together or an interleaved pair fails on the vec0 primary key
with self.upsert_lock:
self.execute_write(f"DELETE FROM {table} WHERE id IN ({ids})", event_ids)
self.execute_write(
f"INSERT INTO {table}(id, {column}) VALUES {values}", params
)
+10 -79
View File
@@ -46,42 +46,8 @@ _PROVIDER_LABELS = {
"MIGraphXExecutionProvider": "MIGraphX", "MIGraphXExecutionProvider": "MIGraphX",
"OpenVINOExecutionProvider": "OpenVINO", "OpenVINOExecutionProvider": "OpenVINO",
"CPUExecutionProvider": "CPU", "CPUExecutionProvider": "CPU",
"LighterANE": "Neural Engine",
} }
# lighter (https://github.com/fieldwork-ai/lighter) places an ONNX Runtime plugin
# execution provider in a container started with --device lighter.sh/ane=all,
# which runs models on a Mac's Neural Engine; LIGHTER_ANE_EP names where it is
LIGHTER_ANE_EP_NAME = "LighterANE"
LIGHTER_ANE_LIBRARY = "/usr/lib/lighter/liblighter_ane_ep.so"
def get_lighter_ane_devices() -> list[Any]:
"""Get the Neural Engine devices lighter's provider offers, registering it once.
Returns:
The provider's ONNX Runtime devices, or an empty list without lighter's device
"""
library = os.environ.get("LIGHTER_ANE_EP", LIGHTER_ANE_LIBRARY)
if not os.path.exists(library):
return []
devices = [d for d in ort.get_ep_devices() if d.ep_name == LIGHTER_ANE_EP_NAME]
if not devices:
try:
ort.register_execution_provider_library(LIGHTER_ANE_EP_NAME, library)
except Exception as e:
logger.warning(
f"Failed to load the Neural Engine provider from {library}: {e}"
)
return []
devices = [d for d in ort.get_ep_devices() if d.ep_name == LIGHTER_ANE_EP_NAME]
return devices
def is_arm64_platform() -> bool: def is_arm64_platform() -> bool:
"""Check if we're running on an ARM platform.""" """Check if we're running on an ARM platform."""
@@ -89,46 +55,30 @@ def is_arm64_platform() -> bool:
return machine in ("aarch64", "arm64", "armv8", "armv7l") return machine in ("aarch64", "arm64", "armv8", "armv7l")
def get_ort_session_options( def get_ort_session_options(model_type: str | None = None) -> ort.SessionOptions | None:
model_type: str | None = None, cpu: bool = False
) -> ort.SessionOptions | None:
"""Get ONNX Runtime session options with appropriate settings. """Get ONNX Runtime session options with appropriate settings.
Args: Args:
model_type: Model being loaded, used to pin its graph optimization level. model_type: Model being loaded, used to pin its graph optimization level.
cpu: Whether the session runs on the CPU execution provider.
Returns: Returns:
SessionOptions for enrichment models, or None for default settings. SessionOptions with a pinned optimization level, or None for default settings.
""" """
# Import here to avoid circular imports # Import here to avoid circular imports
from frigate.embeddings.types import ( from frigate.embeddings.types import EnrichmentModelTypeEnum
ENRICHMENT_FOR_MODEL_TYPE,
EnrichmentModelTypeEnum,
)
if model_type not in ENRICHMENT_FOR_MODEL_TYPE:
return None
sess_options = ort.SessionOptions()
if cpu:
# idle intra-op threads otherwise spin for ~1s after every run, keeping
# most cores busy while an enrichment runs even once a second
sess_options.add_session_config_entry("session.intra_op.allow_spinning", "0")
if model_type == EnrichmentModelTypeEnum.jina_v2.value: if model_type == EnrichmentModelTypeEnum.jina_v2.value:
# below EXTENDED the CUDA EP returns an identical vector for every image, # below EXTENDED the CUDA EP returns an identical vector for every image,
# and ORT_ENABLE_ALL fails to build on CPU with a SimplifiedLayerNormFusion error # and ORT_ENABLE_ALL fails to build on CPU with a SimplifiedLayerNormFusion error
sess_options.graph_optimization_level = ( level = ort.GraphOptimizationLevel.ORT_ENABLE_EXTENDED
ort.GraphOptimizationLevel.ORT_ENABLE_EXTENDED
)
elif model_type == EnrichmentModelTypeEnum.jina_v1.value: elif model_type == EnrichmentModelTypeEnum.jina_v1.value:
# aggressive optimizations create or expect nodes that don't exist # aggressive optimizations create or expect nodes that don't exist
sess_options.graph_optimization_level = ( level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
ort.GraphOptimizationLevel.ORT_ENABLE_BASIC else:
) return None
sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = level
return sess_options return sess_options
@@ -739,23 +689,6 @@ def get_optimized_runner(
if rknn_path: if rknn_path:
return _record_runner(model_path, model_type, RKNNModelRunner(rknn_path)) return _record_runner(model_path, model_type, RKNNModelRunner(rknn_path))
if device != "CPU" and (ane_devices := get_lighter_ane_devices()):
sess_options = get_ort_session_options(model_type) or ort.SessionOptions()
sess_options.add_provider_for_devices(ane_devices, {})
try:
session = ort.InferenceSession(model_path, sess_options=sess_options)
except Exception as e:
logger.warning(
f"Failed to load {model_path} on the Neural Engine, using the default providers: {e}"
)
else:
return _record_runner(
model_path,
model_type,
ONNXModelRunner(session, model_type=model_type),
)
providers, options = get_ort_providers(device == "CPU", device, **kwargs) providers, options = get_ort_providers(device == "CPU", device, **kwargs)
if providers[0] == "CPUExecutionProvider": if providers[0] == "CPUExecutionProvider":
@@ -805,9 +738,7 @@ def get_optimized_runner(
ONNXModelRunner( ONNXModelRunner(
ort.InferenceSession( ort.InferenceSession(
model_path, model_path,
sess_options=get_ort_session_options( sess_options=get_ort_session_options(model_type),
model_type, cpu=providers[0] == "CPUExecutionProvider"
),
providers=providers, providers=providers,
provider_options=options, provider_options=options,
), ),
+14 -30
View File
@@ -47,17 +47,21 @@ class ModelTypeEnum(str, Enum):
yologeneric = "yolo-generic" yologeneric = "yolo-generic"
# the scene of the model used by cameras that don't name one class SceneEnum(str, Enum):
DEFAULT_SCENE = "default" """The camera environment a detection model is intended for."""
SCENE_PATTERN = r"^[A-Za-z0-9_-]+$"
all = "all"
indoor = "indoor"
outdoor = "outdoor"
indoor_thermal = "indoor_thermal"
outdoor_thermal = "outdoor_thermal"
class ModelConfig(BaseModel): class ModelConfig(BaseModel):
scene: str = Field( scene: SceneEnum = Field(
default=DEFAULT_SCENE, default=SceneEnum.all,
pattern=SCENE_PATTERN,
title="Model scene", title="Model scene",
description="A name for the camera environment this model is used for, such as 'thermal'. Cameras select a model by setting detect.scene to a matching value, and the 'default' model is used by any camera that does not set one.", description="The camera environment this model is used for. Cameras select a model by setting detect.scene to a matching value, and 'all' is used by any camera that does not set one.",
) )
devices: list[str] = Field( devices: list[str] = Field(
default_factory=list, default_factory=list,
@@ -119,7 +123,6 @@ class ModelConfig(BaseModel):
_all_attributes: list[str] = PrivateAttr() _all_attributes: list[str] = PrivateAttr()
_all_attribute_logos: list[str] = PrivateAttr() _all_attribute_logos: list[str] = PrivateAttr()
_model_hash: str = PrivateAttr() _model_hash: str = PrivateAttr()
_plus_id: str | None = PrivateAttr(default=None)
@property @property
def merged_labelmap(self) -> dict[int, str]: def merged_labelmap(self) -> dict[int, str]:
@@ -145,11 +148,6 @@ class ModelConfig(BaseModel):
def model_hash(self) -> str: def model_hash(self) -> str:
return self._model_hash return self._model_hash
@property
def plus_id(self) -> str | None:
"""The Frigate+ model id, once a plus:// path has been resolved."""
return self._plus_id
def __init__(self, **config): def __init__(self, **config):
super().__init__(**config) super().__init__(**config)
@@ -180,34 +178,20 @@ class ModelConfig(BaseModel):
os.makedirs(MODEL_CACHE_DIR, exist_ok=True) os.makedirs(MODEL_CACHE_DIR, exist_ok=True)
model_id = self.path[7:] model_id = self.path[7:]
self._plus_id = model_id
self.path = os.path.join(MODEL_CACHE_DIR, model_id) self.path = os.path.join(MODEL_CACHE_DIR, model_id)
model_info_path = f"{self.path}.json" model_info_path = f"{self.path}.json"
# download the model if it doesn't exist # download the model if it doesn't exist
if not os.path.isfile(self.path): if not os.path.isfile(self.path):
try: download_url = plus_api.get_model_download_url(model_id)
download_url = plus_api.get_model_download_url(model_id) r = requests.get(download_url)
r = requests.get(download_url)
except requests.exceptions.ConnectionError as e:
raise ValueError(
f"Unable to connect to Frigate+ to download model {model_id}"
) from e
with open(self.path, "wb") as f: with open(self.path, "wb") as f:
f.write(r.content) f.write(r.content)
# download the model info if it doesn't exist # download the model info if it doesn't exist
if not os.path.isfile(model_info_path): if not os.path.isfile(model_info_path):
try:
model_info = plus_api.get_model_info(model_id)
except requests.exceptions.ConnectionError as e:
raise ValueError(
f"Unable to connect to Frigate+ to download model info for {model_id}"
) from e
with open(model_info_path, "w") as f: with open(model_info_path, "w") as f:
json.dump(model_info, f) json.dump(plus_api.get_model_info(model_id), f)
model_info = load_plus_model_info(model_id) model_info = load_plus_model_info(model_id)
-26
View File
@@ -25,7 +25,6 @@ SYS_ROOT = "/sys"
DEV_ROOT = "/dev" DEV_ROOT = "/dev"
PROC_ROOT = "/proc" PROC_ROOT = "/proc"
ETC_ROOT = "/etc" ETC_ROOT = "/etc"
LIB_ROOT = "/usr/lib"
# a Coral reports as Global Unichip until its firmware is loaded, then as Google # a Coral reports as Global Unichip until its firmware is loaded, then as Google
CORAL_USB_IDS = {("1a6e", "089a"), ("18d1", "9302")} CORAL_USB_IDS = {("1a6e", "089a"), ("18d1", "9302")}
@@ -274,16 +273,6 @@ def detect_memryx() -> DetectionHardware | None:
return _hardware("memryx", "memryx", "MemryX MX3", units) return _hardware("memryx", "memryx", "MemryX MX3", units)
def detect_deepx() -> DetectionHardware | None:
"""Find DEEPX NPUs by their device nodes."""
units = _dev_units("dxrt*", "deepx:PCIe:{index}", "PCIe")
if not units:
return None
return _hardware("deepx", "deepx", "DEEPX NPU", units)
def detect_rockchip() -> DetectionHardware | None: def detect_rockchip() -> DetectionHardware | None:
"""Find a Rockchip NPU by reading the SoC from the device tree.""" """Find a Rockchip NPU by reading the SoC from the device tree."""
compatible = _read(f"{PROC_ROOT}/device-tree/compatible") compatible = _read(f"{PROC_ROOT}/device-tree/compatible")
@@ -318,19 +307,6 @@ def detect_synaptics() -> DetectionHardware | None:
return _hardware("synaptics", "synaptics", "Synaptics NPU", units) return _hardware("synaptics", "synaptics", "Synaptics NPU", units)
def detect_lighter_ane() -> DetectionHardware | None:
"""Find a Mac's Neural Engine by the provider library lighter's device places."""
library = os.environ.get(
"LIGHTER_ANE_EP", f"{LIB_ROOT}/lighter/liblighter_ane_ep.so"
)
if not os.path.exists(library):
return None
# runs through onnx, whose session picks lighter's provider when it is present
units = [HardwareUnit(device="onnx", label="Neural Engine")]
return _hardware("onnx:lighter", "onnx", "Apple Neural Engine", units)
def detect_cpu() -> DetectionHardware: def detect_cpu() -> DetectionHardware:
"""The CPU, which is always available.""" """The CPU, which is always available."""
units = [HardwareUnit(device="cpu", label="CPU")] units = [HardwareUnit(device="cpu", label="CPU")]
@@ -343,7 +319,6 @@ PROBES = (
detect_coral_usb, detect_coral_usb,
detect_hailo, detect_hailo,
detect_memryx, detect_memryx,
detect_deepx,
detect_intel_npu, detect_intel_npu,
detect_intel_gpu, detect_intel_gpu,
detect_nvidia_gpu, detect_nvidia_gpu,
@@ -352,7 +327,6 @@ PROBES = (
detect_rockchip, detect_rockchip,
detect_axengine, detect_axengine,
detect_synaptics, detect_synaptics,
detect_lighter_ane,
detect_cpu, detect_cpu,
) )
File diff suppressed because it is too large Load Diff
+1 -2
View File
@@ -1,7 +1,7 @@
import json import json
import logging import logging
import os import os
from typing import Any, ClassVar, Literal from typing import Any, Literal
import numpy as np import numpy as np
import zmq import zmq
@@ -21,7 +21,6 @@ class ZmqDetectorConfig(BaseDetectorConfig):
model_config = ConfigDict( model_config = ConfigDict(
title="ZMQ IPC", title="ZMQ IPC",
) )
device_spec_field: ClassVar[str] = "endpoint"
type: Literal[DETECTOR_KEY] type: Literal[DETECTOR_KEY]
endpoint: str = Field( endpoint: str = Field(
-6
View File
@@ -10,7 +10,6 @@ from json.decoder import JSONDecodeError
from multiprocessing.synchronize import Event as MpEvent from multiprocessing.synchronize import Event as MpEvent
from typing import Any from typing import Any
import cv2
import regex import regex
from pathvalidate import ValidationError, sanitize_filename from pathvalidate import ValidationError, sanitize_filename
@@ -52,11 +51,6 @@ class EmbeddingProcess(FrigateProcess):
from .maintainer import EmbeddingMaintainer from .maintainer import EmbeddingMaintainer
self.pre_run_setup(self.config.logger) self.pre_run_setup(self.config.logger)
# frame conversions and crops here are one-off calls, and OpenCV's
# per-core thread pool burns ~8x the CPU of running them inline
cv2.setNumThreads(1)
maintainer = EmbeddingMaintainer( maintainer = EmbeddingMaintainer(
self.config, self.config,
self.metrics, self.metrics,
+40 -43
View File
@@ -6,10 +6,9 @@ import logging
import os import os
import threading import threading
import time import time
from typing import Any
import numpy as np import numpy as np
from peewee import DatabaseError, DoesNotExist, IntegrityError from peewee import DoesNotExist, IntegrityError
from PIL import Image from PIL import Image
from playhouse.shortcuts import model_to_dict from playhouse.shortcuts import model_to_dict
@@ -208,10 +207,12 @@ class Embeddings:
embedding = self.vision_embedding([thumbnail])[0] embedding = self.vision_embedding([thumbnail])[0]
if upsert: if upsert:
self.db.upsert_embeddings( self.db.execute_sql(
"vec_thumbnails", """
"thumbnail_embedding", INSERT OR REPLACE INTO vec_thumbnails(id, thumbnail_embedding)
{event_id: serialize(embedding)}, VALUES(?, ?)
""",
(event_id, serialize(embedding)),
) )
self.image_inference_speed.update(datetime.datetime.now().timestamp() - start) self.image_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -250,12 +251,19 @@ class Embeddings:
embeddings = self.vision_embedding(valid_thumbs) embeddings = self.vision_embedding(valid_thumbs)
if upsert: if upsert:
items = {} items = []
for i in range(len(valid_ids)): for i in range(len(valid_ids)):
items[valid_ids[i]] = serialize(embeddings[i]) items.append(valid_ids[i])
items.append(serialize(embeddings[i]))
self.image_eps.update() self.image_eps.update()
self.db.upsert_embeddings("vec_thumbnails", "thumbnail_embedding", items) self.db.execute_sql(
"""
INSERT OR REPLACE INTO vec_thumbnails(id, thumbnail_embedding)
VALUES {}
""".format(", ".join(["(?, ?)"] * len(valid_ids))),
items,
)
duration = datetime.datetime.now().timestamp() - start duration = datetime.datetime.now().timestamp() - start
self.image_inference_speed.update(duration / len(valid_ids)) self.image_inference_speed.update(duration / len(valid_ids))
@@ -269,10 +277,12 @@ class Embeddings:
embedding = self.text_embedding([description])[0] embedding = self.text_embedding([description])[0]
if upsert: if upsert:
self.db.upsert_embeddings( self.db.execute_sql(
"vec_descriptions", """
"description_embedding", INSERT OR REPLACE INTO vec_descriptions(id, description_embedding)
{event_id: serialize(embedding)}, VALUES(?, ?)
""",
(event_id, serialize(embedding)),
) )
self.text_inference_speed.update(datetime.datetime.now().timestamp() - start) self.text_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -292,14 +302,19 @@ class Embeddings:
if upsert: if upsert:
ids = list(event_descriptions.keys()) ids = list(event_descriptions.keys())
items = {} items = []
for i in range(len(ids)): for i in range(len(ids)):
items[ids[i]] = serialize(embeddings[i]) items.append(ids[i])
items.append(serialize(embeddings[i]))
self.text_eps.update() self.text_eps.update()
self.db.upsert_embeddings( self.db.execute_sql(
"vec_descriptions", "description_embedding", items """
INSERT OR REPLACE INTO vec_descriptions(id, description_embedding)
VALUES {}
""".format(", ".join(["(?, ?)"] * len(ids))),
items,
) )
self.text_inference_speed.update(datetime.datetime.now().timestamp() - start) self.text_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -307,17 +322,6 @@ class Embeddings:
return embeddings return embeddings
def reindex(self) -> None: def reindex(self) -> None:
"""Rebuild every tracked object embedding from scratch."""
totals: dict[str, Any] = {"status": "indexing"}
try:
self._reindex(totals)
except DatabaseError:
logger.exception("Unable to reindex tracked object embeddings")
totals["status"] = "failed"
self.requestor.send_data(UPDATE_EMBEDDINGS_REINDEX_PROGRESS, totals)
def _reindex(self, totals: dict[str, Any]) -> None:
logger.info("Indexing tracked object embeddings...") logger.info("Indexing tracked object embeddings...")
self.db.drop_embeddings_tables() self.db.drop_embeddings_tables()
@@ -342,24 +346,17 @@ class Embeddings:
batch_size = 32 batch_size = 32
current_page = 1 current_page = 1
totals.update( totals = {
{ "thumbnails": 0,
"thumbnails": 0, "descriptions": 0,
"descriptions": 0, "processed_objects": total_events - 1 if total_events < batch_size else 0,
"processed_objects": total_events - 1 "total_objects": total_events,
if total_events < batch_size "time_remaining": 0 if total_events < batch_size else -1,
else 0, "status": "indexing",
"total_objects": total_events, }
"time_remaining": 0 if total_events < batch_size else -1,
"status": "indexing",
}
)
self.requestor.send_data(UPDATE_EMBEDDINGS_REINDEX_PROGRESS, totals) self.requestor.send_data(UPDATE_EMBEDDINGS_REINDEX_PROGRESS, totals)
# a single batch sends no progress, so the first message shows it nearly done
totals["processed_objects"] = 0
events = ( events = (
Event.select() Event.select()
.order_by(Event.start_time.desc()) .order_by(Event.start_time.desc())
+12 -36
View File
@@ -11,11 +11,7 @@ from typing import Any
from peewee import DoesNotExist from peewee import DoesNotExist
from frigate.comms.config_updater import ConfigSubscriber from frigate.comms.config_updater import ConfigSubscriber
from frigate.comms.detections_updater import ( from frigate.comms.detections_updater import DetectionSubscriber, DetectionTypeEnum
DetectionPublisher,
DetectionSubscriber,
DetectionTypeEnum,
)
from frigate.comms.embeddings_updater import ( from frigate.comms.embeddings_updater import (
EmbeddingsRequestEnum, EmbeddingsRequestEnum,
EmbeddingsResponder, EmbeddingsResponder,
@@ -72,7 +68,7 @@ from frigate.events.types import (
RegenerateDescriptionEnum, RegenerateDescriptionEnum,
) )
from frigate.genai import GenAIClientManager from frigate.genai import GenAIClientManager
from frigate.models import Event, Recordings, ReviewSegment, Timeline, Trigger from frigate.models import Event, Recordings, ReviewSegment, Trigger
from frigate.types import TrackedObjectUpdateTypesEnum from frigate.types import TrackedObjectUpdateTypesEnum
from frigate.util.builtin import serialize from frigate.util.builtin import serialize
from frigate.util.file import get_event_thumbnail_bytes from frigate.util.file import get_event_thumbnail_bytes
@@ -143,7 +139,7 @@ class EmbeddingMaintainer(threading.Thread):
), ),
load_vec_extension=True, load_vec_extension=True,
) )
models = [Event, Recordings, ReviewSegment, Timeline, Trigger] models = [Event, Recordings, ReviewSegment, Trigger]
db.bind(models) db.bind(models)
self.genai_manager = GenAIClientManager(config) self.genai_manager = GenAIClientManager(config)
@@ -172,7 +168,6 @@ class EmbeddingMaintainer(threading.Thread):
) )
self.review_subscriber = ReviewDataSubscriber("") self.review_subscriber = ReviewDataSubscriber("")
self.detection_subscriber = DetectionSubscriber(DetectionTypeEnum.video.value) self.detection_subscriber = DetectionSubscriber(DetectionTypeEnum.video.value)
self.detection_publisher = DetectionPublisher(DetectionTypeEnum.all.value)
self.embeddings_responder = EmbeddingsResponder() self.embeddings_responder = EmbeddingsResponder()
self.frame_manager = SharedMemoryFrameManager() self.frame_manager = SharedMemoryFrameManager()
@@ -256,11 +251,7 @@ class EmbeddingMaintainer(threading.Thread):
): ):
self.post_processors.append( self.post_processors.append(
AudioTranscriptionPostProcessor( AudioTranscriptionPostProcessor(
self.config, self.config, self.requestor, self.embeddings, metrics
self.requestor,
self.embeddings,
metrics,
self.genai_manager,
) )
) )
@@ -332,9 +323,6 @@ class EmbeddingMaintainer(threading.Thread):
"""Apply camera config updates and register newly enabled processors.""" """Apply camera config updates and register newly enabled processors."""
updated_topics = self.config_updater.check_for_updates() updated_topics = self.config_updater.check_for_updates()
for camera in updated_topics.get("remove", []):
self.frame_manager.close_camera(camera)
if updated_topics.keys() & GENAI_UPDATE_TOPICS: if updated_topics.keys() & GENAI_UPDATE_TOPICS:
self._sync_genai_processors() self._sync_genai_processors()
@@ -364,7 +352,6 @@ class EmbeddingMaintainer(threading.Thread):
self.event_end_subscriber.stop() self.event_end_subscriber.stop()
self.recordings_subscriber.stop() self.recordings_subscriber.stop()
self.detection_subscriber.stop() self.detection_subscriber.stop()
self.detection_publisher.stop()
self.event_metadata_publisher.stop() self.event_metadata_publisher.stop()
self.event_metadata_subscriber.stop() self.event_metadata_subscriber.stop()
self.embeddings_responder.stop() self.embeddings_responder.stop()
@@ -520,7 +507,7 @@ class EmbeddingMaintainer(threading.Thread):
if update is None: if update is None:
return return
source_type, event_type, camera, frame_name, data, frame_time = update source_type, event_type, camera, frame_name, data = update
logger.debug( logger.debug(
f"Received update - source_type: {source_type}, camera: {camera}, data label: {data.get('label') if data else 'None'}" f"Received update - source_type: {source_type}, camera: {camera}, data label: {data.get('label') if data else 'None'}"
@@ -549,7 +536,7 @@ class EmbeddingMaintainer(threading.Thread):
# Create our own thumbnail based on the bounding box and the frame time # Create our own thumbnail based on the bounding box and the frame time
try: try:
yuv_frame = self.frame_manager.get( yuv_frame = self.frame_manager.get(
frame_name, camera_config.frame_shape_yuv, frame_time frame_name, camera_config.frame_shape_yuv
) )
except FileNotFoundError: except FileNotFoundError:
logger.debug(f"Frame {frame_name} not found for camera {camera}") logger.debug(f"Frame {frame_name} not found for camera {camera}")
@@ -586,6 +573,8 @@ class EmbeddingMaintainer(threading.Thread):
PostProcessDataEnum.tracked_object, PostProcessDataEnum.tracked_object,
) )
self.frame_manager.close(frame_name)
def _process_finalized(self) -> None: def _process_finalized(self) -> None:
"""Process the end of an event.""" """Process the end of an event."""
while True: while True:
@@ -794,7 +783,7 @@ class EmbeddingMaintainer(threading.Thread):
if topic is None: if topic is None:
return return
camera, frame_name, frame_time, _, motion_boxes, _ = data camera, frame_name, _, _, motion_boxes, _ = data
if not camera or camera not in self.config.cameras: if not camera or camera not in self.config.cameras:
return return
@@ -818,7 +807,7 @@ class EmbeddingMaintainer(threading.Thread):
try: try:
yuv_frame = self.frame_manager.get( yuv_frame = self.frame_manager.get(
frame_name, camera_config.frame_shape_yuv, frame_time frame_name, camera_config.frame_shape_yuv
) )
except FileNotFoundError: except FileNotFoundError:
pass pass
@@ -842,6 +831,8 @@ class EmbeddingMaintainer(threading.Thread):
{"camera": camera, "motion": motion_boxes}, yuv_frame {"camera": camera, "motion": motion_boxes}, yuv_frame
) )
self.frame_manager.close(frame_name)
def _process_deferred_results(self) -> None: def _process_deferred_results(self) -> None:
"""Drain results from deferred processors and perform IPC side-effects.""" """Drain results from deferred processors and perform IPC side-effects."""
for processor in self.realtime_processors: for processor in self.realtime_processors:
@@ -856,21 +847,6 @@ class EmbeddingMaintainer(threading.Thread):
f"{result['camera']}/classification/{result['model_name']}", f"{result['camera']}/classification/{result['model_name']}",
result["state"], result["state"],
) )
# the first state verified after startup is not a change
if result["previous_state"] is not None:
self.detection_publisher.publish(
(
result["camera"],
{
"model": result["model_name"],
"from": result["previous_state"],
"to": result["state"],
"timestamp": result["timestamp"],
},
),
DetectionTypeEnum.classification_state.value,
)
elif result["processor"] == "object": elif result["processor"] == "object":
object_id = result["object_id"] object_id = result["object_id"]
camera = result["camera"] camera = result["camera"]
+13 -13
View File
@@ -2,10 +2,10 @@
import logging import logging
import os import os
import shutil
import threading import threading
import warnings
from transformers import AutoTokenizer, CLIPImageProcessorPil from transformers import AutoFeatureExtractor, AutoTokenizer
from transformers.utils.logging import disable_progress_bar from transformers.utils.logging import disable_progress_bar
from frigate.comms.inter_process import InterProcessRequestor from frigate.comms.inter_process import InterProcessRequestor
@@ -21,6 +21,12 @@ from frigate.util.downloader import ModelDownloader
from .base_embedding import BaseEmbedding from .base_embedding import BaseEmbedding
warnings.filterwarnings(
"ignore",
category=FutureWarning,
message="The class CLIPFeatureExtractor is deprecated",
)
# disables the progress bar for downloading tokenizers and feature extractors # disables the progress bar for downloading tokenizers and feature extractors
disable_progress_bar() disable_progress_bar()
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -52,13 +58,6 @@ class JinaV1TextEmbedding(BaseEmbedding):
self._lock = threading.Lock() self._lock = threading.Lock()
files_names = list(self.download_urls.keys()) + [self.tokenizer_file] files_names = list(self.download_urls.keys()) + [self.tokenizer_file]
# an interrupted download leaves the hub cache without the saved tokenizer
tokenizer_path = os.path.join(self.download_path, self.tokenizer_file)
if os.path.isdir(tokenizer_path) and not os.path.exists(
os.path.join(tokenizer_path, "tokenizer_config.json")
):
shutil.rmtree(tokenizer_path)
if not all( if not all(
os.path.exists(os.path.join(self.download_path, n)) for n in files_names os.path.exists(os.path.join(self.download_path, n)) for n in files_names
): ):
@@ -93,7 +92,7 @@ class JinaV1TextEmbedding(BaseEmbedding):
tokenizer = AutoTokenizer.from_pretrained( tokenizer = AutoTokenizer.from_pretrained(
self.model_name, self.model_name,
trust_remote_code=False, trust_remote_code=True,
cache_dir=f"{MODEL_CACHE_DIR}/{self.model_name}/tokenizer", cache_dir=f"{MODEL_CACHE_DIR}/{self.model_name}/tokenizer",
clean_up_tokenization_spaces=True, clean_up_tokenization_spaces=True,
) )
@@ -124,8 +123,9 @@ class JinaV1TextEmbedding(BaseEmbedding):
f"{MODEL_CACHE_DIR}/{self.model_name}/tokenizer" f"{MODEL_CACHE_DIR}/{self.model_name}/tokenizer"
) )
self.tokenizer = AutoTokenizer.from_pretrained( self.tokenizer = AutoTokenizer.from_pretrained(
tokenizer_path, self.model_name,
trust_remote_code=False, cache_dir=tokenizer_path,
trust_remote_code=True,
clean_up_tokenization_spaces=True, clean_up_tokenization_spaces=True,
) )
@@ -209,7 +209,7 @@ class JinaV1ImageEmbedding(BaseEmbedding):
if self.downloader: if self.downloader:
self.downloader.wait_for_download() self.downloader.wait_for_download()
self.feature_extractor = CLIPImageProcessorPil.from_pretrained( self.feature_extractor = AutoFeatureExtractor.from_pretrained(
f"{MODEL_CACHE_DIR}/{self.model_name}", f"{MODEL_CACHE_DIR}/{self.model_name}",
) )

Some files were not shown because too many files have changed in this diff Show More