add default icon, observe measure div, fix tests

* fix record status never returning online after a record ffmpeg restart

The record ffmpeg restart paths sent `offline` directly, so the cached record status still read `online` and the recovery was never published. Detect and record also shared one resend timestamp, and detect's resend always ran first, so record's periodic resend never fired either. Record's offline now goes through the cache and each status has its own timestamp.

* don't publish record online when the record process has exited

.cspell/frigate-dictionary.txt

@@ -55,7 +55,6 @@ Dahua
 datasheet
 debconf
 deci
-deepstack
 defragment
 devcontainer
 DEVICEMAP

.github/DISCUSSION_TEMPLATE/beta-support.yml

@@ -26,8 +26,8 @@ body:
     id: version
     attributes:
       label: Beta Version
-      description: Visible on the System Metrics page in the Web UI. Please include the full version including the build identifier (eg. 0.18.0-beta1, 0.18.0-8b72c7a, etc.)
-      placeholder: "0.18.0-beta1"
+      description: Visible on the System Metrics page in the Web UI. Please include the full version including the build identifier (eg. 0.19.0-beta1, 0.19.0-8b72c7a, etc.)
+      placeholder: "0.19.0-beta1"
     validations:
       required: true
   - type: dropdown

.github/DISCUSSION_TEMPLATE/report-a-bug.yml

@@ -6,7 +6,9 @@ body:
       value: |
         Use this form to submit a reproducible bug in Frigate or Frigate's UI.
 
-        **⚠️ If you are running a beta version (0.18.0-beta or similar), please use the [Beta Support template](https://github.com/blakeblackshear/frigate/discussions/new?category=beta-support) instead.**
+        If you are running on Proxmox, please see the [Proxmox FAQ](https://github.com/blakeblackshear/frigate/discussions/23916) and reproduce the issue on a standard Docker install first (bare metal, or a VM running plain Debian/Ubuntu) before submitting here.
+
+        **⚠️ If you are running a beta version (0.19.0-beta or similar), please use the [Beta Support template](https://github.com/blakeblackshear/frigate/discussions/new?category=beta-support) instead.**
 
         Before submitting your bug report, please ask the AI with the "Ask AI" button on the [official documentation site][ai] about your issue, [search the discussions][discussions], look at recent open and closed [pull requests][prs], read the [official Frigate documentation][docs], and read the [Frigate FAQ][faq] pinned at the Discussion page to see if your bug has already been fixed by the developers or reported by the community.
 

.github/actions/setup/action.yml

@@ -17,9 +17,14 @@ runs:
       shell: bash
     # This creates a virtual volume at /var/lib/docker to maximize the size
     # As of 2/14/2024, this results in 97G for docker images
+    # Runners no longer have a separate /mnt disk, so the temp PV is also carved
+    # from root and temp-reserve-mb is what actually stays free on root. Keep 4G
+    # there for setup-qemu/buildx caches in ~/.docker and the tool cache
     - name: Maximize build space
       uses: easimon/maximize-build-space@master
       with:
+        root-reserve-mb: 8192
+        temp-reserve-mb: 4096
         remove-dotnet: 'true'
         remove-android: 'true'
         remove-haskell: 'true'

.github/workflows/ci.yml

@@ -23,7 +23,7 @@ jobs:
     name: AMD64 Build
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -49,7 +49,7 @@ jobs:
       - amd64_build
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -144,9 +144,11 @@ jobs:
             done'
       - name: Assert device access grants
         run: |
-          # a fake accelerator node created after boot, then the oneshot re-run
+          # a fake accelerator node created after boot, then the oneshot re-run.
+          # /command is on PATH only for s6-supervised services, and the
+          # with-contenv shebang resolves its execline helpers through PATH
           docker exec frigate mknod /dev/apex_9 c 120 99
-          docker exec frigate /etc/s6-overlay/s6-rc.d/init-devices/run
+          docker exec frigate sh -c 'export PATH=/command:$PATH; exec /etc/s6-overlay/s6-rc.d/init-devices/run'
           acl=$(docker exec frigate getfacl -p /dev/apex_9)
           echo "$acl"
           echo "$acl" | grep -q "user:frigate:rw-"
@@ -154,7 +156,7 @@ jobs:
           # the usb tree gets recursive grants plus a default ACL that
           # newly created nodes inherit (the Coral re-enumeration path)
           docker exec frigate sh -c 'mkdir -p /dev/bus/usb/001 && mknod /dev/bus/usb/001/002 c 189 1'
-          docker exec frigate /etc/s6-overlay/s6-rc.d/init-devices/run
+          docker exec frigate sh -c 'export PATH=/command:$PATH; exec /etc/s6-overlay/s6-rc.d/init-devices/run'
           docker exec frigate getfacl -p /dev/bus/usb/001 | grep -q "user:frigate:rwx"
           docker exec frigate sh -c 'mknod /dev/bus/usb/001/099 c 189 98 && chmod 664 /dev/bus/usb/001/099'
           inherited=$(docker exec frigate getfacl -p /dev/bus/usb/001/099)
@@ -171,7 +173,7 @@ jobs:
           # hardware that is absent must stay silent: the literal table entries
           # are not globs, so nullglob does not drop them and only an existence
           # check keeps them from warning on every boot
-          out=$(docker exec frigate /etc/s6-overlay/s6-rc.d/init-devices/run)
+          out=$(docker exec frigate sh -c 'export PATH=/command:$PATH; exec /etc/s6-overlay/s6-rc.d/init-devices/run')
           echo "$out"
           if echo "$out" | grep -q "WARN"; then
             echo "grant warned about device nodes that do not exist"; exit 1
@@ -301,7 +303,7 @@ jobs:
           # /run must allow exec: S6_READ_ONLY_ROOT has s6 copy its service
           # scripts there and run them, and --tmpfs defaults to noexec
           docker run -d --name frigate-ro --shm-size 256m \
-            --read-only --tmpfs /tmp:rw,size=1g --tmpfs /run:exec,nosuid,nodev,mode=0755 \
+            --read-only --tmpfs /tmp:rw,size=1g --tmpfs /run:exec,nosuid,nodev,mode=0755,uid=1000,gid=1000 \
             --user 1000:1000 \
             --security-opt no-new-privileges:true \
             -v /tmp/frigate-config-ro:/config \
@@ -395,7 +397,7 @@ jobs:
           # setfacl under a read-only rootfs, which nothing else covers:
           # init-devices exits early under --user, so that path is never reached
           docker exec frigate-rod mknod /dev/apex_9 c 120 99
-          docker exec frigate-rod /etc/s6-overlay/s6-rc.d/init-devices/run
+          docker exec frigate-rod sh -c 'export PATH=/command:$PATH; exec /etc/s6-overlay/s6-rc.d/init-devices/run'
           docker exec frigate-rod getfacl -p /dev/apex_9 | grep -q "user:frigate:rw-"
           docker rm -f frigate-rod
       - name: "Assert switching that install to user: still starts"
@@ -403,7 +405,7 @@ jobs:
           # the config dir above now holds a go2rtc-owned go2rtc_homekit.yml,
           # which user: keeps readable but not writable (no supplementary groups)
           docker run -d --name frigate-rod-user --shm-size 256m \
-            --read-only --tmpfs /tmp:rw,size=1g --tmpfs /run:exec,nosuid,nodev,mode=0755 \
+            --read-only --tmpfs /tmp:rw,size=1g --tmpfs /run:exec,nosuid,nodev,mode=0755,uid=1000,gid=1000 \
             --user 1000:1000 \
             -v /tmp/frigate-config-rod:/config \
             -v /tmp/frigate-media-rod:/media/frigate \
@@ -424,7 +426,7 @@ jobs:
     name: ARM Build
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -459,7 +461,7 @@ jobs:
     name: Jetson Jetpack 6
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -490,7 +492,7 @@ jobs:
       - amd64_build
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -531,7 +533,7 @@ jobs:
       - arm64_build
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -556,7 +558,7 @@ jobs:
       - arm64_build
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up QEMU and Buildx
@@ -588,7 +590,7 @@ jobs:
         with:
           string: ${{ github.repository }}
       - name: Log in to the Container registry
-        uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
         with:
           registry: ghcr.io
           username: ${{ github.actor }}

.github/workflows/pull_request.yml

@@ -16,10 +16,10 @@ jobs:
     name: Web - Lint
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 20.x
       - run: npm install
@@ -35,10 +35,10 @@ jobs:
     name: Web - Test
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 20.x
       - run: npm install
@@ -46,18 +46,15 @@ jobs:
       - name: Build web
         run: npm run build
         working-directory: ./web
-      # - name: Test
-      #   run: npm run test
-      #   working-directory: ./web
 
   web_e2e:
     name: Web - E2E Tests
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 20.x
       - run: npm install
@@ -86,11 +83,11 @@ jobs:
     name: Python Checks
     steps:
       - name: Check out the repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
       - name: Set up Python ${{ env.DEFAULT_PYTHON }}
-        uses: actions/setup-python@v5.4.0
+        uses: actions/setup-python@v7.0.0
         with:
           python-version: ${{ env.DEFAULT_PYTHON }}
       - name: Install requirements
@@ -109,10 +106,10 @@ jobs:
     name: Python Tests
     steps:
       - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           persist-credentials: false
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 20.x
       - name: Install devcontainer cli

.github/workflows/release.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
       - id: lowercaseRepo
@@ -18,7 +18,7 @@ jobs:
         with:
           string: ${{ github.repository }}
       - name: Log in to the Container registry
-        uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
         with:
           registry: ghcr.io
           username: ${{ github.actor }}

AGENTS.md

@@ -160,7 +160,7 @@ When reviewing code, do NOT comment on:
 
 ### Code Quality
 
-- **Linting**: ESLint (see `web/.eslintrc.cjs`)
+- **Linting**: ESLint (see `web/eslint.config.js`)
 - **Formatting**: Prettier with Tailwind CSS plugin
 - **Type Safety**: TypeScript strict mode enabled
 

CODEOWNERS

@@ -5,3 +5,4 @@
 /docker/rockchip/ @MarcA711
 /docker/rocm/ @harakas
 /docker/hailo8l/ @spanner3003
+/docker/deepx/ @sixfab

docker/deepx/user_installation.sh

@@ -0,0 +1,131 @@
+#!/bin/bash
+
+# Installs the DEEPX NPU kernel driver and the DX-RT runtime on the Docker host,
+# then enables the vendor's dxrt.service. A container cannot load kernel
+# modules, so this runs outside the image; the driver creates the /dev/dxrt*
+# nodes and the daemon multiplexes the NPU across host and container.
+#
+# Driver, runtime and firmware versions must agree or inference hangs instead
+# of failing at startup. The set this script installs is pinned in
+# driver_version, runtime_version and firmware_version below; move them
+# together, never one at a time.
+#
+# DEEPX NPU support in Frigate is maintained by Sixfab (https://sixfab.com).
+
+set -euo pipefail
+
+driver_version="v2.6.0"
+# the commit the tag resolves to, since DEEPX signs neither tags nor releases
+# and this is compiled and installed as root. Update both together
+driver_commit="7074748e7104f470b02f517583abba652b3f05fa"
+firmware_version="v2.7.4"
+
+sudo apt-get update
+sudo apt-get install -y git build-essential "linux-headers-$(uname -r)" pciutils wget
+
+if ! lspci -d 1ff4: | grep -q .; then
+    echo "No DEEPX device found on the PCIe bus (lspci -d 1ff4:)."
+    echo "Check that the module is seated correctly before continuing."
+    exit 1
+fi
+
+# fetch the pinned commit rather than cloning the tag, so a retag cannot swap
+# in different source. The build directory is reused so a second run after a
+# failure does not stop on the directory already being there
+mkdir -p dx_rt_npu_linux_driver
+cd dx_rt_npu_linux_driver
+git init -q
+git remote get-url origin > /dev/null 2>&1 ||
+    git remote add origin https://github.com/DEEPX-AI/dx_rt_npu_linux_driver.git
+git fetch --depth 1 origin "${driver_commit}"
+git checkout -q FETCH_HEAD
+
+fetched_commit=$(git rev-parse HEAD)
+if [[ "${fetched_commit}" != "${driver_commit}" ]]; then
+    echo "Fetched commit ${fetched_commit} does not match pinned driver_commit ${driver_commit}."
+    echo "Refusing to build unverified driver source."
+    exit 1
+fi
+
+cd modules
+
+sudo ./build.sh -c install --reload
+
+sudo depmod -A
+
+# dx_dma is the PCIe transport, dxrt_driver the NPU driver on top of it
+for module in dx_dma dxrt_driver; do
+    if ! sudo modprobe "${module}"; then
+        echo "Unable to load the ${module} kernel module, common reasons are:"
+        echo "- Secure Boot is enabled and is rejecting the unsigned module."
+        echo "- The running kernel does not match the installed linux-headers."
+        exit 1
+    fi
+done
+
+if ! compgen -G "/dev/dxrt*" > /dev/null; then
+    echo "Modules loaded but no /dev/dxrt* device node appeared."
+    echo "Run ./sanity_check.sh from the driver repo to diagnose."
+    exit 1
+fi
+
+runtime_version="v3.4.0"
+declare -A runtime_sha256=(
+    [amd64]="736cfef009ce9e974ab1ab610d867239d19d72a426a53e367ddcbd53297b6e20"
+    [arm64]="eb6107f5f02f2ad76ae89f414e8b5f346f34fbc6f0888236136853a26be6f6a0"
+)
+
+runtime_release="${runtime_version#v}"
+deb_arch=$(dpkg --print-architecture)
+deb_file="/tmp/libdxrt-bin_${runtime_release}_${deb_arch}.deb"
+
+wget -qO "${deb_file}" \
+    "https://raw.githubusercontent.com/DEEPX-AI/dx_rt/${runtime_version}/release/${runtime_release}/libdxrt-bin_${runtime_release}_${deb_arch}.deb"
+
+expected_sha256="${runtime_sha256[${deb_arch}]:-}"
+if [[ -z "${expected_sha256}" ]]; then
+    echo "No pinned SHA-256 for architecture ${deb_arch}; refusing to install."
+    exit 1
+fi
+if [[ "$(sha256sum "${deb_file}" | cut -d' ' -f1)" != "${expected_sha256}" ]]; then
+    echo "SHA-256 mismatch for ${deb_file}; refusing to install."
+    exit 1
+fi
+
+sudo dpkg -i "${deb_file}"
+sudo ldconfig
+rm -f "${deb_file}"
+
+sudo cp /usr/share/libdxrt-bin/service/dxrt.service /etc/systemd/system/
+
+# With an endpoint set, dxrtd binds that path only, so the socket goes in a
+# directory Frigate can mount (kept across restarts so the mount stays valid)
+# and a symlink at the default /tmp path keeps host tools that do not set the
+# variable working through their own fallback.
+sudo mkdir -p /etc/systemd/system/dxrt.service.d
+sudo tee /etc/systemd/system/dxrt.service.d/frigate.conf > /dev/null <<'UNIT'
+[Service]
+RuntimeDirectory=dxrt
+RuntimeDirectoryMode=0755
+RuntimeDirectoryPreserve=yes
+Environment=DXRT_DYNAMIC_IPC_ENDPOINT=/run/dxrt/dxrt_dynamic_ipc.sock
+ExecStartPost=/bin/ln -sfn /run/dxrt/dxrt_dynamic_ipc.sock /tmp/dxrt_dynamic_ipc.sock
+UNIT
+
+sudo systemctl daemon-reload
+sudo systemctl enable dxrt.service
+sudo systemctl restart dxrt.service
+
+if ! sudo systemctl is-active --quiet dxrt.service; then
+    echo "dxrt.service did not start. Check: sudo journalctl -u dxrt.service"
+    exit 1
+fi
+
+echo "DEEPX driver and runtime installation complete."
+echo "Driver version:   $(modinfo -F version dxrt_driver) (expected ${driver_version#v})"
+echo "Runtime version:  ${runtime_release}"
+echo "Device node(s):   $(echo /dev/dxrt*)"
+echo
+echo "This driver expects NPU firmware ${firmware_version}. Check it with:"
+echo "  dxrt-cli --status"
+echo "Update the module if it does not match before starting Frigate."

docker/main/requirements-dev.txt

@@ -1,4 +1,4 @@
 ruff == 0.15.20
 
 # types
-types-peewee == 3.17.*
+types-peewee == 4.0.*

docker/main/requirements-wheels.txt

@@ -1,17 +1,17 @@
-aiofiles == 24.1.*
-click == 8.1.*
+aiofiles == 25.1.*
+click == 8.5.*
 # FastAPI
 aiohttp == 3.12.*
 starlette == 0.47.*
-starlette-context == 0.4.*
+starlette-context == 0.5.*
 fastapi[standard-no-fastapi-cloud-cli] == 0.116.*
-uvicorn == 0.35.*
+uvicorn == 0.52.*
 slowapi == 0.1.*
-joserfc == 1.2.*
-cryptography == 44.0.*
+joserfc == 1.6.*
+cryptography == 46.0.*
 pathvalidate == 3.3.*
 markupsafe == 3.0.*
-python-multipart == 0.0.26
+python-multipart == 0.0.31
 # Classification Model Training
 tensorflow == 2.19.* ; platform_machine == 'aarch64'
 tensorflow-cpu == 2.19.* ; platform_machine == 'x86_64'
@@ -26,15 +26,15 @@ psutil == 7.1.*
 pydantic == 2.10.*
 git+https://github.com/fbcotter/py3nvml#egg=py3nvml
 pytz == 2025.*
-pyzmq == 26.2.*
-ruamel.yaml == 0.18.*
+pyzmq == 27.1.*
+ruamel.yaml == 0.19.*
 tzlocal == 5.2
-requests == 2.32.*
+requests == 2.33.*
 types-requests == 2.32.*
 norfair == 2.3.*
 setproctitle == 1.3.*
 ws4py == 0.5.*
-unidecode == 1.3.*
+unidecode == 1.4.*
 titlecase == 2.4.*
 # Image Manipulation
 numpy == 1.26.*
@@ -43,7 +43,7 @@ opencv-contrib-python == 4.11.0.*
 scipy == 1.16.*
 # OpenVino & ONNX
 openvino == 2025.4.*
-onnxruntime == 1.22.*
+onnxruntime == 1.30.*
 # Embeddings
 transformers == 4.45.*
 # Generative AI
@@ -51,26 +51,26 @@ google-genai == 1.58.*
 ollama == 0.6.*
 openai == 1.65.*
 # push notifications
-py-vapid == 1.9.*
+py-vapid == 1.9.4
 pywebpush == 2.0.*
 # alpr
-pyclipper == 1.3.*
+pyclipper == 1.4.*
 shapely == 2.0.*
 rapidfuzz==3.12.*
 # HailoRT
-argcomplete==2.0.*
+argcomplete==3.7.*
 contextlib2==0.6.*
 future==0.18.*
-netaddr==0.8.*
+netaddr==1.3.*
 netifaces==0.10.*
-prometheus-client == 0.21.*
+prometheus-client == 0.26.*
 # TFLite
 tflite_runtime @ https://github.com/frigate-nvr/TFlite-builds/releases/download/v2.17.1/tflite_runtime-2.17.1-cp311-cp311-linux_x86_64.whl; platform_machine == 'x86_64'
 tflite_runtime @ https://github.com/feranick/TFlite-builds/releases/download/v2.17.1/tflite_runtime-2.17.1-cp311-cp311-linux_aarch64.whl; platform_machine == 'aarch64'
 # audio transcription
-sherpa-onnx==1.12.*
-faster-whisper==1.1.*
+sherpa-onnx==1.13.*
+faster-whisper==1.2.*
 librosa==0.11.*
 soundfile==0.13.*
 # Memory profiling
-memray == 1.15.*
+memray == 1.20.*

docker/main/rootfs/etc/s6-overlay/s6-rc.d/init-devices/run

@@ -37,6 +37,7 @@ device_globs=(
     "/dev/nvmap"
     "/dev/nvidia*"
     "/dev/memx*"
+    "/dev/dxrt*"
 )
 
 IFS=',' read -ra extra_globs <<< "${DEVICE_ACL_PATHS:-}"

docker/main/rootfs/usr/local/go2rtc/create_config.py

@@ -15,6 +15,7 @@ from frigate.const import (
 )
 from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode
 from frigate.util.config import find_config_file, resolve_ffmpeg_path
+from frigate.util.live_streams import raw_transcode_streams
 from frigate.util.services import (
     is_go2rtc_arbitrary_exec_allowed,
     is_restricted_go2rtc_source,
@@ -174,6 +175,17 @@ for name in list(go2rtc_config.get("streams", {})):
             del go2rtc_config["streams"][name]
             continue
 
+# add transcoded live streams; a user stream with the same name wins here and
+# fails Frigate's config validation
+transcoded_streams = raw_transcode_streams(config)
+
+if transcoded_streams:
+    if go2rtc_config.get("streams") is None:
+        go2rtc_config["streams"] = {}
+
+    for name, source in transcoded_streams.items():
+        go2rtc_config["streams"].setdefault(name, source)
+
 # add birdseye restream stream if enabled
 if config.get("birdseye", {}).get("restream", False):
     birdseye: dict[str, Any] = config.get("birdseye")

docker/main/rootfs/usr/local/nginx/conf/nginx.conf

@@ -166,7 +166,9 @@ http {
             include auth_request.conf;
             types {
                 video/mp4 mp4;
-                image/jpeg jpg;
+                image/jpeg jpg jpeg;
+                image/png png;
+                image/webp webp;
             }
 
             expires 7d;
@@ -341,13 +343,6 @@ http {
                 add_header Cache-Control "public";
             }
 
-            location /fonts/ {
-                access_log off;
-                expires 1y;
-                include security_headers.conf;
-                add_header Cache-Control "public";
-            }
-
             location /locales/ {
                 access_log off;
                 include security_headers.conf;
@@ -374,7 +369,7 @@ http {
             sub_filter '"/BASE_PATH/assets/' '"$http_x_ingress_path/assets/';
             sub_filter '"/BASE_PATH/locales/' '"$http_x_ingress_path/locales/';
             sub_filter '"/BASE_PATH/monacoeditorwork/' '"$http_x_ingress_path/assets/';
-            sub_filter 'return"/BASE_PATH/"' 'return window.baseUrl';
+            sub_filter 'return`/BASE_PATH/`' 'return window.baseUrl';
             sub_filter '<body>' '<body><script>window.baseUrl="$http_x_ingress_path/";</script>';
             sub_filter_types text/css application/javascript;
             sub_filter_once off;

docker/tensorrt/requirements-amd64.txt

@@ -1,6 +1,5 @@
 # Nvidia ONNX Runtime GPU Support
 --extra-index-url 'https://pypi.nvidia.com'
-cython==3.0.*; platform_machine == 'x86_64'
 nvidia-cuda-cupti-cu12==12.8.90; platform_machine == 'x86_64'
 nvidia-cublas-cu12==12.8.4.1; platform_machine == 'x86_64'
 nvidia-cudnn-cu12==9.8.0.87; platform_machine == 'x86_64'

docs/data/object_detectors_models.yaml

@@ -36,13 +36,13 @@ edgeTPU:
             height: 320 # <--- should match the imgsize of the model, typically 320
             path: /config/model_cache/yolov9-s-relu6-best_320_int8_edgetpu.tflite
             labelmap_path: /config/labels-coco17.txt
-hailo8l:
-  title: Hailo-8/Hailo-8L
+hailo:
+  title: Hailo
   models:
     - key: yolo
       label: YOLO
       recommended: true
-      download: If no custom model path or URL is provided, the Hailo detector automatically downloads the default model (YOLOv6n) from the Hailo Model Zoo on first startup based on the detected hardware. Once cached under `/config/model_cache/hailo`, the model works fully offline.
+      download: If no custom model path or URL is provided, the Hailo detector automatically downloads the default model (YOLOv6n) from the Hailo Model Zoo on first startup, choosing the build that matches the attached device. Once cached under `/config/model_cache`, the model works fully offline.
       ui: |-
         Navigate to **Settings > System > Detection models** and select **Hailo** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure the model settings:
 
@@ -60,7 +60,7 @@ hailo8l:
       yaml: |-
         models:
           - devices:
-              - hailo8l:PCIe
+              - hailo:PCIe
             width: 320
             height: 320
             input_tensor: nhwc
@@ -101,7 +101,7 @@ hailo8l:
       yaml: |-
         models:
           - devices:
-              - hailo8l:PCIe
+              - hailo:PCIe
             width: 300
             height: 300
             input_tensor: nhwc
@@ -824,24 +824,38 @@ cpu:
         models:
           - devices:
               - cpu:3
-deepstack:
-  title: DeepStack / CodeProject.AI
+xdna2:
+  title: AMD XDNA2
   models:
-    - key: yolo
-      label: YOLO
+    - key: yolov9
+      label: YOLOv9
       recommended: true
-      download: This detector runs object detection over the network against a CodeProject.AI or DeepStack server, so no model is downloaded into Frigate itself. Visit the [CodeProject.AI official website](https://www.codeproject.com/Articles/5322557/CodeProject-AI-Server-AI-the-easy-way) to download and install the AI server on your preferred device (e.g. Raspberry Pi, Nvidia Jetson, or other compatible hardware) before configuring the detector.
+      download: |-
+        Prepare the model using the frigate-xdna setup instructions linked above. For local YOLO models, Frigate must have access to the same ONNX file bytes as the sidecar. The example below uses YOLOv9-C at 320x320. Frigate+ models may instead use the same `plus://MODEL_ID` in Frigate and the sidecar.
       ui: |-
-        Navigate to **Settings > System > Detection models** and add a model. The CodeProject.AI server is not reported by the hardware probe, so set `devices` to `deepstack:http://<your_codeproject_ai_server_ip>:<port>/v1/vision/detection` in YAML.
+        Navigate to **Settings > System > Detection models** and add a model. The ZMQ endpoint is not reported by the hardware probe, so set `devices` to `zmq:tcp://xdna:5555` in YAML. Then, on the same model, open the **Custom Model** tab and configure:
 
-        | Field         | Value                                                                  |
-        | ------------- | ---------------------------------------------------------------------- |
-        | **API URL**   | `http://<your_codeproject_ai_server_ip>:<port>/v1/vision/detection`    |
-        | **API Timeout** | `0.1` (seconds)                                                      |
+        | Field                                    | Value                                      |
+        | ---------------------------------------- | ------------------------------------------ |
+        | **Custom object detector model path**    | `/config/models/yolov9-c-320.onnx`       |
+        | **Label map for custom object detector** | `/labelmap/coco-80.txt`                   |
+        | **Object detection model input width**   | `320`                                    |
+        | **Object detection model input height**  | `320`                                    |
+        | **Model Input Pixel Color Format**       | `rgb` (Frigate's default value)          |
+        | **Model Input Tensor Shape**             | `nchw`                                   |
+        | **Model Input D Type**                   | `float`                                  |
+        | **Object Detection Model Type**          | `yolo-generic`                           |
       yaml: |-
         models:
           - devices:
-              - deepstack:http://<your_codeproject_ai_server_ip>:<port>/v1/vision/detection
+              - zmq:tcp://xdna:5555
+            model_type: yolo-generic
+            width: 320
+            height: 320
+            input_tensor: nchw
+            input_dtype: float
+            path: /config/models/yolov9-c-320.onnx
+            labelmap_path: /labelmap/coco-80.txt
 memryx:
   title: MemryX
   models:
@@ -985,6 +999,65 @@ memryx:
             # The .zip file must contain:
             # ├── ssdlite_mobilenet.dfp          (a file ending with .dfp)
             # └── ssdlite_mobilenet_post.onnx    (optional; only if the model includes a cropped post-processing network)
+deepx:
+  title: DEEPX NPU
+  models:
+    - key: yolo
+      label: YOLO
+      recommended: true
+      download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The recommended model is `yolox-s_640x640_ppu.dxnn`. Its Post-Processing Unit (PPU) compile moves candidate selection onto the NPU, which makes it the fastest ModelZoo model measured through Frigate (about 13 ms on a DX-M1). The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration; prefer a `PPU` variant whenever the ModelZoo offers one. PPU models must be compiled with DX-COM 2.4.0 or later, which writes the head layout Frigate reads into the file.
+      ui: |-
+        Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
+
+        | Field                                    | Value                                                |
+        | ---------------------------------------- | ---------------------------------------------------- |
+        | **Custom object detector model path**    | `/config/model_cache/deepx/yolox-s_640x640_ppu.dxnn` |
+        | **Label map for custom object detector** | `/labelmap/coco-80.txt`                              |
+        | **Object detection model input width**   | `640`                                                |
+        | **Object detection model input height**  | `640`                                                |
+        | **Model Input Pixel Color Format**       | `rgb` (Frigate's default value)                      |
+        | **Model Input Tensor Shape**             | `nhwc` (Frigate's default value)                     |
+        | **Model Input D Type**                   | `int` (Frigate's default value)                      |
+        | **Object Detection Model Type**          | `yolo-generic`                                       |
+
+        Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
+      yaml: |-
+        models:
+          - devices:
+              - deepx:PCIe:0
+            path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
+            labelmap_path: /labelmap/coco-80.txt
+            model_type: yolo-generic
+            width: 640
+            height: 640
+    - key: yolox
+      label: YOLOX
+      recommended: false
+      download: No model is bundled with Frigate. Download a pre-compiled YOLOX `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), then bind-mount it into the container and point the model's `path` at it. The `_ppu` variant is faster and also works with the `yolo-generic` model type; the plain export needs `yolox` so its raw head is decoded.
+      ui: |-
+        Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
+
+        | Field                                    | Value                                    |
+        | ---------------------------------------- | ---------------------------------------- |
+        | **Custom object detector model path**    | `/config/model_cache/deepx/yolox-s_640x640.dxnn`|
+        | **Label map for custom object detector** | `/labelmap/coco-80.txt`                  |
+        | **Object detection model input width**   | `640`                                    |
+        | **Object detection model input height**  | `640`                                    |
+        | **Model Input Pixel Color Format**       | `rgb` (Frigate's default value)          |
+        | **Model Input Tensor Shape**             | `nhwc` (Frigate's default value)         |
+        | **Model Input D Type**                   | `int` (Frigate's default value)          |
+        | **Object Detection Model Type**          | `yolox`                                  |
+
+        The width and height must match the resolution the `.dxnn` file was compiled for.
+      yaml: |-
+        models:
+          - devices:
+              - deepx:PCIe:0
+            path: /config/model_cache/deepx/yolox-s_640x640.dxnn
+            labelmap_path: /labelmap/coco-80.txt
+            model_type: yolox
+            width: 640
+            height: 640
 tensorrt:
   title: TensorRT
   models:
@@ -1033,7 +1106,7 @@ synaptics:
     - key: ssd
       label: SSD MobileNet
       recommended: true
-      download: A synap model is provided in the container at `/mobilenet.synap` and is used by this detector type by default. The model comes from the [Synap-release Github](https://github.com/synaptics-astra/synap-release/tree/v1.5.0/models/dolphin/object_detection/coco/model/mobilenet224_full80).
+      download: A synap model is provided in the container at `/synaptics/mobilenet.synap` and is used by this detector type by default. The model comes from the [Synap-release Github](https://github.com/synaptics-astra/synap-release/tree/v1.5.0/models/dolphin/object_detection/coco/model/mobilenet224_full80).
       ui: |-
         Navigate to **Settings > System > Detection models** and select **Synaptics NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
 

docs/docs/configuration/advanced/reference.md

@@ -152,9 +152,10 @@ auth:
 models:
     # Optional: the camera environment this model is for (default: shown below)
     # Cameras select a model by setting detect -> scene to a matching value, and
-    # a model with a scene of all is used by any camera that does not set one.
-    # Valid values are all, indoor, outdoor, indoor_thermal, outdoor_thermal
-  - scene: all
+    # the model with a scene of default is used by any camera that does not set one.
+    # Any name made up of letters, numbers, _ and - is valid, such as thermal.
+    # Models that use the same model file are combined into one model.
+  - scene: default
     # Required: hardware this model runs on, as <detector> or <detector>:<device>
     # See https://docs.frigate.video/configuration/object_detectors for the
     # detectors available and the devices each one accepts. All of a model's
@@ -293,9 +294,9 @@ ffmpeg:
     # Optional: output args for detect streams (default: shown below)
     detect: -threads 2 -f rawvideo -pix_fmt yuv420p
     # Optional: output args for record streams (default: shown below)
-    record: preset-record-generic
+    record: preset-record-generic-audio-aac
     # Optional: output args for sub stream record streams (default: the record output args above)
-    # record_sub: preset-record-generic
+    # record_sub: preset-record-generic-audio-aac
   # Optional: Time in seconds to wait before ffmpeg retries connecting to the camera. (default: shown below)
   # If set too low, frigate will retry a connection to the camera's stream too frequently, using up the limited streams some cameras can allow at once
   # If set too high, then if a ffmpeg crash or camera stream timeout occurs, you could potentially lose up to a maximum of retry_interval second(s) of footage
@@ -316,9 +317,9 @@ detect:
   # Optional: height of the frame for the input with the detect role (default: use native stream resolution)
   height: 720
   # Optional: the environment this camera looks at, which picks the model it runs on
-  # (default: the model with a scene of all)
-  # Valid values are all, indoor, outdoor, indoor_thermal, outdoor_thermal
-  scene: outdoor
+  # (default: the model with a scene of default)
+  # Must match the scene of a configured model
+  scene: thermal
   # Optional: desired fps for your camera for the input with the detect role (default: shown below)
   # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
   fps: 5
@@ -800,7 +801,7 @@ lpr:
 # to Google or OpenAI's LLMs to generate descriptions. GenAI features can be configured at
 # the camera level to enhance privacy for indoor cameras.
 # NOTE: genai is a map of named providers. Each key is a name you choose for the provider,
-#       and each role (chat, descriptions, embeddings) may be assigned to exactly one provider.
+#       and each role (chat, descriptions, embeddings, transcribe) may be assigned to exactly one provider.
 genai:
   # Required: name of the provider (chosen by you, used to reference it elsewhere)
   my_provider:
@@ -813,11 +814,13 @@ genai:
     # Required: The model to use with the provider.
     model: gemini-1.5-flash
     # Optional: Roles this provider handles (default: shown below)
-    # Each role (chat, descriptions, embeddings) must be assigned to exactly one provider.
+    # Each role (chat, descriptions, embeddings, transcribe) must be assigned to exactly
+    # one provider.
     roles:
       - chat
       - descriptions
       - embeddings
+      - transcribe
     # Optional additional args to pass to the GenAI Provider (default: None)
     provider_options:
       keep_alive: -1
@@ -830,13 +833,19 @@ genai:
 audio_transcription:
   # Optional: Enable live and speech event audio transcription (default: shown below)
   enabled: False
+  # Optional: The transcription backend (default: shown below)
+  # Either 'whisper' for Frigate's built-in local models, or the name of a genai
+  # provider that has 'transcribe' in its roles. device and model_size are ignored
+  # when a genai provider is named.
+  model: whisper
   # Optional: The device to run the models on for live transcription. (default: shown below)
   device: CPU
   # Optional: Set the model size used for live transcription. (default: shown below)
   model_size: small
   # Optional: Set the language used for transcription translation. (default: shown below)
-  # List of language codes: https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10
-  language: en
+  # Use 'auto' to let the model detect the language, or a language code from
+  # https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10
+  language: auto
 
 # Optional: Configuration for classification models
 classification:
@@ -892,6 +901,21 @@ live:
   streams:
     main_stream: main_stream_name
     sub_stream: sub_stream_name
+  # Optional: Lower-quality live streams transcoded by go2rtc while someone is watching.
+  # NOTE: Set at the camera level only.
+  transcode:
+    # Optional: Enable transcoded streams (default: shown below)
+    enabled: False
+    # Optional: go2rtc stream to transcode (default: the first live stream)
+    source: main_stream_name
+    # Optional: One transcoded stream per quality (default: shown below)
+    qualities:
+      - height: 720
+        bitrate: 1200
+      - height: 480
+        bitrate: 500
+      - height: 360
+        bitrate: 250
   # Optional: Set the height of the jsmpeg stream. (default: 720)
   # This must be less than or equal to the height of the detect stream. Lower resolutions
   # reduce bandwidth required for viewing the jsmpeg stream. Width is computed to match known aspect ratio.

docs/docs/configuration/audio_detectors.md

@@ -204,7 +204,7 @@ Frequently-heard labels like `speech` can generate a lot of events, and each eve
 
 ### Audio Transcription
 
-Frigate supports fully local audio transcription using either `sherpa-onnx` or OpenAI's open-source Whisper models via `faster-whisper`. The goal of this feature is to support Semantic Search for `speech` audio events. Frigate is not intended to act as a continuous, fully-automatic speech transcription service. Automatically transcribing all speech (or queuing many audio events for transcription) requires substantial CPU (or GPU) resources and is impractical on most systems. For this reason, transcriptions for events are initiated manually from the UI or the API rather than being run continuously in the background.
+Frigate supports fully local audio transcription using either `sherpa-onnx` or OpenAI's open-source Whisper models via `faster-whisper`, and can alternatively offload transcription to a [GenAI provider](#genai-provider). The goal of this feature is to support Semantic Search for `speech` audio events. Frigate is not intended to act as a continuous, fully-automatic speech transcription service. Automatically transcribing all speech (or queuing many audio events for transcription) requires substantial CPU (or GPU) resources and is impractical on most systems. For this reason, transcriptions for events are initiated manually from the UI or the API rather than being run continuously in the background.
 
 :::info
 
@@ -224,6 +224,7 @@ To enable transcription, configure it globally and optionally disable for specif
 **Global:** Navigate to <NavPath path="Settings > Enrichments > Audio transcription" />.
 
 - Set **Enable audio transcription** to on
+- Set **Audio transcription model or GenAI provider name** to `whisper` for Frigate's built-in local models, or to the name of a GenAI provider
 - Set **Transcription device** to the desired device
 - Set **Model size** to the desired size
 
@@ -235,6 +236,7 @@ To enable transcription, configure it globally and optionally disable for specif
 ```yaml
 audio_transcription:
   enabled: True
+  model: whisper
   device: ...
   model_size: ...
 ```
@@ -263,20 +265,88 @@ The optional config parameters that can be set at the global level include:
 - **`enabled`**: Enable or disable the audio transcription feature.
   - Default: `False`
   - It is recommended to only configure the features at the global level, and enable it at the individual camera level.
+- **`model`**: The transcription backend.
+  - Default: `whisper`
+  - `whisper` uses Frigate's built-in local models, described by `device` and `model_size` below.
+  - Any other value must name a key in your `genai` config whose entry has `transcribe` in its `roles`. See [GenAI Provider](#genai-provider).
 - **`device`**: Device to use to run transcription and translation models.
   - Default: `CPU`
   - This can be `CPU` or `GPU`. The `sherpa-onnx` models are lightweight and run on the CPU only. The `whisper` models can run on GPU but are only supported on CUDA hardware.
+  - Ignored when `model` names a GenAI provider.
 - **`model_size`**: The size of the model used for live transcription.
   - Default: `small`
   - This can be `small` or `large`. The `small` setting uses `sherpa-onnx` models that are fast, lightweight, and always run on the CPU but are not as accurate as the `whisper` model.
-  - This config option applies to **live transcription only**. Recorded `speech` events will always use a different `whisper` model (and can be accelerated for CUDA hardware if available with `device: GPU`).
-- **`language`**: Defines the language used by `whisper` to translate `speech` audio events (and live audio only if using the `large` model).
-  - Default: `en`
-  - You must use a valid [language code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10).
+  - This config option applies to **live transcription only**. With `model: whisper`, recorded `speech` events always use a different `whisper` model (and can be accelerated for CUDA hardware if available with `device: GPU`).
+  - Ignored when `model` names a GenAI provider.
+- **`language`**: Defines the language used to transcribe and translate `speech` audio events (and live audio only if using the `large` model or a GenAI provider).
+  - Default: `auto`
+  - `auto` lets the model detect the language itself, which most models do well. Set an explicit language only if detection is picking the wrong one.
+  - Otherwise you must use a valid [language code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py#L10).
   - Transcriptions for `speech` events are translated.
   - Live audio is translated only if you are using the `large` model. The `small` `sherpa-onnx` model is English-only.
 
-The only field that is valid at the camera level is `enabled`.
+The only field that is valid at the camera level is `enabled`. In particular `model` is global only: the transcription backend is a process-wide resource shared by every camera.
+
+#### GenAI Provider
+
+Frigate can send audio to a GenAI provider for transcription when that provider has the `transcribe` role. This is useful if you already run a GenAI provider, or if you do not have the CPU/GPU headroom for a local whisper model. Supported providers are **OpenAI**, **Azure OpenAI**, **Gemini**, and **llama.cpp** with an audio-capable model (a dedicated ASR model such as Qwen3-ASR, or a general multimodal model that accepts audio). Ollama is not supported as it has no audio input.
+
+To use a GenAI provider for audio transcription:
+
+1. Configure a GenAI provider with `transcribe` in its `roles`.
+2. Set the audio transcription model to that GenAI config key (e.g. `whisper_cloud`).
+
+<ConfigTabs>
+<TabItem value="ui">
+
+Navigate to <NavPath path="Settings > Enrichments > Audio transcription" />.
+
+| Field                                                | Description                                                                                              |
+| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
+| **Audio transcription model or GenAI provider name** | Set to the GenAI config key (e.g. `whisper_cloud`) to use a configured GenAI provider for transcription |
+
+The GenAI provider must also be configured with the `transcribe` role under <NavPath path="Settings > Enrichments > Generative AI" />.
+
+</TabItem>
+<TabItem value="yaml">
+
+```yaml
+genai:
+  whisper_cloud:
+    provider: openai
+    api_key: your-api-key
+    model: gpt-transcribe
+    roles:
+      - transcribe
+
+audio_transcription:
+  enabled: True
+  model: whisper_cloud
+  language: en
+```
+
+</TabItem>
+</ConfigTabs>
+
+:::warning
+
+**Give `transcribe` its own `genai` entry.** A `genai` entry has a single `model` string that is shared by every role it holds, so `roles: [descriptions, transcribe]` would send the same model name to both the chat endpoint and the transcription endpoint. Transcription models and chat models are almost never the same model, so define a dedicated entry as shown above.
+
+:::
+
+:::warning
+
+**Live transcription against a metered provider is billed continuously.** In live mode Frigate uploads an overlapping ~2 second window of audio roughly once per second, per camera, for as long as audio stays above that camera's `audio.min_volume`. Windows below that threshold are never uploaded, which is what keeps a quiet camera near zero requests, but a camera pointed at a busy street will keep sending.
+
+Three things keep this opt-in: `transcribe` is not one of the default roles, live transcription is off by default, and the volume gate suppresses silence. Transcription of recorded `speech` events is unaffected - it remains a manual, one-request-per-event action.
+
+:::
+
+`device` and `model_size` have no effect on this path and no local model is ever downloaded.
+
+`language` defaults to `auto`, which sends no language hint and lets the model detect it. Most audio models detect language well, so leave it on `auto` unless detection is picking the wrong one.
+
+When set explicitly, it is sent as the transcription endpoint's native `language` parameter for OpenAI, Azure, and llama.cpp, and as part of the prompt for Gemini. This matters for dedicated ASR models such as Qwen3-ASR: they read the prompt as contextual biasing rather than as an instruction, so a language named in the prompt is ignored, while the endpoint parameter is honored.
 
 #### Live transcription
 
@@ -292,6 +362,8 @@ Results can be error-prone due to a number of factors, including:
 
 For speech sources close to the camera with minimal background noise, use the `small` model.
 
+A [GenAI provider](#genai-provider) is generally the most accurate option for live transcription, at the cost of a network round trip per window. That round trip has to stay under about a second to keep up with the audio; if it does not, Frigate drops the oldest buffered audio rather than letting the backlog grow.
+
 If you have CUDA hardware, you can experiment with the `large` `whisper` model on GPU. Performance is not quite as fast as the `sherpa-onnx` `small` model, but live transcription is far more accurate. Using the `large` model with CPU will likely be too slow for real-time transcription.
 
 #### Transcription and translation of `speech` audio events
@@ -308,7 +380,7 @@ Only one `speech` event may be transcribed at a time. Frigate does not automatic
 
 :::
 
-Recorded `speech` events will always use a `whisper` model, regardless of the `model_size` config setting. Without a supported Nvidia GPU, generating transcriptions for longer `speech` events may take a fair amount of time, so be patient.
+With `model: whisper`, recorded `speech` events always use a `whisper` model, regardless of the `model_size` config setting. Without a supported Nvidia GPU, generating transcriptions for longer `speech` events may take a fair amount of time, so be patient. With a [GenAI provider](#genai-provider), the recorded clip is sent to the provider instead and no local model is used.
 
 #### FAQ
 

docs/docs/configuration/custom_classification/state_classification.md

@@ -85,6 +85,14 @@ An optional config, `save_attempts`, can be set as a key under the model name. T
 </TabItem>
 </ConfigTabs>
 
+## Review items
+
+When a model's state changes while its camera has an active review item, the change is recorded on that review item. This includes changes in the few seconds before the item starts, such as a garage door opening just before the car is detected. State changes never create or extend review items on their own, and the first state reported after Frigate starts is not recorded as a change.
+
+Recorded changes appear in the review item's data as `classification_state_changes` (see the [`frigate/reviews`](/integrations/mqtt#frigatereviews) MQTT topic) and are passed to [GenAI review summaries](/configuration/genai/genai_review) as facts, so a description can note that a gate was opened during the activity.
+
+Change times are most accurate with `motion: true`. A model that only runs on an `interval` notices a change at its next run, so the change may be recorded late or attached to a later review item.
+
 ## Training the model
 
 Creating and training the model is done within the Frigate UI using the `Classification` page. The process consists of three steps:

docs/docs/configuration/ffmpeg_presets.md

@@ -17,17 +17,19 @@ Hardware acceleration arguments tell FFmpeg to decode your camera's video stream
 
 See [the hardware acceleration docs](/configuration/hardware_acceleration_video.md) for details on setting up hardware acceleration for your GPU / iGPU, then select the preset that matches your hardware.
 
-| Preset (YAML config)  | UI Label                | Usage                             | Notes                                                           |
-| --------------------- | ----------------------- | --------------------------------- | --------------------------------------------------------------- |
-| preset-rpi-64-h264    | Raspberry Pi (H.264)    | 64-bit Raspberry Pi, H.264 stream |                                                                 |
-| preset-rpi-64-h265    | Raspberry Pi (H.265)    | 64-bit Raspberry Pi, H.265 stream |                                                                 |
-| preset-vaapi          | VAAPI (Intel/AMD GPU)   | Intel or AMD GPU via VAAPI        | Check the hwaccel docs to ensure the correct driver is selected |
-| preset-intel-qsv-h264 | Intel QuickSync (H.264) | Intel QuickSync, H.264 stream     | If you have issues, use the VAAPI preset instead                |
-| preset-intel-qsv-h265 | Intel QuickSync (H.265) | Intel QuickSync, H.265 stream     | If you have issues, use the VAAPI preset instead                |
-| preset-nvidia         | NVIDIA GPU              | NVIDIA GPU                        |                                                                 |
-| preset-jetson-h264    | NVIDIA Jetson (H.264)   | NVIDIA Jetson, H.264 stream       |                                                                 |
-| preset-jetson-h265    | NVIDIA Jetson (H.265)   | NVIDIA Jetson, H.265 stream       |                                                                 |
-| preset-rkmpp          | Rockchip RKMPP          | Rockchip MPP                      | Use an image with the `-rk` suffix and run in privileged mode   |
+| Preset (YAML config)      | UI Label                | Usage                                         | Notes                                                           |
+| ------------------------- | ----------------------- | --------------------------------------------- | --------------------------------------------------------------- |
+| preset-rpi-64-h264        | Raspberry Pi (H.264)    | 64-bit Raspberry Pi, H.264 stream             |                                                                 |
+| preset-rpi-64-h265        | Raspberry Pi (H.265)    | 64-bit Raspberry Pi, H.265 stream             |                                                                 |
+| preset-apple-silicon-h264 | Apple Silicon (H.264)   | Apple Silicon Mac under lighter, H.264 stream | Needs the `lighter.sh/video` device                             |
+| preset-apple-silicon-h265 | Apple Silicon (H.265)   | Apple Silicon Mac under lighter, H.265 stream | Needs the `lighter.sh/video` device                             |
+| preset-vaapi              | VAAPI (Intel/AMD GPU)   | Intel or AMD GPU via VAAPI                    | Check the hwaccel docs to ensure the correct driver is selected |
+| preset-intel-qsv-h264     | Intel QuickSync (H.264) | Intel QuickSync, H.264 stream                 | If you have issues, use the VAAPI preset instead                |
+| preset-intel-qsv-h265     | Intel QuickSync (H.265) | Intel QuickSync, H.265 stream                 | If you have issues, use the VAAPI preset instead                |
+| preset-nvidia             | NVIDIA GPU              | NVIDIA GPU                                    |                                                                 |
+| preset-jetson-h264        | NVIDIA Jetson (H.264)   | NVIDIA Jetson, H.264 stream                   |                                                                 |
+| preset-jetson-h265        | NVIDIA Jetson (H.265)   | NVIDIA Jetson, H.265 stream                   |                                                                 |
+| preset-rkmpp              | Rockchip RKMPP          | Rockchip MPP                                  | Use an image with the `-rk` suffix and run in privileged mode   |
 
 <ConfigTabs>
 <TabItem value="ui">

docs/docs/configuration/genai/config.md

@@ -43,7 +43,7 @@ genai:
 
 The examples on this page all use `my_provider`, but the name is arbitrary and is only used to reference the provider elsewhere in the config (for example, `semantic_search.model`).
 
-Each provider handles one or more **roles**: `chat`, `descriptions`, and `embeddings`. A provider handles all three by default, and each role may be assigned to exactly one provider. Define a single provider if you want it to do everything, or split the roles across several providers using the `roles` option.
+Each provider handles one or more **roles**: `chat`, `descriptions`, `embeddings`, and `transcribe`. A provider handles the first three by default; `transcribe` must always be listed explicitly, and is not available on Ollama, which has no audio input. Each role may be assigned to exactly one provider. Define a single provider if you want it to do everything, or split the roles across several providers using the `roles` option.
 
 If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
 
@@ -63,11 +63,11 @@ Running Generative AI models on CPU is not recommended, as high inference times
 
 You must use a vision-capable model with Frigate. The following models are recommended for local deployment of the `descriptions` and `chat` roles:
 
-| Model               | Notes                                                                                                                                                                |
-| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `qwen3-vl`          | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object.                                              |
-| `qwen3.6`/`qwen3.8` | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
-| `gemma4`            | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action.                                     |
+| Model               | Review [frame mode](/configuration/genai/genai_review#frame-mode) | Notes                                                                                                                                                                |
+| ------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `qwen3-vl`          | `frames`                                                              | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object. Follows a sequence of frames on its own.     |
+| `qwen3.6`/`qwen3.8` | `frames`                                                              | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
+| `gemma4`            | `annotated_frames`                                                    | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action. Loses track of activity that repeats or reverses, so it benefits from annotated frames. |
 
 #### Embedding models
 
@@ -77,6 +77,17 @@ The `embeddings` role needs a different kind of model. Text queries are matched
 | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `qwen3-vl-embedding` | Multimodal embeddings for [Semantic Search](/configuration/semantic_search#genai-provider). Must be served by llama.cpp started with `--embeddings` and `--mmproj`. |
 
+#### Transcription models
+
+The `transcribe` role needs a model that accepts audio input. A text-only or vision-only model cannot serve this role. The following are recommended for local deployment of the `transcribe` role:
+
+| Model       | Notes                                                                                                                                                                                                      |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `qwen3-asr` | Dedicated speech recognition model covering 30 languages, and the better choice for transcription quality. It only transcribes, so it cannot be shared with the `descriptions` or `chat` roles.            |
+| `gemma4`    | General multimodal model that accepts audio as well as images, so one served model can cover `transcribe` alongside the other roles. Transcript quality is below `qwen3-asr`, particularly on noisy audio. |
+
+Both must be served by llama.cpp started with the matching audio `--mmproj`. llama.cpp only reports audio support when an audio projector is loaded. Without it Frigate sees the model as text-only and the `transcribe` role is unavailable in the UI. Frigate transcribes through the server's `/v1/audio/transcriptions` route, which llama.cpp serves for any audio-capable model.
+
 :::info
 
 Each model is available in multiple parameter sizes (3b, 4b, 8b, etc.). Larger sizes are more capable of complex tasks and understanding of situations, but requires more memory and computational resources. It is recommended to try multiple models and experiment to see which performs best.

docs/docs/configuration/genai/review_summaries.md

@@ -192,6 +192,45 @@ review:
 </TabItem>
 </ConfigTabs>
 
+### Frame Mode
+
+Review items are sent to the model as a sequence of still frames. Some models follow that sequence well on their own; others lose track of activity that repeats or reverses, and describe a single trip when the subject actually made several. The `frame_mode` option controls how those frames are presented.
+
+- `frames` (default): the prompt followed by the frames, exactly as earlier versions of Frigate sent them.
+- `annotated_frames`: each frame is preceded by its frame number and elapsed time, along with notes describing what the object tracker recorded at that moment, such as an object being first detected, starting to move, turning around, stopping, or no longer being detected.
+
+The notes come from tracking data rather than from the images, so they describe activity the model may not have picked up on its own. In testing with a person carrying three waste bins to the curb one at a time, `gemma4` described a single trip on every attempt with `frames`, and consistently described multiple trips with `annotated_frames`. Models that already handle these sequences well, such as the `qwen3-vl` family, gain little and should stay on `frames`.
+
+Changes reported by [state classification](/configuration/custom_classification/state_classification#review-items) models during the review item are listed in the prompt in both modes. `annotated_frames` also notes each change before the frame where it happened.
+
+Annotated mode also caps the number of frames, since the notes already establish the order of events and extra near-duplicate frames tend to crowd out the middle of a clip. Longer review items are sampled more sparsely as a result, and typically use fewer tokens than `frames` mode for the same item.
+
+:::note
+
+Annotated mode needs tracking data for the review item. If none is available, Frigate falls back to sending plain frames for that item.
+
+:::
+
+<ConfigTabs>
+<TabItem value="ui">
+
+Navigate to <NavPath path="Settings > Global configuration > Review" />.
+
+- Set **GenAI config > Frame mode** to the desired mode (e.g., `annotated_frames`)
+
+</TabItem>
+<TabItem value="yaml">
+
+```yaml {4}
+review:
+  genai:
+    enabled: true
+    frame_mode: annotated_frames
+```
+
+</TabItem>
+</ConfigTabs>
+
 ### Response Style
 
 Different models respond to the built-in prompt with very different writing styles: some produce natural narration while others sound short and mechanical. The `response_style` option selects a writing style preset that rewords the prompt's instructions for the user-facing fields (the title, short summary, and scene description). Presets replace those instructions rather than adding extra ones, so the model never receives competing style directions.

docs/docs/configuration/hardware_acceleration_video.md

@@ -43,6 +43,10 @@ Frigate supports presets for optimal hardware accelerated video decoding:
 
 - [RKNN](#rockchip-platform): Frigate can utilize the media engine in RockChip SOCs to accelerate video decoding.
 
+**Apple Silicon Mac** <CommunityBadge />
+
+- [lighter](#apple-silicon-mac-lighter): Frigate can utilize the media engine in Apple Silicon Macs to accelerate video decoding, when running under the lighter container runtime.
+
 **Other Hardware**
 
 Depending on your system, these presets may not be compatible, and you may need to use manual hwaccel args to take advantage of your hardware. More information on hardware accelerated decoding for ffmpeg can be found here: https://trac.ffmpeg.org/wiki/HWAccelIntro
@@ -499,7 +503,7 @@ cameras:
 
 ## Synaptics
 
-Hardware accelerated video de-/encoding is supported on Synpatics SL-series SoC.
+Hardware accelerated video de-/encoding is supported on Synaptics SL-series SoC.
 
 ### Prerequisites
 
@@ -533,3 +537,35 @@ output_args:
 Make sure that your SoC supports hardware acceleration for your input stream and your input stream is h264 encoding. For example, if your camera streams with h264 encoding, your SoC must be able to de- and encode with it. If you are unsure whether your SoC meets the requirements, take a look at the datasheet.
 
 :::
+
+## Apple Silicon Mac (lighter)
+
+[lighter](https://github.com/fieldwork-ai/lighter) is an open-source container runtime for macOS. It gives a container the Mac's media engine as a standard V4L2 decoder, backed by VideoToolbox, so Frigate decodes H.264 and H.265 streams in hardware with the ffmpeg it already ships. It works on M1 and newer Macs with lighter 0.9.2 or newer.
+
+Give the container the video device. With Docker Compose:
+
+```yaml {4-5}
+services:
+  frigate:
+    ...
+    devices:
+      - lighter.sh/video=all
+```
+
+Or with `docker run`, add `--device lighter.sh/video=all`.
+
+Then set the preset for the codec your cameras stream. The decoder is specific to the codec, so if your cameras mix H.264 and H.265, set the preset for the most common codec globally and override it on the other cameras:
+
+```yaml
+ffmpeg:
+  hwaccel_args: preset-apple-silicon-h264
+
+cameras:
+  garage: # an H.265 camera
+    ffmpeg:
+      hwaccel_args: preset-apple-silicon-h265
+```
+
+The presets decode on the media engine and encode the Birdseye restream and timelapses there too. Scaling to the detect resolution runs on the CPU, as ffmpeg's V4L2 decoders cannot scale.
+
+lighter can also run object detection on the Mac's Neural Engine; see [Apple Neural Engine (lighter)](object_detectors.md#apple-neural-engine-lighter).

docs/docs/configuration/license_plate_recognition.md

@@ -378,10 +378,10 @@ Navigate to <NavPath path="Settings > Camera configuration > Object detection" /
 
 Navigate to <NavPath path="Settings > Camera configuration > Objects" />.
 
-| Field                                          | Description         |
-| ---------------------------------------------- | ------------------- |
-| **Objects to track**                           | Add `license_plate` |
-| **Object filters > License Plate > Threshold** | Set to `0.7`        |
+| Field                                                     | Description         |
+| --------------------------------------------------------- | ------------------- |
+| **Objects to track**                                      | Add `license_plate` |
+| **Object filters > License Plate > Confidence threshold** | Set to `0.7`        |
 
 Navigate to <NavPath path="Settings > Camera configuration > Motion detection" />.
 

docs/docs/configuration/live.md

@@ -28,6 +28,24 @@ WebRTC may use an external STUN server for NAT traversal. MSE and HLS streaming
 
 :::
 
+### Selecting a streaming technology
+
+Frigate [defaults to MSE](#why-does-frigate-prefer-mse-over-webrtc-for-live-view) for restreamed cameras by design. To use WebRTC, select it explicitly from a camera's single-camera Live view settings (the settings menu in the camera's Live view header on desktop, or the settings drawer on mobile). Three related controls work together:
+
+- **Stream**: _what_ to play. This lists the [streams you've configured](#setting-streams-for-live-ui) (for example `Main Stream` and `Sub Stream`).
+- **Force low-bandwidth mode**: a switch that always plays Frigate's built-in low-bandwidth feed (the stream assigned the `detect` role, using JSMpeg) instead of the selected stream. It works anywhere without go2rtc and is useful on slow or metered connections. While it is enabled, the stream and streaming technology selectors are disabled; your stream and technology choices are restored when you turn it off.
+- **Streaming Technology**: _how_ to play the selected stream, listing **MSE** and **WebRTC**. It is only shown for a restreamed stream.
+
+- The choices are saved **per device, per camera** in your browser's local storage.
+- **WebRTC is only selectable when it can actually work for that stream.** When it can't, the option is shown disabled with the reason inline, and a more detailed reason (the failing codecs, or why the connectivity check failed) is logged to your browser's console. Common reasons:
+  - **Not configured**: no `candidates` or `ice_servers` are set under `go2rtc.webrtc` (see [WebRTC extra configuration](#webrtc-extra-configuration)).
+  - **Could not connect**: e.g. port `8555` isn't reachable, or a STUN/TURN server is misconfigured. Frigate runs a one-time WebRTC connectivity check when the Live view opens; the option may briefly show as "checking" while it runs.
+  - **Unsupported video codec**: the stream's video codec can't be played over WebRTC in your browser, most commonly H.265/HEVC in Firefox or Edge.
+  - **Unsupported audio codec**: WebRTC needs opus or G.711 audio, so a stream whose playback audio is only AAC (without an added opus/G.711 track) can't carry audio over WebRTC. See [Audio Support](#audio-support) for how to add one.
+  - **Unsupported browser**: the browser doesn't support WebRTC.
+
+When WebRTC isn't available, Frigate automatically uses MSE (or falls back to JSMpeg), so live view keeps working regardless of the selection.
+
 ### Camera Settings Recommendations
 
 If you are using go2rtc, you should adjust the following settings in your camera's firmware for the best experience with Live view:
@@ -74,7 +92,7 @@ go2rtc:
 
 ### Setting Streams For Live UI
 
-You can configure Frigate to allow manual selection of the stream you want to view in the Live UI. For example, you may want to view your camera's substream on mobile devices, but the full resolution stream on desktop devices. Setting the streams list will populate a dropdown in the UI's Live view that allows you to choose between the streams. This stream setting is _per device_ and is saved in your browser's local storage.
+You can configure Frigate to allow manual selection of the stream you want to view in the Live UI. For example, you may want to view your camera's substream on mobile devices, but the full resolution stream on desktop devices. Setting the streams list will populate a dropdown in the UI's Live view that allows you to choose between the streams. This stream setting is _per device_ and is saved in your browser's local storage. When a camera has more than one stream, the dropdown also offers **Auto**, which is used until you pick a specific stream. Auto starts on the first stream, steps down the list when your connection can't keep up, and steps back up when it recovers. To retry the top stream right away, select **Try highest quality** under the stream picker. List streams from highest to lowest quality, and avoid names that are plain numbers (such as `720`), which the browser sorts ahead of the others. In the UI, drag streams to reorder them, or use **Auto order** to sort them by measured bitrate.
 
 Additionally, when creating and editing camera groups in the UI, you can choose the stream you want to use for your camera group's Live dashboard.
 
@@ -140,6 +158,26 @@ cameras:
 </TabItem>
 </ConfigTabs>
 
+### Transcoded streams
+
+When a camera has no suitable sub stream, Frigate can add lower-quality streams that go2rtc transcodes to H.264 while someone is watching. They appear in the stream list like any other stream, so Auto mode can step down to them. Enable them under <NavPath path="Settings > Camera configuration > Live playback" />, or in YAML:
+
+```yaml
+cameras:
+  test_cam:
+    live:
+      transcode:
+        enabled: true
+        source: test_cam # optional, defaults to the first live stream
+        qualities:
+          - height: 720
+            bitrate: 1200 # kbps
+          - height: 480
+            bitrate: 500
+```
+
+Each quality becomes a go2rtc stream named `<camera>_transcode_<height>p`. go2rtc picks a hardware encoder automatically and falls back to the CPU, which costs CPU for each transcode while it is being watched. Check go2rtc's `api/ffmpeg/hardware` page to see which encoder it found. Using a sub stream as the `source` lowers the cost.
+
 ### WebRTC extra configuration:
 
 WebRTC works by creating a TCP or UDP connection on port `8555`. However, it requires additional configuration:
@@ -157,6 +195,17 @@ WebRTC works by creating a TCP or UDP connection on port `8555`. However, it req
         - stun:8555
   ```
 
+- The web UI uses the STUN and TURN servers in `ice_servers` and falls back to Google's public STUN server when none are set:
+
+  ```yaml title="config.yml"
+  go2rtc:
+    webrtc:
+      ice_servers:
+        - urls: [turn:turn.example.com:3478]
+          username: frigate
+          credential: password
+  ```
+
 - For access through Tailscale, the Frigate system's Tailscale IP must be added as a WebRTC candidate. Tailscale IPs all start with `100.`, and are reserved within the `100.64.0.0/10` CIDR block.
 
 - Note that some browsers may not support H.265 (HEVC). You can check your browser's current version for H.265 compatibility [here](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#codecs-madness).
@@ -206,6 +255,8 @@ For devices that support two way talk, Frigate can be configured to use the feat
 - Ensure you access Frigate via https (may require [opening port 8971](/frigate/installation/#ports)).
 - For the Home Assistant Frigate card, [follow the docs](http://card.camera/#/usage/2-way-audio) for the correct source.
 
+The two-way talk control in the single-camera Live view is only enabled when WebRTC is available; if WebRTC isn't configured or can't connect, the control is shown disabled.
+
 To use the Reolink Doorbell with two way talk, you should use the [recommended Reolink configuration](/configuration/camera_specific#reolink-cameras)
 
 As a starting point to check compatibility for your camera, view the list of cameras supported for two-way talk on the [go2rtc repository](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#two-way-audio). For cameras in the category `ONVIF Profile T`, you can use the [ONVIF Conformant Products Database](https://www.onvif.org/conformant-products/)'s FeatureList to check for the presence of `AudioOutput`. A camera that supports `ONVIF Profile T` _usually_ supports this, but due to inconsistent support, a camera that explicitly lists this feature may still not work. If no entry for your camera exists on the database, it is recommended not to buy it or to consult with the manufacturer's support on the feature availability.
@@ -332,6 +383,13 @@ When your browser runs into problems playing back your camera streams, it will l
     - `Safari reported InvalidStateError.`
     - `Safari reported decoding errors.`
 
+- **mse-codec**
+  - What it means: go2rtc has no codec for this stream that the browser can play.
+  - What to try: Pick a stream with a codec the browser supports (H.264 is the most compatible), or use a browser that supports the stream's codec. In Auto, Frigate skips this stream for the rest of the session.
+
+  - Possible console messages from the player code:
+    - `mse: streams: codecs not matched: ...`
+
 - **stalled**
   - What it means: Playback has stalled because the player has fallen too far behind live (extended buffering or no data arriving).
   - What to try: This is usually indicative of the browser struggling to decode too many high-resolution streams at once. Try selecting a lower-bandwidth stream (substream), reduce the number of live streams open, improve the network connection, or lower the camera resolution. Also check your camera's keyframe (I-frame) interval: shorter intervals make playback start and recover faster. You can also try increasing the timeout value in <NavPath path="Settings > UI" /> .

docs/docs/configuration/non_root.md

@@ -313,8 +313,16 @@ To remove root from the container entirely, add Docker's `user:`:
 
 ```yaml
     user: "1000:1000" # NOT compatible with PUID/PGID, see the run modes table
+    tmpfs:
+      - /tmp:size=256m
+      - /tmp/cache:size=1000000000
+      - /run:exec,nosuid,nodev,mode=0755,uid=1000,gid=1000,size=16m # uid must match user:
 ```
 
+`/run` has to be owned by that uid as well. s6 writes its runtime state there before anything else starts, and with no root in the container a root-owned `/run` stops it during init with `cannot create /run/test of writability`. Keep `uid` and `gid` in the tmpfs options matching `user:`, and don't carry that pair back into the default mode, where a root-owned `/run` is what keeps the unprivileged services out of s6's runtime state.
+
+This only bites once root is genuinely gone. s6's init helper is setuid, so `user:` on its own still lets init regain root and correct `/run` itself. The `no-new-privileges:true` above is what blocks that, which is also what makes the `/run` ownership mandatory. Dropping it would hide the problem by handing init root again.
+
 Two things change, and the first one will break a working install if you skip it. The startup device grants can't run, because there is no root left to run them, so every device you pass stops working until you grant that uid access yourself with `group_add:` or a udev rule; see [Manual setup](#manual-setup). Expect this to surface as a driver error rather than a permission error, like `No VA display found` from VAAPI. And every service then runs as that one uid, so go2rtc no longer gets its own restricted user. `/config` and `/media/frigate` have to be owned by that uid already, since Frigate never adjusts ownership in this mode. Switching an existing install over also leaves `/config/go2rtc_homekit.yml` owned by the go2rtc user, which this mode can't write; `chown` it to your uid or HomeKit pairing changes stop persisting. Frigate warns and starts either way.
 
 This mode can also take `cap_drop: [ALL]`, which the default mode cannot: starting as root needs `CAP_CHOWN` for the ownership sweep, `CAP_SETUID` and `CAP_SETGID` to drop to the runtime user, and `CAP_FOWNER` for the device grants.

docs/docs/configuration/object_detectors.md

@@ -22,17 +22,20 @@ Frigate supports multiple different detectors that work on different types of ha
 **Most Hardware**
 
 - [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB, Mini PCIe, and m.2 formats allowing for a wide range of compatibility with devices.
-- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
+- [Hailo](#hailo): The Hailo-8, Hailo-8L and Hailo-8R AI Acceleration modules are available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
 - <CommunityBadge /> [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
+- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, offering broad compatibility across various platforms.
 
 **AMD**
 
 - [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection.
 - [ONNX](#onnx): ROCm will automatically be detected and used as a detector in the `-rocm` Frigate image when a supported ONNX model is configured.
+- <CommunityBadge /> [XDNA2](#amd-xdna2): AMD Ryzen AI / XDNA2 NPUs can run object detection through the community-maintained `frigate-xdna` ZMQ sidecar.
 
 **Apple Silicon**
 
 - [Apple Silicon](#apple-silicon-detector): Apple Silicon can run on M1 and newer Apple Silicon devices.
+- <CommunityBadge /> [ONNX](#apple-neural-engine-lighter): the ONNX detector runs on the Neural Engine of M1 and newer Macs when Frigate runs under the lighter container runtime.
 
 **Intel**
 
@@ -101,32 +104,36 @@ Coral EdgeTPU and MemryX accelerators can only be opened by one process, so thos
 
 ### Running more than one model
 
-Cameras can be split across models by scene, which is useful when indoor and outdoor cameras benefit from differently trained models. Each model declares the `scene` it is for, and each camera picks one with `detect -> scene`:
+Cameras can be split across models by scene, which is useful when some cameras benefit from a differently trained model, such as thermal cameras. Each model declares the `scene` it is for, and each camera picks one with `detect -> scene`:
 
 ```yaml
 models:
-  - scene: outdoor
-    path: plus://your-outdoor-model
+  - scene: default
+    path: plus://your-model
     devices:
       - edgetpu:pci:0
-  - scene: indoor
-    path: /config/model_cache/indoor.onnx
+  - scene: thermal
+    path: /config/model_cache/thermal.onnx
     model_type: yolo-generic
     devices:
       - openvino:GPU
 
 cameras:
   driveway:
-    detect:
-      scene: outdoor
     ...
-  hallway:
+  backyard_thermal:
     detect:
-      scene: indoor
+      scene: thermal
     ...
 ```
 
-Available scenes are `all`, `indoor`, `outdoor`, `indoor_thermal`, and `outdoor_thermal`. A model with a scene of `all` is used by every camera that does not set one, and `all` is the default when a model does not declare a scene. Changing a camera's scene requires a restart.
+A scene is any name made up of letters, numbers, `_`, and `-`. The model with a scene of `default` is used by every camera that does not set one (or sets a scene that no model is configured for), and `default` is used when a model does not declare a scene. Changing a camera's scene requires a restart.
+
+:::warning
+
+Scenes are for running **different** models. Do not configure the same model under several scenes to dedicate a detector to specific cameras: every detector of a model already serves every camera using it, and splitting them only leaves some detectors idle while others fall behind. Frigate detects models that use the same model file, even under a different path or file name, combines them into one model with all of their hardware, and logs a warning.
+
+:::
 
 ### Choosing a model size
 
@@ -285,9 +292,9 @@ models:
 
 ---
 
-## Hailo-8
+## Hailo
 
-This detector is available for use with both Hailo-8 and Hailo-8L AI Acceleration Modules. The integration automatically detects your hardware architecture via the Hailo CLI and selects the appropriate default model if no custom model is specified.
+This detector is available for use with the Hailo-8, Hailo-8L and Hailo-8R AI Acceleration Modules. The integration identifies which of them is attached and selects the matching default model if no custom model is specified.
 
 See the [installation docs](../frigate/installation.md#hailo-8) for information on configuring the Hailo hardware.
 
@@ -308,11 +315,11 @@ The HailoRT runtime is not part of the Frigate image. It is downloaded and insta
 When configuring the Hailo detector, you have two options to specify the model: a local **path** or a **URL**.
 If both are provided, the detector will first check for the model at the given local path. If the file is not found, it will download the model from the specified URL. The model file is cached under `/config/model_cache/hailo`.
 
-<ModelConfigDropdown detectorTitle="Hailo-8/Hailo-8L" models={objectDetectorsModels.hailo8l.models} />
+<ModelConfigDropdown detectorTitle="Hailo" models={objectDetectorsModels.hailo.models} />
 
 For additional ready-to-use models, please visit: https://github.com/hailo-ai/hailo_model_zoo
 
-Hailo8 supports all models in the Hailo Model Zoo that include HailoRT post-processing. You're welcome to choose any of these pre-configured models for your implementation.
+Hailo supports all models in the Hailo Model Zoo that include HailoRT post-processing. You're welcome to choose any of these pre-configured models for your implementation.
 
 > **Note:**
 > The config.path parameter can accept either a local file path or a URL ending with .hef. When provided, the detector will first check if the path is a local file path. If the file exists locally, it will use it directly. If the file is not found locally or if a URL was provided, it will attempt to download the model from the specified URL.
@@ -362,7 +369,7 @@ Intel NPUs cannot be used under Home Assistant OS, which does not include the NP
 
 :::warning
 
-The network-based detectors (Deepstack and the Apple Silicon client) are being reworked. Their extra options no longer have a place in the config, so only the endpoint carried in the device string is honored right now: Deepstack ignores `api_key` and `api_timeout`, and the Apple Silicon client ignores `request_timeout_ms` and `linger_ms`. Anything else is dropped when your config is migrated.
+The Apple Silicon detector client is being reworked. Its extra options no longer have a place in the config, so only the endpoint carried in the device string is honored right now, and `request_timeout_ms` and `linger_ms` are ignored. Anything else is dropped when your config is migrated.
 
 :::
 
@@ -483,7 +490,7 @@ See [ONNX supported models](#onnx) for supported models, there are some caveats:
 
 ## ONNX
 
-ONNX is an open format for building machine learning models, Frigate supports running ONNX models on CPU, OpenVINO, ROCm, and TensorRT. On startup Frigate will automatically try to use a GPU if one is available.
+ONNX is an open format for building machine learning models, Frigate supports running ONNX models on CPU, OpenVINO, ROCm, TensorRT, and a Mac's Neural Engine. On startup Frigate will automatically try to use a GPU if one is available.
 
 :::info
 
@@ -499,6 +506,9 @@ If the correct build is used for your GPU then the GPU will be detected and used
   - Nvidia GPUs will automatically be detected and used with the ONNX detector in the `-tensorrt` Frigate image.
   - Jetson devices will automatically be detected and used with the ONNX detector in the `-tensorrt-jp6` Frigate image.
 
+- **Apple Silicon Mac** <CommunityBadge />
+  - The Neural Engine will automatically be detected and used with the ONNX detector when Frigate runs under lighter with its Neural Engine device. See [Apple Neural Engine (lighter)](#apple-neural-engine-lighter).
+
 :::
 
 :::tip
@@ -514,6 +524,22 @@ models:
 
 :::
 
+### Apple Neural Engine (lighter) {#apple-neural-engine-lighter}
+
+[lighter](https://github.com/fieldwork-ai/lighter) is an open-source container runtime for macOS. A container started with its `lighter.sh/ane` device gets an ONNX Runtime execution provider that runs models on the Mac's Neural Engine, and the ONNX detector uses it automatically, with the same models and configuration as on any other hardware. It works on M1 and newer Macs with lighter 0.9.2 or newer.
+
+Give the Frigate container the Neural Engine device. With Docker Compose:
+
+```yaml
+services:
+  frigate:
+    image: ghcr.io/blakeblackshear/frigate:stable-standard-arm64
+    devices:
+      - lighter.sh/ane=all
+```
+
+Or with `docker run`, add `--device lighter.sh/ane=all`. Frigate then reports the Neural Engine under **Settings > System > Detection models**, and the ONNX detector's model loads on it. lighter can also decode camera streams on the Mac's media engine; see [Video Decoding](hardware_acceleration_video.md#apple-silicon-mac-lighter).
+
 ### Configuration {#configuration-onnx}
 
 <ModelConfigDropdown detectorTitle="ONNX" models={objectDetectorsModels.onnx.models} />
@@ -540,32 +566,30 @@ A TensorFlow Lite model is provided in the container at `/cpu_model.tflite` and
 
 When using CPU detectors, you can add one CPU detector per camera. Adding more detectors than the number of cameras should not improve performance.
 
-## Deepstack / CodeProject.AI Server Detector
-
-:::warning
-
-The network-based detectors (Deepstack and the Apple Silicon client) are being reworked. Their extra options no longer have a place in the config, so only the endpoint carried in the device string is honored right now: Deepstack ignores `api_key` and `api_timeout`, and the Apple Silicon client ignores `request_timeout_ms` and `linger_ms`. Anything else is dropped when your config is migrated.
-
-:::
-
-The Deepstack / CodeProject.AI Server detector for Frigate allows you to integrate Deepstack and CodeProject.AI object detection capabilities into Frigate. CodeProject.AI and DeepStack are open-source AI platforms that can be run on various devices such as the Raspberry Pi, Nvidia Jetson, and other compatible hardware. It is important to note that the integration is performed over the network, so the inference times may not be as fast as native Frigate detectors, but it still provides an efficient and reliable solution for object detection and tracking.
-
-### Setup {#setup-deepstack}
-
-To get started with CodeProject.AI, visit their [official website](https://www.codeproject.com/Articles/5322557/CodeProject-AI-Server-AI-the-easy-way) to follow the instructions to download and install the AI server on your preferred device. Detailed setup instructions for CodeProject.AI are outside the scope of the Frigate documentation.
-
-To integrate CodeProject.AI into Frigate, configure the detector as follows:
-
-### Configuration {#configuration-deepstack}
-
-<ModelConfigDropdown detectorTitle="DeepStack" models={objectDetectorsModels.deepstack.models} />
-
-Replace `<your_codeproject_ai_server_ip>` and `<port>` with the IP address and port of your CodeProject.AI server.
-
-To verify that the integration is working correctly, start Frigate and observe the logs for any error messages related to CodeProject.AI. Additionally, you can check the Frigate web interface to see if the objects detected by CodeProject.AI are being displayed and tracked properly.
-
 # Community Supported Detectors
 
+## AMD XDNA2
+
+AMD Ryzen AI / XDNA2 NPUs can be used through the community-maintained
+[frigate-xdna](https://github.com/mitchins/frigate-xdna) detector sidecar.
+The sidecar runs separately from Frigate and connects using Frigate's ZMQ
+detector interface.
+
+Currently qualified on **Ryzen AI Max 300 / Strix Halo**. Other XDNA2 devices
+are not yet qualified; XDNA1 is unsupported.
+
+Follow the frigate-xdna setup instructions to prepare and start the sidecar
+before starting Frigate.
+
+### Configuration {#configuration-xdna2}
+
+Using the detector config below will connect Frigate to the sidecar:
+
+<ModelConfigDropdown detectorTitle="AMD XDNA2" models={objectDetectorsModels.xdna2.models} />
+
+The example assumes Frigate and the sidecar share a Docker network where the
+sidecar is named `xdna`.
+
 ## MemryX MX3
 
 This detector is available for use with the MemryX MX3 accelerator M.2 module. Frigate supports the MX3 on compatible hardware platforms, providing efficient and high-performance object detection.
@@ -636,6 +660,63 @@ For detailed instructions on compiling models, refer to the [MemryX Compiler](ht
 
 ---
 
+## DEEPX NPU
+
+This detector is available for use with the DEEPX NPU, both the DX-M1 M.2 module and the DX-M1M on the Sixfab AI HAT+ for the Raspberry Pi 5. The configuration below applies unchanged to either form factor. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
+
+See the [installation docs](../frigate/installation.md#deepx-npu) for information on installing the DEEPX kernel driver and runtime on the host and passing the NPU through to the container.
+
+To run a model on a DEEPX NPU, list a `deepx` device on that model.
+
+:::info
+
+The DX-RT Python bindings are not part of the Frigate image. They are downloaded and installed into `/config/.local` the first time a DEEPX device is configured, verified against pinned checksums, and updated automatically when a Frigate release pins a new version. If the container has no internet access, see [Detector runtimes](/frigate/network_requirements#detector-runtimes) for how to provide the files yourself.
+
+:::
+
+### Configuration {#configuration-deepx}
+
+<ModelConfigDropdown detectorTitle="DEEPX" models={objectDetectorsModels.deepx.models} />
+
+Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Two model types are supported:
+
+- `yolo-generic` for YOLO object detection models, the recommended default. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do models compiled with DEEPX's Post-Processing Unit (PPU) support.
+- `yolox` for YOLOX models compiled without PPU support, whose raw head needs Frigate's YOLOX decoder. A YOLOX model compiled with PPU support works under either `yolox` or `yolo-generic`.
+
+The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler. The recommended starting point is `yolox-s_640x640_ppu.dxnn`, the fastest ModelZoo model measured through Frigate:
+
+```yaml
+models:
+  - devices:
+      - deepx:PCIe:0
+    path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
+    labelmap_path: /labelmap/coco-80.txt
+    model_type: yolo-generic
+    width: 640
+    height: 640
+```
+
+For PPU models, use a `.dxnn` compiled with DX-COM 2.4.0 or later. Frigate reads the PPU head layout the compiler writes into the file and refuses to load a PPU model without it.
+
+`model_type` must be set to `yolo-generic` or `yolox` to match the model; `yolo-generic` is the recommended default unless the model is a raw YOLOX export. Frigate defaults it to `ssd`, which this detector does not support, so the detector refuses to start on a model that leaves it unset.
+
+`width` and `height` must match the resolution the model was compiled for. Quantization parameters are baked into the `.dxnn` file at compile time, so no normalization is applied on the host and Frigate's default `input_tensor`, `input_pixel_format`, and `input_dtype` values do not need to be overridden.
+
+A DEEPX device is `PCIe:<index>`, as reported on the detector settings page. The NPU daemon multiplexes across processes, so the same device may be listed more than once to run additional inference processes against it:
+
+```yaml
+models:
+  - devices:
+      - deepx:PCIe:0
+      - deepx:PCIe:0
+```
+
+#### Label maps
+
+The object detection models in the DEEPX ModelZoo are trained on the standard 80-class COCO label set, so `labelmap_path` must be set to `/labelmap/coco-80.txt`. Frigate's default label map uses an extended 91-class COCO scheme, and leaving it in place will cause detections to be reported as the wrong object type. For `yolo-generic` models the label map is also what the detector uses to tell the output layout, so a label map with the wrong number of classes is reported as an error at startup.
+
+---
+
 ## NVidia TensorRT Detector
 
 Nvidia Jetson devices may be used for object detection using the TensorRT libraries. Due to the size of the additional libraries, this detector is only provided in images with the `-tensorrt-jp6` tag suffix, e.g. `ghcr.io/blakeblackshear/frigate:stable-tensorrt-jp6`. This detector is designed to work with Yolo models for object detection.

docs/docs/configuration/object_filters.md

@@ -45,10 +45,10 @@ Any detection below `min_score` will be immediately thrown out and never tracked
 
 Navigate to <NavPath path="Settings > Global configuration > Objects" /> to set score filters globally.
 
-| Field                                   | Description                                                      |
-| --------------------------------------- | ---------------------------------------------------------------- |
-| **Object filters > Person > Min Score** | Minimum score for a single detection to initiate tracking        |
-| **Object filters > Person > Threshold** | Minimum computed (median) score to be considered a true positive |
+| Field                                              | Description                                                      |
+| -------------------------------------------------- | ---------------------------------------------------------------- |
+| **Object filters > Person > Minimum confidence**   | Minimum score for a single detection to initiate tracking        |
+| **Object filters > Person > Confidence threshold** | Minimum computed (median) score to be considered a true positive |
 
 To override score filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" /> and select the camera.
 
@@ -103,12 +103,12 @@ Conceptually, a ratio of 1 is a square, 0.5 is a "tall skinny" box, and 2 is a "
 
 Navigate to <NavPath path="Settings > Global configuration > Objects" /> to set shape filters globally.
 
-| Field                                   | Description                                                              |
-| --------------------------------------- | ------------------------------------------------------------------------ |
-| **Object filters > Person > Min Area**  | Minimum bounding box area in pixels (or decimal for percentage of frame) |
-| **Object filters > Person > Max Area**  | Maximum bounding box area in pixels (or decimal for percentage of frame) |
-| **Object filters > Person > Min Ratio** | Minimum width/height ratio of the bounding box                           |
-| **Object filters > Person > Max Ratio** | Maximum width/height ratio of the bounding box                           |
+| Field                                              | Description                                                              |
+| -------------------------------------------------- | ------------------------------------------------------------------------ |
+| **Object filters > Person > Minimum object area**  | Minimum bounding box area in pixels (or decimal for percentage of frame) |
+| **Object filters > Person > Maximum object area**  | Maximum bounding box area in pixels (or decimal for percentage of frame) |
+| **Object filters > Person > Minimum aspect ratio** | Minimum width/height ratio of the bounding box                           |
+| **Object filters > Person > Maximum aspect ratio** | Maximum width/height ratio of the bounding box                           |
 
 To override shape filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" /> and select the camera.
 

docs/docs/configuration/objects.md

@@ -70,14 +70,14 @@ Object filters help reduce false positives by constraining the size, shape, and
 
 Navigate to <NavPath path="Settings > Global configuration > Objects" />.
 
-| Field                                   | Description                                                              |
-| --------------------------------------- | ------------------------------------------------------------------------ |
-| **Object filters > Person > Min Area**  | Minimum bounding box area in pixels (or decimal for percentage of frame) |
-| **Object filters > Person > Max Area**  | Maximum bounding box area in pixels (or decimal for percentage of frame) |
-| **Object filters > Person > Min Ratio** | Minimum width/height ratio of the bounding box                           |
-| **Object filters > Person > Max Ratio** | Maximum width/height ratio of the bounding box                           |
-| **Object filters > Person > Min Score** | Minimum score for the object to initiate tracking                        |
-| **Object filters > Person > Threshold** | Minimum computed score to be considered a true positive                  |
+| Field                                              | Description                                                              |
+| -------------------------------------------------- | ------------------------------------------------------------------------ |
+| **Object filters > Person > Minimum object area**  | Minimum bounding box area in pixels (or decimal for percentage of frame) |
+| **Object filters > Person > Maximum object area**  | Maximum bounding box area in pixels (or decimal for percentage of frame) |
+| **Object filters > Person > Minimum aspect ratio** | Minimum width/height ratio of the bounding box                           |
+| **Object filters > Person > Maximum aspect ratio** | Maximum width/height ratio of the bounding box                           |
+| **Object filters > Person > Minimum confidence**   | Minimum score for the object to initiate tracking                        |
+| **Object filters > Person > Confidence threshold** | Minimum computed score to be considered a true positive                  |
 
 To override filters for a specific camera, navigate to <NavPath path="Settings > Camera configuration > Objects" />.
 

docs/docs/configuration/profiles.md

@@ -191,14 +191,12 @@ cameras:
     detect:
       enabled: false
     record:
-      enabled: false
+      enabled: true
     profiles:
       away:
         enabled: true
         detect:
           enabled: true
-        record:
-          enabled: true
       home:
         enabled: false
 ```
@@ -251,6 +249,12 @@ Leaving the `objects` section empty (or omitting `track`) does not clear the lis
 
 Fields that require a Frigate restart to take effect cannot be overridden by profiles, since profiles are applied at runtime without restarting. Those fields are hidden when editing a profile override and can only be changed on the base configuration.
 
+### Why can't a profile enable recording when it's disabled in the base config?
+
+Frigate only sets up a camera's recording stream at startup when recording is enabled in the base config, so enabling it later from a profile has no effect. The same applies to turning recording on from the UI or MQTT.
+
+To keep recording off by default, leave `record.enabled: true` in the base config and create a profile that sets `record.enabled: false`. Activate that profile and it will be restored automatically when Frigate starts.
+
 ### Can I schedule profiles to be enabled or disabled at certain times?
 
 Not within Frigate itself. Frigate is an NVR, not an automation platform, so it intentionally does not include a scheduler for activating profiles. Instead, activate profiles from an automation platform that already handles time- and event-based triggers well, such as [Home Assistant](https://www.home-assistant.io/) or [Node-RED](https://nodered.org/). These integrate with Frigate and give you far more robust and flexible scheduling than a built-in scheduler could.

docs/docs/configuration/record.md

@@ -450,7 +450,7 @@ For advanced use cases, the [custom export HTTP API](../integrations/api/export-
 POST /export/custom/{camera_name}/start/{start_time}/end/{end_time}
 ```
 
-The request body accepts `ffmpeg_input_args` and `ffmpeg_output_args` to control encoding, frame rate, filters, and other FFmpeg options. If neither is provided, Frigate defaults to time-lapse output settings (25x speed, 30 FPS).
+The request body accepts `ffmpeg_input_args` and `ffmpeg_output_args` to control encoding, frame rate, filters, and other FFmpeg options. If neither is provided, Frigate defaults to time-lapse output settings (25x speed, 30 FPS) with audio removed (`-an`). When providing your own `ffmpeg_input_args`, include `-an` if you want audio stripped from the export.
 
 The following example exports a time-lapse at 60x speed with 25 FPS:
 

docs/docs/configuration/restream.md

@@ -197,7 +197,7 @@ For cameras that support two-way talk, go2rtc will automatically establish an au
 To prevent this, you must configure two separate stream instances:
 
 1. One stream instance with `#backchannel=0` for Frigate's viewing, recording, and detection (prevents go2rtc from establishing the blocking backchannel)
-2. A second stream instance without `#backchannel=0` for two-way talk functionality (can be used by Frigate's WebRTC viewer or other applications)
+2. A second stream instance with no `#` parameters at all for two-way talk functionality (can be used by Frigate's WebRTC viewer or other applications)
 
 Configuration example:
 
@@ -215,6 +215,8 @@ In this configuration:
 - `front_door` stream is used by Frigate for viewing, recording, and detection. The `#backchannel=0` parameter prevents go2rtc from establishing the audio output backchannel, so it won't block two-way talk access.
 - `front_door_twoway` stream is used for two-way talk functionality. This stream can be used by Frigate's WebRTC viewer when two-way talk is enabled, or by other applications (like Home Assistant Advanced Camera Card) that need access to the camera's audio output channel.
 
+Any `#` parameter on a bare `rtsp://` source disables the backchannel unless the URL explicitly contains `#backchannel=1`. A two-way talk stream with something like `#video=h264` on it silently loses two-way audio, and Frigate will report that two-way talk is unavailable for that stream.
+
 ## Security: Restricted Stream Sources
 
 For security reasons, the `echo:`, `expr:`, and `exec:` stream sources are disabled by default in go2rtc. These sources allow arbitrary command execution and can pose security risks if misconfigured.

docs/docs/configuration/semantic_search.md

@@ -245,8 +245,8 @@ Triggers are best configured through the Frigate UI.
 1. Navigate to <NavPath path="Settings > Enrichments > Triggers" /> and select a camera from the dropdown menu.
 2. Click **Add Trigger** to create a new trigger or use the pencil icon to edit an existing one.
 3. In the **Create Trigger** wizard:
-   - Enter a **Name** for the trigger (e.g., "Red Car Alert").
-   - Enter a descriptive **Friendly Name** for the trigger (e.g., "Red car on the driveway camera").
+   - Enter a **Name** for the trigger (e.g., "Red Car Alert"). Frigate derives the trigger's
+     internal **ID** from this name, which can be revealed and edited with the show/hide toggle.
    - Select the **Type** (`Thumbnail` or `Description`).
    - For `Thumbnail`, select an image to trigger this action when a similar thumbnail image is detected, based on the threshold.
    - For `Description`, enter text to trigger this action when a similar tracked object description is detected.

docs/docs/configuration/zones.md

@@ -28,7 +28,7 @@ During testing, enable the Zones option for the [Debug view](/usage/live#the-sin
 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
 2. Under the **Zones** section, click the plus icon to add a new zone.
 3. Click on the camera's latest image to create the points for the zone boundary. Click the first point again to close the polygon.
-4. Configure zone options such as **Friendly name**, **Objects**, **Loitering time**, and **Inertia** in the zone editor.
+4. Configure zone options such as **Name**, **Objects**, **Loitering Time**, and **Inertia** in the zone editor.
 5. Press **Save** when finished.
 
 </TabItem>
@@ -200,7 +200,7 @@ When using loitering zones, a review item will behave in the following way:
 
 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
 2. Edit or create the zone (e.g., `sidewalk`).
-   - Set **Loitering time** to the desired number of seconds (e.g., `4`)
+   - Set **Loitering Time** to the desired number of seconds (e.g., `4`)
    - Under **Objects**, add the relevant object types (e.g., `person`)
 
 </TabItem>
@@ -291,7 +291,7 @@ Accurate real-world distance measurements are required to estimate speeds. These
 
 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
 2. Create or edit a zone with exactly 4 points aligned to the ground plane.
-3. In the zone editor, enter the real-world **Distances** between each pair of consecutive points.
+3. In the zone editor, enable **Speed Estimation** and enter the real-world **Line A distance**, **Line B distance**, **Line C distance**, and **Line D distance** between each pair of consecutive points.
    - For example, if the distance between the first and second points is 10 meters, between the second and third is 12 meters, etc.
 4. Distances are measured in meters (metric) or feet (imperial), depending on the **Unit system** setting.
 
@@ -358,7 +358,7 @@ Zones can be configured with a minimum speed requirement, meaning an object must
 
 1. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> and select the desired camera.
 2. Edit or create the zone with distances configured.
-   - Set **Speed threshold** to the desired minimum speed (e.g., `20`)
+   - Set **Speed Threshold** to the desired minimum speed (e.g., `20`)
    - The unit is kph or mph, depending on the **Unit system** setting
 
 </TabItem>

docs/docs/development/contributing.md

@@ -204,11 +204,20 @@ Light guidelines and advice:
 npm run lint
 ```
 
-- Add to unit tests and ensure they pass. As much as possible, you should strive to _increase_ test coverage whenever making changes. This will help ensure features do not accidentally become broken in the future.
-- If you run into error messages like "TypeError: Cannot read properties of undefined (reading 'context')" when running tests, this may be due to these issues (https://github.com/vitest-dev/vitest/issues/1910, https://github.com/vitest-dev/vitest/issues/1652) in vitest, but I haven't been able to resolve them.
+- Ensure the backend [unit tests](#unit-tests) pass. Your PR cannot be merged unless tests pass.
+
+```shell
+python3 -u -m unittest
+```
+
+- Ensure the end-to-end tests pass. They run in Playwright against a production build with mocked API data, so they don't need a running Frigate instance. Add or update tests in `web/e2e/specs/` when you change UI behavior.
 
 ```console
-npm run test
+# First-time setup
+npx playwright install chromium
+
+# Build the app and run all tests
+npm run e2e:build && npm run e2e
 ```
 
 - Test in different browsers. Firefox, Chrome, and Safari all have different quirks that make them unique targets to interact with.

docs/docs/frigate/glossary.md

@@ -54,7 +54,7 @@ An object filter mask drops any [bounding box](#bounding-box) whose bottom cente
 
 ## Min Score
 
-The lowest score a detected object can have to be kept during tracking. Anything scoring below the minimum is assumed to be a [false positive](#false-positive) and discarded.
+The lowest score a detected object can have to be kept during tracking. Anything scoring below the minimum is assumed to be a [false positive](#false-positive) and discarded. Set with `min_score` in the config, shown as **Minimum confidence** in the settings UI.
 
 ## Model
 
@@ -86,7 +86,7 @@ A more specific identity assigned to a [tracked object](#tracked-object-event-in
 
 ## Threshold
 
-The median score an object must reach to be considered a true positive.
+The median score an object must reach to be considered a true positive. Set with `threshold` in the config, shown as **Confidence threshold** in the settings UI.
 
 ## Top Score
 

docs/docs/frigate/hardware.md

@@ -54,7 +54,7 @@ Frigate supports multiple different detectors that work on different types of ha
 
 **Most Hardware**
 
-- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices offering a wide range of compatibility with devices.
+- [Hailo](#hailo-8): The Hailo-8, Hailo-8L and Hailo-8R AI Acceleration modules are available in m.2 format with a HAT for RPi devices offering a wide range of compatibility with devices.
   - [Supports many model architectures](../../configuration/object_detectors#configuration-hailo)
   - Runs best with tiny or small size models
 
@@ -65,14 +65,26 @@ Frigate supports multiple different detectors that work on different types of ha
   - [Supports many model architectures](../../configuration/object_detectors#memryx-mx3)
   - Runs best with tiny, small, or medium-size models
 
+- <CommunityBadge /> [DEEPX](#deepx-npu): The DEEPX NPU is available in m.2 format and as a HAT+ for the Raspberry Pi 5, allowing for a wide range of compatibility with devices.
+  - [Supports YOLO model architectures](../../configuration/object_detectors#deepx-npu)
+  - Runs best with tiny or small size models
+  - Runs efficiently on low power hardware
+
 **AMD**
 
 - [ROCm](#rocm---amd-gpu): ROCm can run on AMD Discrete GPUs to provide efficient object detection
   - [Supports limited model architectures](../../configuration/object_detectors#amdrocm-gpu-detector)
   - Runs best on discrete AMD GPUs
+- <CommunityBadge /> [XDNA2 (Ryzen AI)](#amd-xdna2): AMD XDNA2 NPU (sub-watt power AI/ML processor separate to the GPU) inside Strix and other "AI" branded AMD platforms
+  - Has only been tested with YOLOv9, in theory other graphs may be compiled too.
+  - Runs via ZMQ proxy which adds some latency, only recommended for local connection
 
 **Apple Silicon**
 
+- [ONNX via lighter](#apple-silicon): The ONNX detector runs on the Neural Engine of M1 and newer Macs when Frigate runs in the lighter container runtime
+  - [Supports the same model architectures as the ONNX detector](../../configuration/object_detectors#apple-neural-engine-lighter)
+  - Runs inside the Frigate container, with no separate detector process to set up
+  - The recommended way to run Frigate on a Mac
 - [Apple Silicon](#apple-silicon): Apple Silicon is usable on all M1 and newer Apple Silicon devices to provide efficient and fast object detection
   - [Supports primarily ssdlite and mobilenet model architectures](../../configuration/object_detectors#apple-silicon-detector)
   - Runs well with any size models including large
@@ -111,12 +123,13 @@ Frigate supports multiple different detectors that work on different types of ha
 
 ### Hailo-8
 
-Frigate supports both the Hailo-8 and Hailo-8L AI Acceleration Modules on compatible hardware platforms, including the Raspberry Pi 5 with the PCIe hat from the AI kit. The Hailo detector integration in Frigate automatically identifies your hardware type and selects the appropriate default model when a custom model isn’t provided.
+Frigate supports the Hailo-8, Hailo-8L and Hailo-8R AI Acceleration Modules on compatible hardware platforms, including the Raspberry Pi 5 with the PCIe hat from the AI kit. The Hailo detector integration in Frigate identifies which of them is attached and selects the matching default model when a custom model isn’t provided.
 
 **Default Model Configuration:**
 
-- **Hailo-8L:** Default model is **YOLOv6n**.
-- **Hailo-8:** Default model is **YOLOv6n**.
+- **Hailo-8L:** Default model is **YOLOv6n**, compiled for the Hailo-8L.
+- **Hailo-8:** Default model is **YOLOv6n**, compiled for the Hailo-8.
+- **Hailo-8R:** Default model is the **Hailo-8** build of **YOLOv6n**, since the Hailo Model Zoo publishes no Hailo-8R build.
 
 In real-world deployments, even with multiple cameras running concurrently, Frigate has demonstrated consistent performance. Testing on x86 platforms, with dual PCIe lanes, yields further improvements in FPS, throughput, and latency compared to the Raspberry Pi setup.
 
@@ -205,7 +218,13 @@ Inference is done with the `onnx` detector type. Speeds will vary greatly depend
 
 ### Apple Silicon
 
-With the [Apple Silicon](../configuration/object_detectors.md#apple-silicon-detector) detector Frigate can take advantage of the NPU in M1 and newer Apple Silicon.
+Frigate on a Mac is best run in the [lighter](https://github.com/fieldwork-ai/lighter) container runtime, where the [ONNX detector](../configuration/object_detectors.md#apple-neural-engine-lighter) runs on the Neural Engine of M1 and newer Macs from inside the Frigate container. There is no separate detector process to install or keep running, and the same container can decode video on the Mac's media engine.
+
+| Name | YOLOv9 Inference Time                  | YOLO-NAS Inference Time | RF-DETR Inference Time |
+| ---- | -------------------------------------- | ----------------------- | ---------------------- |
+| M1   | t-320: 3.3 ms s-320: 7 ms s-640: 13 ms | 320: 6.6 ms             | Nano-320: 38 ms        |
+
+Alternatively, with the [Apple Silicon](../configuration/object_detectors.md#apple-silicon-detector) detector Frigate can take advantage of the NPU in M1 and newer Apple Silicon.
 
 :::warning
 
@@ -256,6 +275,32 @@ The MX3 is a pipelined architecture, where the maximum frames per second support
 
 Inference speeds may vary depending on the host platform. The above data was measured on an **Intel 13700 CPU**. Platforms like Raspberry Pi, Orange Pi, and other ARM-based SBCs have different levels of processing capability, which may limit total FPS.
 
+### DEEPX NPU
+
+Frigate supports the DEEPX NPU in both of its form factors: the **DX-M1** M.2 module, which works on x86 (Intel/AMD) and ARM-based SBCs such as the Raspberry Pi 5, and the **DX-M1M** on the [Sixfab AI HAT+](https://docs.sixfab.com/docs/ai-hat-plus-raspberry-pi-5-quickstart) for the Raspberry Pi 5. Both use the same driver and runtime, so the configuration is identical for either one. DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
+
+The DEEPX driver and runtime run on the Docker host rather than inside the Frigate container and must be installed before the NPU can be used. See the [installation docs](installation.md#deepx-npu) for the setup steps and [the detector docs](/configuration/object_detectors#deepx-npu) for the configuration.
+
+Frigate does not bundle a model for this detector. Models use DEEPX's `.dxnn` format, and pre-compiled YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Prefer a model with a `_ppu` suffix whenever one is available for the architecture you want: these run part of the post-processing on the NPU itself and are considerably faster, roughly 2.5x for the same architecture and input size. **YOLOX-S with PPU is the recommended starting point.**
+
+Inference times for a few recommended models, measured through Frigate's own stats on a DX-M1:
+
+| Model             | Input Size | DX-M1 Inference Time |
+| ----------------- | ---------- | -------------------- |
+| YOLOX-S (PPU)     | 640        | ~ 13 ms              |
+| YOLOv9-t (PPU)    | 640        | ~ 18 ms              |
+| YOLOv4 (PPU)      | 512        | ~ 20 ms              |
+| YOLOX-S           | 640        | ~ 34 ms              |
+| YOLOv9-s          | 640        | ~ 39 ms              |
+
+Other ModelZoo YOLO variants are also supported but have not been measured. Inference speeds vary with the host platform, so a slower host such as a Raspberry Pi 5 will report higher times than those above.
+
+:::note
+
+A few ModelZoo models can not be used with Frigate: SSD models (they are trained on Pascal VOC, so their labels do not match Frigate's), DAMO-YOLO models, face and pose models, and the PPU builds of YOLOv7.
+
+:::
+
 ### Nvidia Jetson
 
 Jetson devices are supported via the TensorRT or ONNX detectors when running Jetpack 6. It will [make use of the Jetson's hardware media engine](/configuration/hardware_acceleration_video#nvidia-jetson) when configured with the [appropriate presets](/configuration/ffmpeg_presets#hwaccel-presets), and will make use of the Jetson's GPU and DLA for object detection when configured with the [TensorRT detector](/configuration/object_detectors#nvidia-tensorrt-detector).
@@ -296,6 +341,32 @@ The inference time of a rk3588 with all 3 cores enabled is typically 25-30 ms fo
 | ---------------- | ----------------------------------- |
 | yolov9-tiny      | ~ 4 ms                              |
 
+### AMD Ryzen AI / XDNA2
+
+Frigate supports AMD XDNA2 NPUs through the community-maintained
+frigate-xdna ZMQ sidecar. It works with stock Frigate and supports
+Frigate+ models or compatible local YOLO ONNX models. Models are compiled
+once on the target system and cached for subsequent use.
+
+Currently qualified on **Ryzen AI Max 300 / Strix Halo**. Other XDNA2
+devices are not yet qualified; XDNA1 is unsupported.
+
+Measured YOLOv9 detector latency on Strix Halo:
+
+| Model | 320 | 640 |
+| ----- | ---: | ---: |
+| YOLOv9-T | ~7.4 ms | unsupported |
+| YOLOv9-S | ~9.0 ms | ~20.0 ms |
+| YOLOv9-M | ~13.1 ms | ~34.4 ms |
+| YOLOv9-C | ~14.1 ms | ~35.2 ms |
+| YOLOv9-E | ~69.4 ms | ~224.8 ms |
+
+**YOLOv9-C at 320 is the recommended quality/performance balance.**
+C at 640 is also usable where the lower throughput is acceptable.
+
+Setup, model preparation, and compatibility details are available
+[in the frigate-xdna documentation](https://github.com/mitchins/frigate-xdna).
+
 ## What does Frigate use the CPU for and what does it use a detector for? (ELI5 Version)
 
 This is taken from a [user question on reddit](https://www.reddit.com/r/homeassistant/comments/q8mgau/comment/hgqbxh5/?utm_source=share&utm_medium=web2x&context=3). Modified slightly for clarity.

docs/docs/frigate/installation.md

@@ -122,7 +122,7 @@ Additionally, the USB Coral draws a considerable amount of power. If using any o
 
 ### Hailo-8
 
-The Hailo-8 and Hailo-8L AI accelerators are available in both M.2 and HAT form factors for the Raspberry Pi. The M.2 version typically connects to a carrier board for PCIe, which then interfaces with the Raspberry Pi 5 as part of the AI Kit. The HAT version can be mounted directly onto compatible Raspberry Pi models. Both form factors have been successfully tested on x86 platforms as well, making them versatile options for various computing environments.
+The Hailo-8, Hailo-8L and Hailo-8R AI accelerators are available in both M.2 and HAT form factors for the Raspberry Pi. The M.2 version typically connects to a carrier board for PCIe, which then interfaces with the Raspberry Pi 5 as part of the AI Kit. The HAT version can be mounted directly onto compatible Raspberry Pi models. Both form factors have been successfully tested on x86 platforms as well, making them versatile options for various computing environments.
 
 The HailoRT runtime is not part of the Frigate image; Frigate downloads and installs it at first start once a Hailo detector is configured. Containers without internet access can provide the files themselves, see [Detector runtimes](/frigate/network_requirements#detector-runtimes).
 
@@ -300,7 +300,7 @@ If you are using `docker run`, add this option to your command `--device /dev/ha
 
 #### Configuration
 
-Finally, configure [hardware object detection](/configuration/object_detectors#hailo-8) to complete the setup.
+Finally, configure [hardware object detection](/configuration/object_detectors#hailo) to complete the setup.
 
 ### MemryX MX3
 
@@ -381,6 +381,99 @@ If you can't use Docker Compose, you can run the container with something simila
 
 Finally, configure [hardware object detection](/configuration/object_detectors#memryx-mx3) to complete the setup.
 
+### DEEPX NPU
+
+The DEEPX NPU is available in two form factors, and Frigate supports both:
+
+- **DX-M1** in the M.2 2280 form factor (like an NVMe SSD), for x86 (Intel/AMD) PCs, the Raspberry Pi 5, and other ARM SBCs with an exposed PCIe M.2 slot.
+- **DX-M1M** on the [Sixfab AI HAT+](https://docs.sixfab.com/docs/ai-hat-plus-raspberry-pi-5-quickstart), a HAT+ board that connects to the Raspberry Pi 5 over PCIe Gen 3 x1.
+
+Both present the NPU through the same PCIe driver and DX-RT runtime, so the setup below and the detector configuration are identical for either one. Nothing needs to change when moving between them.
+
+DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
+
+#### Versions
+
+A DEEPX install has several separately versioned pieces, and they all have to agree. The driver, the runtime, and the daemon live on the Docker host; Frigate itself carries only the Python bindings, which it downloads on first start:
+
+| Component      | Version  | Installed on | Installed by              |
+| -------------- | -------- | ------------ | ------------------------- |
+| Kernel driver  | `v2.6.0` | Host  | `user_installation.sh`    |
+| DX-RT runtime  | `v3.4.0` | Host  | `user_installation.sh`    |
+| NPU firmware   | `v2.7.4` | The module   | Flashed from the host     |
+| DX-RT bindings | `v3.4.0` | Frigate      | Downloaded at first start |
+
+:::warning
+
+A version mismatch does not produce a startup error. It typically shows up as inference requests that are accepted but never return a result, so detections simply stop appearing while Frigate looks healthy. If that happens after a Frigate upgrade, check every version in the table before anything else.
+
+:::
+
+The installation script installs the DX-RT runtime on the host and enables `dxrt.service`, so the daemon starts at boot and any other program on the host can share the NPU with Frigate. Check the firmware version with `dxrt-cli --status` and update the module if it does not match the table above.
+
+#### Installation
+
+The DEEPX kernel driver must be installed on the host rather than in the container, because containers share the host kernel and cannot load kernel modules. Installing it creates the `/dev/dxrt*` device nodes that are passed through to Frigate. The same script installs the DX-RT runtime and enables `dxrt.service`, the daemon that owns the NPU and hands work to it on behalf of Frigate and anything else on the host.
+
+1. Copy or download [this script](https://github.com/blakeblackshear/frigate/blob/dev/docker/deepx/user_installation.sh).
+2. Ensure it has execution permissions with `sudo chmod +x user_installation.sh`
+3. Run the script with `./user_installation.sh`
+4. **Restart your computer** to complete driver installation.
+
+Confirm the NPU is visible before continuing:
+
+```bash
+ls /dev/dxrt*
+```
+
+Then confirm the daemon is running and listening in `/run/dxrt`:
+
+```bash
+systemctl is-active dxrt.service
+ls /run/dxrt/
+```
+
+#### Setup
+
+To set up Frigate, follow the default installation instructions, for example: `ghcr.io/blakeblackshear/frigate:stable`
+
+#### Docker configuration
+
+Frigate needs the NPU device node and the directory holding the daemon's socket:
+
+```yaml
+services:
+  frigate:
+    devices:
+      - /dev/dxrt0:/dev/dxrt0
+    volumes:
+      - /run/dxrt:/run/dxrt
+```
+
+If you can't use Docker Compose, add `--device /dev/dxrt0:/dev/dxrt0 -v /run/dxrt:/run/dxrt` to your `docker run` command.
+
+Add one `--device` per NPU, contiguously from `/dev/dxrt0`, since the client stops enumerating at the first gap.
+
+The installation script configures `dxrt.service` to place its socket in `/run/dxrt` through a systemd drop-in. Mounting the directory rather than the socket file means the container sees the new socket after `dxrt.service` is restarted, rather than holding on to a deleted one.
+
+`dxrtd` listens on an abstract socket as well, but that one does not cross into a container, so Frigate names the filesystem socket through `DXRT_DYNAMIC_IPC_ENDPOINT` on your behalf. Set that variable on the container yourself only if the daemon listens somewhere else, which means you also set it for `dxrtd` through its own systemd drop-in. The script writes `/etc/systemd/system/dxrt.service.d/frigate.conf` for exactly that, and has `dxrt.service` link the socket to `/tmp/dxrt_dynamic_ipc.sock` when it starts, so the host's own `dxrt-cli` and `dxtop` keep finding it at the default path they fall back to.
+
+:::note
+
+The DX-RT client exits when `dxrt.service` stops, so restart the Frigate container after restarting `dxrt.service`.
+
+:::
+
+The device node is needed as well as the socket, because the client opens the NPU directly even though the daemon arbitrates access. Without it, inference fails with `Device not found`.
+
+`/dev/shm` does not need sharing.
+
+The DX-RT python bindings are not shipped in the Frigate image. Frigate downloads them on first start when a DEEPX detector is configured, and caches them under `/config`.
+
+#### Configuration
+
+Finally, configure [hardware object detection](/configuration/object_detectors#deepx-npu) to complete the setup.
+
 ### Rockchip platform
 
 Make sure that you use a linux distribution that comes with the rockchip BSP kernel 5.10 or 6.1 and necessary drivers (especially rkvdec2 and rknpu). To check, enter the following commands:

docs/docs/frigate/network_requirements.md

@@ -47,7 +47,7 @@ If you are using one of the following hardware detectors and have not provided y
 | Detector                                                           | Model Downloaded     | Source                   |
 | ------------------------------------------------------------------ | -------------------- | ------------------------ |
 | [Rockchip RKNN](/configuration/object_detectors#rockchip-platform) | RKNN detection model | GitHub                   |
-| [Hailo 8 / 8L](/configuration/object_detectors#hailo-8)            | YOLOv6n (.hef)       | Hailo Model Zoo (AWS S3) |
+| [Hailo 8 / 8L / 8R](/configuration/object_detectors#hailo)         | YOLOv6n (.hef)       | Hailo Model Zoo (AWS S3) |
 | [AXERA AXEngine](/configuration/object_detectors)                  | Detection model      | HuggingFace              |
 
 :::note
@@ -60,16 +60,16 @@ The default CPU, EdgeTPU, and OpenVINO object detection models are bundled into
 
 The SDKs for a few hardware detectors are not shipped in the Frigate image. They are downloaded the first time that detector is configured, verified against checksums pinned in the Frigate release, and installed into the Frigate user's home directory (`/config/.local` by default). Once installed they are not downloaded again until a Frigate release pins a new version.
 
-| Detector                                                       | Version | Files                                                                                                                                                                                                 | Source                                                                                     |
-| -------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
-| [Hailo 8 / 8L](/configuration/object_detectors#hailo-8)        | 4.21.0  | `hailort-debian12-amd64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_x86_64.whl` on x86, `hailort-debian12-arm64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_aarch64.whl` on arm64 | [GitHub release](https://github.com/frigate-nvr/hailort/releases/tag/v4.21.0)              |
-| [MemryX MX3](/configuration/object_detectors#memryx-mx3)       | 2.1.0   | `mx_accl_frigate-2.1.0.zip` (the release source archive, renamed)                                                                                                                                     | [GitHub archive](https://github.com/memryx/mx_accl_frigate/archive/refs/tags/v2.1.0.zip)   |
-| [AXERA AXEngine](/configuration/object_detectors#axera)        | 0.1.3   | `axengine-0.1.3-py3-none-any.whl`                                                                                                                                                                     | [GitHub release](https://github.com/AXERA-TECH/pyaxengine/releases/tag/0.1.3-frigate)      |
+| Detector                                                   | Version | Files                                                                                                                                                                                 | Source                                                                                   |
+| ---------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
+| [Hailo 8 / 8L / 8R](/configuration/object_detectors#hailo) | 4.21.0  | `hailort-debian12-amd64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_x86_64.whl` on x86, `hailort-debian12-arm64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_aarch64.whl` on arm64 | [GitHub release](https://github.com/frigate-nvr/hailort/releases/tag/v4.21.0)            |
+| [MemryX MX3](/configuration/object_detectors#memryx-mx3)   | 2.1.0   | `mx_accl_frigate-2.1.0.zip` (the release source archive, renamed)                                                                                                                     | [GitHub archive](https://github.com/memryx/mx_accl_frigate/archive/refs/tags/v2.1.0.zip) |
+| [AXERA AXEngine](/configuration/object_detectors#axera)    | 0.1.3   | `axengine-0.1.3-py3-none-any.whl`                                                                                                                                                     | [GitHub release](https://github.com/AXERA-TECH/pyaxengine/releases/tag/0.1.3-frigate)    |
 
 If the container cannot reach GitHub, provide the files yourself:
 
 1. Download the files for your architecture on a machine with internet access.
-2. Place them, with exactly the file names listed above, in `/config/model_cache/runtimes/<detector>/`, where `<detector>` is the detector `type` from your config (`hailo8l`, `memryx`, or `axengine`).
+2. Place them, with exactly the file names listed above, in `/config/model_cache/runtimes/<detector>/`, where `<detector>` is the detector named in your config's `devices` (`hailo`, `memryx`, or `axengine`).
 3. Start Frigate. Files whose checksum matches are installed without any download; a file with the wrong checksum is discarded and downloaded again, so a failed startup log names the file to replace.
 
 The `GITHUB_ENDPOINT` mirror variable below applies to these downloads as well.
@@ -142,16 +142,12 @@ When [notifications](/configuration/notifications) are enabled and users have re
 
 If an [MQTT broker](/integrations/mqtt) is configured, Frigate maintains a connection to the broker's host and port. This is typically a local network connection, but will require internet if you use a cloud-hosted MQTT broker.
 
-### DeepStack / CodeProject.AI
-
-When using the [DeepStack detector plugin](/configuration/object_detectors), Frigate sends images to the configured API endpoint for inference. This is typically local but depends on where the service is hosted.
-
 ## WebRTC (STUN)
 
 For [WebRTC live streaming](/configuration/live), Frigate uses STUN for NAT traversal:
 
 - **go2rtc** defaults to a local STUN listener (`stun:8555`), no internet required.
-- **The web UI's WebRTC player** includes a fallback to Google's public STUN server (`stun:stun.l.google.com:19302`), which requires internet.
+- **The web UI** uses the servers in `go2rtc.webrtc.ice_servers` for its WebRTC player and for the WebRTC connectivity check it runs when the Live view loads. If none are set, it uses Google's public STUN server (`stun:stun.l.google.com:19302`), which requires internet access from the browser. Set `ice_servers` to a STUN or TURN server on your network to avoid this.
 
 ## Home Assistant Supervisor
 

docs/docs/integrations/mqtt.md

@@ -11,6 +11,12 @@ MQTT requires a network connection to your broker. This is typically local, but
 
 :::
 
+:::note
+
+Wherever a topic below includes a camera, mask, or zone name, use its `ID` from the config, not its `friendly_name`. For example, a camera with `friendly_name: "Back Yard"` and ID `back_yard` publishes to `frigate/back_yard/...`, not `frigate/Back Yard/...`.
+
+:::
+
 ## General Frigate Topics
 
 ### `frigate/available`
@@ -212,6 +218,7 @@ An `update` with the same ID will be published when:
 - The severity changes from `detection` to `alert`
 - Additional objects are detected
 - An object is recognized via face, lpr, etc.
+- A [state classification](/configuration/custom_classification/state_classification#review-items) model changes state
 
 When the review activity has ended a final `end` message is published.
 
@@ -235,7 +242,8 @@ When the review activity has ended a final `end` message is published.
       "objects": ["person", "car"],
       "sub_labels": [],
       "zones": [],
-      "audio": []
+      "audio": [],
+      "classification_state_changes": []
     }
   },
   "after": {
@@ -254,7 +262,16 @@ When the review activity has ended a final `end` message is published.
       "objects": ["person", "car"],
       "sub_labels": ["Bob"],
       "zones": ["front_yard"],
-      "audio": []
+      "audio": [],
+      "classification_state_changes": [
+        // verified changes of state classification models on this camera
+        {
+          "model": "front_gate",
+          "from": "closed",
+          "to": "open",
+          "timestamp": 1718987131.52
+        }
+      ]
     }
   }
 }

docs/docs/integrations/third_party_extensions.md

@@ -27,6 +27,10 @@ The [Advanced Camera Card](https://card.camera/#/README) is a Home Assistant das
 It supports automatically setting the sub labels in Frigate for person objects that are detected and recognized.
 This is a fork (with fixed errors and new features) of [original Double Take](https://github.com/jakowenko/double-take) project which, unfortunately, isn't being maintained by author.
 
+## [frigate-abr](https://github.com/007hacky007/frigate-abr)
+
+[frigate-abr](https://github.com/007hacky007/frigate-abr) is a drop-in Docker image of Frigate that adds adaptive bitrate (ABR) playback for recordings: a sidecar transcodes footage to lower quality tiers on demand, for reviewing over slow remote connections. Segments are transcoded when played and cached, so no additional stream is recorded. Frigate itself is not modified.
+
 ## [Frigate Notify](https://github.com/0x2142/frigate-notify)
 
 [Frigate Notify](https://github.com/0x2142/frigate-notify) is a simple app designed to send notifications from Frigate to your favorite platforms. Intended to be used with standalone Frigate installations - Home Assistant not required, MQTT is optional but recommended.

docs/docs/plus/faq.md

@@ -21,7 +21,13 @@ Yes. Models and metadata are stored in the `model_cache` directory within the co
 
 ### Can I keep using my Frigate+ models even if I do not renew my subscription?
 
-Yes. Subscriptions to Frigate+ provide access to the infrastructure used to train the models. Models trained with your subscription are yours to keep and use forever. However, do note that the terms and conditions prohibit you from sharing, reselling, or creating derivative products from the models.
+Yes. Subscriptions to Frigate+ provide access to the infrastructure used to train the models. Models you train during an active subscription remain licensed for your continued use even after your subscription ends — models already in your model cache will keep working indefinitely. An active subscription is required to train new models and download new versions.
+
+### Can I use Frigate+ models commercially?
+
+A standard subscription covers use on camera systems you own or operate, including for your business. A shop, restaurant, warehouse, or office running Frigate+ at its own locations (including multiple locations) is exactly the kind of use the subscription is for.
+What the standard subscription does not cover is using Frigate+ models to provide a product or service to others. If you're deploying models at your customers' sites, bundling them with hardware you sell, or running them as part of a hosted or managed service, even if your customers never receive the model files themselves, you'll need a commercial license.
+Note that professional installers are fine under standard subscriptions when each customer holds their own Frigate+ subscription. The commercial license is for cases where your license powers your customers' sites.
 
 ### Why can't I submit images to Frigate+?
 

docs/docs/plus/first_model.md

@@ -63,20 +63,20 @@ Frigate+ models generally have much higher scores than the default model provide
 <ConfigTabs>
 <TabItem value="ui">
 
-Navigate to <NavPath path="Settings > Global configuration > Objects" />. Under **Object filters**, set **Min Score** and **Threshold** for each object type, then click **Save**.
+Navigate to <NavPath path="Settings > Global configuration > Objects" />. Under **Object filters**, set **Minimum confidence** and **Confidence threshold** for each object type, then click **Save**.
 
-| Object            | Min Score | Threshold |
-| ----------------- | --------- | --------- |
-| **dog**           | .7        | .9        |
-| **cat**           | .65       | .8        |
-| **face**          | .7        |           |
-| **package**       | .65       | .9        |
-| **license_plate** | .6        |           |
-| **amazon**        | .75       |           |
-| **ups**           | .75       |           |
-| **fedex**         | .75       |           |
-| **person**        | .65       | .85       |
-| **car**           | .65       | .85       |
+| Object            | Minimum confidence | Confidence threshold |
+| ----------------- | ------------------ | -------------------- |
+| **dog**           | .7                 | .9                   |
+| **cat**           | .65                | .8                   |
+| **face**          | .7                 |                      |
+| **package**       | .65                | .9                   |
+| **license_plate** | .6                 |                      |
+| **amazon**        | .75                |                      |
+| **ups**           | .75                |                      |
+| **fedex**         | .75                |                      |
+| **person**        | .65                | .85                  |
+| **car**           | .65                | .85                  |
 
 </TabItem>
 <TabItem value="yaml">

docs/docs/plus/index.md

@@ -41,7 +41,7 @@ Rockchip models are automatically converted as of 0.17. For 0.16, YOLOv9 onnx mo
 
 ## Supported detector types
 
-Currently, Frigate+ models support CPU (`cpu`), Google Coral (`edgetpu`), OpenVino (`openvino`), ONNX (`onnx`), Hailo (`hailo8l`), and Rockchip (`rknn`) detectors.
+Currently, Frigate+ models support CPU (`cpu`), Google Coral (`edgetpu`), OpenVino (`openvino`), ONNX (`onnx`), Hailo (`hailo`), and Rockchip (`rknn`) detectors.
 
 | Hardware                                                                         | Recommended Detector Type | Recommended Model Type |
 | -------------------------------------------------------------------------------- | ------------------------- | ---------------------- |
@@ -50,7 +50,7 @@ Currently, Frigate+ models support CPU (`cpu`), Google Coral (`edgetpu`), OpenVi
 | [Intel](/configuration/object_detectors.md#openvino-detector)                    | `openvino`                | `yolov9`               |
 | [NVidia GPU](/configuration/object_detectors#onnx)                               | `onnx`                    | `yolov9`               |
 | [AMD ROCm GPU](/configuration/object_detectors#amdrocm-gpu-detector)             | `onnx`                    | `yolov9`               |
-| [Hailo8/Hailo8L/Hailo8R](/configuration/object_detectors#hailo-8)                | `hailo8l`                 | `yolov9`               |
+| [Hailo8/Hailo8L/Hailo8R](/configuration/object_detectors#hailo)                  | `hailo`                   | `yolov9`               |
 | [Rockchip NPU](/configuration/object_detectors#rockchip-platform)                | `rknn`                    | `yolov9`               |
 
 ## Improving your model
@@ -65,11 +65,11 @@ Some users may find that Frigate+ models result in more false positives initiall
 
 Frigate+ models support a more relevant set of objects for security cameras. The labels for annotation in Frigate+ are configurable by editing the camera in the Cameras section of Frigate+. Currently, the following objects are supported:
 
-- **People**: `person`, `face`
-- **Vehicles**: `car`, `motorcycle`, `bicycle`, `boat`, `school_bus`, `license_plate`
+- **People**: `person`, `face`, `baby`
+- **Vehicles**: `car`, `motorcycle`, `bicycle`, `boat`, `school_bus`, `garbage truck`, `license_plate`
 - **Delivery Logos**: `amazon`, `usps`, `ups`, `fedex`, `dhl`, `an_post`, `purolator`, `postnl`, `nzpost`, `postnord`, `gls`, `dpd`, `canada_post`, `royal_mail`
-- **Animals**: `dog`, `cat`, `deer`, `horse`, `bird`, `raccoon`, `fox`, `bear`, `cow`, `squirrel`, `goat`, `rabbit`, `skunk`, `kangaroo`
-- **Other**: `package`, `waste_bin`, `bbq_grill`, `robot_lawnmower`, `umbrella`
+- **Animals**: `dog`, `cat`, `deer`, `horse`, `bird`, `raccoon`, `fox`, `bear`, `cow`, `squirrel`, `goat`, `rabbit`, `skunk`, `kangaroo`, `possum`, `rodent`
+- **Other**: `package`, `waste_bin`, `bbq_grill`, `robot_lawnmower`, `umbrella`, `baby_stroller`
 
 Other object types available in the default Frigate model are not available. Additional object types will be added in future releases.
 
@@ -77,9 +77,12 @@ Other object types available in the default Frigate model are not available. Add
 
 Candidate labels are also available for annotation. These labels don't have enough data to be included in the model yet, but using them will help add support sooner. You can enable these labels by editing the camera settings.
 
-Where possible, these labels are mapped to existing labels during training. For example, any `baby` labels are mapped to `person` until support for new labels is added.
+Where possible, these labels are mapped to existing labels during training. For example, any `duck` labels are mapped to `bird` until support for new labels is added.
 
-The candidate labels are: `baby`, `bpost`, `badger`, `possum`, `rodent`, `chicken`, `groundhog`, `boar`, `hedgehog`, `tractor`, `golf cart`, `garbage truck`, `bus`, `sports ball`, `la_poste`, `lawnmower`, `heron`, `rickshaw`, `wombat`, `auspost`, `aramex`, `bobcat`, `mustelid`, `transoflex`, `airplane`, `drone`, `mountain_lion`, `crocodile`, `turkey`, `baby_stroller`, `monkey`, `coyote`, `porcupine`, `parcelforce`, `sheep`, `snake`, `helicopter`, `lizard`, `duck`, `hermes`, `cargus`, `fan_courier`, `sameday`
+- **Vehicles**: `tractor`, `golf_cart`, `bus`, `airplane`, `helicopter`, `rickshaw`, `scooter`
+- **Delivery Logos**: `bpost`, `auspost`, `aramex`, `transoflex`, `parcelforce`, `hermes`, `cargus`, `fan_courier`, `sameday`, `la_poste`
+- **Animals**: `badger`, `chicken`, `duck`, `turkey`, `groundhog`, `boar`, `hedgehog`, `wombat`, `bobcat`, `mustelid`, `mountain_lion`, `crocodile`, `monkey`, `coyote`, `porcupine`, `sheep`, `snake`, `lizard`, `heron`, `elk`, `moose`, `pig`, `donkey`, `civet`
+- **Other**: `sports_ball`, `drone`, `lawnmower`
 
 Candidate labels are not available for automatic suggestions.
 

docs/docs/troubleshooting/faqs.md

@@ -39,6 +39,20 @@ To do this efficiently the following setup is required:
 
 When this is done correctly, the GPU will do the decoding and scaling which will result in a small increase in CPU usage but with better results.
 
+### How can I rotate my camera's video feed?
+
+Rotation is best done in the camera's firmware settings (usually called rotate, flip, or corridor mode) so the video arrives already rotated and no extra processing is needed. Check there first.
+
+If your camera does not support rotation, go2rtc's ffmpeg module can rotate the stream with the `#rotate` parameter (`90`, `180`, `270`, or `-90`), but this is not recommended: rotation requires transcoding (re-encoding) the video, which significantly increases CPU usage, especially for high resolution streams.
+
+```yaml
+go2rtc:
+  streams:
+    my_camera: "ffmpeg:rtsp://user:password@192.168.1.10:554/stream#video=h264#hardware#rotate=90"
+```
+
+Point the camera's inputs at the restream as described in the [restream docs](/configuration/restream.md), and swap `detect -> width` and `detect -> height` to match the rotated resolution.
+
 ### My mjpeg stream or snapshots look green and crazy
 
 This almost always means that the width/height defined for your camera are not correct. Double check the resolution with VLC or another player. Also make sure you don't have the width and height values backwards.
@@ -170,6 +184,6 @@ Filters and masks only hide the incorrect result - they don't teach Frigate what
 
 ### Where do I see problems Frigate has detected?
 
-Open System > Health. The Notices list keeps a record of problems Frigate has found, and you can dismiss any entry to acknowledge it. Ongoing conditions, such as an offline camera or recordings deleted before their retention period, appear in the status bar for admins until they clear, and the status bar links to the Notices list while it has undismissed entries. On mobile, tap the warning icon in the bottom navigation bar to see them.
+Open System > Health. The Notices list keeps a record of problems Frigate has found. Acknowledge an entry to hide it until the problem happens again, or mute it to hide it for good. Hidden entries stay listed under Show hidden in the filter. Ongoing conditions, such as an offline camera or recordings deleted before their retention period, appear in the status bar for admins until they clear, and the status bar links to the Notices list while it has entries showing. On mobile, tap the warning icon in the bottom navigation bar to see them.
 
 The Hardware section below the notices shows whether the detection hardware, hardware acceleration, and enrichment devices in your config were found and are being used, so a GPU that silently fell back to the CPU shows up as a warning. Run stream checks to probe every camera's streams for the same problems the camera wizard reports.

docs/docs/troubleshooting/go2rtc.md

@@ -78,7 +78,9 @@ go2rtc:
 
 :::warning
 
-The `#`-modifiers (`#video=`, `#audio=`, `#hardware`, `#backchannel=0`, …) **only take effect on a source that is prefixed with `ffmpeg:`**. Adding them to a bare `rtsp://…#audio=opus` source does nothing: go2rtc ignores them. Likewise, when a source references another stream by name (e.g. `ffmpeg:back#audio=aac`), the name must match the stream key **exactly** (it is case sensitive), or the transcode is silently never produced. This is the single most common configuration mistake. In the Frigate UI, the **Use compatibility mode (ffmpeg)** toggle adds the `ffmpeg:` prefix for you.
+The transcoding modifiers (`#video=`, `#audio=`, `#hardware`, …) **only take effect on a source that is prefixed with `ffmpeg:`**. Adding them to a bare `rtsp://…#audio=opus` source does nothing: go2rtc ignores them. Likewise, when a source references another stream by name (e.g. `ffmpeg:back#audio=aac`), the name must match the stream key **exactly** (it is case sensitive), or the transcode is silently never produced. This is the single most common configuration mistake. In the Frigate UI, the **Use compatibility mode (ffmpeg)** toggle adds the `ffmpeg:` prefix for you.
+
+A bare `rtsp://` source reads a different set of modifiers: `#backchannel=`, `#media=`, `#timeout=`, and `#transport=`. These do nothing on an `ffmpeg:` source. Adding **any** modifier to a bare `rtsp://` source also disables the camera's backchannel unless the URL explicitly contains `#backchannel=1`, so a stream dedicated to two-way talk should carry no modifiers at all.
 
 :::
 
@@ -153,7 +155,7 @@ WebRTC is only attempted when MSE fails or when using a camera's two-way talk fe
 
 - **Codec mismatch**: WebRTC cannot carry H.265 or AAC. The stream backing the WebRTC view must provide Opus (or PCMA/PCMU) audio and H.264 video. Add an `ffmpeg:back#audio=opus` source as shown above.
 - **Port `8555` not reachable, or no candidates set**: WebRTC needs port `8555` (both TCP and UDP) open and a reachable candidate advertised. On Docker installs running on a custom/overlay network, go2rtc may advertise unreachable container IPs as ICE candidates; setting `webrtc.filters.candidates: []` and supplying only your host's LAN IP resolves this. See [WebRTC extra configuration](/configuration/live#webrtc-extra-configuration).
-- **Two-way talk** additionally requires a secure context (HTTPS or the authenticated port `8971`, because browsers block microphone access on plain HTTP). The camera's RTSP backchannel must also be handled correctly: go2rtc seizes the backchannel by default, which blocks two-way audio for other consumers and can inject static. Disable it on the primary stream with `#backchannel=0` and use a separate dedicated stream for talk, as documented in [preventing go2rtc from blocking two-way audio](/configuration/restream#two-way-talk-restream).
+- **Two-way talk** additionally requires a secure context (HTTPS or the authenticated port `8971`, because browsers block microphone access on plain HTTP). The camera's RTSP backchannel must also be handled correctly: go2rtc seizes the backchannel by default, which blocks two-way audio for other consumers and can inject static. Disable it on the primary stream with `#backchannel=0` and use a separate dedicated stream for talk, carrying no `#` modifiers of any kind, as documented in [preventing go2rtc from blocking two-way audio](/configuration/restream#two-way-talk-restream).
 
 ## High CPU usage
 

docs/docs/troubleshooting/recordings.md

@@ -397,19 +397,11 @@ dmesg | grep -i -E "gpu|drm|reset|hang"
 
 Messages like `trying reset from guc_exec_queue_timedout_job` or similar GPU reset/hang messages indicate a driver or hardware issue. Ensure your kernel and GPU drivers (especially Intel) are up to date.
 
-#### Step 6: Verify hardware acceleration configuration
-
-An incorrect `hwaccel_args` preset can cause ffmpeg to fail silently or consume excessive CPU, starving the detector of resources.
-
-- After upgrading Frigate, verify your preset matches your hardware (e.g., `preset-intel-qsv-h264` instead of the deprecated `preset-vaapi`).
-- For h265 cameras, use the corresponding h265 preset (e.g., `preset-intel-qsv-h265`).
-- Note that `hwaccel_args` are only relevant for the detect stream. Frigate does not decode the record stream.
-
-#### Step 7: Verify go2rtc stream configuration
+#### Step 6: Verify go2rtc stream configuration
 
 Ensure that the ffmpeg source names in your go2rtc configuration match the correct camera stream. A misconfigured stream name (e.g., copying a config from one camera to another without updating the stream reference) will cause the wrong stream to be used or the stream to fail entirely.
 
-#### Step 8: Check system resources
+#### Step 7: Check system resources
 
 If none of the above apply, the issue may be a general resource constraint. Monitor the following on your host:
 

docs/docusaurus.config.ts

@@ -3,6 +3,9 @@ import * as path from "node:path";
 import type { Config, PluginConfig } from "@docusaurus/types";
 import type * as OpenApiPlugin from "docusaurus-plugin-openapi-docs";
 
+// Bump when a new stable release ships
+const STABLE_VERSION = "0.18";
+
 const config: Config = {
   title: "Frigate",
   tagline: "NVR With Realtime Object Detection for IP Cameras",
@@ -23,17 +26,17 @@ const config: Config = {
     mermaid: true,
   },
   i18n: {
-    defaultLocale: 'en',
-    locales: ['en'],
+    defaultLocale: "en",
+    locales: ["en"],
     localeConfigs: {
       en: {
-        label: 'English',
-      }
+        label: "English",
+      },
     },
   },
   themeConfig: {
     announcementBar: {
-      id: 'frigate_plus',
+      id: "frigate_plus",
       content: `
         <span style="margin-right: 8px; display: inline-block; animation: pulse 2s infinite;">🚀</span>
         Get more relevant and accurate detections with Frigate+ models.
@@ -45,8 +48,8 @@ const config: Config = {
             50%       { transform: scale(1.1); }
           }
         </style>`,
-      backgroundColor: '#005f73',
-      textColor: '#e0fbfc',
+      backgroundColor: "#005f73",
+      textColor: "#e0fbfc",
       isCloseable: false,
     },
     docs: {
@@ -83,15 +86,15 @@ const config: Config = {
       },
     },
     prism: {
-      magicComments:[
+      magicComments: [
         {
-          className: 'theme-code-block-highlighted-line',
-          line: 'highlight-next-line',
-          block: {start: 'highlight-start', end: 'highlight-end'},
+          className: "theme-code-block-highlighted-line",
+          line: "highlight-next-line",
+          block: { start: "highlight-start", end: "highlight-end" },
         },
         {
-          className: 'code-block-error-line',
-          line: 'highlight-error-line',
+          className: "code-block-error-line",
+          line: "highlight-error-line",
         },
       ],
       additionalLanguages: ["bash", "json"],
@@ -131,6 +134,11 @@ const config: Config = {
         srcDark: "img/branding/logo-dark.svg",
       },
       items: [
+        {
+          href: "https://github.com/blakeblackshear/frigate/releases",
+          label: `${STABLE_VERSION}`,
+          position: "left",
+        },
         {
           to: "/",
           activeBasePath: "docs",
@@ -148,19 +156,19 @@ const config: Config = {
           position: "right",
         },
         {
-          type: 'localeDropdown',
-          position: 'right',
+          type: "localeDropdown",
+          position: "right",
           dropdownItemsAfter: [
             {
-              label: '简体中文（社区翻译）',
-              href: 'https://docs.frigate-cn.video',
-            }
-          ]
+              label: "简体中文（社区翻译）",
+              href: "https://docs.frigate-cn.video",
+            },
+          ],
         },
         {
-          href: 'https://github.com/blakeblackshear/frigate',
-          label: 'GitHub',
-          position: 'right',
+          href: "https://github.com/blakeblackshear/frigate",
+          label: "GitHub",
+          position: "right",
         },
       ],
     },

docs/package.json

@@ -18,17 +18,17 @@
     "write-heading-ids": "docusaurus write-heading-ids"
   },
   "dependencies": {
-    "@docusaurus/core": "^3.7.0",
-    "@docusaurus/plugin-content-docs": "^3.7.0",
-    "@docusaurus/preset-classic": "^3.7.0",
-    "@docusaurus/theme-mermaid": "^3.7.0",
+    "@docusaurus/core": "^3.10.2",
+    "@docusaurus/plugin-content-docs": "^3.10.2",
+    "@docusaurus/preset-classic": "^3.10.2",
+    "@docusaurus/theme-mermaid": "^3.10.2",
     "@inkeep/docusaurus": "^2.0.16",
     "@mdx-js/react": "^3.1.0",
     "@types/js-yaml": "^4.0.9",
     "clsx": "^2.1.1",
-    "docusaurus-plugin-openapi-docs": "^4.5.1",
-    "docusaurus-theme-openapi-docs": "^4.5.1",
-    "js-yaml": "^4.1.1",
+    "docusaurus-plugin-openapi-docs": "^5.2.0",
+    "docusaurus-theme-openapi-docs": "^5.2.0",
+    "js-yaml": "^4.3.2",
     "marked": "^16.4.2",
     "prism-react-renderer": "^2.4.1",
     "raw-loader": "^4.0.2",
@@ -48,11 +48,11 @@
     ]
   },
   "devDependencies": {
-    "@docusaurus/module-type-aliases": "^3.7.0",
-    "@docusaurus/types": "^3.7.0",
+    "@docusaurus/module-type-aliases": "^3.10.2",
+    "@docusaurus/types": "^3.10.2",
     "@types/react": "^18.3.27"
   },
   "engines": {
-    "node": ">=18.0"
+    "node": ">=20.19"
   }
 }

docs/static/frigate-api.yaml

@@ -412,6 +412,39 @@ paths:
       security:
         - frigateAdminAuth: []
       x-required-role: admin
+  /go2rtc/streams/{stream_name}/bitrate:
+    get:
+      tags:
+        - Camera
+      summary: Go2Rtc Stream Bitrate
+      description: |-
+        **Access:** Admin role required.
+
+        Measure a go2rtc stream's bitrate over a few seconds.
+      operationId: 
+        go2rtc_stream_bitrate_go2rtc_streams__stream_name__bitrate_get
+      parameters:
+        - name: stream_name
+          in: path
+          required: true
+          schema:
+            type: string
+            title: Stream Name
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema: {}
+        '422':
+          description: Validation Error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPValidationError'
+      security:
+        - frigateAdminAuth: []
+      x-required-role: admin
   /ffprobe:
     get:
       tags:
@@ -4174,19 +4207,20 @@ paths:
         Get notices, most severe first.
 
         Args:
-            include_dismissed: Also return dismissed notices, for the history view
+            include_hidden: Also return acknowledged and muted notices, for the
+                hidden list
 
         Returns:
             The notices
       operationId: get_notices_notices_get
       parameters:
-        - name: include_dismissed
+        - name: include_hidden
           in: query
           required: false
           schema:
             type: boolean
             default: false
-            title: Include Dismissed
+            title: Include Hidden
       responses:
         '200':
           description: Successful Response
@@ -4221,16 +4255,16 @@ paths:
       security:
         - frigateAdminAuth: []
       x-required-role: admin
-  /notices/dismissed_checks:
+  /notices/muted_checks:
     get:
       tags:
         - Notices
-      summary: Get Dismissed Checks
+      summary: Get Muted Checks
       description: |-
         **Access:** Admin role required.
 
-        Get the dismissed config and stream check rows, newest first.
-      operationId: get_dismissed_checks_notices_dismissed_checks_get
+        Get the muted config and stream check rows, newest first.
+      operationId: get_muted_checks_notices_muted_checks_get
       responses:
         '200':
           description: Successful Response
@@ -4240,16 +4274,16 @@ paths:
       security:
         - frigateAdminAuth: []
       x-required-role: admin
-  /notices/dismissed:
+  /notices/hidden:
     delete:
       tags:
         - Notices
-      summary: Purge Dismissed
+      summary: Unhide All Notices
       description: |-
         **Access:** Admin role required.
 
-        Delete every dismissed notice and check row so each can show again.
-      operationId: purge_dismissed_notices_dismissed_delete
+        Show every acknowledged and muted notice and check row again.
+      operationId: unhide_all_notices_notices_hidden_delete
       responses:
         '200':
           description: Successful Response
@@ -4259,18 +4293,83 @@ paths:
       security:
         - frigateAdminAuth: []
       x-required-role: admin
-  /notices/{notice_id}/dismiss:
+  /notices/{notice_id}/acknowledge:
     post:
       tags:
         - Notices
-      summary: Dismiss Notice
+      summary: Acknowledge Notice
       description: |-
         **Access:** Admin role required.
 
-        Hide a notice or a config or stream check row.
+        Hide a notice until it happens again.
 
-        It stays hidden if the same problem happens again.
-      operationId: dismiss_notice_notices__notice_id__dismiss_post
+        Config and stream check rows and the update notice never repeat, so they
+        can only be muted.
+      operationId: acknowledge_notice_notices__notice_id__acknowledge_post
+      parameters:
+        - name: notice_id
+          in: path
+          required: true
+          schema:
+            type: string
+            title: Notice Id
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema: {}
+        '422':
+          description: Validation Error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPValidationError'
+      security:
+        - frigateAdminAuth: []
+      x-required-role: admin
+  /notices/{notice_id}/mute:
+    post:
+      tags:
+        - Notices
+      summary: Mute Notice
+      description: |-
+        **Access:** Admin role required.
+
+        Hide a notice or a config or stream check row for good.
+      operationId: mute_notice_notices__notice_id__mute_post
+      parameters:
+        - name: notice_id
+          in: path
+          required: true
+          schema:
+            type: string
+            title: Notice Id
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema: {}
+        '422':
+          description: Validation Error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPValidationError'
+      security:
+        - frigateAdminAuth: []
+      x-required-role: admin
+  /notices/{notice_id}/hidden:
+    delete:
+      tags:
+        - Notices
+      summary: Unhide Notice
+      description: |-
+        **Access:** Admin role required.
+
+        Show an acknowledged or muted notice or check row again.
+      operationId: unhide_notice_notices__notice_id__hidden_delete
       parameters:
         - name: notice_id
           in: path
@@ -4394,6 +4493,16 @@ paths:
               - type: 'null'
             default: 100
             title: Limit
+        - name: offset
+          in: query
+          required: false
+          schema:
+            anyOf:
+              - type: integer
+                minimum: 0
+              - type: 'null'
+            default: 0
+            title: Offset
         - name: after
           in: query
           required: false
@@ -4699,6 +4808,16 @@ paths:
               - type: 'null'
             default: 50
             title: Limit
+        - name: offset
+          in: query
+          required: false
+          schema:
+            anyOf:
+              - type: integer
+                minimum: 0
+              - type: 'null'
+            default: 0
+            title: Offset
         - name: cameras
           in: query
           required: false
@@ -7665,6 +7784,11 @@ components:
           type: boolean
           title: Skip Save
           default: false
+        replace_paths:
+          items:
+            type: string
+          type: array
+          title: Replace Paths
       type: object
       title: AppConfigSetBody
     AppPostLoginBody:

frigate/__main__.py

@@ -99,6 +99,10 @@ def main() -> None:
         print("***    End Config Validation Errors                       ***")
         print("*************************************************************")
 
+        # force a non-zero exit code for config failures
+        if args.validate_config:
+            sys.exit(1)
+
         # attempt to start Frigate in recovery mode
         try:
             config = FrigateConfig.load(install=True, safe_load=True)

frigate/api/app.py

@@ -2,7 +2,6 @@
 
 import asyncio
 import copy
-import json
 import logging
 import os
 import platform
@@ -11,7 +10,6 @@ import urllib
 from datetime import datetime, timedelta
 from functools import reduce
 from io import StringIO
-from pathlib import Path as FilePath
 from typing import Any
 
 import aiofiles
@@ -56,6 +54,7 @@ from frigate.jobs.media_sync import (
     start_media_sync_job,
 )
 from frigate.models import Event, Timeline
+from frigate.plus import load_plus_model_info
 from frigate.stats.prometheus import get_metrics, update_metrics
 from frigate.types import JobStatusTypesEnum
 from frigate.util.builtin import (
@@ -64,6 +63,7 @@ from frigate.util.builtin import (
     flatten_config_data,
     load_labels,
     process_config_query_string,
+    split_config_key_path,
     update_yaml_file_bulk,
 )
 from frigate.util.config import (
@@ -71,6 +71,10 @@ from frigate.util.config import (
     find_config_file,
     redact_credential,
 )
+from frigate.util.live_streams import (
+    generated_transcode_streams,
+    sync_transcode_streams,
+)
 from frigate.util.object_names import get_categorized_object_names
 from frigate.util.schema import get_config_schema
 from frigate.util.services import (
@@ -394,6 +398,11 @@ def config(request: Request):
         model_dict["non_logo_attributes"] = model.non_logo_attributes
         model_dict["labelmap"] = model.merged_labelmap
 
+        # report the configured reference rather than the resolved cache path,
+        # so saving the config back doesn't lose the Frigate+ model
+        if model.plus_id:
+            model_dict["path"] = f"plus://{model.plus_id}"
+
         if not config["plus"]["enabled"]:
             continue
 
@@ -401,13 +410,7 @@ def config(request: Request):
         model_dict["plus"] = None
 
         if model.path:
-            model_json_path = FilePath(model.path).with_suffix(".json")
-
-            try:
-                with open(model_json_path) as f:
-                    model_dict["plus"] = json.load(f)
-            except (FileNotFoundError, json.JSONDecodeError):
-                pass
+            model_dict["plus"] = load_plus_model_info(os.path.basename(model.path))
 
     return JSONResponse(content=config)
 
@@ -436,6 +439,8 @@ def ffmpeg_presets():
         hwaccel_presets = [
             "preset-rpi-64-h264",
             "preset-rpi-64-h265",
+            "preset-apple-silicon-h264",
+            "preset-apple-silicon-h265",
             "preset-jetson-h264",
             "preset-jetson-h265",
             "preset-rkmpp",
@@ -815,6 +820,17 @@ def _config_set_in_memory(request: Request, body: AppConfigSetBody) -> JSONRespo
         )
 
 
+def _config_path_exists(data: Any, key_path: str) -> bool:
+    """Return whether a dotted config path is present in parsed yaml."""
+    for key in split_config_key_path(key_path):
+        if not isinstance(data, dict) or key not in data:
+            return False
+
+        data = data[key]
+
+    return True
+
+
 @router.put("/config/set", dependencies=[Depends(require_role(["admin"]))])
 def config_set(request: Request, body: AppConfigSetBody):
     config_file = find_config_file()
@@ -869,6 +885,19 @@ def config_set(request: Request, body: AppConfigSetBody):
                         status_code=400,
                     )
 
+                # delete replaced paths first so their maps are rewritten in
+                # the order sent; update_yaml would otherwise keep old order
+                if body.replace_paths:
+                    old_yaml = ruamel.yaml.YAML(typ="safe").load(old_raw_config) or {}
+                    updates = {
+                        **{
+                            path: ""
+                            for path in body.replace_paths
+                            if _config_path_exists(old_yaml, path)
+                        },
+                        **updates,
+                    }
+
                 # apply all updates in a single operation
                 update_yaml_file_bulk(config_file, updates)
 
@@ -932,9 +961,15 @@ def config_set(request: Request, body: AppConfigSetBody):
             if request.app.dispatcher is not None:
                 request.app.dispatcher.clear_runtime_state_for_yaml_keys(updates.keys())
 
+            go2rtc_synced = True
+
             if body.requires_restart == 0 or body.update_topic:
                 old_config: FrigateConfig = request.app.frigate_config
                 swap_runtime_config(request.app, config)
+                go2rtc_synced = sync_transcode_streams(
+                    generated_transcode_streams(old_config),
+                    generated_transcode_streams(config),
+                )
 
                 if body.update_topic:
                     if body.update_topic.startswith("config/cameras/"):
@@ -994,6 +1029,7 @@ def config_set(request: Request, body: AppConfigSetBody):
                             if body.requires_restart == 0
                             else "Config successfully updated, restart to apply"
                         ),
+                        "go2rtc_synced": go2rtc_synced,
                     }
                 ),
                 status_code=200,

frigate/api/auth.py

@@ -130,23 +130,18 @@ def require_admin_by_default():
         if path.startswith(EXEMPT_PREFIXES):
             return
 
-        # Dynamic camera path exemption:
-        # Any path whose first segment matches a configured camera name should
-        # bypass the global admin requirement. These endpoints enforce access
-        # via route-level dependencies (e.g. require_camera_access) to ensure
-        # per-camera authorization. This allows non-admin authenticated users
-        # (e.g. viewer role) to access camera-specific resources without
-        # needing admin privileges.
-        try:
-            if path.startswith("/"):
-                first_segment = path.split("/", 2)[1]
-                if (
-                    first_segment
-                    and first_segment in request.app.frigate_config.cameras
-                ):
-                    return
-        except Exception:
-            pass
+        # Camera routes enforce per-camera access via route-level dependencies
+        # (e.g. require_camera_access). Match on the route template, not the raw
+        # path, so a camera named like another namespace (e.g. "faces") can't
+        # waive the admin check for that namespace's routes.
+        route = request.scope.get("route")
+        if (
+            route is not None
+            and route.path.startswith("/{camera_name}")
+            and request.path_params.get("camera_name")
+            in request.app.frigate_config.cameras
+        ):
+            return
 
         # For all other paths, require admin role
         # Internal port requests have admin role set automatically
@@ -324,11 +319,17 @@ def get_remote_addr(request: Request):
             network = ipaddress.ip_network(proxy)
         except ValueError:
             logger.warning(f"Unable to parse trusted network: {proxy}")
+            continue
         trusted_proxies.append(network)
 
     # return the first remote address that is not trusted
     for addr in route:
-        ip = ipaddress.ip_address(addr.strip())
+        try:
+            ip = ipaddress.ip_address(addr.strip())
+        except ValueError:
+            logger.debug("Invalid address in X-Forwarded-For header")
+            return direct_addr or "127.0.0.1"
+
         logger.debug(f"Checking {ip} (v{ip.version})")
         trusted = False
         for trusted_proxy in trusted_proxies:
@@ -473,12 +474,11 @@ def create_encoded_jwt(user, role, expiration, secret):
 
 def set_jwt_cookie(response: Response, cookie_name, encoded_jwt, max_age, secure):
     # TODO: ideally this would set secure as well, but that requires TLS
-    # SameSite is intentionally left unset (browsers default to Lax). Setting
-    # SameSite=Lax/Strict would stop the cookie from being sent in cross-origin
-    # iframes, breaking embedded views such as the Home Assistant Frigate card.
-    # CSRF is instead mitigated by requiring a custom X-CSRF-TOKEN header, which
-    # cross-origin pages cannot set without a CORS preflight that Frigate never
-    # grants (see check_csrf in api/fastapi_app.py).
+    # Starlette sets SameSite=Lax by default. The cookie is still sent to
+    # same-site iframes (e.g. Home Assistant on the same host or domain), but
+    # not to cross-site ones. CSRF is also mitigated by requiring a custom
+    # X-CSRF-TOKEN header, which cross-origin pages cannot set without a CORS
+    # preflight that Frigate never grants (see check_csrf in api/fastapi_app.py).
     response.set_cookie(
         key=cookie_name,
         value=encoded_jwt,
@@ -786,6 +786,13 @@ def auth(request: Request):
 
         user = token.claims.get("sub")
         role = token.claims.get("role")
+
+        # the token keeps the role it was issued with, so a role removed from
+        # the config since then must send the user back through login
+        if role not in auth_config.roles:
+            logger.debug("jwt role %s is not in the config", role)
+            return fail_response
+
         current_time = int(time.time())
 
         # if the jwt is expired

frigate/api/camera.py

@@ -39,6 +39,11 @@ from frigate.util.builtin import clean_camera_user_pass, get_record_segment_time
 from frigate.util.camera_cleanup import cleanup_camera_db, cleanup_camera_files
 from frigate.util.config import find_config_file
 from frigate.util.image import run_ffmpeg_snapshot
+from frigate.util.live_streams import (
+    generated_transcode_streams,
+    measure_stream_bitrate,
+    sync_transcode_streams,
+)
 from frigate.util.services import (
     analyze_record_keyframes,
     ffprobe_stream,
@@ -248,6 +253,34 @@ def go2rtc_delete_stream(stream_name: str):
         )
 
 
+@router.get(
+    "/go2rtc/streams/{stream_name}/bitrate",
+    dependencies=[Depends(require_role(["admin"]))],
+)
+async def go2rtc_stream_bitrate(request: Request, stream_name: str):
+    """Measure a go2rtc stream's bitrate over a few seconds."""
+    config: FrigateConfig = request.app.frigate_config
+    known = set(config.go2rtc.model_dump().get("streams") or {}) | set(
+        generated_transcode_streams(config)
+    )
+
+    if stream_name not in known:
+        return JSONResponse(
+            content={"success": False, "message": "Unknown stream"},
+            status_code=404,
+        )
+
+    kbps = await asyncio.to_thread(measure_stream_bitrate, stream_name)
+
+    if kbps is None:
+        return JSONResponse(
+            content={"success": False, "message": "Stream sent no data"},
+            status_code=502,
+        )
+
+    return JSONResponse(content={"success": True, "kbps": round(kbps)})
+
+
 @router.get("/ffprobe", dependencies=[Depends(require_role(["admin"]))])
 def ffprobe(request: Request, paths: str = "", detailed: bool = False):
     path_param = paths
@@ -1342,6 +1375,12 @@ async def delete_camera(
     except Exception:
         logger.debug("Failed to remove go2rtc stream for %s", camera_name)
 
+    await asyncio.to_thread(
+        sync_transcode_streams,
+        generated_transcode_streams(frigate_config),
+        generated_transcode_streams(request.app.frigate_config),
+    )
+
     return JSONResponse(
         content={
             "success": True,

frigate/api/config_util.py

@@ -51,6 +51,7 @@ def swap_runtime_config(app: FastAPI, config: FrigateConfig) -> None:
 
     if app.stats_emitter is not None:
         app.stats_emitter.config = config
+        app.stats_emitter.hardware_stats.set_config(config)
 
     if app.dispatcher is not None:
         app.dispatcher.config = config

frigate/api/defs/query/events_query_parameters.py

@@ -14,6 +14,7 @@ class EventsQueryParams(BaseModel):
     zone: str | None = "all"
     zones: str | None = "all"
     limit: int | None = 100
+    offset: int | None = Field(0, ge=0)
     after: float | None = None
     before: float | None = None
     time_range: str | None = DEFAULT_TIME_RANGE
@@ -55,6 +56,7 @@ class EventsSearchQueryParams(BaseModel):
         deprecated=True,
     )
     limit: int | None = 50
+    offset: int | None = Field(0, ge=0)
     cameras: str | None = "all"
     labels: str | None = "all"
     sub_labels: str | None = "all"

frigate/api/defs/request/app_body.py

@@ -10,6 +10,8 @@ class AppConfigSetBody(BaseModel):
     update_topic: str | None = None
     config_data: dict[str, Any] | None = None
     skip_save: bool = False
+    # paths rewritten whole, so a map saves in the order sent
+    replace_paths: list[str] = Field(default_factory=list)
 
 
 class GenAIProbeBody(BaseModel):

frigate/api/event.py

@@ -129,6 +129,7 @@ def events(
         zones = zone
 
     limit = params.limit
+    offset = params.offset
     after = params.after
     before = params.before
     time_range = params.time_range
@@ -361,11 +362,15 @@ def events(
     else:
         order_by = Event.start_time.desc()
 
+    # offset paging needs a stable order when scores or speeds tie
+    tiebreaker = [Event.id] if sort and sort.startswith(("score", "speed")) else []
+
     events = (
         Event.select(*selected_columns)
         .where(reduce(operator.and_, clauses))
-        .order_by(order_by)
+        .order_by(order_by, *tiebreaker)
         .limit(limit)
+        .offset(offset)
         .dicts()
         .iterator()
     )
@@ -534,6 +539,7 @@ def events_search(
     search_type = params.search_type
     include_thumbnails = params.include_thumbnails
     limit = params.limit
+    offset = params.offset
     sort = params.sort
 
     # Filters
@@ -840,6 +846,9 @@ def events_search(
     if search_results:
         events_query = events_query.where(Event.id << list(search_results.keys()))
 
+    # sorts below are stable, so this orders ties for offset paging
+    events_query = events_query.order_by(Event.id)
+
     # Fetch events and process them in a single pass
     processed_events = []
     for event in events_query.dicts():
@@ -897,7 +906,7 @@ def events_search(
         processed_events.sort(key=lambda x: x["start_time"], reverse=True)
 
     # Limit the number of events returned
-    processed_events = processed_events[:limit]
+    processed_events = processed_events[offset:][:limit]
 
     return JSONResponse(content=processed_events)
 

frigate/api/export.py

@@ -71,6 +71,7 @@ from frigate.jobs.export import (
 from frigate.models import Export, ExportCase, Previews, Recordings
 from frigate.record.export import (
     DEFAULT_TIME_LAPSE_FFMPEG_ARGS,
+    DEFAULT_TIME_LAPSE_FFMPEG_INPUT_ARGS,
     ChaptersEnum,
     ExportStreamEnum,
     PlaybackSourceEnum,
@@ -1037,6 +1038,10 @@ def export_recording_custom(
     if camera_validation_error is not None:
         return camera_validation_error
 
+    # Validate user-provided ffmpeg args to prevent injection and add to cases.
+    # Admin users are trusted and skip validation.
+    is_admin = request.headers.get("remote-role", "") == "admin"
+
     playback_source = body.source
     friendly_name = body.name
     existing_image, image_validation_error = _sanitize_existing_image(body.image_path)
@@ -1047,6 +1052,16 @@ def export_recording_custom(
     cpu_fallback = body.cpu_fallback
 
     export_case_id = body.export_case_id
+
+    if export_case_id is not None and not is_admin:
+        return JSONResponse(
+            content={
+                "success": False,
+                "message": "Only admins can attach exports to an existing case.",
+            },
+            status_code=403,
+        )
+
     case_validation_error = _validate_export_case(export_case_id)
     if case_validation_error is not None:
         return case_validation_error
@@ -1063,10 +1078,6 @@ def export_recording_custom(
             status_code=400,
         )
 
-    # Validate user-provided ffmpeg args to prevent injection.
-    # Admin users are trusted and skip validation.
-    is_admin = request.headers.get("remote-role", "") == "admin"
-
     if not is_admin:
         for args_label, args_value in [
             ("input", ffmpeg_input_args),
@@ -1087,7 +1098,7 @@ def export_recording_custom(
 
     # Set default values if not provided (timelapse defaults)
     if ffmpeg_input_args is None:
-        ffmpeg_input_args = ""
+        ffmpeg_input_args = DEFAULT_TIME_LAPSE_FFMPEG_INPUT_ARGS
 
     if ffmpeg_output_args is None:
         ffmpeg_output_args = DEFAULT_TIME_LAPSE_FFMPEG_ARGS

frigate/api/media.py

@@ -63,7 +63,6 @@ from frigate.util.recording_coverage import (
     null_audio_glitches,
     plan_clip,
     resolve_coverage,
-    stream_has_audio,
 )
 
 logger = logging.getLogger(__name__)
@@ -189,7 +188,7 @@ async def camera_ptz_info(request: Request, camera_name: str):
         future = asyncio.run_coroutine_threadsafe(
             request.app.onvif.get_camera_info(camera_name), request.app.onvif.loop
         )
-        result = future.result()
+        result = await asyncio.wrap_future(future)
         return JSONResponse(content=result)
     else:
         return JSONResponse(
@@ -258,15 +257,15 @@ async def latest_frame(
 
                 frame = request.app.camera_error_image
 
-        height = int(params.height or str(frame.shape[0]))
-        width = int(height * frame.shape[1] / frame.shape[0])
-
         if frame is None:
             return JSONResponse(
                 content={"success": False, "message": "Unable to get valid frame"},
                 status_code=500,
             )
 
+        height = int(params.height or str(frame.shape[0]))
+        width = int(height * frame.shape[1] / frame.shape[0])
+
         if height < 1 or width < 1:
             return JSONResponse(
                 content="Invalid height / width requested :: {} / {}".format(
@@ -681,15 +680,10 @@ async def _vod_response(
         end_ts,
         force_discontinuity,
     )
-    intervals = resolve_coverage(camera_name, start_ts, end_ts)
-
     # rows contradicting their stream's audio composition are
     # truncated-shutdown glitches
-    main_audio = stream_has_audio(intervals, main=True)
-    sub_audio = stream_has_audio(intervals, main=False)
-
     spans = build_spans(
-        null_audio_glitches(intervals, main_audio, sub_audio),
+        null_audio_glitches(resolve_coverage(camera_name, start_ts, end_ts)),
         stream_preference,
     )
 
@@ -886,9 +880,8 @@ async def vod_event(
     # If the recordings are not found and the event started more than 5 minutes ago, set has_clip to false
     if (
         event.start_time < datetime.now().timestamp() - 300
-        and type(vod_response) is tuple
-        and len(vod_response) == 2
-        and vod_response[1] == 404
+        and isinstance(vod_response, JSONResponse)
+        and vod_response.status_code == 404
     ):
         Event.update(has_clip=False).where(Event.id == event_id).execute()
 
@@ -956,64 +949,80 @@ async def event_snapshot(
     event_complete = False
     jpg_bytes = None
     frame_time = 0
+
     try:
         event = Event.get(Event.id == event_id, Event.end_time != None)
-        event_complete = True
         await require_camera_access(event.camera, request=request)
+    except DoesNotExist:
+        event = None
+
+    if event is not None:
+        event_complete = True
+
         if not event.has_snapshot:
             return JSONResponse(
                 content={"success": False, "message": "Snapshot not available"},
                 status_code=404,
             )
-        snapshot_settings = _resolve_snapshot_settings(
-            request.app.frigate_config.cameras[event.camera].snapshots, params
-        )
-        jpg_bytes, frame_time = get_event_snapshot_bytes(
-            event,
-            ext="jpg",
-            timestamp=snapshot_settings["timestamp"],
-            bounding_box=snapshot_settings["bounding_box"],
-            crop=snapshot_settings["crop"],
-            height=snapshot_settings["height"],
-            quality=snapshot_settings["quality"],
-            timestamp_style=request.app.frigate_config.cameras[
-                event.camera
-            ].timestamp_style,
-            colormap=request.app.frigate_config.model_for_camera(event.camera).colormap,
-        )
-    except DoesNotExist:
-        # see if the object is currently being tracked
+
         try:
-            camera_states: list[CameraState] = (
-                request.app.detected_frames_processor.get_camera_states()
+            snapshot_settings = _resolve_snapshot_settings(
+                request.app.frigate_config.cameras[event.camera].snapshots, params
+            )
+            jpg_bytes, frame_time = get_event_snapshot_bytes(
+                event,
+                ext="jpg",
+                timestamp=snapshot_settings["timestamp"],
+                bounding_box=snapshot_settings["bounding_box"],
+                crop=snapshot_settings["crop"],
+                height=snapshot_settings["height"],
+                quality=snapshot_settings["quality"],
+                timestamp_style=request.app.frigate_config.cameras[
+                    event.camera
+                ].timestamp_style,
+                colormap=request.app.frigate_config.model_for_camera(
+                    event.camera
+                ).colormap,
             )
-            for camera_state in camera_states:
-                if event_id in camera_state.tracked_objects:
-                    tracked_obj = camera_state.tracked_objects.get(event_id)
-                    if tracked_obj is not None:
-                        snapshot_settings = _resolve_snapshot_settings(
-                            camera_state.camera_config.snapshots, params
-                        )
-                        jpg_bytes, frame_time = tracked_obj.get_img_bytes(
-                            ext="jpg",
-                            timestamp=snapshot_settings["timestamp"],
-                            bounding_box=snapshot_settings["bounding_box"],
-                            crop=snapshot_settings["crop"],
-                            height=snapshot_settings["height"],
-                            quality=snapshot_settings["quality"],
-                        )
-                        await require_camera_access(camera_state.name, request=request)
         except Exception:
             return JSONResponse(
-                content={"success": False, "message": "Ongoing event not found"},
+                content={"success": False, "message": "Unknown error occurred"},
                 status_code=404,
             )
-    except Exception:
-        return JSONResponse(
-            content={"success": False, "message": "Unknown error occurred"},
-            status_code=404,
+    else:
+        # see if the object is currently being tracked
+        camera_states: list[CameraState] = (
+            request.app.detected_frames_processor.get_camera_states()
         )
 
+        for camera_state in camera_states:
+            tracked_obj = camera_state.tracked_objects.get(event_id)
+
+            if tracked_obj is None:
+                continue
+
+            await require_camera_access(camera_state.name, request=request)
+
+            try:
+                snapshot_settings = _resolve_snapshot_settings(
+                    camera_state.camera_config.snapshots, params
+                )
+                jpg_bytes, frame_time = tracked_obj.get_img_bytes(
+                    ext="jpg",
+                    timestamp=snapshot_settings["timestamp"],
+                    bounding_box=snapshot_settings["bounding_box"],
+                    crop=snapshot_settings["crop"],
+                    height=snapshot_settings["height"],
+                    quality=snapshot_settings["quality"],
+                )
+            except Exception:
+                return JSONResponse(
+                    content={"success": False, "message": "Ongoing event not found"},
+                    status_code=404,
+                )
+
+            break
+
     if jpg_bytes is None:
         return JSONResponse(
             content={"success": False, "message": "Live frame not available"},
@@ -1062,19 +1071,25 @@ async def event_thumbnail(
 
     if not thumbnail_bytes:
         # see if the object is currently being tracked
-        try:
-            camera_states = request.app.detected_frames_processor.get_camera_states()
-            for camera_state in camera_states:
-                if event_id in camera_state.tracked_objects:
-                    tracked_obj = camera_state.tracked_objects.get(event_id)
-                    if tracked_obj is not None:
-                        await require_camera_access(camera_state.name, request=request)
-                        thumbnail_bytes = tracked_obj.get_thumbnail(extension.value)
-        except Exception:
-            return JSONResponse(
-                content={"success": False, "message": "Event not found"},
-                status_code=404,
-            )
+        camera_states = request.app.detected_frames_processor.get_camera_states()
+
+        for camera_state in camera_states:
+            tracked_obj = camera_state.tracked_objects.get(event_id)
+
+            if tracked_obj is None:
+                continue
+
+            await require_camera_access(camera_state.name, request=request)
+
+            try:
+                thumbnail_bytes = tracked_obj.get_thumbnail(extension.value)
+            except Exception:
+                return JSONResponse(
+                    content={"success": False, "message": "Event not found"},
+                    status_code=404,
+                )
+
+            break
 
     if not thumbnail_bytes:
         return JSONResponse(

frigate/api/notices.py

@@ -14,17 +14,18 @@ router = APIRouter(tags=[Tags.notices])
 
 
 @router.get("/notices", dependencies=[Depends(require_role(["admin"]))])
-def get_notices(request: Request, include_dismissed: bool = False) -> JSONResponse:
+def get_notices(request: Request, include_hidden: bool = False) -> JSONResponse:
     """Get notices, most severe first.
 
     Args:
-        include_dismissed: Also return dismissed notices, for the history view
+        include_hidden: Also return acknowledged and muted notices, for the
+            hidden list
 
     Returns:
         The notices
     """
     return JSONResponse(
-        content=request.app.notice_registry.active(include_dismissed=include_dismissed)
+        content=request.app.notice_registry.active(include_hidden=include_hidden)
     )
 
 
@@ -34,37 +35,64 @@ def get_notice_stats(request: Request) -> JSONResponse:
     return JSONResponse(content=request.app.notice_registry.stats())
 
 
-@router.get(
-    "/notices/dismissed_checks", dependencies=[Depends(require_role(["admin"]))]
-)
-def get_dismissed_checks(request: Request) -> JSONResponse:
-    """Get the dismissed config and stream check rows, newest first."""
-    return JSONResponse(content=request.app.notice_registry.dismissed_checks())
+@router.get("/notices/muted_checks", dependencies=[Depends(require_role(["admin"]))])
+def get_muted_checks(request: Request) -> JSONResponse:
+    """Get the muted config and stream check rows, newest first."""
+    return JSONResponse(content=request.app.notice_registry.muted_checks())
 
 
-@router.delete("/notices/dismissed", dependencies=[Depends(require_role(["admin"]))])
-def purge_dismissed(request: Request) -> JSONResponse:
-    """Delete every dismissed notice and check row so each can show again."""
-    request.app.notice_registry.purge_dismissed()
-    return JSONResponse(
-        content={"success": True, "message": "Dismissed notices cleared"}
-    )
+@router.delete("/notices/hidden", dependencies=[Depends(require_role(["admin"]))])
+def unhide_all_notices(request: Request) -> JSONResponse:
+    """Show every acknowledged and muted notice and check row again."""
+    request.app.notice_registry.unhide_all()
+    return JSONResponse(content={"success": True, "message": "Notices shown again"})
 
 
 # model notice ids contain a slash, so the id is a path parameter
 @router.post(
-    "/notices/{notice_id:path}/dismiss",
+    "/notices/{notice_id:path}/acknowledge",
     dependencies=[Depends(require_role(["admin"]))],
 )
-def dismiss_notice(request: Request, notice_id: str) -> JSONResponse:
-    """Hide a notice or a config or stream check row.
+def acknowledge_notice(request: Request, notice_id: str) -> JSONResponse:
+    """Hide a notice until it happens again.
 
-    It stays hidden if the same problem happens again.
+    Config and stream check rows and the update notice never repeat, so they
+    can only be muted.
     """
-    if not request.app.notice_registry.dismiss(notice_id):
+    if not request.app.notice_registry.acknowledge(notice_id):
         return JSONResponse(
             content={"success": False, "message": "Notice not found"},
             status_code=404,
         )
 
-    return JSONResponse(content={"success": True, "message": "Notice dismissed"})
+    return JSONResponse(content={"success": True, "message": "Notice acknowledged"})
+
+
+@router.post(
+    "/notices/{notice_id:path}/mute",
+    dependencies=[Depends(require_role(["admin"]))],
+)
+def mute_notice(request: Request, notice_id: str) -> JSONResponse:
+    """Hide a notice or a config or stream check row for good."""
+    if not request.app.notice_registry.mute(notice_id):
+        return JSONResponse(
+            content={"success": False, "message": "Notice not found"},
+            status_code=404,
+        )
+
+    return JSONResponse(content={"success": True, "message": "Notice muted"})
+
+
+@router.delete(
+    "/notices/{notice_id:path}/hidden",
+    dependencies=[Depends(require_role(["admin"]))],
+)
+def unhide_notice(request: Request, notice_id: str) -> JSONResponse:
+    """Show an acknowledged or muted notice or check row again."""
+    if not request.app.notice_registry.unhide(notice_id):
+        return JSONResponse(
+            content={"success": False, "message": "Notice not found"},
+            status_code=404,
+        )
+
+    return JSONResponse(content={"success": True, "message": "Notice shown again"})

frigate/api/record.py

@@ -412,11 +412,8 @@ async def no_recordings(
     if not camera_list:
         return JSONResponse(content=[])
 
-    before = params.before or datetime.datetime.now().timestamp()
-    after = (
-        params.after
-        or (datetime.datetime.now() - datetime.timedelta(hours=1)).timestamp()
-    )
+    before = params.before or datetime.now().timestamp()
+    after = params.after or (datetime.now() - timedelta(hours=1)).timestamp()
     scale = params.scale
 
     recordings: list[tuple[float, float]] = []

frigate/api/review.py

@@ -44,6 +44,22 @@ logger = logging.getLogger(__name__)
 router = APIRouter(tags=[Tags.review])
 
 
+def get_label_clause(label: str, include_audio: bool = True):
+    """Build a clause matching a label within a review segment's data.
+
+    Verified objects are stored with a `-verified` suffix (eg. `person-verified`)
+    so that variant is matched as well.
+    """
+    clause = (ReviewSegment.data["objects"].cast("text") % f'*"{label}"*') | (
+        ReviewSegment.data["objects"].cast("text") % f'*"{label}-verified"*'
+    )
+
+    if include_audio:
+        clause |= ReviewSegment.data["audio"].cast("text") % f'*"{label}"*'
+
+    return clause
+
+
 @router.get(
     "/review",
     response_model=list[ReviewSegmentResponse],
@@ -93,10 +109,7 @@ async def review(
         filtered_labels = labels.split(",")
 
         for label in filtered_labels:
-            label_clauses.append(
-                (ReviewSegment.data["objects"].cast("text") % f'*"{label}"*')
-                | (ReviewSegment.data["audio"].cast("text") % f'*"{label}"*')
-            )
+            label_clauses.append(get_label_clause(label))
         clauses.append(reduce(operator.or_, label_clauses))
 
     if zones != "all":
@@ -239,10 +252,7 @@ async def review_summary(
         filtered_labels = labels.split(",")
 
         for label in filtered_labels:
-            label_clauses.append(
-                (ReviewSegment.data["objects"].cast("text") % f'*"{label}"*')
-                | (ReviewSegment.data["audio"].cast("text") % f'*"{label}"*')
-            )
+            label_clauses.append(get_label_clause(label))
         clauses.append(reduce(operator.or_, label_clauses))
     if zones != "all":
         # use matching so segments with multiple zones
@@ -340,9 +350,8 @@ async def review_summary(
         filtered_labels = labels.split(",")
 
         for label in filtered_labels:
-            label_clauses.append(
-                ReviewSegment.data["objects"].cast("text") % f'*"{label}"*'
-            )
+            label_clauses.append(get_label_clause(label, include_audio=False))
+
         clauses.append(reduce(operator.or_, label_clauses))
 
     # Find the time range of available data

frigate/app.py

@@ -49,7 +49,6 @@ from frigate.debug_replay import (
     DebugReplayManager,
     cleanup_replay_cameras,
 )
-from frigate.detectors.detector_config import SceneEnum
 from frigate.detectors.detector_types import api_types
 from frigate.detectors.device import build_detector_config, runner_names
 from frigate.embeddings import EmbeddingProcess, EmbeddingsContext
@@ -108,7 +107,7 @@ class FrigateApp:
         self.metrics_manager = manager
         self.audio_process: mp.Process | None = None
         self.stop_event = stop_event
-        self.detection_queues: dict[SceneEnum, Queue] = {
+        self.detection_queues: dict[str, Queue] = {
             model.scene: mp.Queue() for model in config.models
         }
         self.detectors: dict[str, ObjectDetectProcess] = {}
@@ -395,7 +394,7 @@ class FrigateApp:
                 logger.error("Unable to prepare the %s runtime: %s", detector_type, err)
 
     def start_detectors(self) -> None:
-        model_cameras: dict[SceneEnum, list[str]] = {
+        model_cameras: dict[str, list[str]] = {
             model.scene: [] for model in self.config.models
         }
 
@@ -483,7 +482,7 @@ class FrigateApp:
 
     def start_audio_processor(self) -> None:
         self.audio_process = AudioProcessor(
-            self.config, self.camera_metrics, self.stop_event
+            self.config, self.camera_metrics, self.embeddings_metrics, self.stop_event
         )
         self.audio_process.start()
         self.processes["audio_detector"] = self.audio_process.pid or 0
@@ -556,10 +555,20 @@ class FrigateApp:
                 "output",
                 lambda: OutputProcess(self.config, self.stop_event),
             ),
+            (
+                "audio_process",
+                "audio_detector",
+                lambda: AudioProcessor(
+                    self.config,
+                    self.camera_metrics,
+                    self.embeddings_metrics,
+                    self.stop_event,
+                ),
+            ),
         ]
 
         for attr, key, factory in specs:
-            if not hasattr(self, attr):
+            if getattr(self, attr, None) is None:
                 continue
 
             def on_restart(

frigate/camera/activity_manager.py

@@ -104,12 +104,13 @@ class CameraActivityManager:
         all_objects: list[dict[str, Any]] = []
 
         for camera in new_activity.keys():
-            if camera not in self.config.cameras:
+            camera_config = self.config.cameras.get(camera)
+            if camera_config is None:
                 continue
 
             # handle cameras that were added dynamically
             if camera not in self.camera_all_object_counts:
-                self.__init_camera(self.config.cameras[camera])
+                self.__init_camera(camera_config)
 
             new_objects = new_activity[camera].get("objects", [])
             all_objects.extend(new_objects)
@@ -234,12 +235,13 @@ class AudioActivityManager:
         now = datetime.datetime.now().timestamp()
 
         for camera in new_activity.keys():
-            if camera not in self.config.cameras:
+            camera_config = self.config.cameras.get(camera)
+            if camera_config is None:
                 continue
 
             # handle cameras that were added dynamically
             if camera not in self.current_audio_detections:
-                self.__init_camera(self.config.cameras[camera])
+                self.__init_camera(camera_config)
 
             new_detections = new_activity[camera].get("detections", [])
             if self.compare_audio_activity(camera, new_detections, now):

frigate/camera/maintainer.py

@@ -15,7 +15,6 @@ from frigate.config.camera.updater import (
     CameraConfigUpdateSubscriber,
 )
 from frigate.const import REPLAY_CAMERA_PREFIX
-from frigate.detectors.detector_config import SceneEnum
 from frigate.models import Regions
 from frigate.object_detection.util import detection_frame_size
 from frigate.util.builtin import empty_and_close_queue
@@ -31,7 +30,7 @@ class CameraMaintainer(threading.Thread):
     def __init__(
         self,
         config: FrigateConfig,
-        detection_queues: dict[SceneEnum, Queue],
+        detection_queues: dict[str, Queue],
         detected_frames_queue: Queue,
         camera_metrics: DictProxy,
         ptz_metrics: dict[str, PTZMetrics],

frigate/comms/detections_updater.py

@@ -12,6 +12,7 @@ class DetectionTypeEnum(str, Enum):
     video = "video"
     audio = "audio"
     lpr = "lpr"
+    classification_state = "classification_state"
 
 
 class DetectionPublisher(Publisher):

frigate/comms/dispatcher.py

@@ -782,7 +782,9 @@ class Dispatcher:
         try:
             payload = int(payload)
         except ValueError:
-            f"Received unsupported value for motion contour area: {payload}"
+            logger.warning(
+                f"Received unsupported value for motion contour area: {payload}"
+            )
             return
 
         motion_settings = self.config.cameras[camera_name].motion
@@ -799,7 +801,9 @@ class Dispatcher:
         try:
             payload = int(payload)
         except ValueError:
-            f"Received unsupported value for motion threshold: {payload}"
+            logger.warning(
+                f"Received unsupported value for motion threshold: {payload}"
+            )
             return
 
         motion_settings = self.config.cameras[camera_name].motion
@@ -814,7 +818,9 @@ class Dispatcher:
     def _on_global_notification_command(self, payload: str) -> None:
         """Callback for global notification topic."""
         if payload != "ON" and payload != "OFF":
-            f"Received unsupported value for all notification: {payload}"
+            logger.warning(
+                f"Received unsupported value for all notification: {payload}"
+            )
             return
 
         notification_settings = self.config.notifications

frigate/config/camera/detect.py

@@ -1,6 +1,6 @@
 from pydantic import Field, model_validator
 
-from frigate.detectors.detector_config import SceneEnum
+from frigate.detectors.detector_config import DEFAULT_SCENE, SCENE_PATTERN
 
 from ..base import FrigateBaseModel
 
@@ -62,10 +62,11 @@ class DetectConfig(FrigateBaseModel):
         title="Detect width",
         description="Width (pixels) of frames used for the detect stream; leave empty to use the native stream resolution.",
     )
-    scene: SceneEnum = Field(
-        default=SceneEnum.all,
+    scene: str = Field(
+        default=DEFAULT_SCENE,
+        pattern=SCENE_PATTERN,
         title="Detect scene",
-        description="The environment this camera looks at, used to pick which of the configured models runs on it. Cameras left on 'all' run the model configured with a scene of 'all'.",
+        description="The environment this camera looks at, used to pick which of the configured models runs on it. Cameras left on 'default' run the model configured with a scene of 'default'.",
     )
     fps: int = Field(
         default=5,

frigate/config/camera/genai.py

@@ -1,7 +1,7 @@
 from enum import Enum
-from typing import Any
+from typing import Any, Self
 
-from pydantic import Field
+from pydantic import Field, model_validator
 
 from ..base import FrigateBaseModel
 from ..env import EnvString
@@ -21,6 +21,17 @@ class GenAIRoleEnum(str, Enum):
     chat = "chat"
     descriptions = "descriptions"
     embeddings = "embeddings"
+    transcribe = "transcribe"
+
+
+# Providers that can accept audio input for the transcribe role. Ollama has no
+# audio input support, so claiming the role there would fail at request time.
+TRANSCRIBE_CAPABLE_PROVIDERS = {
+    GenAIProviderEnum.openai,
+    GenAIProviderEnum.azure_openai,
+    GenAIProviderEnum.gemini,
+    GenAIProviderEnum.llamacpp,
+}
 
 
 class GenAIConfig(FrigateBaseModel):
@@ -52,7 +63,7 @@ class GenAIConfig(FrigateBaseModel):
             GenAIRoleEnum.chat,
         ],
         title="Roles",
-        description="GenAI roles (chat, descriptions, embeddings); one provider per role.",
+        description="GenAI roles (chat, descriptions, embeddings, transcribe); one provider per role. Only chat, descriptions, and embeddings are granted by default; transcribe must be listed explicitly.",
     )
     provider_options: dict[str, Any] = Field(
         default={},
@@ -66,3 +77,17 @@ class GenAIConfig(FrigateBaseModel):
         description="Runtime options passed to the provider for each inference call.",
         json_schema_extra={"additionalProperties": {}},
     )
+
+    @model_validator(mode="after")
+    def validate_transcribe_provider(self) -> Self:
+        """Reject the transcribe role on providers that cannot accept audio input."""
+        if (
+            GenAIRoleEnum.transcribe in self.roles
+            and self.provider not in TRANSCRIBE_CAPABLE_PROVIDERS
+        ):
+            raise ValueError(
+                f"GenAI provider '{self.provider.value}' does not support audio input "
+                "and cannot be given the 'transcribe' role."
+            )
+
+        return self

frigate/config/camera/live.py

@@ -1,8 +1,57 @@
-from pydantic import Field
+from pydantic import Field, field_validator
+
+from frigate.util.live_streams import DEFAULT_TRANSCODE_QUALITIES
 
 from ..base import FrigateBaseModel
 
-__all__ = ["CameraLiveConfig"]
+__all__ = ["CameraLiveConfig", "LiveTranscodeConfig", "LiveTranscodeQualityConfig"]
+
+
+class LiveTranscodeQualityConfig(FrigateBaseModel):
+    height: int = Field(
+        ge=144,
+        le=2160,
+        title="Height",
+        description="Output height in pixels; width follows the source aspect ratio.",
+    )
+    bitrate: int = Field(
+        ge=64,
+        title="Bitrate",
+        description="Target and maximum video bitrate in kbps.",
+    )
+
+
+class LiveTranscodeConfig(FrigateBaseModel):
+    enabled: bool = Field(
+        default=False,
+        title="Enable transcoded streams",
+        description="Add lower-quality live streams that go2rtc transcodes in real time while someone is watching.",
+    )
+    source: str | None = Field(
+        default=None,
+        title="Source stream",
+        description="go2rtc stream to transcode. Defaults to the first live stream.",
+    )
+    qualities: list[LiveTranscodeQualityConfig] = Field(
+        default_factory=lambda: [
+            LiveTranscodeQualityConfig(**quality)
+            for quality in DEFAULT_TRANSCODE_QUALITIES
+        ],
+        title="Qualities",
+        description="One transcoded stream is added per quality.",
+    )
+
+    @field_validator("qualities")
+    @classmethod
+    def validate_unique_heights(
+        cls, qualities: list[LiveTranscodeQualityConfig]
+    ) -> list[LiveTranscodeQualityConfig]:
+        heights = [quality.height for quality in qualities]
+
+        if len(heights) != len(set(heights)):
+            raise ValueError("Transcoded stream heights must be unique.")
+
+        return qualities
 
 
 class CameraLiveConfig(FrigateBaseModel):
@@ -11,6 +60,11 @@ class CameraLiveConfig(FrigateBaseModel):
         title="Live stream names",
         description="Mapping of configured stream names to restream/go2rtc names used for live playback.",
     )
+    transcode: LiveTranscodeConfig = Field(
+        default_factory=LiveTranscodeConfig,
+        title="Transcoded streams",
+        description="Lower-quality live streams transcoded on demand by go2rtc.",
+    )
     height: int = Field(
         default=720,
         title="Live height",

frigate/config/camera/review.py

@@ -9,6 +9,7 @@ __all__ = [
     "DetectionsConfig",
     "AlertsConfig",
     "ImageSourceEnum",
+    "ReviewFrameModeEnum",
     "ReviewResponseStyleEnum",
 ]
 
@@ -20,6 +21,13 @@ class ImageSourceEnum(str, Enum):
     recordings = "recordings"
 
 
+class ReviewFrameModeEnum(str, Enum):
+    """How review frames are presented to the GenAI provider."""
+
+    frames = "frames"
+    annotated_frames = "annotated_frames"
+
+
 class ReviewResponseStyleEnum(str, Enum):
     """Writing style presets for GenAI review descriptions."""
 
@@ -153,6 +161,11 @@ class GenAIReviewConfig(FrigateBaseModel):
         description="Preferred language to request from the GenAI provider for generated responses.",
         default=None,
     )
+    frame_mode: ReviewFrameModeEnum = Field(
+        default=ReviewFrameModeEnum.frames,
+        title="Frame mode",
+        description="How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses.",
+    )
     response_style: ReviewResponseStyleEnum = Field(
         default=ReviewResponseStyleEnum.default,
         title="Response style",

frigate/config/classification.py

@@ -5,6 +5,7 @@ from pydantic import ConfigDict, Field, field_validator
 from .base import FrigateBaseModel
 
 __all__ = [
+    "AudioTranscriptionModelEnum",
     "CameraFaceRecognitionConfig",
     "CameraLicensePlateRecognitionConfig",
     "CameraAudioTranscriptionConfig",
@@ -20,6 +21,10 @@ class SemanticSearchModelEnum(str, Enum):
     jinav2 = "jinav2"
 
 
+class AudioTranscriptionModelEnum(str, Enum):
+    whisper = "whisper"
+
+
 class EnrichmentsDeviceEnum(str, Enum):
     GPU = "GPU"
     CPU = "CPU"
@@ -53,10 +58,35 @@ class AudioTranscriptionConfig(FrigateBaseModel):
         description="Enable or disable automatic audio transcription for all cameras; can be overridden per-camera.",
     )
     language: str = Field(
-        default="en",
+        default="auto",
         title="Transcription language",
-        description="Language code used for transcription/translation (for example 'en' for English). See https://whisper-api.com/docs/languages/ for supported language codes.",
+        description="Language code used for transcription/translation (for example 'en' for English), or 'auto' to let the model detect it. See https://whisper-api.com/docs/languages/ for supported language codes.",
     )
+    model: AudioTranscriptionModelEnum | str | None = Field(
+        default=AudioTranscriptionModelEnum.whisper,
+        title="Audio transcription model or GenAI provider name",
+        description="The transcription backend: 'whisper' for Frigate's built-in local models, or the name of a GenAI provider with the transcribe role.",
+    )
+
+    @field_validator("model", mode="before")
+    @classmethod
+    def coerce_model_enum(cls, v):
+        # An absent value ("model:" with nothing after it, or an explicit null)
+        # means unspecified, so fall back to the built-in backend. Left as None
+        # it would pass the GenAI-provider validation, which only inspects
+        # strings, and then be treated as a provider name that resolves to no
+        # client, turning transcription into a silent no-op.
+        if v is None or (isinstance(v, str) and not v.strip()):
+            return AudioTranscriptionModelEnum.whisper
+
+        if isinstance(v, str):
+            try:
+                return AudioTranscriptionModelEnum(v)
+            except ValueError:
+                return v
+
+        return v
+
     device: EnrichmentsDeviceEnum = Field(
         default=EnrichmentsDeviceEnum.CPU,
         title="Transcription device",

frigate/config/config.py

@@ -19,7 +19,7 @@ from ruamel.yaml import YAML
 
 from frigate.const import REGEX_JSON
 from frigate.detectors import ModelConfig
-from frigate.detectors.detector_config import SceneEnum
+from frigate.detectors.detector_config import DEFAULT_SCENE
 from frigate.detectors.device import DeviceParseError, DeviceSpec, parse_device
 from frigate.plus import PlusApi
 from frigate.util.builtin import (
@@ -35,6 +35,13 @@ from frigate.util.config import (
     migrate_frigate_config,
 )
 from frigate.util.image import create_mask
+from frigate.util.live_streams import (
+    default_transcode_source,
+    is_transcode_stream_name,
+    transcode_stream_name,
+    transcode_streams,
+)
+from frigate.util.runtime_deps import sha256_of
 from frigate.util.services import auto_detect_hwaccel
 
 from .auth import AuthConfig
@@ -56,6 +63,7 @@ from .camera.timestamp import TimestampStyleConfig
 from .camera_group import CameraGroupConfig
 from .classification import (
     AudioTranscriptionConfig,
+    AudioTranscriptionModelEnum,
     ClassificationConfig,
     FaceRecognitionConfig,
     LicensePlateRecognitionConfig,
@@ -281,11 +289,78 @@ def verify_config_roles(camera_config: CameraConfig) -> None:
         )
 
 
+def apply_live_transcode_streams(
+    frigate_config: FrigateConfig, camera_config: CameraConfig
+) -> None:
+    """Fold a camera's transcoded streams into its live stream list.
+
+    Enabled qualities missing from live.streams are appended, and entries the
+    user placed keep their position. Transcoded names that are no longer
+    generated are dropped unless they name a real go2rtc stream.
+    """
+    live = camera_config.live
+    transcode = live.transcode
+    go2rtc_streams = frigate_config.go2rtc.model_dump().get("streams") or {}
+    generated: dict[str, str] = {}
+
+    if transcode.enabled:
+        if transcode.source is None:
+            transcode.source = default_transcode_source(
+                camera_config.name, live.streams
+            )
+
+        if transcode.source not in go2rtc_streams:
+            raise ValueError(
+                f"Camera {camera_config.name} has transcoded streams enabled, but its source {transcode.source} is not a go2rtc stream."
+            )
+
+        generated = transcode_streams(
+            camera_config.name,
+            transcode.source,
+            [quality.model_dump() for quality in transcode.qualities],
+        )
+
+        for name in generated:
+            if name in go2rtc_streams:
+                raise ValueError(
+                    f"Camera {camera_config.name} generates transcoded stream {name}, which collides with a go2rtc stream of the same name."
+                )
+
+    streams = {
+        label: name
+        for label, name in live.streams.items()
+        if name in generated
+        or name in go2rtc_streams
+        or not is_transcode_stream_name(camera_config.name, name)
+    }
+    placed = set(streams.values())
+
+    for quality in transcode.qualities if transcode.enabled else []:
+        name = transcode_stream_name(camera_config.name, quality.height)
+
+        if name in placed:
+            continue
+
+        label = f"{quality.height}p"
+
+        if label in streams:
+            raise ValueError(
+                f"Camera {camera_config.name} already has a live stream named {label}; rename it or place the transcoded stream under another name."
+            )
+
+        streams[label] = name
+
+    live.streams = streams
+
+
 def verify_valid_live_stream_names(
     frigate_config: FrigateConfig, camera_config: CameraConfig
 ) -> ValueError | None:
     """Verify that a restream exists to use for live view."""
     for _, stream_name in camera_config.live.streams.items():
+        if is_transcode_stream_name(camera_config.name, stream_name):
+            continue
+
         if (
             stream_name
             not in frigate_config.go2rtc.model_dump().get("streams", {}).keys()
@@ -535,7 +610,7 @@ class FrigateConfig(FrigateBaseModel):
     models: list[ModelConfig] = Field(
         default_factory=_default_models,
         title="Detection models",
-        description="Object detection models and the hardware each one runs on. Cameras pick a model by matching their detect.scene against a model's scene.",
+        description="Object detection models and the hardware each one runs on. Cameras pick a model by matching their detect.scene against a model's scene, falling back to the 'default' model.",
     )
 
     # GenAI config (named provider configs: name -> GenAIConfig)
@@ -650,7 +725,9 @@ class FrigateConfig(FrigateBaseModel):
     )
 
     _plus_api: PlusApi
-    _model_devices: dict[SceneEnum, list[DeviceSpec]]
+    _model_devices: dict[str, list[DeviceSpec]]
+    # scene -> model, including the scenes of duplicate models folded into another
+    _scene_models: dict[str, ModelConfig]
     _camera_models: dict[str, ModelConfig]
     _all_attributes: list[str]
     _all_attribute_logos: list[str]
@@ -685,7 +762,7 @@ class FrigateConfig(FrigateBaseModel):
     def primary_model(self) -> ModelConfig:
         """The model used when no specific camera is in play."""
         for model in self.models:
-            if model.scene == SceneEnum.all:
+            if model.scene == DEFAULT_SCENE:
                 return model
 
         return self.models[0]
@@ -707,7 +784,7 @@ class FrigateConfig(FrigateBaseModel):
 
         if model is None:
             camera = self.cameras.get(camera_name)
-            scene = camera.detect.scene if camera is not None else SceneEnum.all
+            scene = camera.detect.scene if camera is not None else DEFAULT_SCENE
             model = self._resolve_camera_model(camera_name, scene)
             self._camera_models[camera_name] = model
 
@@ -767,14 +844,14 @@ class FrigateConfig(FrigateBaseModel):
         if not self.models:
             raise ValueError("At least one model must be configured under models")
 
-        model_devices: dict[SceneEnum, list[DeviceSpec]] = {}
+        model_devices: dict[str, list[DeviceSpec]] = {}
         # device string -> the scene of the model that already claimed it
-        claimed_devices: dict[str, SceneEnum] = {}
+        claimed_devices: dict[str, str] = {}
 
         for index, model in enumerate(self.models):
-            scene = model.scene.value
+            scene = model.scene
 
-            if model.scene in model_devices:
+            if scene in model_devices:
                 raise ValueError(
                     f"Multiple models are configured with a scene of '{scene}'. Each model must use a different scene."
                 )
@@ -803,17 +880,20 @@ class FrigateConfig(FrigateBaseModel):
                     other = claimed_devices[device.raw]
                     where = (
                         f"twice by model '{scene}'"
-                        if other == model.scene
-                        else f"by both the '{other.value}' and '{scene}' models"
+                        if other == scene
+                        else f"by both the '{other}' and '{scene}' models"
                     )
                     raise ValueError(
                         f"Device '{device.raw}' is used {where}, but it can only run one detection process."
                     )
 
-                claimed_devices[device.raw] = model.scene
+                claimed_devices[device.raw] = scene
 
             self.models[index] = self._load_model(model, devices[0].detector)
-            model_devices[model.scene] = devices
+            model_devices[scene] = devices
+
+        self._scene_models = {model.scene: model for model in self.models}
+        self._consolidate_duplicate_models(model_devices)
 
         attributes: set[str] = set()
         attribute_logos: set[str] = set()
@@ -837,36 +917,107 @@ class FrigateConfig(FrigateBaseModel):
         }
         self._all_labels = labels
 
-    def _resolve_camera_model(self, name: str, scene: SceneEnum) -> ModelConfig:
+    def _consolidate_duplicate_models(
+        self, model_devices: dict[str, list[DeviceSpec]]
+    ) -> None:
+        """Fold models that load the same model file into a single model.
+
+        Separate scenes for one model only split the same work across separate
+        detection queues, so each device serves fewer cameras and is slower
+        overall than one shared model. The duplicate's devices are moved to the
+        model it duplicates and its scene resolves to that model.
+
+        Args:
+            model_devices: Scene to parsed devices, updated in place
+        """
+        kept: list[ModelConfig] = []
+        hashes: dict[str, str | None] = {}
+
+        def file_hash(path: str) -> str | None:
+            if path not in hashes:
+                hashes[path] = sha256_of(path) if os.path.isfile(path) else None
+
+            return hashes[path]
+
+        def same_model(a: ModelConfig, b: ModelConfig) -> bool:
+            # a model's devices all share a detector, so folding across
+            # detectors would produce an invalid model
+            if model_devices[a.scene][0].detector != model_devices[b.scene][0].detector:
+                return False
+
+            if not a.path or not b.path:
+                return False
+
+            if os.path.realpath(a.path) == os.path.realpath(b.path):
+                return True
+
+            a_hash = file_hash(a.path)
+            return a_hash is not None and a_hash == file_hash(b.path)
+
+        for model in self.models:
+            original = next((other for other in kept if same_model(other, model)), None)
+
+            if original is None:
+                kept.append(model)
+                continue
+
+            # keep the default model so cameras without a scene still find it
+            if model.scene == DEFAULT_SCENE:
+                kept[kept.index(original)] = model
+                original, model = model, original
+
+            logger.warning(
+                "Models '%s' and '%s' use the same model file, so they have been combined into the '%s' model. Defining one model under several scenes to assign detectors to specific cameras is slower and less efficient than letting every detector serve every camera. Remove the '%s' model and list its devices under the '%s' model instead",
+                original.scene,
+                model.scene,
+                original.scene,
+                model.scene,
+                original.scene,
+            )
+            original.devices = [*original.devices, *model.devices]
+            model_devices[original.scene] = [
+                *model_devices[original.scene],
+                *model_devices.pop(model.scene),
+            ]
+            self._scene_models[model.scene] = original
+
+            # anything already folded into the duplicate follows it
+            for scene, target in self._scene_models.items():
+                if target is model:
+                    self._scene_models[scene] = original
+
+        self.models = kept
+
+    def _resolve_camera_model(self, name: str, scene: str) -> ModelConfig:
         """Resolve which model a camera runs on.
 
         A camera may name a scene no model is configured for, which is valid as
-        long as an 'all' model is there to fall back to.
+        long as a 'default' model is there to fall back to.
 
         Args:
             name: Name of the camera
-            scene: The camera's detect scene, which defaults to 'all'
+            scene: The camera's detect scene, which defaults to 'default'
 
         Returns:
             The model the camera runs on
         """
-        by_scene = {model.scene: model for model in self.models}
-        model = by_scene.get(scene)
+        model = self._scene_models.get(scene)
 
         if model is not None:
             return model
 
-        default = by_scene.get(SceneEnum.all)
+        default = self._scene_models.get(DEFAULT_SCENE)
 
         if default is None:
             raise ValueError(
-                f"Camera '{name}' has a detect scene of '{scene.value}', but no model is configured for that scene or for 'all'."
+                f"Camera '{name}' has a detect scene of '{scene}', but no model is configured for that scene or for '{DEFAULT_SCENE}'."
             )
 
         logger.warning(
-            "Camera '%s' has a detect scene of '%s', but no model is configured for that scene, so the 'all' model is used",
+            "Camera '%s' has a detect scene of '%s', but no model is configured for that scene, so the '%s' model is used",
             name,
-            scene.value,
+            scene,
+            DEFAULT_SCENE,
         )
         return default
 
@@ -884,7 +1035,7 @@ class FrigateConfig(FrigateBaseModel):
         # set notifications state
         self.notifications.enabled_in_config = self.notifications.enabled
 
-        # validate genai: each role (chat, descriptions, embeddings) at most once
+        # validate genai: each role (chat, descriptions, embeddings, transcribe) at most once
         role_to_name: dict[GenAIRoleEnum, str] = {}
         for name, genai_cfg in self.genai.items():
             for role in genai_cfg.roles:
@@ -977,6 +1128,8 @@ class FrigateConfig(FrigateBaseModel):
                 "face_recognition": ["enabled", "min_area"],
                 "lpr": ["enabled", "expire_time", "min_area", "enhancement"],
                 "audio_transcription": ["enabled", "live_enabled"],
+                # transcode is camera-level only
+                "live": ["streams", "height", "quality"],
             }
 
             for section in allowed_fields_map:
@@ -997,6 +1150,10 @@ class FrigateConfig(FrigateBaseModel):
             camera_model = self._resolve_camera_model(name, camera_config.detect.scene)
             self._camera_models[name] = camera_model
 
+            # point cameras at the model their duplicate scene was folded into
+            if camera_config.detect.scene in self._scene_models:
+                camera_config.detect.scene = camera_model.scene
+
             if camera_config.ffmpeg.hwaccel_args == "auto":
                 camera_config.ffmpeg.hwaccel_args = self.ffmpeg.hwaccel_args
 
@@ -1205,6 +1362,8 @@ class FrigateConfig(FrigateBaseModel):
             if not camera_config.live.streams:
                 camera_config.live.streams = {name: name}
 
+            apply_live_transcode_streams(self, camera_config)
+
             # generate the ffmpeg commands
             camera_config.create_ffmpeg_cmds()
             self.cameras[name] = camera_config
@@ -1245,6 +1404,35 @@ class FrigateConfig(FrigateBaseModel):
         for model in self.models:
             model.create_colormap(colored_labels)
 
+        # validate audio_transcription.model when it is a GenAI provider name.
+        # this runs here rather than beside the semantic_search check because the
+        # global->camera merge above is what resolves camera-level enablement.
+        transcription_active = self.audio_transcription.enabled or any(
+            camera.audio_transcription.enabled for camera in self.cameras.values()
+        )
+
+        if (
+            transcription_active
+            and isinstance(self.audio_transcription.model, str)
+            and not isinstance(
+                self.audio_transcription.model, AudioTranscriptionModelEnum
+            )
+        ):
+            if self.audio_transcription.model not in self.genai:
+                raise ValueError(
+                    f"audio_transcription.model '{self.audio_transcription.model}' is not a "
+                    "valid GenAI config key. Must match a key in genai config."
+                )
+
+            if (
+                GenAIRoleEnum.transcribe
+                not in self.genai[self.audio_transcription.model].roles
+            ):
+                raise ValueError(
+                    f"GenAI provider '{self.audio_transcription.model}' must have "
+                    "'transcribe' in its roles for audio transcription."
+                )
+
         # Check audio transcription and audio detection requirements
         if self.audio_transcription.enabled:
             # If audio transcription is enabled globally, at least one camera must have audio detection enabled

frigate/data_processing/common/license_plate/mixin.py

@@ -176,6 +176,7 @@ class LicensePlateProcessingMixin:
         """
         input_shape = [3, 48, 320]
         num_images = len(images)
+        outputs: list[np.ndarray] = []
 
         for index in range(0, num_images, self.batch_size):
             input_h, input_w = input_shape[1], input_shape[2]
@@ -195,11 +196,11 @@ class LicensePlateProcessingMixin:
                 norm_image = norm_image[np.newaxis, :]
                 norm_images.append(norm_image)
 
-        try:
-            outputs = self.model_runner.recognition_model(norm_images)  # type: ignore[arg-type]
-        except Exception as e:
-            logger.warning(f"Error running LPR recognition model: {e}")
-            return [], []
+            try:
+                outputs.extend(self.model_runner.recognition_model(norm_images))  # type: ignore[arg-type]
+            except Exception as e:
+                logger.warning(f"Error running LPR recognition model: {e}")
+                return [], []
 
         return self.ctc_decoder(outputs)
 

frigate/data_processing/post/audio_transcription.py

@@ -10,6 +10,7 @@ from peewee import DoesNotExist
 
 from frigate.comms.inter_process import InterProcessRequestor
 from frigate.config import FrigateConfig
+from frigate.config.classification import AudioTranscriptionModelEnum
 from frigate.const import (
     CACHE_DIR,
     MODEL_CACHE_DIR,
@@ -18,8 +19,13 @@ from frigate.const import (
 )
 from frigate.data_processing.types import PostProcessDataEnum
 from frigate.embeddings.embeddings import Embeddings
+from frigate.genai.manager import GenAIClientManager
 from frigate.types import TrackedObjectUpdateTypesEnum
-from frigate.util.audio import get_audio_from_recording
+from frigate.util.audio import (
+    clean_transcript,
+    get_audio_from_recording,
+    resolve_language,
+)
 
 from ..types import DataProcessorMetrics
 from .api import PostProcessorApi
@@ -34,15 +40,25 @@ class AudioTranscriptionPostProcessor(PostProcessorApi):
         requestor: InterProcessRequestor,
         embeddings: Embeddings,
         metrics: DataProcessorMetrics,
+        genai_manager: GenAIClientManager | None = None,
     ):
         super().__init__(config, metrics, None)
         self.config = config
         self.requestor = requestor
         self.embeddings = embeddings
+        self.genai_manager = genai_manager
         self.recognizer = None
         self.transcription_lock = threading.Lock()
         self.transcription_thread: threading.Thread | None = None
         self.transcription_running = False
+        self._use_genai = not isinstance(
+            config.audio_transcription.model, AudioTranscriptionModelEnum
+        )
+
+        if self._use_genai:
+            # never build the local recognizer on the GenAI path; WhisperModel
+            # downloads several hundred MB on first use
+            return
 
         # faster-whisper handles model downloading automatically
         self.model_path = os.path.join(MODEL_CACHE_DIR, "whisper")
@@ -147,6 +163,31 @@ class AudioTranscriptionPostProcessor(PostProcessorApi):
             logger.error(f"Error in audio transcription post-processing: {e}")
 
     def __transcribe_audio(self, audio_data: bytes) -> str | None:
+        """Transcribe WAV audio data with the configured backend."""
+        if self._use_genai:
+            return self.__transcribe_audio_genai(audio_data)
+
+        return self.__transcribe_audio_whisper(audio_data)
+
+    def __transcribe_audio_genai(self, audio_data: bytes) -> str | None:
+        """Hand the WAV bytes to the GenAI provider holding the transcribe role."""
+        client = self.genai_manager.transcribe_client if self.genai_manager else None
+
+        if not client:
+            logger.error(
+                "audio_transcription.model is '%s' (GenAI provider) but no transcribe "
+                "client is configured. Ensure the GenAI provider has 'transcribe' in its roles",
+                self.config.audio_transcription.model,
+            )
+            return None
+
+        text = client.transcribe(
+            audio_data,
+            language=resolve_language(self.config.audio_transcription.language),
+        )
+        return clean_transcript(text) or None
+
+    def __transcribe_audio_whisper(self, audio_data: bytes) -> str | None:
         """Transcribe WAV audio data using faster-whisper."""
         if not self.recognizer:
             logger.debug("Recognizer not initialized")
@@ -160,7 +201,7 @@ class AudioTranscriptionPostProcessor(PostProcessorApi):
 
             segments, info = self.recognizer.transcribe(
                 temp_wav,
-                language=self.config.audio_transcription.language,
+                language=resolve_language(self.config.audio_transcription.language),
                 beam_size=5,
             )
 

frigate/data_processing/post/object_descriptions.py

@@ -253,7 +253,7 @@ class ObjectDescriptionProcessor(PostProcessorApi):
             # Crop snapshot based on region
             # provide full image if region doesn't exist (manual events)
             height, width = img.shape[:2]
-            x1_rel, y1_rel, width_rel, height_rel = event.data.get(  # type: ignore[attr-defined]
+            x1_rel, y1_rel, width_rel, height_rel = event.data.get(
                 "region", [0, 0, 1, 1]
             )
             x1, y1 = int(x1_rel * width), int(y1_rel * height)

frigate/data_processing/post/review_annotations.py

@@ -0,0 +1,416 @@
+"""Frame annotations derived from object tracking data.
+
+Builds short notes describing what changed during a review item, keyed to the
+frames sampled from it. Everything here comes from data already recorded
+(each event's `path_data` trajectory, the timeline's stationary/active
+changes, and the review item's state classification changes), so the notes
+can be stated to the model as fact rather than as something it must perceive.
+"""
+
+import logging
+import math
+from collections.abc import Sequence
+from typing import Any
+
+from frigate.models import Event, Timeline
+
+logger = logging.getLogger(__name__)
+
+# Movement smaller than this (normalized frame units) between two path points
+# is treated as the object holding still rather than travelling.
+STILL_THRESHOLD = 0.02
+
+# A heading change beyond this (dot product against the leg's own heading)
+# counts as the object turning back rather than curving.
+REVERSAL_DOT = -0.3
+
+# A run of travel shorter than this (normalized frame units) is treated as
+# milling about rather than going somewhere. Without it, a subject pacing in
+# one spot produces a burst of contradictory "turns around" notes on a single
+# frame.
+MIN_LEG_DISTANCE = 0.08
+
+# Movement that begins within this many seconds of detection is folded into
+# the detection note, so each arrival reads as one event instead of several.
+DETECT_MOVE_MERGE_SECONDS = 2.0
+
+STATE_CHANGE_PHRASES = {
+    "stationary": "has stopped moving",
+    "active": "starts moving again",
+}
+
+Point = tuple[float, float, float]
+Leg = tuple[int, int]
+
+
+def describe_position(x: float, y: float) -> str:
+    """Name a normalized frame position in plain terms."""
+    horizontal = "left" if x < 0.34 else ("right" if x > 0.66 else "center")
+    vertical = "top" if y < 0.34 else ("bottom" if y > 0.66 else "middle")
+
+    if horizontal == "center" and vertical == "middle":
+        return "the middle of the frame"
+
+    if horizontal == "center":
+        return f"the {vertical} of the frame"
+
+    if vertical == "middle":
+        return f"the {horizontal} of the frame"
+
+    return f"the {vertical} {horizontal} of the frame"
+
+
+def describe_heading(dx: float, dy: float) -> str:
+    """Name a direction of travel in frame terms.
+
+    y grows downward in normalized coordinates, so a falling y reads as moving
+    toward the top of the frame.
+    """
+    parts = []
+
+    if abs(dy) > abs(dx) * 0.4:
+        parts.append("down" if dy > 0 else "up")
+
+    if abs(dx) > abs(dy) * 0.4:
+        parts.append("right" if dx > 0 else "left")
+
+    return " and ".join(parts) if parts else "in place"
+
+
+def event_name(event: dict[str, Any]) -> str:
+    """Name an object for the notes, e.g. 'a person' or 'waste bin "Compost"'.
+
+    Objects are never numbered or given track identifiers. Frigate opens a new
+    tracked object whenever a subject is re-detected, so the tracking data
+    cannot say whether two entries are the same subject, and the notes stay
+    ambiguous rather than implying either answer.
+    """
+    label = str(event["label"]).replace("_", " ").replace("-verified", "")
+    sub_label = event.get("sub_label")
+
+    if sub_label:
+        return f'{label} "{sub_label}"'
+
+    article = "an" if label[:1].lower() in "aeiou" else "a"
+    return f"{article} {label}"
+
+
+def describe_classification_change(change: dict[str, Any]) -> str:
+    """Phrase a state classification change, e.g. 'front gate changed from
+    closed to open'."""
+    model = str(change["model"]).replace("_", " ")
+    before = str(change["from"]).replace("_", " ")
+    after = str(change["to"]).replace("_", " ")
+    return f"{model} changed from {before} to {after}"
+
+
+def path_legs(points: list[Point]) -> list[Leg]:
+    """Split a trajectory into runs of travel in a consistent direction.
+
+    A leg ends when the subject starts moving back against the direction that
+    leg established, and only once the leg has covered MIN_LEG_DISTANCE, so
+    jitter around a standing subject does not register as a turn.
+
+    Returns (start, end) index pairs into `points`.
+    """
+    legs: list[Leg] = []
+    start = 0
+
+    for i in range(1, len(points)):
+        lx = points[i][0] - points[start][0]
+        ly = points[i][1] - points[start][1]
+        leg_distance = (lx * lx + ly * ly) ** 0.5
+
+        if leg_distance < MIN_LEG_DISTANCE:
+            continue
+
+        sx = points[i][0] - points[i - 1][0]
+        sy = points[i][1] - points[i - 1][1]
+        step = (sx * sx + sy * sy) ** 0.5
+
+        if step < STILL_THRESHOLD:
+            continue
+
+        dot = (lx / leg_distance) * (sx / step) + (ly / leg_distance) * (sy / step)
+
+        if dot < REVERSAL_DOT:
+            legs.append((start, i - 1))
+            start = i - 1
+
+    if start < len(points) - 1:
+        legs.append((start, len(points) - 1))
+
+    return [
+        (a, b)
+        for a, b in legs
+        if ((points[b][0] - points[a][0]) ** 2 + (points[b][1] - points[a][1]) ** 2)
+        ** 0.5
+        >= MIN_LEG_DISTANCE
+    ]
+
+
+def path_points(path_data: list[Any]) -> list[Point]:
+    """Flatten path_data into (x, y, timestamp) tuples, or [] if malformed."""
+    try:
+        return [(p[0][0], p[0][1], p[1]) for p in path_data or []]
+    except (IndexError, TypeError):
+        logger.debug("Malformed path_data, skipping trajectory notes")
+        return []
+
+
+def leg_start_time(points: list[Point], leg: Leg) -> float:
+    """When a leg's movement actually began.
+
+    path_data always keeps an object's first two samples, so a leg can open
+    with points recorded long before the object moved. The first sample that
+    has left the leg's origin is the earliest evidence of movement.
+    """
+    a, b = leg
+    x0, y0, t0 = points[a]
+
+    for x, y, t in points[a + 1 : b + 1]:
+        if math.hypot(x - x0, y - y0) >= STILL_THRESHOLD:
+            return t
+
+    return t0
+
+
+def leg_heading(points: list[Point], leg: Leg) -> str:
+    a, b = leg
+    return describe_heading(points[b][0] - points[a][0], points[b][1] - points[a][1])
+
+
+def leg_phrase(points: list[Point], leg: Leg, first: bool) -> str:
+    """Describe the start of a leg, e.g. 'turns around at ... and heads left'."""
+    heading = leg_heading(points, leg)
+    place = describe_position(points[leg[0]][0], points[leg[0]][1])
+
+    if first:
+        return f"starts moving {heading} from {place}"
+
+    return f"turns around at {place} and heads {heading}"
+
+
+def path_moments(path_data: list[Any]) -> list[tuple[float, str]]:
+    """Key moments in one trajectory as (timestamp, phrase).
+
+    Emits one note per leg of travel. Where the last leg ends is left out:
+    path_data only records significant movement, so its final point cannot
+    distinguish an object coming to rest from one leaving the frame.
+    """
+    points = path_points(path_data)
+
+    if len(points) < 2:
+        return []
+
+    return [
+        (leg_start_time(points, leg), leg_phrase(points, leg, index == 0))
+        for index, leg in enumerate(path_legs(points))
+    ]
+
+
+def build_timeline(
+    events: list[dict[str, Any]],
+    span_end: float,
+    state_changes: Sequence[dict[str, Any]] = (),
+) -> list[tuple[float, str]]:
+    """All annotated moments across every event, in time order.
+
+    Only changes are noted, since those are what sparse frames miss; an
+    object's state at the end of the clip is visible in the last frame.
+    `span_end` is the timestamp of the last sampled frame, and moments past it
+    describe nothing the model can see. A track ending means the object
+    stopped being detected, which may or may not mean it left the frame.
+
+    `state_changes` are timeline rows (timestamp, source_id, class_type); the
+    stationary and active ones become "has stopped moving" / "starts moving
+    again". Frigate only marks an object stationary after it has been still
+    for a while, which the past-tense wording reflects.
+
+    Each object keeps its own notes. Folding an object into the note of the
+    person moving it ("alongside ...") was tried and made models lose track of
+    where the object went.
+    """
+    changes_by_event: dict[str, list[tuple[float, str]]] = {}
+
+    for change in state_changes:
+        phrase = STATE_CHANGE_PHRASES.get(change["class_type"])
+
+        if phrase:
+            changes_by_event.setdefault(change["source_id"], []).append(
+                (change["timestamp"], phrase)
+            )
+
+    timeline: list[tuple[float, str]] = []
+
+    for event in sorted(events, key=lambda e: e["start_time"]):
+        # Frame extraction can come up short at the end of a clip, leaving
+        # objects that only appear after the last frame we actually have.
+        if event["start_time"] > span_end:
+            continue
+
+        points = path_points(event.get("path_data") or [])
+        legs = path_legs(points) if len(points) >= 2 else []
+        name = event_name(event)
+        detected_at = event["start_time"]
+        where = describe_position(points[0][0], points[0][1]) if points else "the frame"
+        merges = bool(legs) and leg_start_time(points, legs[0]) - detected_at <= (
+            DETECT_MOVE_MERGE_SECONDS
+        )
+        remaining = list(enumerate(legs))
+        moments: list[tuple[float, str]] = []
+
+        if merges:
+            heading = leg_heading(points, legs[0])
+            timeline.append(
+                (detected_at, f"{name} first detected at {where}, moving {heading}")
+            )
+            remaining = remaining[1:]
+        else:
+            timeline.append((detected_at, f"{name} first detected at {where}"))
+
+        for index, leg in remaining:
+            moments.append(
+                (
+                    leg_start_time(points, leg),
+                    leg_phrase(points, leg, index == 0),
+                )
+            )
+
+        moments.extend(
+            (timestamp, phrase)
+            for timestamp, phrase in changes_by_event.get(event["id"], [])
+            if timestamp >= detected_at
+        )
+
+        for timestamp, phrase in moments:
+            if timestamp <= span_end:
+                timeline.append((timestamp, f"{name} {phrase}"))
+
+        if event["end_time"] and event["end_time"] <= span_end:
+            timeline.append((event["end_time"], f"{name} is no longer detected"))
+
+    return sorted(timeline, key=lambda m: m[0])
+
+
+def annotations_by_frame(
+    timeline: list[tuple[float, str]], frame_times: list[float]
+) -> dict[int, list[str]]:
+    """Bucket timeline moments onto the frame that follows each one.
+
+    A moment is attached to the first frame at or after it happened, so the
+    note always precedes the image in which the change becomes visible.
+    """
+    buckets: dict[int, list[str]] = {}
+
+    if not frame_times:
+        return buckets
+
+    for timestamp, phrase in timeline:
+        index = next(
+            (i for i, ft in enumerate(frame_times) if ft >= timestamp),
+            len(frame_times) - 1,
+        )
+        buckets.setdefault(index, []).append(phrase)
+
+    return buckets
+
+
+def get_tracked_events(detection_ids: list[str]) -> list[dict[str, Any]]:
+    """Load the tracked objects behind a review item's detections."""
+    if not detection_ids:
+        return []
+
+    rows = list(
+        Event.select(
+            Event.id,
+            Event.label,
+            Event.sub_label,
+            Event.start_time,
+            Event.end_time,
+            Event.data,
+        )
+        .where(Event.id << detection_ids)
+        .dicts()
+        .iterator()
+    )
+
+    return [
+        {
+            "id": row["id"],
+            "label": row["label"],
+            "sub_label": row["sub_label"],
+            "start_time": row["start_time"],
+            "end_time": row["end_time"],
+            "path_data": (row["data"] or {}).get("path_data") or [],
+        }
+        for row in rows
+        if row["start_time"] is not None
+    ]
+
+
+def get_state_changes(detection_ids: list[str]) -> list[dict[str, Any]]:
+    """Stationary/active changes the timeline recorded for these objects."""
+    if not detection_ids:
+        return []
+
+    return list(
+        Timeline.select(Timeline.timestamp, Timeline.source_id, Timeline.class_type)
+        .where(
+            (Timeline.source_id << detection_ids)
+            & (Timeline.class_type << list(STATE_CHANGE_PHRASES))
+        )
+        .dicts()
+        .iterator()
+    )
+
+
+def build_frame_captions(
+    detection_ids: list[str],
+    frame_times: list[float],
+    classification_changes: Sequence[dict[str, Any]] = (),
+) -> list[str]:
+    """A caption for each sampled frame, in frame order.
+
+    Every frame gets its index and elapsed time so the model can tell them
+    apart; frames where something changed also carry the tracker and state
+    classification notes for that moment. Returns an empty list when there is
+    nothing to say, which callers treat as a reason to fall back to sending
+    plain frames.
+    """
+    if not frame_times:
+        return []
+
+    span_end = frame_times[-1]
+    timeline: list[tuple[float, str]] = []
+    events = get_tracked_events(detection_ids)
+
+    # audio and manual review items can have state changes but no tracked objects
+    if events:
+        timeline.extend(
+            (timestamp, f"[tracker] {note}")
+            for timestamp, note in build_timeline(
+                events, span_end, get_state_changes(detection_ids)
+            )
+        )
+
+    timeline.extend(
+        (change["timestamp"], f"[state] {describe_classification_change(change)}")
+        for change in classification_changes
+        if change["timestamp"] <= span_end
+    )
+    buckets = annotations_by_frame(sorted(timeline, key=lambda m: m[0]), frame_times)
+
+    if not buckets:
+        return []
+
+    total = len(frame_times)
+    origin = frame_times[0]
+    captions: list[str] = []
+
+    for index, timestamp in enumerate(frame_times):
+        lines = [f"Frame {index + 1} of {total} (+{timestamp - origin:.1f}s):"]
+        lines.extend(buckets.get(index, []))
+        captions.append("\n".join(lines))
+
+    return captions

frigate/data_processing/post/review_descriptions.py

@@ -8,7 +8,7 @@ import os
 import shutil
 import threading
 from pathlib import Path
-from typing import Any
+from typing import Any, cast
 
 import cv2
 from peewee import DoesNotExist
@@ -19,7 +19,11 @@ from frigate.comms.embeddings_updater import EmbeddingsRequestEnum
 from frigate.comms.inter_process import InterProcessRequestor
 from frigate.config import FrigateConfig
 from frigate.config.camera import CameraConfig
-from frigate.config.camera.review import GenAIReviewConfig, ImageSourceEnum
+from frigate.config.camera.review import (
+    GenAIReviewConfig,
+    ImageSourceEnum,
+    ReviewFrameModeEnum,
+)
 from frigate.const import (
     ATTRIBUTE_LABEL_DISPLAY_MAP,
     CACHE_DIR,
@@ -36,6 +40,7 @@ from frigate.util.image import get_image_from_recording
 
 from ..post.api import PostProcessorApi
 from ..types import DataProcessorMetrics
+from .review_annotations import build_frame_captions, describe_classification_change
 
 logger = logging.getLogger(__name__)
 
@@ -43,6 +48,7 @@ RECORDING_BUFFER_EXTENSION_PERCENT = 0.10
 MIN_RECORDING_DURATION = 10
 MAX_IMAGE_TOKENS = 24000
 MAX_FRAMES_PER_SECOND = 1
+MAX_ANNOTATED_FRAMES = 28
 
 
 class ReviewDescriptionProcessor(PostProcessorApi):
@@ -67,6 +73,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         duration: float,
         image_source: ImageSourceEnum = ImageSourceEnum.preview,
         height: int = 480,
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
     ) -> int:
         """Calculate optimal number of frames based on event duration, context size,
         image source, and resolution.
@@ -80,6 +87,8 @@ class ReviewDescriptionProcessor(PostProcessorApi):
           - MAX_FRAMES_PER_SECOND x duration, to avoid drowning short events in
             near-duplicate frames where the model latches onto the redundant middle
             and skips the start/end action
+          - MAX_ANNOTATED_FRAMES in annotated mode, where the tracking notes
+            already carry the sequence
         """
         client = self.genai_manager.description_client
 
@@ -125,6 +134,10 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         max_frames_by_tokens = int(image_token_budget / tokens_per_image)
         max_frames_by_duration = int(duration * MAX_FRAMES_PER_SECOND)
         max_frames = min(max_frames_by_tokens, max_frames_by_duration)
+
+        if frame_mode == ReviewFrameModeEnum.annotated_frames:
+            max_frames = min(max_frames, MAX_ANNOTATED_FRAMES)
+
         return max(max_frames, 3)
 
     def process_data(
@@ -166,6 +179,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 return
 
             image_source = camera_config.review.genai.image_source
+            frame_mode = camera_config.review.genai.frame_mode
 
             if image_source == ImageSourceEnum.recordings:
                 buffer_extension = get_recording_buffer_extension(
@@ -174,40 +188,43 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 final_data["start_time"] -= buffer_extension
                 final_data["end_time"] += buffer_extension
 
-                thumbs = self.get_recording_frames(
+                frames = self.get_recording_frames(
                     camera,
                     final_data["start_time"],
                     final_data["end_time"],
                     height=480,  # Use 480p for good balance between quality and token usage
+                    frame_mode=frame_mode,
                 )
 
-                if not thumbs:
+                if not frames:
                     # Fallback to preview frames if no recordings available
                     logger.warning(
                         f"No recording frames found for {camera}, falling back to preview frames"
                     )
-                    thumbs = self.get_preview_frames_as_bytes(
+                    frames = self.get_preview_frames_as_bytes(
                         camera,
                         final_data["start_time"],
                         final_data["end_time"],
                         final_data["thumb_path"],
                         id,
                         camera_config.review.genai.debug_save_thumbnails,
+                        frame_mode,
                     )
                 elif camera_config.review.genai.debug_save_thumbnails:
-                    self.save_debug_recording_frames(id, thumbs)
+                    self.save_debug_recording_frames(id, frames)
             else:
                 # Use preview frames
-                thumbs = self.get_preview_frames_as_bytes(
+                frames = self.get_preview_frames_as_bytes(
                     camera,
                     final_data["start_time"],
                     final_data["end_time"],
                     final_data["thumb_path"],
                     id,
                     camera_config.review.genai.debug_save_thumbnails,
+                    frame_mode,
                 )
 
-            self.start_analysis(camera_config, final_data, thumbs)
+            self.start_analysis(camera_config, final_data, frames)
 
     def handle_request(self, topic: str, request_data: dict[str, Any]) -> str | None:
         if topic == EmbeddingsRequestEnum.regenerate_review_description.value:
@@ -237,6 +254,10 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                     "start_time": r["start_time"],
                     "end_time": r["end_time"],
                     "metadata": r["data"]["metadata"],
+                    "state_changes": [
+                        describe_classification_change(change)
+                        for change in sorted_classification_state_changes(r["data"])
+                    ],
                 }
                 for r in (
                     ReviewSegment.select(
@@ -281,6 +302,9 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 primary_item["start_time"] = primary_seg["start_time"]
                 primary_item["end_time"] = primary_seg["end_time"]
 
+                if primary_seg["state_changes"]:
+                    primary_item["state_changes"] = primary_seg["state_changes"]
+
                 # Find overlapping contextual items from other cameras
                 primary_start = primary_seg["start_time"]
                 primary_end = primary_seg["end_time"]
@@ -301,14 +325,25 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                     seg_end = seg["end_time"]
 
                     if seg_start < primary_end and primary_start < seg_end:
-                        # Avoid duplicates if same camera has multiple overlapping segments
-                        if seg_camera not in seen_contextual_cameras:
-                            contextual_item = copy.deepcopy(seg["metadata"])
-                            contextual_item["camera"] = seg_camera
-                            contextual_item["start_time"] = seg_start
-                            contextual_item["end_time"] = seg_end
-                            contextual_items.append(contextual_item)
-                            seen_contextual_cameras.add(seg_camera)
+                        # Avoid duplicates if same camera has multiple overlapping
+                        # segments. One with state changes is kept as its own item
+                        # so each change stays within its item's time range.
+                        if (
+                            seg_camera in seen_contextual_cameras
+                            and not seg["state_changes"]
+                        ):
+                            continue
+
+                        contextual_item = copy.deepcopy(seg["metadata"])
+                        contextual_item["camera"] = seg_camera
+                        contextual_item["start_time"] = seg_start
+                        contextual_item["end_time"] = seg_end
+
+                        if seg["state_changes"]:
+                            contextual_item["state_changes"] = seg["state_changes"]
+
+                        contextual_items.append(contextual_item)
+                        seen_contextual_cameras.add(seg_camera)
 
                 # Add context array to primary item
                 primary_item["context"] = contextual_items
@@ -386,31 +421,51 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         buffer_extension = get_recording_buffer_extension(
             final_data["end_time"] - final_data["start_time"]
         )
-        thumbs = self.get_recording_frames(
+        frames = self.get_recording_frames(
             str(review.camera),
             final_data["start_time"] - buffer_extension,
             final_data["end_time"] + buffer_extension,
             height=480,
+            frame_mode=camera_config.review.genai.frame_mode,
         )
 
-        if not thumbs:
+        if not frames:
             logger.error(
                 "No recording frames are available for review item %s", review_id
             )
             return
 
         if camera_config.review.genai.debug_save_thumbnails:
-            self.save_debug_recording_frames(review_id, thumbs)
+            self.save_debug_recording_frames(review_id, frames)
 
-        self.start_analysis(camera_config, final_data, thumbs)
+        self.start_analysis(camera_config, final_data, frames)
 
     def start_analysis(
         self,
         camera_config: CameraConfig,
         final_data: dict[str, Any],
-        thumbs: list[bytes],
+        frames: list[tuple[bytes, float]],
     ) -> None:
         """Kick off description generation for a review item in the background."""
+        thumbs = [frame for frame, _ in frames]
+        captions: list[str] = []
+
+        if (
+            camera_config.review.genai.frame_mode
+            == ReviewFrameModeEnum.annotated_frames
+        ):
+            captions = build_frame_captions(
+                final_data["data"].get("detections") or [],
+                [timestamp for _, timestamp in frames],
+                sorted_classification_state_changes(final_data["data"]),
+            )
+
+            if not captions:
+                logger.debug(
+                    "No tracking annotations for review item %s, sending plain frames",
+                    final_data["id"],
+                )
+
         self.review_desc_dps.update()
         threading.Thread(
             target=run_analysis,
@@ -421,19 +476,22 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 camera_config,
                 final_data,
                 thumbs,
+                captions,
                 camera_config.review.genai,
                 sorted(self.config.all_labels),
                 self.config.all_attributes,
             ),
         ).start()
 
-    def save_debug_recording_frames(self, review_id: str, thumbs: list[bytes]) -> None:
+    def save_debug_recording_frames(
+        self, review_id: str, frames: list[tuple[bytes, float]]
+    ) -> None:
         """Write the recording frames sent to the provider out for debugging."""
         Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir(
             parents=True, exist_ok=True
         )
 
-        for idx, frame_bytes in enumerate(thumbs):
+        for idx, (frame_bytes, _) in enumerate(frames):
             with open(
                 os.path.join(CLIPS_DIR, f"genai-requests/{review_id}/{idx}.jpg"),
                 "wb",
@@ -445,7 +503,9 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         camera: str,
         start_time: float,
         end_time: float,
-    ) -> list[str]:
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
+    ) -> list[tuple[str, float]]:
+        """Preview frame paths paired with the time each one was captured."""
         preview_dir = os.path.join(CACHE_DIR, "preview_frames")
         file_start = f"preview_{camera}-"
         start_file = f"{file_start}{start_time}.webp"
@@ -477,18 +537,29 @@ class ReviewDescriptionProcessor(PostProcessorApi):
 
         frame_count = len(all_frames)
         desired_frame_count = self.calculate_frame_count(
-            camera, duration=end_time - start_time
+            camera,
+            duration=end_time - start_time,
+            frame_mode=frame_mode,
         )
 
+        def with_timestamp(path: str) -> tuple[str, float]:
+            # Preview frames are named preview_<camera>-<timestamp>.webp
+            stem = os.path.basename(path).removesuffix(".webp")
+
+            try:
+                return (path, float(stem.removeprefix(file_start)))
+            except ValueError:
+                return (path, start_time)
+
         if frame_count <= desired_frame_count:
-            return all_frames
+            return [with_timestamp(f) for f in all_frames]
 
         selected_frames = []
         step_size = (frame_count - 1) / (desired_frame_count - 1)
 
         for i in range(desired_frame_count):
             index = round(i * step_size)
-            selected_frames.append(all_frames[index])
+            selected_frames.append(with_timestamp(all_frames[index]))
 
         return selected_frames
 
@@ -498,11 +569,12 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         start_time: float,
         end_time: float,
         height: int = 480,
-    ) -> list[bytes]:
-        """Get frames from recordings at specified timestamps."""
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
+    ) -> list[tuple[bytes, float]]:
+        """Get frames from recordings paired with the time each was captured."""
         duration = end_time - start_time
         desired_frame_count = self.calculate_frame_count(
-            camera, duration, ImageSourceEnum.recordings, height
+            camera, duration, ImageSourceEnum.recordings, height, frame_mode
         )
 
         # Calculate evenly spaced timestamps throughout the duration
@@ -528,7 +600,8 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                     .get()
                 )
 
-                time_in_segment = ts - recording.start_time
+                # start_time is a DateTimeField holding a unix timestamp
+                time_in_segment = ts - cast(float, recording.start_time)
                 return get_image_from_recording(
                     self.config.ffmpeg,
                     recording.path,
@@ -539,7 +612,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
             except DoesNotExist:
                 return None
 
-        frames = []
+        frames: list[tuple[bytes, float]] = []
 
         for timestamp in timestamps:
             try:
@@ -552,7 +625,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                     image_data = extract_frame_from_recording(rounded_timestamp)
 
                 if image_data:
-                    frames.append(image_data)
+                    frames.append((image_data, timestamp))
                 else:
                     logger.warning(
                         f"No recording found for {camera} at timestamp {timestamp}"
@@ -573,7 +646,8 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         thumb_path_fallback: str,
         review_id: str,
         save_debug: bool,
-    ) -> list[bytes]:
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
+    ) -> list[tuple[bytes, float]]:
         """Get preview frames and convert them to JPEG bytes.
 
         Args:
@@ -585,14 +659,14 @@ class ReviewDescriptionProcessor(PostProcessorApi):
             save_debug: Whether to save debug thumbnails
 
         Returns:
-            List of JPEG image bytes
+            List of (JPEG image bytes, capture timestamp) pairs
         """
-        frame_paths = self.get_cache_frames(camera, start_time, end_time)
+        frame_paths = self.get_cache_frames(camera, start_time, end_time, frame_mode)
         if not frame_paths:
-            frame_paths = [thumb_path_fallback]
+            frame_paths = [(thumb_path_fallback, start_time)]
 
-        thumbs = []
-        for idx, thumb_path in enumerate(frame_paths):
+        thumbs: list[tuple[bytes, float]] = []
+        for idx, (thumb_path, timestamp) in enumerate(frame_paths):
             thumb_data = cv2.imread(thumb_path)
 
             if thumb_data is None:
@@ -605,7 +679,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 ".jpg", thumb_data, [int(cv2.IMWRITE_JPEG_QUALITY), 100]
             )
             if ret:
-                thumbs.append(jpg.tobytes())
+                thumbs.append((jpg.tobytes(), timestamp))
 
             if save_debug:
                 Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir(
@@ -633,6 +707,39 @@ def get_recording_buffer_extension(duration: float) -> float:
     return buffer_extension
 
 
+def sorted_classification_state_changes(
+    review_data: dict[str, Any],
+) -> list[dict[str, Any]]:
+    """A review item's state classification changes in time order."""
+    return sorted(
+        review_data.get("classification_state_changes") or [],
+        key=lambda change: change["timestamp"],
+    )
+
+
+def format_classification_state_changes(
+    changes: list[dict[str, Any]], start_time: float, end_time: float
+) -> list[str]:
+    """Phrase state classification changes with their timing in the activity.
+
+    Changes are attached while the review item is active, which runs past its
+    end_time by the review cutoff, and a few seconds before its start.
+    """
+    lines = []
+
+    for change in changes:
+        if change["timestamp"] < start_time:
+            when = "just before the activity started"
+        elif change["timestamp"] > end_time:
+            when = "after the activity ended"
+        else:
+            when = f"{round(change['timestamp'] - start_time)}s into the activity"
+
+        lines.append(f"{describe_classification_change(change)}, {when}")
+
+    return lines
+
+
 def run_analysis(
     requestor: InterProcessRequestor,
     genai_client: GenAIClient,
@@ -640,6 +747,7 @@ def run_analysis(
     camera_config: CameraConfig,
     final_data: dict[str, Any],
     thumbs: list[bytes],
+    frame_captions: list[str],
     genai_config: GenAIReviewConfig,
     labelmap_objects: list[str],
     attribute_labels: list[str],
@@ -687,6 +795,13 @@ def run_analysis(
                 unified_objects.append(object_type)
 
     analytics_data["unified_objects"] = unified_objects
+    analytics_data["classification_state_changes"] = (
+        format_classification_state_changes(
+            sorted_classification_state_changes(final_data["data"]),
+            final_data["start_time"],
+            final_data["end_time"],
+        )
+    )
 
     metadata = genai_client.generate_review_description(
         analytics_data,
@@ -696,6 +811,7 @@ def run_analysis(
         genai_config.debug_save_thumbnails,
         genai_config.activity_context_prompt,
         genai_config.response_style,
+        frame_captions,
     )
     review_inference_speed.update(datetime.datetime.now().timestamp() - start)
 

frigate/data_processing/post/semantic_trigger.py

@@ -237,7 +237,7 @@ class SemanticTriggerProcessor(PostProcessorApi):
                     return
 
                 # Skip the event if not an object
-                if event.data.get("type") != "object":  # type: ignore[attr-defined]
+                if event.data.get("type") != "object":
                     return
 
                 thumbnail_bytes = get_event_thumbnail_bytes(event)

frigate/data_processing/real_time/audio_transcription.py

@@ -1,16 +1,19 @@
 """Handle processing audio for speech transcription using sherpa-onnx with FFmpeg pipe."""
 
+import collections
 import logging
 import os
 import queue
 import threading
-from typing import Any
+import time
+from typing import TYPE_CHECKING, Any
 
 import numpy as np
 
 from frigate.comms.inter_process import InterProcessRequestor
 from frigate.config import CameraConfig, FrigateConfig
-from frigate.const import MODEL_CACHE_DIR
+from frigate.config.classification import AudioTranscriptionModelEnum
+from frigate.const import AUDIO_DURATION, MODEL_CACHE_DIR
 from frigate.data_processing.common.audio_transcription.model import (
     AudioTranscriptionModelRunner,
 )
@@ -18,12 +21,38 @@ from frigate.data_processing.real_time.whisper_online import (
     FasterWhisperASR,
     OnlineASRProcessor,
 )
+from frigate.util.audio import (
+    clean_transcript,
+    pcm16_to_wav,
+    resolve_language,
+    stitch_transcripts,
+)
 
 from ..types import DataProcessorMetrics
 from .api import RealTimeProcessorApi
 
+if TYPE_CHECKING:
+    # importing frigate.genai eagerly would pull the provider SDKs into the
+    # audio process even when transcription runs on a local model
+    from frigate.genai.manager import GenAIClientManager
+
 logger = logging.getLogger(__name__)
 
+# Number of ~0.975s audio detector chunks per GenAI request. The window advances
+# one chunk at a time, so two chunks means a 50% overlap: every word lands whole
+# in at least one window, which whisper-family models need to avoid hallucinating
+# on a clipped clip. The cadence is fixed by the audio detector's frame size, so
+# this is a constant rather than a config knob.
+GENAI_WINDOW_CHUNKS = 2
+
+# Bound the queue at ~30s of audio so a slow or hung provider cannot grow it
+# without limit. The producer is the ffmpeg read thread and must never block.
+AUDIO_QUEUE_MAXSIZE = int(30 / AUDIO_DURATION)
+
+# A backed-up queue drops a chunk per cycle, so warning on each one would spam
+# the log once a second per camera for as long as the provider stays slow.
+AUDIO_DROP_WARN_INTERVAL = 10.0
+
 
 class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
     def __init__(
@@ -31,9 +60,10 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
         config: FrigateConfig,
         camera_config: CameraConfig,
         requestor: InterProcessRequestor,
-        model_runner: AudioTranscriptionModelRunner,
+        model_runner: AudioTranscriptionModelRunner | None,
         metrics: DataProcessorMetrics,
         stop_event: threading.Event,
+        genai_manager: "GenAIClientManager | None" = None,
     ):
         super().__init__(config, metrics)
         self.config = config
@@ -42,11 +72,31 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
         self.stream: Any = None
         self.whisper_model: FasterWhisperASR | None = None
         self.model_runner = model_runner
+        self.genai_manager = genai_manager
         self.transcription_segments: list[str] = []
-        self.audio_queue: queue.Queue[tuple[dict[str, Any], np.ndarray]] = queue.Queue()
+        self.audio_queue: queue.Queue[tuple[dict[str, Any], np.ndarray]] = queue.Queue(
+            maxsize=AUDIO_QUEUE_MAXSIZE
+        )
         self.stop_event = stop_event
+        self._use_genai = not isinstance(
+            config.audio_transcription.model, AudioTranscriptionModelEnum
+        )
+        # sliding window of raw int16 chunks; the deque's maxlen is what evicts
+        # the oldest chunk and so produces the overlap
+        self._genai_window: collections.deque[np.ndarray] = collections.deque(
+            maxlen=GENAI_WINDOW_CHUNKS
+        )
+        self._genai_committed = ""
+        # set by the producer when it discards a chunk, so the consumer knows the
+        # audio it is about to receive is not contiguous with what it buffered
+        self._audio_dropped = threading.Event()
+        self._last_drop_warning = 0.0
 
     def __build_recognizer(self) -> None:
+        if self._use_genai:
+            # nothing local to load; never import sherpa or FasterWhisperASR
+            return
+
         try:
             if self.config.audio_transcription.model_size == "large":
                 # Whisper models need to be per-process and can only run one stream at a time
@@ -64,7 +114,7 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
                 self.stream = OnlineASRProcessor(
                     asr=self.whisper_model,
                 )
-            else:
+            elif self.model_runner is not None:
                 logger.debug(f"Loading sherpa stream for {self.camera_config.name}")
                 self.stream = self.model_runner.model.create_stream()
             logger.debug(
@@ -76,6 +126,15 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
             )
 
     def __process_audio_stream(self, audio_data: np.ndarray) -> tuple[str, bool] | None:
+        # must precede both the model_runner guard (model_runner is None on this
+        # path) and the float32 normalization below (GenAI wants untouched int16)
+        if self._use_genai:
+            return self.__process_audio_genai(audio_data)
+
+        if self.model_runner is None:
+            logger.debug("Audio transcription (live) model runner not initialized")
+            return None
+
         if (
             self.model_runner.model is None
             and self.config.audio_transcription.model_size == "small"
@@ -140,6 +199,82 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
             logger.error(f"Error processing audio stream: {e}")
             return None
 
+    def __process_audio_genai(self, audio_data: np.ndarray) -> tuple[str, bool] | None:
+        """Transcribe a sliding overlapped window through the GenAI provider."""
+        client = self.genai_manager.transcribe_client if self.genai_manager else None
+
+        if not client:
+            logger.error(
+                "audio_transcription.model is '%s' (GenAI provider) but no transcribe "
+                "client is configured. Ensure the GenAI provider has 'transcribe' in its roles",
+                self.config.audio_transcription.model,
+            )
+            return None
+
+        if self._audio_dropped.is_set():
+            self._audio_dropped.clear()
+
+            # Chunks were discarded between what is buffered and this one, so
+            # concatenating them would splice non-adjacent audio into one window
+            # and destroy the overlap the stitcher depends on.
+            self._genai_window.clear()
+
+            if self._genai_committed:
+                # the transcript has a gap in it; close the utterance out rather
+                # than stitching across missing speech
+                return self.__end_genai_utterance()
+
+        self._genai_window.append(audio_data)
+
+        if len(self._genai_window) < GENAI_WINDOW_CHUNKS:
+            # wait for a full window so the first request is never a clipped clip
+            return None
+
+        window = np.concatenate(list(self._genai_window))
+
+        # Silence gate, using the same threshold audio detection uses. Gate the
+        # whole window rather than individual chunks; this is the primary cost
+        # and privacy brake and is what keeps a quiet camera near zero requests.
+        window_as_float = window.astype(np.float32)
+        rms = float(np.sqrt(np.mean(np.absolute(np.square(window_as_float)))))
+
+        if rms < self.camera_config.audio.min_volume:
+            logger.debug(
+                f"Window RMS {rms:.1f} below min_volume, skipping transcription"
+            )
+            return self.__end_genai_utterance()
+
+        text = client.transcribe(
+            pcm16_to_wav(window),
+            language=resolve_language(self.config.audio_transcription.language),
+        )
+
+        # cleaning has to come first: a silent window often comes back as the
+        # model's preamble alone, which is silence, not a word to commit
+        cleaned = clean_transcript(text)
+
+        if not cleaned:
+            return self.__end_genai_utterance()
+
+        self._genai_committed = stitch_transcripts(self._genai_committed, cleaned)
+
+        # no VAD on this path, so mirror the whisper branch's heuristic endpoint
+        is_endpoint = (
+            self._genai_committed.endswith((".", "!", "?"))
+            and len(self._genai_committed) > 300
+        )
+
+        logger.debug(f"GenAI transcription: '{self._genai_committed}'")
+
+        return self._genai_committed, is_endpoint
+
+    def __end_genai_utterance(self) -> tuple[str, bool] | None:
+        """Close out the current utterance when a window carries no speech."""
+        if not self._genai_committed:
+            return None
+
+        return self._genai_committed, True
+
     def process_frame(self, obj_data: dict[str, Any], frame: np.ndarray) -> None:
         pass
 
@@ -148,8 +283,38 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
             logger.debug("No audio data provided for transcription")
             return None
 
-        # enqueue audio data for processing in the thread
-        self.audio_queue.put((obj_data, audio))
+        # enqueue audio data for processing in the thread. never block: the
+        # producer is the ffmpeg read thread that audio detection depends on,
+        # so on a backlog drop the oldest chunk instead.
+        try:
+            self.audio_queue.put_nowait((obj_data, audio))
+        except queue.Full:
+            try:
+                self.audio_queue.get_nowait()
+                self.audio_queue.task_done()
+            except queue.Empty:
+                pass
+
+            # the stream now has a hole in it, which the consumer has to know
+            # about before it splices the next chunk onto what it already holds
+            self._audio_dropped.set()
+
+            now = time.monotonic()
+
+            if now - self._last_drop_warning >= AUDIO_DROP_WARN_INTERVAL:
+                self._last_drop_warning = now
+                logger.warning(
+                    "Audio transcription queue for %s is full, dropping audio. The "
+                    "provider is not keeping up with the %.2fs chunk rate",
+                    self.camera_config.name,
+                    AUDIO_DURATION,
+                )
+
+            try:
+                self.audio_queue.put_nowait((obj_data, audio))
+            except queue.Full:
+                pass
+
         return None
 
     def run(self) -> None:
@@ -205,6 +370,14 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
                 break
 
     def reset(self) -> None:
+        if self._use_genai:
+            self._genai_committed = ""
+            # stale audio carried across an utterance boundary would be
+            # re-transcribed into the next one
+            self._genai_window.clear()
+            logger.debug("Stream reset")
+            return
+
         if self.config.audio_transcription.model_size == "large":
             # get final output from whisper
             output = self.stream.finish()
@@ -218,7 +391,7 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
             # reset whisper
             self.stream.init()
             self.transcription_segments = []
-        else:
+        elif self.model_runner is not None:
             # reset sherpa
             self.model_runner.model.reset(self.stream)
 
@@ -226,6 +399,24 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
 
     def check_unload_model(self) -> None:
         # regularly called in the loop in audio maintainer
+        if self._use_genai:
+            # no model to unload, but this is the hook that fires when
+            # live_enabled flips off. guard on emptiness: called ~1x/s per camera.
+            if self._genai_committed or self._genai_window:
+                logger.debug(
+                    f"Clearing GenAI transcription state for {self.camera_config.name}"
+                )
+                self.clear_audio_queue()
+                self._genai_committed = ""
+                self._genai_window.clear()
+
+                self.requestor.send_data(
+                    f"{self.camera_config.name}/audio/transcription",
+                    "",
+                )
+
+            return
+
         if (
             self.config.audio_transcription.model_size == "large"
             and self.whisper_model is not None
@@ -270,6 +461,10 @@ class AudioTranscriptionRealTimeProcessor(RealTimeProcessorApi):
         self, topic: str, request_data: dict[str, Any]
     ) -> dict[str, Any] | None:
         if topic == "clear_audio_recognizer":
+            if self._use_genai:
+                self.reset()
+                return {"message": "Audio transcription state cleared", "success": True}
+
             self.stream = None
             self.__build_recognizer()
             return {"message": "Audio recognizer cleared and rebuilt", "success": True}

frigate/data_processing/real_time/custom_classification.py

@@ -91,8 +91,23 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
         self.tensor_input_details = self.interpreter.get_input_details()
         self.tensor_output_details = self.interpreter.get_output_details()
         self.labelmap = load_labels(labelmap_path, prefill=0, indexed=False)
+        self._forget_unknown_states()
         self.classifications_per_second.start()
 
+    def _forget_unknown_states(self) -> None:
+        """Drop verified states that are not labels of the loaded model.
+
+        A retrained model can rename or remove labels. Keeping a state it can
+        no longer produce would report its first verified state as a change
+        from that obsolete label.
+        """
+        labels = set(self.labelmap.values())
+        self.state_history = {
+            camera: history
+            for camera, history in self.state_history.items()
+            if history["current_state"] in labels
+        }
+
     def __update_metrics(self, duration: float) -> None:
         self.classifications_per_second.update()
         if self.inference_speed:
@@ -134,15 +149,20 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
         # Don't save if state is stable (detected_state == current_state) AND score is 100%
         return False
 
-    def verify_state_change(self, camera: str, detected_state: str) -> str | None:
+    def verify_state_change(
+        self, camera: str, detected_state: str, timestamp: float
+    ) -> tuple[str | None, float] | None:
         """
         Verify state change requires 3 consecutive identical states before publishing.
-        Returns state to publish or None if verification not complete.
+        Returns (previous state, time the new state was first seen) once verified,
+        or None if verification not complete. The previous state is None for the
+        first state verified on a camera.
         """
         if camera not in self.state_history:
             self.state_history[camera] = {
                 "current_state": None,
                 "pending_state": None,
+                "pending_since": 0.0,
                 "consecutive_count": 0,
             }
 
@@ -157,12 +177,14 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
             verification["consecutive_count"] += 1
 
             if verification["consecutive_count"] >= 3:
+                previous_state = verification["current_state"]
                 verification["current_state"] = detected_state
                 verification["pending_state"] = None
                 verification["consecutive_count"] = 0
-                return detected_state
+                return previous_state, verification["pending_since"]
         else:
             verification["pending_state"] = detected_state
+            verification["pending_since"] = timestamp
             verification["consecutive_count"] = 1
             logger.debug(
                 f"New state '{detected_state}' detected for {camera}, need {3 - verification['consecutive_count']} more consecutive detections"
@@ -340,16 +362,19 @@ class CustomStateClassificationProcessor(DeferredRealtimeProcessorApi):
             )
             return
 
-        verified_state = self.verify_state_change(camera, detected_state)
+        verified = self.verify_state_change(camera, detected_state, timestamp)
 
-        if verified_state is not None:
+        if verified is not None:
+            previous_state, changed_at = verified
             self._emit_result(
                 {
                     "type": "classification",
                     "processor": "state",
                     "model_name": self.model_config.name,
                     "camera": camera,
-                    "state": verified_state,
+                    "state": detected_state,
+                    "previous_state": previous_state,
+                    "timestamp": changed_at,
                 }
             )
 

frigate/db/sqlitevecq.py

@@ -1,8 +1,10 @@
 import logging
 import sqlite3
+import threading
 from typing import Any
 
 import regex
+from peewee import DatabaseError
 from playhouse.sqliteq import SqliteQueueDatabase
 
 logger = logging.getLogger(__name__)
@@ -17,6 +19,7 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
         self.load_vec_extension: bool = load_vec_extension
         # no extension necessary, sqlite will load correctly for each platform
         self.sqlite_vec_path = "/usr/local/lib/vec0"
+        self.upsert_lock = threading.Lock()
         super().__init__(*args, **kwargs)
 
     def _connect(self, *args: Any, **kwargs: Any) -> sqlite3.Connection:
@@ -53,6 +56,22 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
 
         conn.create_function("REGEXP", 2, regexp)
 
+    def execute_write(self, sql: str, params: Any = None) -> None:
+        """Run a write and wait for it, so that failures are raised here.
+
+        SqliteQueueDatabase hands non-SELECT statements to a writer thread and
+        stores any exception on the cursor it returns, so callers that ignore
+        that cursor never learn the write failed.
+        """
+        self.execute_sql(sql, params).fetchall()
+
+    def _table_exists(self, table: str) -> bool:
+        cursor = self.execute_sql(
+            "SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?",
+            (table,),
+        )
+        return cursor.fetchone() is not None
+
     def _delete_embeddings(self, table: str, event_ids: list[str]) -> None:
         """Delete embeddings for the given events, if the table exists.
 
@@ -63,17 +82,17 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
             return
 
         # the embeddings tables are only created once semantic search has run
-        cursor = self.execute_sql(
-            "SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?",
-            (table,),
-        )
-
-        if cursor.fetchone() is None:
+        if not self._table_exists(table):
             logger.debug("Skipping %s cleanup, table does not exist", table)
             return
 
         ids = ",".join(["?" for _ in event_ids])
-        self.execute_sql(f"DELETE FROM {table} WHERE id IN ({ids})", event_ids)
+
+        # callers treat cleanup as best effort, so log rather than propagate
+        try:
+            self.execute_write(f"DELETE FROM {table} WHERE id IN ({ids})", event_ids)
+        except DatabaseError:
+            logger.exception("Failed to delete embeddings from %s", table)
 
     def delete_embeddings_thumbnail(self, event_ids: list[str]) -> None:
         self._delete_embeddings("vec_thumbnails", event_ids)
@@ -81,25 +100,67 @@ class SqliteVecQueueDatabase(SqliteQueueDatabase):
     def delete_embeddings_description(self, event_ids: list[str]) -> None:
         self._delete_embeddings("vec_descriptions", event_ids)
 
+    def _restore_vec_info_table(self, table: str) -> None:
+        """Recreate the _info shadow table a legacy vec0 table is missing.
+
+        sqlite-vec added _info in 0.1.6 and drops it unconditionally when a
+        table is destroyed, so tables written by Frigate 0.17 and earlier fail
+        to drop. An empty stub is enough, and leaving it unseeded keeps the
+        table reading as pre-0.1.10 if the drop does not follow.
+        """
+        if not self._table_exists(table) or self._table_exists(f"{table}_info"):
+            return
+
+        logger.debug("Restoring the %s_info shadow table before dropping", table)
+        self.execute_write(
+            f'CREATE TABLE "{table}_info" (key TEXT PRIMARY KEY, value ANY)'
+        )
+
     def drop_embeddings_tables(self) -> None:
-        self.execute_sql("""
-            DROP TABLE vec_descriptions;
-        """)
-        self.execute_sql("""
-            DROP TABLE vec_thumbnails;
-        """)
+        for table in ("vec_descriptions", "vec_thumbnails"):
+            self._restore_vec_info_table(table)
+            self.execute_write(f"DROP TABLE IF EXISTS {table}")
 
     def create_embeddings_tables(self) -> None:
         """Create vec0 virtual table for embeddings"""
-        self.execute_sql("""
+        self.execute_write("""
             CREATE VIRTUAL TABLE IF NOT EXISTS vec_thumbnails USING vec0(
                 id TEXT PRIMARY KEY,
                 thumbnail_embedding FLOAT[768] distance_metric=cosine
             );
         """)
-        self.execute_sql("""
+        self.execute_write("""
             CREATE VIRTUAL TABLE IF NOT EXISTS vec_descriptions USING vec0(
                 id TEXT PRIMARY KEY,
                 description_embedding FLOAT[768] distance_metric=cosine
             );
         """)
+
+    def upsert_embeddings(
+        self, table: str, column: str, embeddings: dict[str, bytes]
+    ) -> None:
+        """Write embeddings for the given event ids, replacing any that exist.
+
+        vec0 implements neither REPLACE nor UPSERT, so rows that are already
+        there have to be deleted first.
+        """
+        if not embeddings:
+            return
+
+        event_ids = list(embeddings.keys())
+        ids = ",".join(["?" for _ in event_ids])
+        params: list[Any] = []
+
+        for event_id in event_ids:
+            params.extend((event_id, embeddings[event_id]))
+
+        values = ", ".join(["(?, ?)"] * len(event_ids))
+
+        # reindexing and live embedding run on separate threads, and each write
+        # is queued separately, so the delete and the insert have to be held
+        # together or an interleaved pair fails on the vec0 primary key
+        with self.upsert_lock:
+            self.execute_write(f"DELETE FROM {table} WHERE id IN ({ids})", event_ids)
+            self.execute_write(
+                f"INSERT INTO {table}(id, {column}) VALUES {values}", params
+            )

frigate/detectors/detection_runners.py

@@ -46,8 +46,42 @@ _PROVIDER_LABELS = {
     "MIGraphXExecutionProvider": "MIGraphX",
     "OpenVINOExecutionProvider": "OpenVINO",
     "CPUExecutionProvider": "CPU",
+    "LighterANE": "Neural Engine",
 }
 
+# lighter (https://github.com/fieldwork-ai/lighter) places an ONNX Runtime plugin
+# execution provider in a container started with --device lighter.sh/ane=all,
+# which runs models on a Mac's Neural Engine; LIGHTER_ANE_EP names where it is
+LIGHTER_ANE_EP_NAME = "LighterANE"
+LIGHTER_ANE_LIBRARY = "/usr/lib/lighter/liblighter_ane_ep.so"
+
+
+def get_lighter_ane_devices() -> list[Any]:
+    """Get the Neural Engine devices lighter's provider offers, registering it once.
+
+    Returns:
+        The provider's ONNX Runtime devices, or an empty list without lighter's device
+    """
+    library = os.environ.get("LIGHTER_ANE_EP", LIGHTER_ANE_LIBRARY)
+
+    if not os.path.exists(library):
+        return []
+
+    devices = [d for d in ort.get_ep_devices() if d.ep_name == LIGHTER_ANE_EP_NAME]
+
+    if not devices:
+        try:
+            ort.register_execution_provider_library(LIGHTER_ANE_EP_NAME, library)
+        except Exception as e:
+            logger.warning(
+                f"Failed to load the Neural Engine provider from {library}: {e}"
+            )
+            return []
+
+        devices = [d for d in ort.get_ep_devices() if d.ep_name == LIGHTER_ANE_EP_NAME]
+
+    return devices
+
 
 def is_arm64_platform() -> bool:
     """Check if we're running on an ARM platform."""
@@ -247,15 +281,20 @@ class CudaGraphRunner(BaseModelRunner):
             EnrichmentModelTypeEnum.yolov9_license_plate.value,
         ]
 
+    # ORT performs two regular runs before it starts capturing, but on some
+    # driver / cuDNN combinations the arena still has to extend on the run that
+    # captures, and cudaMalloc is not allowed during capture. Running with
+    # capture disabled first keeps those allocations outside of the capture.
+    GRAPH_FREE_WARMUP_RUNS = 2
+
     def __init__(self, session: ort.InferenceSession, cuda_device_id: int):
         self._session = session
         self._cuda_device_id = cuda_device_id
-        self._captured = False
+        self._prepared = False
         self._io_binding: ort.IOBinding | None = None
         self._input_name: str | None = None
         self._output_names: list[str] | None = None
         self._input_ortvalue: ort.OrtValue | None = None
-        self._output_ortvalues: ort.OrtValue | None = None
 
     def get_input_names(self) -> list[str]:
         """Get input names for the model."""
@@ -265,35 +304,41 @@ class CudaGraphRunner(BaseModelRunner):
         """Get the input width of the model."""
         return self._session.get_inputs()[0].shape[3]
 
+    def _prepare(self, input_name: str, tensor_input: np.ndarray) -> None:
+        """Bind CUDA buffers and warm the session up with capture disabled."""
+        self._io_binding = self._session.io_binding()
+        self._input_name = input_name
+        self._output_names = [o.name for o in self._session.get_outputs()]
+
+        self._input_ortvalue = ort.OrtValue.ortvalue_from_numpy(
+            tensor_input, "cuda", self._cuda_device_id
+        )
+        self._io_binding.bind_ortvalue_input(self._input_name, self._input_ortvalue)
+
+        for name in self._output_names:
+            # Bind outputs to CUDA and allow ORT to allocate appropriately
+            self._io_binding.bind_output(name, "cuda", self._cuda_device_id)
+
+        # gpu_graph_id -1 disables capture and replay for the run
+        warmup_options = ort.RunOptions()
+        warmup_options.add_run_config_entry("gpu_graph_id", "-1")
+
+        for _ in range(self.GRAPH_FREE_WARMUP_RUNS):
+            self._session.run_with_iobinding(self._io_binding, warmup_options)
+
+        self._prepared = True
+
     def run(self, input: dict[str, Any]):
         # Extract the single tensor input (assuming one input)
         input_name = list(input.keys())[0]
-        tensor_input = input[input_name]
-        tensor_input = np.ascontiguousarray(tensor_input)
+        tensor_input = np.ascontiguousarray(input[input_name])
 
-        if not self._captured:
-            # Prepare IOBinding with CUDA buffers and let ORT allocate outputs on device
-            self._io_binding = self._session.io_binding()
-            self._input_name = input_name
-            self._output_names = [o.name for o in self._session.get_outputs()]
+        if not self._prepared:
+            self._prepare(input_name, tensor_input)
+        else:
+            # Replay using updated input
+            self._input_ortvalue.update_inplace(tensor_input)
 
-            self._input_ortvalue = ort.OrtValue.ortvalue_from_numpy(
-                tensor_input, "cuda", self._cuda_device_id
-            )
-            self._io_binding.bind_ortvalue_input(self._input_name, self._input_ortvalue)
-
-            for name in self._output_names:
-                # Bind outputs to CUDA and allow ORT to allocate appropriately
-                self._io_binding.bind_output(name, "cuda", self._cuda_device_id)
-
-            # First IOBinding run to allocate, execute, and capture CUDA Graph
-            ro = ort.RunOptions()
-            self._session.run_with_iobinding(self._io_binding, ro)
-            self._captured = True
-            return self._io_binding.copy_outputs_to_cpu()
-
-        # Replay using updated input, copy results to CPU
-        self._input_ortvalue.update_inplace(tensor_input)
         ro = ort.RunOptions()
         self._session.run_with_iobinding(self._io_binding, ro)
         return self._io_binding.copy_outputs_to_cpu()
@@ -678,6 +723,23 @@ def get_optimized_runner(
         if rknn_path:
             return _record_runner(model_path, model_type, RKNNModelRunner(rknn_path))
 
+    if device != "CPU" and (ane_devices := get_lighter_ane_devices()):
+        sess_options = get_ort_session_options(model_type) or ort.SessionOptions()
+        sess_options.add_provider_for_devices(ane_devices, {})
+
+        try:
+            session = ort.InferenceSession(model_path, sess_options=sess_options)
+        except Exception as e:
+            logger.warning(
+                f"Failed to load {model_path} on the Neural Engine, using the default providers: {e}"
+            )
+        else:
+            return _record_runner(
+                model_path,
+                model_type,
+                ONNXModelRunner(session, model_type=model_type),
+            )
+
     providers, options = get_ort_providers(device == "CPU", device, **kwargs)
 
     if providers[0] == "CPUExecutionProvider":

frigate/detectors/detector_config.py

@@ -3,14 +3,14 @@ import json
 import logging
 import os
 from enum import Enum
-from typing import Any, ClassVar
+from typing import ClassVar
 
 import requests
 from pydantic import BaseModel, ConfigDict, Field
 from pydantic.fields import PrivateAttr
 
 from frigate.const import DEFAULT_ATTRIBUTE_LABEL_MAP, MODEL_CACHE_DIR
-from frigate.plus import PlusApi
+from frigate.plus import PlusApi, load_plus_model_info
 from frigate.util.builtin import generate_color_palette, load_labels
 
 logger = logging.getLogger(__name__)
@@ -47,21 +47,17 @@ class ModelTypeEnum(str, Enum):
     yologeneric = "yolo-generic"
 
 
-class SceneEnum(str, Enum):
-    """The camera environment a detection model is intended for."""
-
-    all = "all"
-    indoor = "indoor"
-    outdoor = "outdoor"
-    indoor_thermal = "indoor_thermal"
-    outdoor_thermal = "outdoor_thermal"
+# the scene of the model used by cameras that don't name one
+DEFAULT_SCENE = "default"
+SCENE_PATTERN = r"^[A-Za-z0-9_-]+$"
 
 
 class ModelConfig(BaseModel):
-    scene: SceneEnum = Field(
-        default=SceneEnum.all,
+    scene: str = Field(
+        default=DEFAULT_SCENE,
+        pattern=SCENE_PATTERN,
         title="Model scene",
-        description="The camera environment this model is used for. Cameras select a model by setting detect.scene to a matching value, and 'all' is used by any camera that does not set one.",
+        description="A name for the camera environment this model is used for, such as 'thermal'. Cameras select a model by setting detect.scene to a matching value, and the 'default' model is used by any camera that does not set one.",
     )
     devices: list[str] = Field(
         default_factory=list,
@@ -123,6 +119,7 @@ class ModelConfig(BaseModel):
     _all_attributes: list[str] = PrivateAttr()
     _all_attribute_logos: list[str] = PrivateAttr()
     _model_hash: str = PrivateAttr()
+    _plus_id: str | None = PrivateAttr(default=None)
 
     @property
     def merged_labelmap(self) -> dict[int, str]:
@@ -148,6 +145,11 @@ class ModelConfig(BaseModel):
     def model_hash(self) -> str:
         return self._model_hash
 
+    @property
+    def plus_id(self) -> str | None:
+        """The Frigate+ model id, once a plus:// path has been resolved."""
+        return self._plus_id
+
     def __init__(self, **config):
         super().__init__(**config)
 
@@ -178,6 +180,7 @@ class ModelConfig(BaseModel):
         os.makedirs(MODEL_CACHE_DIR, exist_ok=True)
 
         model_id = self.path[7:]
+        self._plus_id = model_id
         self.path = os.path.join(MODEL_CACHE_DIR, model_id)
         model_info_path = f"{self.path}.json"
 
@@ -190,12 +193,13 @@ class ModelConfig(BaseModel):
 
         # download the model info if it doesn't exist
         if not os.path.isfile(model_info_path):
-            model_info = plus_api.get_model_info(model_id)
             with open(model_info_path, "w") as f:
-                json.dump(model_info, f)
-        else:
-            with open(model_info_path) as f:
-                model_info: dict[str, Any] = json.load(f)
+                json.dump(plus_api.get_model_info(model_id), f)
+
+        model_info = load_plus_model_info(model_id)
+
+        if model_info is None:
+            raise ValueError(f"Unable to read the model info for {model_id}")
 
         if detector and detector not in model_info["supportedDetectors"]:
             raise ValueError(f"Model does not support detector type of {detector}")

frigate/detectors/hardware.py

@@ -25,6 +25,7 @@ SYS_ROOT = "/sys"
 DEV_ROOT = "/dev"
 PROC_ROOT = "/proc"
 ETC_ROOT = "/etc"
+LIB_ROOT = "/usr/lib"
 
 # a Coral reports as Global Unichip until its firmware is loaded, then as Google
 CORAL_USB_IDS = {("1a6e", "089a"), ("18d1", "9302")}
@@ -259,8 +260,8 @@ def detect_hailo() -> DetectionHardware | None:
 
     # the hailo runtime schedules across every attached device itself, so there
     # is nothing to address individually
-    units = [HardwareUnit(device="hailo8l:PCIe", label=os.path.basename(nodes[0]))]
-    return _hardware("hailo8l", "hailo8l", "Hailo", units)
+    units = [HardwareUnit(device="hailo:PCIe", label=os.path.basename(nodes[0]))]
+    return _hardware("hailo", "hailo", "Hailo", units)
 
 
 def detect_memryx() -> DetectionHardware | None:
@@ -273,6 +274,16 @@ def detect_memryx() -> DetectionHardware | None:
     return _hardware("memryx", "memryx", "MemryX MX3", units)
 
 
+def detect_deepx() -> DetectionHardware | None:
+    """Find DEEPX NPUs by their device nodes."""
+    units = _dev_units("dxrt*", "deepx:PCIe:{index}", "PCIe")
+
+    if not units:
+        return None
+
+    return _hardware("deepx", "deepx", "DEEPX NPU", units)
+
+
 def detect_rockchip() -> DetectionHardware | None:
     """Find a Rockchip NPU by reading the SoC from the device tree."""
     compatible = _read(f"{PROC_ROOT}/device-tree/compatible")
@@ -307,6 +318,19 @@ def detect_synaptics() -> DetectionHardware | None:
     return _hardware("synaptics", "synaptics", "Synaptics NPU", units)
 
 
+def detect_lighter_ane() -> DetectionHardware | None:
+    """Find a Mac's Neural Engine by the provider library lighter's device places."""
+    library = os.environ.get(
+        "LIGHTER_ANE_EP", f"{LIB_ROOT}/lighter/liblighter_ane_ep.so"
+    )
+    if not os.path.exists(library):
+        return None
+
+    # runs through onnx, whose session picks lighter's provider when it is present
+    units = [HardwareUnit(device="onnx", label="Neural Engine")]
+    return _hardware("onnx:lighter", "onnx", "Apple Neural Engine", units)
+
+
 def detect_cpu() -> DetectionHardware:
     """The CPU, which is always available."""
     units = [HardwareUnit(device="cpu", label="CPU")]
@@ -319,6 +343,7 @@ PROBES = (
     detect_coral_usb,
     detect_hailo,
     detect_memryx,
+    detect_deepx,
     detect_intel_npu,
     detect_intel_gpu,
     detect_nvidia_gpu,
@@ -327,6 +352,7 @@ PROBES = (
     detect_rockchip,
     detect_axengine,
     detect_synaptics,
+    detect_lighter_ane,
     detect_cpu,
 )
 

frigate/detectors/plugins/deepstack.py

@@ -1,105 +0,0 @@
-import io
-import logging
-from typing import Literal
-
-import numpy as np
-import requests
-from PIL import Image
-from pydantic import ConfigDict, Field
-
-from frigate.detectors.detection_api import DetectionApi
-from frigate.detectors.detector_config import BaseDetectorConfig
-
-logger = logging.getLogger(__name__)
-
-DETECTOR_KEY = "deepstack"
-
-
-class DeepstackDetectorConfig(BaseDetectorConfig):
-    """DeepStack/CodeProject.AI detector that sends images to a remote DeepStack HTTP API for inference. Not recommended."""
-
-    model_config = ConfigDict(
-        title="DeepStack",
-    )
-
-    type: Literal[DETECTOR_KEY]
-    api_url: str = Field(
-        default="http://localhost:80/v1/vision/detection",
-        title="DeepStack API URL",
-        description="The URL of the DeepStack API.",
-    )
-    api_timeout: float = Field(
-        default=0.1,
-        title="DeepStack API timeout (in seconds)",
-        description="Maximum time allowed for a DeepStack API request.",
-    )
-    api_key: str = Field(
-        default="",
-        title="DeepStack API key (if required)",
-        description="Optional API key for authenticated DeepStack services.",
-    )
-
-
-class DeepStack(DetectionApi):
-    type_key = DETECTOR_KEY
-
-    def __init__(self, detector_config: DeepstackDetectorConfig):
-        self.api_url = detector_config.api_url
-        self.api_timeout = detector_config.api_timeout
-        self.api_key = detector_config.api_key
-        self.labels = detector_config.model.merged_labelmap
-        self.session = requests.Session()
-
-    def get_label_index(self, label_value):
-        if label_value.lower() == "truck":
-            label_value = "car"
-        for index, value in self.labels.items():
-            if value == label_value.lower():
-                return index
-        return -1
-
-    def detect_raw(self, tensor_input):
-        image_data = np.squeeze(tensor_input).astype(np.uint8)
-        image = Image.fromarray(image_data)
-        self.w, self.h = image.size
-        with io.BytesIO() as output:
-            image.save(output, format="JPEG")
-            image_bytes = output.getvalue()
-        data = {"api_key": self.api_key}
-
-        try:
-            response = self.session.post(
-                self.api_url,
-                data=data,
-                files={"image": image_bytes},
-                timeout=self.api_timeout,
-            )
-        except requests.exceptions.RequestException as ex:
-            logger.error("Error calling deepstack API: %s", ex)
-            return np.zeros((20, 6), np.float32)
-
-        response_json = response.json()
-        detections = np.zeros((20, 6), np.float32)
-        if response_json.get("predictions") is None:
-            logger.debug(f"Error in parsing response json: {response_json}")
-            return detections
-
-        for i, detection in enumerate(response_json.get("predictions")):
-            logger.debug(f"Response: {detection}")
-            if detection["confidence"] < 0.4:
-                logger.debug("Break due to confidence < 0.4")
-                break
-            label = self.get_label_index(detection["label"])
-            if label < 0:
-                logger.debug("Break due to unknown label")
-                break
-            detections[i] = [
-                label,
-                float(detection["confidence"]),
-                detection["y_min"] / self.h,
-                detection["x_min"] / self.w,
-                detection["y_max"] / self.h,
-                detection["x_max"] / self.w,
-            ]
-
-        return detections

frigate/detectors/plugins/hailo.py

@@ -53,7 +53,7 @@ def preprocess_tensor(image: np.ndarray, model_w: int, model_h: int) -> np.ndarr
 
 
 # ----------------- Global Constants ----------------- #
-DETECTOR_KEY = "hailo8l"
+DETECTOR_KEY = "hailo"
 ARCH = None
 H8_DEFAULT_MODEL = "yolov6n.hef"
 H8L_DEFAULT_MODEL = "yolov6n.hef"
@@ -469,10 +469,10 @@ class HailoDetector(DetectionApi):
 
 # ----------------- HailoDetectorConfig Class ----------------- #
 class HailoDetectorConfig(BaseDetectorConfig):
-    """Hailo-8/Hailo-8L detector using HEF models and the HailoRT SDK for inference on Hailo hardware."""
+    """Hailo detector using HEF models and the HailoRT SDK for inference on Hailo hardware."""
 
     model_config = ConfigDict(
-        title="Hailo-8/Hailo-8L",
+        title="Hailo",
     )
 
     type: Literal[DETECTOR_KEY]

frigate/detectors/plugins/zmq_ipc.py

@@ -1,7 +1,7 @@
 import json
 import logging
 import os
-from typing import Any, Literal
+from typing import Any, ClassVar, Literal
 
 import numpy as np
 import zmq
@@ -21,6 +21,7 @@ class ZmqDetectorConfig(BaseDetectorConfig):
     model_config = ConfigDict(
         title="ZMQ IPC",
     )
+    device_spec_field: ClassVar[str] = "endpoint"
 
     type: Literal[DETECTOR_KEY]
     endpoint: str = Field(

frigate/embeddings/embeddings.py

@@ -6,9 +6,10 @@ import logging
 import os
 import threading
 import time
+from typing import Any
 
 import numpy as np
-from peewee import DoesNotExist, IntegrityError
+from peewee import DatabaseError, DoesNotExist, IntegrityError
 from PIL import Image
 from playhouse.shortcuts import model_to_dict
 
@@ -207,12 +208,10 @@ class Embeddings:
         embedding = self.vision_embedding([thumbnail])[0]
 
         if upsert:
-            self.db.execute_sql(
-                """
-                INSERT OR REPLACE INTO vec_thumbnails(id, thumbnail_embedding)
-                VALUES(?, ?)
-                """,
-                (event_id, serialize(embedding)),
+            self.db.upsert_embeddings(
+                "vec_thumbnails",
+                "thumbnail_embedding",
+                {event_id: serialize(embedding)},
             )
 
         self.image_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -251,19 +250,12 @@ class Embeddings:
         embeddings = self.vision_embedding(valid_thumbs)
 
         if upsert:
-            items = []
+            items = {}
             for i in range(len(valid_ids)):
-                items.append(valid_ids[i])
-                items.append(serialize(embeddings[i]))
+                items[valid_ids[i]] = serialize(embeddings[i])
                 self.image_eps.update()
 
-            self.db.execute_sql(
-                """
-                INSERT OR REPLACE INTO vec_thumbnails(id, thumbnail_embedding)
-                VALUES {}
-                """.format(", ".join(["(?, ?)"] * len(valid_ids))),
-                items,
-            )
+            self.db.upsert_embeddings("vec_thumbnails", "thumbnail_embedding", items)
 
         duration = datetime.datetime.now().timestamp() - start
         self.image_inference_speed.update(duration / len(valid_ids))
@@ -277,12 +269,10 @@ class Embeddings:
         embedding = self.text_embedding([description])[0]
 
         if upsert:
-            self.db.execute_sql(
-                """
-                INSERT OR REPLACE INTO vec_descriptions(id, description_embedding)
-                VALUES(?, ?)
-                """,
-                (event_id, serialize(embedding)),
+            self.db.upsert_embeddings(
+                "vec_descriptions",
+                "description_embedding",
+                {event_id: serialize(embedding)},
             )
 
         self.text_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -302,19 +292,14 @@ class Embeddings:
 
         if upsert:
             ids = list(event_descriptions.keys())
-            items = []
+            items = {}
 
             for i in range(len(ids)):
-                items.append(ids[i])
-                items.append(serialize(embeddings[i]))
+                items[ids[i]] = serialize(embeddings[i])
                 self.text_eps.update()
 
-            self.db.execute_sql(
-                """
-                INSERT OR REPLACE INTO vec_descriptions(id, description_embedding)
-                VALUES {}
-                """.format(", ".join(["(?, ?)"] * len(ids))),
-                items,
+            self.db.upsert_embeddings(
+                "vec_descriptions", "description_embedding", items
             )
 
         self.text_inference_speed.update(datetime.datetime.now().timestamp() - start)
@@ -322,6 +307,17 @@ class Embeddings:
         return embeddings
 
     def reindex(self) -> None:
+        """Rebuild every tracked object embedding from scratch."""
+        totals: dict[str, Any] = {"status": "indexing"}
+
+        try:
+            self._reindex(totals)
+        except DatabaseError:
+            logger.exception("Unable to reindex tracked object embeddings")
+            totals["status"] = "failed"
+            self.requestor.send_data(UPDATE_EMBEDDINGS_REINDEX_PROGRESS, totals)
+
+    def _reindex(self, totals: dict[str, Any]) -> None:
         logger.info("Indexing tracked object embeddings...")
 
         self.db.drop_embeddings_tables()
@@ -346,17 +342,24 @@ class Embeddings:
             batch_size = 32
         current_page = 1
 
-        totals = {
-            "thumbnails": 0,
-            "descriptions": 0,
-            "processed_objects": total_events - 1 if total_events < batch_size else 0,
-            "total_objects": total_events,
-            "time_remaining": 0 if total_events < batch_size else -1,
-            "status": "indexing",
-        }
+        totals.update(
+            {
+                "thumbnails": 0,
+                "descriptions": 0,
+                "processed_objects": total_events - 1
+                if total_events < batch_size
+                else 0,
+                "total_objects": total_events,
+                "time_remaining": 0 if total_events < batch_size else -1,
+                "status": "indexing",
+            }
+        )
 
         self.requestor.send_data(UPDATE_EMBEDDINGS_REINDEX_PROGRESS, totals)
 
+        # a single batch sends no progress, so the first message shows it nearly done
+        totals["processed_objects"] = 0
+
         events = (
             Event.select()
             .order_by(Event.start_time.desc())

frigate/embeddings/maintainer.py

@@ -11,7 +11,11 @@ from typing import Any
 from peewee import DoesNotExist
 
 from frigate.comms.config_updater import ConfigSubscriber
-from frigate.comms.detections_updater import DetectionSubscriber, DetectionTypeEnum
+from frigate.comms.detections_updater import (
+    DetectionPublisher,
+    DetectionSubscriber,
+    DetectionTypeEnum,
+)
 from frigate.comms.embeddings_updater import (
     EmbeddingsRequestEnum,
     EmbeddingsResponder,
@@ -68,7 +72,7 @@ from frigate.events.types import (
     RegenerateDescriptionEnum,
 )
 from frigate.genai import GenAIClientManager
-from frigate.models import Event, Recordings, ReviewSegment, Trigger
+from frigate.models import Event, Recordings, ReviewSegment, Timeline, Trigger
 from frigate.types import TrackedObjectUpdateTypesEnum
 from frigate.util.builtin import serialize
 from frigate.util.file import get_event_thumbnail_bytes
@@ -139,7 +143,7 @@ class EmbeddingMaintainer(threading.Thread):
             ),
             load_vec_extension=True,
         )
-        models = [Event, Recordings, ReviewSegment, Trigger]
+        models = [Event, Recordings, ReviewSegment, Timeline, Trigger]
         db.bind(models)
 
         self.genai_manager = GenAIClientManager(config)
@@ -168,6 +172,7 @@ class EmbeddingMaintainer(threading.Thread):
         )
         self.review_subscriber = ReviewDataSubscriber("")
         self.detection_subscriber = DetectionSubscriber(DetectionTypeEnum.video.value)
+        self.detection_publisher = DetectionPublisher(DetectionTypeEnum.all.value)
         self.embeddings_responder = EmbeddingsResponder()
         self.frame_manager = SharedMemoryFrameManager()
 
@@ -251,7 +256,11 @@ class EmbeddingMaintainer(threading.Thread):
         ):
             self.post_processors.append(
                 AudioTranscriptionPostProcessor(
-                    self.config, self.requestor, self.embeddings, metrics
+                    self.config,
+                    self.requestor,
+                    self.embeddings,
+                    metrics,
+                    self.genai_manager,
                 )
             )
 
@@ -352,6 +361,7 @@ class EmbeddingMaintainer(threading.Thread):
         self.event_end_subscriber.stop()
         self.recordings_subscriber.stop()
         self.detection_subscriber.stop()
+        self.detection_publisher.stop()
         self.event_metadata_publisher.stop()
         self.event_metadata_subscriber.stop()
         self.embeddings_responder.stop()
@@ -847,6 +857,21 @@ class EmbeddingMaintainer(threading.Thread):
                         f"{result['camera']}/classification/{result['model_name']}",
                         result["state"],
                     )
+
+                    # the first state verified after startup is not a change
+                    if result["previous_state"] is not None:
+                        self.detection_publisher.publish(
+                            (
+                                result["camera"],
+                                {
+                                    "model": result["model_name"],
+                                    "from": result["previous_state"],
+                                    "to": result["state"],
+                                    "timestamp": result["timestamp"],
+                                },
+                            ),
+                            DetectionTypeEnum.classification_state.value,
+                        )
                 elif result["processor"] == "object":
                     object_id = result["object_id"]
                     camera = result["camera"]

frigate/events/audio.py

@@ -11,6 +11,7 @@ from typing import Any
 
 import numpy as np
 
+from frigate.camera import CameraMetrics
 from frigate.comms.detections_updater import DetectionPublisher, DetectionTypeEnum
 from frigate.comms.inter_process import InterProcessRequestor
 from frigate.config import CameraConfig, CameraInput, FrigateConfig
@@ -19,6 +20,7 @@ from frigate.config.camera.updater import (
     CameraConfigUpdateEnum,
     CameraConfigUpdateSubscriber,
 )
+from frigate.config.classification import AudioTranscriptionModelEnum
 from frigate.const import (
     AUDIO_DURATION,
     AUDIO_FORMAT,
@@ -35,6 +37,7 @@ from frigate.data_processing.common.audio_transcription.model import (
 from frigate.data_processing.real_time.audio_transcription import (
     AudioTranscriptionRealTimeProcessor,
 )
+from frigate.data_processing.types import DataProcessorMetrics
 from frigate.ffmpeg_presets import parse_preset_input
 from frigate.log import LogPipe, suppress_stderr_during
 from frigate.util.builtin import get_ffmpeg_arg_list, load_labels
@@ -85,6 +88,7 @@ class AudioProcessor(FrigateProcess):
         self,
         config: FrigateConfig,
         camera_metrics: DictProxy,
+        embeddings_metrics: DataProcessorMetrics,
         stop_event: MpEvent,
     ):
         super().__init__(
@@ -92,8 +96,38 @@ class AudioProcessor(FrigateProcess):
         )
 
         self.camera_metrics = camera_metrics
+        self.embeddings_metrics = embeddings_metrics
         self.config = config
 
+    def spawn_if_needed(self, camera: CameraConfig) -> None:
+        """Start an audio maintainer for the camera once everything it needs
+        has arrived. Returning early leaves the camera for the next poll."""
+        name = camera.name
+        if name is None or name in self.audio_threads:
+            return
+        if not camera.enabled or not camera.audio.enabled:
+            return
+        # ffmpeg update may not have arrived yet
+        if not any("audio" in i.roles for i in camera.ffmpeg.inputs):
+            return
+        # the camera maintainer creates metrics on its own poll of the same
+        # add update and may not have gotten there yet
+        metrics = self.camera_metrics.get(name)
+        if metrics is None:
+            return
+        thread = AudioEventMaintainer(
+            camera,
+            self.config,
+            metrics,
+            self.embeddings_metrics,
+            self.transcription_model_runner,
+            self.stop_event,  # type: ignore[arg-type]
+            self.genai_manager,
+        )
+        self.audio_threads[name] = thread
+        thread.start()
+        self.logger.info(f"Audio maintainer started for {name}")
+
     def __stop_audio_thread(self, camera: str) -> None:
         thread = self.audio_threads.pop(camera, None)
         if thread is None:
@@ -112,18 +146,31 @@ class AudioProcessor(FrigateProcess):
 
         threading.current_thread().name = "process:audio_manager"
 
+        self.transcription_model_runner: AudioTranscriptionModelRunner | None = None
+        self.genai_manager: Any = None
+
         if any(
             c.enabled_in_config and c.audio_transcription.enabled
             for c in self.config.cameras.values()
         ):
-            self.transcription_model_runner: AudioTranscriptionModelRunner | None = (
-                AudioTranscriptionModelRunner(
+            if isinstance(
+                self.config.audio_transcription.model, AudioTranscriptionModelEnum
+            ):
+                # AudioTranscriptionModelRunner.__init__ unconditionally fetches
+                # sherpa-onnx or whisper weights, so only build it on the local path
+                self.transcription_model_runner = AudioTranscriptionModelRunner(
                     self.config.audio_transcription.device or "AUTO",
                     self.config.audio_transcription.model_size,
                 )
-            )
-        else:
-            self.transcription_model_runner = None
+            else:
+                # imported here rather than at module scope: frigate.genai pulls in
+                # numpy, the provider SDKs, frigate.models, and the prompt builders,
+                # and this process runs at PROCESS_PRIORITY_HIGH. built after the
+                # fork because SDK clients hold sockets and TLS state that must not
+                # cross it; clients themselves stay lazy behind the role property.
+                from frigate.genai.manager import GenAIClientManager
+
+                self.genai_manager = GenAIClientManager(self.config)
 
         config_subscriber = CameraConfigUpdateSubscriber(
             self.config,
@@ -136,28 +183,8 @@ class AudioProcessor(FrigateProcess):
             ],
         )
 
-        def spawn_if_needed(camera: CameraConfig) -> None:
-            name = camera.name
-            if name is None or name in self.audio_threads:
-                return
-            if not camera.enabled or not camera.audio.enabled:
-                return
-            # ffmpeg update may not have arrived yet; wait for next poll
-            if not any("audio" in i.roles for i in camera.ffmpeg.inputs):
-                return
-            thread = AudioEventMaintainer(
-                camera,
-                self.config,
-                self.camera_metrics,
-                self.transcription_model_runner,
-                self.stop_event,  # type: ignore[arg-type]
-            )
-            self.audio_threads[name] = thread
-            thread.start()
-            self.logger.info(f"Audio maintainer started for {name}")
-
         for camera in self.config.cameras.values():
-            spawn_if_needed(camera)
+            self.spawn_if_needed(camera)
 
         self.logger.info(f"Audio processor started (pid: {self.pid})")
 
@@ -167,15 +194,14 @@ class AudioProcessor(FrigateProcess):
             updated_topics = config_subscriber.check_for_updates()
 
             # stop maintainers for removed cameras so their ffmpeg process is
-            # torn down and they stop touching camera_metrics (which the camera
-            # maintainer has already popped for the removed camera)
+            # torn down
             for removed_camera in updated_topics.get(
                 CameraConfigUpdateEnum.remove.name, []
             ):
                 self.__stop_audio_thread(removed_camera)
 
             for camera in self.config.cameras.values():
-                spawn_if_needed(camera)
+                self.spawn_if_needed(camera)
 
         config_subscriber.stop()
 
@@ -197,15 +223,21 @@ class AudioEventMaintainer(threading.Thread):
         self,
         camera: CameraConfig,
         config: FrigateConfig,
-        camera_metrics: DictProxy,
+        metrics: CameraMetrics,
+        embeddings_metrics: DataProcessorMetrics,
         audio_transcription_model_runner: AudioTranscriptionModelRunner | None,
         stop_event: threading.Event,
+        genai_manager: Any = None,
     ) -> None:
         super().__init__(name=f"{camera.name}_audio_event_processor")
 
         self.config = config
         self.camera_config = camera
-        self.camera_metrics = camera_metrics
+        # hold the metrics object rather than indexing the manager dict per
+        # chunk, which costs an IPC round trip and breaks once the camera
+        # maintainer pops the entry on removal
+        self.metrics = metrics
+        self.embeddings_metrics = embeddings_metrics
         self.stop_event = stop_event
         # per-camera stop signal so a single maintainer can be torn down at
         # runtime (e.g. on camera removal) without stopping the whole process
@@ -222,6 +254,7 @@ class AudioEventMaintainer(threading.Thread):
         self.logpipe = LogPipe(f"ffmpeg.{self.camera_config.name}.audio")
         self.audio_listener: subprocess.Popen[Any] | None = None
         self.audio_transcription_model_runner = audio_transcription_model_runner
+        self.genai_manager = genai_manager
         self.transcription_processor = None
         self.transcription_thread = None
 
@@ -238,9 +271,9 @@ class AudioEventMaintainer(threading.Thread):
         )
         self.detection_publisher = DetectionPublisher(DetectionTypeEnum.audio.value)
 
-        if (
-            self.camera_config.audio_transcription.enabled
-            and self.audio_transcription_model_runner is not None
+        if self.camera_config.audio_transcription.enabled and (
+            self.audio_transcription_model_runner is not None
+            or self.genai_manager is not None
         ):
             # init the transcription processor for this camera
             self.transcription_processor = AudioTranscriptionRealTimeProcessor(
@@ -248,8 +281,9 @@ class AudioEventMaintainer(threading.Thread):
                 camera_config=self.camera_config,
                 requestor=self.requestor,
                 model_runner=self.audio_transcription_model_runner,
-                metrics=self.camera_metrics[self.camera_config.name],
+                metrics=self.embeddings_metrics,
                 stop_event=self.stop_event,
+                genai_manager=self.genai_manager,
             )
 
             self.transcription_thread = threading.Thread(
@@ -273,8 +307,8 @@ class AudioEventMaintainer(threading.Thread):
         audio_as_float: np.ndarray = audio.astype(np.float32)
         rms, dBFS = self.calculate_audio_levels(audio_as_float)
 
-        self.camera_metrics[self.camera_config.name].audio_rms.value = rms
-        self.camera_metrics[self.camera_config.name].audio_dBFS.value = dBFS
+        self.metrics.audio_rms.value = rms
+        self.metrics.audio_dBFS.value = dBFS
 
         audio_detections: list[tuple[str, float]] = []
 
@@ -359,7 +393,6 @@ class AudioEventMaintainer(threading.Thread):
                 return
 
             time.sleep(self.camera_config.ffmpeg.retry_interval)
-            self.logpipe.dump()
             self.start_or_restart_ffmpeg()
 
         if self.audio_listener is None or self.audio_listener.stdout is None:

frigate/events/cleanup.py

@@ -37,7 +37,7 @@ class EventCleanup(threading.Thread):
         if self.removed_camera_labels is None:
             self.removed_camera_labels = list(
                 Event.select(Event.label)
-                .where(Event.camera.not_in(self.camera_keys))  # type: ignore[arg-type,call-arg,misc]
+                .where(Event.camera.not_in(self.camera_keys))
                 .distinct()
                 .execute()
             )
@@ -89,7 +89,7 @@ class EventCleanup(threading.Thread):
                     Event.thumbnail,
                 )
                 .where(
-                    Event.camera.not_in(self.camera_keys),  # type: ignore[arg-type,call-arg,misc]
+                    Event.camera.not_in(self.camera_keys),
                     Event.start_time < expire_after,
                     Event.label == event.label,
                     Event.retain_indefinitely == False,
@@ -111,7 +111,7 @@ class EventCleanup(threading.Thread):
 
             # update the clips attribute for the db entry
             query = Event.select(Event.id).where(
-                Event.camera.not_in(self.camera_keys),  # type: ignore[arg-type,call-arg,misc]
+                Event.camera.not_in(self.camera_keys),
                 Event.start_time < expire_after,
                 Event.label == event.label,
                 Event.retain_indefinitely == False,
@@ -218,7 +218,7 @@ class EventCleanup(threading.Thread):
                 Event.camera,
             )
             .where(
-                Event.camera.not_in(self.camera_keys),  # type: ignore[arg-type,call-arg,misc]
+                Event.camera.not_in(self.camera_keys),
                 Event.start_time < expire_after,
                 Event.retain_indefinitely == False,
             )
@@ -249,7 +249,7 @@ class EventCleanup(threading.Thread):
 
         # update the clips attribute for the db entry
         query = Event.select(Event.id).where(
-            Event.camera.not_in(self.camera_keys),  # type: ignore[arg-type,call-arg,misc]
+            Event.camera.not_in(self.camera_keys),
             Event.start_time < expire_after,
             Event.retain_indefinitely == False,
         )
@@ -365,6 +365,7 @@ class EventCleanup(threading.Thread):
                     chunk = ids_to_delete[i : i + CHUNK_SIZE]
                     logger.debug(f"Deleting {len(chunk)} events from the database")
                     Event.delete().where(Event.id << chunk).execute()
+                    Timeline.delete().where(Timeline.source_id << chunk).execute()
 
                     # embeddings are always cleaned up, even when semantic search
                     # is disabled, so that they don't outlive their events

frigate/ffmpeg_presets.py

@@ -84,6 +84,8 @@ _user_agent_args = [
 PRESETS_HW_ACCEL_DECODE = {
     "preset-rpi-64-h264": "-c:v:1 h264_v4l2m2m",
     "preset-rpi-64-h265": "-c:v:1 hevc_v4l2m2m",
+    "preset-apple-silicon-h264": "-c:v h264_v4l2m2m",
+    "preset-apple-silicon-h265": "-c:v hevc_v4l2m2m",
     FFMPEG_HWACCEL_VAAPI: "-hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device {3} -hwaccel_output_format vaapi",
     "preset-intel-qsv-h264": f"-hwaccel qsv -qsv_device {{3}} -hwaccel_output_format qsv -c:v h264_qsv{' -bsf:v dump_extra' if LIBAVFORMAT_VERSION_MAJOR >= 61 else ''}",  # https://trac.ffmpeg.org/ticket/9766#comment:17
     "preset-intel-qsv-h265": f"-load_plugin hevc_hw -hwaccel qsv -qsv_device {{3}} -hwaccel_output_format qsv{' -bsf:v dump_extra' if LIBAVFORMAT_VERSION_MAJOR >= 61 else ''}",  # https://trac.ffmpeg.org/ticket/9766#comment:17
@@ -120,9 +122,12 @@ PRESETS_HW_ACCEL_DECODE["preset-rk-h265"] = PRESETS_HW_ACCEL_DECODE[
 PRESETS_HW_ACCEL_SCALE = {
     "preset-rpi-64-h264": "-r {0} -vf fps={0},scale={1}:{2}",
     "preset-rpi-64-h265": "-r {0} -vf fps={0},scale={1}:{2}",
+    # ffmpeg's v4l2m2m decoders cannot scale, so frames are scaled on the CPU
+    "preset-apple-silicon-h264": "-r {0} -vf fps={0},scale={1}:{2}",
+    "preset-apple-silicon-h265": "-r {0} -vf fps={0},scale={1}:{2}",
     FFMPEG_HWACCEL_VAAPI: "-r {0} -vf fps={0},scale_vaapi=w={1}:h={2},hwdownload,format=nv12",
-    "preset-intel-qsv-h264": "-r {0} -vf vpp_qsv=w={1}:h={2}:format=nv12,hwdownload,format=nv12,fps={0},format=yuv420p",
-    "preset-intel-qsv-h265": "-r {0} -vf vpp_qsv=w={1}:h={2}:format=nv12,hwdownload,format=nv12,fps={0},format=yuv420p",
+    "preset-intel-qsv-h264": "-r {0} -vf fps={0},vpp_qsv=w={1}:h={2}:format=nv12,hwdownload,format=nv12,format=yuv420p",
+    "preset-intel-qsv-h265": "-r {0} -vf fps={0},vpp_qsv=w={1}:h={2}:format=nv12,hwdownload,format=nv12,format=yuv420p",
     FFMPEG_HWACCEL_NVIDIA: "-r {0} -vf fps={0},scale_cuda=w={1}:h={2},hwdownload,format=nv12",
     "preset-jetson-h264": "-r {0}",  # scaled in decoder
     "preset-jetson-h265": "-r {0}",  # scaled in decoder
@@ -150,6 +155,8 @@ PRESETS_HW_ACCEL_SCALE["preset-rk-h265"] = PRESETS_HW_ACCEL_SCALE[FFMPEG_HWACCEL
 PRESETS_HW_ACCEL_ENCODE_BIRDSEYE = {
     "preset-rpi-64-h264": "{0} -hide_banner {1} -c:v h264_v4l2m2m {2}",
     "preset-rpi-64-h265": "{0} -hide_banner {1} -c:v hevc_v4l2m2m {2}",
+    "preset-apple-silicon-h264": "{0} -hide_banner {1} -c:v h264_v4l2m2m {2}",
+    "preset-apple-silicon-h265": "{0} -hide_banner {1} -c:v h264_v4l2m2m {2}",
     # -vaapi_device is required in addition to -hwaccel_device: this is the only
     # birdseye preset that uses hwupload, and ffmpeg 8 initializes filters before
     # the decoder creates a device, so hwupload cannot see an -hwaccel_device one.
@@ -184,6 +191,8 @@ PRESETS_HW_ACCEL_ENCODE_BIRDSEYE["preset-rk-h264"] = PRESETS_HW_ACCEL_ENCODE_BIR
 PRESETS_HW_ACCEL_ENCODE_TIMELAPSE = {
     "preset-rpi-64-h264": "{0} -hide_banner {1} -c:v h264_v4l2m2m -pix_fmt yuv420p {2}",
     "preset-rpi-64-h265": "{0} -hide_banner {1} -c:v hevc_v4l2m2m -pix_fmt yuv420p {2}",
+    "preset-apple-silicon-h264": "{0} -hide_banner {1} -c:v h264_v4l2m2m -pix_fmt yuv420p {2}",
+    "preset-apple-silicon-h265": "{0} -hide_banner {1} -c:v hevc_v4l2m2m -pix_fmt yuv420p {2}",
     FFMPEG_HWACCEL_VAAPI: "{0} -hide_banner -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device {3} {1} -c:v h264_vaapi {2}",
     "preset-intel-qsv-h264": "{0} -hide_banner {1} -c:v h264_qsv -profile:v high -level:v 4.1 -async_depth:v 1 {2}",
     "preset-intel-qsv-h265": "{0} -hide_banner {1} -c:v hevc_qsv -profile:v main -level:v 4.1 -async_depth:v 1 {2}",

frigate/genai/__init__.py

@@ -105,8 +105,21 @@ class GenAIClient:
         debug_save: bool,
         activity_context_prompt: str,
         response_style: str = "default",
+        frame_captions: list[str] | None = None,
     ) -> ReviewMetadata | None:
-        """Generate a description for the review item activity."""
+        """Generate a description for the review item activity.
+
+        `frame_captions` holds one caption per thumbnail for the annotated
+        frame mode; each is sent directly before its frame.
+        """
+        if frame_captions and len(frame_captions) != len(thumbnails):
+            logger.warning(
+                "Got %d frame captions for %d thumbnails, sending plain frames",
+                len(frame_captions),
+                len(thumbnails),
+            )
+            frame_captions = None
+
         context_prompt = build_review_description_prompt(
             review_data,
             thumbnails,
@@ -114,6 +127,7 @@ class GenAIClient:
             preferred_language,
             activity_context_prompt,
             response_style,
+            frame_captions,
         )
 
         logger.debug(
@@ -129,9 +143,30 @@ class GenAIClient:
             ) as f:
                 f.write(context_prompt)
 
+            if frame_captions:
+                # One file per frame, numbered to match the image it precedes
+                # (0.txt goes with 0.jpg), so the debug folder replays without
+                # having to re-derive the mapping.
+                for index, caption in enumerate(frame_captions):
+                    with open(
+                        os.path.join(
+                            CLIPS_DIR,
+                            "genai-requests",
+                            review_data["id"],
+                            f"{index}.txt",
+                        ),
+                        "w",
+                    ) as f:
+                        f.write(caption)
+
         response_format = build_review_description_response_format(concerns)
 
-        response = self._send(context_prompt, thumbnails, response_format)
+        response = self._send(
+            context_prompt,
+            thumbnails,
+            response_format,
+            image_captions=frame_captions,
+        )
 
         if debug_save and response:
             with open(
@@ -269,6 +304,7 @@ class GenAIClient:
         images: list[bytes],
         response_format: dict | None = None,
         enable_thinking: bool = False,
+        image_captions: list[str] | None = None,
     ) -> str | None:
         """Submit a request to the provider.
 
@@ -276,6 +312,10 @@ class GenAIClient:
         ``supports_toggleable_thinking``. Description-style callers leave it
         at the default (off) since synthesis tasks don't benefit from
         reasoning traces.
+
+        ``image_captions`` carries one caption per image, to be placed
+        immediately before its image so the model can tell the frames apart.
+        Providers build their request order with ``interleave_images``.
         """
         return None
 
@@ -298,6 +338,11 @@ class GenAIClient:
         """Whether the configured model can generate embeddings via embed()."""
         return False
 
+    @property
+    def supports_transcription(self) -> bool:
+        """Whether the configured model can transcribe audio via transcribe()."""
+        return False
+
     def list_models(self) -> list[str]:
         """Return the list of model names available from this provider.
 
@@ -305,6 +350,21 @@ class GenAIClient:
         """
         return []
 
+    def list_model_capabilities(self) -> dict[str, dict[str, bool]]:
+        """Return capability flags for each model the provider serves.
+
+        Only providers whose backend advertises capabilities per model can
+        populate this; llama.cpp reports input modalities for every model it
+        serves, so one request describes them all. An empty mapping means "no
+        per-model information available", and callers fall back to this
+        client's own capability properties, which describe only the configured
+        model. A model absent from a non-empty mapping means the same thing.
+
+        Returns:
+            Model name (including aliases) to its capability flags
+        """
+        return {}
+
     def get_context_size(self) -> int:
         """Get the context window size for this provider in tokens."""
         return 4096
@@ -336,6 +396,33 @@ class GenAIClient:
         )
         return []
 
+    def transcribe(
+        self,
+        audio: bytes,
+        language: str | None = None,
+        mime_type: str = "audio/wav",
+    ) -> str | None:
+        """Transcribe speech audio to text.
+
+        Audio is passed as a self-describing blob rather than raw samples so
+        every provider receives a container it can declare, and WAV framing
+        lives in one place instead of in each plugin.
+
+        Args:
+            audio: The encoded audio payload (WAV bytes by default)
+            language: Optional ISO language hint for the provider
+            mime_type: Media type of ``audio``
+
+        Returns:
+            The transcript, or None when the provider cannot produce one
+        """
+        logger.warning(
+            "%s does not support transcription. "
+            "This method should be overridden by the provider implementation.",
+            self.__class__.__name__,
+        )
+        return None
+
     def chat_with_tools(
         self,
         messages: list[dict[str, Any]],

frigate/genai/manager.py

@@ -110,6 +110,12 @@ class GenAIClientManager:
         name = self._role_map.get(GenAIRoleEnum.embeddings)
         return self._get_client(name) if name else None
 
+    @property
+    def transcribe_client(self) -> "GenAIClient | None":
+        """Client configured for the transcribe role."""
+        name = self._role_map.get(GenAIRoleEnum.transcribe)
+        return self._get_client(name) if name else None
+
     def role_info(self) -> dict[str, dict[str, Any]]:
         """Return the model selected for each configured role and its context size.
 
@@ -144,5 +150,11 @@ class GenAIClientManager:
                 "roles": [r.value for r in genai_cfg.roles],
                 "supports_toggleable_thinking": client.supports_toggleable_thinking,
                 "supports_embeddings": client.supports_embeddings,
+                "supports_transcription": client.supports_transcription,
+                # Capabilities of the configured model are above; this maps every
+                # model the provider serves to its own, so the UI can react to a
+                # model selected but not yet saved. Empty when the provider
+                # cannot report capabilities without loading a model.
+                "model_capabilities": client.list_model_capabilities(),
             }
         return result

frigate/genai/plugins/azure-openai.py

@@ -13,6 +13,19 @@ overrides what is genuinely Azure-specific:
 - Context size: Azure does not expose a per-model ``max_model_len`` field
   reliably, so we keep the historical 128K default rather than the
   model-name heuristic used by OpenAI.
+
+Transcription is inherited too: :class:`openai.AzureOpenAI` exposes the same
+``audio.transcriptions.create``. Two Azure-specific caveats apply when using
+the ``transcribe`` role:
+
+- ``model`` must be the Azure *deployment* name, not the underlying model name.
+- The ``api-version`` parsed from ``base_url`` must be 2024-06-01 or later;
+  earlier versions have no transcriptions route and the 404 surfaces only as a
+  generic provider error.
+- Because ``model`` is a deployment name, the inherited check that picks
+  ``languages`` over ``language`` for gpt-transcribe cannot fire unless the
+  deployment happens to be named after the model. Name the deployment
+  ``gpt-transcribe`` to get the right field, or leave the language on ``auto``.
 """
 
 import logging