administrator/frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2026-06-21 03:41:55 +03:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Merge branch 'dev' into dev

Browse Source
This commit is contained in:
Dmytro Marchuk 2026-06-07 08:10:15 +03:00 committed by GitHub
commit f00dd5c7af
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
708 changed files with 44080 additions and 7465 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.cspell/frigate-dictionary.txt
View File

@ -162,6 +162,7 @@ mpegts
mqtt
mse
msenc
muxing
namedtuples
nbytes
nchw
@ -197,6 +198,8 @@ OWASP
paddleocr
paho
passwordless
PCMA
PCMU
popleft
posthog
postprocess
@ -222,7 +225,9 @@ radeontop
rawvideo
rcond
RDONLY
realmonitor
rebranded
recvonly
referer
reindex
Reolink
@ -239,8 +244,11 @@ rocminfo
rootfs
rtmp
RTSP
rtsps
rtspx
ruamel
scroller
sendonly
setproctitle
setpts
shms
@ -251,6 +259,7 @@ SNDMORE
socs
sqliteq
sqlitevecq
Srtp
ssdlite
statm
stimeout

401
.github/copilot-instructions.md vendored
View File

@ -1,401 +0,0 @@
# GitHub Copilot Instructions for Frigate NVR
This document provides coding guidelines and best practices for contributing to Frigate NVR, a complete and local NVR designed for Home Assistant with AI object detection.
## Project Overview
Frigate NVR is a realtime object detection system for IP cameras that uses:
- **Backend**: Python 3.13+ with FastAPI, OpenCV, TensorFlow/ONNX
- **Frontend**: React with TypeScript, Vite, TailwindCSS
- **Architecture**: Multiprocessing design with ZMQ and MQTT communication
- **Focus**: Minimal resource usage with maximum performance
## Code Review Guidelines
When reviewing code, do NOT comment on:
- Missing imports - Static analysis tooling catches these
- Code formatting - Ruff (Python) and Prettier (TypeScript/React) handle formatting
- Minor style inconsistencies already enforced by linters
## Python Backend Standards
### Python Requirements
- **Compatibility**: Python 3.13+
- **Language Features**: Use modern Python features:
- Pattern matching
- Type hints (comprehensive typing preferred)
- f-strings (preferred over `%` or `.format()`)
- Dataclasses
- Async/await patterns
### Code Quality Standards
- **Formatting**: Ruff (configured in `pyproject.toml`)
- **Linting**: Ruff with rules defined in project config
- **Type Checking**: Use type hints consistently
- **Testing**: unittest framework - use `python3 -u -m unittest` to run tests
- **Language**: American English for all code, comments, and documentation
### Logging Standards
- **Logger Pattern**: Use module-level logger
```python
import logging
logger = logging.getLogger(__name__)
```
- **Format Guidelines**:
- No periods at end of log messages
- No sensitive data (keys, tokens, passwords)
- Use lazy logging: `logger.debug("Message with %s", variable)`
- **Log Levels**:
- `debug`: Development and troubleshooting information
- `info`: Important runtime events (startup, shutdown, state changes)
- `warning`: Recoverable issues that should be addressed
- `error`: Errors that affect functionality but don't crash the app
- `exception`: Use in except blocks to include traceback
### Error Handling
- **Exception Types**: Choose most specific exception available
- **Try/Catch Best Practices**:
- Only wrap code that can throw exceptions
- Keep try blocks minimal - process data after the try/except
- Avoid bare exceptions except in background tasks
Bad pattern:
```python
try:
data = await device.get_data() # Can throw
# ❌ Don't process data inside try block
processed = data.get("value", 0) * 100
result = processed
except DeviceError:
logger.error("Failed to get data")
```
Good pattern:
```python
try:
data = await device.get_data() # Can throw
except DeviceError:
logger.error("Failed to get data")
return
# ✅ Process data outside try block
processed = data.get("value", 0) * 100
result = processed
```
### Async Programming
- **External I/O**: All external I/O operations must be async
- **Best Practices**:
- Avoid sleeping in loops - use `asyncio.sleep()` not `time.sleep()`
- Avoid awaiting in loops - use `asyncio.gather()` instead
- No blocking calls in async functions
- Use `asyncio.create_task()` for background operations
- **Thread Safety**: Use proper synchronization for shared state
### Documentation Standards
- **Module Docstrings**: Concise descriptions at top of files
```python
"""Utilities for motion detection and analysis."""
```
- **Function Docstrings**: Required for public functions and methods
```python
async def process_frame(frame: ndarray, config: Config) -> Detection:
"""Process a video frame for object detection.
Args:
frame: The video frame as numpy array
config: Detection configuration
Returns:
Detection results with bounding boxes
"""
```
- **Comment Style**:
- Explain the "why" not just the "what"
- Keep lines under 88 characters when possible
- Use clear, descriptive comments
### File Organization
- **API Endpoints**: `frigate/api/` - FastAPI route handlers
- **Configuration**: `frigate/config/` - Configuration parsing and validation
- **Detectors**: `frigate/detectors/` - Object detection backends
- **Events**: `frigate/events/` - Event management and storage
- **Utilities**: `frigate/util/` - Shared utility functions
## Frontend (React/TypeScript) Standards
### Internationalization (i18n)
- **CRITICAL**: Never write user-facing strings directly in components
- **Always use react-i18next**: Import and use the `t()` function
```tsx
import { useTranslation } from "react-i18next";
function MyComponent() {
const { t } = useTranslation(["views/live"]);
return <div>{t("camera_not_found")}</div>;
}
```
- **Translation Files**: Add English strings to the appropriate json files in `web/public/locales/en`
- **Namespaces**: Organize translations by feature/view (e.g., `views/live`, `common`, `views/system`)
### Code Quality
- **Linting**: ESLint (see `web/.eslintrc.cjs`)
- **Formatting**: Prettier with Tailwind CSS plugin
- **Type Safety**: TypeScript strict mode enabled
- **Testing**: Vitest for unit tests
### Component Patterns
- **UI Components**: Use Radix UI primitives (in `web/src/components/ui/`)
- **Styling**: TailwindCSS with `cn()` utility for class merging
- **State Management**: React hooks (useState, useEffect, useCallback, useMemo)
- **Data Fetching**: Custom hooks with proper loading and error states
### ESLint Rules
Key rules enforced:
- `react-hooks/rules-of-hooks`: error
- `react-hooks/exhaustive-deps`: error
- `no-console`: error (use proper logging or remove)
- `@typescript-eslint/no-explicit-any`: warn (always use proper types instead of `any`)
- Unused variables must be prefixed with `_`
- Comma dangles required for multiline objects/arrays
### File Organization
- **Pages**: `web/src/pages/` - Route components
- **Views**: `web/src/views/` - Complex view components
- **Components**: `web/src/components/` - Reusable components
- **Hooks**: `web/src/hooks/` - Custom React hooks
- **API**: `web/src/api/` - API client functions
- **Types**: `web/src/types/` - TypeScript type definitions
## Testing Requirements
### Backend Testing
- **Framework**: Python unittest
- **Run Command**: `python3 -u -m unittest`
- **Location**: `frigate/test/`
- **Coverage**: Aim for comprehensive test coverage of core functionality
- **Pattern**: Use `TestCase` classes with descriptive test method names
```python
class TestMotionDetection(unittest.TestCase):
def test_detects_motion_above_threshold(self):
# Test implementation
```
### Test Best Practices
- Always have a way to test your work and confirm your changes
- Write tests for bug fixes to prevent regressions
- Test edge cases and error conditions
- Mock external dependencies (cameras, APIs, hardware)
- Use fixtures for test data
## Development Commands
### Python Backend
```bash
# Run all tests
python3 -u -m unittest
# Run specific test file
python3 -u -m unittest frigate.test.test_ffmpeg_presets
# Check formatting (Ruff)
ruff format --check frigate/
# Apply formatting
ruff format frigate/
# Run linter
ruff check frigate/
```
### Frontend (from web/ directory)
```bash
# Start dev server (AI agents should never run this directly unless asked)
npm run dev
# Build for production
npm run build
# Run linter
npm run lint
# Fix linting issues
npm run lint:fix
# Format code
npm run prettier:write
```
### Docker Development
AI agents should never run these commands directly unless instructed.
```bash
# Build local image
make local
# Build debug image
make debug
```
## Common Patterns
### API Endpoint Pattern
```python
from fastapi import APIRouter, Request
from frigate.api.defs.tags import Tags
router = APIRouter(tags=[Tags.Events])
@router.get("/events")
async def get_events(request: Request, limit: int = 100):
"""Retrieve events from the database."""
# Implementation
```
### Configuration Access
```python
# Access Frigate configuration
config: FrigateConfig = request.app.frigate_config
camera_config = config.cameras["front_door"]
```
### Database Queries
```python
from frigate.models import Event
# Use Peewee ORM for database access
events = (
Event.select()
.where(Event.camera == camera_name)
.order_by(Event.start_time.desc())
.limit(limit)
)
```
## Common Anti-Patterns to Avoid
### ❌ Avoid These
```python
# Blocking operations in async functions
data = requests.get(url) # ❌ Use async HTTP client
time.sleep(5) # ❌ Use asyncio.sleep()
# Hardcoded strings in React components
<div>Camera not found</div> # ❌ Use t("camera_not_found")
# Missing error handling
data = await api.get_data() # ❌ No exception handling
# Bare exceptions in regular code
try:
value = await sensor.read()
except Exception: # ❌ Too broad
logger.error("Failed")
# Returning exceptions in JSON responses
except ValueError as e:
return JSONResponse(
content={"success": False, "message": str(e)},
)
```
### ✅ Use These Instead
```python
# Async operations
import aiohttp
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
data = await response.json()
await asyncio.sleep(5) # ✅ Non-blocking
# Translatable strings in React
const { t } = useTranslation();
<div>{t("camera_not_found")}</div> # ✅ Translatable
# Proper error handling
try:
data = await api.get_data()
except ApiException as err:
logger.error("API error: %s", err)
raise
# Specific exceptions
try:
value = await sensor.read()
except SensorException as err: # ✅ Specific
logger.exception("Failed to read sensor")
# Safe error responses
except ValueError:
logger.exception("Invalid parameters for API request")
return JSONResponse(
content={
"success": False,
"message": "Invalid request parameters",
},
)
```
## Project-Specific Conventions
### Configuration Files
- Main config: `config/config.yml`
### Directory Structure
- Backend code: `frigate/`
- Frontend code: `web/`
- Docker files: `docker/`
- Documentation: `docs/`
- Database migrations: `migrations/`
### Code Style Conformance
Always conform new and refactored code to the existing coding style in the project:
- Follow established patterns in similar files
- Match indentation and formatting of surrounding code
- Use consistent naming conventions (snake_case for Python, camelCase for TypeScript)
- Maintain the same level of verbosity in comments and docstrings
## Additional Resources
- Documentation: https://docs.frigate.video
- Main Repository: https://github.com/blakeblackshear/frigate
- Home Assistant Integration: https://github.com/blakeblackshear/frigate-hass-integration

1
.github/copilot-instructions.md vendored Symbolic link
View File

@ -0,0 +1 @@
AGENTS.md

439
AGENTS.md Normal file
View File

@ -0,0 +1,439 @@
# Agent Instructions for Frigate NVR
This document provides coding guidelines and best practices for contributing to Frigate NVR, a complete and local NVR designed for Home Assistant with AI object detection.
## Project Overview
Frigate NVR is a realtime object detection system for IP cameras that uses:
- **Backend**: Python 3.13+ with FastAPI, OpenCV, TensorFlow/ONNX
- **Frontend**: React with TypeScript, Vite, TailwindCSS
- **Architecture**: Multiprocessing design with ZMQ and MQTT communication
- **Focus**: Minimal resource usage with maximum performance
## Code Review Guidelines
When reviewing code, do NOT comment on:
- Missing imports - Static analysis tooling catches these
- Code formatting - Ruff (Python) and Prettier (TypeScript/React) handle formatting
- Minor style inconsistencies already enforced by linters
## Python Backend Standards
### Python Requirements
- **Compatibility**: Python 3.13+
- **Language Features**: Use modern Python features:
- Pattern matching
- Type hints (comprehensive typing preferred)
- f-strings (preferred over `%` or `.format()`)
- Dataclasses
- Async/await patterns
### Code Quality Standards
- **Formatting**: Ruff (configured in `pyproject.toml`)
- **Linting**: Ruff with rules defined in project config
- **Type Checking**: Use type hints consistently
- **Testing**: unittest framework - use `python3 -u -m unittest` to run tests
- **Language**: American English for all code, comments, and documentation
### Logging Standards
- **Logger Pattern**: Use module-level logger
```python
import logging
logger = logging.getLogger(__name__)
```
- **Format Guidelines**:
- No periods at end of log messages
- No sensitive data (keys, tokens, passwords)
- Use lazy logging: `logger.debug("Message with %s", variable)`
- **Log Levels**:
- `debug`: Development and troubleshooting information
- `info`: Important runtime events (startup, shutdown, state changes)
- `warning`: Recoverable issues that should be addressed
- `error`: Errors that affect functionality but don't crash the app
- `exception`: Use in except blocks to include traceback
### Error Handling
- **Exception Types**: Choose most specific exception available
- **Try/Catch Best Practices**:
- Only wrap code that can throw exceptions
- Keep try blocks minimal - process data after the try/except
- Avoid bare exceptions except in background tasks
Bad pattern:
```python
try:
data = await device.get_data() # Can throw
# ❌ Don't process data inside try block
processed = data.get("value", 0) * 100
result = processed
except DeviceError:
logger.error("Failed to get data")
```
Good pattern:
```python
try:
data = await device.get_data() # Can throw
except DeviceError:
logger.error("Failed to get data")
return
# ✅ Process data outside try block
processed = data.get("value", 0) * 100
result = processed
```
### Async Programming
- **External I/O**: All external I/O operations must be async
- **Best Practices**:
- Avoid sleeping in loops - use `asyncio.sleep()` not `time.sleep()`
- Avoid awaiting in loops - use `asyncio.gather()` instead
- No blocking calls in async functions
- Use `asyncio.create_task()` for background operations
- **Thread Safety**: Use proper synchronization for shared state
### Documentation Standards
- **Module Docstrings**: Concise descriptions at top of files
```python
"""Utilities for motion detection and analysis."""
```
- **Function Docstrings**: Required for public functions and methods
```python
async def process_frame(frame: ndarray, config: Config) -> Detection:
"""Process a video frame for object detection.
Args:
frame: The video frame as numpy array
config: Detection configuration
Returns:
Detection results with bounding boxes
"""
```
- **Comment Style**:
- Explain the "why" not just the "what"
- Keep lines under 88 characters when possible
- Use clear, descriptive comments
### File Organization
- **API Endpoints**: `frigate/api/` - FastAPI route handlers
- **Configuration**: `frigate/config/` - Configuration parsing and validation
- **Detectors**: `frigate/detectors/` - Object detection backends
- **Events**: `frigate/events/` - Event management and storage
- **Utilities**: `frigate/util/` - Shared utility functions
## Frontend (React/TypeScript) Standards
### Internationalization (i18n)
- **CRITICAL**: Never write user-facing strings directly in components
- **Always use react-i18next**: Import and use the `t()` function
```tsx
import { useTranslation } from "react-i18next";
function MyComponent() {
const { t } = useTranslation(["views/live"]);
return <div>{t("camera_not_found")}</div>;
}
```
- **Translation Files**: Add English strings to the appropriate json files in `web/public/locales/en`
- **Namespaces**: Organize translations by feature/view (e.g., `views/live`, `common`, `views/system`)
### Code Quality
- **Linting**: ESLint (see `web/.eslintrc.cjs`)
- **Formatting**: Prettier with Tailwind CSS plugin
- **Type Safety**: TypeScript strict mode enabled
### Component Patterns
- **UI Components**: Use Radix UI primitives (in `web/src/components/ui/`)
- **Styling**: TailwindCSS with `cn()` utility for class merging
- **State Management**: React hooks (useState, useEffect, useCallback, useMemo)
- **Data Fetching**: Custom hooks with proper loading and error states
### ESLint Rules
Key rules enforced:
- `react-hooks/rules-of-hooks`: error
- `react-hooks/exhaustive-deps`: error
- `no-console`: error (use proper logging or remove)
- `@typescript-eslint/no-explicit-any`: warn (always use proper types instead of `any`)
- Unused variables must be prefixed with `_`
- Comma dangles required for multiline objects/arrays
### File Organization
- **Pages**: `web/src/pages/` - Route components
- **Views**: `web/src/views/` - Complex view components
- **Components**: `web/src/components/` - Reusable components
- **Hooks**: `web/src/hooks/` - Custom React hooks
- **API**: `web/src/api/` - API client functions
- **Types**: `web/src/types/` - TypeScript type definitions
## Testing Requirements
### Backend Testing
- **Framework**: Python unittest
- **Run Command**: `python3 -u -m unittest`
- **Location**: `frigate/test/`
- **Coverage**: Aim for comprehensive test coverage of core functionality
- **Pattern**: Use `TestCase` classes with descriptive test method names
```python
class TestMotionDetection(unittest.TestCase):
def test_detects_motion_above_threshold(self):
# Test implementation
```
### Test Best Practices
- Always have a way to test your work and confirm your changes
- Write tests for bug fixes to prevent regressions
- Test edge cases and error conditions
- Mock external dependencies (cameras, APIs, hardware)
- Use fixtures for test data
## Development Commands
### Python Backend
```bash
# Run all tests
python3 -u -m unittest
# Run specific test file
python3 -u -m unittest frigate.test.test_ffmpeg_presets
# Check formatting (Ruff)
ruff format --check frigate/
# Apply formatting
ruff format frigate/
# Run linter
ruff check frigate/
# Type check
python3 -u -m mypy --config-file frigate/mypy.ini frigate
```
### Frontend (from web/ directory)
```bash
# Start dev server (AI agents should never run this directly unless asked)
npm run dev
# Build for production
npm run build
# Run linter
npm run lint
# Fix linting issues
npm run lint:fix
# Format code
npm run prettier:write
# E2E: first-time setup
npm install
npx playwright install chromium
# E2E: build the app and run all tests
npm run e2e:build && npm run e2e
# E2E: interactive UI for debugging
npm run e2e:ui
# E2E: run a specific spec
npx playwright test --config e2e/playwright.config.ts e2e/specs/live.spec.ts
# E2E: filter by name, or run only desktop/mobile
npx playwright test --config e2e/playwright.config.ts --grep="severity tab"
npx playwright test --config e2e/playwright.config.ts --project=desktop
# E2E: regenerate mock data after backend model changes (from repo root)
PYTHONPATH=. python3 web/e2e/fixtures/mock-data/generate-mock-data.py
# Regenerate config translations from Pydantic models — outputs to
# web/public/locales/en/config/{global,cameras}.json. NEVER edit those
# JSON files by hand; change the Pydantic field title/description and
# re-run this script. (from repo root)
python3 generate_config_translations.py
# Extract i18n keys from source into the locale files after adding
# new t() calls. Use the :ci variant to verify the locale files are
# in sync with source (fails if extraction would change anything).
npm run i18n:extract
npm run i18n:extract:ci
```
### Docker Development
AI agents should never run these commands directly unless instructed.
```bash
# Build local image
make local
# Build debug image
make debug
```
## Common Patterns
### API Endpoint Pattern
```python
from fastapi import APIRouter, Request
from frigate.api.defs.tags import Tags
router = APIRouter(tags=[Tags.Events])
@router.get("/events")
async def get_events(request: Request, limit: int = 100):
"""Retrieve events from the database."""
# Implementation
```
### Configuration Access
```python
# Access Frigate configuration
config: FrigateConfig = request.app.frigate_config
camera_config = config.cameras["front_door"]
```
### Database Queries
```python
from frigate.models import Event
# Use Peewee ORM for database access
events = (
Event.select()
.where(Event.camera == camera_name)
.order_by(Event.start_time.desc())
.limit(limit)
)
```
## Common Anti-Patterns to Avoid
### ❌ Avoid These
```python
# Blocking operations in async functions
data = requests.get(url) # ❌ Use async HTTP client
time.sleep(5) # ❌ Use asyncio.sleep()
# Hardcoded strings in React components
<div>Camera not found</div> # ❌ Use t("camera_not_found")
# Missing error handling
data = await api.get_data() # ❌ No exception handling
# Bare exceptions in regular code
try:
value = await sensor.read()
except Exception: # ❌ Too broad
logger.error("Failed")
# Returning exceptions in JSON responses
except ValueError as e:
return JSONResponse(
content={"success": False, "message": str(e)},
)
```
### ✅ Use These Instead
```python
# Async operations
import aiohttp
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
data = await response.json()
await asyncio.sleep(5) # ✅ Non-blocking
# Translatable strings in React
const { t } = useTranslation();
<div>{t("camera_not_found")}</div> # ✅ Translatable
# Proper error handling
try:
data = await api.get_data()
except ApiException as err:
logger.error("API error: %s", err)
raise
# Specific exceptions
try:
value = await sensor.read()
except SensorException as err: # ✅ Specific
logger.exception("Failed to read sensor")
# Safe error responses
except ValueError:
logger.exception("Invalid parameters for API request")
return JSONResponse(
content={
"success": False,
"message": "Invalid request parameters",
},
)
```
## WebSocket Broadcasts
Outbound WebSocket broadcasts go through a per-recipient classifier in `frigate/comms/ws.py` that enforces camera-level access. **The classifier is fail-closed: any topic it doesn't recognize is dropped for every client.** New outbound topics must be classified there or they'll silently disappear.
## Project-Specific Conventions
### Configuration Files
- Main config: `config/config.yml`
### Directory Structure
- Backend code: `frigate/`
- Frontend code: `web/`
- Docker files: `docker/`
- Documentation: `docs/`
- Database migrations: `migrations/`
### Code Style Conformance
Always conform new and refactored code to the existing coding style in the project:
- Follow established patterns in similar files
- Match indentation and formatting of surrounding code
- Use consistent naming conventions (snake_case for Python, camelCase for TypeScript)
- Maintain the same level of verbosity in comments and docstrings
## Additional Resources
- Documentation: https://docs.frigate.video
- Main Repository: https://github.com/blakeblackshear/frigate
- Home Assistant Integration: https://github.com/blakeblackshear/frigate-hass-integration

1
CLAUDE.md Symbolic link
View File

@ -0,0 +1 @@
AGENTS.md

4
docker/main/Dockerfile
View File

@ -265,8 +265,8 @@ ENV PATH="/usr/local/go2rtc/bin:/usr/local/tempio/bin:/usr/local/nginx/sbin:${PA
RUN --mount=type=bind,source=docker/main/install_deps.sh,target=/deps/install_deps.sh \
/deps/install_deps.sh
ENV DEFAULT_FFMPEG_VERSION="7.0"
ENV INCLUDED_FFMPEG_VERSIONS="${DEFAULT_FFMPEG_VERSION}:5.0"
ENV DEFAULT_FFMPEG_VERSION="8.0"
ENV INCLUDED_FFMPEG_VERSIONS="${DEFAULT_FFMPEG_VERSION}:7.0:5.0"
RUN wget -q https://bootstrap.pypa.io/get-pip.py -O get-pip.py \
&& sed -i 's/args.append("setuptools")/args.append("setuptools==77.0.3")/' get-pip.py \

12
docker/main/install_deps.sh
View File

@ -52,9 +52,13 @@ if [[ "${TARGETARCH}" == "amd64" ]]; then
tar -xf ffmpeg.tar.xz -C /usr/lib/ffmpeg/5.0 --strip-components 1 amd64/bin/ffmpeg amd64/bin/ffprobe
rm -rf ffmpeg.tar.xz
mkdir -p /usr/lib/ffmpeg/7.0
wget -qO ffmpeg.tar.xz "https://github.com/NickM-27/FFmpeg-Builds/releases/download/autobuild-2026-03-19-13-03/ffmpeg-n7.1.3-43-g5a1f107b4c-linux64-gpl-7.1.tar.xz"
wget -qO ffmpeg.tar.xz "https://github.com/NickM-27/FFmpeg-Builds/releases/download/autobuild-2024-09-19-12-51/ffmpeg-n7.0.2-18-g3e6cec1286-linux64-gpl-7.0.tar.xz"
tar -xf ffmpeg.tar.xz -C /usr/lib/ffmpeg/7.0 --strip-components 1 amd64/bin/ffmpeg amd64/bin/ffprobe
rm -rf ffmpeg.tar.xz
mkdir -p /usr/lib/ffmpeg/8.0
wget -qO ffmpeg.tar.xz "https://github.com/NickM-27/FFmpeg-Builds/releases/download/autobuild-2026-06-02-14-20/ffmpeg-n8.1.1-9-g58d4114d36-linux64-gpl-8.1.tar.xz"
tar -xf ffmpeg.tar.xz -C /usr/lib/ffmpeg/8.0 --strip-components 1 amd64/bin/ffmpeg amd64/bin/ffprobe
rm -rf ffmpeg.tar.xz
fi
# ffmpeg -> arm64
@ -64,9 +68,13 @@ if [[ "${TARGETARCH}" == "arm64" ]]; then
tar -xf ffmpeg.tar.xz -C /usr/lib/ffmpeg/5.0 --strip-components 1 arm64/bin/ffmpeg arm64/bin/ffprobe
rm -f ffmpeg.tar.xz
mkdir -p /usr/lib/ffmpeg/7.0
wget -qO ffmpeg.tar.xz "https://github.com/NickM-27/FFmpeg-Builds/releases/download/autobuild-2026-03-19-13-03/ffmpeg-n7.1.3-43-g5a1f107b4c-linuxarm64-gpl-7.1.tar.xz"
wget -qO ffmpeg.tar.xz "https://github.com/NickM-27/FFmpeg-Builds/releases/download/autobuild-2024-09-19-12-51/ffmpeg-n7.0.2-18-g3e6cec1286-linuxarm64-gpl-7.0.tar.xz"
tar -xf ffmpeg.tar.xz -C /usr/lib/ffmpeg/7.0 --strip-components 1 arm64/bin/ffmpeg arm64/bin/ffprobe
rm -f ffmpeg.tar.xz
mkdir -p /usr/lib/ffmpeg/8.0
wget -qO ffmpeg.tar.xz "https://github.com/NickM-27/FFmpeg-Builds/releases/download/autobuild-2026-06-02-14-20/ffmpeg-n8.1.1-9-g58d4114d36-linuxarm64-gpl-8.1.tar.xz"
tar -xf ffmpeg.tar.xz -C /usr/lib/ffmpeg/8.0 --strip-components 1 arm64/bin/ffmpeg arm64/bin/ffprobe
rm -f ffmpeg.tar.xz
fi
# arch specific packages

13
docker/main/rootfs/usr/local/ffmpeg/get_ffmpeg_path.py
View File

@ -5,11 +5,7 @@ from typing import Any
from ruamel.yaml import YAML
sys.path.insert(0, "/opt/frigate")
from frigate.const import (
DEFAULT_FFMPEG_VERSION,
INCLUDED_FFMPEG_VERSIONS,
)
from frigate.util.config import find_config_file
from frigate.util.config import find_config_file, resolve_ffmpeg_path
sys.path.remove("/opt/frigate")
@ -29,9 +25,4 @@ except FileNotFoundError:
config: dict[str, Any] = {}
path = config.get("ffmpeg", {}).get("path", "default")
if path == "default":
print(f"/usr/lib/ffmpeg/{DEFAULT_FFMPEG_VERSION}/bin/ffmpeg")
elif path in INCLUDED_FFMPEG_VERSIONS:
print(f"/usr/lib/ffmpeg/{path}/bin/ffmpeg")
else:
print(f"{path}/bin/ffmpeg")
print(resolve_ffmpeg_path(path, "ffmpeg"))

11
docker/main/rootfs/usr/local/go2rtc/create_config.py
View File

@ -11,12 +11,10 @@ sys.path.insert(0, "/opt/frigate")
from frigate.config.env import substitute_frigate_vars
from frigate.const import (
BIRDSEYE_PIPE,
DEFAULT_FFMPEG_VERSION,
INCLUDED_FFMPEG_VERSIONS,
LIBAVFORMAT_VERSION_MAJOR,
)
from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode
from frigate.util.config import find_config_file
from frigate.util.config import find_config_file, resolve_ffmpeg_path
from frigate.util.services import is_restricted_go2rtc_source
sys.path.remove("/opt/frigate")
@ -81,12 +79,7 @@ if go2rtc_config.get("rtsp", {}).get("password") is not None:
# ensure ffmpeg path is set correctly
path = config.get("ffmpeg", {}).get("path", "default")
if path == "default":
ffmpeg_path = f"/usr/lib/ffmpeg/{DEFAULT_FFMPEG_VERSION}/bin/ffmpeg"
elif path in INCLUDED_FFMPEG_VERSIONS:
ffmpeg_path = f"/usr/lib/ffmpeg/{path}/bin/ffmpeg"
else:
ffmpeg_path = f"{path}/bin/ffmpeg"
ffmpeg_path = resolve_ffmpeg_path(path, "ffmpeg")
if go2rtc_config.get("ffmpeg") is None:
go2rtc_config["ffmpeg"] = {"bin": ffmpeg_path}

1
docker/main/rootfs/usr/local/nginx/conf/nginx.conf
View File

@ -252,6 +252,7 @@ http {
include proxy.conf;
proxy_cache api_cache;
proxy_cache_key "$scheme$proxy_host$request_uri|$role|$groups|$user";
proxy_cache_lock on;
proxy_cache_use_stale updating;
proxy_cache_valid 200 5s;

73
docs/docs/configuration/reference.md → docs/docs/configuration/advanced/reference.md
View File

@ -147,6 +147,13 @@ auth:
# NOTE: changing this value will not automatically update password hashes, you
# will need to change each user password for it to apply
hash_iterations: 600000
# Optional: Map roles to the list of cameras each role can access (default: none)
# NOTE: An empty list grants the role access to all cameras. Roles defined here can be
# referenced by proxy header role mapping or assigned to native users.
roles:
my_custom_role:
- front_door
- back_yard
# Optional: model modifications
# NOTE: The default values are for the EdgeTPU detector.
@ -166,6 +173,9 @@ model:
# Required: Object detection model input tensor format
# Valid values are nhwc or nchw (default: shown below)
input_tensor: nhwc
# Optional: Data type of the model input tensor
# Valid values are float, float_denorm, or int (default: shown below)
input_dtype: int
# Required: Object detection model type, currently only used with the OpenVINO detector
# Valid values are ssd, yolox, yolonas (default: shown below)
model_type: ssd
@ -196,6 +206,8 @@ audio:
# - 500 - medium sensitivity
# - 1000 - low sensitivity
min_volume: 500
# Optional: Number of threads to use for audio detection (default: shown below)
num_threads: 2
# Optional: Types of audio to listen for (default: shown below)
listen:
- bark
@ -257,7 +269,7 @@ birdseye:
# More information about presets at https://docs.frigate.video/configuration/ffmpeg_presets
ffmpeg:
# Optional: ffmpeg binary path (default: shown below)
# can also be set to `7.0` or `5.0` to specify one of the included versions
# can also be set to `8.0` or `5.0` to specify one of the included versions
# or can be set to any path that holds `bin/ffmpeg` & `bin/ffprobe`
path: "default"
# Optional: global ffmpeg args (default: shown below)
@ -469,6 +481,8 @@ review:
- Animals in the garden
# Optional: Preferred response language (default: English)
preferred_language: English
# Optional: Save thumbnails sent to the GenAI provider for review/debugging purposes (default: shown below)
debug_save_thumbnails: False
# Optional: Motion configuration
# NOTE: Can be overridden at the camera level
@ -500,6 +514,8 @@ motion:
# - 30 - medium sensitivity
# - 50 - low sensitivity
contour_area: 10
# Optional: Alpha blending factor used in frame differencing for motion calculation (default: shown below)
delta_alpha: 0.2
# Optional: Alpha value passed to cv2.accumulateWeighted when averaging frames to determine the background (default: shown below)
# Higher values mean the current frame impacts the average a lot, and a new object will be averaged into the background faster.
# Low values will cause things like moving shadows to be detected as motion for longer.
@ -572,6 +588,8 @@ record:
timelapse_args: "-vf setpts=0.04*PTS -r 30"
# Optional: Global hardware acceleration settings for timelapse exports. (default: inherit)
hwaccel_args: auto
# Optional: Maximum number of export jobs to process at the same time (default: shown below)
max_concurrent: 3
# Optional: Recording Preview Settings
preview:
# Optional: Quality of recording preview (default: shown below).
@ -638,6 +656,11 @@ snapshots:
retain:
# Required: Default retention days (default: shown below)
default: 10
# Optional: Mode for retention. (default: shown below)
# all - save all snapshots regardless of activity
# motion - save snapshots for any detected motion
# active_objects - save snapshots for active/moving objects
mode: motion
# Optional: Per object retention days
objects:
person: 15
@ -714,15 +737,23 @@ lpr:
enhancement: 0
# Optional: Save plate images to /media/frigate/clips/lpr for debugging purposes (default: shown below)
debug_save_plates: False
# Optional: List of regex replacement rules to normalize detected plates (default: shown below)
replace_rules: {}
# Optional: List of regex replacement rules to normalize detected plates before matching (default: none)
replace_rules:
# Required: regex pattern to match in the detected plate
- pattern: "O"
# Required: string to replace the matched pattern with
replacement: "0"
# Optional: Configuration for AI / LLM provider
# Optional: Configuration for AI / LLM providers
# WARNING: Depending on the provider, this will send thumbnails over the internet
# to Google or OpenAI's LLMs to generate descriptions. GenAI features can be configured at
# the camera level to enhance privacy for indoor cameras.
# NOTE: genai is a map of named providers. Each key is a name you choose for the provider,
# and each role (chat, descriptions, embeddings) may be assigned to exactly one provider.
genai:
# Required: Provider must be one of ollama, gemini, or openai
# Required: name of the provider (chosen by you, used to reference it elsewhere)
my_provider:
# Required: Provider must be one of ollama, openai, azure_openai, gemini, or llamacpp
provider: ollama
# Required if provider is ollama. May also be used for an OpenAI API compatible backend with the openai provider.
base_url: http://localhost::11434
@ -730,6 +761,12 @@ genai:
api_key: "{FRIGATE_GENAI_API_KEY}"
# Required: The model to use with the provider.
model: gemini-1.5-flash
# Optional: Roles this provider handles (default: shown below)
# Each role (chat, descriptions, embeddings) must be assigned to exactly one provider.
roles:
- chat
- descriptions
- embeddings
# Optional additional args to pass to the GenAI Provider (default: None)
provider_options:
keep_alive: -1
@ -840,8 +877,8 @@ cameras:
# Required: name of the camera
back:
# Optional: Enable/Disable the camera (default: shown below).
# If disabled: config is used but no live stream and no capture etc.
# Events/Recordings are still viewable.
# When False, ffmpeg is not started and the camera is hidden from the UI
# (except Camera Management). Re-enabling requires a Frigate restart.
enabled: True
# Optional: camera type used for some Frigate features (default: shown below)
# Options are "generic" and "lpr"
@ -908,6 +945,9 @@ cameras:
inertia: 3
# Optional: Number of seconds that an object must loiter to be considered in the zone (default: shown below)
loitering_time: 0
# Optional: Minimum speed required for an object to be considered present in the zone (default: none)
# In real-world units if distances are set. Used for speed-based zone triggers.
speed_threshold: 2.5
# Optional: List of objects that can trigger this zone (default: all tracked objects)
objects:
- person
@ -945,6 +985,9 @@ cameras:
order: 0
# Optional: Whether or not to show the camera in the Frigate UI (default: shown below)
dashboard: True
# Optional: Whether this camera is visible in review (the review page and its camera
# filter, motion review, and the history view) (default: shown below)
review: True
# Optional: connect to ONVIF camera
# to enable PTZ controls.
@ -1083,22 +1126,6 @@ ui:
# Optional: Set the time format used.
# Options are browser, 12hour, or 24hour (default: shown below)
time_format: browser
# Optional: Set the date style for a specified length.
# Options are: full, long, medium, short
# Examples:
# short: 2/11/23
# medium: Feb 11, 2023
# full: Saturday, February 11, 2023
# (default: shown below).
date_style: short
# Optional: Set the time style for a specified length.
# Options are: full, long, medium, short
# Examples:
# short: 8:14 PM
# medium: 8:15:22 PM
# full: 8:15:22 PM Mountain Standard Time
# (default: shown below).
time_style: medium
# Optional: Set the unit system to either "imperial" or "metric" (default: metric)
# Used in the UI and in MQTT topics
unit_system: metric

41
docs/docs/configuration/advanced.md → docs/docs/configuration/advanced/system.md
View File

@ -1,7 +1,6 @@
---
id: advanced
title: Advanced Options
sidebar_label: Advanced Options
id: system
title: System
---
import ConfigTabs from "@site/src/components/ConfigTabs";
@ -172,7 +171,7 @@ Custom models may also require different input tensor formats. The colorspace co
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detection model" /> to configure the model path, dimensions, and input format.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and open the **Custom Model** tab to configure the model path, dimensions, and input format.
| Field | Description |
| --------------------------------------------- | ------------------------------------ |
@ -202,7 +201,7 @@ model:
:::warning
If the labelmap is customized then the labels used for alerts will need to be adjusted as well. See [alert labels](../configuration/review.md#restricting-alerts-to-specific-labels) for more info.
If the labelmap is customized then the labels used for alerts will need to be adjusted as well. See [alert labels](../review.md#restricting-alerts-to-specific-labels) for more info.
:::
@ -234,26 +233,16 @@ Some labels have special handling and modifications can disable functionality.
## Network Configuration
Changes to Frigate's internal network configuration can be made by bind mounting nginx.conf into the container. For example:
```yaml
services:
frigate:
container_name: frigate
...
volumes:
...
- /path/to/your/nginx.conf:/usr/local/nginx/conf/nginx.conf
```
Frigate exposes a few networking options. IPv6 and the listen ports are set in the `networking` configuration (or from the Settings UI); more advanced changes require [customizing the bundled Nginx configuration](#customizing-the-nginx-configuration).
### Enabling IPv6
IPv6 is disabled by default. Enable it in the Frigate configuration.
By default Frigate listens on IPv4 only. To also listen on IPv6 — on port `5000`, and on `8971` when TLS is configured — enable it in the `networking` configuration.
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Networking" /> and expand **IPv6 configuration**, then enable **Enable IPv6**.
Navigate to <NavPath path="Settings > System > Networking" /> and enable **IPv6**.
</TabItem>
<TabItem value="yaml">
@ -261,7 +250,7 @@ Navigate to <NavPath path="Settings > System > Networking" /> and expand **IPv6
```yaml
networking:
ipv6:
enabled: True
enabled: true
```
</TabItem>
@ -300,6 +289,20 @@ This setting is for advanced users. For the majority of use cases it's recommend
:::
### Customizing the Nginx configuration
More advanced changes to Frigate's internal network configuration can be made by bind mounting your own `nginx.conf` into the container. For example:
```yaml
services:
frigate:
container_name: frigate
...
volumes:
...
- /path/to/your/nginx.conf:/usr/local/nginx/conf/nginx.conf
```
## Base path
By default, Frigate runs at the root path (`/`). However some setups require to run Frigate under a custom path prefix (e.g. `/frigate`), especially when Frigate is located behind a reverse proxy that requires path-based routing.

2
docs/docs/configuration/autotracking.md
View File

@ -167,7 +167,7 @@ A fast [detector](object_detectors.md) is recommended. CPU detectors will not pe
A full-frame zone in `required_zones` is not recommended, especially if you've calibrated your camera and there are `movement_weights` defined in the configuration file. Frigate will continue to autotrack an object that has entered one of the `required_zones`, even if it moves outside of that zone.
Some users have found it helpful to adjust the zone `inertia` value. See the [configuration reference](index.md).
Some users have found it helpful to adjust the zone `inertia` value. See the [configuration reference](advanced/reference.md).
## Zooming

9
docs/docs/configuration/cameras.md
View File

@ -67,7 +67,7 @@ Additional cameras are simply added under the camera configuration section.
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > Camera configuration > Management" /> and use the add camera button to configure each additional camera.
Navigate to <NavPath path="Settings > Global configuration > Camera management" /> and use the add camera button to configure each additional camera.
</TabItem>
<TabItem value="yaml">
@ -143,6 +143,11 @@ If your ONVIF camera does not require authentication credentials, you may still
:::
If a camera connects but fails to authenticate, two optional fields can help:
- `tls_insecure`: Skips TLS certificate verification and sends the ONVIF password as plaintext (`PasswordText`) instead of a hashed digest (`PasswordDigest`). Some cameras reject the digest token and only accept plaintext. This weakens connection security, so only enable it on a trusted local network.
- `ignore_time_mismatch`: ONVIF authentication tokens include a timestamp, and a camera will reject the token if its clock differs too much from Frigate's. Enabling this makes Frigate compensate for the time offset so authentication can still succeed. Running NTP on both the camera and the Frigate host is the recommended fix; only use this in a "safe" environment, as it slightly weakens token validation.
If your camera has multiple ONVIF profiles, you can specify which one to use for PTZ control with the `profile` option, matched by token or name. When not set, Frigate selects the first profile with a valid PTZ configuration. Check the Frigate debug logs (`frigate.ptz.onvif: debug`) to see available profile names and tokens for your camera.
An ONVIF-capable camera that supports relative movement within the field of view (FOV) can also be configured to automatically track moving objects and keep them in the center of the frame. For autotracking setup, see the [autotracking](autotracking.md) docs.
@ -174,7 +179,7 @@ The FeatureList on the [ONVIF Conformant Products Database](https://www.onvif.or
| Hikvision DS-2DE3A404IWG-E/W | ✅ | ✅ | |
| Reolink | ✅ | ❌ | |
| Speco O8P32X | ✅ | ❌ | |
| Sunba 405-D20X | ✅ | ❌ | Incomplete ONVIF support reported on original, and 4k models. All models are suspected incompatable. |
| Sunba 405-D20X | ✅ | ❌ | Incomplete ONVIF support reported on original, and 4k models. All models are suspected incompatible. |
| Tapo | ✅ | ❌ | Many models supported, ONVIF Service Port: 2020 |
| Uniview IPC672LR-AX4DUPK | ✅ | ❌ | Firmware says FOV relative movement is supported, but camera doesn't actually move when sending ONVIF commands |
| Uniview IPC6612SR-X33-VG | ✅ | ✅ | Leave `calibrate_on_startup` as `False`. A user has reported that zooming with `absolute` is working. |

20
docs/docs/configuration/index.md → docs/docs/configuration/config.md
View File

@ -1,5 +1,5 @@
---
id: index
id: config
title: Frigate Configuration
---
@ -57,7 +57,7 @@ VS Code supports JSON schemas for automatically validating configuration files.
## Environment Variable Substitution
Frigate supports the use of environment variables starting with `FRIGATE_` **only** where specifically indicated in the [reference config](./reference.md). For example, the following values can be replaced at runtime by using environment variables:
Frigate supports the use of environment variables starting with `FRIGATE_` **only** where specifically indicated in the [reference config](./advanced/reference.md). For example, the following values can be replaced at runtime by using environment variables:
```yaml
mqtt:
@ -92,7 +92,7 @@ genai:
## Common configuration examples
Here are some common starter configuration examples. These can be configured through the Settings UI or via YAML. Refer to the [reference config](./reference.md) for detailed information about all config values.
Here are some common starter configuration examples. These can be configured through the Settings UI or via YAML. Refer to the [reference config](./advanced/reference.md) for detailed information about all config values.
### Raspberry Pi Home Assistant App with USB Coral
@ -110,10 +110,10 @@ Here are some common starter configuration examples. These can be configured thr
1. Navigate to <NavPath path="Settings > System > MQTT" /> and configure the MQTT connection to your Home Assistant Mosquitto broker
2. Navigate to <NavPath path="Settings > Global configuration > FFmpeg" /> and set **Hardware acceleration arguments** to `Raspberry Pi (H.264)`
3. Navigate to <NavPath path="Settings > System > Detector hardware" /> and add a detector with **Type** `EdgeTPU` and **Device** `usb`
3. Navigate to <NavPath path="Settings > System > Detectors and model" /> and add a detector with **Type** `EdgeTPU` and **Device** `usb`
4. Navigate to <NavPath path="Settings > Global configuration > Recording" /> and set **Enable recording** to on, **Motion retention > Retention days** to `7`, **Alert retention > Event retention > Retention days** to `30`, **Alert retention > Event retention > Retention mode** to `motion`, **Detection retention > Event retention > Retention days** to `30`, **Detection retention > Event retention > Retention mode** to `motion`
5. Navigate to <NavPath path="Settings > Global configuration > Snapshots" /> and set **Enable snapshots** to on, **Snapshot retention > Default retention** to `30`
6. Navigate to <NavPath path="Settings > Camera configuration > Management" /> and add your camera with the appropriate RTSP stream URL
6. Navigate to <NavPath path="Settings > Global configuration > Camera management" /> and add your camera with the appropriate RTSP stream URL
7. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> to add a motion mask for the camera timestamp
</TabItem>
@ -189,10 +189,10 @@ cameras:
1. Navigate to <NavPath path="Settings > System > MQTT" /> and set **Enable MQTT** to off
2. Navigate to <NavPath path="Settings > Global configuration > FFmpeg" /> and set **Hardware acceleration arguments** to `VAAPI (Intel/AMD GPU)`
3. Navigate to <NavPath path="Settings > System > Detector hardware" /> and add a detector with **Type** `EdgeTPU` and **Device** `usb`
3. Navigate to <NavPath path="Settings > System > Detectors and model" /> and add a detector with **Type** `EdgeTPU` and **Device** `usb`
4. Navigate to <NavPath path="Settings > Global configuration > Recording" /> and set **Enable recording** to on, **Motion retention > Retention days** to `7`, **Alert retention > Event retention > Retention days** to `30`, **Alert retention > Event retention > Retention mode** to `motion`, **Detection retention > Event retention > Retention days** to `30`, **Detection retention > Event retention > Retention mode** to `motion`
5. Navigate to <NavPath path="Settings > Global configuration > Snapshots" /> and set **Enable snapshots** to on, **Snapshot retention > Default retention** to `30`
6. Navigate to <NavPath path="Settings > Camera configuration > Management" /> and add your camera with the appropriate RTSP stream URL
6. Navigate to <NavPath path="Settings > Global configuration > Camera management" /> and add your camera with the appropriate RTSP stream URL
7. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> to add a motion mask for the camera timestamp
</TabItem>
@ -266,11 +266,11 @@ cameras:
1. Navigate to <NavPath path="Settings > System > MQTT" /> and configure the connection to your MQTT broker
2. Navigate to <NavPath path="Settings > Global configuration > FFmpeg" /> and set **Hardware acceleration arguments** to `VAAPI (Intel/AMD GPU)`
3. Navigate to <NavPath path="Settings > System > Detector hardware" /> and add a detector with **Type** `openvino` and **Device** `AUTO`
4. Navigate to <NavPath path="Settings > System > Detection model" /> and configure the OpenVINO model path and settings
3. Navigate to <NavPath path="Settings > System > Detectors and model" /> and add a detector with **Type** `openvino` and **Device** `AUTO`
4. On the same page, in the **Custom Model** tab, configure the OpenVINO model path and settings
5. Navigate to <NavPath path="Settings > Global configuration > Recording" /> and set **Enable recording** to on, **Motion retention > Retention days** to `7`, **Alert retention > Event retention > Retention days** to `30`, **Alert retention > Event retention > Retention mode** to `motion`, **Detection retention > Event retention > Retention days** to `30`, **Detection retention > Event retention > Retention mode** to `motion`
6. Navigate to <NavPath path="Settings > Global configuration > Snapshots" /> and set **Enable snapshots** to on, **Snapshot retention > Default retention** to `30`
7. Navigate to <NavPath path="Settings > Camera configuration > Management" /> and add your camera with the appropriate RTSP stream URL
7. Navigate to <NavPath path="Settings > Global configuration > Camera management" /> and add your camera with the appropriate RTSP stream URL
8. Navigate to <NavPath path="Settings > Camera configuration > Masks / Zones" /> to add a motion mask for the camera timestamp
</TabItem>

9
docs/docs/configuration/custom_classification/object_classification.md
View File

@ -149,9 +149,16 @@ For more detail, see [Frigate Tip: Best Practices for Training Face and Custom C
- **The wizard is just the starting point**: You don't need to find and label every class upfront. Missing classes will naturally appear in Recent Classifications, and those images tend to be more valuable because they represent new conditions and edge cases.
- **Problem framing**: Keep classes visually distinct and relevant to the chosen object types.
- **Preprocessing**: Ensure examples reflect object crops similar to Frigate's boxes; keep the subject centered.
- **Labels**: Keep label names short and consistent; include a `none` class if you plan to ignore uncertain predictions for sub labels.
- **Crop size**: Aim for crops of at least 100×100 pixels (a 10,000 pixel area). Crops smaller than ~80×80 get stretched 3-7× by the model's 224×224 input resize and tend to collapse into a generic "blob" region of feature space where identity becomes unreliable. If most of your detections are small because the camera is far from the subject, consider repositioning the camera for closer crops.
- **Class balance**: Aim to keep your largest class within ~3× the count of your smallest. Beyond that, the model becomes biased toward the dominant class and tends to default borderline predictions to it (the "everything looks like Buddy" failure mode).
- **Threshold**: Tune `threshold` per model to reduce false assignments. Start at `0.8` and adjust based on validation.
:::tip `none` works differently from named classes
Named classes work best with visually uniform examples — every Buddy photo should look like Buddy. The `none` class needs the opposite: visual diversity across sizes, framings, and qualities, because at inference it has to absorb everything that isn't one of your named classes. Don't apply the same "only keep large, well-framed images" rule to `none` that you would to a named class. Mix in small crops, partial views, and false positives deliberately - otherwise the model has no signal for "small/ambiguous thing = not one of my known classes" and will force those crops into a named class by default.
:::
## Debugging Classification Models
To troubleshoot issues with object classification models, enable debug logging to see detailed information about classification attempts, scores, and consensus calculations.

11
docs/docs/configuration/genai/config.md
View File

@ -49,15 +49,14 @@ You should have at least 8 GB of RAM available (or VRAM if running on GPU) to ru
### Model Types: Instruct vs Thinking
Most vision-language models are available as **instruct** models, which are fine-tuned to follow instructions and respond concisely to prompts. However, some models (such as certain Qwen-VL or minigpt variants) offer both **instruct** and **thinking** versions.
Vision-language models come in **instruct** variants (fine-tuned to follow instructions and respond concisely), **thinking** variants (fine-tuned for free-form, speculative reasoning), and **hybrid** variants that support both modes per request. Most modern vision-language models are hybrid.
- **Instruct models** are always recommended for use with Frigate. These models generate direct, relevant, actionable descriptions that best fit Frigate's object and event summary use case.
- **Reasoning / Thinking models** are fine-tuned for more free-form, open-ended, and speculative outputs, which are typically not concise and may not provide the practical summaries Frigate expects. For this reason, Frigate does **not** recommend or support using thinking models.
Frigate manages reasoning per task automatically:
Some models are labeled as **hybrid** (capable of both thinking and instruct tasks). In these cases, it is recommended to disable reasoning / thinking, which is generally model specific (see your models documentation).
- **Description tasks** (object descriptions, review descriptions, review summaries) are synthesis-only and benefit from concise, direct output, so Frigate disables thinking for these calls when the model exposes a per-request toggle.
- **Chat** lets you toggle thinking on or off from the composer when the configured model supports it.
**Recommendation:**
Always select the `-instruct` or documented instruct/tagged variant of any model you use in your Frigate configuration. If in doubt, refer to your model provider's documentation or model library for guidance on the correct model variant to use.
You can use a pure instruct, hybrid, or thinking-capable model with Frigate — no extra configuration is required to disable thinking for descriptions.
### llama.cpp

70
docs/docs/configuration/go2rtc.md Normal file
View File

@ -0,0 +1,70 @@
---
id: go2rtc
title: go2rtc
---
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
Frigate uses the bundled go2rtc to power a number of key features:
- WebRTC or MSE for live viewing with audio, higher resolutions and frame rates than the jsmpeg stream which is limited to the detect stream and does not support audio
- Live stream support for cameras in Home Assistant Integration
- RTSP relay for use with other consumers to reduce the number of connections to your camera streams
:::tip[Most users no longer need to configure go2rtc by hand]
The **camera setup wizard** is the recommended way to add cameras. Click **Add Camera** in <NavPath path="Settings > Global configuration > Camera management" />, and the wizard probes your camera and writes its configuration for you — including the go2rtc restream and the live stream mapping — so go2rtc is set up automatically.
This guide is mainly useful if you are **upgrading from an older version and have existing cameras that don't yet use go2rtc**, or if you want to fine-tune a stream by hand (for example, to transcode a codec your browser can't play). The [go2rtc troubleshooting guide](/troubleshooting/go2rtc) applies regardless of how your cameras were added.
:::
## Adding a go2rtc stream manually
If you added your cameras with the wizard, go2rtc is already configured — you can skip straight to [troubleshooting](/troubleshooting/go2rtc). The steps below are for upgrading users with existing cameras that aren't using go2rtc yet, or for anyone who prefers to configure a stream by hand.
Configure go2rtc to connect to your camera by adding the stream you want to use for live view. Avoid changing any other parts of your config at this step. Note that go2rtc supports [many different stream types](https://github.com/AlexxIT/go2rtc/tree/v1.9.13#module-streams), not just rtsp.
:::tip
For the best experience, set the stream name under `go2rtc` to match the name of your camera so that Frigate will automatically map it and be able to use better live view options for the camera.
See [the live view docs](/configuration/live#setting-streams-for-live-ui) for more information.
:::
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > go2rtc Streams" /> and click **Add stream**. Give the stream a name (use the camera's name so Frigate can auto-map it - for example, if your camera's name is `back`, use `back` as the go2rtc stream name), then paste the camera's stream URL into the **Source** field. Save the section.
</TabItem>
<TabItem value="yaml">
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
```
</TabItem>
</ConfigTabs>
After adding this to the config, restart Frigate and try to watch the live stream for a single camera by clicking on it from the dashboard. It should look much clearer and more fluent than the original jsmpeg stream.
### Next steps
1. If the stream you added to go2rtc is also used by Frigate for the `record` or `detect` role, you can migrate your config to pull from the RTSP restream to reduce the number of connections to your camera as shown [here](/configuration/restream#reduce-connections-to-camera).
2. You can [set up WebRTC](/configuration/live#webrtc-extra-configuration) if your camera supports two-way talk. Note that WebRTC only supports specific audio formats and may require opening ports on your router.
3. If your camera supports two-way talk, you must configure your stream with `#backchannel=0` to prevent go2rtc from blocking other applications from accessing the camera's audio output. See [preventing go2rtc from blocking two-way audio](/configuration/restream#two-way-talk-restream) in the restream documentation.
## Troubleshooting
If your stream won't play, has no audio, uses excessive CPU, or otherwise misbehaves, see the dedicated [go2rtc troubleshooting guide](/troubleshooting/go2rtc). It walks through how to isolate where the problem is and covers the most common issues — unsupported codecs, H.265/HEVC, audio, WebRTC and two-way talk, hardware-accelerated transcoding with FFmpeg 8, and camera-specific quirks.
## Homekit Configuration
To add camera streams to Homekit Frigate must be configured in docker to use `host` networking mode. Once that is done, you can use the go2rtc WebUI (accessed via port 1984, which is disabled by default) to share export a camera to Homekit. Any changes made will automatically be saved to `/config/go2rtc_homekit.yml`.

6
docs/docs/configuration/hardware_acceleration_video.md
View File

@ -72,7 +72,7 @@ Frigate can utilize most Intel integrated GPUs and Arc GPUs to accelerate video
:::note
The default driver is `iHD`. You may need to change the driver to `i965` by adding the following environment variable `LIBVA_DRIVER_NAME=i965` to your docker-compose file or [in the `config.yml` for HA App users](advanced.md#environment_vars).
The default driver is `iHD`. You may need to change the driver to `i965` by adding the following environment variable `LIBVA_DRIVER_NAME=i965` to your docker-compose file or [in the `config.yml` for HA App users](advanced/system.md#environment_vars).
See [The Intel Docs](https://www.intel.com/content/www/us/en/support/articles/000005505/processors.html) to figure out what generation your CPU is.
@ -169,7 +169,7 @@ Frigate can utilize modern AMD integrated GPUs and AMD GPUs to accelerate video
### Configuring Radeon Driver
You need to change the driver to `radeonsi` by adding the following environment variable `LIBVA_DRIVER_NAME=radeonsi` to your docker-compose file or [in the `config.yml` for HA App users](advanced.md#environment_vars).
You need to change the driver to `radeonsi` by adding the following environment variable `LIBVA_DRIVER_NAME=radeonsi` to your docker-compose file or [in the `config.yml` for HA App users](advanced/system.md#environment_vars).
### Via VAAPI
@ -193,7 +193,7 @@ ffmpeg:
## NVIDIA GPUs
While older GPUs may work, it is recommended to use modern, supported GPUs. NVIDIA provides a [matrix of supported GPUs and features](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new). If your card is on the list and supports CUVID/NVDEC, it will most likely work with Frigate for decoding. However, you must also use [a driver version that will work with FFmpeg](https://github.com/FFmpeg/nv-codec-headers/blob/master/README). Older driver versions may be missing symbols and fail to work, and older cards are not supported by newer driver versions. The only way around this is to [provide your own FFmpeg](/configuration/advanced#custom-ffmpeg-build) that will work with your driver version, but this is unsupported and may not work well if at all.
While older GPUs may work, it is recommended to use modern, supported GPUs. NVIDIA provides a [matrix of supported GPUs and features](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new). If your card is on the list and supports CUVID/NVDEC, it will most likely work with Frigate for decoding. However, you must also use [a driver version that will work with FFmpeg](https://github.com/FFmpeg/nv-codec-headers/blob/master/README). Older driver versions may be missing symbols and fail to work, and older cards are not supported by newer driver versions. The only way around this is to [provide your own FFmpeg](/configuration/advanced/system#custom-ffmpeg-build) that will work with your driver version, but this is unsupported and may not work well if at all.
A more complete list of cards and their compatible drivers is available in the [driver release readme](https://download.nvidia.com/XFree86/Linux-x86_64/525.85.05/README/supportedchips.html).

58
docs/docs/configuration/live.md
View File

@ -11,7 +11,7 @@ Frigate intelligently displays your camera streams on the Live view dashboard. B
### Live View technologies
Frigate intelligently uses three different streaming technologies to display your camera streams on the dashboard and the single camera view, switching between available modes based on network bandwidth, player errors, or required features like two-way talk. The highest quality and fluency of the Live view requires the bundled `go2rtc` to be configured as shown in the [step by step guide](/guides/configuring_go2rtc).
Frigate intelligently uses three different streaming technologies to display your camera streams on the dashboard and the single camera view, switching between available modes based on network bandwidth, player errors, or required features like two-way talk. The highest quality and fluency of the Live view requires the bundled `go2rtc` to be [configured](/configuration/go2rtc).
The jsmpeg live view will use more browser and client GPU resources. Using go2rtc is highly recommended and will provide a superior experience.
@ -88,8 +88,18 @@ Configure a "friendly name" for your stream followed by the go2rtc stream name.
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Camera configuration > Live playback" />, then select your camera.
- Under **Live stream names**, add entries mapping a friendly name to each go2rtc stream name (e.g., `Main Stream` mapped to `test_cam`, `Sub Stream` mapped to `test_cam_sub`).
1. Navigate to <NavPath path="Settings > Camera configuration > Live playback" /> and select your camera.
2. Under **Live stream names**, click **Add stream** to add a new entry.
3. In the **Stream name** field, enter a friendly name that will appear in the Live UI's stream dropdown (e.g., `Main Stream`).
4. In the **go2rtc stream** field, open the dropdown and select the go2rtc stream this name should map to (e.g., `test_cam`). The dropdown lists every stream configured under `go2rtc.streams`. If the go2rtc stream hasn't been created yet, you can type the name and choose **Use "..."** to save a custom value.
5. Repeat for each additional stream you want to expose (e.g., `Sub Stream` → `test_cam_sub`).
6. Use the trash icon on a row to remove a stream, then **Save** the section.
:::tip
Configure your go2rtc streams first under <NavPath path="Settings > System > go2rtc streams" /> so the dropdown is populated with valid options.
:::
</TabItem>
<TabItem value="yaml">
@ -257,19 +267,47 @@ cameras:
</TabItem>
</ConfigTabs>
### Disabling cameras
### Camera state
Cameras can be temporarily disabled through the Frigate UI and through [MQTT](/integrations/mqtt#frigatecamera_nameenabledset) to conserve system resources. When disabled, Frigate's ffmpeg processes are terminated — recording stops, object detection is paused, and the Live dashboard displays a blank image with a disabled message. Review items, tracked objects, and historical footage for disabled cameras can still be accessed via the UI.
Each camera has three possible states, surfaced as a status selector in **Settings → Global configuration → Camera management**:
:::note
- **On** — streams are processed normally. Object detection, recording, and Live view are active.
- **Off** — Frigate's ffmpeg processes are paused. Recording stops, object detection is paused, and the Live dashboard displays a blank image with a "Camera is off" message. The camera is still visible in the Live dashboard and its past review items, tracked objects, and historical footage remain accessible via the UI. The Off state persists across Frigate restarts via a `.runtime_state.json` file alongside `config.yml` (see [Runtime toggle persistence](#runtime-toggle-persistence)).
- **Disabled** — the change is saved to your configuration file (`enabled: False`). The camera stops immediately, Frigate stops ffmpeg processes, and all live and historical UI elements for the camera are no longer visible but remains retained on disk. The camera is still listed in **Settings → Global configuration → Camera management** so it can be re-enabled. **A restart of Frigate is required to bring a disabled camera back to On.**
Disabling a camera via the Frigate UI or MQTT is temporary and does not persist through restarts of Frigate.
#### Turning a camera on or off
:::
Turning a camera off is temporary and does not require a restart. The available controls are:
For restreamed cameras, go2rtc remains active but does not use system resources for decoding or processing unless there are active external consumers (such as the Advanced Camera Card in Home Assistant using a go2rtc source).
- The power button in the single-camera Live view header
- The right-click context menu on a camera tile on the Live dashboard
- The Camera management settings pane (status set to **Off**)
- The mobile settings drawer on the single-camera Live view (admin users only)
- The [MQTT topic](/integrations/mqtt#frigatecamera_nameenabledset) `frigate/<camera_name>/enabled/set` with payload `ON` or `OFF`
- The Home Assistant integration via the [`camera.turn_on` / `camera.turn_off` actions](/integrations/home-assistant#camera-api)
Note that disabling a camera through the config file (`enabled: False`) removes all related UI elements, including historical footage access. To retain access while disabling the camera, keep it enabled in the config and use the UI or MQTT to disable it temporarily.
#### Disabling a camera
Disabling a camera saves the change to your configuration file. Navigate to **Settings → Global configuration → Camera management** and set the camera's status to **Disabled**. Runtime processing stops immediately; the change persists across restarts.
Re-enabling a disabled camera requires a restart of Frigate so that the ffmpeg processes and other camera-scoped resources can be initialized. The UI will prompt you to restart when you switch a disabled camera back to On.
#### Restream behavior
For both Off and Disabled cameras, go2rtc remains active but does not use system resources for decoding or processing unless there are active external consumers (such as the Advanced Camera Card in Home Assistant using a go2rtc source).
#### Choosing Off versus Disabled
If you want a camera's historical data (review items, tracked objects, footage) to stay accessible in the UI while you stop processing, set the camera to **Off**. If you want the camera fully removed from the Live dashboard, review filters, and other UI surfaces, set it to **Disabled**. The Disabled state still keeps the camera in Camera management so it can be re-enabled later; if you want to remove all traces of a camera including its configuration, delete it via Camera management instead.
#### Runtime toggle persistence
The Live view toggles for **camera on/off**, **detect**, **recordings**, **snapshots**, and **audio detection** — along with the equivalent MQTT `/set` topics — write the new state to `.runtime_state.json` next to your `config.yml`. The file is replayed on Frigate startup so your last-known toggle states survive a restart. Two interactions worth knowing:
- **Settings UI saves win.** When you save a field through **Settings → Global configuration**, the matching entry is cleared from `.runtime_state.json` so the new value in your config file is the durable source.
- **Switching profiles clears all runtime overrides.** Activating or deactivating a [profile](/configuration/profiles) is treated as a deliberate state change, so the file is wiped to avoid stale overrides replaying on top of the new profile.
If you hand-edit `config.yml` while runtime overrides exist, the overrides will still replay on restart. Delete `.runtime_state.json` to reset to the YAML-defined defaults.
### Live player error messages

4
docs/docs/configuration/motion_detection.md
View File

@ -197,3 +197,7 @@ This option is handy when you want to prevent large transient changes from trigg
When the skip threshold is exceeded, **no motion is reported** for that frame, meaning **nothing is recorded** for that frame. That means you can miss something important, like a PTZ camera auto-tracking an object or activity while the camera is moving. If you prefer to guarantee that every frame is saved, leave this unset and accept occasional recordings containing scene noise — they typically only take up a few megabytes and are quick to scan in the timeline UI.
:::
## Reviewing Detected Motion
To review what the detector picked up — or to search past recordings for motion in a specific region — see [Reviewing Motion](/usage/review#reviewing-motion) on the Review page.

90
docs/docs/configuration/object_detectors.md
View File

@ -72,7 +72,7 @@ This does not affect using hardware for accelerating other tasks such as [semant
# Officially Supported Detectors
Frigate provides a number of builtin detector types. By default, Frigate will use a single OpenVINO detector running on the CPU. Other detectors may require additional configuration as described below. When using multiple detectors they will run in dedicated processes, but pull from a common queue of detection requests from across all cameras.
Frigate provides a number of builtin detector types. By default, Frigate will use a single CPU detector. Other detectors may require additional configuration as described below. When using multiple detectors they will run in dedicated processes, but pull from a common queue of detection requests from across all cameras.
## Edge TPU Detector
@ -91,7 +91,7 @@ See [common Edge TPU troubleshooting steps](/troubleshooting/edgetpu) if the Edg
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then set device to `usb`.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then set device to `usb`.
</TabItem>
<TabItem value="yaml">
@ -111,7 +111,7 @@ detectors:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add** to add multiple detectors, specifying `usb:0` and `usb:1` as the device for each.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add** to add multiple detectors, specifying `usb:0` and `usb:1` as the device for each.
</TabItem>
<TabItem value="yaml">
@ -136,7 +136,7 @@ _warning: may have [compatibility issues](https://github.com/blakeblackshear/fri
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then leave the device field empty.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then leave the device field empty.
</TabItem>
<TabItem value="yaml">
@ -156,7 +156,7 @@ detectors:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then set device to `pci`.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then set device to `pci`.
</TabItem>
<TabItem value="yaml">
@ -176,7 +176,7 @@ detectors:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add** to add multiple detectors, specifying `pci:0` and `pci:1` as the device for each.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add** to add multiple detectors, specifying `pci:0` and `pci:1` as the device for each.
</TabItem>
<TabItem value="yaml">
@ -199,7 +199,7 @@ detectors:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add** to add multiple detectors with different device types (e.g., `usb` and `pci`).
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add** to add multiple detectors with different device types (e.g., `usb` and `pci`).
</TabItem>
<TabItem value="yaml">
@ -246,7 +246,7 @@ After placing the downloaded files for the tflite model and labels in your confi
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then set device to `usb`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure the model settings:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **EdgeTPU** from the detector type dropdown and click **Add**, then set device to `usb`. Then on the same page, in the **Custom Model** tab, configure the model settings:
| Field | Value |
| ---------------------------------------- | ----------------------------------------------------------------- |
@ -309,7 +309,7 @@ Use this configuration for YOLO-based models. When no custom model path or URL i
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **Hailo-8/Hailo-8L** from the detector type dropdown and click **Add**, then set device to `PCIe`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure the model settings:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **Hailo-8/Hailo-8L** from the detector type dropdown and click **Add**, then set device to `PCIe`. Then on the same page, in the **Custom Model** tab, configure the model settings:
| Field | Value |
| ---------------------------------------- | ----------------------- |
@ -365,7 +365,7 @@ For SSD-based models, provide either a model path or URL to your compiled SSD mo
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **Hailo-8/Hailo-8L** from the detector type dropdown and click **Add**, then set device to `PCIe`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure the model settings:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **Hailo-8/Hailo-8L** from the detector type dropdown and click **Add**, then set device to `PCIe`. Then on the same page, in the **Custom Model** tab, configure the model settings:
| Field | Value |
| --------------------------------------- | ------ |
@ -410,7 +410,7 @@ The Hailo detector supports all YOLO models compiled for Hailo hardware that inc
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **Hailo-8/Hailo-8L** from the detector type dropdown and click **Add**, then set device to `PCIe`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure the model settings to match your custom model dimensions and format.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **Hailo-8/Hailo-8L** from the detector type dropdown and click **Add**, then set device to `PCIe`. Then on the same page, in the **Custom Model** tab, configure the model settings to match your custom model dimensions and format.
</TabItem>
<TabItem value="yaml">
@ -465,7 +465,7 @@ When using many cameras one detector may not be enough to keep up. Multiple dete
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **OpenVINO** from the detector type dropdown and click **Add** to add multiple detectors, each targeting `GPU` or `NPU`.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **OpenVINO** from the detector type dropdown and click **Add** to add multiple detectors, each targeting `GPU` or `NPU`.
</TabItem>
<TabItem value="yaml">
@ -508,7 +508,7 @@ Use the model configuration shown below when using the OpenVINO detector with th
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU` (or `NPU`). Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU` (or `NPU`). Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------ |
@ -558,7 +558,7 @@ After placing the downloaded onnx model in your config folder, use the following
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------------- |
@ -620,7 +620,7 @@ After placing the downloaded onnx model in your config folder, use the following
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU` (or `NPU`). Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU` (or `NPU`). Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------------------------- |
@ -660,7 +660,7 @@ Note that the labelmap uses a subset of the complete COCO label set that has onl
#### RF-DETR
[RF-DETR](https://github.com/roboflow/rf-detr) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-rf-detr-model) for more informatoin on downloading the RF-DETR model for use in Frigate.
[RF-DETR](https://github.com/roboflow/rf-detr) is a DETR based model. The ONNX exported models are supported, but not included by default. See [the models section](#downloading-rf-detr-model) for more information on downloading the RF-DETR model for use in Frigate.
:::warning
@ -676,7 +676,7 @@ After placing the downloaded onnx model in your `config/model_cache` folder, use
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `GPU`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| --------------------------------------- | --------------------------------- |
@ -728,7 +728,7 @@ After placing the downloaded onnx model in your config/model_cache folder, use t
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `CPU`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **OpenVINO** from the detector type dropdown and click **Add**, then set device to `CPU`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ---------------------------------- |
@ -807,7 +807,7 @@ Using the detector config below will connect to the client:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ZMQ IPC** from the detector type dropdown and click **Add**, then set the endpoint to `tcp://host.docker.internal:5555`.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ZMQ IPC** from the detector type dropdown and click **Add**, then set the endpoint to `tcp://host.docker.internal:5555`.
</TabItem>
<TabItem value="yaml">
@ -841,7 +841,7 @@ When Frigate is started with the following config it will connect to the detecto
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ZMQ IPC** from the detector type dropdown and click **Add**, then set the endpoint to `tcp://host.docker.internal:5555`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ZMQ IPC** from the detector type dropdown and click **Add**, then set the endpoint to `tcp://host.docker.internal:5555`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------------------------- |
@ -1002,7 +1002,7 @@ When using many cameras one detector may not be enough to keep up. Multiple dete
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ONNX** from the detector type dropdown and click **Add** to add multiple detectors.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ONNX** from the detector type dropdown and click **Add** to add multiple detectors.
</TabItem>
<TabItem value="yaml">
@ -1050,7 +1050,7 @@ After placing the downloaded onnx model in your config folder, use the following
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ONNX** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ONNX** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------------- |
@ -1109,7 +1109,7 @@ After placing the downloaded onnx model in your config folder, use the following
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ONNX** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ONNX** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------------------------- |
@ -1158,7 +1158,7 @@ After placing the downloaded onnx model in your config folder, use the following
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ONNX** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ONNX** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------------------------- |
@ -1207,7 +1207,7 @@ After placing the downloaded onnx model in your `config/model_cache` folder, use
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ONNX** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ONNX** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| --------------------------------------- | --------------------------------- |
@ -1252,7 +1252,7 @@ After placing the downloaded onnx model in your `config/model_cache` folder, use
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **ONNX** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **ONNX** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------- |
@ -1328,7 +1328,7 @@ A TensorFlow Lite model is provided in the container at `/cpu_model.tflite` and
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **CPU** from the detector type dropdown and click **Add**. Configure the number of threads and click **Add** again to add additional CPU detectors as needed (one per camera is recommended).
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **CPU** from the detector type dropdown and click **Add**. Configure the number of threads and click **Add** again to add additional CPU detectors as needed (one per camera is recommended).
</TabItem>
<TabItem value="yaml">
@ -1364,7 +1364,7 @@ To integrate CodeProject.AI into Frigate, configure the detector as follows:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **DeepStack** from the detector type dropdown and click **Add**. Set the API URL to point to your CodeProject.AI server (e.g., `http://<your_codeproject_ai_server_ip>:<port>/v1/vision/detection`).
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **DeepStack** from the detector type dropdown and click **Add**. Set the API URL to point to your CodeProject.AI server (e.g., `http://<your_codeproject_ai_server_ip>:<port>/v1/vision/detection`).
</TabItem>
<TabItem value="yaml">
@ -1403,7 +1403,7 @@ To configure the MemryX detector, use the following example configuration:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`.
</TabItem>
<TabItem value="yaml">
@ -1423,7 +1423,7 @@ detectors:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **MemryX** from the detector type dropdown and click **Add** to add multiple detectors, specifying `PCIe:0`, `PCIe:1`, `PCIe:2`, etc. as the device for each.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **MemryX** from the detector type dropdown and click **Add** to add multiple detectors, specifying `PCIe:0`, `PCIe:1`, `PCIe:2`, etc. as the device for each.
</TabItem>
<TabItem value="yaml">
@ -1467,7 +1467,7 @@ Below is the recommended configuration for using the **YOLO-NAS** (small) model
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------------- |
@ -1515,7 +1515,7 @@ Below is the recommended configuration for using the **YOLOv9** (small) model wi
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------------- |
@ -1562,7 +1562,7 @@ Below is the recommended configuration for using the **YOLOX** (small) model wit
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ----------------------- |
@ -1609,7 +1609,7 @@ Below is the recommended configuration for using the **SSDLite MobileNet v2** mo
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **MemryX** from the detector type dropdown and click **Add**, then set device to `PCIe:0`. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ----------------------- |
@ -1768,7 +1768,7 @@ Use the config below to work with generated TRT models:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **TensorRT** from the detector type dropdown and click **Add**, then set the device to `0` (the default GPU index). Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **TensorRT** from the detector type dropdown and click **Add**, then set the device to `0` (the default GPU index). Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ------------------------------------------------------------ |
@ -1825,7 +1825,7 @@ Use the model configuration shown below when using the synaptics detector with t
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **Synaptics** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **Synaptics** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ---------------------------- |
@ -1879,7 +1879,7 @@ When using many cameras one detector may not be enough to keep up. Multiple dete
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **RKNN** from the detector type dropdown and click **Add** to add multiple detectors, each with `num_cores` set to `0` for automatic selection.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **RKNN** from the detector type dropdown and click **Add** to add multiple detectors, each with `num_cores` set to `0` for automatic selection.
</TabItem>
<TabItem value="yaml">
@ -1921,7 +1921,7 @@ This `config.yml` shows all relevant options to configure the detector and expla
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **RKNN** from the detector type dropdown and click **Add**. Set `num_cores` to `0` for automatic selection (increase for better performance on multicore NPUs, e.g., set to `3` on rk3588).
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **RKNN** from the detector type dropdown and click **Add**. Set `num_cores` to `0` for automatic selection (increase for better performance on multicore NPUs, e.g., set to `3` on rk3588).
</TabItem>
<TabItem value="yaml">
@ -1958,7 +1958,7 @@ The inference time was determined on a rk3588 with 3 NPU cores.
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ----------------------------------------------------------------------- |
@ -2004,7 +2004,7 @@ The pre-trained YOLO-NAS weights from DeciAI are subject to their license and ca
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------------------- |
@ -2044,7 +2044,7 @@ model: # required
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ---------------------------------------------- |
@ -2138,7 +2138,7 @@ Once completed, configure the detector as follows:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **DeGirum** from the detector type dropdown and click **Add**. Set the location to your AI server (e.g., service name, container name, or `host:port`), the zoo to `degirum/public`, and provide your authentication token if needed.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **DeGirum** from the detector type dropdown and click **Add**. Set the location to your AI server (e.g., service name, container name, or `host:port`), the zoo to `degirum/public`, and provide your authentication token if needed.
</TabItem>
<TabItem value="yaml">
@ -2181,7 +2181,7 @@ It is also possible to eliminate the need for an AI server and run the hardware
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **DeGirum** from the detector type dropdown and click **Add**. Set the location to `@local`, the zoo to `degirum/public`, and provide your authentication token.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **DeGirum** from the detector type dropdown and click **Add**. Set the location to `@local`, the zoo to `degirum/public`, and provide your authentication token.
</TabItem>
<TabItem value="yaml">
@ -2218,7 +2218,7 @@ If you do not possess whatever hardware you want to run, there's also the option
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **DeGirum** from the detector type dropdown and click **Add**. Set the location to `@cloud`, the zoo to `degirum/public`, and provide your authentication token.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **DeGirum** from the detector type dropdown and click **Add**. Set the location to `@cloud`, the zoo to `degirum/public`, and provide your authentication token.
</TabItem>
<TabItem value="yaml">
@ -2274,7 +2274,7 @@ Use the model configuration shown below when using the axengine detector with th
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and select **AXEngine NPU** from the detector type dropdown and click **Add**. Then navigate to <NavPath path="Settings > System > Detection model" /> and configure:
Navigate to <NavPath path="Settings > System > Detectors and model" /> and select **AXEngine NPU** from the detector type dropdown and click **Add**. Then on the same page, in the **Custom Model** tab, configure:
| Field | Value |
| ---------------------------------------- | ----------------------- |

2
docs/docs/configuration/objects.md
View File

@ -158,4 +158,4 @@ Models for both CPU and EdgeTPU (Coral) are bundled in the image. You can use yo
- EdgeTPU Model: `/edgetpu_model.tflite`
- Labels: `/labelmap.txt`
You also need to update the [model config](advanced.md#model) if they differ from the defaults.
You also need to update the [model config](advanced/system.md#model) if they differ from the defaults.

40
docs/docs/configuration/profiles.md
View File

@ -33,10 +33,10 @@ The easiest way to define profiles is to use the Frigate UI. Profiles can also b
<ConfigTabs>
<TabItem value="ui">
1. **Create a profile** — Navigate to <NavPath path="Settings > Camera configuration > Profiles" />. Click the **Add Profile** button, enter a name (and optionally a profile ID).
1. **Create a profile** — Navigate to <NavPath path="Settings > Global configuration > Profiles" />. Click the **Add Profile** button, enter a name (and optionally a profile ID).
2. **Configure overrides** — Navigate to a camera configuration section (e.g. Motion detection, Record, Notifications). In the top right, two buttons will appear - choose a camera and a profile from the profile selector to edit overrides for that camera and section. Only the fields you change will be stored as overrides — fields that require a restart are hidden since profiles are applied at runtime. You can click the **Remove Profile Override** button to clear overrides.
3. **Activate a profile** — Use the **Profiles** option in Frigate's main menu to choose a profile. Alternatively, in Settings, navigate to <NavPath path="Settings > Camera configuration > Profiles" />, then choose a profile in the Active Profile dropdown to activate it. The active profile is also shown in the status bar at the bottom of the screen on desktop browsers.
4. **Delete a profile** — Navigate to <NavPath path="Settings > Camera configuration > Profiles" />, then click the trash icon for a profile. This removes the profile definition and all camera overrides associated with it.
3. **Activate a profile** — Use the **Profiles** option in Frigate's main menu to choose a profile. Alternatively, in Settings, navigate to <NavPath path="Settings > Global configuration > Profiles" />, then choose a profile in the Active Profile dropdown to activate it. The active profile is also shown in the status bar at the bottom of the screen on desktop browsers.
4. **Delete a profile** — Navigate to <NavPath path="Settings > Global configuration > Profiles" />, then click the trash icon for a profile. This removes the profile definition and all camera overrides associated with it.
</TabItem>
<TabItem value="yaml">
@ -126,7 +126,11 @@ Only the fields you explicitly set in a profile override are applied. All other
## Activating Profiles
Profiles can be activated and deactivated from the Frigate UI. Open the Settings cog and select **Profiles** from the submenu to see all defined profiles. From there you can activate any profile or deactivate the current one. The active profile is indicated in the UI so you always know which profile is in effect.
Profiles can be activated and deactivated via the Frigate UI, [MQTT](/integrations/mqtt#frigateprofileset), or the Home Assistant integration.
In the Frigate UI, open the Settings cog and select **Profiles** from the submenu to see all defined profiles. From there you can activate any profile or deactivate the current one. The active profile is indicated in the UI so you always know which profile is in effect.
Activating or deactivating a profile clears any [runtime toggle overrides](/configuration/live#runtime-toggle-persistence) so the profile's settings aren't silently undone by a stale toggle from before the switch.
## Example: Home / Away Setup
@ -135,10 +139,10 @@ A common use case is having different detection and notification settings based
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Camera configuration > Profiles" /> and create two profiles: **Home** and **Away**.
1. Navigate to <NavPath path="Settings > Global configuration > Profiles" /> and create two profiles: **Home** and **Away**.
2. From to the Camera configuration section in Settings, choose the **front_door** camera, and select the **Away** profile from the profile dropdown. Then, enable notifications from the Notifications pane, and set alert labels to `person` and `car` from the Review pane. Then, from the profile dropdown choose **Home** profile, then navigate to Notifications to disable notifications.
3. For the **indoor_cam** camera, perform similar steps - configure the **Away** profile to enable the camera, detection, and recording. Configure the **Home** profile to disable the camera entirely for privacy.
4. Activate the desired profile from <NavPath path="Settings > Camera configuration > Profiles" /> or from the **Profiles** option in Frigate's main menu.
4. Activate the desired profile from <NavPath path="Settings > Global configuration > Profiles" /> or from the **Profiles** option in Frigate's main menu.
</TabItem>
<TabItem value="yaml">
@ -207,3 +211,27 @@ In this example:
- **Away profile**: The front door camera enables notifications and tracks specific alert labels. The indoor camera is fully enabled with detection and recording.
- **Home profile**: The front door camera disables notifications. The indoor camera is completely disabled for privacy.
- **No profile active**: All cameras use their base configuration values.
## FAQ
### Can I define a zone or mask in a profile but not have it in the base config?
No. Profiles are pure overrides. Every zone and mask defined under a profile must reference an entry that already exists on the base camera config. Configurations that introduce profile-only zones or masks are rejected at startup.
If you want a zone or mask to be active only under a specific profile, define it on the base config with `enabled: false`, then enable it in that profile's overrides.
### How do I revert a profile zone or mask override back to the base configuration?
Delete the override. In the Frigate UI, edit the profile and use the "Revert override" action (the trash can icon) on the zone or mask. The base entry is left untouched, and once the override is removed the profile inherits the base values for that zone or mask.
### Can multiple profiles be active at the same time?
No. Only one profile can be active at a time. Activating a new profile automatically deactivates the current one.
### What happens to my profile overrides if I delete a zone or mask from the base?
When you delete a base zone or mask in the Frigate UI, any profile overrides for that entry are deleted automatically as part of the same operation. If you remove a base entry by editing your config file directly and leave a profile override behind, the config will fail validation at startup until the orphaned override is removed as well.
### Why are some settings missing when I configure a profile override?
Fields that require a Frigate restart to take effect cannot be overridden by profiles, since profiles are applied at runtime without restarting. Those fields are hidden when editing a profile override and can only be changed on the base configuration.

6
docs/docs/configuration/record.md
View File

@ -11,6 +11,12 @@ Recordings can be enabled and are stored at `/media/frigate/recordings`. The fol
New recording segments are written from the camera stream to cache, they are only moved to disk if they match the setup recording retention policy.
:::tip
To keep a specific clip beyond your retention window, [export](/usage/exports) it rather than increasing retention for the whole camera. Exports are saved separately and are never removed by retention.
:::
H265 recordings can be viewed in Chrome 108+, Edge and Safari only. All other browsers require recordings to be encoded with H264.
## Common recording configurations

6
docs/docs/configuration/review.md
View File

@ -23,7 +23,7 @@ In 0.14 and later, all of that is bundled into a single review item which starts
## Alerts and Detections
Not every segment of video captured by Frigate may be of the same level of interest to you. Video of people who enter your property may be a different priority than those walking by on the sidewalk. For this reason, Frigate 0.14 categorizes review items as _alerts_ and _detections_. By default, all person and car objects are considered alerts. You can refine categorization of your review items by configuring required zones for them.
Not every segment of video captured by Frigate may be of the same level of interest to you. Video of people who enter your property may be a different priority than those walking by on the sidewalk. For this reason, Frigate categorizes review items as _alerts_ and _detections_. By default, all person and car objects are considered alerts. You can refine categorization of your review items by configuring required zones for them.
:::note
@ -130,3 +130,7 @@ By default a review item will be created if any `review -> alerts -> labels` and
Because zones don't apply to audio, audio labels will always be marked as a detection by default.
:::
## Reviewing Motion
The Review page can also surface periods of motion that didn't produce a tracked object, and lets you search past recordings for motion in a region you draw. See [Reviewing Motion](/usage/review#reviewing-motion) in the Usage docs for how to use **Motion Previews** and **Motion Search**, and [Tuning Motion Detection](motion_detection.md) for configuring the underlying motion detector.

7
docs/docs/configuration/semantic_search.md
View File

@ -222,12 +222,7 @@ See the [Hardware Accelerated Enrichments](/configuration/hardware_acceleration_
## Usage and Best Practices
1. Semantic Search is used in conjunction with the other filters available on the Explore page. Use a combination of traditional filtering and Semantic Search for the best results.
2. Use the thumbnail search type when searching for particular objects in the scene. Use the description search type when attempting to discern the intent of your object.
3. Because of how the AI models Frigate uses have been trained, the comparison between text and image embedding distances generally means that with multi-modal (`thumbnail` and `description`) searches, results matching `description` will appear first, even if a `thumbnail` embedding may be a better match. Play with the "Search Type" setting to help find what you are looking for. Note that if you are generating descriptions for specific objects or zones only, this may cause search results to prioritize the objects with descriptions even if the the ones without them are more relevant.
4. Make your search language and tone closely match exactly what you're looking for. If you are using thumbnail search, **phrase your query as an image caption**. Searching for "red car" may not work as well as "red sedan driving down a residential street on a sunny day".
5. Semantic search on thumbnails tends to return better results when matching large subjects that take up most of the frame. Small things like "cat" tend to not work well.
6. Experiment! Find a tracked object you want to test and start typing keywords and phrases to see what works for you.
For tips on getting the best results from Semantic Search — choosing between thumbnail and description search, phrasing queries effectively, and combining search with the other Explore filters — see [Usage and best practices](/usage/explore#usage-and-best-practices) in the Usage docs.
## Triggers

6
docs/docs/frigate/installation.md
View File

@ -600,7 +600,7 @@ There are several variants of the App available:
If you are using hardware acceleration for ffmpeg, you **may** need to use the _Full Access_ variant of the App. This is because the Frigate App runs in a container with limited access to the host system. The _Full Access_ variant allows you to disable _Protection mode_ and give Frigate full access to the host system.
You can also edit the Frigate configuration file through the [VS Code App](https://github.com/hassio-addons/addon-vscode) or similar. In that case, the configuration file will be at `/addon_configs/<addon_directory>/config.yml`, where `<addon_directory>` is specific to the variant of the Frigate App you are running. See the list of directories [here](../configuration/index.md#accessing-app-config-dir).
You can also edit the Frigate configuration file through the [VS Code App](https://github.com/hassio-addons/addon-vscode) or similar. In that case, the configuration file will be at `/addon_configs/<addon_directory>/config.yml`, where `<addon_directory>` is specific to the variant of the Frigate App you are running. See the list of directories [here](../configuration/config.md#accessing-app-config-dir).
## Kubernetes
@ -749,7 +749,7 @@ Failure to remap port 5000 on the host will result in the WebUI and all API endp
:::
Docker containers on macOS can be orchestrated by either [Docker Desktop](https://docs.docker.com/desktop/setup/install/mac-install/) or [OrbStack](https://orbstack.dev) (native swift app). The difference in inference speeds is negligable, however CPU, power consumption and container start times will be lower on OrbStack because it is a native Swift application.
Docker containers on macOS can be orchestrated by either [Docker Desktop](https://docs.docker.com/desktop/setup/install/mac-install/) or [OrbStack](https://orbstack.dev) (native Swift app). The difference in inference speeds is negligible, however CPU, power consumption and container start times will be lower on OrbStack because it is a native Swift application.
To allow Frigate to use the Apple Silicon Neural Engine / Processing Unit (NPU) the host must be running [Apple Silicon Detector](../configuration/object_detectors.md#apple-silicon-detector) on the host (outside Docker)
@ -768,7 +768,7 @@ services:
- /path/to/your/recordings:/recordings
ports:
- "8971:8971"
# If exposing on macOS map to a diffent host port like 5001 or any orher port with no conflicts
# If exposing on macOS map to a different host port like 5001 or any other port with no conflicts
# - "5001:5000" # Internal unauthenticated access. Expose carefully.
- "8554:8554" # RTSP feeds
extra_hosts:

2
docs/docs/frigate/network_requirements.md
View File

@ -32,7 +32,7 @@ The following models are downloaded automatically the first time their associate
| [License plate recognition](/configuration/license_plate_recognition) | PaddleOCR (detection, classification, recognition) + YOLOv9 plate detector | GitHub |
| [Bird classification](/configuration/bird_classification) | MobileNetV2 bird model + label map | GitHub |
| [Custom classification](/configuration/custom_classification/state_classification) (training) | MobileNetV2 ImageNet base weights (via Keras) | Google storage |
| [Audio transcription](/configuration/advanced) | Whisper or Sherpa-ONNX streaming model | HuggingFace / OpenAI |
| [Audio transcription](/configuration/advanced/system) | Whisper or Sherpa-ONNX streaming model | HuggingFace / OpenAI |
### Hardware-Specific Detector Models

116
docs/docs/guides/configuring_go2rtc.md
View File

@ -1,116 +0,0 @@
---
id: configuring_go2rtc
title: Configuring go2rtc
---
Use of the bundled go2rtc is optional. You can still configure FFmpeg to connect directly to your cameras. However, adding go2rtc to your configuration is required for the following features:
- WebRTC or MSE for live viewing with audio, higher resolutions and frame rates than the jsmpeg stream which is limited to the detect stream and does not support audio
- Live stream support for cameras in Home Assistant Integration
- RTSP relay for use with other consumers to reduce the number of connections to your camera streams
## Setup a go2rtc stream
First, you will want to configure go2rtc to connect to your camera stream by adding the stream you want to use for live view in your Frigate config file. Avoid changing any other parts of your config at this step. Note that go2rtc supports [many different stream types](https://github.com/AlexxIT/go2rtc/tree/v1.9.13#module-streams), not just rtsp.
:::tip
For the best experience, you should set the stream name under `go2rtc` to match the name of your camera so that Frigate will automatically map it and be able to use better live view options for the camera.
See [the live view docs](../configuration/live.md#setting-streams-for-live-ui) for more information.
:::
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
```
After adding this to the config, restart Frigate and try to watch the live stream for a single camera by clicking on it from the dashboard. It should look much clearer and more fluent than the original jsmpeg stream.
### What if my video doesn't play?
- Check Logs:
- Access the go2rtc logs in the Frigate UI under Logs in the sidebar.
- If go2rtc is having difficulty connecting to your camera, you should see some error messages in the log.
- Check go2rtc Web Interface: if you don't see any errors in the logs, try viewing the camera through go2rtc's web interface.
- Navigate to port 1984 in your browser to access go2rtc's web interface.
- If using Frigate through Home Assistant, enable the web interface at port 1984.
- If using Docker, forward port 1984 before accessing the web interface.
- Click `stream` for the specific camera to see if the camera's stream is being received.
- Check Video Codec:
- If the camera stream works in go2rtc but not in your browser, the video codec might be unsupported.
- If using H265, switch to H264. Refer to [video codec compatibility](https://github.com/AlexxIT/go2rtc/tree/v1.9.13#codecs-madness) in go2rtc documentation.
- If unable to switch from H265 to H264, or if the stream format is different (e.g., MJPEG), re-encode the video using [FFmpeg parameters](https://github.com/AlexxIT/go2rtc/tree/v1.9.13#source-ffmpeg). It supports rotating and resizing video feeds and hardware acceleration. Keep in mind that transcoding video from one format to another is a resource intensive task and you may be better off using the built-in jsmpeg view.
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
- "ffmpeg:back#video=h264#hardware"
```
- Switch to FFmpeg if needed:
- Some camera streams may need to use the ffmpeg module in go2rtc. This has the downside of slower startup times, but has compatibility with more stream types.
```yaml
go2rtc:
streams:
back:
- ffmpeg:rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
```
- If you can see the video but do not have audio, this is most likely because your camera's audio stream codec is not AAC.
- If possible, update your camera's audio settings to AAC in your camera's firmware.
- If your cameras do not support AAC audio, you will need to tell go2rtc to re-encode the audio to AAC on demand if you want audio. This will use additional CPU and add some latency. To add AAC audio on demand, you can update your go2rtc config as follows:
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
- "ffmpeg:back#audio=aac"
```
If you need to convert **both** the audio and video streams, you can use the following:
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
- "ffmpeg:back#video=h264#audio=aac#hardware"
```
When using the ffmpeg module, you would add AAC audio like this:
```yaml
go2rtc:
streams:
back:
- "ffmpeg:rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2#video=copy#audio=copy#audio=aac#hardware"
```
:::warning
To access the go2rtc stream externally when utilizing the Frigate App (for
instance through VLC), you must first enable the RTSP Restream port.
You can do this by visiting the Frigate App configuration page within Home
Assistant and revealing the hidden options under the "Show disabled ports"
section.
:::
### Next steps
1. If the stream you added to go2rtc is also used by Frigate for the `record` or `detect` role, you can migrate your config to pull from the RTSP restream to reduce the number of connections to your camera as shown [here](/configuration/restream#reduce-connections-to-camera).
2. You can [set up WebRTC](/configuration/live#webrtc-extra-configuration) if your camera supports two-way talk. Note that WebRTC only supports specific audio formats and may require opening ports on your router.
3. If your camera supports two-way talk, you must configure your stream with `#backchannel=0` to prevent go2rtc from blocking other applications from accessing the camera's audio output. See [preventing go2rtc from blocking two-way audio](/configuration/restream#two-way-talk-restream) in the restream documentation.
## Homekit Configuration
To add camera streams to Homekit Frigate must be configured in docker to use `host` networking mode. Once that is done, you can use the go2rtc WebUI (accessed via port 1984, which is disabled by default) to share export a camera to Homekit. Any changes made will automatically be saved to `/config/go2rtc_homekit.yml`.

25
docs/docs/guides/getting_started.md
View File

@ -144,7 +144,7 @@ At this point you should be able to start Frigate and a basic config will be cre
### Step 2: Add a camera
Click the **Add Camera** button in <NavPath path="Settings > Camera configuration > Management" /> to use the camera setup wizard to get your first camera added into Frigate.
Click the **Add Camera** button in <NavPath path="Settings > Global configuration > Camera management" /> to use the camera setup wizard to get your first camera added into Frigate.
### Step 3: Configure hardware acceleration (recommended)
@ -204,8 +204,8 @@ You need to refer to **Configure hardware acceleration** above to enable the con
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > System > Detector hardware" /> and add a detector with **Type** `OpenVINO` and **Device** `GPU`
2. Navigate to <NavPath path="Settings > System > Detection model" /> and configure the model settings for OpenVINO:
1. Navigate to <NavPath path="Settings > System > Detectors and model" /> and add a detector with **Type** `OpenVINO` and **Device** `GPU`
2. On the same page, in the **Custom Model** tab, configure the model settings for OpenVINO:
| Field | Value |
| ---------------------------------------- | ------------------------------------------ |
@ -273,7 +273,7 @@ services:
<ConfigTabs>
<TabItem value="ui">
Navigate to <NavPath path="Settings > System > Detector hardware" /> and add a detector with **Type** `EdgeTPU` and **Device** `usb`.
Navigate to <NavPath path="Settings > System > Detectors and model" /> and add a detector with **Type** `EdgeTPU` and **Device** `usb`.
</TabItem>
<TabItem value="yaml">
@ -301,7 +301,7 @@ cameras:
More details on available detectors can be found [here](../configuration/object_detectors.md).
Restart Frigate and you should start seeing detections for `person`. If you want to track other objects, they can be configured in <NavPath path="Settings > Global configuration > Objects" /> or via the [configuration file reference](../configuration/reference.md).
Restart Frigate and you should start seeing detections for `person`. If you want to track other objects, they can be configured in <NavPath path="Settings > Global configuration > Objects" /> or via the [configuration file reference](../configuration/advanced/reference.md).
### Step 5: Setup motion masks
@ -388,21 +388,20 @@ If you only plan to use Frigate for recording, it is still recommended to define
:::
By default, Frigate will retain video of all tracked objects for 10 days. The full set of options for recording can be found [here](../configuration/reference.md).
By default, Frigate will retain video of all tracked objects for 10 days. The full set of options for recording can be found [here](../configuration/advanced/reference.md).
### Step 7: Complete config
At this point you have a complete config with basic functionality.
- View [common configuration examples](../configuration/index.md#common-configuration-examples) for a list of common configuration examples.
- View [full config reference](../configuration/reference.md) for a complete list of configuration options.
- View [common configuration examples](../configuration/config.md#common-configuration-examples) for a list of common configuration examples.
- View [full config reference](../configuration/advanced/reference.md) for a complete list of configuration options.
### Follow up
Now that you have a working install, you can use the following documentation for additional features:
1. [Configuring go2rtc](configuring_go2rtc.md) - Additional live view options and RTSP relay
2. [Zones](../configuration/zones.md)
3. [Review](../configuration/review.md)
4. [Masks](../configuration/masks.md)
5. [Home Assistant Integration](../integrations/home-assistant.md) - Integrate with Home Assistant
1. [Zones](../configuration/zones.md)
2. [Review](../configuration/review.md)
3. [Masks](../configuration/masks.md)
4. [Home Assistant Integration](../integrations/home-assistant.md) - Integrate with Home Assistant

2
docs/docs/guides/reverse_proxy.md
View File

@ -12,7 +12,7 @@ Before setting up a reverse proxy, check if any of the built-in functionality in
|-|-|
|TLS|Please see the `tls` [configuration option](../configuration/tls.md)|
|Authentication|Please see the [authentication](../configuration/authentication.md) documentation|
|IPv6|[Enabling IPv6](../configuration/advanced.md#enabling-ipv6)
|IPv6|[Enabling IPv6](../configuration/advanced/system.md#enabling-ipv6)
**Note about TLS**
When using a reverse proxy, the TLS session is usually terminated at the proxy, sending the internal request over plain HTTP. If this is the desired behavior, TLS must first be disabled in Frigate, or you will encounter an HTTP 400 error: "The plain HTTP request was sent to HTTPS port."

10
docs/docs/integrations/home-assistant.md
View File

@ -195,7 +195,7 @@ For clips to be castable to media devices, audio is required and may need to be
## Camera API
To disable a camera dynamically
To turn a camera off (pauses Frigate's processing of the stream; does not persist across Frigate restarts; see [Camera state](/configuration/live#camera-state)):
```
action: camera.turn_off
@ -204,7 +204,7 @@ target:
entity_id: camera.back_deck_cam # your Frigate camera entity ID
```
To enable a camera that has been disabled dynamically
To turn a camera back on:
```
action: camera.turn_on
@ -213,6 +213,12 @@ target:
entity_id: camera.back_deck_cam # your Frigate camera entity ID
```
:::note
These actions toggle Frigate's runtime On/Off state. To permanently disable a camera, set its status to **Disabled** in **Settings → Camera Management** in the Frigate UI.
:::
## Notification API
Many people do not want to expose Frigate to the web, so the integration creates some public API endpoints that can be used for notifications.

14
docs/docs/integrations/mqtt.md
View File

@ -306,7 +306,7 @@ Publishes the current health status of each role that is enabled (`audio`, `dete
- `online`: Stream is running and being processed
- `offline`: Stream is offline and is being restarted
- `disabled`: Camera is currently disabled
- `disabled`: Camera is currently turned off (either at runtime via the `enabled/set` topic, or persistently via the configuration file). See [Camera state](/configuration/live#camera-state) for the distinction.
### `frigate/<camera_name>/<object_name>`
@ -368,15 +368,15 @@ The published value is the detected state class name (e.g., `open`, `closed`, `o
### `frigate/<camera_name>/enabled/set`
Topic to turn Frigate's processing of a camera on and off. Expected values are `ON` and `OFF`.
Topic to turn Frigate's processing of a camera on or off at runtime. Expected values are `ON` and `OFF`. The change is persisted across Frigate restarts (see [Runtime toggle persistence](/configuration/live#runtime-toggle-persistence)). To permanently change the configured value, use **Settings → Global configuration → Camera management** in the Frigate UI. See [Camera state](/configuration/live#camera-state) for the difference between turning a camera off and disabling it.
### `frigate/<camera_name>/enabled/state`
Topic with current state of processing for a camera. Published values are `ON` and `OFF`.
Topic with current runtime state of processing for a camera. Published values are `ON` and `OFF`.
### `frigate/<camera_name>/detect/set`
Topic to turn object detection for a camera on and off. Expected values are `ON` and `OFF`.
Topic to turn object detection for a camera on and off. Expected values are `ON` and `OFF`. The change is persisted across Frigate restarts (see [Runtime toggle persistence](/configuration/live#runtime-toggle-persistence)).
### `frigate/<camera_name>/detect/state`
@ -384,7 +384,7 @@ Topic with current state of object detection for a camera. Published values are
### `frigate/<camera_name>/audio/set`
Topic to turn audio detection for a camera on and off. Expected values are `ON` and `OFF`.
Topic to turn audio detection for a camera on and off. Expected values are `ON` and `OFF`. The change is persisted across Frigate restarts (see [Runtime toggle persistence](/configuration/live#runtime-toggle-persistence)).
### `frigate/<camera_name>/audio/state`
@ -392,7 +392,7 @@ Topic with current state of audio detection for a camera. Published values are `
### `frigate/<camera_name>/recordings/set`
Topic to turn recordings for a camera on and off. Expected values are `ON` and `OFF`.
Topic to turn recordings for a camera on and off. Expected values are `ON` and `OFF`. The change is persisted across Frigate restarts (see [Runtime toggle persistence](/configuration/live#runtime-toggle-persistence)).
### `frigate/<camera_name>/recordings/state`
@ -400,7 +400,7 @@ Topic with current state of recordings for a camera. Published values are `ON` a
### `frigate/<camera_name>/snapshots/set`
Topic to turn snapshots for a camera on and off. Expected values are `ON` and `OFF`.
Topic to turn snapshots for a camera on and off. Expected values are `ON` and `OFF`. The change is persisted across Frigate restarts (see [Runtime toggle persistence](/configuration/live#runtime-toggle-persistence)).
### `frigate/<camera_name>/snapshots/state`

4
docs/docs/integrations/plus.md
View File

@ -3,6 +3,8 @@ id: plus
title: Frigate+
---
import NavPath from "@site/src/components/NavPath";
For more information about how to use Frigate+ to improve your model, see the [Frigate+ docs](/plus/).
:::info
@ -57,7 +59,7 @@ You can view all of your submitted images at [https://plus.frigate.video](https:
Once you have [requested your first model](../plus/first_model.md) and gotten your own model ID, it can be used with a special model path. No other information needs to be configured for Frigate+ models because it fetches the remaining config from Frigate+ automatically.
You can either choose the new model from the Frigate+ pane in the Settings page of the Frigate UI, or manually set the model at the root level in your config:
You can either choose the new model from the <NavPath path="Settings > System > Detectors and model" /> pane in the Frigate UI (the **Frigate+ Model** tab), or manually set the model at the root level in your config:
```yaml
detectors: ...

2
docs/docs/troubleshooting/cpu.md
View File

@ -24,7 +24,7 @@ Video decoding is one of the most CPU-intensive tasks in Frigate. While an AI ac
### Configuration
Frigate provides preset configurations for common hardware acceleration scenarios. Set up `hwaccel_args` based on your hardware in your [configuration](../configuration/reference) as described in the [getting started guide](../guides/getting_started).
Frigate provides preset configurations for common hardware acceleration scenarios. Set up `hwaccel_args` based on your hardware in your [configuration](../configuration/advanced/reference) as described in the [getting started guide](../guides/getting_started).
### Troubleshooting Hardware Acceleration

18
docs/docs/troubleshooting/dummy-camera.md
View File

@ -3,6 +3,8 @@ id: dummy-camera
title: Analyzing Object Detection
---
import NavPath from "@site/src/components/NavPath";
Frigate provides several tools for investigating object detection and tracking behavior: reviewing recorded detections through the UI, using the built-in Debug Replay feature, and manually setting up a dummy camera for advanced scenarios.
## Reviewing Detections in the UI
@ -37,6 +39,8 @@ The per-clip variation is typically quite low and is mostly an artifact of keyfr
Debug Replay lets you re-run Frigate's detection pipeline against a section of recorded video without manually configuring a dummy camera. It automatically extracts the recording, creates a temporary camera with the same detection settings as the original, and loops the clip through the pipeline so you can observe detections in real time.
Debug Replay isn't intended to be a one-stop pane for all Frigate diagnostics or a comprehensive debugging environment for every Frigate feature. It merely makes it easier to spin up a "dummy camera" and perform some common adjustments in real-time. You'll still need to use the normal tools (logs, an MQTT client, etc) to debug your feature.
### When to use
- Reproducing a detection or tracking issue from a specific time range
@ -49,11 +53,25 @@ Only one replay session can be active at a time. If a session is already running
:::
### Starting Debug Replay
Debug Replay can be started from several places in the UI. The starting point determines the time range that gets replayed.
- **History — Actions menu.** Navigate to <NavPath path="History > {camera}" />, open the **Actions** menu in the toolbar, and choose **Debug Replay**. From here you can pick a preset (**Last 1 Minute**, **Last 5 Minutes**), select a range directly on the timeline with **From Timeline**, or enter exact start and end times with **Custom**. This is the most flexible option and the best choice when you want to add padding around a detection. On mobile, the same options appear in the Actions drawer.
- **History — Detail Stream event menu.** While viewing a review item in the Detail Stream, open the menu on a tracked object's event card and choose **Debug Replay**. The replay range is set automatically to that object's start and end times.
- **Explore — search result menu.** From an Explore card, open the kebab menu and choose **Debug Replay**. The range is taken from the tracked object's lifecycle.
- **Explore — Tracking Details Actions menu.** Open a tracked object's **Tracking Details** dialog, then choose **Debug Replay** from the Actions menu. Same automatic range as the search result menu.
- **Exports — export card menu.** From <NavPath path="Exports" />, open the menu on an export and choose **Debug Replay** to loop the exported clip through the detection pipeline for the camera it was exported from.
The Detail Stream, Explore, and Exports entry points use the underlying recording or export's bounds with a small amount of padding. This can be convenient for quick checks, but if a detection is short or you want extra "settle" time for motion and the detector, start the replay from the History Actions menu instead and widen the range manually.
### Variables to consider
- The replay will not always produce identical results to the original run. Different frames may be selected on replay, which can change detections and tracking.
- Motion detection depends on the exact frames used; small frame shifts can change motion regions and therefore what gets passed to the detector.
- Object detection is not fully deterministic: models and post-processing can yield slightly different results across runs.
- In cases where a detection is short and a replay may only be a small number of frames, it is recommended to manually add some padding before and after the detection so that the motion and object detectors have time to settle into the scene. Rather than starting Debug Replay from Explore, navigate to History for your camera, choose Debug Replay from the Actions menu, and click the "From Timeline" or "Custom" option.
- The replay camera inherits the source camera's zones. Any automations that trigger on those zone names will fire for the replay camera as well. This can be helpful when debugging zone behavior, but may be unexpected. You can add a condition on the source camera's name in your automation if you want to exclude replay triggers.
Treat the replay as a close approximation rather than an exact reproduction. Run multiple loops and examine the debug overlays and logs to understand the behavior.

2
docs/docs/troubleshooting/faqs.md
View File

@ -55,7 +55,7 @@ If you see repeated "On connect called" messages in your logs, check for another
### Error: Database Is Locked
SQLite does not work well on a network share, if the `/media` folder is mapped to a network share then [this guide](../configuration/advanced.md#database) should be used to move the database to a location on the internal drive.
SQLite does not work well on a network share, if the `/media` folder is mapped to a network share then [this guide](../configuration/advanced/system.md#database) should be used to move the database to a location on the internal drive.
### Unable to publish to MQTT: client is not connected

235
docs/docs/troubleshooting/go2rtc.md Normal file
View File

@ -0,0 +1,235 @@
---
id: go2rtc
title: Troubleshooting go2rtc
---
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
This page covers common problems with the bundled [go2rtc](/configuration/go2rtc) and how to resolve them, whether your cameras were added with the setup wizard or configured by hand.
When a stream won't play or behaves oddly, the most important first step is to figure out **where** in the pipeline it breaks. Frigate's live view is a chain — _camera → go2rtc → your browser_ — and each stage fails for different reasons. Work through the checks below in order, then jump to the matching problem category.
## Start by isolating the problem
### 1. Read the go2rtc logs
Access the go2rtc logs in the Frigate UI under <NavPath path="System Logs" /> in the sidebar (select the **go2rtc** tab). If go2rtc cannot connect to your camera you will usually see a clear error here — `401 Unauthorized` (bad or incorrectly encoded credentials), `Connection refused` / `timeout` (wrong IP, port, or the camera is at its connection limit), or `404 Not Found` (wrong RTSP path, or the referenced stream name does not exist).
### 2. Test the stream in the go2rtc web interface
If the logs look clean, open go2rtc's own web interface on port `1984`. This is the single most useful diagnostic, because it takes Frigate's UI out of the equation entirely.
- If using Frigate through Home Assistant, enable the web interface at port `1984` (it is disabled by default — see [Home Assistant ports](#home-assistant-and-port-access)).
- If using Docker, forward port `1984` before accessing the web interface.
Open the stream page for your camera (`http://<frigate_host>:1984/stream.html?src=back`) and try each player link:
- **If nothing plays here**, the problem is between the camera and go2rtc (codec, credentials, or transport), _not_ your browser. Fix it at the source before touching anything in Frigate.
- **If a player works here but Frigate's live view does not**, the problem is browser/codec related — compare the **MSE** and **WebRTC** links. Frigate prefers MSE and only attempts WebRTC when MSE fails (or for two-way talk). If `mode=mse` plays but `mode=webrtc` does not, you have a [WebRTC codec problem](#webrtc-and-two-way-talk); if neither plays, your browser cannot decode the codec (commonly H.265 — see [H.265 / HEVC cameras](#h265--hevc-cameras)).
### 3. Inspect the negotiated codecs
You can view detailed stream info — including the exact video and audio codecs go2rtc negotiated with the camera — at `http://frigate_ip:5000/api/go2rtc/streams` (or `http://frigate_ip:5000/api/go2rtc/streams/back` for a single camera). This is the authoritative answer to "what is my camera actually sending?" and is far more reliable than guessing from the camera's web UI. It also shows whether the audio track is `sendonly`/`recvonly`, which matters for [two-way talk](#webrtc-and-two-way-talk).
### 4. Fix the codec with the FFmpeg module
If the camera plays in go2rtc but not in your browser, the video or audio codec is unsupported. Browsers can reliably play **H.264** video and **AAC** audio; many cannot play H.265/HEVC, and some camera audio (G.711/PCM, MJPEG containers, etc.) is not playable at all. The fix is to have go2rtc re-encode the stream on demand using its FFmpeg module.
In the Frigate UI this is the **Use compatibility mode (ffmpeg)** toggle on a stream source; in YAML it is the `ffmpeg:` prefix on the source URL.
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > System > go2rtc Streams" /> and expand your camera's stream.
2. On the source you want to convert, click the **Use compatibility mode (ffmpeg)** button (the sliders icon next to the URL). This routes the source through go2rtc's FFmpeg module and reveals the transcoding options.
3. Set **Video** to **Transcode to H.264** if your browser can't play the camera's video codec (e.g. H.265). Leave it on **Copy** to pass the video through untouched — this is much cheaper and should be your default whenever only the audio needs converting.
4. Set **Audio** to **Transcode to AAC** (for MSE) or **Transcode to Opus** (for WebRTC) if the camera's audio codec is unsupported. Leave it on **Copy** to keep the original, or **Exclude** to drop audio entirely.
5. When transcoding **video**, set **Hardware acceleration** to **Automatic (recommended)** so the encode runs on your GPU instead of the CPU. See [hardware-accelerated transcoding](#hardware-accelerated-transcoding-with-ffmpeg-8) for an important FFmpeg 8 caveat.
6. **Save** the section, then reload the live view.
</TabItem>
<TabItem value="yaml">
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
# transcode video to H.264 on the GPU; only needed if the browser can't play the source codec
- "ffmpeg:back#video=h264#hardware"
```
To convert audio only (leaving video untouched), or to convert both:
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
- "ffmpeg:back#audio=aac" # audio only — preferred when the video already plays
# or, to convert both video and audio:
# - "ffmpeg:back#video=h264#audio=aac#hardware"
```
</TabItem>
</ConfigTabs>
:::warning
The `#`-modifiers (`#video=`, `#audio=`, `#hardware`, `#backchannel=0`, …) **only take effect on a source that is prefixed with `ffmpeg:`**. Adding them to a bare `rtsp://…#audio=opus` source does nothing — go2rtc ignores them. Likewise, when a source references another stream by name (e.g. `ffmpeg:back#audio=aac`), the name must match the stream key **exactly** (it is case sensitive), or the transcode is silently never produced. This is the single most common configuration mistake. In the Frigate UI, the **Use compatibility mode (ffmpeg)** toggle adds the `ffmpeg:` prefix for you.
:::
Transcoding video is resource intensive. Always prefer `#video=copy` (the **Copy** option) and only convert the track that is actually unsupported. If you must transcode video and have no hardware encoder available, the built-in jsmpeg view may be the better option.
## Live view is black, buffering, or stuck in "low-bandwidth mode"
When the live view shows a black screen, spins forever, or repeatedly drops to the lower-quality jsmpeg player ("low-bandwidth mode"), the stream almost always contains something the browser cannot decode over MSE — usually H.265 video or a non-AAC audio track. Confirm this in the go2rtc web UI (port `1984`): if MSE won't play there, Frigate can't play it either, since it uses the same pipeline.
The fix is to produce an **H.264 + AAC** stream, either by changing your camera's firmware codecs or by transcoding in go2rtc (see [Fix the codec with the FFmpeg module](#4-fix-the-codec-with-the-ffmpeg-module)). A few other things worth checking:
- **Set the camera's I-frame (keyframe) interval to match its frame rate** (or "1x" on Reolink), and avoid "smart"/"+" codecs like _H.264+_ or _H.265+_. A long keyframe interval delays the first decodable frame past Frigate's startup timeout, which forces the fallback to jsmpeg. See [camera settings recommendations](/configuration/live#camera-settings-recommendations).
- **A spinner that never clears, even though video plays in VLC**, is often an unplayable _audio_ track stalling playback. Drop or transcode the audio (see below).
- **Remote/VPN viewing that buffers** while the LAN is fine is usually latency/jitter exceeding MSE's startup buffer — set up [WebRTC](/configuration/live#webrtc-extra-configuration), which drops late frames instead of buffering.
The general live-view behavior (smart streaming, the MSE → WebRTC → jsmpeg fallback chain, and how to read browser console errors) is documented in detail in the [Live view FAQ](/configuration/live#live-view-faq).
## H.265 / HEVC cameras
H.265/HEVC playback in the browser is unreliable and version-dependent. WebRTC does not support H.265 on some browsers, and MSE/HEVC support varies by browser, OS, and whether a hardware decoder is present. An H.265 stream that plays fine in VLC, the go2rtc web UI, and Frigate's recordings can still be blank in a live view.
For dependable live viewing, use **H.264** for the stream the live view consumes:
- Point the live view at the camera's H.264 **substream** and keep the H.265 main stream for recording only, or
- Transcode H.265 → H.264 in go2rtc with the FFmpeg module and `#hardware` (software HEVC transcoding is very CPU heavy).
Treat browser HEVC playback as best-effort. See also [H.265 cameras via Safari](/configuration/camera_specific#h265-cameras-via-safari).
## No audio in Live view
Live view audio has strict codec requirements that differ by player: **MSE requires AAC, PCMA, or PCMU**, and **WebRTC requires Opus, PCMA, or PCMU**. Many cameras default to a codec outside these sets (or to PCM/G.711), so the player loads video only and no audio control appears.
The most robust approach is to provide both an AAC track (for MSE) and an Opus track (for WebRTC) on the same stream by transcoding audio with the FFmpeg module while copying the video:
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > System > go2rtc Streams" /> and expand the camera's stream.
2. Add a second **Source** that references the stream by name (e.g. the URL `ffmpeg:back`), enable **Use compatibility mode (ffmpeg)**, and set **Audio** to **Transcode to Opus** for WebRTC support.
3. Keep the original source as **Source 1** so MSE can use the camera's AAC (or transcode the first source's audio to AAC if the camera doesn't provide it).
4. **Save** the section.
</TabItem>
<TabItem value="yaml">
```yaml
go2rtc:
streams:
back:
- rtsp://user:password@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2 # video + AAC for MSE
- "ffmpeg:back#audio=opus" # adds an Opus track for WebRTC
```
If the camera's native audio isn't AAC either, transcode both:
```yaml
go2rtc:
streams:
back:
- "ffmpeg:rtsp://user:password@10.0.10.10:554/live0#video=copy#audio=aac" # video copy + AAC for MSE
- "ffmpeg:back#audio=opus" # Opus for WebRTC
```
</TabItem>
</ConfigTabs>
Setting the camera firmware to AAC (and H.264) avoids transcoding entirely and is always preferable when the camera supports it. For more detail and examples, see [Audio Support](/configuration/live#audio-support).
## WebRTC and two-way talk
WebRTC is only attempted when MSE fails or when using a camera's two-way talk feature; the "All Cameras" dashboard never uses it. When it doesn't work, the cause is almost always one of:
- **Codec mismatch** — WebRTC cannot carry H.265 or AAC. The stream backing the WebRTC view must provide Opus (or PCMA/PCMU) audio and H.264 video. Add an `ffmpeg:back#audio=opus` source as shown above.
- **Port `8555` not reachable, or no candidates set** — WebRTC needs port `8555` (both TCP and UDP) open and a reachable candidate advertised. On Docker installs running on a custom/overlay network, go2rtc may advertise unreachable container IPs as ICE candidates; setting `webrtc.filters.candidates: []` and supplying only your host's LAN IP resolves this. See [WebRTC extra configuration](/configuration/live#webrtc-extra-configuration).
- **Two-way talk** additionally requires a secure context (HTTPS or the authenticated port `8971`, because browsers block microphone access on plain HTTP). The camera's RTSP backchannel must also be handled correctly — go2rtc seizes the backchannel by default, which blocks two-way audio for other consumers and can inject static. Disable it on the primary stream with `#backchannel=0` and use a separate dedicated stream for talk, as documented in [preventing go2rtc from blocking two-way audio](/configuration/restream#two-way-talk-restream).
## High CPU usage
If go2rtc is using a lot of CPU, it is almost always transcoding in software. An FFmpeg source with a codec modifier like `#video=h264` or `#audio=aac` but **no** `#hardware` re-encodes on the CPU. (Frigate's `ffmpeg.hwaccel_args` only applies to Frigate's own detect/record processes — it does _not_ accelerate go2rtc's transcodes.)
To keep CPU usage down:
- Only transcode the track that is genuinely unsupported, and use `#video=copy` to pass video through untouched whenever possible.
- When you must transcode video, always add `#hardware` (the **Automatic** hardware option in the UI) so the encode runs on the GPU. Note the [FFmpeg 8 device requirement](#hardware-accelerated-transcoding-with-ffmpeg-8) below.
- Don't restream a high-resolution main stream just to feed the live view — even with `#video=copy`, muxing a 4K/8MP+ stream is inherently expensive. Use the camera's lower-resolution substream for live and detect, and let Frigate pull the main stream directly for recording.
## Connection, authentication, and complex passwords
If go2rtc logs `401 Unauthorized` for a URL that works in VLC, the password almost certainly contains reserved URL characters. **Frigate URL-encodes passwords for its own `cameras.ffmpeg.inputs`, but it does not touch what you write under `go2rtc.streams`** — go2rtc parses that URL itself. You must URL-encode special characters yourself in the `go2rtc.streams` section (`@` → `%40`, `#` → `%23`, `?` → `%3F`, `%` → `%25`, etc.).
Note the asymmetry: under `cameras.ffmpeg.inputs` you should use the **raw** password (Frigate encodes it for you) — pre-encoding it there causes a double-encode and fails. See [Handling Complex Passwords](/configuration/restream#handling-complex-passwords).
Repeated `401`/`Connection refused` errors can also mean the camera hit its **concurrent connection limit** or triggered a login lockout. Routing all roles through a single [RTSP restream](/configuration/restream#reduce-connections-to-camera) means the camera only ever sees one connection from go2rtc.
## Stream names must match everywhere
A surprising number of "the better live options aren't available" or `404 Not Found` problems come down to a name mismatch. The same string must be used consistently:
- the **go2rtc stream key** (`go2rtc.streams.<name>`),
- any `ffmpeg:<name>#…` source that references it,
- the camera's restream input path (`rtsp://127.0.0.1:8554/<name>`), and
- the camera name itself (so Frigate auto-maps it for MSE/WebRTC) — or an explicit `live -> streams` mapping pointing at the go2rtc stream **name** (never a path).
If you rename or remove a go2rtc stream while experimenting and the live stream selector then shows a blank entry, clear your browser's site data for the Frigate URL — the selected stream is cached per-device in local storage.
## Camera-specific behavior
Several camera brands have well-known quirks with go2rtc. Rather than repeat them here, see the [camera-specific configuration](/configuration/camera_specific) page, which covers them in detail. The highlights:
- **Reolink** — RTSP is unreliable on many models; the **http-flv** stream through the FFmpeg module is recommended, and you must enable HTTP/RTMP in the camera and **reboot** it. 6MP+ models stream H.265 over http-flv-enhanced, which requires FFmpeg 8.0. See [Reolink Cameras](/configuration/camera_specific#reolink-cameras).
- **TP-Link Tapo** — use go2rtc's native `tapo://` source for stability and two-way audio; a stale RTSP credential can often be revived by clicking play once in the go2rtc web UI.
- **Ubiquiti/UniFi Protect** — use the `rtspx://` scheme (not `rtsps://…?enableSrtp`).
- **Amcrest/Dahua** — use the `/cam/realmonitor?channel=1&subtype=N` scheme, where `subtype=0` is the main stream. See [Amcrest & Dahua](/configuration/camera_specific#amcrest--dahua).
## Non-RTSP sources and the FFmpeg module
go2rtc's native zero-copy handling only supports well-formed RTSP H.264/H.265. Anything else — MJPEG, HTTP/HTTP-FLV, RTMP, or unusual codecs — must be handed to the FFmpeg module by prefixing the source with `ffmpeg:`. This is also necessary for some camera streams to be parsed at all, at the cost of slightly slower startup. MJPEG and other non-H.264 sources additionally need `#video=h264` (with `#hardware`) before they can be used for the `record`, `detect`, or restream roles. See [MJPEG Cameras](/configuration/camera_specific#mjpeg-cameras) for a complete example.
## Hardware-accelerated transcoding with FFmpeg 8
Frigate 0.18 ships **FFmpeg 8.0** as the default, and FFmpeg 8 is stricter about hardware-accelerated filtering than earlier versions. Whenever go2rtc transcodes video with hardware acceleration (any source using `#hardware`, `#hardware=vaapi`, or the **Automatic** hardware option in the UI), it builds a filter chain that uploads frames to the GPU with the `hwupload` filter. FFmpeg 8 now refuses to do this unless it is told **which device** to use — earlier versions selected one automatically. The result is that an otherwise-working transcode fails to start, the live view never loads, and go2rtc logs:
```
[hwupload] A hardware device reference is required to upload frames to.
[AVFilterGraph] Error initializing filters
Error opening output files: Invalid argument
```
The fix is to tell go2rtc's bundled FFmpeg which hardware device to use via the `go2rtc -> ffmpeg -> global` option. For **VAAPI**-based acceleration — which covers most Intel and AMD GPUs, and is what go2rtc selects automatically on that hardware — point it at your render device:
```yaml
go2rtc:
ffmpeg:
global: "-vaapi_device /dev/dri/renderD128"
streams:
back:
- "ffmpeg:rtsp://user:password@10.0.10.10:554/live0#video=h264#hardware"
```
`/dev/dri/renderD128` is the usual render node; on a system with more than one GPU you may need `renderD129` (or higher), and the device must be passed into the container (e.g. `devices: - /dev/dri:/dev/dri` in Docker Compose).
If you use a **different hardware acceleration backend**, you will likely need to specify its device in the same way, using the option that matches that backend instead of `-vaapi_device`. See the [go2rtc FFmpeg source documentation](https://github.com/AlexxIT/go2rtc/tree/v1.9.13#source-ffmpeg) and the upstream report ([go2rtc issue #1984](https://github.com/AlexxIT/go2rtc/issues/1984)) for background and other examples.
:::tip
If you don't transcode in go2rtc with hardware acceleration, this does not affect you. If you want to avoid the change entirely, you can pin Frigate (and the go2rtc it bundles) back to FFmpeg 7.0 by setting `ffmpeg -> path: "7.0"` in your config.
:::
## Home Assistant and port access
When running Frigate as a Home Assistant add-on, the go2rtc API (port `1984`), the RTSP restream (port `8554`), and WebRTC (port `8555`) are **disabled and hidden by default**. To use them — for example to reach the go2rtc web interface for troubleshooting, or to open a go2rtc stream externally in an app like VLC — go to <NavPath path="Settings > Add-ons > Frigate > Configuration > Network" />, click **Show disabled ports**, enable the port you need, and save. Use the host's IP address rather than an mDNS name like `homeassistant.local`.
If live view works in the Frigate UI but not in Home Assistant, the most common cause is the go2rtc stream name not matching the camera name — name the primary go2rtc stream exactly like the camera, or add a `live -> streams` mapping, so the integration can resolve the restream.

95
docs/docs/usage/explore.md Normal file
View File

@ -0,0 +1,95 @@
---
id: explore
title: Explore
---
import NavPath from "@site/src/components/NavPath";
**Explore** is where you browse and search every **tracked object** Frigate has saved. By default it groups recent objects by label; when [Semantic Search](/configuration/semantic_search) is enabled, you can also search by natural-language description or visual similarity. Selecting any object opens a detail pane with its snapshot, lifecycle, and metadata.
This page describes how to _use_ the Explore view. For how the underlying features are _configured_, see [Semantic Search](/configuration/semantic_search) and [Generative AI descriptions](/configuration/genai/genai_objects).
## Browsing tracked objects
The default view shows your most recent tracked objects grouped into rows by label — _Person_, _Car_, _Dog_, and so on — each row labeled with the object type and a count. The arrow at the end of a row opens the full, filterable grid for that label.
Clicking a thumbnail opens its [detail dialog](#tracked-object-details); right-clicking or long-pressing a thumbnail opens an [actions menu](#actions-and-bulk-selection). You can switch to a denser grid layout and adjust the number of columns from the view's settings.
## Searching
When [Semantic Search](/configuration/semantic_search) is enabled, a search bar appears that combines two things in one input:
- **Natural-language search** — type a free-text query and press Enter to run a semantic search over your tracked objects.
- **Filter tokens** — type a `key:` to get suggestions, then a value, to add a structured filter. Each filter becomes a removable chip, and you can chain several together.
You can save a search with the star icon and reload it later, and clear everything with the clear-search icon. A help popover explains the token syntax, for example:
```
cameras:front_door label:person before:01012024 time_range:3:00PM-4:00PM
```
### Filter reference
The most common filter tokens are:
| Filter | Description |
| ---------------------------- | ---------------------------------------------------------------------------------- |
| **Cameras** | Limit to one or more cameras. |
| **Labels** | Object labels (person, car, etc.). |
| **Sub Labels** | Recognized sub labels (e.g. a recognized face or name). |
| **Attributes** | Classification attributes applied to the object. |
| **Recognized License Plate** | Match a recognized plate. |
| **Zones** | Objects that entered specific zones. |
| **Before / After** | Restrict to a date range. |
| **Time Range** | Restrict to a time of day (`HH:MM-HH:MM`). |
| **Min / Max Score** | Restrict by the object's confidence score. |
| **Min / Max Speed** | Restrict by estimated speed (when speed estimation is configured). |
| **Has Snapshot / Has Clip** | Only objects that saved a snapshot or recording. |
| **Submitted to Frigate+** | Only objects already submitted (when Frigate+ is enabled). |
| **Search Type** | Whether semantic search matches the object's **Thumbnail** or its **Description**. |
### Sorting
When a filter or search is active, a **Sort** control lets you order results by **date**, **object score**, or **estimated speed** (ascending or descending). When a semantic query or similarity search is active, results can also be ordered by **relevance**.
### Thumbnail and description search
- The **Search Type** setting controls whether a text query is matched against each object's **thumbnail** or its **description**. Each result indicates which one it matched and the confidence.
Natural-language search, thumbnail search, and description search all require [Semantic Search](/configuration/semantic_search) to be enabled.
## Tracked Object Details
Selecting an object opens the **Tracked Object Details** dialog. Use the arrows (or the left/right keys) to step to the previous or next object. The dialog has two tabs:
- **Snapshot** or **Thumbnail** — the saved snapshot (or thumbnail).
- **Tracking Details** — the object's lifecycle, available when the object has a recording. It lists each significant moment (detected, entered a zone, became active or stationary, left, and so on); clicking a moment plays that part of the recording with the bounding box overlaid. A settings popover lets you show all zones and adjust the annotation offset.
The details pane shows the object's **label**, **scores**, **camera**, **timestamp**, estimated **speed**, any **recognized license plate** and **classification attributes**, and its **description**. Admins can edit the sub label, license plate, and attributes inline.
The **description** can be edited by hand, and — when [Generative AI descriptions](/configuration/genai/genai_objects) are enabled and the object's lifecycle has ended — regenerated from the snapshot or from thumbnails. For `speech` objects, a **Transcribe** action is available when audio transcription is enabled. When [Frigate+](/integrations/plus) is enabled, admins can submit a snapshot to improve their model directly from this pane.
## Actions and bulk selection
Right-clicking or long-pressing an object (in the grid or its thumbnail) opens an actions menu with options to **download** the video, snapshot, or a clean snapshot; **view tracking details**; **find similar**; **add a trigger**; **view in History**; and **delete the tracked object**.
:::note
Deleting a tracked object removes its snapshot, embeddings, and tracking-details entries, but the recorded footage of that object in [History](/usage/history) is **not** deleted.
:::
To act on many objects at once, Ctrl/Cmd-click or right-click to start a selection (selected tiles gain a blue ring), then use the toolbar to select all, clear the selection, or delete (admins).
## Semantic Search - Usage and best practices {#usage-and-best-practices}
1. Semantic Search is used in conjunction with the other filters available on the Explore page. Use a combination of traditional filtering and Semantic Search for the best results.
2. Use the thumbnail search type when searching for particular objects in the scene. Use the description search type when attempting to discern the intent of your object.
3. Because of how the AI models Frigate uses have been trained, the comparison between text and image embedding distances generally means that with multi-modal (`thumbnail` and `description`) searches, results matching `description` will appear first, even if a `thumbnail` embedding may be a better match. Play with the "Search Type" setting to help find what you are looking for. Note that if you are generating descriptions for specific objects or zones only, this may cause search results to prioritize the objects with descriptions even if the the ones without them are more relevant.
4. Make your search language and tone closely match exactly what you're looking for. If you are using thumbnail search, **phrase your query as an image caption**. Searching for "red car" may not work as well as "red sedan driving down a residential street on a sunny day".
5. Semantic search on thumbnails tends to return better results when matching large subjects that take up most of the frame. Small things like "cat" tend to not work well.
6. Experiment! Find a tracked object you want to test and start typing keywords and phrases to see what works for you.
## Triggers
From an object's actions menu, **Add trigger** sets up a per-camera trigger that uses Semantic Search to automate an action (a notification, sub label, or attribute) whenever a similar object appears. Triggers require Semantic Search and are managed under <NavPath path="Settings > Enrichments > Triggers" />. See [Triggers](/configuration/semantic_search#triggers) for full configuration and best practices.

43
docs/docs/usage/exports.md Normal file
View File

@ -0,0 +1,43 @@
---
id: exports
title: Exports
---
**Exports** are how you keep a specific piece of footage permanently.
Frigate's recordings are governed by your [retention settings](/configuration/record): once footage ages past its retention window — or, depending on your configuration, once it is only kept where motion, alerts, or detections occurred — it is deleted to free up disk space. An **export** saves a copy of a chosen time range to a separate location that is **never removed by retention**, so it stays available until you delete it yourself.
This is the answer to the common question _"how do I stop Frigate from deleting an important clip?"_ Instead of increasing retention for an entire camera (which uses far more storage to protect a single moment), export just the footage you want to keep.
:::tip
Exports are stored under `/media/frigate/exports`, separate from your recordings, and are not counted against or removed by recording retention. They remain on disk until you delete them, so be aware that they accumulate over time.
:::
## Creating an export
There are a few ways to create an export:
- **From Review** — select (right click or long-press) an individual review item directly, and choose Export from the header menu. You can also select multiple review items and export them all at once, optionally grouping them into a [case](#cases).
- **From History** — open the **Actions** menu and choose **Export**. You can export a preset duration (the last 1, 4, 8, 12, or 24 hours), enter a custom start and end time, or select a range directly on the timeline. A **multi-camera** option lets you export the same time range across several cameras at once.
In every case you can give the export a name. Frigate then saves the footage from your recordings as a single video file. Larger ranges take time to process; the export is marked _in progress_ until it finishes, and you can keep using Frigate while it runs.
## Managing exports
All of your exports live on the **Exports** page, reachable from the main navigation, where you can search for one by name. Each export offers the following actions:
- **Play** it in the browser,
- **Download** it to save the footage outside of Frigate,
- **Share** it — copies a direct link to the export (or uses your device's share sheet),
- **Rename** it, and
- **Delete** it — deleting is the only way an export is removed.
You can also select multiple exports at once to **delete** them in bulk, or to **add them to** (or **remove them from**) a [case](#cases).
## Cases
A **case** groups related exports together — for example, all the clips from a single incident across multiple cameras. On the **Exports** page you can create a case with a name and description, add existing exports to it (or create a new case while exporting), and **download the entire case as a single archive** to hand off as one package.
Exports that don't belong to a case appear under **Uncategorized Exports**. Deleting a case lets you either keep its exports (they move back to uncategorized) or delete them along with the case.

69
docs/docs/usage/history.md Normal file
View File

@ -0,0 +1,69 @@
---
id: history
title: History
---
import NavPath from "@site/src/components/NavPath";
**History** is Frigate's full-resolution recording viewer. Unlike Live, Review, and Explore, there is no menu item for it — you reach it from within another view, then scrub the timeline, switch cameras, inspect a tracked object's lifecycle, and export or share any moment.
This page describes how to _use_ the History view. For how recordings are _configured_ (retention, pre/post capture), see [Recording](/configuration/record).
## Opening History
You can open History from several places:
- **From [Review](/usage/review):** clicking a review item opens its recording, scrubbed to just before the activity on that camera.
- **From [Live](/usage/live):** the **History** button in a camera's single-camera view opens that camera about 30 seconds in the past.
- **From a share link:** opening a shared timestamp link (see [Share Timestamp](#the-actions-menu) below) jumps straight to that camera and moment.
Use the **Back** button to return where you came from, or the **Live** button to jump to the current camera's live view.
## Timeline, Events, and Detail
A toggle (a drawer on mobile) switches the side panel between three modes:
- **Timeline** — a scrubbable vertical timeline of the selected camera, annotated with a motion line, review-item markers, and gaps where no recording exists.
- **Events** — a scrollable list of the camera's review items for the time range; clicking one seeks the player to it.
- **Detail** — the [tracking details inspector](#the-detail-view) for the objects in view.
While you are selecting a range to export, the panel temporarily switches to Timeline.
## Scrubbing and previews
Drag the timeline handlebar to move through time; the main player and any secondary camera previews scrub together so everything stays in sync. Press the zoom buttons on the timeline to change its zoom level (from coarse to fine segments). Sections of the timeline with no recordings are shown as gaps.
On desktop, when more than one camera is available, a **row of secondary previews** shows the other cameras at the same moment. Clicking one of them makes it the main camera at the current timestamp, so you can follow activity across cameras without losing your place. On mobile, use the camera drawer to switch cameras.
## Filtering and the calendar
You can filter History by **cameras** and **date**. The calendar behaves the same as it does in [Review](/usage/review#filtering-and-the-calendar): an **underline** under a day means recordings exist for that day, and a **colored dot** (red for unreviewed alerts, orange for unreviewed detections) marks days with unreviewed activity.
## The Detail view
The **Detail** mode turns the side panel into a tracking details inspector. It lists one card per review item, each showing the item's severity, start time, the object labels involved, a count of tracked objects, and the duration. The active card is highlighted as the video plays, and clicking a card seeks to it.
Expanding a card reveals the **lifecycle** of each tracked object — a row for each significant moment (detected, entered a zone, became active, became stationary, left, and so on), with a progress line that follows the current playback position. Hovering a row shows that moment's score, ratio, and area, and clicking a row seeks the video to that exact timestamp.
The **Detail View Settings** at the bottom let you toggle whether the active item's objects expand automatically, and adjust the **annotation offset** — a fine timing correction that aligns the bounding-box overlays with the recorded video when your camera's snapshot and recording timestamps drift. Admins can save the offset to the camera's configuration.
## The Actions menu
On desktop, the **Actions** menu (the film icon) collects the things you can do with the footage you are viewing:
- **Export** — save a clip of a chosen time range so it is never removed by retention. The dialog pre-selects the last hour; adjust the range or drag the timeline handles, then export. See [Exports](/usage/exports) for managing and downloading exports.
- **Share Timestamp** — generate a link to the current moment (or a custom timestamp) to share with another Frigate user. This is an internal link, not a public share URL.
- **Motion Search** — scan this camera's recordings for changes in a region you draw. This is the same tool documented under [Reviewing Motion](/usage/review#motion-search).
- **Debug Replay** (admins) — replay a recorded range back through Frigate's detection pipeline to see how it would be processed.
You can also capture an instant snapshot of the current frame, and submit a frame to [Frigate+](/integrations/plus) directly from the player (admins only).
## AI review summaries
When [Generative AI review](/configuration/genai/genai_review) is configured, Frigate can generate a title, description, and threat classification for review items and surface them as you scrub through History. A review item that has an AI summary exposes its details in a few places:
- **Over the video** — when the item is on screen, a popup appears over the player.
- **In the Events side panel** — items with a summary show the title below the thumbnail.
- **In the Detail side panel** — the item's card shows the title alongside its tracking details.
Clicking any of these opens the **AI Analysis** dialog with the generated detail and any flagged concerns for that item.

118
docs/docs/usage/live.md Normal file
View File

@ -0,0 +1,118 @@
---
id: live
title: Live View
---
import NavPath from "@site/src/components/NavPath";
**Live view** is Frigate's real-time dashboard and the page you land on by default. It shows all of your cameras at a glance, streams your most recent alerts across the top, and lets you open any camera in a full-resolution single-camera view with audio, two-way talk, PTZ, and on-demand recording controls.
This page describes how to _use_ the Live view. For how to _configure_ live streaming — go2rtc, stream selection, smart streaming, WebRTC, and audio — see the [Live View configuration](/configuration/live) docs.
## The dashboard at a glance
The default **All Cameras** dashboard shows every camera, with a filmstrip of recent **alerts** scrolling across the top. Clicking an alert opens it in [Review](/usage/review); each card also has a check button to mark it reviewed without leaving the dashboard.
By default Frigate uses **smart streaming**: a camera's image updates roughly once per minute while nothing is happening, and switches to a full live stream the moment activity is detected. This conserves bandwidth and resources. You can change this per camera or per group (see [Streaming settings](#streaming-settings-and-the-right-click-menu) below), and the behavior is explained in detail under [Live view technologies](/configuration/live#live-view-technologies).
On mobile, a toggle in the header switches between a **grid** layout and a single-column **list** layout. On desktop a **fullscreen** button is available in the lower-right corner.
## Switching dashboards and camera groups
The icon rail (top-left on desktop, a horizontal strip on mobile) switches between dashboards:
- The **home** icon is the **All Cameras** dashboard, which shows every camera enabled for the dashboard.
- Each **camera group** you create appears as its own icon. Selecting a group shows only that group's cameras.
Camera groups are useful for organizing cameras by location (for example, _Front of House_ or _Backyard_) and for giving each group its own dashboard layout and streaming preferences.
You can also view [Birdseye](/configuration/birdseye) on the dashboard, or open it directly at `http://<frigate_host>:5000/#birdseye`. Clicking a camera inside the Birdseye view jumps to that camera's live feed.
## Creating and editing camera groups
Admins can manage groups from the pencil icon next to the group rail, which opens the **Camera Groups** dialog. From there you can add a group, or edit and delete existing ones. When creating a group you choose:
- a **Name** (spaces are converted to underscores),
- the **cameras** to include — each camera has a toggle and a gear that opens its [streaming settings](#streaming-settings-and-the-right-click-menu), and
- an **icon** used for the group's button in the rail.
Deleting a group also clears any custom layout you saved for it.
## Rearranging a camera group layout
On desktop and tablet, each camera group has its own freely-arrangeable grid. Enter **Edit Layout** mode from the layout button in the lower-right corner: camera tiles gain a drag handle and corner resize handles. Drag a tile to reposition it and drag a corner to resize it (the aspect ratio is preserved). Exit edit mode to save. The layout is stored in your browser per device, so each device can have its own arrangement.
The default **All Cameras** dashboard is not manually arrangeable — it automatically sizes tiles based on each camera's aspect ratio (wide cameras span two columns, tall cameras span two rows).
## Reading the tile indicators
Each camera tile surfaces its current state with a few overlays:
- A **pulsing red dot** in the corner means **motion is currently detected** on that camera.
- A **red outline** around the tile means an **active tracked object** is on that camera.
- A small **label chip** lists the object types currently detected (for example, _Person_, _Car_).
- A **camera-name label** appears when you have enabled always-on camera names, or when a camera is offline or disabled.
- A **Stream Offline** or **Camera is off** placeholder appears when no frames are being received or the camera has been turned off.
You can optionally overlay live streaming statistics (stream type, bandwidth, latency, and frame counts) on a tile to diagnose playback issues.
## Streaming settings and the right-click menu
Right-clicking (or long-pressing) a camera tile opens a context menu with quick controls: an **audio volume** control for streams that support audio, **Mute / Unmute all cameras**, **show or hide streaming statistics**, the **debug view**, **notification** options, and — for admins — turning the camera on or off.
A **Low-bandwidth mode** notice may also appear in the context menu with a **Reset** option appears when Frigate has fallen back to the lower-quality jsmpeg stream — see the [Live view FAQ](/configuration/live#live-view-faq) for why this happens.
For non-default groups, the context menu also exposes **Streaming Settings** for that camera, which let you choose:
- the **stream** to display (the dropdown lists the streams you configured under [`live -> streams`](/configuration/live#setting-streams-for-live-ui), and indicates whether audio is available),
- the **streaming method** — **No Streaming**, **Smart Streaming** (recommended), or **Continuous Streaming** (higher bandwidth), and
- **compatibility mode**, for devices that have trouble rendering the default player.
These settings are saved per group and per device in your browser, not in your config file.
## The single-camera view
Clicking a camera tile opens its full-resolution single-camera view. The top bar provides:
- **Back** (also the `Esc` key) to return to the dashboard,
- **History** to jump to the [recordings](/usage/history) for this camera, starting about 30 seconds in the past,
- **Fullscreen** and **Picture-in-Picture** (if supported by your browser),
- **Two-way talk** (the microphone button — requires a supported camera and WebRTC; keyboard shortcut `t`), and
- **Camera audio muting** (the speaker button; keyboard shortcut `m`).
You can pinch or scroll to zoom into the feed. A **settings** gear provides a **stream** selector (with audio and two-way-talk availability indicators), **Play in background**, **Show stats**, and a **Debug view** that overlays Frigate's detection regions and bounding boxes.
:::tip
Two-way talk and camera audio have specific codec and port requirements. See [Audio Support](/configuration/live#audio-support) and [WebRTC](/configuration/live#webrtc-extra-configuration) for setup details.
:::
## Camera controls
Admins get a row of toggles in the single-camera view (a settings drawer on mobile) to turn camera features on and off in real time:
- **Camera** on/off,
- **Object detection**,
- **Recording** (only available when recording is enabled in the camera's config),
- **Snapshots**,
- **Audio detection**,
- **Live audio transcription** (when audio detection is enabled), and
- **Autotracking** (for [autotracking-capable PTZ cameras](/configuration/autotracking)).
These toggles change runtime behavior immediately. Whether a change persists across a restart depends on the feature — see the relevant configuration page.
## On-demand recording and snapshots
The single-camera view can capture footage on demand:
- **Start on-demand recording** begins a manual recording based on the camera's recording retention settings (the button pulses while active). If recording is disabled for the camera, only a snapshot is saved. Use **End on-demand recording** to stop.
- **Download instant snapshot** saves a still image of the current frame.
See [Recording](/configuration/record) and [Snapshots](/configuration/snapshots) for how retention is configured, and [Exports](/usage/exports) for keeping a clip permanently.
## PTZ controls
For ONVIF cameras that support it, a control panel provides pan/tilt arrows, **zoom**, **focus**, and saved **presets**. You can also enable a **click-to-move / drag-to-zoom** overlay: click a point in the frame to center the camera there, or drag a box to pan and zoom to that area (dragging top-left to bottom-right zooms in, the reverse zooms out).
For continuous, automatic tracking of a moving object, see [Autotracking](/configuration/autotracking).

140
docs/docs/usage/review.md Normal file
View File

@ -0,0 +1,140 @@
---
id: review
title: Review
---
import NavPath from "@site/src/components/NavPath";
**Review** is where you triage what happened on your cameras. It groups activity into **review items** — segments of time on a single camera that bundle together the objects and audio that were active at once — and sorts them into **Alerts**, **Detections**, and **Motion**. From here you can scrub through activity, mark items as reviewed, filter, export, and jump to the full recording in [History](/usage/history).
This page describes how to _use_ the Review view. For how alerts and detections are _configured_ (labels, zones, required zones, retention), see the [Review configuration](/configuration/review) docs.
:::info
Review items are only created for a camera when **recording is enabled** for that camera. See [Recording](/configuration/record).
:::
## Alerts, Detections, and Motion
Not every segment of video captured by Frigate is of the same level of interest. The people who enter your property may be a higher priority than those just walking by on the sidewalk. For this reason, Frigate sorts **review items** by importance into **alerts** and **detections**, with a separate **Motion** category for significant motion.
The toggle at the top of the page switches between these three severities. One is always selected.
| Tab | Indicator color | What it shows |
| -------------- | --------------- | ---------------------------------------------------------------------------------------------------------------- |
| **Alerts** | dark red | The activity you most want to see. By default, all `person` and `car` tracked objects are alerts. |
| **Detections** | orange | Everything else Frigate tracked that wasn't promoted to an alert. |
| **Motion** | yellow | Periods of significant motion, with the ability to filter to periods which did **not** produce a tracked object. |
This same color coding is used for the ring around a selected item and the dots on the calendar. How an object is categorized as an alert vs. a detection — and how required zones refine that — is covered in [Alerts and Detections](/configuration/review#alerts-and-detections).
The **Alerts** and **Detections** tabs show a count next to their label. With **Show Reviewed** turned off (the default), this is the number of items still left to review; with it on, the count reflects every item in the selected time range.
## Marking items as reviewed
Review items are shown as a grid of thumbnail cards next to a vertical activity timeline. Hovering a card (desktop) or swiping to the right (mobile) plays a short preview inline.
- **Clicking** a card opens its recording in [History](/usage/history) and marks the item as reviewed.
- The object chip on each card is **gray** when the item is unreviewed and turns **green** once it has been reviewed.
- The **Mark these items as reviewed** button marks everything currently shown as reviewed at once.
Reviewed state is tracked per user, so marking an item reviewed does not hide it for other users.
## Selecting and acting on multiple items
To act on several items at once, start a selection by **Ctrl/Cmd-clicking** a card (desktop) or **long-pressing** one (mobile). Selected cards gain a colored ring matching their severity. Keyboard shortcuts speed this up: `Ctrl+A` selects all, `R` marks the selection reviewed, and `Esc` clears it.
With items selected, an action bar appears with options to:
- **Export** the selected items (a single item exports directly; multiple items open the batch [export](/usage/exports) dialog),
- **Mark as reviewed** or **Mark as unreviewed**, and
- **Delete** them (admins only).
## Filtering and the calendar
Use the filter controls in the header to narrow what's shown. The available filters depend on the tab: Alerts and Detections can be filtered by **cameras**, **date**, **labels**, **zones**, and whether items are already reviewed; the Motion tab can be filtered by **cameras**, **date**, and **motion only**.
The **calendar** filter lets you jump to a specific day (it shows **Last 24 Hours** until you pick one). On each day:
- An **underline** under the day number means **recordings exist** for that day. Days without recordings are dimmed.
- A **colored dot** under the day number means there is **unreviewed activity** that day — a **red dot** for unreviewed alerts, or an **orange dot** for unreviewed detections when there are no unreviewed alerts. Motion is not represented by a dot.
Future dates are disabled, and the week start and time zone follow your configuration.
## Reviewing Motion
The Review page also can show periods of motion that didn't produce a tracked object, and provides a way to search past recordings for motion in a specific region. These tools complement the alerts and detections workflow above — see [Tuning Motion Detection](/configuration/motion_detection) for how the underlying motion detector is configured.
The **Motion** tab itself shows a multi-camera grid scrubbed to a shared point in time, with a draggable timeline and a playback-speed selector. A camera tile gains a colored ring when a review item or significant motion overlaps the current time, and clicking a tile opens that camera's recording at that moment. Each camera's options menu (the kebab in the corner of its tile) is where you open **Motion Previews** and **Motion Search**, described below.
### Motion Previews
The Motion Previews pane shows preview clips for periods of significant motion that did not produce a tracked object. It is useful for spotting things that motion detection picked up but object detection did not, which can help validate tuning or catch missed objects.
On the <NavPath path="Review > Motion" /> page, click the kebab menu on a camera and choose **Motion Previews**. Each card represents a continuous range of motion-only activity and plays back the recorded preview for that range. A heatmap overlay dims areas of the frame with no motion so the moving regions stand out.
The pane provides a few controls:
- **Speed** — speeds up or slows down all of the preview clips at once.
- **Dim** — controls how strongly non-motion areas are darkened by the heatmap overlay. Higher values increase motion area visibility.
- **Filter** — opens a 16×16 grid overlaid on a snapshot of the camera. Select one or more cells to only show clips with motion in those regions. This is helpful for filtering out motion in areas like a busy street while keeping motion in your driveway.
Clicking a preview clip seeks the recording player to that timestamp so you can review the full footage.
### Motion Search
Motion Search lets you scan recorded footage for changes inside a region of interest you draw on the camera. Unlike Motion Previews, which surfaces what Frigate's motion detector flagged in real time, Motion Search re-analyzes the saved recordings, so it can find changes that were missed (for example, an object that appeared while motion detection was paused by `lightning_threshold`, or in a region that is normally motion-masked).
To start a search, open the Actions menu in [History](/usage/history) or click the kebab menu on a camera in the <NavPath path="Review > Motion" /> page and choose **Motion Search**. In the dialog:
1. Pick the camera and time range to scan. In the date pickers, days that have recordings available are underlined.
2. Draw a polygon on the camera frame to define the region of interest.
3. Adjust the search parameters if needed:
| Field | Description |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Sensitivity Threshold** | Per-pixel luminance change required to count as motion inside the ROI. Behaves like Frigate's motion detection `threshold` setting. |
| **Minimum Change Area** | Minimum size of a single moving region, as a percentage of the ROI, for a frame to count as significant. Raise it to ignore small movements (leaves, distant motion); lower it when your subject covers only a small slice of the ROI. Every result shows the percentage it scored, so you can use those values to tune this. |
| **Maximum Results** | Maximum number of matching timestamps to return. The search stops once it reaches this many results, so a lower value finishes sooner while a higher value scans further into the range. |
| **Parallel mode** | Decode multiple recording ranges at the same time. Speeds up large time ranges at the cost of higher decoding and CPU usage. |
Motion Search samples each recording's keyframes automatically, so there is no frame-rate or sampling setting to tune.
Once running, Frigate scans the recording segments that overlap the time range and reports timestamps where changes were detected inside the polygon, along with the percentage of the ROI that changed. Clicking a result seeks the player to that moment so you can review what happened.
The results panel shows the time range being scanned, a live progress bar with the timestamp currently being analyzed, and the running result count. A collapsible **Search Metrics** section reports how many segments were scanned and processed, how many were skipped because no motion was recorded in the ROI (using the stored motion heatmap), how many frames were decoded, and the total search time. Skipping segments with no recorded motion in the selected ROI is what makes searching long time ranges practical.
#### Common use cases
Frigate's main use case is to record and surface tracked objects, so Motion Search is most useful for the cases where object detection produced nothing — there is no object to find in Explore, but you suspect something happened.
- **Locating an unattributed change.** You know something appeared, disappeared, or moved in a window of footage — a package now gone, a gate left open — but no detection points to it. A search returns the candidate timestamps instead of scrubbing the timeline by hand.
- **An object that was never detected.** Something Frigate doesn't have a model label for, an object too small or distant to be detected, or movement in a region where detection isn't running. The activity left no tracked object but did change the pixels, so a search can still find it.
- **Activity while detection was effectively paused.** Changes that occurred while object detection was disabled, motion was suppressed by `skip_motion_threshold`, or inside an area covered by a motion mask, won't appear as review items or tracked objects but can be recovered by searching the recordings directly.
#### Examples
These show how to choose the ROI and **Minimum Change Area** for two common goals. Minimum Change Area is the size of a single moving region as a percentage of the ROI you draw, so the right value depends on how much of the ROI your subject — and its movement between samples — covers.
Because samples are a second or more apart, a moving subject usually appears in two places at once in the comparison, so even ordinary motion often scores tens of percent and a low threshold lets in almost everything. The most reliable approach is to **run a search, look at the percentage each result scored, and set Minimum Change Area just below the values for the events you care about.** The default is 20%; the suggestions below are starting points.
- **When did this item first appear (or disappear)?** A package was dropped off, a car parked, or a trash can was moved, and you want the exact moment. Draw a **tight ROI** around the spot the item occupies and **raise Minimum Change Area** (start around 40–60%). Because the item fills most of a tight ROI, its arrival or removal is a large change, while smaller nearby motion (shadows, a passing pedestrian) stays below the threshold. The **earliest result** is when it appeared; if you only care about that moment, a low Maximum Results finishes faster. If you get no hits, the ROI is probably looser than the item — lower the threshold or tighten the ROI.
- **What's been getting into the garden?** Something has been trampling a flower bed overnight and no object was ever tracked. Draw a **looser ROI** covering the whole bed and use a **lower Minimum Change Area than the case above** — start near the 20% default and lower it (toward 5–10%) only if a small or distant subject is missed, since it covers just a slice of a large region. Expect more results to scan through — step through the timestamps and jump to each to see what triggered it. If wind-blown plants add noise, raise Minimum Change Area or the Sensitivity Threshold.
#### Expected performance
Motion Search analyzes the saved recordings on demand rather than reading a pre-built index, so a search over a long range takes longer than browsing Motion Previews. Cost scales mainly with how much footage has to be examined: segments with no recorded motion in your ROI are skipped using the stored motion heatmap (shown as "segments skipped" in the status panel), so a quiet range finishes quickly while a busy one takes longer.
To increase the speed of searches:
- Draw a tight ROI. Because **Minimum Change Area** is measured as a percentage of the region you draw, a tight ROI around where you expect the change makes the object fill a larger share of the area, so it clears the threshold more easily. A loose ROI makes the same object a small fraction of the region, so it can fall below the threshold and be missed — forcing you to lower Minimum Change Area, which lets in more noise.
- Narrow the time range to the window you care about, so there is less footage to examine.
- Lower **Maximum Results** when you only need the first few hits. Because the search stops once it reaches that many results, a smaller value lets a busy range finish early instead of scanning the whole window.
- Use Parallel mode to shorten wall-clock time on multi-core systems, at the cost of higher decoding and CPU usage while it runs.
## AI review summaries
When [Generative AI review](/configuration/genai/genai_review) is configured, Frigate can generate a title, description, and threat classification for review items and surface them automatically in Review and History. Clicking the summary chip opens an **AI Analysis** dialog with the generated detail and any flagged concerns.
In Review, an additional icon appears on unreviewed items that the AI classified as **suspicious** (Level 1) or **critical** (Level 2), so the activity that most warrants attention stands out before you open it. The icon goes away once the item has been reviewed.

4
docs/scripts/lib/nav_map.py
View File

@ -63,8 +63,8 @@ SYSTEM_NAV: dict[str, tuple[str, str]] = {
"environment_vars": ("System", "Environment variables"),
"telemetry": ("System", "Telemetry"),
"birdseye": ("System", "Birdseye"),
"detectors": ("System", "Detector hardware"),
"model": ("System", "Detection model"),
"detectors": ("System", "Detectors and model"),
"model": ("System", "Detectors and model"),
}
# All known top-level config section keys

70
docs/sidebars.ts
View File

@ -17,26 +17,31 @@ const sidebars: SidebarsConfig = {
],
Guides: [
"guides/getting_started",
"guides/configuring_go2rtc",
"guides/ha_notifications",
"guides/ha_network_storage",
"guides/reverse_proxy",
],
Configuration: {
"Configuration Files": [
"configuration/index",
"configuration/reference",
{
type: "link",
label: "Go2RTC Configuration Reference",
href: "https://github.com/AlexxIT/go2rtc/tree/v1.9.13#configuration",
} as PropSidebarItemLink,
Usage: [
"usage/live",
"usage/review",
"usage/history",
"usage/explore",
"usage/exports",
],
Detectors: [
Configuration: [
"configuration/config",
{
type: "category",
label: "Detectors",
items: [
"configuration/object_detectors",
"configuration/audio_detectors",
],
Enrichments: [
},
{
type: "category",
label: "Enrichments",
items: [
"configuration/semantic_search",
"configuration/face_recognition",
"configuration/license_plate_recognition",
@ -69,7 +74,11 @@ const sidebars: SidebarsConfig = {
],
},
],
Cameras: [
},
{
type: "category",
label: "Cameras",
items: [
"configuration/cameras",
"configuration/review",
"configuration/record",
@ -81,27 +90,53 @@ const sidebars: SidebarsConfig = {
"configuration/autotracking",
"configuration/camera_specific",
],
Objects: [
},
{
type: "category",
label: "Objects",
items: [
"configuration/object_filters",
"configuration/masks",
"configuration/zones",
"configuration/objects",
"configuration/stationary_objects",
],
"Hardware Acceleration": [
},
{
type: "category",
label: "Hardware Acceleration",
items: [
"configuration/hardware_acceleration_video",
"configuration/hardware_acceleration_enrichments",
],
"Extra Configuration": [
},
{
type: "category",
label: "Extra Configuration",
items: [
"configuration/authentication",
"configuration/notifications",
"configuration/profiles",
"configuration/go2rtc",
"configuration/ffmpeg_presets",
"configuration/pwa",
"configuration/tls",
"configuration/advanced",
],
},
{
type: "category",
label: "Advanced Configuration",
items: [
"configuration/advanced/system",
"configuration/advanced/reference",
{
type: "link",
label: "Go2RTC Configuration Reference",
href: "https://github.com/AlexxIT/go2rtc/tree/v1.9.13#configuration",
} as PropSidebarItemLink,
],
},
],
Integrations: [
"integrations/plus",
"integrations/home-assistant",
@ -130,6 +165,7 @@ const sidebars: SidebarsConfig = {
],
Troubleshooting: [
"troubleshooting/faqs",
"troubleshooting/go2rtc",
"troubleshooting/recordings",
"troubleshooting/dummy-camera",
{

91
docs/static/frigate-api.yaml vendored
View File

@ -2058,6 +2058,47 @@ paths:
application/json:
schema:
$ref: "#/components/schemas/HTTPValidationError"
/genai/models:
get:
tags:
- App
summary: List available GenAI models
description: Returns available models for each configured GenAI provider.
operationId: genai_models_genai_models_get
responses:
"200":
description: Successful Response
content:
application/json:
schema: {}
/genai/probe:
post:
tags:
- App
summary: Probe a GenAI provider without saving config
description: >-
Builds a transient client from the request body and returns its
available models. Used to validate provider credentials in the UI
before saving the configuration. Requires admin role.
operationId: genai_probe_genai_probe_post
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/GenAIProbeBody"
responses:
"200":
description: Successful Response
content:
application/json:
schema: {}
"422":
description: Validation Error
content:
application/json:
schema:
$ref: "#/components/schemas/HTTPValidationError"
/vainfo:
get:
tags:
@ -7031,6 +7072,39 @@ components:
"john_doe": ["face1.webp", "face2.jpg"],
"jane_smith": ["face3.png"]
}
GenAIProbeBody:
properties:
provider:
type: string
enum:
- openai
- azure_openai
- gemini
- ollama
- llamacpp
title: Provider
description: GenAI provider to probe
api_key:
anyOf:
- type: string
- type: "null"
title: API Key
description: API key for the provider (when applicable)
base_url:
anyOf:
- type: string
- type: "null"
title: Base URL
description: Base URL for self-hosted or compatible providers
provider_options:
type: object
title: Provider Options
description: Additional provider-specific options
default: {}
type: object
required:
- provider
title: GenAIProbeBody
GenerateObjectExamplesBody:
properties:
model_name:
@ -7214,13 +7288,6 @@ components:
title: Min Area
description: Minimum change area as a percentage of the ROI
default: 5
frame_skip:
type: integer
maximum: 30
minimum: 1
title: Frame Skip
description: "Process every Nth frame (1=all frames, 5=every 5th frame)"
default: 5
parallel:
type: boolean
title: Parallel
@ -7306,6 +7373,16 @@ components:
anyOf:
- $ref: "#/components/schemas/MotionSearchMetricsResponse"
- type: "null"
scanning_timestamp:
anyOf:
- type: number
- type: "null"
title: Scanning Timestamp
progress:
anyOf:
- type: number
- type: "null"
title: Progress
type: object
required:
- success

180
frigate/api/app.py
View File

@ -34,15 +34,18 @@ from frigate.api.auth import (
from frigate.api.defs.query.app_query_parameters import AppTimelineHourlyQueryParameters
from frigate.api.defs.request.app_body import (
AppConfigSetBody,
GenAIProbeBody,
MediaSyncBody,
)
from frigate.api.defs.tags import Tags
from frigate.config import FrigateConfig
from frigate.config import FrigateConfig, GenAIConfig, GenAIProviderEnum
from frigate.config.camera.updater import (
CameraConfigUpdateEnum,
CameraConfigUpdateTopic,
)
from frigate.const import REDACTED_CREDENTIAL_SENTINEL
from frigate.ffmpeg_presets import FFMPEG_HWACCEL_VAAPI, _gpu_selector
from frigate.genai import PROVIDERS, load_providers
from frigate.jobs.media_sync import (
get_current_media_sync_job,
get_media_sync_job_by_id,
@ -59,7 +62,11 @@ from frigate.util.builtin import (
process_config_query_string,
update_yaml_file_bulk,
)
from frigate.util.config import apply_section_update, find_config_file
from frigate.util.config import (
apply_section_update,
find_config_file,
redact_credential,
)
from frigate.util.schema import get_config_schema
from frigate.util.services import (
get_nvidia_driver_info,
@ -75,6 +82,14 @@ logger = logging.getLogger(__name__)
router = APIRouter(tags=[Tags.app])
# Short timeout for the /genai/probe path. The probe is interactive — fail
# fast on hung providers rather than holding an API worker thread.
_PROBE_TIMEOUT_SECONDS = 10
# Outer cap that returns control to the caller even if the underlying sync
# HTTP call ignores its timeout. The sync work continues in the background
# thread; only the response is bounded.
_PROBE_OUTER_TIMEOUT_SECONDS = 15
@router.get(
"/", response_class=PlainTextResponse, dependencies=[Depends(allow_public())]
@ -170,6 +185,95 @@ def genai_models(request: Request):
return JSONResponse(content=request.app.genai_manager.list_models())
@router.post(
"/genai/probe",
dependencies=[Depends(require_role(["admin"]))],
summary="Probe a GenAI provider without saving config",
description=(
"Builds a transient client from the request body and returns its "
"available models. Used to validate provider credentials in the UI "
"before saving the configuration."
),
)
async def genai_probe(body: GenAIProbeBody):
load_providers()
provider_cls = PROVIDERS.get(body.provider)
if not provider_cls:
return JSONResponse(
status_code=400,
content={"success": False, "message": "Unknown provider"},
)
# The OpenAI-compatible SDKs accept "timeout" as a constructor kwarg via
# provider_options; other plugins use GenAIClient.timeout passed below.
# Don't inject timeout for Gemini — its HttpOptions interprets the value
# in milliseconds and would clash with the plugin's own default.
probe_provider_options: dict[str, Any] = dict(body.provider_options or {})
if body.provider in (GenAIProviderEnum.openai, GenAIProviderEnum.azure_openai):
probe_provider_options.setdefault("timeout", _PROBE_TIMEOUT_SECONDS)
try:
transient_cfg = GenAIConfig(
provider=body.provider,
api_key=body.api_key,
base_url=body.base_url,
provider_options=probe_provider_options,
# model is required by the schema but irrelevant for listing.
model="probe",
roles=[],
)
except ValidationError:
logger.exception("GenAI probe: invalid configuration")
return JSONResponse(
status_code=400,
content={"success": False, "message": "Invalid provider configuration"},
)
try:
client = provider_cls(
transient_cfg,
timeout=_PROBE_TIMEOUT_SECONDS,
validate_model=False,
)
except Exception:
logger.exception("GenAI probe: failed to construct client")
return JSONResponse(
content={
"success": False,
"message": "Failed to connect to provider",
},
)
try:
models = await asyncio.wait_for(
asyncio.to_thread(client.list_models),
timeout=_PROBE_OUTER_TIMEOUT_SECONDS,
)
except asyncio.TimeoutError:
return JSONResponse(
content={"success": False, "message": "Probe timed out"},
)
except Exception:
logger.exception("GenAI probe: list_models failed")
return JSONResponse(
content={"success": False, "message": "Provider returned no models"},
)
if not models:
return JSONResponse(
content={
"success": False,
"message": (
"No models returned. Check the API key, base URL, and "
"that the provider is reachable."
),
},
)
return JSONResponse(content={"success": True, "models": models})
@router.get("/config", dependencies=[Depends(allow_any_authenticated())])
def config(request: Request):
config_obj: FrigateConfig = request.app.frigate_config
@ -185,26 +289,24 @@ def config(request: Request):
if request.headers.get("remote-role") != "admin":
config.pop("environment_vars", None)
# remove mqtt credentials
config["mqtt"].pop("password", None)
config["mqtt"].pop("user", None)
# redact mqtt credentials
redact_credential(config["mqtt"], "password")
# remove the proxy secret
config["proxy"].pop("auth_secret", None)
# redact proxy secret
redact_credential(config["proxy"], "auth_secret")
# remove genai api keys
for genai_name, genai_cfg in config.get("genai", {}).items():
# redact genai api keys
for _genai_name, genai_cfg in config.get("genai", {}).items():
if isinstance(genai_cfg, dict):
genai_cfg.pop("api_key", None)
redact_credential(genai_cfg, "api_key")
for camera_name, camera in request.app.frigate_config.cameras.items():
camera_dict = config["cameras"][camera_name]
# remove onvif credentials
# redact onvif credentials
onvif_dict = camera_dict.get("onvif", {})
if onvif_dict:
onvif_dict.pop("user", None)
onvif_dict.pop("password", None)
redact_credential(onvif_dict, "password")
# clean paths
for input in camera_dict.get("ffmpeg", {}).get("inputs", []):
@ -581,6 +683,10 @@ def _config_set_in_memory(request: Request, body: AppConfigSetBody) -> JSONRespo
_restore_masked_camera_paths(body.config_data, request.app.frigate_config)
updates = flatten_config_data(body.config_data)
updates = {k: ("" if v is None else v) for k, v in updates.items()}
# Drop any field whose value is still the redaction sentinel
updates = {
k: v for k, v in updates.items() if v != REDACTED_CREDENTIAL_SENTINEL
}
if not updates:
return JSONResponse(
@ -644,6 +750,40 @@ def _config_set_in_memory(request: Request, body: AppConfigSetBody) -> JSONRespo
settings,
)
# detect resize also republishes motion + objects so other
# processes pick up the rebuilt masks, and fires refresh so
# the camera maintainer recycles the camera process to pick
# up the new ffmpeg cmd / SHM sizing
if field == "detect":
cam_cfg = config.cameras.get(camera)
if cam_cfg is not None:
if cam_cfg.motion is not None:
request.app.config_publisher.publish_update(
CameraConfigUpdateTopic(
CameraConfigUpdateEnum.motion, camera
),
cam_cfg.motion,
)
request.app.config_publisher.publish_update(
CameraConfigUpdateTopic(
CameraConfigUpdateEnum.objects, camera
),
cam_cfg.objects,
)
if cam_cfg.zones:
request.app.config_publisher.publish_update(
CameraConfigUpdateTopic(
CameraConfigUpdateEnum.zones, camera
),
cam_cfg.zones,
)
request.app.config_publisher.publish_update(
CameraConfigUpdateTopic(
CameraConfigUpdateEnum.refresh, camera
),
cam_cfg,
)
return JSONResponse(
content={"success": True, "message": "Config applied in-memory"},
status_code=200,
@ -691,6 +831,13 @@ def config_set(request: Request, body: AppConfigSetBody):
updates = flatten_config_data(body.config_data)
# Convert None values to empty strings for deletion (e.g., when deleting masks)
updates = {k: ("" if v is None else v) for k, v in updates.items()}
# Drop sentinel-valued fields so untouched credential
# placeholders don't clobber the saved YAML value.
updates = {
k: v
for k, v in updates.items()
if v != REDACTED_CREDENTIAL_SENTINEL
}
if not updates:
return JSONResponse(
@ -761,6 +908,11 @@ def config_set(request: Request, body: AppConfigSetBody):
status_code=500,
)
# drop runtime overrides for any fields the user just rewrote in
# yaml so a stale override doesn't silently win after restart
if request.app.dispatcher is not None:
request.app.dispatcher.clear_runtime_state_for_yaml_keys(updates.keys())
if body.requires_restart == 0 or body.update_topic:
old_config: FrigateConfig = request.app.frigate_config
request.app.frigate_config = config
@ -774,6 +926,8 @@ def config_set(request: Request, body: AppConfigSetBody):
if request.app.dispatcher is not None:
request.app.dispatcher.config = config
for comm in request.app.dispatcher.comms:
comm.config = config
if body.update_topic:
if body.update_topic.startswith("config/cameras/"):

118
frigate/api/camera.py
View File

@ -529,6 +529,68 @@ def _extract_fps(r_frame_rate: str) -> float | None:
return None
def _build_digest_transport(username: str, password: str) -> AsyncTransport:
"""Build a zeep transport backed by an httpx client using HTTP digest auth."""
auth = httpx.DigestAuth(username, password)
client = httpx.AsyncClient(auth=auth, timeout=10.0)
return AsyncTransport(client=client)
async def _connect_onvif_camera(
host: str,
port: int,
username: str,
password: str,
wsdl_base: str | None,
auth_type: str,
) -> ONVIFCamera:
"""Connect to an ONVIF device, trying both WS-Security password encodings.
Cameras disagree on whether the WS-Security UsernameToken should carry a
hashed PasswordDigest or a plaintext PasswordText. The wizard can't know
which a given camera expects, so we try PasswordDigest first (the common
case) and fall back to PasswordText when the device rejects the token. This
is independent of auth_type, which controls HTTP transport-level auth.
"""
first_error: Fault | None = None
# encrypt=True -> PasswordDigest, encrypt=False -> PasswordText
for encrypt in (True, False):
onvif_camera = ONVIFCamera(
host,
port,
username or "",
password or "",
wsdl_dir=wsdl_base,
encrypt=encrypt,
)
try:
await onvif_camera.update_xaddrs()
except Fault as e:
# A SOAP fault here is how a camera signals the wrong password
# encoding, so retry with the other encoding before giving up.
logger.debug(
"ONVIF connect with %s rejected, trying alternate encoding",
"PasswordDigest" if encrypt else "PasswordText",
)
if first_error is None:
first_error = e
continue
if auth_type == "digest" and username and password:
transport = _build_digest_transport(username, password)
for service in ("devicemgmt", "media", "ptz"):
if hasattr(onvif_camera, service):
getattr(onvif_camera, service).zeep_client.transport = transport
logger.debug("Configured digest authentication")
return onvif_camera
# Both encodings failed authentication; surface the original fault.
raise first_error
@router.get(
"/onvif/probe",
dependencies=[Depends(require_role(["admin"]))],
@ -605,34 +667,10 @@ async def onvif_probe(
except Exception:
wsdl_base = None
onvif_camera = ONVIFCamera(
host, port, username or "", password or "", wsdl_dir=wsdl_base
onvif_camera = await _connect_onvif_camera(
host, port, username, password, wsdl_base, auth_type
)
# Configure digest authentication if requested
if auth_type == "digest" and username and password:
# Create httpx client with digest auth
auth = httpx.DigestAuth(username, password)
client = httpx.AsyncClient(auth=auth, timeout=10.0)
# Replace the transport in the zeep client
transport = AsyncTransport(client=client)
# Update the xaddr before setting transport
await onvif_camera.update_xaddrs()
# Replace transport in all services
if hasattr(onvif_camera, "devicemgmt"):
onvif_camera.devicemgmt.zeep_client.transport = transport
if hasattr(onvif_camera, "media"):
onvif_camera.media.zeep_client.transport = transport
if hasattr(onvif_camera, "ptz"):
onvif_camera.ptz.zeep_client.transport = transport
logger.debug("Configured digest authentication")
else:
await onvif_camera.update_xaddrs()
# Get device information
device_info = {
"manufacturer": "Unknown",
@ -644,10 +682,9 @@ async def onvif_probe(
# Update transport for device service if digest auth
if auth_type == "digest" and username and password:
auth = httpx.DigestAuth(username, password)
client = httpx.AsyncClient(auth=auth, timeout=10.0)
transport = AsyncTransport(client=client)
device_service.zeep_client.transport = transport
device_service.zeep_client.transport = _build_digest_transport(
username, password
)
device_info_resp = await device_service.GetDeviceInformation()
manufacturer = getattr(device_info_resp, "Manufacturer", None) or (
@ -685,10 +722,9 @@ async def onvif_probe(
# Update transport for media service if digest auth
if auth_type == "digest" and username and password:
auth = httpx.DigestAuth(username, password)
client = httpx.AsyncClient(auth=auth, timeout=10.0)
transport = AsyncTransport(client=client)
media_service.zeep_client.transport = transport
media_service.zeep_client.transport = _build_digest_transport(
username, password
)
profiles = await media_service.GetProfiles()
profiles_count = len(profiles) if profiles else 0
@ -720,10 +756,9 @@ async def onvif_probe(
# Update transport for PTZ service if digest auth
if auth_type == "digest" and username and password:
auth = httpx.DigestAuth(username, password)
client = httpx.AsyncClient(auth=auth, timeout=10.0)
transport = AsyncTransport(client=client)
ptz_service.zeep_client.transport = transport
ptz_service.zeep_client.transport = _build_digest_transport(
username, password
)
# Check if PTZ service is available
try:
@ -876,10 +911,9 @@ async def onvif_probe(
# Update transport for media service if digest auth
if auth_type == "digest" and username and password:
auth = httpx.DigestAuth(username, password)
client = httpx.AsyncClient(auth=auth, timeout=10.0)
transport = AsyncTransport(client=client)
media_service.zeep_client.transport = transport
media_service.zeep_client.transport = _build_digest_transport(
username, password
)
if profiles_count and media_service:
for p in profiles or []:

523
frigate/api/chat.py
View File

@ -35,9 +35,13 @@ from frigate.api.defs.response.chat_response import (
ToolCall,
)
from frigate.api.defs.tags import Tags
from frigate.api.event import events
from frigate.api.event import _build_attribute_filter_clause, events
from frigate.config import FrigateConfig
from frigate.config.ui import UnitSystemEnum
from frigate.genai.prompts import (
build_chat_system_prompt,
get_attribute_classifications,
get_tool_definitions,
)
from frigate.genai.utils import build_assistant_message_for_conversation
from frigate.jobs.vlm_watch import (
get_vlm_watch_job,
@ -68,390 +72,6 @@ class VLMMonitorRequest(BaseModel):
zones: List[str] = []
def get_tool_definitions(
semantic_search_enabled: bool = False,
) -> List[Dict[str, Any]]:
"""
Get OpenAI-compatible tool definitions for Frigate.
Returns a list of tool definitions that can be used with OpenAI-compatible
function calling APIs. When semantic search is enabled, the search_objects
tool exposes an additional `semantic_query` parameter for descriptive
queries (e.g. "person riding a lawn mower") and find_similar_objects is
included.
"""
search_objects_properties: Dict[str, Any] = {
"camera": {
"type": "string",
"description": "Camera name to filter by (optional).",
},
"label": {
"type": "string",
"description": (
"Generic object class to filter by — one of the tracked detector "
"labels such as 'person', 'package', 'car', 'dog', 'bird'. Use "
"this for broad queries like 'show me all cars today'. Combine "
"with semantic_query when the user also describes appearance or "
"behavior (e.g. label='person', semantic_query='riding a lawn "
"mower')."
),
},
"sub_label": {
"type": "string",
"description": (
"Filter by a DISCRETE NAMED entity recognized in the detection. "
"Use this for: a known person's name ('John'), a delivery "
"company ('Amazon', 'UPS'), a recognized animal species or "
"breed ('blue jay', 'cardinal', 'golden retriever'), or a "
"license plate string. When filtering by a specific name, set "
"only sub_label and leave label unset. Do NOT use sub_label "
"for descriptions of appearance, clothing, or actions — those "
"belong in semantic_query."
),
},
"after": {
"type": "string",
"description": "Start time in ISO 8601 format (e.g., '2024-01-01T00:00:00Z').",
},
"before": {
"type": "string",
"description": "End time in ISO 8601 format (e.g., '2024-01-01T23:59:59Z').",
},
"zones": {
"type": "array",
"items": {"type": "string"},
"description": "List of zone names to filter by.",
},
"limit": {
"type": "integer",
"description": "Maximum number of objects to return (default: 25).",
"default": 25,
},
}
if semantic_search_enabled:
search_objects_properties["semantic_query"] = {
"type": "string",
"description": (
"Optional natural-language description of a PHYSICAL "
"CHARACTERISTIC, APPEARANCE, or ACTIVITY the user mentioned, "
"used to semantically narrow results. Only set this when the "
"user describes something beyond what label and sub_label can "
"express on their own.\n"
"USE for descriptive phrases like: 'riding a lawn mower', "
"'wearing a red jacket', 'carrying a package', 'walking a "
"dog', 'on a bicycle', 'holding an umbrella'.\n"
"DO NOT USE for:\n"
"- specific named people, pets, or delivery companies → use sub_label\n"
"- animal species or breed names like 'blue jay', 'cardinal', "
"'golden retriever' → use sub_label\n"
"- license plate strings → use sub_label\n"
"- generic object queries like 'all cars today' or 'every "
"person' → use label alone with no semantic_query\n"
"When set, combine with label/time/camera/zone filters as "
"usual (e.g. label='person', semantic_query='riding a lawn "
"mower', after='2024-05-01T00:00:00Z')."
),
}
search_objects_description = (
"Search the historical record of detected objects in Frigate. "
"Use this ONLY for questions about the PAST — e.g. 'did anyone come by today?', "
"'when was the last car?', 'show me detections from yesterday'. "
"Do NOT use this for monitoring or alerting requests about future events — "
"use start_camera_watch instead for those. "
"An 'object' in Frigate represents a tracked detection (e.g., a person, package, car).\n\n"
"Choose filters based on what the user is asking for:\n"
"- Generic class query ('show me all cars today'): set `label` only.\n"
"- Specific NAMED entity (known person, delivery company, animal "
"species/breed like 'blue jay' or 'golden retriever', license "
"plate): set `sub_label` only and leave `label` unset.\n"
)
if semantic_search_enabled:
search_objects_description += (
"- Physical CHARACTERISTIC, APPEARANCE, or ACTIVITY that is not a "
"discrete name ('person riding a lawn mower', 'someone in a red "
"jacket', 'person carrying a package'): set `semantic_query` with "
"the descriptive phrase, optionally alongside `label` for the "
"object class. Do NOT put descriptive phrases in sub_label."
)
return [
{
"type": "function",
"function": {
"name": "search_objects",
"description": search_objects_description,
"parameters": {
"type": "object",
"properties": search_objects_properties,
},
"required": [],
},
},
{
"type": "function",
"function": {
"name": "find_similar_objects",
"description": (
"Find tracked objects that are visually and semantically similar "
"to a specific past event. Use this when the user references a "
"particular object they have seen and wants to find other "
"sightings of the same or similar one ('that green car', 'the "
"person in the red jacket', 'the package that was delivered'). "
"Prefer this over search_objects whenever the user's intent is "
"'find more like this specific one.' Use search_objects first "
"only if you need to locate the anchor event. Requires semantic "
"search to be enabled."
),
"parameters": {
"type": "object",
"properties": {
"event_id": {
"type": "string",
"description": "The id of the anchor event to find similar objects to.",
},
"after": {
"type": "string",
"description": "Start time in ISO 8601 format (e.g., '2024-01-01T00:00:00Z').",
},
"before": {
"type": "string",
"description": "End time in ISO 8601 format (e.g., '2024-01-01T23:59:59Z').",
},
"cameras": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of cameras to restrict to. Defaults to all.",
},
"labels": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of labels to restrict to. Defaults to the anchor event's label.",
},
"sub_labels": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of sub_labels (names) to restrict to.",
},
"zones": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of zones. An event matches if any of its zones overlap.",
},
"similarity_mode": {
"type": "string",
"enum": ["visual", "semantic", "fused"],
"description": "Which similarity signal(s) to use. 'fused' (default) combines visual and semantic.",
"default": "fused",
},
"min_score": {
"type": "number",
"description": "Drop matches with a similarity score below this threshold (0.0-1.0).",
},
"limit": {
"type": "integer",
"description": "Maximum number of matches to return (default: 10).",
"default": 10,
},
},
"required": ["event_id"],
},
},
},
{
"type": "function",
"function": {
"name": "set_camera_state",
"description": (
"Change a camera's feature state (e.g., turn detection on/off, enable/disable recordings). "
"Use camera='*' to apply to all cameras at once. "
"Only call this tool when the user explicitly asks to change a camera setting. "
"Requires admin privileges."
),
"parameters": {
"type": "object",
"properties": {
"camera": {
"type": "string",
"description": "Camera name to target, or '*' to target all cameras.",
},
"feature": {
"type": "string",
"enum": [
"detect",
"record",
"snapshots",
"audio",
"motion",
"enabled",
"birdseye",
"birdseye_mode",
"improve_contrast",
"ptz_autotracker",
"motion_contour_area",
"motion_threshold",
"notifications",
"audio_transcription",
"review_alerts",
"review_detections",
"object_descriptions",
"review_descriptions",
"profile",
],
"description": (
"The feature to change. Most features accept ON or OFF. "
"birdseye_mode accepts CONTINUOUS, MOTION, or OBJECTS. "
"motion_contour_area and motion_threshold accept a number. "
"profile accepts a profile name or 'none' to deactivate (requires camera='*')."
),
},
"value": {
"type": "string",
"description": "The value to set. ON or OFF for toggles, a number for thresholds, a profile name or 'none' for profile.",
},
},
"required": ["camera", "feature", "value"],
},
},
},
{
"type": "function",
"function": {
"name": "get_live_context",
"description": (
"Get the current live image and detection information for a camera: objects being tracked, "
"zones, timestamps. Use this to understand what is visible in the live view. "
"Call this when answering questions about what is happening right now on a specific camera."
),
"parameters": {
"type": "object",
"properties": {
"camera": {
"type": "string",
"description": "Camera name to get live context for.",
},
},
"required": ["camera"],
},
},
},
{
"type": "function",
"function": {
"name": "start_camera_watch",
"description": (
"Start a continuous VLM watch job that monitors a camera and sends a notification "
"when a specified condition is met. Use this when the user wants to be alerted about "
"a future event, e.g. 'tell me when guests arrive' or 'notify me when the package is picked up'. "
"Only one watch job can run at a time. Returns a job ID."
),
"parameters": {
"type": "object",
"properties": {
"camera": {
"type": "string",
"description": "Camera ID to monitor.",
},
"condition": {
"type": "string",
"description": (
"Natural-language description of the condition to watch for, "
"e.g. 'a person arrives at the front door'."
),
},
"max_duration_minutes": {
"type": "integer",
"description": "Maximum time to watch before giving up (minutes, default 60).",
"default": 60,
},
"labels": {
"type": "array",
"items": {"type": "string"},
"description": "Object labels that should trigger a VLM check (e.g. ['person', 'car']). If omitted, any detection on the camera triggers a check.",
},
"zones": {
"type": "array",
"items": {"type": "string"},
"description": "Zone names to filter by. If specified, only detections in these zones trigger a VLM check.",
},
},
"required": ["camera", "condition"],
},
},
},
{
"type": "function",
"function": {
"name": "stop_camera_watch",
"description": (
"Cancel the currently running VLM watch job. Use this when the user wants to "
"stop a previously started watch, e.g. 'stop watching the front door'."
),
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "get_profile_status",
"description": (
"Get the current profile status including the active profile and "
"timestamps of when each profile was last activated. Use this to "
"determine time periods for recap requests — e.g. when the user asks "
"'what happened while I was away?', call this first to find the relevant "
"time window based on profile activation history."
),
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "get_recap",
"description": (
"Get a recap of all activity (alerts and detections) for a given time period. "
"Use this after calling get_profile_status to retrieve what happened during "
"a specific window — e.g. 'what happened while I was away?'. Returns a "
"chronological list of activity with camera, objects, zones, and GenAI-generated "
"descriptions when available. Summarize the results for the user."
),
"parameters": {
"type": "object",
"properties": {
"after": {
"type": "string",
"description": "Start of the time period in ISO 8601 format (e.g. '2025-03-15T08:00:00').",
},
"before": {
"type": "string",
"description": "End of the time period in ISO 8601 format (e.g. '2025-03-15T17:00:00').",
},
"cameras": {
"type": "string",
"description": "Comma-separated camera IDs to include, or 'all' for all cameras. Default is 'all'.",
},
"severity": {
"type": "string",
"enum": ["alert", "detection"],
"description": "Filter by severity level. Omit to include both alerts and detections.",
},
},
"required": ["after", "before"],
},
},
},
]
@router.get(
"/chat/tools",
dependencies=[Depends(allow_any_authenticated())],
@ -460,10 +80,13 @@ def get_tool_definitions(
)
def get_tools(request: Request) -> JSONResponse:
"""Get list of available tools for LLM function calling."""
semantic_search_enabled = bool(
getattr(request.app.frigate_config.semantic_search, "enabled", False)
config = request.app.frigate_config
semantic_search_enabled = bool(getattr(config.semantic_search, "enabled", False))
attribute_classifications = get_attribute_classifications(config)
tools = get_tool_definitions(
semantic_search_enabled=semantic_search_enabled,
attribute_classifications=attribute_classifications,
)
tools = get_tool_definitions(semantic_search_enabled=semantic_search_enabled)
return JSONResponse(content={"tools": tools})
@ -554,11 +177,14 @@ async def _execute_search_objects(
elif zones is None:
zones = "all"
attribute = arguments.get("attribute")
# Build query parameters compatible with EventsQueryParams
query_params = EventsQueryParams(
cameras=arguments.get("camera", "all"),
labels=arguments.get("label", "all"),
sub_labels=arguments.get("sub_label", "all"), # case-insensitive on the backend
attributes=attribute if attribute else "all",
zones=zones,
zone=zones,
after=after,
@ -626,6 +252,7 @@ async def _execute_search_objects_semantic(
label = arguments.get("label")
sub_label = arguments.get("sub_label")
attribute = arguments.get("attribute")
zones = arguments.get("zones")
if isinstance(zones, list) and zones:
@ -668,6 +295,10 @@ async def _execute_search_objects_semantic(
if sub_label:
# case-insensitive match to mirror events() behavior
clauses.append(fn.LOWER(Event.sub_label.cast("text")) == sub_label.lower())
if attribute:
attribute_clause = _build_attribute_filter_clause(attribute)
if attribute_clause is not None:
clauses.append(attribute_clause)
if zones:
zone_clauses = [Event.zones.cast("text") % f'*"{zone}"*' for zone in zones]
clauses.append(reduce(operator.or_, zone_clauses))
@ -916,9 +547,21 @@ async def _execute_get_live_context(
camera: str,
allowed_cameras: List[str],
) -> Dict[str, Any]:
# Reject wildcards explicitly so models retry with a real camera name
# instead of silently fanning out across every camera.
if camera in ("*", "all"):
return {
"error": (
"get_live_context requires a single camera name; wildcards "
"are not supported. Call this tool once per camera."
),
"available_cameras": allowed_cameras,
}
if camera not in allowed_cameras:
return {
"error": f"Camera '{camera}' not found or access denied",
"available_cameras": allowed_cameras,
}
if camera not in request.app.frigate_config.cameras:
@ -1090,7 +733,14 @@ async def _execute_tool_internal(
"Arguments: %s",
json.dumps(arguments),
)
return {"error": "Camera parameter is required"}
return {
"error": (
"get_live_context requires a single camera name; "
"wildcards and empty values are not supported. "
"Call this tool once per camera."
),
"available_cameras": allowed_cameras,
}
return await _execute_get_live_context(request, camera, allowed_cameras)
elif tool_name == "start_camera_watch":
return await _execute_start_camera_watch(request, arguments)
@ -1481,72 +1131,19 @@ async def chat_completion(
config = request.app.frigate_config
semantic_search_enabled = bool(getattr(config.semantic_search, "enabled", False))
tools = get_tool_definitions(semantic_search_enabled=semantic_search_enabled)
attribute_classifications = get_attribute_classifications(config)
tools = get_tool_definitions(
semantic_search_enabled=semantic_search_enabled,
attribute_classifications=attribute_classifications,
)
conversation = []
current_datetime = datetime.now()
current_date_str = current_datetime.strftime("%Y-%m-%d")
current_time_str = current_datetime.strftime("%I:%M:%S %p")
cameras_info = []
has_speed_zone = False
for camera_id in allowed_cameras:
if camera_id not in config.cameras:
continue
camera_config = config.cameras[camera_id]
friendly_name = (
camera_config.friendly_name
if camera_config.friendly_name
else camera_id.replace("_", " ").title()
system_prompt = build_chat_system_prompt(
config=config,
allowed_cameras=allowed_cameras,
semantic_search_enabled=semantic_search_enabled,
attribute_classifications=attribute_classifications,
)
zone_names = list(camera_config.zones.keys())
if not has_speed_zone:
has_speed_zone = any(
zone.distances for zone in camera_config.zones.values()
)
if zone_names:
cameras_info.append(
f" - {friendly_name} (ID: {camera_id}, zones: {', '.join(zone_names)})"
)
else:
cameras_info.append(f" - {friendly_name} (ID: {camera_id})")
cameras_section = ""
if cameras_info:
cameras_section = (
"\n\nAvailable cameras:\n"
+ "\n".join(cameras_info)
+ "\n\nWhen users refer to cameras by their friendly name (e.g., 'Back Deck Camera'), use the corresponding camera ID (e.g., 'back_deck_cam') in tool calls."
)
speed_units_section = ""
if has_speed_zone:
speed_unit = (
"mph" if config.ui.unit_system == UnitSystemEnum.imperial else "km/h"
)
speed_units_section = f"\n\nReport object speeds to the user in {speed_unit}."
semantic_search_section = ""
if semantic_search_enabled:
semantic_search_section = (
"\n\nWhen routing a search_objects call, pick filters by the shape of the user's request:\n"
"- Generic class ('show me all cars today'): set `label` only.\n"
"- Specific named entity — a known person ('John'), delivery company ('Amazon'), animal species/breed ('blue jay', 'cardinal', 'golden retriever'), or license plate: set `sub_label` only and leave `label` unset.\n"
"- Physical characteristic, appearance, or activity that is NOT a discrete name ('find me people riding a lawn mower', 'someone in a red jacket', 'a person carrying a package'): set `semantic_query` with the descriptive phrase, optionally combined with `label` for the object class. Never put descriptive phrases in `sub_label`."
)
system_prompt = f"""You are a helpful assistant for Frigate, a security camera NVR system. You help users answer questions about their cameras, detected objects, and events.
Current server local date and time: {current_date_str} at {current_time_str}
Do not start your response with phrases like "I will check...", "Let me see...", or "Let me look...". Answer directly.
Always present times to the user in the server's local timezone. When tool results include start_time_local and end_time_local, use those exact strings when listing or describing detection times—do not convert or invent timestamps. Do not use UTC or ISO format with Z for the user-facing answer unless the tool result only provides Unix timestamps without local time fields.
When users ask about "today", "yesterday", "this week", etc., use the current date above as reference.
When searching for objects or events, use ISO 8601 format for dates (e.g., {current_date_str}T00:00:00Z for the start of today).
Always be accurate with time calculations based on the current date provided.
When a user refers to a specific object they have seen or describe with identifying details ("that green car", "the person in the red jacket", "a package left today"), prefer the find_similar_objects tool over search_objects. Use search_objects first only to locate the anchor event, then pass its id to find_similar_objects. For generic queries like "show me all cars today", keep using search_objects. If a user message begins with [attached_event:<id>], treat that event id as the anchor for any similarity or "tell me more" request in the same message and call find_similar_objects with that id.{semantic_search_section}{cameras_section}{speed_units_section}"""
conversation.append(
{
@ -1595,6 +1192,7 @@ When a user refers to a specific object they have seen or describe with identify
messages=conversation,
tools=tools if tools else None,
tool_choice="auto",
enable_thinking=body.enable_thinking,
):
if await request.is_disconnected():
logger.debug("Client disconnected, stopping chat stream")
@ -1607,6 +1205,13 @@ When a user refers to a specific object they have seen or describe with identify
)
+ b"\n"
)
elif kind == "reasoning_delta":
yield (
json.dumps({"type": "reasoning", "delta": value}).encode(
"utf-8"
)
+ b"\n"
)
elif kind == "stats":
yield (
json.dumps({"type": "stats", **value}).encode("utf-8")
@ -1682,6 +1287,7 @@ When a user refers to a specific object they have seen or describe with identify
messages=conversation,
tools=tools if tools else None,
tool_choice="auto",
enable_thinking=body.enable_thinking,
)
if response.get("finish_reason") == "error":
@ -1707,6 +1313,7 @@ When a user refers to a specific object they have seen or describe with identify
final_content = response.get("content") or ""
if body.stream:
final_reasoning = response.get("reasoning")
async def stream_body() -> Any:
if tool_calls:
@ -1721,6 +1328,15 @@ When a user refers to a specific object they have seen or describe with identify
).encode("utf-8")
+ b"\n"
)
# Emit the full reasoning trace up front when the
# underlying client did not stream it
if final_reasoning:
yield (
json.dumps(
{"type": "reasoning", "delta": final_reasoning}
).encode("utf-8")
+ b"\n"
)
# Stream content in word-sized chunks for smooth UX
for part in chunk_content(final_content):
yield (
@ -1741,6 +1357,7 @@ When a user refers to a specific object they have seen or describe with identify
message=ChatMessageResponse(
role="assistant",
content=final_content,
reasoning=response.get("reasoning"),
tool_calls=None,
),
finish_reason=response.get("finish_reason", "stop"),

8
frigate/api/classification.py
View File

@ -280,7 +280,7 @@ async def create_face(request: Request, name: str):
success response with details about the registration, or an error if face recognition
is not enabled or the image cannot be processed.""",
)
async def register_face(request: Request, name: str, file: UploadFile):
def register_face(request: Request, name: str, file: UploadFile):
if not request.app.frigate_config.face_recognition.enabled:
return JSONResponse(
status_code=400,
@ -288,7 +288,7 @@ async def register_face(request: Request, name: str, file: UploadFile):
)
context: EmbeddingsContext = request.app.embeddings
result = None if context is None else context.register_face(name, await file.read())
result = None if context is None else context.register_face(name, file.file.read())
if not isinstance(result, dict):
return JSONResponse(
@ -313,7 +313,7 @@ async def register_face(request: Request, name: str, file: UploadFile):
registered faces in the system. Returns the recognized face name and confidence score,
or an error if face recognition is not enabled or the image cannot be processed.""",
)
async def recognize_face(request: Request, file: UploadFile):
def recognize_face(request: Request, file: UploadFile):
if not request.app.frigate_config.face_recognition.enabled:
return JSONResponse(
status_code=400,
@ -321,7 +321,7 @@ async def recognize_face(request: Request, file: UploadFile):
)
context: EmbeddingsContext = request.app.embeddings
result = context.recognize_face(await file.read())
result = context.recognize_face(file.file.read())
if not isinstance(result, dict):
return JSONResponse(

108
frigate/api/debug_replay.py
View File

@ -6,11 +6,18 @@ from datetime import datetime
from fastapi import APIRouter, Depends, Request
from fastapi.responses import JSONResponse
from peewee import DoesNotExist
from pydantic import BaseModel, Field
from frigate.api.auth import require_role
from frigate.api.defs.tags import Tags
from frigate.jobs.debug_replay import start_debug_replay_job
from frigate.jobs.debug_replay import (
ExportDebugReplaySource,
RecordingDebugReplaySource,
start_debug_replay_job,
)
from frigate.models import Export
from frigate.util.services import get_video_properties
logger = logging.getLogger(__name__)
@ -25,6 +32,12 @@ class DebugReplayStartBody(BaseModel):
end_time: float = Field(title="End timestamp")
class DebugReplayStartFromExportBody(BaseModel):
"""Request body for starting a debug replay session from an export."""
export_id: str = Field(title="Export id")
class DebugReplayStartResponse(BaseModel):
"""Response for starting a debug replay session."""
@ -73,13 +86,100 @@ class DebugReplayStopResponse(BaseModel):
async def start_debug_replay(request: Request, body: DebugReplayStartBody):
"""Start a debug replay session asynchronously."""
replay_manager = request.app.replay_manager
internal_port = request.app.frigate_config.networking.listen.internal
if type(internal_port) is str:
internal_port = int(internal_port.split(":")[-1])
source = RecordingDebugReplaySource(
source_camera=body.camera,
start_ts=body.start_time,
end_ts=body.end_time,
internal_port=internal_port,
)
try:
job_id = await asyncio.to_thread(
start_debug_replay_job,
source_camera=body.camera,
start_ts=body.start_time,
end_ts=body.end_time,
source=source,
frigate_config=request.app.frigate_config,
config_publisher=request.app.config_publisher,
replay_manager=replay_manager,
)
except RuntimeError:
return JSONResponse(
content={
"success": False,
"message": "A replay session is already active",
},
status_code=409,
)
except ValueError:
logger.exception("Rejected debug replay start request")
return JSONResponse(
content={
"success": False,
"message": "Invalid debug replay parameters",
},
status_code=400,
)
return JSONResponse(
content={
"success": True,
"replay_camera": replay_manager.replay_camera_name,
"job_id": job_id,
},
status_code=202,
)
@router.post(
"/debug_replay/start_from_export",
response_model=DebugReplayStartResponse,
status_code=202,
responses={
400: {"description": "Invalid export, time range, or no recordings"},
404: {"description": "Export not found"},
409: {"description": "A replay session is already active"},
},
dependencies=[Depends(require_role(["admin"]))],
summary="Start debug replay from an export",
description="Start a debug replay session covering an existing export's "
"time range. The end time is derived from the export's video duration.",
)
async def start_debug_replay_from_export(
request: Request, body: DebugReplayStartFromExportBody
):
"""Start a debug replay session from an existing export."""
try:
export: Export = Export.get(Export.id == body.export_id)
except DoesNotExist:
return JSONResponse(
content={"success": False, "message": "Export not found"},
status_code=404,
)
properties = await get_video_properties(
request.app.frigate_config.ffmpeg, export.video_path, get_duration=True
)
duration = properties.get("duration", -1)
if duration is None or duration <= 0:
return JSONResponse(
content={
"success": False,
"message": "Could not determine export duration",
},
status_code=400,
)
replay_manager = request.app.replay_manager
source = ExportDebugReplaySource(export=export, duration=float(duration))
try:
job_id = await asyncio.to_thread(
start_debug_replay_job,
source=source,
frigate_config=request.app.frigate_config,
config_publisher=request.app.config_publisher,
replay_manager=replay_manager,

9
frigate/api/defs/request/app_body.py
View File

@ -2,6 +2,8 @@ from typing import Any, Dict, List, Optional
from pydantic import BaseModel, Field
from frigate.config import GenAIProviderEnum
class AppConfigSetBody(BaseModel):
requires_restart: int = 1
@ -10,6 +12,13 @@ class AppConfigSetBody(BaseModel):
skip_save: bool = False
class GenAIProbeBody(BaseModel):
provider: GenAIProviderEnum
api_key: Optional[str] = None
base_url: Optional[str] = None
provider_options: Dict[str, Any] = Field(default_factory=dict)
class AppPutPasswordBody(BaseModel):
password: str
old_password: Optional[str] = None

7
frigate/api/defs/request/chat_body.py
View File

@ -36,3 +36,10 @@ class ChatCompletionRequest(BaseModel):
default=False,
description="If true, stream the final assistant response in the body as newline-delimited JSON.",
)
enable_thinking: Optional[bool] = Field(
default=None,
description=(
"Per-request thinking toggle. None means use the provider default. "
"Ignored by providers that do not expose a per-request thinking switch."
),
)

4
frigate/api/defs/response/chat_response.py
View File

@ -20,6 +20,10 @@ class ChatMessageResponse(BaseModel):
content: Optional[str] = Field(
default=None, description="Message content (None if tool calls present)"
)
reasoning: Optional[str] = Field(
default=None,
description="Separated reasoning/thinking trace if the model emitted one",
)
tool_calls: Optional[list[ToolCallInvocation]] = Field(
default=None, description="Tool calls if LLM wants to call tools"
)

2
frigate/api/export.py
View File

@ -398,7 +398,7 @@ class _StreamingZipBuffer:
def _unique_archive_name(export: Export, used: set[str]) -> str:
base = sanitize_filename(export.name) if export.name else None
if not base:
base = f"{export.camera}_{int(datetime.datetime.timestamp(export.date))}"
base = f"{export.camera}_{int(export.date)}"
candidate = f"{base}.mp4"
counter = 1

8
frigate/api/fastapi_app.py
View File

@ -1,3 +1,4 @@
import asyncio
import logging
import re
from typing import Optional
@ -36,7 +37,7 @@ from frigate.comms.event_metadata_updater import (
from frigate.config import FrigateConfig
from frigate.config.camera.updater import CameraConfigUpdatePublisher
from frigate.config.profile_manager import ProfileManager
from frigate.debug_replay import DebugReplayManager
from frigate.debug_replay import DebugReplayManager, debug_replay_auto_stop_watchdog
from frigate.embeddings import EmbeddingsContext
from frigate.genai import GenAIClientManager
from frigate.ptz.onvif import OnvifController
@ -116,6 +117,11 @@ def create_fastapi_app(
@app.on_event("startup")
async def startup():
logger.info("FastAPI started")
asyncio.create_task(
debug_replay_auto_stop_watchdog(
replay_manager, frigate_config, config_publisher
)
)
# Rate limiter (used for login endpoint)
if frigate_config.auth.failed_login_rate_limit is None:

12
frigate/api/motion_search.py
View File

@ -41,12 +41,6 @@ class MotionSearchRequest(BaseModel):
le=100.0,
description="Minimum change area as a percentage of the ROI",
)
frame_skip: int = Field(
default=5,
ge=1,
le=30,
description="Process every Nth frame (1=all frames, 5=every 5th frame)",
)
parallel: bool = Field(
default=False,
description="Enable parallel scanning across segments",
@ -97,6 +91,8 @@ class MotionSearchStatusResponse(BaseModel):
total_frames_processed: Optional[int] = None
error_message: Optional[str] = None
metrics: Optional[MotionSearchMetricsResponse] = None
scanning_timestamp: Optional[float] = None
progress: Optional[float] = None
@router.post(
@ -151,7 +147,6 @@ async def start_motion_search(
polygon_points=body.polygon_points,
threshold=body.threshold,
min_area=body.min_area,
frame_skip=body.frame_skip,
parallel=body.parallel,
max_results=body.max_results,
)
@ -231,6 +226,9 @@ async def get_motion_search_status_endpoint(
if job.metrics:
response_content["metrics"] = job.metrics.to_dict()
response_content["scanning_timestamp"] = job.scanning_timestamp
response_content["progress"] = job.progress
return JSONResponse(content=response_content)

26
frigate/api/record.py
View File

@ -299,22 +299,36 @@ async def no_recordings(
.iterator()
)
# Convert recordings to list of (start, end) tuples
# Convert recordings to list of (start, end) tuples, ordered by start_time
recordings = [(r["start_time"], r["end_time"]) for r in data]
# Merge overlapping/adjacent recordings into covered intervals. The query
# orders by start_time, so a single pass merges them
covered: list[tuple[float, float]] = []
for rec_start, rec_end in recordings:
if covered and rec_start <= covered[-1][1]:
covered[-1] = (covered[-1][0], max(covered[-1][1], rec_end))
else:
covered.append((rec_start, rec_end))
# Iterate through time segments and check if each has any recording
no_recording_segments = []
current = after
current_gap_start = None
idx = 0
covered_count = len(covered)
while current < before:
segment_end = min(current + scale, before)
# Check if this segment overlaps with any recording
has_recording = any(
rec_start < segment_end and rec_end > current
for rec_start, rec_end in recordings
)
# Advance past covered intervals that end before this segment begins;
# they cannot overlap this or any later segment.
while idx < covered_count and covered[idx][1] <= current:
idx += 1
# A covered interval overlaps the segment when it starts before the
# segment ends (its end is already known to be > current).
has_recording = idx < covered_count and covered[idx][0] < segment_end
if not has_recording:
# This segment has no recordings

22
frigate/api/review.py
View File

@ -605,9 +605,10 @@ def motion_activity(
if not filtered:
return JSONResponse(content=[])
camera_list = list(filtered)
clauses.append((Recordings.camera << camera_list))
else:
clauses.append((Recordings.camera << allowed_cameras))
camera_list = list(allowed_cameras)
clauses.append((Recordings.camera << camera_list))
data: list[Recordings] = (
Recordings.select(
@ -635,13 +636,11 @@ def motion_activity(
df.set_index(["start_time"], inplace=True)
# normalize data
motion = (
df["motion"]
.resample(f"{scale}s")
.apply(lambda x: max(x, key=abs, default=0.0))
.fillna(0.0)
.to_frame()
)
motion = df["motion"].resample(f"{scale}s").max().fillna(0.0).to_frame()
if len(camera_list) == 1:
cameras = df["camera"].resample(f"{scale}s").first().fillna("")
else:
cameras = df["camera"].resample(f"{scale}s").agg(lambda x: ",".join(set(x)))
df = motion.join(cameras)
@ -658,6 +657,11 @@ def motion_activity(
else:
df.iloc[i : i + chunk, 0] = 0.0
# Drop resample gap-fill buckets. The resample above emits a row for every
# {scale}s bucket spanning the range, and buckets with no recording get a
# motion of 0 (from fillna) and an empty camera (from joining an empty set).
df = df[df["camera"] != ""]
# change types for output
df.index = df.index.astype(int) // (10**9)
normalized = df.reset_index().to_dict("records")

18
frigate/app.py
View File

@ -343,12 +343,24 @@ class FrigateApp:
)
self.dispatcher.profile_manager = self.profile_manager
def restore_active_profile(self) -> None:
"""Re-activate the persisted profile after subscribers are connected.
ZMQ PUB/SUB drops messages with no subscribers, so activation must
run after every config_updater subscriber is up.
"""
if self.profile_manager is None:
return
persisted = ProfileManager.load_persisted_profile()
if persisted and any(
persisted in cam.profiles for cam in self.config.cameras.values()
):
logger.info("Restoring persisted profile '%s'", persisted)
self.profile_manager.activate_profile(persisted)
# runtime overrides are layered on top via restore_runtime_state()
self.profile_manager.activate_profile(
persisted, clear_runtime_overrides=False
)
def start_detectors(self) -> None:
for name in self.config.cameras.keys():
@ -612,6 +624,10 @@ class FrigateApp:
self.start_record_cleanup()
self.start_watchdog()
# restore persisted runtime overrides on top of config
self.restore_active_profile()
self.dispatcher.restore_runtime_state()
self.init_auth()
try:

54
frigate/camera/maintainer.py
View File

@ -14,6 +14,7 @@ from frigate.config.camera.updater import (
CameraConfigUpdateEnum,
CameraConfigUpdateSubscriber,
)
from frigate.const import REPLAY_CAMERA_PREFIX
from frigate.models import Regions
from frigate.util.builtin import empty_and_close_queue
from frigate.util.image import SharedMemoryFrameManager, UntrackedSharedMemory
@ -50,6 +51,7 @@ class CameraMaintainer(threading.Thread):
[
CameraConfigUpdateEnum.add,
CameraConfigUpdateEnum.remove,
CameraConfigUpdateEnum.refresh,
],
)
self.shm_count = self.__calculate_shm_frame_count()
@ -202,6 +204,25 @@ class CameraMaintainer(threading.Thread):
capture_process.terminate()
capture_process.join()
def __unlink_camera_frame_slots(self, camera: str) -> None:
"""Drop the camera's per-frame YUV SHM segments from this
process's frame_manager and unlink them at the OS level.
Safe to call after the camera's capture/processor subprocesses
have been joined — they no longer hold mappings, so unlink frees
the segments immediately. Other long-lived processes that opened
these slots will continue using their existing mappings until
they call frame_manager.get with a shape that no longer fits
(the get path drops and reopens stale refs).
"""
prefix = f"{camera}_frame"
names = [n for n in list(self.frame_manager.shm_store) if n.startswith(prefix)]
for name in names:
try:
self.frame_manager.delete(name)
except Exception as exc:
logger.debug("Could not unlink SHM %s: %s", name, exc)
def __stop_camera_process(self, camera: str) -> None:
camera_process = self.camera_processes.get(camera)
if camera_process is not None:
@ -253,12 +274,45 @@ class CameraMaintainer(threading.Thread):
for camera in updated_cameras:
self.__stop_camera_capture_process(camera)
self.__stop_camera_process(camera)
self.__unlink_camera_frame_slots(camera)
self.capture_processes.pop(camera, None)
self.camera_processes.pop(camera, None)
self.camera_stop_events.pop(camera, None)
self.region_grids.pop(camera, None)
self.camera_metrics.pop(camera, None)
self.ptz_metrics.pop(camera, None)
elif update_type == CameraConfigUpdateEnum.refresh.name:
# Recycle replay cameras so detect width/height/fps
# propagate through ffmpeg args, SHM sizing, and the
# region grid. Regular cameras detect change still
# requires a full restart.
for camera in updated_cameras:
if not camera.startswith(REPLAY_CAMERA_PREFIX):
continue
new_config = self.update_subscriber.camera_configs.get(camera)
if new_config is None:
# remove arrived in the same batch
continue
if (
camera not in self.camera_processes
and camera not in self.capture_processes
):
continue
# rebuild ffmpeg cmds on the shared config so the
# new subprocesses spawn with current args
new_config.recreate_ffmpeg_cmds()
self.__stop_camera_capture_process(camera)
self.__stop_camera_process(camera)
self.__unlink_camera_frame_slots(camera)
self.capture_processes.pop(camera, None)
self.camera_processes.pop(camera, None)
self.__start_camera_processor(camera, new_config, runtime=True)
self.__start_camera_capture(camera, new_config, runtime=True)
# ensure the capture processes are done
for camera in self.capture_processes.keys():

48
frigate/camera/state.py
View File

@ -45,6 +45,7 @@ class CameraState:
self.frame_cache: dict[float, dict[str, Any]] = {}
self.zone_objects: defaultdict[str, list[Any]] = defaultdict(list)
self._current_frame = np.zeros(self.camera_config.frame_shape_yuv, np.uint8)
self._last_frame_shape: tuple[int, int] = self.camera_config.frame_shape_yuv
self.current_frame_lock = threading.Lock()
self.current_frame_time = 0.0
self.motion_boxes: list[tuple[int, int, int, int]] = []
@ -303,6 +304,42 @@ class CameraState:
def on(self, event_type: str, callback: Callable[..., Any]) -> None:
self.callbacks[event_type].append(callback)
def _discard_stale_resolution_state(
self, current_detections: dict[str, dict[str, Any]]
) -> bool:
"""Drop tracked state when the camera's detect resolution has
changed, and signal the caller to skip this batch if it contains
out-of-bounds boxes from the pre-recycle detect process.
Returns True when the batch should be skipped entirely.
"""
# detect resolution changed — drop tracked state so old-grid
# boxes don't leak through end-callbacks
current_shape = self.camera_config.frame_shape_yuv
if current_shape != self._last_frame_shape:
logger.debug(
f"{self.name}: detect resolution changed {self._last_frame_shape} -> {current_shape}, dropping tracked state"
)
with self.current_frame_lock:
self.tracked_objects.clear()
self.motion_boxes = []
self.regions = []
self._last_frame_shape = current_shape
# drop in-flight batches from the pre-recycle detect process
# whose boxes exceed the current detect resolution
detect = self.camera_config.detect
if detect.width is not None and detect.height is not None:
for obj in current_detections.values():
box = obj.get("box")
if box and (box[2] > detect.width or box[3] > detect.height):
logger.debug(
f"{self.name}: dropping stale-resolution detection batch (box {box} exceeds {detect.width}x{detect.height})"
)
return True
return False
def update(
self,
frame_name: str,
@ -311,6 +348,9 @@ class CameraState:
motion_boxes: list[tuple[int, int, int, int]],
regions: list[tuple[int, int, int, int]],
) -> None:
if self._discard_stale_resolution_state(current_detections):
return
current_frame = self.frame_manager.get(
frame_name, self.camera_config.frame_shape_yuv
)
@ -332,12 +372,16 @@ class CameraState:
current_detections[id],
)
# add initial frame to frame cache
# Skip caching when the frame buffer isn't readable — e.g.
# frame_manager.get returned None because the SHM segment was
# unlinked or hasn't been recreated yet during a camera
# add/remove cycle.
if current_frame is not None:
logger.debug(
f"{self.name}: New object, adding {frame_time} to frame cache for {id}"
)
self.frame_cache[frame_time] = {
"frame": np.copy(current_frame), # type: ignore[arg-type]
"frame": np.copy(current_frame),
"object_id": id,
}

62
frigate/comms/dispatcher.py
View File

@ -3,11 +3,13 @@
import datetime
import json
import logging
from collections.abc import Iterable
from typing import Any, Callable, Optional, cast
from frigate.camera import PTZMetrics
from frigate.camera.activity_manager import AudioActivityManager, CameraActivityManager
from frigate.comms.base_communicator import Communicator
from frigate.comms.runtime_state import RuntimeStatePersistence
from frigate.comms.webpush import WebPushClient
from frigate.config import BirdseyeModeEnum, FrigateConfig
from frigate.config.camera.updater import (
@ -67,6 +69,7 @@ class Dispatcher:
self.embeddings_reindex: dict[str, Any] = {}
self.birdseye_layout: dict[str, Any] = {}
self.audio_transcription_state: str = "idle"
self._runtime_state = RuntimeStatePersistence()
self._camera_settings_handlers: dict[str, Callable] = {
"audio": self._on_audio_command,
"audio_transcription": self._on_audio_transcription_command,
@ -397,6 +400,60 @@ class Dispatcher:
for comm in self.comms:
comm.stop()
def restore_runtime_state(self) -> None:
"""Replay persisted runtime overrides through the camera settings handlers.
Called once after Frigate startup completes so processing threads can
receive the resulting ``config_updater`` broadcasts. Unknown cameras
and topics are skipped; handler exceptions are logged and replay
continues for remaining entries.
"""
state = self._runtime_state.load()
for camera_name, features in state.items():
if camera_name not in self.config.cameras:
continue
for topic, value in features.items():
handler = self._camera_settings_handlers.get(topic)
if handler is None:
continue
payload = "ON" if value else "OFF"
try:
handler(camera_name, payload)
except Exception:
logger.exception(
"Failed to restore runtime state %s.%s=%s",
camera_name,
topic,
payload,
)
continue
logger.info(
"Restored runtime state: %s.%s=%s",
camera_name,
topic,
payload,
)
def clear_runtime_state_for_yaml_keys(self, dotted_keys: Iterable[str]) -> None:
"""Clear stored runtime overrides for YAML keys that were just rewritten.
Called by ``/api/config/set`` after a successful YAML save so an
explicit settings-UI save isn't silently overridden by an older
runtime toggle on the next restart.
"""
self._runtime_state.clear_for_yaml_keys(dotted_keys)
def clear_runtime_state(self) -> None:
"""Wipe every stored runtime override.
Called when a profile is activated or deactivated. A profile switch
changes the layer below the runtime overrides, so the stored
"steady state" is no longer valid and must be reset; otherwise a
subsequent restart would replay stale overrides on top of the new
profile-derived in-memory state.
"""
self._runtime_state.clear_all()
def _on_detect_command(self, camera_name: str, payload: str) -> None:
"""Callback for detect topic."""
detect_settings = self.config.cameras[camera_name].detect
@ -428,6 +485,7 @@ class Dispatcher:
CameraConfigUpdateTopic(CameraConfigUpdateEnum.detect, camera_name),
detect_settings,
)
self._runtime_state.set(camera_name, "detect", detect_settings.enabled)
self.publish(f"{camera_name}/detect/state", payload, retain=True)
def _on_enabled_command(self, camera_name: str, payload: str) -> None:
@ -452,6 +510,7 @@ class Dispatcher:
CameraConfigUpdateTopic(CameraConfigUpdateEnum.enabled, camera_name),
camera_settings.enabled,
)
self._runtime_state.set(camera_name, "enabled", camera_settings.enabled)
self.publish(f"{camera_name}/enabled/state", payload, retain=True)
def _on_motion_command(self, camera_name: str, payload: str) -> None:
@ -614,6 +673,7 @@ class Dispatcher:
CameraConfigUpdateTopic(CameraConfigUpdateEnum.audio, camera_name),
audio_settings,
)
self._runtime_state.set(camera_name, "audio", audio_settings.enabled)
self.publish(f"{camera_name}/audio/state", payload, retain=True)
def _on_audio_transcription_command(self, camera_name: str, payload: str) -> None:
@ -670,6 +730,7 @@ class Dispatcher:
CameraConfigUpdateTopic(CameraConfigUpdateEnum.record, camera_name),
record_settings,
)
self._runtime_state.set(camera_name, "recordings", record_settings.enabled)
self.publish(f"{camera_name}/recordings/state", payload, retain=True)
def _on_snapshots_command(self, camera_name: str, payload: str) -> None:
@ -689,6 +750,7 @@ class Dispatcher:
CameraConfigUpdateTopic(CameraConfigUpdateEnum.snapshots, camera_name),
snapshots_settings,
)
self._runtime_state.set(camera_name, "snapshots", snapshots_settings.enabled)
self.publish(f"{camera_name}/snapshots/state", payload, retain=True)
def _on_ptz_command(self, camera_name: str, payload: str | bytes) -> None:

163
frigate/comms/runtime_state.py Normal file
View File

@ -0,0 +1,163 @@
"""Persistence layer for dispatcher runtime state overrides."""
import json
import logging
import os
from collections.abc import Iterable
from typing import Any
from filelock import FileLock, Timeout
from frigate.util.config import find_config_file
logger = logging.getLogger(__name__)
class RuntimeStatePersistence:
"""Persist last-known runtime states for dispatcher toggles.
Stores boolean overrides applied to camera-level toggles by the dispatcher.
Overrides are replayed at startup on top of the YAML-derived in-memory
config, so changes made via MQTT or the live-view UI survive a restart.
"""
# Maps dispatcher topic name -> YAML key suffix under cameras.<cam>
TRACKED_TOPICS: dict[str, str] = {
"enabled": "enabled",
"detect": "detect.enabled",
"snapshots": "snapshots.enabled",
"recordings": "record.enabled",
"audio": "audio.enabled",
}
_SUFFIX_TO_TOPIC: dict[str, str] = {v: k for k, v in TRACKED_TOPICS.items()}
def __init__(self) -> None:
self._path = os.path.join(
os.path.dirname(find_config_file()), ".runtime_state.json"
)
self._lock_path = f"{self._path}.lock"
self._lock_timeout = 5
def load(self) -> dict[str, dict[str, bool]]:
"""Return {camera: {topic: bool}} or {} if missing/corrupt."""
try:
with FileLock(self._lock_path, timeout=self._lock_timeout):
data = self._read_locked()
except Timeout:
logger.error("Timed out acquiring runtime state lock for load")
return {}
cameras = data.get("cameras", {})
if not isinstance(cameras, dict):
return {}
# Filter out malformed camera entries so callers can trust the shape.
return {
name: features
for name, features in cameras.items()
if isinstance(features, dict)
}
def set(self, camera: str, topic: str, value: bool) -> None:
"""Persist a single (camera, topic, value). No-op if topic untracked."""
if topic not in self.TRACKED_TOPICS:
return
try:
with FileLock(self._lock_path, timeout=self._lock_timeout):
data = self._read_locked()
cameras = data.setdefault("cameras", {})
if not isinstance(cameras, dict):
cameras = {}
data["cameras"] = cameras
cam = cameras.setdefault(camera, {})
if not isinstance(cam, dict):
cam = {}
cameras[camera] = cam
cam[topic] = bool(value)
self._write_locked(data)
except Timeout:
logger.error("Timed out persisting runtime state for %s/%s", camera, topic)
except OSError:
logger.exception("Failed to persist runtime state for %s/%s", camera, topic)
def clear_all(self) -> None:
"""Wipe every stored runtime override.
Called when the "layer below" changes in a way that invalidates all
runtime overrides for the current session (currently: profile
activation or deactivation).
"""
try:
with FileLock(self._lock_path, timeout=self._lock_timeout):
if not os.path.exists(self._path):
return
self._write_locked({"cameras": {}})
except Timeout:
logger.error("Timed out clearing runtime state")
except OSError:
logger.exception("Failed to clear runtime state")
def clear_for_yaml_keys(self, dotted_keys: Iterable[str]) -> None:
"""Remove stored entries whose YAML key was just rewritten.
Each dotted key must be of the form ``cameras.<camera>.<suffix>``.
Keys that don't match a tracked topic are ignored.
"""
to_remove: list[tuple[str, str]] = []
for key in dotted_keys:
parts = key.split(".")
if len(parts) < 3 or parts[0] != "cameras":
continue
camera = parts[1]
suffix = ".".join(parts[2:])
topic = self._SUFFIX_TO_TOPIC.get(suffix)
if topic is not None:
to_remove.append((camera, topic))
if not to_remove:
return
try:
with FileLock(self._lock_path, timeout=self._lock_timeout):
data = self._read_locked()
cameras = data.get("cameras")
if not isinstance(cameras, dict):
return
changed = False
for camera, topic in to_remove:
cam = cameras.get(camera)
if isinstance(cam, dict) and topic in cam:
del cam[topic]
changed = True
if not cam:
del cameras[camera]
if changed:
self._write_locked(data)
except Timeout:
logger.error("Timed out clearing runtime state for YAML keys")
except OSError:
logger.exception("Failed to clear runtime state for YAML keys")
def _read_locked(self) -> dict[str, Any]:
"""Read the JSON file while the FileLock is held.
Returns ``{}`` on a missing or corrupt file so the caller can write a
fresh structure on the next mutation.
"""
if not os.path.exists(self._path):
return {}
try:
with open(self._path, "r") as f:
data = json.load(f)
except (OSError, json.JSONDecodeError):
logger.exception(
"Failed to read runtime state file %s; starting fresh", self._path
)
return {}
return data if isinstance(data, dict) else {}
def _write_locked(self, data: dict[str, Any]) -> None:
"""Atomically write the JSON file while the FileLock is held."""
tmp_path = f"{self._path}.tmp"
with open(tmp_path, "w") as f:
json.dump(data, f, indent=2, sort_keys=True)
os.replace(tmp_path, self._path)

358
frigate/comms/ws.py
View File

@ -34,6 +34,8 @@ from frigate.const import (
UPDATE_REVIEW_DESCRIPTION,
UPSERT_REVIEW_SEGMENT,
)
from frigate.models import User
from frigate.output.ws_auth import ws_has_camera_access
logger = logging.getLogger(__name__)
@ -66,6 +68,7 @@ _WS_VIEWER_TOPICS = frozenset(
"audioTranscriptionState",
"birdseyeLayout",
"embeddingsReindexProgress",
"jobState",
}
)
@ -102,6 +105,321 @@ def _check_ws_authorization(
return topic in _WS_VIEWER_TOPICS
# ---- Outbound filtering ---------------------------------------------------
#
# Every WebSocket broadcast is classified into one of a small set of scopes,
# then materialized per recipient. Connections with restricted roles only see
# data for cameras they are authorized to access; admin and full-access roles
# behave as today.
# Topics that are safe to broadcast to every authenticated client.
_WS_GLOBAL_OUTBOUND_TOPICS = frozenset(
{
"model_state",
"embeddings_reindex_progress",
"audio_transcription_state",
"profile/state",
"notifications/state",
"notification_test",
}
)
# Topics that restricted roles must never receive. Birdseye composites span
# all cameras, so the existing JSMPEG policy already restricts birdseye access
# to unrestricted roles; the layout broadcast follows the same rule.
_WS_UNRESTRICTED_ONLY_TOPICS = frozenset(
{
"birdseye_layout",
}
)
# Topics whose payload (parsed as JSON) names a single owning camera at the
# given key path. Used to scope events, reviews, triggers, etc.
_WS_PAYLOAD_CAMERA_TOPICS: dict[str, tuple[str, ...]] = {
"events": ("after", "camera"),
"reviews": ("after", "camera"),
"tracked_object_update": ("camera",),
"triggers": ("camera",),
"camera_monitoring": ("camera",),
}
# Topics whose payload is a dict keyed by camera name; filter keys per
# recipient.
_WS_RESHAPE_BY_CAMERA_KEY_TOPICS = frozenset(
{
"camera_activity",
"audio_detections",
}
)
# Topics whose payload is a dict keyed by job_type, where each entry may
# contain a "camera" or "source_camera" field, or a nested ``results.jobs``
# list of per-camera sub-jobs (export broadcasts).
_WS_RESHAPE_JOB_STATE_TOPICS = frozenset(
{
"job_state",
}
)
# Topics whose payload mixes global aggregates with a ``cameras`` sub-dict
# keyed by camera name. Aggregates and detector data stay; per-camera entries
# are filtered.
_WS_RESHAPE_STATS_TOPICS = frozenset(
{
"stats",
}
)
def _collect_zone_names(config: FrigateConfig) -> set[str]:
"""Return the set of all zone names defined across cameras."""
names: set[str] = set()
for camera in config.cameras.values():
zones = getattr(camera, "zones", None) or {}
names.update(zones.keys())
return names
def _parse_json_payload(payload: Any) -> Any:
"""Return payload parsed as JSON if it is a string, else as-is."""
if isinstance(payload, str):
try:
return json.loads(payload)
except (ValueError, TypeError):
return None
return payload
def _scope_job_entry_to_allowed(entry: Any, allowed: set[str]) -> dict[str, Any] | None:
"""Filter a single job_state entry to the recipient's allowed cameras.
Returns the (possibly reshaped) entry, or None to drop it. Four shapes
are handled:
* Top-level ``camera`` or ``source_camera`` (motion_search, vlm_watch,
export sub-job dicts): drop the entry if not allowed.
* Nested ``results.jobs`` list of per-camera sub-jobs (the aggregated
export broadcast): filter the list; drop the entry if nothing remains.
* Nested ``results.camera`` or ``results.source_camera`` (debug_replay,
which puts replay-specific fields inside ``results``): drop the entry
if not allowed.
* No camera anywhere (e.g. ``media_sync``): treat as global and keep.
"""
if not isinstance(entry, dict):
return None
cam = entry.get("camera") or entry.get("source_camera")
if cam is None:
results = entry.get("results")
if isinstance(results, dict):
sub_jobs = results.get("jobs")
if isinstance(sub_jobs, list):
filtered_jobs = [
j
for j in sub_jobs
if isinstance(j, dict)
and (j.get("camera") or j.get("source_camera")) in allowed
]
if not filtered_jobs:
return None
reshaped = dict(entry)
reshaped["results"] = dict(results)
reshaped["results"]["jobs"] = filtered_jobs
return reshaped
cam = results.get("camera") or results.get("source_camera")
if cam is not None:
return entry if cam in allowed else None
return entry
def _extract_payload_camera(payload: Any, path: tuple[str, ...]) -> str | None:
"""Walk the dotted path through a (possibly JSON-encoded) payload."""
cur = _parse_json_payload(payload)
for key in path:
if not isinstance(cur, dict):
return None
cur = cur.get(key)
return cur if isinstance(cur, str) else None
def _classify_outbound(
topic: str, all_cameras: set[str], all_zones: set[str]
) -> tuple[str, Any]:
"""Classify an outbound topic into (kind, extra).
kind values:
- "global" : send to every authenticated client
- "drop" : send to nobody (fail-closed for unknowns)
- "unrestricted_only" : send only to admin/full-access roles
- "camera" : extra is the owning camera name
- "payload_camera" : extra is the JSON key path to the camera name
- "reshape_by_camera_key"
- "reshape_job_state"
- "reshape_stats"
"""
if topic in _WS_GLOBAL_OUTBOUND_TOPICS:
return ("global", None)
if topic in _WS_UNRESTRICTED_ONLY_TOPICS:
return ("unrestricted_only", None)
if topic in _WS_RESHAPE_BY_CAMERA_KEY_TOPICS:
return ("reshape_by_camera_key", None)
if topic in _WS_RESHAPE_JOB_STATE_TOPICS:
return ("reshape_job_state", None)
if topic in _WS_RESHAPE_STATS_TOPICS:
return ("reshape_stats", None)
if topic in _WS_PAYLOAD_CAMERA_TOPICS:
return ("payload_camera", _WS_PAYLOAD_CAMERA_TOPICS[topic])
# Topic-prefix based: first segment names the owning camera or zone.
first = topic.split("/", 1)[0]
if first in all_cameras:
return ("camera", first)
if first in all_zones:
# Zone aggregates span cameras; restricted users see nothing here.
return ("unrestricted_only", None)
return ("drop", None)
def _ws_role_header(ws: Any) -> str | None:
"""Return the HTTP_REMOTE_ROLE header value, if any."""
environ = getattr(ws, "environ", None)
if not environ:
return None
value = environ.get("HTTP_REMOTE_ROLE")
return value if isinstance(value, str) else None
def _ws_valid_roles(ws: Any, config: FrigateConfig) -> list[str]:
"""Return the list of recognized roles for this connection."""
header = _ws_role_header(ws)
if not header:
return []
roles = [r.strip() for r in header.split(config.proxy.separator) if r.strip()]
return [r for r in roles if r in config.auth.roles]
def _ws_is_unrestricted(ws: Any, config: FrigateConfig) -> bool:
"""True when the connection has unrestricted camera access.
Mirrors the policy in ``frigate.output.ws_auth``: admin or any role with
an empty allow-list grants full access.
"""
roles = _ws_valid_roles(ws, config)
if not roles:
return False
roles_dict = config.auth.roles
return any(r == "admin" or not roles_dict.get(r) for r in roles)
def _ws_allowed_cameras(ws: Any, config: FrigateConfig) -> set[str]:
"""Return the union of cameras this connection may access across its roles."""
roles = _ws_valid_roles(ws, config)
if not roles:
return set()
all_cameras = set(config.cameras.keys())
allowed: set[str] = set()
for role in roles:
if role == "admin" or not config.auth.roles.get(role):
return all_cameras
allowed.update(User.get_allowed_cameras(role, config.auth.roles, all_cameras))
return allowed
def _wrap_envelope(topic: str, inner_payload: Any) -> str:
"""Re-serialize a (topic, payload) message after payload reshaping.
Frigate's wire format keeps payloads as JSON-encoded strings inside the
outer envelope, mirroring what producers send today.
"""
return json.dumps({"topic": topic, "payload": json.dumps(inner_payload)})
def _materialize_for_ws(
ws: Any,
topic: str,
full_message: str,
scope: tuple[str, Any],
parsed_payload: Any,
config: FrigateConfig,
) -> str | None:
"""Return the JSON string to deliver to ``ws``, or None to skip it."""
kind, extra = scope
has_role = _ws_role_header(ws) is not None
if kind == "drop":
return None
if kind == "global":
# Globals still require an authenticated connection. Missing role
# falls back to viewer semantics (matching the inbound rule).
return full_message
# Beyond globals, an authenticated role header is required (fail-closed).
if not has_role:
return None
if kind == "unrestricted_only":
return full_message if _ws_is_unrestricted(ws, config) else None
if kind == "camera":
return full_message if ws_has_camera_access(ws, extra, config) else None
if kind == "payload_camera":
camera = _extract_payload_camera(parsed_payload, extra)
if camera is None:
return None
return full_message if ws_has_camera_access(ws, camera, config) else None
if kind == "reshape_by_camera_key":
if _ws_is_unrestricted(ws, config):
return full_message
if not isinstance(parsed_payload, dict):
return None
allowed = _ws_allowed_cameras(ws, config)
filtered = {cam: data for cam, data in parsed_payload.items() if cam in allowed}
if not filtered:
return None
return _wrap_envelope(topic, filtered)
if kind == "reshape_job_state":
if _ws_is_unrestricted(ws, config):
return full_message
if not isinstance(parsed_payload, dict):
return None
allowed = _ws_allowed_cameras(ws, config)
filtered_jobs: dict[str, Any] = {}
for job_type, job_payload in parsed_payload.items():
scoped = _scope_job_entry_to_allowed(job_payload, allowed)
if scoped is not None:
filtered_jobs[job_type] = scoped
if not filtered_jobs:
return None
return _wrap_envelope(topic, filtered_jobs)
if kind == "reshape_stats":
if _ws_is_unrestricted(ws, config):
return full_message
if not isinstance(parsed_payload, dict):
return None
allowed = _ws_allowed_cameras(ws, config)
cameras_block = parsed_payload.get("cameras")
if isinstance(cameras_block, dict):
filtered_cameras = {
name: data for name, data in cameras_block.items() if name in allowed
}
reshaped = dict(parsed_payload)
reshaped["cameras"] = filtered_cameras
return _wrap_envelope(topic, reshaped)
return full_message
return None
class WebSocket(WebSocket_): # type: ignore[misc]
def unhandled_error(self, error: Any) -> None:
"""
@ -183,6 +501,10 @@ class WebSocketClient(Communicator):
self.websocket_thread.start()
def publish(self, topic: str, payload: Any, _: bool = False) -> None:
if self.websocket_server is None:
logger.debug("Skipping message, websocket not connected yet")
return
try:
ws_message = json.dumps(
{
@ -195,13 +517,41 @@ class WebSocketClient(Communicator):
logger.debug(f"payload for {topic} wasn't text. Skipping...")
return
if self.websocket_server is None:
logger.debug("Skipping message, websocket not connected yet")
all_cameras = set(self.config.cameras.keys())
all_zones = _collect_zone_names(self.config)
scope = _classify_outbound(topic, all_cameras, all_zones)
if scope[0] == "drop":
return
# Pre-parse payload once for topics that need to read its contents.
parsed_payload: Any = None
if scope[0] in (
"payload_camera",
"reshape_by_camera_key",
"reshape_job_state",
"reshape_stats",
):
parsed_payload = _parse_json_payload(payload)
if parsed_payload is None:
# malformed payload — fail closed
return
manager = self.websocket_server.manager
with manager.lock:
websockets = list(manager.websockets.values())
for ws in websockets:
if getattr(ws, "terminated", False):
continue
message = _materialize_for_ws(
ws, topic, ws_message, scope, parsed_payload, self.config
)
if message is None:
continue
try:
self.websocket_server.manager.broadcast(ws_message)
except ConnectionResetError:
ws.send(message)
except (ConnectionResetError, BrokenPipeError, ValueError):
pass
def stop(self) -> None:

2
frigate/config/camera/camera.py
View File

@ -146,7 +146,7 @@ class CameraConfig(FrigateBaseModel):
timestamp_style: TimestampStyleConfig = Field(
default_factory=TimestampStyleConfig,
title="Timestamp style",
description="Styling options for in-feed timestamps applied to recordings and snapshots.",
description="Styling options for timestamps applied to snapshots and Debug view.",
)
# Options without global fallback

18
frigate/config/camera/ffmpeg.py
View File

@ -3,7 +3,7 @@ from typing import Union
from pydantic import Field, field_validator
from frigate.const import DEFAULT_FFMPEG_VERSION, INCLUDED_FFMPEG_VERSIONS
from frigate.util.config import resolve_ffmpeg_path
from ..base import FrigateBaseModel
from ..env import EnvString
@ -49,7 +49,7 @@ class FfmpegConfig(FrigateBaseModel):
path: str = Field(
default="default",
title="FFmpeg path",
description='Path to the FFmpeg binary to use or a version alias ("5.0" or "7.0").',
description='Path to the FFmpeg binary to use or a version alias ("5.0" or "8.0").',
)
global_args: Union[str, list[str]] = Field(
default=FFMPEG_GLOBAL_ARGS_DEFAULT,
@ -90,21 +90,11 @@ class FfmpegConfig(FrigateBaseModel):
@property
def ffmpeg_path(self) -> str:
if self.path == "default":
return f"/usr/lib/ffmpeg/{DEFAULT_FFMPEG_VERSION}/bin/ffmpeg"
elif self.path in INCLUDED_FFMPEG_VERSIONS:
return f"/usr/lib/ffmpeg/{self.path}/bin/ffmpeg"
else:
return f"{self.path}/bin/ffmpeg"
return resolve_ffmpeg_path(self.path, "ffmpeg")
@property
def ffprobe_path(self) -> str:
if self.path == "default":
return f"/usr/lib/ffmpeg/{DEFAULT_FFMPEG_VERSION}/bin/ffprobe"
elif self.path in INCLUDED_FFMPEG_VERSIONS:
return f"/usr/lib/ffmpeg/{self.path}/bin/ffprobe"
else:
return f"{self.path}/bin/ffprobe"
return resolve_ffmpeg_path(self.path, "ffprobe")
class CameraRoleEnum(str, Enum):

2
frigate/config/camera/genai.py
View File

@ -37,7 +37,7 @@ class GenAIConfig(FrigateBaseModel):
description="Base URL for self-hosted or compatible providers (for example an Ollama instance).",
)
model: str = Field(
default="gpt-4o",
default="",
title="Model",
description="The model to use from the provider for generating descriptions or summaries.",
)

5
frigate/config/camera/ui.py
View File

@ -16,3 +16,8 @@ class CameraUiConfig(FrigateBaseModel):
title="Show in UI",
description="Toggle whether this camera is visible everywhere in the Frigate UI. Disabling this will require manually editing the config to view this camera in the UI again.",
)
review: bool = Field(
default=True,
title="Show in review",
description="Toggle whether this camera is visible in review (the review page and its camera filter, motion review, and the history view).",
)

5
frigate/config/camera/updater.py
View File

@ -26,6 +26,7 @@ class CameraConfigUpdateEnum(str, Enum):
object_genai = "object_genai"
onvif = "onvif"
record = "record"
refresh = "refresh" # signals the camera maintainer to recycle the camera process
remove = "remove" # for removing a camera
review = "review"
review_genai = "review_genai"
@ -84,8 +85,8 @@ class CameraConfigUpdateSubscriber:
self, camera: str, update_type: CameraConfigUpdateEnum, updated_config: Any
) -> None:
if update_type == CameraConfigUpdateEnum.add:
self.config.cameras[camera] = updated_config
self.camera_configs[camera] = updated_config
shared = self.config.cameras.setdefault(camera, updated_config)
self.camera_configs[camera] = shared
return
elif update_type == CameraConfigUpdateEnum.remove:
self.config.cameras.pop(camera, None)

71
frigate/config/config.py
View File

@ -80,12 +80,12 @@ logger = logging.getLogger(__name__)
yaml = YAML()
DEFAULT_DETECTORS = {
"ov": {
"type": "openvino",
"device": "CPU",
}
}
# Pydantic field default applied when an existing config omits `detectors:`.
# Kept as cpu tflite for backwards compatibility with 0.17 configs.
DEFAULT_DETECTORS = {"cpu": {"type": "cpu"}}
# Used by the openvino branch below and rendered into the new-config YAML
# template so first-time setups default to openvino on CPU.
DEFAULT_MODEL = {
"width": 300,
"height": 300,
@ -94,6 +94,7 @@ DEFAULT_MODEL = {
"path": "/openvino-model/ssdlite_mobilenet_v2.xml",
"labelmap_path": "/openvino-model/coco_91cl_bkgr.txt",
}
NEW_CONFIG_DETECTORS = {"ov": {"type": "openvino", "device": "CPU"}}
DEFAULT_DETECT_DIMENSIONS = {"width": 1280, "height": 720}
@ -109,7 +110,7 @@ DEFAULT_CONFIG = f"""
mqtt:
enabled: False
{_render_default_yaml({"detectors": DEFAULT_DETECTORS, "model": DEFAULT_MODEL})}
{_render_default_yaml({"detectors": NEW_CONFIG_DETECTORS, "model": DEFAULT_MODEL})}
cameras: {{}} # No cameras defined, UI wizard should be used
version: {CURRENT_CONFIG_VERSION}
"""
@ -325,6 +326,47 @@ def verify_required_zones_exist(camera_config: CameraConfig) -> None:
)
def verify_profile_overrides_match_base(camera_config: CameraConfig) -> None:
"""Verify that profile zone and mask IDs reference entries defined on the base camera."""
for profile_name, profile in camera_config.profiles.items():
if profile.zones:
for zone_name in profile.zones:
if zone_name not in camera_config.zones:
raise ValueError(
f"Camera '{camera_config.name}' profile '{profile_name}' defines "
f"zone '{zone_name}' that does not exist on the base config"
)
if profile.motion and profile.motion.mask:
for mask_name in profile.motion.mask:
if mask_name not in camera_config.motion.mask:
raise ValueError(
f"Camera '{camera_config.name}' profile '{profile_name}' defines "
f"motion mask '{mask_name}' that does not exist on the base config"
)
if profile.objects:
for mask_name in profile.objects.mask or {}:
if mask_name not in (camera_config.objects.mask or {}):
raise ValueError(
f"Camera '{camera_config.name}' profile '{profile_name}' defines "
f"object mask '{mask_name}' that does not exist on the base config"
)
for label, filter_config in (profile.objects.filters or {}).items():
base_filter = (camera_config.objects.filters or {}).get(label)
profile_filter_masks = (
filter_config.mask if filter_config else None
) or {}
base_filter_masks = (base_filter.mask if base_filter else None) or {}
for mask_name in profile_filter_masks:
if mask_name not in base_filter_masks:
raise ValueError(
f"Camera '{camera_config.name}' profile '{profile_name}' defines "
f"object mask '{mask_name}' for '{label}' that does not exist "
f"on the base config"
)
def verify_autotrack_zones(camera_config: CameraConfig) -> ValueError | None:
"""Verify that required_zones are specified when autotracking is enabled."""
if (
@ -628,15 +670,23 @@ class FrigateConfig(FrigateBaseModel):
# set default min_score for object attributes
for attribute in self.model.all_attributes:
if not self.objects.filters.get(attribute):
existing = self.objects.filters.get(attribute)
if existing is None:
self.objects.filters[attribute] = FilterConfig(min_score=0.7)
elif self.objects.filters[attribute].min_score == 0.5:
self.objects.filters[attribute].min_score = 0.7
elif "min_score" not in existing.model_fields_set:
existing.min_score = 0.7
# auto detect hwaccel args
if self.ffmpeg.hwaccel_args == "auto":
self.ffmpeg.hwaccel_args = auto_detect_hwaccel()
# Resolve global export hwaccel_args so it matches the per-camera
# resolution below. Without this, every camera reads as overriding
# record.export.hwaccel_args because the global stays "auto" while
# the camera value gets resolved to the actual args list.
if self.record.export.hwaccel_args == "auto":
self.record.export.hwaccel_args = self.ffmpeg.hwaccel_args
# Populate global audio filters from listen. Existing user-defined
# entries for labels not in listen are preserved but unused at runtime.
if self.audio.filters is None:
@ -950,6 +1000,7 @@ class FrigateConfig(FrigateBaseModel):
verify_recording_segments_setup_with_reasonable_time(camera_config)
verify_zone_objects_are_tracked(camera_config)
verify_required_zones_exist(camera_config)
verify_profile_overrides_match_base(camera_config)
verify_autotrack_zones(camera_config)
verify_motion_and_detect(camera_config)
verify_objects_track(camera_config, labelmap_objects)

70
frigate/config/profile_manager.py
View File

@ -5,7 +5,7 @@ import json
import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
from typing import Any, Callable, Optional
from frigate.config.camera.updater import (
CameraConfigUpdateEnum,
@ -34,6 +34,45 @@ PROFILE_SECTION_UPDATES: dict[str, CameraConfigUpdateEnum] = {
"zones": CameraConfigUpdateEnum.zones,
}
# Retained MQTT switch topics per profile section, with a payload getter.
# Republished on profile change so MQTT/HA don't show a stale toggle.
SECTION_STATE_TOPICS: dict[str, list[tuple[str, Callable[[Any], Any]]]] = {
"audio": [("audio", lambda c: "ON" if c.audio.enabled else "OFF")],
"birdseye": [
("birdseye", lambda c: "ON" if c.birdseye.enabled else "OFF"),
(
"birdseye_mode",
lambda c: c.birdseye.mode.value.upper() if c.birdseye.enabled else "OFF",
),
],
"detect": [("detect", lambda c: "ON" if c.detect.enabled else "OFF")],
"motion": [
("motion", lambda c: "ON" if c.motion.enabled else "OFF"),
("improve_contrast", lambda c: "ON" if c.motion.improve_contrast else "OFF"),
("motion_threshold", lambda c: c.motion.threshold),
("motion_contour_area", lambda c: c.motion.contour_area),
],
"notifications": [
("notifications", lambda c: "ON" if c.notifications.enabled else "OFF"),
],
"objects": [
("object_descriptions", lambda c: "ON" if c.objects.genai.enabled else "OFF"),
],
"record": [("recordings", lambda c: "ON" if c.record.enabled else "OFF")],
"review": [
("review_alerts", lambda c: "ON" if c.review.alerts.enabled else "OFF"),
(
"review_detections",
lambda c: "ON" if c.review.detections.enabled else "OFF",
),
(
"review_descriptions",
lambda c: "ON" if c.review.genai.enabled else "OFF",
),
],
"snapshots": [("snapshots", lambda c: "ON" if c.snapshots.enabled else "OFF")],
}
PERSISTENCE_FILE = Path(CONFIG_DIR) / ".profiles"
@ -124,11 +163,24 @@ class ProfileManager:
self.config.active_profile = None
self._persist_active_profile(None)
def activate_profile(self, profile_name: Optional[str]) -> Optional[str]:
# drop all runtime overrides so they don't replay stale values on restart
if self.dispatcher is not None:
self.dispatcher.clear_runtime_state()
def activate_profile(
self,
profile_name: Optional[str],
clear_runtime_overrides: bool = True,
) -> Optional[str]:
"""Activate a profile by name, or deactivate if None.
Args:
profile_name: Profile name to activate, or None to deactivate.
clear_runtime_overrides: When True (the default, for user-initiated
activations) drop the dispatcher's runtime override file because
the layer below changed. Startup callers that are replaying a
persisted profile pass False so the runtime state stays
available for the subsequent replay step.
Returns:
None on success, or an error message string on failure.
@ -156,6 +208,11 @@ class ProfileManager:
self.config.active_profile = profile_name
self._persist_active_profile(profile_name)
# a profile switch invalidates the steady-state runtime overrides
if clear_runtime_overrides and self.dispatcher is not None:
self.dispatcher.clear_runtime_state()
logger.info(
"Profile %s",
f"'{profile_name}' activated" if profile_name else "deactivated",
@ -292,6 +349,15 @@ class ProfileManager:
settings,
)
# republish MQTT switch states
if self.dispatcher is not None:
for suffix, get_payload in SECTION_STATE_TOPICS.get(section, ()):
self.dispatcher.publish(
f"{cam_name}/{suffix}/state",
get_payload(cam_config),
retain=True,
)
def _persist_active_profile(self, profile_name: Optional[str]) -> None:
"""Persist the active profile state to disk as JSON."""
try:

2
frigate/config/proxy.py
View File

@ -45,7 +45,7 @@ class ProxyConfig(FrigateBaseModel):
default_role: Optional[str] = Field(
default="viewer",
title="Default role",
description="Default role assigned to proxy-authenticated users when no role mapping applies (admin or viewer).",
description="Default role assigned to proxy-authenticated users when no role mapping applies.",
)
separator: Optional[str] = Field(
default=",",

19
frigate/config/ui.py
View File

@ -5,7 +5,7 @@ from pydantic import Field
from .base import FrigateBaseModel
__all__ = ["TimeFormatEnum", "DateTimeStyleEnum", "UnitSystemEnum", "UIConfig"]
__all__ = ["TimeFormatEnum", "UnitSystemEnum", "UIConfig"]
class TimeFormatEnum(str, Enum):
@ -14,13 +14,6 @@ class TimeFormatEnum(str, Enum):
hours24 = "24hour"
class DateTimeStyleEnum(str, Enum):
full = "full"
long = "long"
medium = "medium"
short = "short"
class UnitSystemEnum(str, Enum):
imperial = "imperial"
metric = "metric"
@ -37,16 +30,6 @@ class UIConfig(FrigateBaseModel):
title="Time format",
description="Time format to use in the UI (browser, 12hour, or 24hour).",
)
date_style: DateTimeStyleEnum = Field(
default=DateTimeStyleEnum.short,
title="Date style",
description="Date style to use in the UI (full, long, medium, short).",
)
time_style: DateTimeStyleEnum = Field(
default=DateTimeStyleEnum.medium,
title="Time style",
description="Time style to use in the UI (full, long, medium, short).",
)
unit_system: UnitSystemEnum = Field(
default=UnitSystemEnum.metric,
title="Unit system",

2
frigate/const.py
View File

@ -21,6 +21,8 @@ PLUS_API_HOST = "https://api.frigate.video"
SHM_FRAMES_VAR = "SHM_MAX_FRAMES"
REDACTED_CREDENTIAL_SENTINEL = "__FRIGATE_SAVED_CREDENTIAL__"
# Attribute & Object constants
DEFAULT_ATTRIBUTE_LABEL_MAP = {

96
frigate/debug_replay.py
View File

@ -5,10 +5,12 @@ frigate.jobs.debug_replay. This module owns only session presence
(active), session metadata, and post-session cleanup.
"""
import asyncio
import logging
import os
import shutil
import threading
import time
from ruamel.yaml import YAML
@ -25,12 +27,23 @@ from frigate.const import (
REPLAY_DIR,
THUMB_DIR,
)
from frigate.jobs.debug_replay import cancel_debug_replay_job, wait_for_runner
from frigate.jobs.debug_replay import (
JOB_TYPE as DEBUG_REPLAY_JOB_TYPE,
)
from frigate.jobs.debug_replay import (
cancel_debug_replay_job,
wait_for_runner,
)
from frigate.jobs.export import JobStatePublisher
from frigate.types import JobStatusTypesEnum
from frigate.util.camera_cleanup import cleanup_camera_db, cleanup_camera_files
from frigate.util.config import find_config_file
logger = logging.getLogger(__name__)
MAX_SESSION_DURATION_SECONDS = 12 * 60 * 60
AUTO_STOP_CHECK_INTERVAL_SECONDS = 60
class DebugReplayManager:
"""Owns the lifecycle pointers for a single debug replay session.
@ -49,6 +62,8 @@ class DebugReplayManager:
self.clip_path: str | None = None
self.start_ts: float | None = None
self.end_ts: float | None = None
self.session_started_at: float | None = None
self._job_state_publisher = JobStatePublisher()
@property
def active(self) -> bool:
@ -73,6 +88,7 @@ class DebugReplayManager:
self.start_ts = start_ts
self.end_ts = end_ts
self.clip_path = None
self.session_started_at = time.time()
def mark_session_ready(self, clip_path: str) -> None:
"""Record the on-disk clip path after the camera has been published."""
@ -94,6 +110,7 @@ class DebugReplayManager:
self.clip_path = None
self.start_ts = None
self.end_ts = None
self.session_started_at = None
def publish_camera(
self,
@ -150,6 +167,7 @@ class DebugReplayManager:
return
replay_name = self.replay_camera_name
source_camera = self.source_camera
# Only publish remove if the camera was actually added to the live
# config (i.e. the runner reached the starting_camera phase).
@ -158,11 +176,27 @@ class DebugReplayManager:
CameraConfigUpdateTopic(CameraConfigUpdateEnum.remove, replay_name),
frigate_config.cameras[replay_name],
)
frigate_config.cameras.pop(replay_name, None)
if replay_name is not None:
self._cleanup_db(replay_name)
self._cleanup_files(replay_name)
self._job_state_publisher.publish(
{
"id": "stopped",
"job_type": DEBUG_REPLAY_JOB_TYPE,
"status": JobStatusTypesEnum.cancelled,
"start_time": None,
"end_time": time.time(),
"error_message": None,
"results": {
"source_camera": source_camera,
"replay_camera_name": replay_name,
},
}
)
self._clear_locked()
logger.info("Debug replay stopped and cleaned up: %s", replay_name)
@ -211,6 +245,10 @@ class DebugReplayManager:
zone_dump.setdefault("coordinates", zone_config.coordinates)
zones_dict[zone_name] = zone_dump
# Extract LPR and face recognition configs
lpr_dict = source_config.lpr.model_dump()
face_recognition_dict = source_config.face_recognition.model_dump()
# Extract motion config (exclude runtime fields)
motion_dict = {}
if source_config.motion is not None:
@ -219,11 +257,23 @@ class DebugReplayManager:
"frame_shape",
"raw_mask",
"mask",
"improved_contrast_enabled",
"enabled_in_config",
"rasterized_mask",
}
)
if source_config.motion.mask:
motion_dict["mask"] = {
mask_id: (
mask_cfg.model_dump(
exclude={"raw_coordinates", "enabled_in_config"}
)
if mask_cfg is not None
else None
)
for mask_id, mask_cfg in source_config.motion.mask.items()
}
return {
"enabled": True,
"ffmpeg": {
@ -248,8 +298,8 @@ class DebugReplayManager:
},
"birdseye": {"enabled": False},
"audio": {"enabled": False},
"lpr": {"enabled": False},
"face_recognition": {"enabled": False},
"lpr": lpr_dict,
"face_recognition": face_recognition_dict,
}
def _cleanup_db(self, camera_name: str) -> None:
@ -308,3 +358,41 @@ def cleanup_replay_cameras() -> None:
shutil.rmtree(REPLAY_DIR)
except Exception as e:
logger.error("Failed to remove replay cache directory: %s", e)
async def debug_replay_auto_stop_watchdog(
manager: DebugReplayManager,
frigate_config: FrigateConfig,
config_publisher: CameraConfigUpdatePublisher,
) -> None:
"""Auto-stop debug replay sessions that exceed MAX_SESSION_DURATION_SECONDS.
Backstop against a session left running for days. The cap is intentionally
generous so realistic tuning and overnight soak workflows aren't disrupted.
"""
while True:
try:
await asyncio.sleep(AUTO_STOP_CHECK_INTERVAL_SECONDS)
started_at = manager.session_started_at
if not manager.active or started_at is None:
continue
if time.time() - started_at < MAX_SESSION_DURATION_SECONDS:
continue
replay_name = manager.replay_camera_name
await asyncio.to_thread(
manager.stop,
frigate_config=frigate_config,
config_publisher=config_publisher,
)
logger.info(
"Debug replay auto-stopped after exceeding max session duration of %d hours: %s",
MAX_SESSION_DURATION_SECONDS // 3600,
replay_name,
)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("Error in debug replay auto-stop watchdog")

34
frigate/detectors/detection_runners.py
View File

@ -15,6 +15,9 @@ from frigate.util.rknn_converter import auto_convert_model, is_rknn_compatible
logger = logging.getLogger(__name__)
# Process-wide lock serializing all OpenVINO compile/inference calls
_OPENVINO_LOCK = threading.Lock()
def is_arm64_platform() -> bool:
"""Check if we're running on an ARM platform."""
@ -282,6 +285,13 @@ class OpenVINOModelRunner(BaseModelRunner):
EnrichmentModelTypeEnum.arcface.value,
]
@staticmethod
def is_detection_model(model_type: str) -> bool:
# Import here to avoid circular imports
from frigate.detectors.detector_config import ModelTypeEnum
return model_type in [m.value for m in ModelTypeEnum]
def __init__(self, model_path: str, device: str, model_type: str, **kwargs):
self.model_path = model_path
self.device = device
@ -310,21 +320,25 @@ class OpenVINOModelRunner(BaseModelRunner):
# Apply performance optimization
self.ov_core.set_property(device, {"PERF_COUNT": "NO"})
if device in ["GPU", "AUTO"]:
if device in ["GPU", "AUTO", "NPU"]:
self.ov_core.set_property(device, {"PERFORMANCE_HINT": "LATENCY"})
# Compile model
if device == "NPU" and OpenVINOModelRunner.is_detection_model(model_type):
try:
self.ov_core.set_property(device, {"NPU_TURBO": "YES"})
except Exception as e:
logger.debug(f"NPU_TURBO not supported by driver: {e}")
# Compile model under the shared lock
with _OPENVINO_LOCK:
self.compiled_model = self.ov_core.compile_model(
model=model_path, device_name=device
)
# Create reusable inference request
self.infer_request = self.compiled_model.create_infer_request()
self.input_tensor: ov.Tensor | None = None
# Thread lock to prevent concurrent inference (needed for JinaV2 which shares
# one runner between text and vision embeddings called from different threads)
self._inference_lock = threading.Lock()
self.input_tensor: ov.Tensor | None = None
if not self.complex_model:
try:
@ -369,9 +383,11 @@ class OpenVINOModelRunner(BaseModelRunner):
Returns:
List of output tensors
"""
# Lock prevents concurrent access to infer_request
# Needed for JinaV2: genai thread (text) + embeddings thread (vision)
with self._inference_lock:
# Shared lock serializes inference across every OpenVINO runner in this
# process — both the shared-runner JinaV2 case (genai text thread +
# embeddings vision thread) and distinct runners running on separate
# threads (e.g. the ArcFace face-model build vs the LPR detector).
with _OPENVINO_LOCK:
from frigate.embeddings.types import EnrichmentModelTypeEnum
if self.model_type in [EnrichmentModelTypeEnum.arcface.value]:

9
frigate/embeddings/maintainer.py
View File

@ -98,10 +98,17 @@ class EmbeddingMaintainer(threading.Thread):
[
CameraConfigUpdateEnum.add,
CameraConfigUpdateEnum.remove,
CameraConfigUpdateEnum.detect,
CameraConfigUpdateEnum.face_recognition,
CameraConfigUpdateEnum.ffmpeg,
CameraConfigUpdateEnum.lpr,
CameraConfigUpdateEnum.motion,
CameraConfigUpdateEnum.objects,
CameraConfigUpdateEnum.object_genai,
CameraConfigUpdateEnum.review,
CameraConfigUpdateEnum.review_genai,
CameraConfigUpdateEnum.semantic_search,
CameraConfigUpdateEnum.zones,
],
)
self.enrichment_config_subscriber = ConfigSubscriber("config/")
@ -232,7 +239,7 @@ class EmbeddingMaintainer(threading.Thread):
)
)
if self.config.audio_transcription.enabled and any(
if any(
c.enabled_in_config and c.audio_transcription.enabled
for c in self.config.cameras.values()
):

59
frigate/events/audio.py
View File

@ -94,13 +94,28 @@ class AudioProcessor(FrigateProcess):
self.camera_metrics = camera_metrics
self.config = config
def __stop_audio_thread(self, camera: str) -> None:
thread = self.audio_threads.pop(camera, None)
if thread is None:
return
thread.stop()
thread.join(10)
if thread.is_alive():
self.logger.warning(f"Audio maintainer thread for {camera} is still alive")
else:
self.logger.info(f"Audio maintainer stopped for {camera}")
def run(self) -> None:
self.pre_run_setup(self.config.logger)
audio_threads: dict[str, AudioEventMaintainer] = {}
self.audio_threads: dict[str, AudioEventMaintainer] = {}
threading.current_thread().name = "process:audio_manager"
if self.config.audio_transcription.enabled:
if any(
c.enabled_in_config and c.audio_transcription.enabled
for c in self.config.cameras.values()
):
self.transcription_model_runner: AudioTranscriptionModelRunner | None = (
AudioTranscriptionModelRunner(
self.config.audio_transcription.device or "AUTO",
@ -117,12 +132,13 @@ class AudioProcessor(FrigateProcess):
CameraConfigUpdateEnum.add,
CameraConfigUpdateEnum.audio,
CameraConfigUpdateEnum.ffmpeg,
CameraConfigUpdateEnum.remove,
],
)
def spawn_if_needed(camera: CameraConfig) -> None:
name = camera.name
if name is None or name in audio_threads:
if name is None or name in self.audio_threads:
return
if not camera.enabled or not camera.audio.enabled:
return
@ -136,7 +152,7 @@ class AudioProcessor(FrigateProcess):
self.transcription_model_runner,
self.stop_event, # type: ignore[arg-type]
)
audio_threads[name] = thread
self.audio_threads[name] = thread
thread.start()
self.logger.info(f"Audio maintainer started for {name}")
@ -145,21 +161,31 @@ class AudioProcessor(FrigateProcess):
self.logger.info(f"Audio processor started (pid: {self.pid})")
# poll for newly added cameras or cameras flipped to audio.enabled at runtime
# poll for newly added/removed cameras or cameras flipped to
# audio.enabled at runtime
while not self.stop_event.wait(timeout=1.0):
config_subscriber.check_for_updates()
updated_topics = config_subscriber.check_for_updates()
# stop maintainers for removed cameras so their ffmpeg process is
# torn down and they stop touching camera_metrics (which the camera
# maintainer has already popped for the removed camera)
for removed_camera in updated_topics.get(
CameraConfigUpdateEnum.remove.name, []
):
self.__stop_audio_thread(removed_camera)
for camera in self.config.cameras.values():
spawn_if_needed(camera)
config_subscriber.stop()
for thread in audio_threads.values():
for thread in self.audio_threads.values():
thread.join(1)
if thread.is_alive():
self.logger.info(f"Waiting for thread {thread.name:s} to exit")
thread.join(10)
for thread in audio_threads.values():
for thread in self.audio_threads.values():
if thread.is_alive():
self.logger.warning(f"Thread {thread.name} is still alive")
@ -181,6 +207,9 @@ class AudioEventMaintainer(threading.Thread):
self.camera_config = camera
self.camera_metrics = camera_metrics
self.stop_event = stop_event
# per-camera stop signal so a single maintainer can be torn down at
# runtime (e.g. on camera removal) without stopping the whole process
self.camera_stop_event = threading.Event()
self.detector = AudioTfl(stop_event, self.camera_config.audio.num_threads)
self.shape = (int(round(AUDIO_DURATION * AUDIO_SAMPLE_RATE)),)
self.chunk_size = int(round(AUDIO_DURATION * AUDIO_SAMPLE_RATE * 2))
@ -206,7 +235,7 @@ class AudioEventMaintainer(threading.Thread):
self.detection_publisher = DetectionPublisher(DetectionTypeEnum.audio.value)
if (
self.config.audio_transcription.enabled
self.camera_config.audio_transcription.enabled
and self.audio_transcription_model_runner is not None
):
# init the transcription processor for this camera
@ -230,7 +259,11 @@ class AudioEventMaintainer(threading.Thread):
self.was_audio_enabled = camera.audio.enabled
def detect_audio(self, audio: np.ndarray) -> None:
if not self.camera_config.audio.enabled or self.stop_event.is_set():
if (
not self.camera_config.audio.enabled
or self.stop_event.is_set()
or self.camera_stop_event.is_set()
):
return
audio_as_float: np.ndarray = audio.astype(np.float32)
@ -349,11 +382,15 @@ class AudioEventMaintainer(threading.Thread):
self.logger.error(f"Error reading audio data from ffmpeg process: {e}")
log_and_restart()
def stop(self) -> None:
"""Signal this maintainer to exit its run loop and clean up."""
self.camera_stop_event.set()
def run(self) -> None:
if self.camera_config.enabled:
self.start_or_restart_ffmpeg()
while not self.stop_event.is_set():
while not self.stop_event.is_set() and not self.camera_stop_event.is_set():
# check if there is an updated config
self.config_subscriber.check_for_updates()

14
frigate/ffmpeg_presets.py
View File

@ -465,16 +465,6 @@ PRESETS_RECORD_OUTPUT = {
"-c:a",
"aac",
],
# NOTE: This preset originally used "-c:a copy" to pass through audio
# without re-encoding. FFmpeg 7.x introduced a threaded pipeline where
# demuxing, encoding, and muxing run in parallel via a Scheduler. This
# broke audio streamcopy from RTSP sources: packets are demuxed correctly
# but silently dropped before reaching the muxer (0 bytes written). The
# issue is specific to RTSP + streamcopy; file inputs and transcoding both
# work. Transcoding AAC audio is very lightweight (~30KiB per 10s segment)
# and adds negligible CPU overhead, so this is an acceptable workaround.
# The benefits of FFmpeg 7.x — particularly the removal of gamma correction
# hacks required by earlier versions — outweigh this trade-off.
"preset-record-generic-audio-copy": [
"-f",
"segment",
@ -486,10 +476,8 @@ PRESETS_RECORD_OUTPUT = {
"1",
"-strftime",
"1",
"-c:v",
"-c",
"copy",
"-c:a",
"aac",
],
"preset-record-mjpeg": [
"-f",

249
frigate/genai/__init__.py
View File

@ -1,21 +1,25 @@
"""Generative AI module for Frigate."""
import datetime
import importlib
import json
import logging
import os
import re
from typing import Any, Callable, Optional
from typing import Any, AsyncGenerator, Callable, Optional
import numpy as np
from playhouse.shortcuts import model_to_dict
from pydantic import ValidationError
from frigate.config import CameraConfig, GenAIConfig, GenAIProviderEnum
from frigate.const import CLIPS_DIR
from frigate.data_processing.post.types import ReviewMetadata
from frigate.genai.manager import GenAIClientManager
from frigate.genai.prompts import (
build_object_description_prompt,
build_review_description_prompt,
build_review_description_response_format,
build_review_summary_prompt,
)
from frigate.models import Event
logger = logging.getLogger(__name__)
@ -46,9 +50,15 @@ def register_genai_provider(key: GenAIProviderEnum) -> Callable:
class GenAIClient:
"""Generative AI client for Frigate."""
def __init__(self, genai_config: GenAIConfig, timeout: int = 120) -> None:
def __init__(
self,
genai_config: GenAIConfig,
timeout: int = 120,
validate_model: bool = True,
) -> None:
self.genai_config: GenAIConfig = genai_config
self.timeout = timeout
self.validate_model = validate_model
self.provider = self._init_provider()
def generate_review_description(
@ -61,75 +71,14 @@ class GenAIClient:
activity_context_prompt: str,
) -> ReviewMetadata | None:
"""Generate a description for the review item activity."""
context_prompt = build_review_description_prompt(
review_data,
thumbnails,
concerns,
preferred_language,
activity_context_prompt,
)
def get_concern_prompt() -> str:
if concerns:
concern_list = "\n - ".join(concerns)
return f"""- `other_concerns` (list of strings): Include a list of any of the following concerns that are occurring:
- {concern_list}"""
else:
return ""
def get_language_prompt() -> str:
if preferred_language:
return f"Provide your answer in {preferred_language}"
else:
return ""
def get_objects_list() -> str:
if review_data["unified_objects"]:
return "\n- " + "\n- ".join(review_data["unified_objects"])
else:
return "\n- (No objects detected)"
context_prompt = f"""
Your task is to analyze a sequence of images taken in chronological order from a security camera.
## Normal Activity Patterns for This Property
{activity_context_prompt}
## Task Instructions
Describe the scene based on observable actions and movements, evaluate the activity against the Activity Indicators above, and assign a potential_threat_level (0, 1, or 2) by applying the threat level indicators consistently.
## Analysis Guidelines
When forming your description:
- **CRITICAL: Only describe objects explicitly listed in "Objects in Scene" below.** Do not infer or mention additional people, vehicles, or objects not present in this list, even if visual patterns suggest them. If only a car is listed, do not describe a person interacting with it unless "person" is also in the objects list.
- **Only describe actions actually visible in the frames.** Do not assume or infer actions that you don't observe happening. If someone walks toward furniture but you never see them sit, do not say they sat. Stick to what you can see across the sequence.
- Describe what you observe: actions, movements, interactions with objects and the environment. Include any observable environmental changes (e.g., lighting changes triggered by activity).
- Note visible details such as clothing, items being carried or placed, tools or equipment present, and how they interact with the property or objects.
- Consider the full sequence chronologically: what happens from start to finish, how duration and actions relate to the location and objects involved.
- **Use the actual timestamp provided in "Activity started at"** below for time of day context—do not infer time from image brightness or darkness. Unusual hours (late night/early morning) should increase suspicion when the observable behavior itself appears questionable. However, recognize that some legitimate activities can occur at any hour.
- **Consider duration as a primary factor**: Apply the duration thresholds defined in the activity patterns above. Brief sequences during normal hours with apparent purpose typically indicate normal activity unless explicit suspicious actions are visible.
- **Weigh all evidence holistically**: Match the activity against the normal and suspicious patterns defined above, then evaluate based on the complete context (zone, objects, time, actions, duration). Apply the threat level indicators consistently. Use your judgment for edge cases.
## Response Field Guidelines
Respond with a JSON object matching the provided schema. Field-specific guidance:
- `observations`: Include the very start of the activity — for example, a vehicle entering the frame or pulling into the driveway — even if it lasts only a few frames and the rest of the clip is dominated by a longer activity. Include each arrival, departure, object handled, and notable change in position or state. Each item is a single concrete fact written as a complete sentence.
- `scene`: Describe how the sequence begins, then the progression of events — all significant movements and actions in order. For example, if a vehicle arrives and then a person exits, describe both sequentially. For named subjects (those with a `←` separator in "Objects in Scene"), always use their name — do not replace them with generic terms. For unnamed objects (e.g., "person", "car"), refer to them naturally with articles (e.g., "a person", "the car"). Your description should align with and support the threat level you assign.
- `title`: Name the primary activity across the observations, together with the location. An activity is what is being done with objects, tools, or surfaces; locomotion through the scene qualifies as the activity only when no other interaction is observed. For named subjects, always use their name. For unnamed objects, refer to them naturally with articles.
- `shortSummary`: Briefly summarize the primary activity across the observations.
- `potential_threat_level`: Must be consistent with your scene description and the activity patterns above.
## Sequence Details
- Camera: {review_data["camera"]}
- Total frames: {len(thumbnails)} (Frame 1 = earliest, Frame {len(thumbnails)} = latest)
- Activity started at {review_data["start"]} and lasted {review_data["duration"]} seconds
- Zones involved: {", ".join(review_data["zones"]) if review_data["zones"] else "None"}
## Objects in Scene
Each line represents a detection state, not necessarily unique individuals. The `←` symbol separates a recognized subject's name from their object type — use only the name (before the `←`) in your response, not the type after it. The same subject may appear across multiple lines if detected multiple times.
**Note: Unidentified objects (without names) are NOT indicators of suspicious activity—they simply mean the system hasn't identified that object.**
{get_objects_list()}
{get_language_prompt()}
"""
logger.debug(
f"Sending {len(thumbnails)} images to create review description on {review_data['camera']}"
)
@ -143,25 +92,7 @@ Each line represents a detection state, not necessarily unique individuals. The
) as f:
f.write(context_prompt)
# Build JSON schema for structured output from ReviewMetadata model
schema = ReviewMetadata.model_json_schema()
schema.get("properties", {}).pop("time", None)
if "time" in schema.get("required", []):
schema["required"].remove("time")
if not concerns:
schema.get("properties", {}).pop("other_concerns", None)
if "other_concerns" in schema.get("required", []):
schema["required"].remove("other_concerns")
response_format = {
"type": "json_schema",
"json_schema": {
"name": "review_metadata",
"strict": True,
"schema": schema,
},
}
response_format = build_review_description_response_format(concerns)
response = self._send(context_prompt, thumbnails, response_format)
@ -240,61 +171,9 @@ Each line represents a detection state, not necessarily unique individuals. The
debug_save: bool,
) -> str | None:
"""Generate a summary of review item descriptions over a period of time."""
time_range = f"{datetime.datetime.fromtimestamp(start_ts).strftime('%B %d, %Y at %I:%M %p')} to {datetime.datetime.fromtimestamp(end_ts).strftime('%B %d, %Y at %I:%M %p')}"
timeline_summary_prompt = f"""
You are a security officer writing a concise security report.
Time range: {time_range}
Input format: Each event is a JSON object with:
- "title", "scene", "confidence", "potential_threat_level" (0-2), "other_concerns", "camera", "time", "start_time", "end_time"
- "context": array of related events from other cameras that occurred during overlapping time periods
**Note: Use the "scene" field for event descriptions in the report. Ignore any "shortSummary" field if present.**
Report Structure - Use this EXACT format:
# Security Summary - {time_range}
## Overview
[Write 1-2 sentences summarizing the overall activity pattern during this period.]
---
## Timeline
[Group events by time periods (e.g., "Morning (6:00 AM - 12:00 PM)", "Afternoon (12:00 PM - 5:00 PM)", "Evening (5:00 PM - 9:00 PM)", "Night (9:00 PM - 6:00 AM)"). Use appropriate time blocks based on when events occurred.]
### [Time Block Name]
**HH:MM AM/PM** | [Camera Name] | [Threat Level Indicator]
- [Event title]: [Clear description incorporating contextual information from the "context" array]
- Context: [If context array has items, mention them here, e.g., "Delivery truck present on Front Driveway Cam (HH:MM AM/PM)"]
- Assessment: [Brief assessment incorporating context - if context explains the event, note it here]
[Repeat for each event in chronological order within the time block]
---
## Summary
[One sentence summarizing the period. If all events are normal/explained: "Routine activity observed." If review needed: "Some activity requires review but no security concerns." If security concerns: "Security concerns requiring immediate attention."]
Guidelines:
- List ALL events in chronological order, grouped by time blocks
- Threat level indicators: ✓ Normal, ⚠️ Needs review, 🔴 Security concern
- Integrate contextual information naturally - use the "context" array to enrich each event's description
- If context explains the event (e.g., delivery truck explains person at door), describe it accordingly (e.g., "delivery person" not "unidentified person")
- Be concise but informative - focus on what happened and what it means
- If contextual information makes an event clearly normal, reflect that in your assessment
- Only create time blocks that have events - don't create empty sections
"""
timeline_summary_prompt += "\n\nEvents:\n"
for event in events:
timeline_summary_prompt += f"\n{event}\n"
if preferred_language:
timeline_summary_prompt += f"\nProvide your answer in {preferred_language}"
timeline_summary_prompt = build_review_summary_prompt(
start_ts, end_ts, events, preferred_language
)
if debug_save:
with open(
@ -326,10 +205,7 @@ Guidelines:
) -> Optional[str]:
"""Generate a description for the frame."""
try:
prompt = camera_config.objects.genai.object_prompts.get(
str(event.label),
camera_config.objects.genai.prompt,
).format(**model_to_dict(event))
prompt = build_object_description_prompt(camera_config, event)
except KeyError as e:
logger.error(f"Invalid key in GenAI prompt: {e}")
return None
@ -346,8 +222,15 @@ Guidelines:
prompt: str,
images: list[bytes],
response_format: Optional[dict] = None,
enable_thinking: bool = False,
) -> Optional[str]:
"""Submit a request to the provider."""
"""Submit a request to the provider.
``enable_thinking`` is honored only by providers that report
``supports_toggleable_thinking``. Description-style callers leave it
at the default (off) since synthesis tasks don't benefit from
reasoning traces.
"""
return None
@property
@ -359,6 +242,11 @@ Guidelines:
"""
return True
@property
def supports_toggleable_thinking(self) -> bool:
"""Whether the configured model exposes a per-request thinking toggle."""
return False
def list_models(self) -> list[str]:
"""Return the list of model names available from this provider.
@ -402,6 +290,7 @@ Guidelines:
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
"""
Send chat messages to LLM with optional tool definitions.
@ -425,11 +314,17 @@ Guidelines:
- 'none': Model must not call tools
- 'required': Model must call at least one tool
- Or a dict specifying a specific tool to call
**kwargs: Additional provider-specific parameters.
enable_thinking: Per-request thinking toggle. None means use the
provider default. Ignored by providers without a per-request
toggle (see `supports_toggleable_thinking`).
Returns:
Dictionary with:
- 'content': Optional[str] - The text response from the LLM, None if tool calls
- 'reasoning': Optional[str] - The separated reasoning/thinking trace
if the model emitted one (e.g. via OpenAI-compatible
`reasoning_content`). None when the model does not surface a
trace or the provider does not parse it.
- 'tool_calls': Optional[List[Dict]] - List of tool calls if LLM wants to call tools.
Each tool call dict has:
- 'id': str - Unique identifier for this tool call
@ -441,6 +336,14 @@ Guidelines:
- 'length': Hit token limit
- 'error': An error occurred
Streaming counterpart `chat_with_tools_stream` yields
``(kind, value)`` tuples where ``kind`` is one of:
- 'content_delta': value is a string fragment of the answer
- 'reasoning_delta': value is a string fragment of the reasoning
trace (emitted before content for thinking models)
- 'stats': value is a usage stats dict
- 'message': value is the final dict shape described above
Raises:
NotImplementedError: If the provider doesn't implement this method.
"""
@ -451,14 +354,50 @@ Guidelines:
)
return {
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
}
async def chat_with_tools_stream(
self,
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> AsyncGenerator[tuple[str, Any], None]:
"""Streaming counterpart to `chat_with_tools`.
Yields ``(kind, value)`` tuples where ``kind`` is one of:
- 'content_delta': value is a string fragment of the answer
- 'reasoning_delta': value is a string fragment of the reasoning
trace (emitted before content for thinking models)
- 'stats': value is a usage stats dict
- 'message': value is the final dict shape described in
`chat_with_tools`
Argument semantics — including ``enable_thinking`` — match
`chat_with_tools`. Providers that don't support streaming should
override this and yield an error 'message' event.
"""
logger.warning(
f"{self.__class__.__name__} does not support chat_with_tools_stream. "
"This method should be overridden by the provider implementation."
)
yield (
"message",
{
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
},
)
def load_providers() -> None:
package_dir = os.path.dirname(__file__)
for filename in os.listdir(package_dir):
plugins_dir = os.path.join(os.path.dirname(__file__), "plugins")
for filename in os.listdir(plugins_dir):
if filename.endswith(".py") and filename != "__init__.py":
module_name = f"frigate.genai.{filename[:-3]}"
module_name = f"frigate.genai.plugins.{filename[:-3]}"
importlib.import_module(module_name)

315
frigate/genai/azure-openai.py
View File

@ -1,315 +0,0 @@
"""Azure OpenAI Provider for Frigate AI."""
import base64
import json
import logging
from typing import Any, AsyncGenerator, Optional
from urllib.parse import parse_qs, urlparse
from openai import AzureOpenAI
from frigate.config import GenAIProviderEnum
from frigate.genai import GenAIClient, register_genai_provider
from frigate.genai.openai import _stats_from_openai_usage
logger = logging.getLogger(__name__)
@register_genai_provider(GenAIProviderEnum.azure_openai)
class OpenAIClient(GenAIClient):
"""Generative AI client for Frigate using Azure OpenAI."""
provider: AzureOpenAI
def _init_provider(self) -> AzureOpenAI | None:
"""Initialize the client."""
try:
parsed_url = urlparse(self.genai_config.base_url or "")
query_params = parse_qs(parsed_url.query)
api_version = query_params.get("api-version", [None])[0]
azure_endpoint = f"{parsed_url.scheme}://{parsed_url.netloc}/"
if not api_version:
logger.warning("Azure OpenAI url is missing API version.")
return None
except Exception as e:
logger.warning("Error parsing Azure OpenAI url: %s", str(e))
return None
return AzureOpenAI(
api_key=self.genai_config.api_key,
api_version=api_version,
azure_endpoint=azure_endpoint,
)
def _send(
self,
prompt: str,
images: list[bytes],
response_format: Optional[dict] = None,
) -> Optional[str]:
"""Submit a request to Azure OpenAI."""
encoded_images = [base64.b64encode(image).decode("utf-8") for image in images]
try:
request_params = {
"model": self.genai_config.model,
"messages": [
{
"role": "user",
"content": [{"type": "text", "text": prompt}]
+ [
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{image}",
"detail": "low",
},
}
for image in encoded_images
],
},
],
"timeout": self.timeout,
**self.genai_config.runtime_options,
}
if response_format:
request_params["response_format"] = response_format
result = self.provider.chat.completions.create(**request_params)
except Exception as e:
logger.warning("Azure OpenAI returned an error: %s", str(e))
return None
if len(result.choices) > 0:
return str(result.choices[0].message.content.strip())
return None
def list_models(self) -> list[str]:
"""Return available model IDs from Azure OpenAI."""
try:
return sorted(m.id for m in self.provider.models.list().data)
except Exception as e:
logger.warning("Failed to list Azure OpenAI models: %s", e)
return []
def get_context_size(self) -> int:
"""Get the context window size for Azure OpenAI."""
return 128000
def chat_with_tools(
self,
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
) -> dict[str, Any]:
try:
openai_tool_choice = None
if tool_choice:
if tool_choice == "none":
openai_tool_choice = "none"
elif tool_choice == "auto":
openai_tool_choice = "auto"
elif tool_choice == "required":
openai_tool_choice = "required"
request_params = {
"model": self.genai_config.model,
"messages": messages,
"timeout": self.timeout,
}
if tools:
request_params["tools"] = tools
if openai_tool_choice is not None:
request_params["tool_choice"] = openai_tool_choice
result = self.provider.chat.completions.create(**request_params) # type: ignore[call-overload]
if (
result is None
or not hasattr(result, "choices")
or len(result.choices) == 0
):
return {
"content": None,
"tool_calls": None,
"finish_reason": "error",
}
choice = result.choices[0]
message = choice.message
content = message.content.strip() if message.content else None
tool_calls = None
if message.tool_calls:
tool_calls = []
for tool_call in message.tool_calls:
try:
arguments = json.loads(tool_call.function.arguments)
except (json.JSONDecodeError, AttributeError) as e:
logger.warning(
f"Failed to parse tool call arguments: {e}, "
f"tool: {tool_call.function.name if hasattr(tool_call.function, 'name') else 'unknown'}"
)
arguments = {}
tool_calls.append(
{
"id": tool_call.id if hasattr(tool_call, "id") else "",
"name": tool_call.function.name
if hasattr(tool_call.function, "name")
else "",
"arguments": arguments,
}
)
finish_reason = "error"
if hasattr(choice, "finish_reason") and choice.finish_reason:
finish_reason = choice.finish_reason
elif tool_calls:
finish_reason = "tool_calls"
elif content:
finish_reason = "stop"
return {
"content": content,
"tool_calls": tool_calls,
"finish_reason": finish_reason,
}
except Exception as e:
logger.warning("Azure OpenAI returned an error: %s", str(e))
return {
"content": None,
"tool_calls": None,
"finish_reason": "error",
}
async def chat_with_tools_stream(
self,
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
) -> AsyncGenerator[tuple[str, Any], None]:
"""
Stream chat with tools; yields content deltas then final message.
Implements streaming function calling/tool usage for Azure OpenAI models.
"""
try:
openai_tool_choice = None
if tool_choice:
if tool_choice == "none":
openai_tool_choice = "none"
elif tool_choice == "auto":
openai_tool_choice = "auto"
elif tool_choice == "required":
openai_tool_choice = "required"
request_params = {
"model": self.genai_config.model,
"messages": messages,
"timeout": self.timeout,
"stream": True,
"stream_options": {"include_usage": True},
}
if tools:
request_params["tools"] = tools
if openai_tool_choice is not None:
request_params["tool_choice"] = openai_tool_choice
# Use streaming API
content_parts: list[str] = []
tool_calls_by_index: dict[int, dict[str, Any]] = {}
finish_reason = "stop"
usage_stats: Optional[dict[str, Any]] = None
stream = self.provider.chat.completions.create(**request_params) # type: ignore[call-overload]
for chunk in stream:
chunk_usage = getattr(chunk, "usage", None)
if chunk_usage is not None:
usage_stats = _stats_from_openai_usage(chunk_usage)
if not chunk or not chunk.choices:
continue
choice = chunk.choices[0]
delta = choice.delta
# Check for finish reason
if choice.finish_reason:
finish_reason = choice.finish_reason
# Extract content deltas
if delta.content:
content_parts.append(delta.content)
yield ("content_delta", delta.content)
# Extract tool calls
if delta.tool_calls:
for tc in delta.tool_calls:
idx = tc.index
fn = tc.function
if idx not in tool_calls_by_index:
tool_calls_by_index[idx] = {
"id": tc.id or "",
"name": fn.name if fn and fn.name else "",
"arguments": "",
}
t = tool_calls_by_index[idx]
if tc.id:
t["id"] = tc.id
if fn and fn.name:
t["name"] = fn.name
if fn and fn.arguments:
t["arguments"] += fn.arguments
# Build final message
full_content = "".join(content_parts).strip() or None
# Convert tool calls to list format
tool_calls_list = None
if tool_calls_by_index:
tool_calls_list = []
for tc in tool_calls_by_index.values():
try:
# Parse accumulated arguments as JSON
parsed_args = json.loads(tc["arguments"])
except (json.JSONDecodeError, Exception):
parsed_args = tc["arguments"]
tool_calls_list.append(
{
"id": tc["id"],
"name": tc["name"],
"arguments": parsed_args,
}
)
finish_reason = "tool_calls"
if usage_stats is not None:
yield ("stats", usage_stats)
yield (
"message",
{
"content": full_content,
"tool_calls": tool_calls_list,
"finish_reason": finish_reason,
},
)
except Exception as e:
logger.warning("Azure OpenAI streaming returned an error: %s", str(e))
yield (
"message",
{
"content": None,
"tool_calls": None,
"finish_reason": "error",
},
)

19
frigate/genai/manager.py
View File

@ -6,7 +6,7 @@ no chat feature is active) are never initialized.
"""
import logging
from typing import TYPE_CHECKING, Optional
from typing import TYPE_CHECKING, Any, Optional
from frigate.config import FrigateConfig
from frigate.config.camera.genai import GenAIConfig, GenAIRoleEnum
@ -108,11 +108,16 @@ class GenAIClientManager:
name = self._role_map.get(GenAIRoleEnum.embeddings)
return self._get_client(name) if name else None
def list_models(self) -> dict[str, list[str]]:
"""Return available models keyed by config entry name."""
result: dict[str, list[str]] = {}
for name in self._configs:
def list_models(self) -> dict[str, dict[str, Any]]:
"""Return per-entry model lists and capabilities, keyed by config entry name."""
result: dict[str, dict[str, Any]] = {}
for name, genai_cfg in self._configs.items():
client = self._get_client(name)
if client:
result[name] = client.list_models()
if not client:
continue
result[name] = {
"models": client.list_models(),
"roles": [r.value for r in genai_cfg.roles],
"supports_toggleable_thinking": client.supports_toggleable_thinking,
}
return result

1
frigate/genai/plugins/__init__.py Normal file
View File

@ -0,0 +1 @@
"""GenAI provider plugins."""

53
frigate/genai/plugins/azure-openai.py Normal file
View File

@ -0,0 +1,53 @@
"""Azure OpenAI Provider for Frigate AI.
Azure OpenAI exposes the same chat completions API as OpenAI once the
client is constructed, so this provider inherits all transport, streaming,
reasoning, and tool-calling logic from :class:`OpenAIClient` and only
overrides what is genuinely Azure-specific:
- Client construction: parses ``api-version`` out of the configured
``base_url`` query string and instantiates :class:`openai.AzureOpenAI`
with ``azure_endpoint`` instead of ``base_url``. Raises if the URL is
malformed; :class:`GenAIClientManager` catches the exception and
disables the provider.
- Context size: Azure does not expose a per-model ``max_model_len`` field
reliably, so we keep the historical 128K default rather than the
model-name heuristic used by OpenAI.
"""
import logging
from urllib.parse import parse_qs, urlparse
from openai import AzureOpenAI
from frigate.config import GenAIProviderEnum
from frigate.genai import register_genai_provider
from frigate.genai.plugins.openai import OpenAIClient
logger = logging.getLogger(__name__)
@register_genai_provider(GenAIProviderEnum.azure_openai)
class AzureOpenAIClient(OpenAIClient):
"""Generative AI client for Frigate using Azure OpenAI."""
def _init_provider(self) -> AzureOpenAI:
"""Initialize the AzureOpenAI client from the configured base_url."""
parsed_url = urlparse(self.genai_config.base_url or "")
query_params = parse_qs(parsed_url.query)
api_version = query_params.get("api-version", [None])[0]
if not api_version:
raise ValueError("Azure OpenAI base_url is missing api-version.")
azure_endpoint = f"{parsed_url.scheme}://{parsed_url.netloc}/"
return AzureOpenAI(
api_key=self.genai_config.api_key,
api_version=api_version,
azure_endpoint=azure_endpoint,
)
def get_context_size(self) -> int:
"""Azure does not reliably surface per-model context size; use 128K."""
return 128000

106
frigate/genai/gemini.py → frigate/genai/plugins/gemini.py
View File

@ -1,5 +1,7 @@
"""Gemini Provider for Frigate AI."""
import base64
import binascii
import json
import logging
from typing import Any, AsyncGenerator, Optional
@ -14,6 +16,27 @@ from frigate.genai import GenAIClient, register_genai_provider
logger = logging.getLogger(__name__)
def _decode_thought_signature(value: Any) -> Optional[bytes]:
"""Decode a base64-encoded thought_signature carried across conversation turns."""
if not value:
return None
if isinstance(value, bytes):
return value
if isinstance(value, str):
try:
return base64.b64decode(value)
except (binascii.Error, ValueError):
return None
return None
def _encode_thought_signature(signature: Optional[bytes]) -> Optional[str]:
"""Encode bytes thought_signature as base64 so it survives JSON-friendly transport."""
if not signature:
return None
return base64.b64encode(signature).decode("ascii")
def _stats_from_gemini_usage(usage: Any) -> Optional[dict[str, Any]]:
"""Build a stats dict from a Gemini usage_metadata object."""
prompt_tokens = getattr(usage, "prompt_token_count", None)
@ -62,6 +85,7 @@ class GeminiClient(GenAIClient):
prompt: str,
images: list[bytes],
response_format: Optional[dict] = None,
enable_thinking: bool = False,
) -> Optional[str]:
"""Submit a request to Gemini."""
contents = [prompt] + [
@ -119,11 +143,14 @@ class GeminiClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
"""
Send chat messages to Gemini with optional tool definitions.
Implements function calling/tool usage for Gemini models.
Implements function calling/tool usage for Gemini models. Thinking is
configured at the model level for Gemini, so ``enable_thinking`` is
accepted for interface parity and ignored.
"""
try:
# Convert messages to Gemini format
@ -165,11 +192,17 @@ class GeminiClient(GenAIClient):
if not isinstance(tc_args, dict):
tc_args = {}
if tc_name:
parts.append(
types.Part.from_function_call(
fc_part = types.Part.from_function_call(
name=tc_name, args=tc_args
)
)
# Thinking-capable Gemini models require the original
# thought_signature to be echoed back on functionCall
# parts after a tool response, or the next request
# fails with INVALID_ARGUMENT.
sig = _decode_thought_signature(tc.get("thought_signature"))
if sig:
fc_part.thought_signature = sig
parts.append(fc_part)
if not parts:
parts.append(types.Part.from_text(text=" "))
gemini_messages.append(types.Content(role="model", parts=parts))
@ -248,6 +281,13 @@ class GeminiClient(GenAIClient):
if tool_config:
config_params["tool_config"] = tool_config
# Ask thinking-capable models (Gemini 2.5+) to include their
# reasoning trace as separate `thought` parts so we can surface
# it on the reasoning channel. Older models ignore this field.
config_params["thinking_config"] = types.ThinkingConfig(
include_thoughts=True
)
# Merge runtime_options
if isinstance(self.genai_config.runtime_options, dict):
config_params.update(self.genai_config.runtime_options)
@ -262,18 +302,23 @@ class GeminiClient(GenAIClient):
if not response or not response.candidates:
return {
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
}
candidate = response.candidates[0]
content = None
reasoning_parts: list[str] = []
tool_calls = None
# Extract content and tool calls from response
# Extract content, reasoning, and tool calls from response
if candidate.content and candidate.content.parts:
for part in candidate.content.parts:
if part.text:
if getattr(part, "thought", False):
reasoning_parts.append(part.text)
else:
content = part.text.strip()
elif part.function_call:
# Handle function call
@ -294,9 +339,14 @@ class GeminiClient(GenAIClient):
"id": part.function_call.name or "",
"name": part.function_call.name or "",
"arguments": arguments,
"thought_signature": _encode_thought_signature(
getattr(part, "thought_signature", None)
),
}
)
reasoning = "".join(reasoning_parts).strip() or None
# Determine finish reason
finish_reason = "error"
if hasattr(candidate, "finish_reason") and candidate.finish_reason:
@ -322,6 +372,7 @@ class GeminiClient(GenAIClient):
return {
"content": content,
"reasoning": reasoning,
"tool_calls": tool_calls,
"finish_reason": finish_reason,
}
@ -330,6 +381,7 @@ class GeminiClient(GenAIClient):
logger.warning("Gemini API error during chat_with_tools: %s", str(e))
return {
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
}
@ -339,6 +391,7 @@ class GeminiClient(GenAIClient):
)
return {
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
}
@ -348,11 +401,14 @@ class GeminiClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> AsyncGenerator[tuple[str, Any], None]:
"""
Stream chat with tools; yields content deltas then final message.
Implements streaming function calling/tool usage for Gemini models.
``enable_thinking`` is accepted for interface parity; Gemini configures
thinking at the model level, so it is ignored here.
"""
try:
# Convert messages to Gemini format
@ -394,11 +450,17 @@ class GeminiClient(GenAIClient):
if not isinstance(tc_args, dict):
tc_args = {}
if tc_name:
parts.append(
types.Part.from_function_call(
fc_part = types.Part.from_function_call(
name=tc_name, args=tc_args
)
)
# Thinking-capable Gemini models require the original
# thought_signature to be echoed back on functionCall
# parts after a tool response, or the next request
# fails with INVALID_ARGUMENT.
sig = _decode_thought_signature(tc.get("thought_signature"))
if sig:
fc_part.thought_signature = sig
parts.append(fc_part)
if not parts:
parts.append(types.Part.from_text(text=" "))
gemini_messages.append(types.Content(role="model", parts=parts))
@ -477,12 +539,19 @@ class GeminiClient(GenAIClient):
if tool_config:
config_params["tool_config"] = tool_config
# Ask thinking-capable models to include their reasoning trace
# as separate `thought` parts (Gemini 2.5+; ignored elsewhere).
config_params["thinking_config"] = types.ThinkingConfig(
include_thoughts=True
)
# Merge runtime_options
if isinstance(self.genai_config.runtime_options, dict):
config_params.update(self.genai_config.runtime_options)
# Use streaming API
content_parts: list[str] = []
reasoning_parts: list[str] = []
tool_calls_by_index: dict[int, dict[str, Any]] = {}
finish_reason = "stop"
usage_stats: Optional[dict[str, Any]] = None
@ -519,10 +588,14 @@ class GeminiClient(GenAIClient):
]:
finish_reason = "error"
# Extract content and tool calls from chunk
# Extract content, reasoning, and tool calls from chunk
if candidate.content and candidate.content.parts:
for part in candidate.content.parts:
if part.text:
if getattr(part, "thought", False):
reasoning_parts.append(part.text)
yield ("reasoning_delta", part.text)
else:
content_parts.append(part.text)
yield ("content_delta", part.text)
elif part.function_call:
@ -553,6 +626,7 @@ class GeminiClient(GenAIClient):
"id": tool_call_id,
"name": tool_call_name,
"arguments": "",
"thought_signature": None,
}
# Accumulate arguments
@ -563,8 +637,16 @@ class GeminiClient(GenAIClient):
else str(arguments)
)
# Capture latest thought_signature for this call
chunk_sig = getattr(part, "thought_signature", None)
if chunk_sig:
tool_calls_by_index[found_index][
"thought_signature"
] = chunk_sig
# Build final message
full_content = "".join(content_parts).strip() or None
full_reasoning = "".join(reasoning_parts).strip() or None
# Convert tool calls to list format
tool_calls_list = None
@ -582,6 +664,9 @@ class GeminiClient(GenAIClient):
"id": tc["id"],
"name": tc["name"],
"arguments": parsed_args,
"thought_signature": _encode_thought_signature(
tc.get("thought_signature")
),
}
)
finish_reason = "tool_calls"
@ -593,6 +678,7 @@ class GeminiClient(GenAIClient):
"message",
{
"content": full_content,
"reasoning": full_reasoning,
"tool_calls": tool_calls_list,
"finish_reason": finish_reason,
},
@ -604,6 +690,7 @@ class GeminiClient(GenAIClient):
"message",
{
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
},
@ -616,6 +703,7 @@ class GeminiClient(GenAIClient):
"message",
{
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
},

185
frigate/genai/llama_cpp.py → frigate/genai/plugins/llama_cpp.py
View File

@ -4,7 +4,7 @@ import base64
import io
import json
import logging
from typing import Any, AsyncGenerator, Optional
from typing import Any, AsyncGenerator, Optional, cast
import httpx
import numpy as np
@ -75,6 +75,29 @@ def _parse_launch_arg(args: list[str], flag: str) -> str | None:
return args[idx + 1]
def _fetch_llama_props(base_url: str, model: str) -> dict[str, Any]:
"""Fetch /props from a llama.cpp server, with llama-swap fallback.
Raises the underlying RequestException if both endpoints fail; callers
decide how to surface the failure.
"""
try:
response = requests.get(
f"{base_url}/props",
params={"model": model},
timeout=10,
)
response.raise_for_status()
return cast(dict[str, Any], response.json())
except Exception:
response = requests.get(
f"{base_url}/upstream/{model}/props",
timeout=10,
)
response.raise_for_status()
return cast(dict[str, Any], response.json())
def _to_jpeg(img_bytes: bytes) -> bytes | None:
"""Convert image bytes to JPEG. llama.cpp/STB does not support WebP."""
try:
@ -99,6 +122,7 @@ class LlamaCppClient(GenAIClient):
_supports_vision: bool
_supports_audio: bool
_supports_tools: bool
_supports_reasoning: bool
_image_token_cache: dict[tuple[int, int], int]
_text_baseline_tokens: int | None
_media_marker: str
@ -112,6 +136,7 @@ class LlamaCppClient(GenAIClient):
self._supports_vision = False
self._supports_audio = False
self._supports_tools = False
self._supports_reasoning = False
self._image_token_cache = {}
self._text_baseline_tokens = None
self._media_marker = "<__media__>"
@ -127,6 +152,10 @@ class LlamaCppClient(GenAIClient):
else:
base_url = base_url.replace("/v1", "") # Strip /v1 if included in base_url
if not self.validate_model:
# Probe path
return base_url
configured_model = self.genai_config.model
info = self._get_model_info(base_url, configured_model)
@ -137,15 +166,17 @@ class LlamaCppClient(GenAIClient):
self._supports_vision = info["supports_vision"]
self._supports_audio = info["supports_audio"]
self._supports_tools = info["supports_tools"]
self._supports_reasoning = info["supports_reasoning"]
self._media_marker = info["media_marker"]
logger.info(
"llama.cpp model '%s' initialized — context: %s, vision: %s, audio: %s, tools: %s",
"llama.cpp model '%s' initialized — context: %s, vision: %s, audio: %s, tools: %s, reasoning: %s",
configured_model,
self._context_size or "unknown",
self._supports_vision,
self._supports_audio,
self._supports_tools,
self._supports_reasoning,
)
return base_url
@ -173,6 +204,7 @@ class LlamaCppClient(GenAIClient):
"supports_vision": False,
"supports_audio": False,
"supports_tools": False,
"supports_reasoning": False,
"media_marker": "<__media__>",
}
@ -239,21 +271,7 @@ class LlamaCppClient(GenAIClient):
info["supports_tools"] = True
try:
try:
response = requests.get(
f"{base_url}/props",
params={"model": configured_model},
timeout=10,
)
response.raise_for_status()
props = response.json()
except Exception:
response = requests.get(
f"{base_url}/upstream/{configured_model}/props",
timeout=10,
)
response.raise_for_status()
props = response.json()
props = _fetch_llama_props(base_url, configured_model)
if info["context_size"] is None:
default_settings = props.get("default_generation_settings", {})
@ -266,10 +284,17 @@ class LlamaCppClient(GenAIClient):
info["supports_vision"] = bool(modalities.get("vision", False))
info["supports_audio"] = bool(modalities.get("audio", False))
chat_caps = props.get("chat_template_caps") or {}
if not info["supports_tools"]:
chat_caps = props.get("chat_template_caps", {})
info["supports_tools"] = bool(chat_caps.get("supports_tools", False))
# llama.cpp does not advertise per-template reasoning support, so
# detect it by looking for the `enable_thinking` toggle variable
# in the Jinja chat template itself.
chat_template = props.get("chat_template") or ""
info["supports_reasoning"] = "enable_thinking" in chat_template
media_marker = props.get("media_marker")
if isinstance(media_marker, str) and media_marker:
info["media_marker"] = media_marker
@ -287,6 +312,7 @@ class LlamaCppClient(GenAIClient):
prompt: str,
images: list[bytes],
response_format: Optional[dict] = None,
enable_thinking: bool = False,
) -> Optional[str]:
"""Submit a request to llama.cpp server."""
if self.provider is None:
@ -314,7 +340,7 @@ class LlamaCppClient(GenAIClient):
)
# Build request payload with llama.cpp native options
payload = {
payload: dict[str, Any] = {
"model": self.genai_config.model,
"messages": [
{
@ -328,6 +354,9 @@ class LlamaCppClient(GenAIClient):
if response_format:
payload["response_format"] = response_format
if self.supports_toggleable_thinking:
payload["chat_template_kwargs"] = {"enable_thinking": enable_thinking}
response = requests.post(
f"{self.provider}/v1/chat/completions",
json=payload,
@ -364,6 +393,10 @@ class LlamaCppClient(GenAIClient):
"""Whether the loaded model supports tool/function calling."""
return self._supports_tools
@property
def supports_toggleable_thinking(self) -> bool:
return self._supports_reasoning
def list_models(self) -> list[str]:
"""Return available model IDs from the llama.cpp server."""
base_url = self.provider or (
@ -491,6 +524,7 @@ class LlamaCppClient(GenAIClient):
tools: Optional[list[dict[str, Any]]],
tool_choice: Optional[str],
stream: bool = False,
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
"""Build request payload for chat completions (sync or stream)."""
openai_tool_choice = None
@ -506,31 +540,47 @@ class LlamaCppClient(GenAIClient):
"messages": messages,
"model": self.genai_config.model,
}
if stream:
payload["stream"] = True
payload["stream_options"] = {"include_usage": True}
payload["timings_per_token"] = True
if tools:
payload["tools"] = tools
if openai_tool_choice is not None:
payload["tool_choice"] = openai_tool_choice
if enable_thinking is not None and self._supports_reasoning:
payload["chat_template_kwargs"] = {"enable_thinking": enable_thinking}
provider_opts = {
k: v for k, v in self.provider_options.items() if k != "context_size"
}
payload.update(provider_opts)
payload.update(self.genai_config.runtime_options)
return payload
def _message_from_choice(self, choice: dict[str, Any]) -> dict[str, Any]:
"""Parse OpenAI-style choice into {content, tool_calls, finish_reason}."""
"""Parse OpenAI-style choice into {content, reasoning, tool_calls, finish_reason}.
llama.cpp's `--reasoning-format` puts the trace in
`message.reasoning_content` (preferred) or `message.thinking`; both
keys are accepted so different builds work without configuration.
"""
message = choice.get("message", {})
content = message.get("content")
content = content.strip() if content else None
reasoning = message.get("reasoning_content") or message.get("thinking")
reasoning = reasoning.strip() if reasoning else None
tool_calls = parse_tool_calls_from_message(message)
finish_reason = choice.get("finish_reason") or (
"tool_calls" if tool_calls else "stop" if content else "error"
)
return {
"content": content,
"reasoning": reasoning,
"tool_calls": tool_calls,
"finish_reason": finish_reason,
}
@ -559,6 +609,31 @@ class LlamaCppClient(GenAIClient):
)
return result if result else None
def _refresh_media_marker(self) -> bool:
"""Re-fetch /props and update the cached media marker if it changed.
The server randomizes the marker per startup (unless LLAMA_MEDIA_MARKER
is set), so a stale marker indicates a restart. Returns True iff the
marker was updated to a new value — used to gate a one-shot retry of
a failed embeddings request.
"""
if self.provider is None:
return False
try:
props = _fetch_llama_props(self.provider, self.genai_config.model)
except Exception as e:
logger.warning("Failed to refresh llama.cpp media marker: %s", e)
return False
marker = props.get("media_marker")
if not isinstance(marker, str) or not marker or marker == self._media_marker:
return False
logger.info("llama.cpp media marker changed (server restart); refreshed")
self._media_marker = marker
return True
def embed(
self,
texts: list[str] | None = None,
@ -583,29 +658,45 @@ class LlamaCppClient(GenAIClient):
EMBEDDING_DIM = 768
content = []
for text in texts:
content.append({"prompt_string": text})
encoded_images: list[str] = []
for img in images:
# llama.cpp uses STB which does not support WebP; convert to JPEG
jpeg_bytes = _to_jpeg(img)
to_encode = jpeg_bytes if jpeg_bytes is not None else img
encoded = base64.b64encode(to_encode).decode("utf-8")
# prompt_string must contain the server's media marker placeholder.
# The marker is randomized per server startup (read from /props).
encoded_images.append(base64.b64encode(to_encode).decode("utf-8"))
def build_content() -> list[dict[str, Any]]:
# prompt_string must contain the server's media marker placeholder
# for each image. The marker is randomized per server startup.
content: list[dict[str, Any]] = []
for text in texts:
content.append({"prompt_string": text})
for encoded in encoded_images:
content.append(
{
"prompt_string": f"{self._media_marker}\n",
"multimodal_data": [encoded], # type: ignore[dict-item]
"multimodal_data": [encoded],
}
)
return content
def post_embeddings() -> requests.Response:
return requests.post(
f"{self.provider}/embeddings",
json={"model": self.genai_config.model, "content": build_content()},
timeout=self.timeout,
)
try:
response = requests.post(
f"{self.provider}/embeddings",
json={"model": self.genai_config.model, "content": content},
timeout=self.timeout,
)
try:
response = post_embeddings()
response.raise_for_status()
except requests.exceptions.RequestException:
# The server may have restarted with a new media marker.
# Refresh from /props; only retry if the marker actually changed.
if not encoded_images or not self._refresh_media_marker():
raise
response = post_embeddings()
response.raise_for_status()
result = response.json()
@ -669,6 +760,7 @@ class LlamaCppClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
"""
Send chat messages to llama.cpp server with optional tool definitions.
@ -686,7 +778,13 @@ class LlamaCppClient(GenAIClient):
"finish_reason": "error",
}
try:
payload = self._build_payload(messages, tools, tool_choice, stream=False)
payload = self._build_payload(
messages,
tools,
tool_choice,
stream=False,
enable_thinking=enable_thinking,
)
response = requests.post(
f"{self.provider}/v1/chat/completions",
json=payload,
@ -734,6 +832,7 @@ class LlamaCppClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> AsyncGenerator[tuple[str, Any], None]:
"""Stream chat with tools via OpenAI-compatible streaming API."""
if self.provider is None:
@ -750,8 +849,15 @@ class LlamaCppClient(GenAIClient):
)
return
try:
payload = self._build_payload(messages, tools, tool_choice, stream=True)
payload = self._build_payload(
messages,
tools,
tool_choice,
stream=True,
enable_thinking=enable_thinking,
)
content_parts: list[str] = []
reasoning_parts: list[str] = []
tool_calls_by_index: dict[int, dict[str, Any]] = {}
finish_reason = "stop"
@ -781,6 +887,15 @@ class LlamaCppClient(GenAIClient):
delta = choices[0].get("delta", {})
if choices[0].get("finish_reason"):
finish_reason = choices[0]["finish_reason"]
# llama.cpp emits separated thinking under
# reasoning_content (preferred) or thinking before any
# content tokens arrive
reasoning_delta = delta.get("reasoning_content") or delta.get(
"thinking"
)
if reasoning_delta:
reasoning_parts.append(reasoning_delta)
yield ("reasoning_delta", reasoning_delta)
if delta.get("content"):
content_parts.append(delta["content"])
yield ("content_delta", delta["content"])
@ -806,6 +921,7 @@ class LlamaCppClient(GenAIClient):
)
full_content = "".join(content_parts).strip() or None
full_reasoning = "".join(reasoning_parts).strip() or None
tool_calls_list = self._streamed_tool_calls_to_list(tool_calls_by_index)
if tool_calls_list:
finish_reason = "tool_calls"
@ -813,6 +929,7 @@ class LlamaCppClient(GenAIClient):
"message",
{
"content": full_content,
"reasoning": full_reasoning,
"tool_calls": tool_calls_list,
"finish_reason": finish_reason,
},

61
frigate/genai/ollama.py → frigate/genai/plugins/ollama.py
View File

@ -98,6 +98,22 @@ class OllamaClient(GenAIClient):
provider: ApiClient | None
provider_options: dict[str, Any]
_supports_thinking_cache: Optional[bool] = None
@property
def supports_toggleable_thinking(self) -> bool:
if self._supports_thinking_cache is not None:
return self._supports_thinking_cache
if self.provider is None:
return False
try:
response = self.provider.show(self.genai_config.model)
capabilities = response.get("capabilities") or []
self._supports_thinking_cache = "thinking" in capabilities
except Exception as e:
logger.debug("Failed to query Ollama model capabilities: %s", e)
self._supports_thinking_cache = False
return self._supports_thinking_cache
def _auth_headers(self) -> dict | None:
if self.genai_config.api_key:
@ -118,6 +134,9 @@ class OllamaClient(GenAIClient):
timeout=self.timeout,
headers=self._auth_headers(),
)
if not self.validate_model:
# Probe path
return client
# ensure the model is available locally
response = client.show(self.genai_config.model)
if response.get("error"):
@ -175,6 +194,7 @@ class OllamaClient(GenAIClient):
prompt: str,
images: list[bytes],
response_format: Optional[dict] = None,
enable_thinking: bool = False,
) -> Optional[str]:
"""Submit a request to Ollama"""
if self.provider is None:
@ -191,6 +211,8 @@ class OllamaClient(GenAIClient):
schema = response_format.get("json_schema", {}).get("schema")
if schema:
ollama_options["format"] = self._clean_schema_for_ollama(schema)
if self.supports_toggleable_thinking:
ollama_options["think"] = enable_thinking
logger.debug(
"Ollama generate request: model=%s, prompt_len=%s, image_count=%s, "
"has_format=%s, options=%s",
@ -271,6 +293,7 @@ class OllamaClient(GenAIClient):
tools: Optional[list[dict[str, Any]]],
tool_choice: Optional[str],
stream: bool = False,
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
"""Build request_messages and params for chat (sync or stream)."""
request_messages = []
@ -309,11 +332,14 @@ class OllamaClient(GenAIClient):
"model": self.genai_config.model,
"messages": request_messages,
**self.provider_options,
**self.genai_config.runtime_options,
}
if stream:
request_params["stream"] = True
if tools:
request_params["tools"] = tools
if enable_thinking is not None and self.supports_toggleable_thinking:
request_params["think"] = enable_thinking
return request_params
def _message_from_response(self, response: dict[str, Any]) -> dict[str, Any]:
@ -336,6 +362,9 @@ class OllamaClient(GenAIClient):
response.get("done"),
)
content = message.get("content", "").strip() if message.get("content") else None
reasoning = (
message.get("thinking", "").strip() if message.get("thinking") else None
)
tool_calls = parse_tool_calls_from_message(message)
finish_reason = "error"
if response.get("done"):
@ -348,6 +377,7 @@ class OllamaClient(GenAIClient):
finish_reason = "stop"
return {
"content": content,
"reasoning": reasoning,
"tool_calls": tool_calls,
"finish_reason": finish_reason,
}
@ -357,6 +387,7 @@ class OllamaClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
if self.provider is None:
logger.warning(
@ -369,7 +400,11 @@ class OllamaClient(GenAIClient):
}
try:
request_params = self._build_request_params(
messages, tools, tool_choice, stream=False
messages,
tools,
tool_choice,
stream=False,
enable_thinking=enable_thinking,
)
response = self.provider.chat(**request_params)
return self._message_from_response(response)
@ -393,6 +428,7 @@ class OllamaClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> AsyncGenerator[tuple[str, Any], None]:
"""Stream chat with tools; yields content deltas then final message.
@ -422,7 +458,11 @@ class OllamaClient(GenAIClient):
"Ollama: tools provided, using non-streaming call for tool support"
)
request_params = self._build_request_params(
messages, tools, tool_choice, stream=False
messages,
tools,
tool_choice,
stream=False,
enable_thinking=enable_thinking,
)
async_client = OllamaAsyncClient(
host=self.genai_config.base_url,
@ -431,6 +471,9 @@ class OllamaClient(GenAIClient):
)
response = await async_client.chat(**request_params)
result = self._message_from_response(response)
reasoning = result.get("reasoning")
if reasoning:
yield ("reasoning_delta", reasoning)
content = result.get("content")
if content:
yield ("content_delta", content)
@ -441,7 +484,11 @@ class OllamaClient(GenAIClient):
return
request_params = self._build_request_params(
messages, tools, tool_choice, stream=True
messages,
tools,
tool_choice,
stream=True,
enable_thinking=enable_thinking,
)
async_client = OllamaAsyncClient(
host=self.genai_config.base_url,
@ -449,6 +496,7 @@ class OllamaClient(GenAIClient):
headers=self._auth_headers(),
)
content_parts: list[str] = []
reasoning_parts: list[str] = []
final_message: dict[str, Any] | None = None
final_chunk: Any = None
stream = await async_client.chat(**request_params)
@ -456,6 +504,10 @@ class OllamaClient(GenAIClient):
if not chunk or "message" not in chunk:
continue
msg = chunk.get("message", {})
reasoning_delta = msg.get("thinking") or ""
if reasoning_delta:
reasoning_parts.append(reasoning_delta)
yield ("reasoning_delta", reasoning_delta)
delta = msg.get("content") or ""
if delta:
content_parts.append(delta)
@ -463,8 +515,10 @@ class OllamaClient(GenAIClient):
if chunk.get("done"):
final_chunk = chunk
full_content = "".join(content_parts).strip() or None
full_reasoning = "".join(reasoning_parts).strip() or None
final_message = {
"content": full_content,
"reasoning": full_reasoning,
"tool_calls": None,
"finish_reason": "stop",
}
@ -481,6 +535,7 @@ class OllamaClient(GenAIClient):
"message",
{
"content": "".join(content_parts).strip() or None,
"reasoning": "".join(reasoning_parts).strip() or None,
"tool_calls": None,
"finish_reason": "stop",
},

43
frigate/genai/openai.py → frigate/genai/plugins/openai.py
View File

@ -38,7 +38,11 @@ class OpenAIClient(GenAIClient):
context_size: Optional[int] = None
def _init_provider(self) -> OpenAI:
"""Initialize the client."""
"""Initialize the client.
Subclasses (e.g. Azure) should raise on configuration errors; the
manager catches construction failures and disables the provider.
"""
# Extract context_size from provider_options as it's not a valid OpenAI client parameter
# It will be used in get_context_size() instead
provider_opts = {
@ -57,6 +61,7 @@ class OpenAIClient(GenAIClient):
prompt: str,
images: list[bytes],
response_format: Optional[dict] = None,
enable_thinking: bool = False,
) -> Optional[str]:
"""Submit a request to OpenAI."""
encoded_images = [base64.b64encode(image).decode("utf-8") for image in images]
@ -183,11 +188,14 @@ class OpenAIClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> dict[str, Any]:
"""
Send chat messages to OpenAI with optional tool definitions.
Implements function calling/tool usage for OpenAI models.
Implements function calling/tool usage for OpenAI models. The OpenAI
chat completions API does not expose a per-request thinking toggle,
so ``enable_thinking`` is accepted for interface parity and ignored.
"""
try:
openai_tool_choice = None
@ -203,6 +211,7 @@ class OpenAIClient(GenAIClient):
"model": self.genai_config.model,
"messages": messages,
"timeout": self.timeout,
**self.genai_config.runtime_options,
}
if tools:
@ -219,7 +228,7 @@ class OpenAIClient(GenAIClient):
}
request_params.update(provider_opts)
result = self.provider.chat.completions.create(**request_params) # type: ignore[call-overload]
result = self.provider.chat.completions.create(**request_params)
if (
result is None
@ -235,6 +244,10 @@ class OpenAIClient(GenAIClient):
choice = result.choices[0]
message = choice.message
content = message.content.strip() if message.content else None
raw_reasoning = getattr(message, "reasoning_content", None) or getattr(
message, "reasoning", None
)
reasoning = raw_reasoning.strip() if raw_reasoning else None
tool_calls = None
if message.tool_calls:
@ -269,6 +282,7 @@ class OpenAIClient(GenAIClient):
return {
"content": content,
"reasoning": reasoning,
"tool_calls": tool_calls,
"finish_reason": finish_reason,
}
@ -277,6 +291,7 @@ class OpenAIClient(GenAIClient):
logger.warning("OpenAI request timed out: %s", str(e))
return {
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
}
@ -284,6 +299,7 @@ class OpenAIClient(GenAIClient):
logger.warning("OpenAI returned an error: %s", str(e))
return {
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
}
@ -293,11 +309,15 @@ class OpenAIClient(GenAIClient):
messages: list[dict[str, Any]],
tools: Optional[list[dict[str, Any]]] = None,
tool_choice: Optional[str] = "auto",
enable_thinking: Optional[bool] = None,
) -> AsyncGenerator[tuple[str, Any], None]:
"""
Stream chat with tools; yields content deltas then final message.
Implements streaming function calling/tool usage for OpenAI models.
The OpenAI chat completions API does not expose a per-request thinking
toggle, so ``enable_thinking`` is accepted for interface parity and
ignored.
"""
try:
openai_tool_choice = None
@ -315,6 +335,7 @@ class OpenAIClient(GenAIClient):
"timeout": self.timeout,
"stream": True,
"stream_options": {"include_usage": True},
**self.genai_config.runtime_options,
}
if tools:
@ -333,11 +354,12 @@ class OpenAIClient(GenAIClient):
# Use streaming API
content_parts: list[str] = []
reasoning_parts: list[str] = []
tool_calls_by_index: dict[int, dict[str, Any]] = {}
finish_reason = "stop"
usage_stats: Optional[dict[str, Any]] = None
stream = self.provider.chat.completions.create(**request_params) # type: ignore[call-overload]
stream = self.provider.chat.completions.create(**request_params)
for chunk in stream:
chunk_usage = getattr(chunk, "usage", None)
@ -354,6 +376,15 @@ class OpenAIClient(GenAIClient):
if choice.finish_reason:
finish_reason = choice.finish_reason
# Extract reasoning deltas (reasoning_content or reasoning,
# depending on the server)
reasoning_delta = getattr(delta, "reasoning_content", None) or getattr(
delta, "reasoning", None
)
if reasoning_delta:
reasoning_parts.append(reasoning_delta)
yield ("reasoning_delta", reasoning_delta)
# Extract content deltas
if delta.content:
content_parts.append(delta.content)
@ -382,6 +413,7 @@ class OpenAIClient(GenAIClient):
# Build final message
full_content = "".join(content_parts).strip() or None
full_reasoning = "".join(reasoning_parts).strip() or None
# Convert tool calls to list format
tool_calls_list = None
@ -410,6 +442,7 @@ class OpenAIClient(GenAIClient):
"message",
{
"content": full_content,
"reasoning": full_reasoning,
"tool_calls": tool_calls_list,
"finish_reason": finish_reason,
},
@ -421,6 +454,7 @@ class OpenAIClient(GenAIClient):
"message",
{
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
},
@ -431,6 +465,7 @@ class OpenAIClient(GenAIClient):
"message",
{
"content": None,
"reasoning": None,
"tool_calls": None,
"finish_reason": "error",
},

744
frigate/genai/prompts.py Normal file
View File

@ -0,0 +1,744 @@
"""Prompt and response-format builders for GenAI features.
Centralizes the per-feature prompt framing and structured-output schema
shaping so provider clients in :mod:`frigate.genai.plugins` only handle
transport.
"""
import datetime
from typing import Any, Dict, List, Optional
from playhouse.shortcuts import model_to_dict
from frigate.config import CameraConfig, FrigateConfig
from frigate.config.classification import ObjectClassificationType
from frigate.config.ui import UnitSystemEnum
from frigate.data_processing.post.types import ReviewMetadata
from frigate.models import Event
def build_review_description_prompt(
review_data: dict[str, Any],
thumbnails: list[bytes],
concerns: list[str],
preferred_language: str | None,
activity_context_prompt: str,
) -> str:
"""Build the prompt for review activity description generation."""
def get_concern_prompt() -> str:
if concerns:
concern_list = "\n - ".join(concerns)
return (
"\n- `other_concerns` (list of strings): Include a list of any of "
"the following concerns that are occurring:\n"
f" - {concern_list}"
)
else:
return ""
def get_language_prompt() -> str:
if preferred_language:
return f"Provide your answer in {preferred_language}"
else:
return ""
def get_objects_list() -> str:
if review_data["unified_objects"]:
return "\n- " + "\n- ".join(review_data["unified_objects"])
else:
return "\n- (No objects detected)"
return f"""
Your task is to analyze a sequence of images taken in chronological order from a security camera.
## Normal Activity Patterns for This Property
{activity_context_prompt}
## Task Instructions
Describe the scene based on observable actions and movements, evaluate the activity against the Activity Indicators above, and assign a potential_threat_level (0, 1, or 2) by applying the threat level indicators consistently.
## Analysis Guidelines
When forming your description:
- **Treat "Objects in Scene" as the list of tracked subjects to describe.** Do not introduce additional people or vehicles that are not present in this list. You may freely reference other items, surfaces, and environmental details visible in the frames when describing what the listed subjects are doing.
- **Describe the most likely activity from visible cues across the sequence** — the subject's path, what they are carrying, and what they interact with. Avoid asserting completed outcomes you do not observe; describe in-progress actions rather than results.
- Describe what you observe: actions, movements, interactions with objects and the environment. Include any observable environmental changes (e.g., lighting changes triggered by activity).
- Note visible details such as clothing, items being carried or placed, tools or equipment present, and how they interact with the property or objects.
- Consider the full sequence chronologically: what happens from start to finish, how duration and actions relate to the location and objects involved.
- **Use the actual timestamp provided in "Activity started at"** below for time of day context—do not infer time from image brightness or darkness. Unusual hours (late night/early morning) should increase suspicion when the observable behavior itself appears questionable. However, recognize that some legitimate activities can occur at any hour.
- **Consider duration as a primary factor**: Apply the duration thresholds defined in the activity patterns above. Brief sequences during normal hours with apparent purpose typically indicate normal activity unless explicit suspicious actions are visible.
- **Weigh all evidence holistically**: Match the activity against the normal and suspicious patterns defined above, then evaluate based on the complete context (zone, objects, time, actions, duration). Apply the threat level indicators consistently. Use your judgment for edge cases.
## Response Field Guidelines
Respond with a JSON object matching the provided schema. Field-specific guidance:
- `observations`: Include the very start of the activity — for example, a vehicle entering the frame or pulling into the driveway — even if it lasts only a few frames and the rest of the clip is dominated by a longer activity. Include each arrival, departure, object handled, and notable change in position or state. Each item is a single concrete fact written as a complete sentence.
- `scene`: Describe how the sequence begins, then the progression of events — all significant movements and actions in order. For example, if a vehicle arrives and then a person exits, describe both sequentially. For named subjects (those with a `←` separator in "Objects in Scene"), always use their name — do not replace them with generic terms. For unnamed objects (e.g., "person", "car"), refer to them naturally with articles (e.g., "a person", "the car"). Your description should align with and support the threat level you assign.
- `title`: Name the primary activity across the observations, together with the location. An activity is what is being done with objects, tools, or surfaces; locomotion through the scene qualifies as the activity only when no other interaction is observed. For named subjects, always use their name. For unnamed objects, refer to them naturally with articles.
- `shortSummary`: Briefly summarize the primary activity across the observations.
- `potential_threat_level`: Must be consistent with your scene description and the activity patterns above.
{get_concern_prompt()}
## Sequence Details
- Camera: {review_data["camera"]}
- Total frames: {len(thumbnails)} (Frame 1 = earliest, Frame {len(thumbnails)} = latest)
- Activity started at {review_data["start"]} and lasted {review_data["duration"]} seconds
- Zones involved: {", ".join(review_data["zones"]) if review_data["zones"] else "None"}
## Objects in Scene
Each line represents a detection state, not necessarily unique individuals. The `←` symbol separates a recognized subject's name from their object type — use only the name (before the `←`) in your response, not the type after it. The same subject may appear across multiple lines if detected multiple times.
**Note: Unidentified objects (without names) are NOT indicators of suspicious activity—they simply mean the system hasn't identified that object.**
{get_objects_list()}
{get_language_prompt()}
"""
def build_review_description_response_format(concerns: list[str]) -> dict[str, Any]:
"""Build the structured-output JSON schema for review descriptions.
Strips the `time` field (populated server-side) and drops
`other_concerns` when no concerns are configured.
"""
schema = ReviewMetadata.model_json_schema()
schema.get("properties", {}).pop("time", None)
if "time" in schema.get("required", []):
schema["required"].remove("time")
if not concerns:
schema.get("properties", {}).pop("other_concerns", None)
if "other_concerns" in schema.get("required", []):
schema["required"].remove("other_concerns")
return {
"type": "json_schema",
"json_schema": {
"name": "review_metadata",
"strict": True,
"schema": schema,
},
}
def build_review_summary_prompt(
start_ts: float,
end_ts: float,
events: list[dict[str, Any]],
preferred_language: str | None,
) -> str:
"""Build the prompt for a multi-event review summary."""
time_range = (
f"{datetime.datetime.fromtimestamp(start_ts).strftime('%B %d, %Y at %I:%M %p')}"
f" to "
f"{datetime.datetime.fromtimestamp(end_ts).strftime('%B %d, %Y at %I:%M %p')}"
)
prompt = f"""
You are a security officer writing a concise security report.
Time range: {time_range}
Input format: Each event is a JSON object with:
- "title", "scene", "confidence", "potential_threat_level" (0-2), "other_concerns", "camera", "time", "start_time", "end_time"
- "context": array of related events from other cameras that occurred during overlapping time periods
**Note: Use the "scene" field for event descriptions in the report. Ignore any "shortSummary" field if present.**
Report Structure - Use this EXACT format:
# Security Summary - {time_range}
## Overview
[Write 1-2 sentences summarizing the overall activity pattern during this period.]
---
## Timeline
[Group events by time periods (e.g., "Morning (6:00 AM - 12:00 PM)", "Afternoon (12:00 PM - 5:00 PM)", "Evening (5:00 PM - 9:00 PM)", "Night (9:00 PM - 6:00 AM)"). Use appropriate time blocks based on when events occurred.]
### [Time Block Name]
**HH:MM AM/PM** | [Camera Name] | [Threat Level Indicator]
- [Event title]: [Clear description incorporating contextual information from the "context" array]
- Context: [If context array has items, mention them here, e.g., "Delivery truck present on Front Driveway Cam (HH:MM AM/PM)"]
- Assessment: [Brief assessment incorporating context - if context explains the event, note it here]
[Repeat for each event in chronological order within the time block]
---
## Summary
[One sentence summarizing the period. If all events are normal/explained: "Routine activity observed." If review needed: "Some activity requires review but no security concerns." If security concerns: "Security concerns requiring immediate attention."]
Guidelines:
- List ALL events in chronological order, grouped by time blocks
- Threat level indicators: ✓ Normal, ⚠️ Needs review, 🔴 Security concern
- Integrate contextual information naturally - use the "context" array to enrich each event's description
- If context explains the event (e.g., delivery truck explains person at door), describe it accordingly (e.g., "delivery person" not "unidentified person")
- Be concise but informative - focus on what happened and what it means
- If contextual information makes an event clearly normal, reflect that in your assessment
- Only create time blocks that have events - don't create empty sections
"""
prompt += "\n\nEvents:\n"
for event in events:
prompt += f"\n{event}\n"
if preferred_language:
prompt += f"\nProvide your answer in {preferred_language}"
return prompt
def build_object_description_prompt(
camera_config: CameraConfig,
event: Event,
) -> str:
"""Build the prompt for a per-object description.
Pulls the per-label override from `objects.genai.object_prompts`, falling
back to the camera default, and interpolates event fields.
Raises:
KeyError: if the user-defined prompt template references an unknown
event field.
"""
template = camera_config.objects.genai.object_prompts.get(
str(event.label),
camera_config.objects.genai.prompt,
)
return template.format(**model_to_dict(event))
def get_attribute_classifications(config: FrigateConfig) -> List[Dict[str, Any]]:
"""Return enabled custom classification models of `attribute` type.
Each entry: {"name": <model name>, "objects": [<object label>, ...]}.
These models attach attribute metadata to events on the listed object
types, which can later be filtered via the search_objects `attribute`
field.
"""
result: List[Dict[str, Any]] = []
for model_key, model_config in config.classification.custom.items():
if not model_config.enabled or model_config.object_config is None:
continue
if (
model_config.object_config.classification_type
!= ObjectClassificationType.attribute
):
continue
result.append(
{
"name": model_config.name or model_key,
"objects": list(model_config.object_config.objects or []),
}
)
return result
def get_tool_definitions(
semantic_search_enabled: bool = False,
attribute_classifications: Optional[List[Dict[str, Any]]] = None,
) -> List[Dict[str, Any]]:
"""
Get OpenAI-compatible tool definitions for Frigate.
Returns a list of tool definitions that can be used with OpenAI-compatible
function calling APIs. When semantic search is enabled, the search_objects
tool exposes an additional `semantic_query` parameter for descriptive
queries (e.g. "person riding a lawn mower") and find_similar_objects is
included. When attribute classification models are configured, an
`attribute` parameter is exposed for filtering by their labels.
"""
search_objects_properties: Dict[str, Any] = {
"camera": {
"type": "string",
"description": "Camera name to filter by (optional).",
},
"label": {
"type": "string",
"description": (
"Generic object class to filter by — one of the tracked detector "
"labels such as 'person', 'package', 'car', 'dog', 'bird'. Use "
"this for broad queries like 'show me all cars today'. Combine "
"with semantic_query when the user also describes appearance or "
"behavior (e.g. label='person', semantic_query='riding a lawn "
"mower')."
),
},
"sub_label": {
"type": "string",
"description": (
"Filter by a DISCRETE NAMED entity recognized in the detection. "
"Use this for: a known person's name ('John'), a delivery "
"company ('Amazon', 'UPS'), a recognized animal species or "
"breed ('blue jay', 'cardinal', 'golden retriever'), or a "
"license plate string. When filtering by a specific name, set "
"only sub_label and leave label unset. Do NOT use sub_label "
"for descriptions of appearance, clothing, or actions — those "
"belong in semantic_query."
),
},
"after": {
"type": "string",
"description": "Start time in ISO 8601 format (e.g., '2024-01-01T00:00:00Z').",
},
"before": {
"type": "string",
"description": "End time in ISO 8601 format (e.g., '2024-01-01T23:59:59Z').",
},
"zones": {
"type": "array",
"items": {"type": "string"},
"description": "List of zone names to filter by.",
},
"limit": {
"type": "integer",
"description": "Maximum number of objects to return (default: 25).",
"default": 25,
},
}
if attribute_classifications:
model_outline = "; ".join(
f"{m['name']} (applies to {', '.join(m['objects']) or 'any object'})"
for m in attribute_classifications
)
search_objects_properties["attribute"] = {
"type": "string",
"description": (
"Filter by a classification attribute label produced by a "
"configured attribute classification model. Use this INSTEAD "
"of semantic_query when the user's request matches one of "
"these classifications. Configured models: "
f"{model_outline}. "
"Set the value to the attribute label that matches the user's "
"phrasing (case-sensitive)."
),
}
if semantic_search_enabled:
search_objects_properties["semantic_query"] = {
"type": "string",
"description": (
"Optional natural-language description of a PHYSICAL "
"CHARACTERISTIC, APPEARANCE, or ACTIVITY the user mentioned, "
"used to semantically narrow results. Only set this when the "
"user describes something beyond what label and sub_label can "
"express on their own.\n"
"USE for descriptive phrases like: 'riding a lawn mower', "
"'wearing a red jacket', 'carrying a package', 'walking a "
"dog', 'on a bicycle', 'holding an umbrella'.\n"
"DO NOT USE for:\n"
"- specific named people, pets, or delivery companies → use sub_label\n"
"- animal species or breed names like 'blue jay', 'cardinal', "
"'golden retriever' → use sub_label\n"
"- license plate strings → use sub_label\n"
"- generic object queries like 'all cars today' or 'every "
"person' → use label alone with no semantic_query\n"
"When set, combine with label/time/camera/zone filters as "
"usual (e.g. label='person', semantic_query='riding a lawn "
"mower', after='2024-05-01T00:00:00Z')."
),
}
search_objects_description = (
"Search the historical record of detected objects in Frigate. "
"Use this ONLY for questions about the PAST — e.g. 'did anyone come by today?', "
"'when was the last car?', 'show me detections from yesterday'. "
"Do NOT use this for monitoring or alerting requests about future events — "
"use start_camera_watch instead for those. "
"An 'object' in Frigate represents a tracked detection (e.g., a person, package, car).\n\n"
"Choose filters based on what the user is asking for:\n"
"- Generic class query ('show me all cars today'): set `label` only.\n"
"- Specific NAMED entity (known person, delivery company, animal "
"species/breed like 'blue jay' or 'golden retriever', license "
"plate): set `sub_label` only and leave `label` unset.\n"
)
if semantic_search_enabled:
search_objects_description += (
"- Physical CHARACTERISTIC, APPEARANCE, or ACTIVITY that is not a "
"discrete name ('person riding a lawn mower', 'someone in a red "
"jacket', 'person carrying a package'): set `semantic_query` with "
"the descriptive phrase, optionally alongside `label` for the "
"object class. Do NOT put descriptive phrases in sub_label."
)
return [
{
"type": "function",
"function": {
"name": "search_objects",
"description": search_objects_description,
"parameters": {
"type": "object",
"properties": search_objects_properties,
},
"required": [],
},
},
{
"type": "function",
"function": {
"name": "find_similar_objects",
"description": (
"Find tracked objects that are visually and semantically similar "
"to a specific past event. Use this when the user references a "
"particular object they have seen and wants to find other "
"sightings of the same or similar one ('that green car', 'the "
"person in the red jacket', 'the package that was delivered'). "
"Prefer this over search_objects whenever the user's intent is "
"'find more like this specific one.' Use search_objects first "
"only if you need to locate the anchor event. Requires semantic "
"search to be enabled."
),
"parameters": {
"type": "object",
"properties": {
"event_id": {
"type": "string",
"description": "The id of the anchor event to find similar objects to.",
},
"after": {
"type": "string",
"description": "Start time in ISO 8601 format (e.g., '2024-01-01T00:00:00Z').",
},
"before": {
"type": "string",
"description": "End time in ISO 8601 format (e.g., '2024-01-01T23:59:59Z').",
},
"cameras": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of cameras to restrict to. Defaults to all.",
},
"labels": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of labels to restrict to. Defaults to the anchor event's label.",
},
"sub_labels": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of sub_labels (names) to restrict to.",
},
"zones": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of zones. An event matches if any of its zones overlap.",
},
"similarity_mode": {
"type": "string",
"enum": ["visual", "semantic", "fused"],
"description": "Which similarity signal(s) to use. 'fused' (default) combines visual and semantic.",
"default": "fused",
},
"min_score": {
"type": "number",
"description": "Drop matches with a similarity score below this threshold (0.0-1.0).",
},
"limit": {
"type": "integer",
"description": "Maximum number of matches to return (default: 10).",
"default": 10,
},
},
"required": ["event_id"],
},
},
},
{
"type": "function",
"function": {
"name": "set_camera_state",
"description": (
"Change a camera's feature state (e.g., turn detection on/off, enable/disable recordings). "
"Use camera='*' to apply to all cameras at once. "
"Only call this tool when the user explicitly asks to change a camera setting. "
"Requires admin privileges."
),
"parameters": {
"type": "object",
"properties": {
"camera": {
"type": "string",
"description": "Camera name to target, or '*' to target all cameras.",
},
"feature": {
"type": "string",
"enum": [
"detect",
"record",
"snapshots",
"audio",
"motion",
"enabled",
"birdseye",
"birdseye_mode",
"improve_contrast",
"ptz_autotracker",
"motion_contour_area",
"motion_threshold",
"notifications",
"audio_transcription",
"review_alerts",
"review_detections",
"object_descriptions",
"review_descriptions",
"profile",
],
"description": (
"The feature to change. Most features accept ON or OFF. "
"birdseye_mode accepts CONTINUOUS, MOTION, or OBJECTS. "
"motion_contour_area and motion_threshold accept a number. "
"profile accepts a profile name or 'none' to deactivate (requires camera='*')."
),
},
"value": {
"type": "string",
"description": "The value to set. ON or OFF for toggles, a number for thresholds, a profile name or 'none' for profile.",
},
},
"required": ["camera", "feature", "value"],
},
},
},
{
"type": "function",
"function": {
"name": "get_live_context",
"description": (
"Get the current live image and detection information for a single camera: objects being tracked, "
"zones, timestamps. Use this to understand what is visible in the live view. "
"Call this when answering questions about what is happening right now on a specific camera. "
"Operates on one camera at a time; call the tool again for each additional camera. "
"Wildcards and empty values are not accepted."
),
"parameters": {
"type": "object",
"properties": {
"camera": {
"type": "string",
"description": (
"Exact name of a single camera to get live context for. "
"Wildcards (e.g. '*', 'all') and empty strings are not accepted."
),
},
},
"required": ["camera"],
},
},
},
{
"type": "function",
"function": {
"name": "start_camera_watch",
"description": (
"Start a continuous VLM watch job that monitors a camera and sends a notification "
"when a specified condition is met. Use this when the user wants to be alerted about "
"a future event, e.g. 'tell me when guests arrive' or 'notify me when the package is picked up'. "
"Only one watch job can run at a time. Returns a job ID."
),
"parameters": {
"type": "object",
"properties": {
"camera": {
"type": "string",
"description": "Camera ID to monitor.",
},
"condition": {
"type": "string",
"description": (
"Natural-language description of the condition to watch for, "
"e.g. 'a person arrives at the front door'."
),
},
"max_duration_minutes": {
"type": "integer",
"description": "Maximum time to watch before giving up (minutes, default 60).",
"default": 60,
},
"labels": {
"type": "array",
"items": {"type": "string"},
"description": "Object labels that should trigger a VLM check (e.g. ['person', 'car']). If omitted, any detection on the camera triggers a check.",
},
"zones": {
"type": "array",
"items": {"type": "string"},
"description": "Zone names to filter by. If specified, only detections in these zones trigger a VLM check.",
},
},
"required": ["camera", "condition"],
},
},
},
{
"type": "function",
"function": {
"name": "stop_camera_watch",
"description": (
"Cancel the currently running VLM watch job. Use this when the user wants to "
"stop a previously started watch, e.g. 'stop watching the front door'."
),
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "get_profile_status",
"description": (
"Get the current profile status including the active profile and "
"timestamps of when each profile was last activated. Use this to "
"determine time periods for recap requests — e.g. when the user asks "
"'what happened while I was away?', call this first to find the relevant "
"time window based on profile activation history."
),
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "get_recap",
"description": (
"Get a recap of all activity (alerts and detections) for a given time period. "
"Use this after calling get_profile_status to retrieve what happened during "
"a specific window — e.g. 'what happened while I was away?'. Returns a "
"chronological list of activity with camera, objects, zones, and GenAI-generated "
"descriptions when available. Summarize the results for the user."
),
"parameters": {
"type": "object",
"properties": {
"after": {
"type": "string",
"description": "Start of the time period in ISO 8601 format (e.g. '2025-03-15T08:00:00').",
},
"before": {
"type": "string",
"description": "End of the time period in ISO 8601 format (e.g. '2025-03-15T17:00:00').",
},
"cameras": {
"type": "string",
"description": "Comma-separated camera IDs to include, or 'all' for all cameras. Default is 'all'.",
},
"severity": {
"type": "string",
"enum": ["alert", "detection"],
"description": "Filter by severity level. Omit to include both alerts and detections.",
},
},
"required": ["after", "before"],
},
},
},
]
def build_chat_system_prompt(
config: FrigateConfig,
allowed_cameras: List[str],
semantic_search_enabled: bool,
attribute_classifications: List[Dict[str, Any]],
) -> str:
"""Build the system prompt for the chat completion endpoint.
Composes the static framing with conditional sections describing the
available cameras, speed units, semantic-search routing guidance, and
configured attribute classifications.
"""
current_datetime = datetime.datetime.now()
current_date_str = current_datetime.strftime("%Y-%m-%d")
current_time_str = current_datetime.strftime("%I:%M:%S %p")
cameras_info: List[str] = []
has_speed_zone = False
for camera_id in allowed_cameras:
if camera_id not in config.cameras:
continue
camera_config = config.cameras[camera_id]
friendly_name = (
camera_config.friendly_name
if camera_config.friendly_name
else camera_id.replace("_", " ").title()
)
zone_names = list(camera_config.zones.keys())
if not has_speed_zone:
has_speed_zone = any(
zone.distances for zone in camera_config.zones.values()
)
if zone_names:
cameras_info.append(
f" - {friendly_name} (ID: {camera_id}, zones: {', '.join(zone_names)})"
)
else:
cameras_info.append(f" - {friendly_name} (ID: {camera_id})")
cameras_section = ""
if cameras_info:
cameras_section = (
"\n\nAvailable cameras:\n"
+ "\n".join(cameras_info)
+ "\n\nWhen users refer to cameras by their friendly name (e.g., 'Back Deck Camera'), use the corresponding camera ID (e.g., 'back_deck_cam') in tool calls."
)
speed_units_section = ""
if has_speed_zone:
speed_unit = (
"mph" if config.ui.unit_system == UnitSystemEnum.imperial else "km/h"
)
speed_units_section = f"\n\nReport object speeds to the user in {speed_unit}."
semantic_search_section = ""
if semantic_search_enabled:
semantic_search_section = (
"\n\nWhen routing a search_objects call, pick filters by the shape of the user's request:\n"
"- Generic class ('show me all cars today'): set `label` only.\n"
"- Specific named entity — a known person ('John'), delivery company ('Amazon'), animal species/breed ('blue jay', 'cardinal', 'golden retriever'), or license plate: set `sub_label` only and leave `label` unset.\n"
"- Physical characteristic, appearance, or activity that is NOT a discrete name ('find me people riding a lawn mower', 'someone in a red jacket', 'a person carrying a package'): set `semantic_query` with the descriptive phrase, optionally combined with `label` for the object class. Never put descriptive phrases in `sub_label`."
)
attribute_classification_section = ""
if attribute_classifications:
model_lines = "\n".join(
f"- {m['name']}: applies to {', '.join(m['objects']) or 'any object'}"
for m in attribute_classifications
)
attribute_classification_section = (
"\n\nAttribute classification models are configured for the following object types:\n"
f"{model_lines}\n"
"When the user's request matches one of these classifications, set the search_objects `attribute` field to the matching label rather than using `semantic_query`. Reserve `semantic_query` for descriptive phrases that fall outside the configured attribute labels."
)
return f"""You are a helpful assistant for Frigate, a security camera NVR system. You help users answer questions about their cameras, detected objects, and events.
Current server local date and time: {current_date_str} at {current_time_str}
Do not start your response with phrases like "I will check...", "Let me see...", or "Let me look...". Answer directly.
Always present times to the user in the server's local timezone. When tool results include start_time_local and end_time_local, use those exact strings when listing or describing detection times—do not convert or invent timestamps. Do not use UTC or ISO format with Z for the user-facing answer unless the tool result only provides Unix timestamps without local time fields.
When users ask about "today", "yesterday", "this week", etc., use the current date above as reference.
When searching for objects or events, use ISO 8601 format for dates (e.g., {current_date_str}T00:00:00Z for the start of today).
Always be accurate with time calculations based on the current date provided.
When a user refers to a specific object they have seen or describe with identifying details ("that green car", "the person in the red jacket", "a package left today"), prefer the find_similar_objects tool over search_objects. Use search_objects first only to locate the anchor event, then pass its id to find_similar_objects. For generic queries like "show me all cars today", keep using search_objects. If a user message begins with [attached_event:<id>], treat that event id as the anchor for any similarity or "tell me more" request in the same message and call find_similar_objects with that id.{semantic_search_section}{attribute_classification_section}{cameras_section}{speed_units_section}"""

8
frigate/genai/utils.py
View File

@ -69,6 +69,14 @@ def build_assistant_message_for_conversation(
"name": tc["name"],
"arguments": json.dumps(tc.get("arguments") or {}),
},
# Gemini-only: opaque signature that must be echoed back on
# the same functionCall part in the next turn. Other providers
# do not set or read this.
**(
{"thought_signature": tc["thought_signature"]}
if tc.get("thought_signature")
else {}
),
}
for tc in tool_calls_raw
]

185
frigate/jobs/debug_replay.py
View File

@ -1,4 +1,4 @@
"""Debug replay startup job: ffmpeg concat + camera config publish.
"""Debug replay startup job: ffmpeg remux + camera config publish.
The runner orchestrates the async portion of starting a debug replay
session. The DebugReplayManager (in frigate.debug_replay) owns session
@ -12,6 +12,7 @@ import os
import subprocess as sp
import threading
import time
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Optional, cast
@ -23,7 +24,7 @@ from frigate.const import REPLAY_CAMERA_PREFIX, REPLAY_DIR
from frigate.jobs.export import JobStatePublisher
from frigate.jobs.job import Job
from frigate.jobs.manager import job_is_running, set_current_job
from frigate.models import Recordings
from frigate.models import Export, Recordings
from frigate.types import JobStatusTypesEnum
from frigate.util.ffmpeg import run_ffmpeg_with_progress
@ -114,6 +115,130 @@ def query_recordings(source_camera: str, start_ts: float, end_ts: float) -> Mode
return cast(ModelSelect, query)
class DebugReplaySource(ABC):
"""Abstract source for a debug replay session.
Provides the camera identity and time range the replay represents,
validates that usable content exists, and supplies the ffmpeg input
args used to build the replay clip.
"""
@property
@abstractmethod
def source_camera(self) -> str:
"""Camera name the replay is derived from."""
@property
@abstractmethod
def start_ts(self) -> float:
"""Unix timestamp marking the start of the replay range."""
@property
@abstractmethod
def end_ts(self) -> float:
"""Unix timestamp marking the end of the replay range."""
@abstractmethod
def validate(self) -> None:
"""Raise ValueError if the source has no usable content."""
@abstractmethod
def ffmpeg_input_args(self, working_dir: str) -> list[str]:
"""Return ffmpeg input args (including -i). May write temp files in working_dir."""
def cleanup(self, working_dir: str) -> None:
"""Remove any temp files the source created in working_dir. Default no-op."""
class RecordingDebugReplaySource(DebugReplaySource):
"""Replay source backed by the Recordings table.
Feeds ffmpeg the internal VOD endpoint so segments with mismatched
SPS/PPS (e.g. across day/night transitions) stitch cleanly via HLS
discontinuities.
"""
def __init__(
self,
source_camera: str,
start_ts: float,
end_ts: float,
internal_port: int,
) -> None:
self._camera = source_camera
self._start_ts = start_ts
self._end_ts = end_ts
self._internal_port = internal_port
@property
def source_camera(self) -> str:
return self._camera
@property
def start_ts(self) -> float:
return self._start_ts
@property
def end_ts(self) -> float:
return self._end_ts
def validate(self) -> None:
if self._end_ts <= self._start_ts:
raise ValueError("End time must be after start time")
if not query_recordings(self._camera, self._start_ts, self._end_ts).count():
raise ValueError(
f"No recordings found for camera '{self._camera}' in the specified time range"
)
def ffmpeg_input_args(self, working_dir: str) -> list[str]:
playlist_url = (
f"http://127.0.0.1:{self._internal_port}/vod/{self._camera}"
f"/start/{self._start_ts}/end/{self._end_ts}/index.m3u8"
)
return [
"-protocol_whitelist",
"pipe,file,http,tcp",
"-i",
playlist_url,
]
class ExportDebugReplaySource(DebugReplaySource):
"""Replay source backed by an existing Export.
Uses the export's video file directly as the ffmpeg input — does not
require recordings to still exist for the time range.
"""
def __init__(self, export: Export, duration: float) -> None:
self._camera = cast(str, export.camera)
# Export.date is declared DateTimeField but Frigate writes raw unix
# timestamps to the column.
self._start_ts = float(cast(Any, export.date))
self._video_path = cast(str, export.video_path)
self._duration = duration
@property
def source_camera(self) -> str:
return self._camera
@property
def start_ts(self) -> float:
return self._start_ts
@property
def end_ts(self) -> float:
return self._start_ts + self._duration
def validate(self) -> None:
if not os.path.exists(self._video_path):
raise ValueError(f"Export video file not found: {self._video_path}")
def ffmpeg_input_args(self, working_dir: str) -> list[str]:
return ["-i", self._video_path]
class DebugReplayJobRunner(threading.Thread):
"""Worker thread that drives the startup job to completion.
@ -126,6 +251,7 @@ class DebugReplayJobRunner(threading.Thread):
def __init__(
self,
job: DebugReplayJob,
source: DebugReplaySource,
frigate_config: FrigateConfig,
config_publisher: CameraConfigUpdatePublisher,
replay_manager: "DebugReplayManager",
@ -133,6 +259,7 @@ class DebugReplayJobRunner(threading.Thread):
) -> None:
super().__init__(daemon=True, name=f"debug_replay_{job.id}")
self.job = job
self.source = source
self.frigate_config = frigate_config
self.config_publisher = config_publisher
self.replay_manager = replay_manager
@ -183,7 +310,6 @@ class DebugReplayJobRunner(threading.Thread):
def run(self) -> None:
replay_name = self.job.replay_camera_name
os.makedirs(REPLAY_DIR, exist_ok=True)
concat_file = os.path.join(REPLAY_DIR, f"{replay_name}_concat.txt")
clip_path = os.path.join(REPLAY_DIR, f"{replay_name}.mp4")
self.job.status = JobStatusTypesEnum.running
@ -192,23 +318,13 @@ class DebugReplayJobRunner(threading.Thread):
self._broadcast(force=True)
try:
recordings = query_recordings(
self.job.source_camera, self.job.start_ts, self.job.end_ts
)
with open(concat_file, "w") as f:
for recording in recordings:
f.write(f"file '{recording.path}'\n")
input_args = self.source.ffmpeg_input_args(REPLAY_DIR)
ffmpeg_cmd = [
self.frigate_config.ffmpeg.ffmpeg_path,
"-hide_banner",
"-y",
"-f",
"concat",
"-safe",
"0",
"-i",
concat_file,
*input_args,
"-c",
"copy",
"-movflags",
@ -285,7 +401,7 @@ class DebugReplayJobRunner(threading.Thread):
self.replay_manager.clear_session()
_remove_silent(clip_path)
finally:
_remove_silent(concat_file)
self.source.cleanup(REPLAY_DIR)
_set_active_runner(None)
def _finalize_cancelled(self, clip_path: str) -> None:
@ -309,52 +425,43 @@ def _remove_silent(path: str) -> None:
def start_debug_replay_job(
*,
source_camera: str,
start_ts: float,
end_ts: float,
source: DebugReplaySource,
frigate_config: FrigateConfig,
config_publisher: CameraConfigUpdatePublisher,
replay_manager: "DebugReplayManager",
) -> str:
"""Validate, create job, start runner. Returns the job id.
Raises ValueError for bad params (camera missing, time range
invalid, no recordings) and RuntimeError if a session is already
active.
Raises ValueError for an invalid source (camera missing, source has
no usable content) and RuntimeError if a session is already active.
"""
if job_is_running(JOB_TYPE) or replay_manager.active:
raise RuntimeError("A replay session is already active")
if source_camera not in frigate_config.cameras:
raise ValueError(f"Camera '{source_camera}' not found")
if source.source_camera not in frigate_config.cameras:
raise ValueError(f"Camera '{source.source_camera}' not found")
if end_ts <= start_ts:
raise ValueError("End time must be after start time")
source.validate()
recordings = query_recordings(source_camera, start_ts, end_ts)
if not recordings.count():
raise ValueError(
f"No recordings found for camera '{source_camera}' in the specified time range"
)
replay_name = f"{REPLAY_CAMERA_PREFIX}{source_camera}"
replay_name = f"{REPLAY_CAMERA_PREFIX}{source.source_camera}"
replay_manager.mark_starting(
source_camera=source_camera,
source_camera=source.source_camera,
replay_camera_name=replay_name,
start_ts=start_ts,
end_ts=end_ts,
start_ts=source.start_ts,
end_ts=source.end_ts,
)
job = DebugReplayJob(
source_camera=source_camera,
source_camera=source.source_camera,
replay_camera_name=replay_name,
start_ts=start_ts,
end_ts=end_ts,
start_ts=source.start_ts,
end_ts=source.end_ts,
)
set_current_job(job)
runner = DebugReplayJobRunner(
job=job,
source=source,
frigate_config=frigate_config,
config_publisher=config_publisher,
replay_manager=replay_manager,

734
frigate/jobs/motion_search.py
View File

@ -3,6 +3,8 @@
import logging
import os
import threading
import time
from collections.abc import Callable, Generator, Iterable
from concurrent.futures import Future, ThreadPoolExecutor, as_completed
from dataclasses import asdict, dataclass, field
from datetime import datetime
@ -19,6 +21,18 @@ from frigate.jobs.manager import (
get_job_by_id,
set_current_job,
)
from frigate.jobs.motion_search_batch import (
build_segment_time_map,
coalesce_runs,
stream_time_to_absolute,
)
from frigate.jobs.motion_search_decode import (
iter_vod_frames,
keyframe_sampling_eligible,
probe_video_dimensions,
probe_vod_keyframe_pts,
resolve_motion_decode_args,
)
from frigate.models import Recordings
from frigate.types import JobStatusTypesEnum
@ -26,6 +40,18 @@ logger = logging.getLogger(__name__)
# Constants
HEATMAP_GRID_SIZE = 16
# Max wall-clock span of one VOD run request (seconds). Bounds per-request size
# and gives streaming/cancel/early-exit granularity.
MAX_RUN_SECONDS = 600.0
# Treat segments within this many seconds end-to-start as time-contiguous.
RUN_GAP_EPSILON = 1.0
# Longest-side pixels for the ROI downscale before motion detection.
SCALE_TARGET = 400
# Minimum wall seconds between intra-run progress broadcasts.
PROGRESS_BROADCAST_INTERVAL = 1.0
# Output frame rate for the fixed-cadence fallback used on long-GOP cameras
# (where keyframe sampling is too sparse). Keyframe cameras ignore this.
FALLBACK_SAMPLE_FPS = 2.0
@dataclass
@ -69,13 +95,16 @@ class MotionSearchJob(Job):
polygon_points: list[list[float]] = field(default_factory=list)
threshold: int = 30
min_area: float = 5.0
frame_skip: int = 5
parallel: bool = False
max_results: int = 25
# Track progress
total_frames_processed: int = 0
# Live progress (ride the existing to_dict() websocket broadcast)
scanning_timestamp: Optional[float] = None
progress: float = 0.0
# Metrics for observability
metrics: Optional[MotionSearchMetrics] = None
@ -100,6 +129,113 @@ def create_polygon_mask(
return mask
def compute_roi_crop_and_scale(
polygon_points: list[list[float]],
frame_width: int,
frame_height: int,
scale_target: int,
) -> tuple[tuple[int, int, int, int], tuple[int, int]]:
"""Compute the ROI crop box and never-upscale scaled dimensions.
Returns ((crop_w, crop_h, crop_x, crop_y), (scaled_w, scaled_h)) in pixels.
The crop is the polygon's bounding box in frame pixels; the scaled size fits
the crop's longest side to ``scale_target`` without ever enlarging it.
"""
xs = [p[0] for p in polygon_points]
ys = [p[1] for p in polygon_points]
# nv12 (4:2:0) hwdownload requires even crop offsets and even crop/scale
# dimensions; otherwise ffmpeg rounds the chroma planes and the raw byte
# stream stops matching the expected frame size. Force even values, and the
# mask is built from these same values so the two stay aligned.
crop_x = int(min(xs) * frame_width)
crop_y = int(min(ys) * frame_height)
crop_x -= crop_x % 2
crop_y -= crop_y % 2
crop_w = max(2, int(max(xs) * frame_width) - crop_x)
crop_h = max(2, int(max(ys) * frame_height) - crop_y)
crop_w -= crop_w % 2
crop_h -= crop_h % 2
longest = max(crop_w, crop_h)
factor = min(1.0, scale_target / longest)
scaled_w = max(2, round(crop_w * factor))
scaled_h = max(2, round(crop_h * factor))
scaled_w -= scaled_w % 2
scaled_h -= scaled_h % 2
return (crop_w, crop_h, crop_x, crop_y), (scaled_w, scaled_h)
def build_scaled_roi_mask(
polygon_points: list[list[float]],
frame_width: int,
frame_height: int,
crop: tuple[int, int, int, int],
scaled: tuple[int, int],
) -> np.ndarray:
"""Rasterize the polygon mask at the scaled ROI size.
Builds the full-resolution mask, crops it to the ROI box, and nearest-
neighbor resizes it to the scaled dimensions so it lines up exactly with the
frames ffmpeg crops and scales.
"""
crop_w, crop_h, crop_x, crop_y = crop
scaled_w, scaled_h = scaled
full_mask = create_polygon_mask(polygon_points, frame_width, frame_height)
cropped = full_mask[crop_y : crop_y + crop_h, crop_x : crop_x + crop_w]
return cv2.resize(cropped, (scaled_w, scaled_h), interpolation=cv2.INTER_NEAREST)
def detect_motion_scaled(
frames: Iterable[tuple[int, np.ndarray]],
mask: np.ndarray,
threshold: int,
min_area: float,
timestamp_fn: Callable[[int], float],
) -> list[MotionSearchResult]:
"""Detect motion across pre-cropped, pre-scaled gray frames.
``frames`` yields (absolute_frame_index, gray_roi_frame); ``mask`` is the
scaled ROI mask. ``min_area`` is a percentage of the masked ROI. Mirrors the
full-res detection math (absdiff -> blur -> threshold -> dilate -> contours)
on the already-reduced frames.
"""
results: list[MotionSearchResult] = []
mask_area = np.count_nonzero(mask)
if mask_area == 0:
return results
min_area_pixels = int((min_area / 100.0) * mask_area)
prev: np.ndarray | None = None
for frame_idx, gray in frames:
masked = cv2.bitwise_and(gray, gray, mask=mask)
if prev is not None:
diff = cv2.absdiff(prev, masked)
diff_blurred = cv2.GaussianBlur(diff, (3, 3), 0)
_, thresh = cv2.threshold(diff_blurred, threshold, 255, cv2.THRESH_BINARY)
thresh_dilated = cv2.dilate(thresh, None, iterations=1) # type: ignore[call-overload]
thresh_masked = cv2.bitwise_and(thresh_dilated, thresh_dilated, mask=mask)
change_pixels = cv2.countNonZero(thresh_masked)
if change_pixels > min_area_pixels:
contours, _ = cv2.findContours(
thresh_masked, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
)
total_change_area = sum(
cv2.contourArea(c)
for c in contours
if cv2.contourArea(c) >= min_area_pixels
)
if total_change_area > 0:
change_percentage = (total_change_area / mask_area) * 100
results.append(
MotionSearchResult(
timestamp=timestamp_fn(frame_idx),
change_percentage=round(change_percentage, 2),
)
)
prev = masked
return results
def compute_roi_bbox_normalized(
polygon_points: list[list[float]],
) -> tuple[float, float, float, float]:
@ -184,6 +320,22 @@ def segment_passes_heatmap_gate(
return heatmap_overlaps_roi(heatmap, roi_bbox)
def resolve_internal_port(config: FrigateConfig) -> int:
"""Return the unauthenticated internal nginx port for VOD requests."""
listen = config.networking.listen.internal
if isinstance(listen, str):
return int(listen.split(":")[-1])
return int(listen)
def build_vod_url(internal_port: int, camera: str, start: float, end: float) -> str:
"""Build the internal VOD HLS URL for a camera time range."""
return (
f"http://127.0.0.1:{internal_port}/vod/{camera}"
f"/start/{start}/end/{end}/index.m3u8"
)
class MotionSearchRunner(threading.Thread):
"""Thread-based runner for motion search jobs with parallel verification."""
@ -206,6 +358,23 @@ class MotionSearchRunner(threading.Thread):
cpu_count = os.cpu_count() or 1
self.max_workers = min(4, cpu_count)
# Resolved once per job in _execute_search
self.ffmpeg_path: str = "ffmpeg"
self.ffprobe_path: str = "ffprobe"
self.decode_args: list[str] = []
# Keyframe sampling decision, decided once per job from the first run's
# GOP. The fallback cadence is a fixed rate (see FALLBACK_SAMPLE_FPS).
self.use_keyframe: bool = True
self.fps_rate: float = FALLBACK_SAMPLE_FPS
# ROI crop/scale + scaled mask, computed once from the VOD-stream
# dimensions (which can differ from the detect resolution).
self.crop: tuple[int, int, int, int] = (0, 0, 0, 0)
self.scaled: tuple[int, int] = (0, 0)
self.scaled_mask: np.ndarray = np.zeros((0, 0), dtype=np.uint8)
self.channels: int = 1
self.internal_port: int = 5000
self._last_progress_broadcast: float = 0.0
def run(self) -> None:
"""Execute the motion search job."""
try:
@ -281,6 +450,9 @@ class MotionSearchRunner(threading.Thread):
if frame_width is None or frame_height is None:
raise ValueError(f"Camera {camera_name} detect dimensions not configured")
self.ffmpeg_path = camera_config.ffmpeg.ffmpeg_path
self.ffprobe_path = camera_config.ffmpeg.ffprobe_path
# Create polygon mask
polygon_mask = create_polygon_mask(
self.job.polygon_points, frame_width, frame_height
@ -384,205 +556,274 @@ class MotionSearchRunner(threading.Thread):
self.metrics.heatmap_roi_skip_segments,
)
if self.job.parallel:
return self._search_motion_parallel(filtered_recordings, polygon_mask)
# Resolve decode backend (allowlisted hwaccel or software), coalesce the
# gate-passing segments into time-contiguous runs, and probe the first
# run's VOD stream once for dimensions + keyframe layout. VOD output is
# what we decode, so crop/scale/mask are computed against it.
self.internal_port = resolve_internal_port(self.config)
self.decode_args = resolve_motion_decode_args(camera_config)
ffprobe_path = self.ffprobe_path
return self._search_motion_sequential(filtered_recordings, polygon_mask)
runs = coalesce_runs(filtered_recordings, MAX_RUN_SECONDS, RUN_GAP_EPSILON)
if not runs:
return []
def _search_motion_parallel(
self,
recordings: list[Recordings],
polygon_mask: np.ndarray,
) -> list[MotionSearchResult]:
"""Search for motion in parallel across segments, streaming results."""
all_results: list[MotionSearchResult] = []
total_frames = 0
next_recording_idx_to_merge = 0
first_run = runs[0]
first_url = build_vod_url(
self.internal_port,
camera_name,
float(first_run[0].start_time),
float(first_run[-1].end_time),
)
dims = probe_video_dimensions(ffprobe_path, first_url)
if dims is None:
raise ValueError(f"Could not probe VOD dimensions for camera {camera_name}")
rec_width, rec_height, _rec_fps = dims
self.crop, self.scaled = compute_roi_crop_and_scale(
self.job.polygon_points, rec_width, rec_height, SCALE_TARGET
)
self.scaled_mask = build_scaled_roi_mask(
self.job.polygon_points, rec_width, rec_height, self.crop, self.scaled
)
self.channels = 1 # always gray output
# Decide keyframe vs fixed-cadence sampling once from the first run's GOP
# (keyframe structure is a per-camera constant).
first_pts = probe_vod_keyframe_pts(ffprobe_path, first_url)
self.use_keyframe = keyframe_sampling_eligible(first_pts)
logger.debug(
"Motion search job %s: starting motion search with %d workers "
"across %d segments",
"Motion search job %s: %d runs, sampling=%s, hwaccel=%s, vod=%dx%d",
self.job.id,
self.max_workers,
len(recordings),
len(runs),
"keyframe" if self.use_keyframe else "cadence",
bool(self.decode_args),
rec_width,
rec_height,
)
# Initialize partial results on the job so they stream to the frontend
self.job.results = {"results": [], "total_frames_processed": 0}
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
futures: dict[Future, int] = {}
completed_segments: dict[int, tuple[list[MotionSearchResult], int]] = {}
for idx, recording in enumerate(recordings):
if self._should_stop():
break
rec_start: float = recording.start_time # type: ignore[assignment]
rec_end: float = recording.end_time # type: ignore[assignment]
future = executor.submit(
self._process_recording_for_motion,
str(recording.path),
rec_start,
rec_end,
self.job.start_time_range,
self.job.end_time_range,
polygon_mask,
self.job.threshold,
self.job.min_area,
self.job.frame_skip,
)
futures[future] = idx
for future in as_completed(futures):
if self._should_stop():
# Cancel remaining futures
for f in futures:
f.cancel()
break
recording_idx = futures[future]
recording = recordings[recording_idx]
try:
results, frames = future.result()
self.metrics.segments_processed += 1
completed_segments[recording_idx] = (results, frames)
while next_recording_idx_to_merge in completed_segments:
segment_results, segment_frames = completed_segments.pop(
next_recording_idx_to_merge
)
all_results.extend(segment_results)
total_frames += segment_frames
self.job.total_frames_processed = total_frames
self.metrics.frames_decoded = total_frames
if segment_results:
deduped = self._deduplicate_results(all_results)
self.job.results = {
"results": [
r.to_dict() for r in deduped[: self.job.max_results]
],
"total_frames_processed": total_frames,
}
return self._search_runs(runs)
def _emit_progress(self, abs_ts: float) -> None:
"""Throttled intra-run progress broadcast (scanning cursor)."""
now = time.monotonic()
if now - self._last_progress_broadcast < PROGRESS_BROADCAST_INTERVAL:
return
self._last_progress_broadcast = now
self.job.scanning_timestamp = abs_ts
self._broadcast_status()
if segment_results and len(deduped) >= self.job.max_results:
self.internal_stop_event.set()
for pending_future in futures:
pending_future.cancel()
break
next_recording_idx_to_merge += 1
if self.internal_stop_event.is_set():
break
except Exception as e:
self.metrics.segments_processed += 1
self.metrics.segments_with_errors += 1
self._broadcast_status()
logger.warning(
"Error processing segment %s: %s",
recording.path,
e,
)
self.job.total_frames_processed = total_frames
self.metrics.frames_decoded = total_frames
logger.debug(
"Motion search job %s: motion search complete, "
"found %d raw results, decoded %d frames, %d segment errors",
self.job.id,
len(all_results),
total_frames,
self.metrics.segments_with_errors,
)
# Sort and deduplicate results
all_results.sort(key=lambda x: x.timestamp)
return self._deduplicate_results(all_results)[: self.job.max_results]
def _search_motion_sequential(
def _detect_with_progress(
self,
recordings: list[Recordings],
polygon_mask: np.ndarray,
indexed_frames: list[tuple[int, np.ndarray]],
timestamp_fn: Callable[[int], float],
) -> list[MotionSearchResult]:
"""Search for motion sequentially across segments, streaming results."""
all_results: list[MotionSearchResult] = []
total_frames = 0
"""Run detection while firing throttled progress as frames are scanned."""
logger.debug(
"Motion search job %s: starting sequential motion search across %d segments",
self.job.id,
len(recordings),
)
def _gen() -> Generator[tuple[int, np.ndarray], None, None]:
for i, frame in indexed_frames:
if not self._should_stop():
self._emit_progress(timestamp_fn(i))
yield i, frame
self.job.results = {"results": [], "total_frames_processed": 0}
for recording in recordings:
if self.cancel_event.is_set():
break
try:
rec_start: float = recording.start_time # type: ignore[assignment]
rec_end: float = recording.end_time # type: ignore[assignment]
results, frames = self._process_recording_for_motion(
str(recording.path),
rec_start,
rec_end,
self.job.start_time_range,
self.job.end_time_range,
polygon_mask,
return detect_motion_scaled(
_gen(),
self.scaled_mask,
self.job.threshold,
self.job.min_area,
self.job.frame_skip,
timestamp_fn,
)
all_results.extend(results)
total_frames += frames
self.job.total_frames_processed = total_frames
self.metrics.frames_decoded = total_frames
self.metrics.segments_processed += 1
def _process_run(
self, run: list[Recordings]
) -> tuple[list[MotionSearchResult], int]:
"""Decode one run's VOD stream and detect motion.
if results:
all_results.sort(key=lambda x: x.timestamp)
deduped = self._deduplicate_results(all_results)[
Keyframe mode compares every decoded keyframe (free recall, since they
are all decoded anyway) paired with its probed PTS; if the decoded and
probed counts disagree (the decoder ignored ``-skip_frame nokey`` or the
stream is corrupt) this run re-runs in the fixed-cadence fallback.
Returns ``(results, frame_count)``.
"""
run_start: float = run[0].start_time # type: ignore[assignment]
run_end: float = run[-1].end_time # type: ignore[assignment]
vod_url = build_vod_url(self.internal_port, self.job.camera, run_start, run_end)
time_map = build_segment_time_map(run)
if self.use_keyframe:
kf_pts = probe_vod_keyframe_pts(self.ffprobe_path, vod_url)
frames = list(
iter_vod_frames(
self.ffmpeg_path,
vod_url,
self.scaled[0],
self.scaled[1],
self.channels,
self.decode_args,
self.crop,
self.scaled,
True,
self._should_stop,
skip_nonkey=True,
fps_rate=None,
)
)
if kf_pts and len(frames) == len(kf_pts):
abs_times = [stream_time_to_absolute(time_map, p) for p in kf_pts]
indexed = list(enumerate(frames))
def _ts_kf(i: int) -> float:
return abs_times[i]
results = self._detect_with_progress(indexed, _ts_kf)
return results, len(frames)
logger.debug(
"Keyframe count mismatch (%d decoded vs %d probed), using cadence",
len(frames),
len(kf_pts),
)
return self._process_run_cadence(vod_url, time_map)
def _process_run_cadence(
self, vod_url: str, time_map: list[tuple[float, float, float]]
) -> tuple[list[MotionSearchResult], int]:
"""Fixed-cadence fallback: fps-filtered VOD decode, evenly spaced times."""
frames = list(
iter_vod_frames(
self.ffmpeg_path,
vod_url,
self.scaled[0],
self.scaled[1],
self.channels,
self.decode_args,
self.crop,
self.scaled,
True,
self._should_stop,
skip_nonkey=False,
fps_rate=self.fps_rate,
)
)
indexed = list(enumerate(frames))
def _ts_fps(i: int) -> float:
return stream_time_to_absolute(time_map, i / self.fps_rate)
results = self._detect_with_progress(indexed, _ts_fps)
return results, len(frames)
def _merge_run(
self,
run: list[Recordings],
run_results: list[MotionSearchResult],
frames: int,
state: dict[str, Any],
) -> bool:
"""Fold one run's output into the running results; stream + dedup.
Returns True once ``max_results`` deduped hits have accumulated.
"""
state["completed_runs"] += 1
state["all_results"].extend(run_results)
state["total_frames"] += frames
self.job.total_frames_processed = state["total_frames"]
self.metrics.frames_decoded = state["total_frames"]
self.metrics.segments_processed += len(run)
self.job.progress = state["completed_runs"] / state["total_runs"]
state["all_results"].sort(key=lambda r: r.timestamp)
deduped = self._deduplicate_results(state["all_results"])[
: self.job.max_results
]
self.job.results = {
"results": [r.to_dict() for r in deduped],
"total_frames_processed": total_frames,
"total_frames_processed": state["total_frames"],
}
self._broadcast_status()
return len(deduped) >= self.job.max_results
if results and len(deduped) >= self.job.max_results:
break
except Exception as e:
self.metrics.segments_processed += 1
self.metrics.segments_with_errors += 1
self._broadcast_status()
logger.warning("Error processing segment %s: %s", recording.path, e)
self.job.total_frames_processed = total_frames
self.metrics.frames_decoded = total_frames
def _search_runs(self, runs: list[list[Recordings]]) -> list[MotionSearchResult]:
"""Decode runs (parallel pool when enabled), merge in order, stream."""
state: dict[str, Any] = {
"all_results": [],
"total_frames": 0,
"completed_runs": 0,
"total_runs": len(runs),
}
self.job.results = {"results": [], "total_frames_processed": 0}
logger.debug(
"Motion search job %s: sequential motion search complete, "
"found %d raw results, decoded %d frames, %d segment errors",
"Motion search job %s: searching %d runs (parallel=%s, workers=%d)",
self.job.id,
len(runs),
self.job.parallel,
self.max_workers,
)
if self.job.parallel and len(runs) > 1:
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
futures: dict[Future, int] = {}
for idx, run in enumerate(runs):
if self._should_stop():
break
futures[executor.submit(self._process_run, run)] = idx
completed: dict[int, tuple[list[MotionSearchResult], int]] = {}
next_idx = 0
for future in as_completed(futures):
if self._should_stop():
break
run_idx = futures[future]
try:
completed[run_idx] = future.result()
except Exception as e:
self.metrics.segments_with_errors += 1
logger.warning("Error processing run %d: %s", run_idx, e)
completed[run_idx] = ([], 0)
while next_idx in completed:
run_results, frames = completed.pop(next_idx)
if self._merge_run(runs[next_idx], run_results, frames, state):
self.internal_stop_event.set()
for pending in futures:
pending.cancel()
break
next_idx += 1
if self.internal_stop_event.is_set():
break
else:
for run in runs:
if self._should_stop():
break
try:
run_results, frames = self._process_run(run)
except Exception as e:
self.metrics.segments_with_errors += 1
self.metrics.segments_processed += len(run)
self._broadcast_status()
logger.warning("Error processing run: %s", e)
continue
if self._merge_run(run, run_results, frames, state):
break
all_results: list[MotionSearchResult] = state["all_results"]
self.job.total_frames_processed = state["total_frames"]
self.metrics.frames_decoded = state["total_frames"]
self.job.progress = 1.0
logger.debug(
"Motion search job %s: complete, %d raw results, %d frames, %d errors",
self.job.id,
len(all_results),
total_frames,
state["total_frames"],
self.metrics.segments_with_errors,
)
all_results.sort(key=lambda x: x.timestamp)
all_results.sort(key=lambda r: r.timestamp)
return self._deduplicate_results(all_results)[: self.job.max_results]
def _deduplicate_results(
@ -602,160 +843,6 @@ class MotionSearchRunner(threading.Thread):
return deduplicated
def _process_recording_for_motion(
self,
recording_path: str,
recording_start: float,
recording_end: float,
search_start: float,
search_end: float,
polygon_mask: np.ndarray,
threshold: int,
min_area: float,
frame_skip: int,
) -> tuple[list[MotionSearchResult], int]:
"""Process a single recording file for motion detection.
This method is designed to be called from a thread pool.
Args:
min_area: Minimum change area as a percentage of the ROI (0-100).
"""
results: list[MotionSearchResult] = []
frames_processed = 0
if not os.path.exists(recording_path):
logger.warning("Recording file not found: %s", recording_path)
return results, frames_processed
cap = cv2.VideoCapture(recording_path)
if not cap.isOpened():
logger.error("Could not open recording: %s", recording_path)
return results, frames_processed
try:
fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
recording_duration = recording_end - recording_start
# Calculate frame range
start_offset = max(0, search_start - recording_start)
end_offset = min(recording_duration, search_end - recording_start)
start_frame = int(start_offset * fps)
end_frame = int(end_offset * fps)
start_frame = max(0, min(start_frame, total_frames - 1))
end_frame = max(0, min(end_frame, total_frames))
if start_frame >= end_frame:
return results, frames_processed
cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame)
# Get ROI bounding box
roi_bbox = cv2.boundingRect(polygon_mask)
roi_x, roi_y, roi_w, roi_h = roi_bbox
prev_frame_gray = None
frame_step = max(frame_skip, 1)
frame_idx = start_frame
while frame_idx < end_frame:
if self._should_stop():
break
ret, frame = cap.read()
if not ret:
frame_idx += 1
continue
if (frame_idx - start_frame) % frame_step != 0:
frame_idx += 1
continue
frames_processed += 1
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Handle frame dimension changes
if gray.shape != polygon_mask.shape:
resized_mask = cv2.resize(
polygon_mask,
(gray.shape[1], gray.shape[0]),
interpolation=cv2.INTER_NEAREST,
)
current_bbox = cv2.boundingRect(resized_mask)
else:
resized_mask = polygon_mask
current_bbox = roi_bbox
roi_x, roi_y, roi_w, roi_h = current_bbox
cropped_gray = gray[roi_y : roi_y + roi_h, roi_x : roi_x + roi_w]
cropped_mask = resized_mask[
roi_y : roi_y + roi_h, roi_x : roi_x + roi_w
]
cropped_mask_area = np.count_nonzero(cropped_mask)
if cropped_mask_area == 0:
frame_idx += 1
continue
# Convert percentage to pixel count for this ROI
min_area_pixels = int((min_area / 100.0) * cropped_mask_area)
masked_gray = cv2.bitwise_and(
cropped_gray, cropped_gray, mask=cropped_mask
)
if prev_frame_gray is not None:
diff = cv2.absdiff(prev_frame_gray, masked_gray) # type: ignore[unreachable]
diff_blurred = cv2.GaussianBlur(diff, (3, 3), 0)
_, thresh = cv2.threshold(
diff_blurred, threshold, 255, cv2.THRESH_BINARY
)
thresh_dilated = cv2.dilate(thresh, None, iterations=1)
thresh_masked = cv2.bitwise_and(
thresh_dilated, thresh_dilated, mask=cropped_mask
)
change_pixels = cv2.countNonZero(thresh_masked)
if change_pixels > min_area_pixels:
contours, _ = cv2.findContours(
thresh_masked, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
)
total_change_area = sum(
cv2.contourArea(c)
for c in contours
if cv2.contourArea(c) >= min_area_pixels
)
if total_change_area > 0:
frame_time_offset = (frame_idx - start_frame) / fps
timestamp = (
recording_start + start_offset + frame_time_offset
)
change_percentage = (
total_change_area / cropped_mask_area
) * 100
results.append(
MotionSearchResult(
timestamp=timestamp,
change_percentage=round(change_percentage, 2),
)
)
prev_frame_gray = masked_gray
frame_idx += 1
finally:
cap.release()
logger.debug(
"Motion search segment complete: %s, %d frames processed, %d results found",
recording_path,
frames_processed,
len(results),
)
return results, frames_processed
# Module-level state for managing per-camera jobs
_motion_search_jobs: dict[str, tuple[MotionSearchJob, threading.Event]] = {}
@ -779,7 +866,6 @@ def start_motion_search_job(
polygon_points: list[list[float]],
threshold: int = 30,
min_area: float = 5.0,
frame_skip: int = 5,
parallel: bool = False,
max_results: int = 25,
) -> str:
@ -794,7 +880,6 @@ def start_motion_search_job(
polygon_points=polygon_points,
threshold=threshold,
min_area=min_area,
frame_skip=frame_skip,
parallel=parallel,
max_results=max_results,
)
@ -812,14 +897,13 @@ def start_motion_search_job(
logger.debug(
"Started motion search job %s for camera %s: "
"time_range=%.1f-%.1f, threshold=%d, min_area=%.1f%%, "
"frame_skip=%d, parallel=%s, max_results=%d, polygon_points=%d vertices",
"parallel=%s, max_results=%d, polygon_points=%d vertices",
job.id,
camera_name,
start_time,
end_time,
threshold,
min_area,
frame_skip,
parallel,
max_results,
len(polygon_points),

75
frigate/jobs/motion_search_batch.py Normal file
View File

@ -0,0 +1,75 @@
"""Pure helpers for VOD-batched motion search.
Coalescing gate-passing segments into time-contiguous runs, mapping a frame's
VOD stream time back to an absolute timestamp, and thinning sample times to a
target interval. No I/O or ffmpeg here so the tricky math stays unit-testable.
"""
from bisect import bisect_right
from typing import Any
def coalesce_runs(
segments: list[Any], max_seconds: float, epsilon: float
) -> list[list[Any]]:
"""Group gate-passing segments into time-contiguous runs.
A run extends while each segment's ``start_time`` is within ``epsilon`` of
the previous segment's ``end_time`` (no recording gap) and the run's total
span stays at or below ``max_seconds``. A gap or the cap starts a new run.
Each segment must expose ``start_time`` / ``end_time``.
"""
runs: list[list[Any]] = []
current: list[Any] = []
for seg in segments:
if not current:
current = [seg]
continue
prev_end = float(current[-1].end_time)
run_start = float(current[0].start_time)
contiguous = abs(float(seg.start_time) - prev_end) <= epsilon
within_cap = (float(seg.end_time) - run_start) <= max_seconds
if contiguous and within_cap:
current.append(seg)
else:
runs.append(current)
current = [seg]
if current:
runs.append(current)
return runs
def build_segment_time_map(
run: list[Any],
) -> list[tuple[float, float, float]]:
"""Build a (stream_offset, abs_start, duration) row per segment in a run.
``stream_offset`` is the segment's start in continuous VOD stream time (the
cumulative sum of preceding segment durations); ``abs_start`` is its absolute
``start_time``. Built from each segment's own duration; for a gap-free run
this makes stream time equal ``run_start + offset``.
"""
rows: list[tuple[float, float, float]] = []
offset = 0.0
for seg in run:
duration = float(seg.end_time) - float(seg.start_time)
rows.append((offset, float(seg.start_time), duration))
offset += duration
return rows
def stream_time_to_absolute(
time_map: list[tuple[float, float, float]], stream_time: float
) -> float:
"""Map a VOD stream time to an absolute timestamp via the run's table.
Binary-searches the segment whose stream range contains ``stream_time`` and
returns ``abs_start + (stream_time - stream_offset)``. Times past the last
segment map into the last segment (clamped at the run edge).
"""
offsets = [row[0] for row in time_map]
idx = bisect_right(offsets, stream_time) - 1
if idx < 0:
idx = 0
stream_offset, abs_start, _duration = time_map[idx]
return abs_start + (stream_time - stream_offset)

382
frigate/jobs/motion_search_decode.py Normal file
View File

@ -0,0 +1,382 @@
"""Hardware-accelerated ffmpeg decode for motion search.
Decodes a recording run's VOD/HLS stream with an ffmpeg subprocess, optionally
selecting only keyframes, and streams raw frames over a pipe for the motion
math. Output is the requested ``pix_fmt`` (gray or ``bgr24``) with optional
crop/scale applied in the filter graph so downstream pixels are unchanged.
"""
import json
import logging
import subprocess as sp
import tempfile
from collections.abc import Callable, Generator
from typing import IO
import numpy as np
from frigate.config import CameraConfig
from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_decode
from frigate.util.services import auto_detect_hwaccel
logger = logging.getLogger(__name__)
# Output-format surfaces that download cleanly to nv12 via the fixed
# ``hwdownload,format=nv12`` step the decode path appends. Other surfaces
# (drm_prime from rkmpp, vulkan, amf) need a different download step, so motion
# search decodes them in software to keep results byte-identical rather than risk
# a wrong-but-valid-sized frame the zero-frame fallback gate would not catch.
_NV12_OUTPUT_FORMATS = frozenset({"vaapi", "cuda", "qsv"})
def _hwaccel_output_format(decode_args: list[str]) -> str | None:
"""Return the ``-hwaccel_output_format`` value in ffmpeg args, or None."""
try:
idx = decode_args.index("-hwaccel_output_format")
except ValueError:
return None
return decode_args[idx + 1] if idx + 1 < len(decode_args) else None
def resolve_motion_decode_args(camera_config: CameraConfig) -> list[str]:
"""Resolve the ffmpeg hwaccel decode args for a camera's recordings.
``auto`` is resolved via ``auto_detect_hwaccel`` and the preset is expanded
by ``parse_preset_hardware_acceleration_decode`` (the same table the live
pipeline uses). Acceleration is kept only when the decoded surface downloads
cleanly to nv12 -- decided by reading ``-hwaccel_output_format`` back from the
resolved args rather than a separate preset allowlist that could drift from
``PRESETS_HW_ACCEL_DECODE``. Anything else (custom args, a software-only
preset, or an nv12-incompatible surface) returns an empty list, meaning
software decode, so results stay byte-identical.
"""
raw = camera_config.ffmpeg.hwaccel_args
preset = auto_detect_hwaccel() if raw == "auto" else raw
# Custom args (a list) decode in software so results stay byte-identical.
if not isinstance(preset, str):
return []
decode_args = parse_preset_hardware_acceleration_decode(
preset,
camera_config.detect.fps,
camera_config.detect.width or 0,
camera_config.detect.height or 0,
camera_config.ffmpeg.gpu,
)
if not decode_args:
return []
if _hwaccel_output_format(decode_args) not in _NV12_OUTPUT_FORMATS:
return []
return decode_args
def _read_exact(stream: IO[bytes], size: int) -> bytes | None:
"""Read exactly ``size`` bytes from a pipe, or None at clean EOF.
Pipe reads can return fewer bytes than requested, so loop until the frame
is complete. A short read at the start of a frame means end-of-stream.
"""
buf = bytearray()
while len(buf) < size:
chunk = stream.read(size - len(buf))
if not chunk:
return None
buf.extend(chunk)
return bytes(buf)
def _terminate(proc: sp.Popen[bytes]) -> None:
"""Stop an ffmpeg decode process promptly."""
# Close the read end first so a blocked ffmpeg write unblocks (ffmpeg then
# sees a broken pipe), then signal it. The resulting ffmpeg write error is
# harmless and goes to the captured stderr.
if proc.stdout is not None:
try:
proc.stdout.close()
except OSError:
pass
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=5)
except sp.TimeoutExpired:
proc.kill()
proc.wait()
KEYFRAME_MAX_GAP_SECONDS = 2.0
def keyframe_sampling_eligible(
keyframe_pts: list[float], max_gap: float = KEYFRAME_MAX_GAP_SECONDS
) -> bool:
"""True if keyframes are dense and regular enough for keyframe-only sampling.
Requires at least two keyframes and no gap longer than ``max_gap`` seconds, so
a multi-second motion event necessarily spans a sampled keyframe.
"""
if len(keyframe_pts) < 2:
return False
gaps = [b - a for a, b in zip(keyframe_pts, keyframe_pts[1:])]
return max(gaps) <= max_gap
VOD_PROTOCOL_ARGS = ["-protocol_whitelist", "pipe,file,http,tcp"]
def build_vod_decode_command(
ffmpeg_path: str,
vod_url: str,
decode_args: list[str],
crop: tuple[int, int, int, int] | None,
scale: tuple[int, int] | None,
gray: bool,
*,
skip_nonkey: bool,
fps_rate: float | None,
) -> list[str]:
"""Build the ffmpeg argv to decode a VOD HLS URL.
``skip_nonkey`` adds ``-skip_frame nokey`` (keyframe-only). ``fps_rate`` adds
an ``fps`` filter for the fixed-cadence fallback. They are mutually
exclusive: keyframe mode passes ``skip_nonkey=True``/``fps_rate=None``; the
fallback passes ``skip_nonkey=False`` with a rate.
"""
filters: list[str] = []
# With hwaccel the decoded frames are GPU surfaces; pull them back to system
# memory before the CPU fps/crop/scale filters and the rawvideo encoder.
if decode_args:
filters.append("hwdownload")
filters.append("format=nv12")
if fps_rate is not None:
filters.append(f"fps={fps_rate}")
if crop is not None:
cw, ch, cx, cy = crop
filters.append(f"crop={cw}:{ch}:{cx}:{cy}")
if scale is not None:
sw, sh = scale
filters.append(f"scale={sw}:{sh}")
pix_fmt = "gray" if gray else "bgr24"
cmd = [ffmpeg_path, "-hide_banner", "-loglevel", "error"]
if skip_nonkey:
cmd += ["-skip_frame", "nokey"]
cmd += [*decode_args, *VOD_PROTOCOL_ARGS, "-i", vod_url, "-an"]
if filters:
cmd += ["-vf", ",".join(filters)]
cmd += ["-vsync", "0", "-f", "rawvideo", "-pix_fmt", pix_fmt, "pipe:"]
return cmd
def _run_vod_decode(
ffmpeg_path: str,
vod_url: str,
out_width: int,
out_height: int,
channels: int,
decode_args: list[str],
crop: tuple[int, int, int, int] | None,
scale: tuple[int, int] | None,
gray: bool,
should_stop: Callable[[], bool],
*,
skip_nonkey: bool,
fps_rate: float | None,
software_retry: bool,
) -> Generator[np.ndarray, None, None]:
"""Run one VOD decode, yielding raw frames; retry in software if empty."""
cmd = build_vod_decode_command(
ffmpeg_path,
vod_url,
decode_args,
crop,
scale,
gray,
skip_nonkey=skip_nonkey,
fps_rate=fps_rate,
)
frame_size = out_width * out_height * channels
stderr_file = tempfile.SpooledTemporaryFile(max_size=65536)
proc = sp.Popen(cmd, stdout=sp.PIPE, stderr=stderr_file)
assert proc.stdout is not None
count = 0
try:
while True:
if should_stop():
break
buf = _read_exact(proc.stdout, frame_size)
if buf is None:
break
if channels == 1:
frame = np.frombuffer(buf, dtype=np.uint8).reshape(
(out_height, out_width)
)
else:
frame = np.frombuffer(buf, dtype=np.uint8).reshape(
(out_height, out_width, channels)
)
count += 1
yield frame
finally:
_terminate(proc)
stderr_file.close()
if count == 0 and software_retry and not should_stop():
logger.warning("Hardware VOD decode produced no frames, retrying in software")
yield from _run_vod_decode(
ffmpeg_path,
vod_url,
out_width,
out_height,
channels,
[],
crop,
scale,
gray,
should_stop,
skip_nonkey=skip_nonkey,
fps_rate=fps_rate,
software_retry=False,
)
def iter_vod_frames(
ffmpeg_path: str,
vod_url: str,
out_width: int,
out_height: int,
channels: int,
decode_args: list[str],
crop: tuple[int, int, int, int] | None,
scale: tuple[int, int] | None,
gray: bool,
should_stop: Callable[[], bool],
*,
skip_nonkey: bool,
fps_rate: float | None,
) -> Generator[np.ndarray, None, None]:
"""Decode a VOD HLS URL and yield raw frames in order.
Pair keyframe-mode output with probed keyframe PTS; pair fallback output with
a fixed cadence. Falls back once to software decode if a hwaccel decode yields
no frames.
"""
yield from _run_vod_decode(
ffmpeg_path,
vod_url,
out_width,
out_height,
channels,
decode_args,
crop,
scale,
gray,
should_stop,
skip_nonkey=skip_nonkey,
fps_rate=fps_rate,
software_retry=bool(decode_args),
)
def probe_vod_keyframe_pts(ffprobe_path: str, vod_url: str) -> list[float]:
"""Return keyframe presentation timestamps (VOD stream time) in order.
Reads packet flags via ffprobe over the VOD URL (no decode). Returns [] on
any failure so the caller can fall back.
"""
cmd = [
ffprobe_path,
"-v",
"error",
*VOD_PROTOCOL_ARGS,
"-i",
vod_url,
"-select_streams",
"v:0",
"-show_packets",
"-show_entries",
"packet=pts_time,flags",
"-of",
"json",
]
try:
completed = sp.run(cmd, capture_output=True, text=True, timeout=120)
except (OSError, sp.SubprocessError):
logger.warning("ffprobe failed for VOD keyframe probe")
return []
if completed.returncode != 0 or not completed.stdout:
return []
try:
packets = json.loads(completed.stdout).get("packets", [])
except json.JSONDecodeError:
return []
pts: list[float] = []
for pkt in packets:
flags = pkt.get("flags", "")
pts_time = pkt.get("pts_time")
if flags.startswith("K") and pts_time is not None:
try:
pts.append(float(pts_time))
except ValueError:
continue
return sorted(pts)
def probe_video_dimensions(
ffprobe_path: str, recording_path: str
) -> tuple[int, int, float] | None:
"""Return (width, height, fps) for a recording's video stream, or None.
Reads stream metadata via ffprobe (no decode). The record stream resolution
can differ from the camera's detect resolution, so this is probed once per
job against a real segment.
"""
cmd = [
ffprobe_path,
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=width,height,avg_frame_rate",
"-of",
"json",
recording_path,
]
try:
completed = sp.run(cmd, capture_output=True, text=True, timeout=30)
except (OSError, sp.SubprocessError):
return None
if completed.returncode != 0 or not completed.stdout:
return None
try:
streams = json.loads(completed.stdout).get("streams", [])
except json.JSONDecodeError:
return None
if not streams:
return None
stream = streams[0]
width = int(stream.get("width", 0) or 0)
height = int(stream.get("height", 0) or 0)
rate = stream.get("avg_frame_rate", "0/0") or "0/0"
try:
num, _, den = rate.partition("/")
fps = float(num) / float(den) if float(den) != 0 else 0.0
except (ValueError, ZeroDivisionError):
fps = 0.0
if width <= 0 or height <= 0:
return None
return width, height, fps

3
frigate/object_detection/base.py
View File

@ -167,8 +167,9 @@ class DetectorRunner(FrigateProcess):
# detect and send the output
self.start_time.value = datetime.datetime.now().timestamp()
mono_start = time.monotonic()
detections = object_detector.detect_raw(input_frame)
duration = datetime.datetime.now().timestamp() - self.start_time.value
duration = time.monotonic() - mono_start
frame_manager.close(connection_id)
if connection_id not in self.outputs:

30
frigate/output/output.py
View File

@ -342,20 +342,30 @@ def move_preview_frames(loc: str) -> None:
preview_holdover = os.path.join(CLIPS_DIR, "preview_restart_cache")
preview_cache = os.path.join(CACHE_DIR, "preview_frames")
try:
if loc == "clips":
shutil.move(preview_cache, preview_holdover)
src = preview_cache
dst = preview_holdover
elif loc == "cache":
if not os.path.exists(preview_holdover):
src = preview_holdover
dst = preview_cache
else:
return
if not os.access(preview_holdover, os.R_OK | os.W_OK):
try:
if not os.path.exists(src):
return
shutil.move(src, dst)
except PermissionError:
logger.error(
"Insufficient permissions on preview restart cache at %s",
preview_holdover,
"Insufficient permissions while moving preview restart cache from %s to %s",
src,
dst,
)
return
shutil.move(preview_holdover, preview_cache)
except shutil.Error:
logger.error("Failed to restore preview cache.")
logger.error(
"Failed to move preview restart cache from %s to %s",
src,
dst,
)

2
frigate/ptz/autotrack.py
View File

@ -1331,6 +1331,8 @@ class PtzAutoTracker:
return self.tracked_object[camera]["region"]
def autotrack_object(self, camera: str, obj: TrackedObject):
if camera not in self.config.cameras:
return
camera_config = self.config.cameras[camera]
if camera_config.onvif.autotracking.enabled:

6
frigate/record/export.py
View File

@ -456,7 +456,7 @@ class RecordingExporter(threading.Thread):
diff = max(0.0, float(self.start_time) - float(preview.start_time))
ffmpeg_cmd = [
"/usr/lib/ffmpeg/7.0/bin/ffmpeg", # hardcode path for exports thumbnail due to missing libwebp support
"/usr/lib/ffmpeg/8.0/bin/ffmpeg", # hardcode path for exports thumbnail due to missing libwebp support
"-hide_banner",
"-loglevel",
"warning",
@ -579,7 +579,9 @@ class RecordingExporter(threading.Thread):
else:
chapters_path = self._build_chapter_metadata_file(recordings)
chapter_args = (
f" -i {chapters_path} -map 0 -map_metadata 1" if chapters_path else ""
f" -i {chapters_path} -map 0 -dn -map_metadata 1"
if chapters_path
else ""
)
ffmpeg_cmd = (
f"{self.config.ffmpeg.ffmpeg_path} -hide_banner {ffmpeg_input}{chapter_args} -c copy -movflags +faststart"

Some files were not shown because too many files have changed in this diff Show More