Compare commits

...

3 Commits

Author SHA1 Message Date
lucaszhu-hue
6330a3b494
Merge a2b92caab0 into 652ea2454f 2026-06-19 23:02:47 +02:00
Josh Hawkins
652ea2454f
Miscellaneous fixes (#23513)
* display zone names consistently using friendly_name or raw id without transformation

* enforce camera-level access on go2rtc live stream websocket endpoints
2026-06-19 10:10:22 -06:00
Lucas Zhu
a2b92caab0 feat(genai): add Atlas Cloud as an OpenAI-compatible GenAI provider
Add an `atlas` GenAI provider backed by Atlas Cloud, an OpenAI-compatible
inference platform serving vision-capable models. The provider subclasses
the existing OpenAIClient and only defaults the base_url to the Atlas
endpoint, reusing all vision, streaming, reasoning, and tool-calling logic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 00:59:49 +08:00
12 changed files with 446 additions and 22 deletions

View File

@ -12,6 +12,44 @@
\[English\] | [简体中文](https://github.com/blakeblackshear/frigate/blob/dev/README_CN.md) \[English\] | [简体中文](https://github.com/blakeblackshear/frigate/blob/dev/README_CN.md)
---
<p align="center">
<a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=frigate">
<img src="docs/static/img/branding/atlas-cloud-logo.png" alt="Atlas Cloud" width="200">
</a>
</p>
<p align="center">
<b><a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=frigate">Atlas Cloud</a></b> is an OpenAI-compatible inference platform that can power Frigate's
<a href="https://docs.frigate.video/configuration/genai/">Generative AI</a> features as a drop-in multimodal LLM backend.
Point the <code>atlas</code> provider at Atlas Cloud and use a vision-capable model
(such as <code>qwen/qwen3-vl-235b-a22b-thinking</code> or <code>Qwen/Qwen3-VL-235B-A22B-Instruct</code>)
to generate natural-language object and review descriptions from detection frames —
no local GPU required. See the <a href="https://docs.frigate.video/configuration/genai/">GenAI configuration docs</a>
to get started, or grab a <a href="https://www.atlascloud.ai/console/coding-plan">coding plan</a>.
</p>
<details>
<summary>Vision-capable Atlas Cloud models for GenAI descriptions</summary>
Frigate's GenAI features require a **vision-capable** model. Good multimodal choices on Atlas Cloud include:
- `qwen/qwen3-vl-235b-a22b-thinking`
- `Qwen/Qwen3-VL-235B-A22B-Instruct`
- `qwen/qwen3-vl-30b-a3b-instruct`
- `qwen/qwen3-vl-30b-a3b-thinking`
- `qwen/qwen3-vl-8b-instruct`
- `google/gemini-3.5-flash`
- `google/gemini-3.1-pro-preview`
The full, always-current model catalog is available at the
[Atlas Cloud console](https://www.atlascloud.ai/console).
</details>
---
A complete and local NVR designed for [Home Assistant](https://www.home-assistant.io) with AI object detection. Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras. A complete and local NVR designed for [Home Assistant](https://www.home-assistant.io) with AI object detection. Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras.
Use of a GPU or AI accelerator is highly recommended. AI accelerators will outperform even the best CPUs with very little overhead. See Frigate's supported [object detectors](https://docs.frigate.video/configuration/object_detectors/). Use of a GPU or AI accelerator is highly recommended. AI accelerators will outperform even the best CPUs with very little overhead. See Frigate's supported [object detectors](https://docs.frigate.video/configuration/object_detectors/).

View File

@ -12,6 +12,43 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
---
<p align="center">
<a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=frigate">
<img src="docs/static/img/branding/atlas-cloud-logo.png" alt="Atlas Cloud" width="200">
</a>
</p>
<p align="center">
<b><a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=frigate">Atlas Cloud</a></b> 是一个兼容 OpenAI 接口的推理平台,可作为即插即用的多模态 LLM 后端,
为 Frigate 的<a href="https://docs.frigate.video/configuration/genai/">生成式 AIGenerative AI</a>功能提供算力支持。
只需将 <code>atlas</code> provider 指向 Atlas Cloud并选用一个支持视觉的模型
(例如 <code>qwen/qwen3-vl-235b-a22b-thinking</code><code>Qwen/Qwen3-VL-235B-A22B-Instruct</code>
即可基于检测帧画面生成自然语言的物体描述与审查摘要,无需本地 GPU。
请参阅 <a href="https://docs.frigate.video/configuration/genai/">GenAI 配置文档</a>开始使用,
或了解 <a href="https://www.atlascloud.ai/console/coding-plan">coding plan</a>
</p>
<details>
<summary>适合做 GenAI 描述的 Atlas Cloud 多模态模型</summary>
Frigate 的 GenAI 功能要求使用**支持视觉**的模型。Atlas Cloud 上推荐的多模态模型包括:
- `qwen/qwen3-vl-235b-a22b-thinking`
- `Qwen/Qwen3-VL-235B-A22B-Instruct`
- `qwen/qwen3-vl-30b-a3b-instruct`
- `qwen/qwen3-vl-30b-a3b-thinking`
- `qwen/qwen3-vl-8b-instruct`
- `google/gemini-3.5-flash`
- `google/gemini-3.1-pro-preview`
完整且实时更新的模型列表请见 [Atlas Cloud 控制台](https://www.atlascloud.ai/console)。
</details>
---
一个完整的本地网络视频录像机NVR专为[Home Assistant](https://www.home-assistant.io)设计,具备 AI 目标/物体检测功能。使用 OpenCV 和 TensorFlow 在本地为 IP 摄像头执行实时物体检测。 一个完整的本地网络视频录像机NVR专为[Home Assistant](https://www.home-assistant.io)设计,具备 AI 目标/物体检测功能。使用 OpenCV 和 TensorFlow 在本地为 IP 摄像头执行实时物体检测。
强烈推荐使用 GPU 或者 AI 加速器(例如[Google Coral 加速器](https://coral.ai/products/) 或者 [Hailo](https://hailo.ai/)等)。它们的运行效率远远高于现在的顶级 CPU并且功耗也极低。 强烈推荐使用 GPU 或者 AI 加速器(例如[Google Coral 加速器](https://coral.ai/products/) 或者 [Hailo](https://hailo.ai/)等)。它们的运行效率远远高于现在的顶级 CPU并且功耗也极低。

View File

@ -386,3 +386,44 @@ genai:
</TabItem> </TabItem>
</ConfigTabs> </ConfigTabs>
### Atlas Cloud
[Atlas Cloud](https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=frigate) is an OpenAI-compatible inference platform that serves a range of vision-capable models, so it can act as a drop-in multimodal backend for Frigate's Generative AI features. The `atlas` provider defaults its base URL to the Atlas Cloud endpoint, so a minimal config only needs your API key and a model.
#### Supported Models
You must use a vision capable model with Frigate. Recommended multimodal models on Atlas Cloud include `qwen/qwen3-vl-235b-a22b-thinking`, `Qwen/Qwen3-VL-235B-A22B-Instruct`, `qwen/qwen3-vl-30b-a3b-instruct`, and `google/gemini-3.5-flash`. The full, always-current catalog is available in the [Atlas Cloud console](https://www.atlascloud.ai/console).
#### Get API Key
To start using Atlas Cloud, create an API key from the [Atlas Cloud console](https://www.atlascloud.ai/console/api-keys).
#### Configuration
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Enrichments > Generative AI" />.
- Set **Provider** to `atlas`
- Set **API key** to your Atlas Cloud API key (or use an environment variable such as `{FRIGATE_ATLAS_API_KEY}`)
- Set **Model** to a vision-capable model (e.g., `qwen/qwen3-vl-235b-a22b-thinking`)
</TabItem>
<TabItem value="yaml">
```yaml
genai:
provider: atlas
api_key: "{FRIGATE_ATLAS_API_KEY}"
model: qwen/qwen3-vl-235b-a22b-thinking
```
</TabItem>
</ConfigTabs>
:::note
The `atlas` provider points to `https://api.atlascloud.ai/v1` by default. To target a different OpenAI-compatible endpoint, set `base_url` explicitly.
:::

Binary file not shown.

After

Width:  |  Height:  |  Size: 131 KiB

View File

@ -12,6 +12,7 @@ import time
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import List, Optional from typing import List, Optional
from urllib.parse import parse_qs, urlparse
from fastapi import APIRouter, Depends, HTTPException, Request, Response from fastapi import APIRouter, Depends, HTTPException, Request, Response
from fastapi.responses import JSONResponse, RedirectResponse from fastapi.responses import JSONResponse, RedirectResponse
@ -26,7 +27,11 @@ from frigate.api.defs.request.app_body import (
AppPutRoleBody, AppPutRoleBody,
) )
from frigate.api.defs.tags import Tags from frigate.api.defs.tags import Tags
from frigate.api.media_auth import check_camera_access, deny_response_for_media_uri from frigate.api.media_auth import (
check_camera_access,
deny_response_for_media_uri,
is_role_restricted,
)
from frigate.config import AuthConfig, NetworkingConfig, ProxyConfig from frigate.config import AuthConfig, NetworkingConfig, ProxyConfig
from frigate.const import CONFIG_DIR, JWT_SECRET_ENV_VAR, PASSWORD_HASH_ALGORITHM from frigate.const import CONFIG_DIR, JWT_SECRET_ENV_VAR, PASSWORD_HASH_ALGORITHM
from frigate.models import User from frigate.models import User
@ -658,6 +663,10 @@ def auth(request: Request):
if deny_status is not None: if deny_status is not None:
return Response("", status_code=deny_status) return Response("", status_code=deny_status)
deny_status = deny_response_for_go2rtc_stream(original_url, role, request)
if deny_status is not None:
return Response("", status_code=deny_status)
return success_response return success_response
# now apply authentication # now apply authentication
@ -757,6 +766,10 @@ def auth(request: Request):
if deny_status is not None: if deny_status is not None:
return Response("", status_code=deny_status) return Response("", status_code=deny_status)
deny_status = deny_response_for_go2rtc_stream(original_url, role, request)
if deny_status is not None:
return Response("", status_code=deny_status)
return success_response return success_response
except Exception as e: except Exception as e:
logger.error(f"Error parsing jwt: {e}") logger.error(f"Error parsing jwt: {e}")
@ -1112,6 +1125,66 @@ def _get_stream_owner_cameras(request: Request, stream_name: str) -> set[str]:
return owner_cameras return owner_cameras
# nginx proxies these paths straight to go2rtc with authentication-only checks
# (see auth_request.conf). Each names the desired stream via the `src` query
# param, so the camera-level check must happen here in the `/auth` subrequest —
# `require_go2rtc_stream_access` only guards the REST `/go2rtc/streams/{name}`
# endpoint, not these proxied live-stream paths.
GO2RTC_STREAM_PROXY_PATHS = frozenset(
{
"/live/mse/api/ws",
"/live/webrtc/api/ws",
"/api/go2rtc/webrtc",
}
)
def deny_response_for_go2rtc_stream(
original_url: Optional[str], role: Optional[str], request: Request
) -> Optional[int]:
"""Block role-restricted users from go2rtc live streams they cannot access.
Returns 403 when any `src` stream named in `original_url` resolves to a
camera outside the role's allow-list (or when no `src` is provided on a
stream-proxy path), otherwise None. Mirrors the resolution logic in
`require_go2rtc_stream_access` so substream names map to their owning
camera correctly.
"""
if not original_url:
return None
parsed = urlparse(original_url)
if parsed.path not in GO2RTC_STREAM_PROXY_PATHS:
return None
frigate_config = request.app.frigate_config
# admin and full-access roles (no allow-list) bypass the camera check
if not role or not is_role_restricted(role, frigate_config):
return None
sources = parse_qs(parsed.query).get("src", [])
if not sources:
# a stream-proxy request naming no stream has nothing legitimate to
# show a restricted user
return 403
allowed_cameras = set(
User.get_allowed_cameras(
role,
frigate_config.auth.roles,
set(frigate_config.cameras.keys()),
)
)
# deny if any requested source resolves outside the allow-list
for src in sources:
if not (_get_stream_owner_cameras(request, src) & allowed_cameras):
return 403
return None
async def require_go2rtc_stream_access( async def require_go2rtc_stream_access(
stream_name: Optional[str] = None, stream_name: Optional[str] = None,
request: Request = None, request: Request = None,

View File

@ -12,6 +12,7 @@ __all__ = ["GenAIConfig", "GenAIProviderEnum", "GenAIRoleEnum"]
class GenAIProviderEnum(str, Enum): class GenAIProviderEnum(str, Enum):
openai = "openai" openai = "openai"
azure_openai = "azure_openai" azure_openai = "azure_openai"
atlas = "atlas"
gemini = "gemini" gemini = "gemini"
ollama = "ollama" ollama = "ollama"
llamacpp = "llamacpp" llamacpp = "llamacpp"

View File

@ -0,0 +1,71 @@
"""Atlas Cloud Provider for Frigate AI.
Atlas Cloud (https://www.atlascloud.ai) is an OpenAI-compatible inference
platform that serves a range of vision-capable models. Because its chat
completions API follows the OpenAI standard, this provider inherits all
transport, vision, streaming, reasoning, and tool-calling logic from
:class:`OpenAIClient` and only overrides what is Atlas-specific:
- Client construction: defaults ``base_url`` to the Atlas Cloud endpoint
when the user has not set one explicitly, so a minimal config (provider +
api_key + model) works out of the box. A user-supplied ``base_url`` still
takes precedence.
- Context size: the Atlas ``/models`` endpoint does not reliably surface a
per-model context window, so we fall back to a conservative default rather
than the model-name heuristic used by OpenAI. It can be overridden via
``provider_options.context_size``.
"""
import logging
from typing import Optional
from openai import OpenAI
from frigate.config import GenAIProviderEnum
from frigate.genai import register_genai_provider
from frigate.genai.plugins.openai import OpenAIClient
logger = logging.getLogger(__name__)
DEFAULT_BASE_URL = "https://api.atlascloud.ai/v1"
# Atlas serves large-context models, but its model listing does not expose a
# per-model context window; default conservatively and let users override via
# provider_options.context_size when they know their model's window.
DEFAULT_CONTEXT_SIZE = 32000
@register_genai_provider(GenAIProviderEnum.atlas)
class AtlasClient(OpenAIClient):
"""Generative AI client for Frigate using Atlas Cloud."""
def _init_provider(self) -> OpenAI:
"""Initialize the OpenAI client pointed at Atlas Cloud.
Defaults ``base_url`` to the Atlas endpoint when the user has not set
one, then defers to the OpenAI implementation for everything else.
"""
if not self.genai_config.base_url:
self.genai_config.base_url = DEFAULT_BASE_URL
return super()._init_provider()
def get_context_size(self) -> int:
"""Return the context window for Atlas models.
A manually specified ``context_size`` in ``provider_options`` always
wins; otherwise fall back to a conservative default since Atlas does
not reliably surface per-model context windows.
"""
if self.context_size is not None:
return self.context_size
provider_context_size: Optional[int] = self.genai_config.provider_options.get(
"context_size"
)
if provider_context_size is not None:
self.context_size = provider_context_size
return self.context_size
self.context_size = DEFAULT_CONTEXT_SIZE
return self.context_size

View File

@ -0,0 +1,175 @@
"""Unit tests for `deny_response_for_go2rtc_stream`.
Covers the camera-level authorization enforced in the `/auth` subrequest for
the nginx-proxied go2rtc live-stream paths (MSE/WebRTC WebSockets and the
WebRTC signaling endpoint). These paths name the stream via the `src` query
param, which the static-media auth in `media_auth` does not inspect.
"""
import types
import unittest
from frigate.api.auth import deny_response_for_go2rtc_stream
from frigate.config import FrigateConfig
_CONFIG = {
"mqtt": {"host": "mqtt"},
"auth": {
"roles": {
"limited_user": ["front_door"],
"dual_user": ["front_door", "back_door"],
}
},
"cameras": {
"front_door": {
"ffmpeg": {
"inputs": [{"path": "rtsp://10.0.0.1:554/video", "roles": ["detect"]}]
},
"detect": {"height": 1080, "width": 1920, "fps": 5},
# go2rtc stream name differs from the camera name (substream)
"live": {"streams": {"Main Stream": "front_door_sub"}},
},
"back_door": {
"ffmpeg": {
"inputs": [{"path": "rtsp://10.0.0.2:554/video", "roles": ["detect"]}]
},
"detect": {"height": 1080, "width": 1920, "fps": 5},
},
"garage": {
"ffmpeg": {
"inputs": [{"path": "rtsp://10.0.0.3:554/video", "roles": ["detect"]}]
},
"detect": {"height": 1080, "width": 1920, "fps": 5},
},
},
}
def _request(config: FrigateConfig) -> types.SimpleNamespace:
return types.SimpleNamespace(app=types.SimpleNamespace(frigate_config=config))
class TestDenyResponseForGo2rtcStream(unittest.TestCase):
def setUp(self) -> None:
self.config = FrigateConfig(**_CONFIG)
self.request = _request(self.config)
def _deny(self, url: str, role: str):
return deny_response_for_go2rtc_stream(url, role, self.request)
# --- non-stream paths pass through ---
def test_non_stream_path_passes_through(self):
self.assertIsNone(
self._deny("http://host/clips/back_door-1.jpg", "limited_user")
)
def test_empty_url_passes_through(self):
self.assertIsNone(self._deny("", "limited_user"))
def test_jsmpeg_path_not_handled_here(self):
# jsmpeg is authorized per-frame in the output pipeline, not here
self.assertIsNone(
self._deny("http://host/live/jsmpeg/back_door", "limited_user")
)
# --- restricted role: allowed vs forbidden cameras ---
def test_mse_allowed_camera(self):
self.assertIsNone(
self._deny("http://host/live/mse/api/ws?src=front_door", "limited_user")
)
def test_mse_forbidden_camera_denied(self):
self.assertEqual(
self._deny("http://host/live/mse/api/ws?src=back_door", "limited_user"),
403,
)
def test_webrtc_ws_forbidden_camera_denied(self):
self.assertEqual(
self._deny("http://host/live/webrtc/api/ws?src=back_door", "limited_user"),
403,
)
def test_webrtc_signaling_forbidden_camera_denied(self):
self.assertEqual(
self._deny("http://host/api/go2rtc/webrtc?src=back_door", "limited_user"),
403,
)
def test_unknown_camera_denied(self):
self.assertEqual(
self._deny("http://host/live/mse/api/ws?src=nonexistent", "limited_user"),
403,
)
def test_missing_src_denied(self):
self.assertEqual(self._deny("http://host/live/mse/api/ws", "limited_user"), 403)
# --- multi-camera role: each assigned camera allowed, others denied ---
def test_multi_camera_role_allows_first_assigned(self):
self.assertIsNone(
self._deny("http://host/live/mse/api/ws?src=front_door", "dual_user")
)
def test_multi_camera_role_allows_second_assigned(self):
self.assertIsNone(
self._deny("http://host/live/mse/api/ws?src=back_door", "dual_user")
)
def test_multi_camera_role_denies_unassigned(self):
# garage is configured but not in dual_user's allow-list
self.assertEqual(
self._deny("http://host/live/mse/api/ws?src=garage", "dual_user"),
403,
)
# --- substream names resolve to their owning camera ---
def test_allowed_substream_resolves_to_owning_camera(self):
# front_door_sub is owned by front_door, which limited_user may access
self.assertIsNone(
self._deny("http://host/live/mse/api/ws?src=front_door_sub", "limited_user")
)
# --- multiple src values: deny if any is forbidden ---
def test_multiple_src_one_forbidden_denied(self):
self.assertEqual(
self._deny(
"http://host/live/mse/api/ws?src=front_door&src=back_door",
"limited_user",
),
403,
)
def test_multiple_src_all_allowed(self):
self.assertIsNone(
self._deny(
"http://host/live/mse/api/ws?src=front_door&src=front_door_sub",
"limited_user",
)
)
# --- privileged roles bypass the check ---
def test_admin_bypasses(self):
self.assertIsNone(
self._deny("http://host/live/mse/api/ws?src=back_door", "admin")
)
def test_builtin_viewer_role_bypasses(self):
# the built-in viewer role is not in the config allow-list map, so it
# is treated as full access
self.assertIsNone(
self._deny("http://host/live/mse/api/ws?src=back_door", "viewer")
)
def test_missing_role_bypasses(self):
self.assertIsNone(self._deny("http://host/live/mse/api/ws?src=back_door", None))
if __name__ == "__main__":
unittest.main()

View File

@ -243,12 +243,7 @@ export default function CameraReviewClassification({
handleZoneToggle("alerts.required_zones", zone.name) handleZoneToggle("alerts.required_zones", zone.name)
} }
/> />
<Label <Label className="font-normal">
className={cn(
"font-normal",
!zone.friendly_name && "smart-capitalize",
)}
>
{zone.friendly_name || zone.name} {zone.friendly_name || zone.name}
</Label> </Label>
</div> </div>

View File

@ -29,8 +29,8 @@ function getZoneDisplayName(zoneName: string, context?: FormContext): string {
} }
} }
} }
// Fallback to cleaning up the zone name // Fallback to the raw zone id verbatim (no friendly_name available)
return String(zoneName).replace(/_/g, " "); return String(zoneName);
} }
export function ZoneSwitchesWidget(props: WidgetProps) { export function ZoneSwitchesWidget(props: WidgetProps) {

View File

@ -1197,14 +1197,7 @@ function LifecycleIconRow({
backgroundColor: `rgb(${color})`, backgroundColor: `rgb(${color})`,
}} }}
/> />
<span <span>{item.data?.zones_friendly_names?.[zidx]}</span>
className={cn(
item.data?.zones_friendly_names?.[zidx] === zone &&
"smart-capitalize",
)}
>
{item.data?.zones_friendly_names?.[zidx]}
</span>
</Badge> </Badge>
); );
})} })}

View File

@ -7,12 +7,12 @@ export function resolveZoneName(
zoneId: string, zoneId: string,
cameraId?: string, cameraId?: string,
) { ) {
if (!config) return String(zoneId).replace(/_/g, " "); if (!config) return String(zoneId);
if (cameraId) { if (cameraId) {
const camera = config.cameras?.[String(cameraId)]; const camera = config.cameras?.[String(cameraId)];
const zone = camera?.zones?.[zoneId]; const zone = camera?.zones?.[zoneId];
return zone?.friendly_name || String(zoneId).replace(/_/g, " "); return zone?.friendly_name || String(zoneId);
} }
for (const camKey in config.cameras) { for (const camKey in config.cameras) {
@ -21,12 +21,12 @@ export function resolveZoneName(
if (!cam?.zones) continue; if (!cam?.zones) continue;
if (Object.prototype.hasOwnProperty.call(cam.zones, zoneId)) { if (Object.prototype.hasOwnProperty.call(cam.zones, zoneId)) {
const zone = cam.zones[zoneId]; const zone = cam.zones[zoneId];
return zone?.friendly_name || String(zoneId).replace(/_/g, " "); return zone?.friendly_name || String(zoneId);
} }
} }
// Fallback: return a cleaned-up zoneId string // Fallback: display the raw zone id verbatim (no friendly_name available)
return String(zoneId).replace(/_/g, " "); return String(zoneId);
} }
export function useZoneFriendlyName(zoneId: string, cameraId?: string): string { export function useZoneFriendlyName(zoneId: string, cameraId?: string): string {