mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-07-29 15:19:00 +03:00
feat(genai): Support multiple GenAI providers and per-camera configuration
This commit is contained in:
mw
@@ -11,18 +11,24 @@ Requests for a description are sent off automatically to your AI provider at the
|
||||
|
||||
Generative AI can be enabled for all cameras or only for specific cameras. If GenAI is disabled for a camera, you can still manually generate descriptions for events using the HTTP API. There are currently 3 native providers available to integrate with Frigate. Other providers that support the OpenAI standard API can also be used. See the OpenAI section below.
|
||||
|
||||
To use Generative AI, you must define a single provider at the global level of your Frigate configuration. If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
|
||||
To use Generative AI, you must define one or more providers at the global level of your Frigate configuration. If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: gemini
|
||||
api_key: "{FRIGATE_GEMINI_API_KEY}"
|
||||
model: gemini-2.0-flash
|
||||
- name: gemini
|
||||
provider: gemini
|
||||
api_key: "{FRIGATE_GEMINI_API_KEY}"
|
||||
model: gemini-2.0-flash
|
||||
- name: ollama
|
||||
provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: qwen3-vl:4b
|
||||
|
||||
cameras:
|
||||
front_camera:
|
||||
genai:
|
||||
enabled: True # <- enable GenAI for your front camera
|
||||
provider: gemini # <- specify which provider to use
|
||||
use_snapshot: True
|
||||
objects:
|
||||
- person
|
||||
@@ -72,9 +78,9 @@ Ollama also supports [cloud models](https://ollama.com/cloud), where your local
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: qwen3-vl:4b
|
||||
- provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: qwen3-vl:4b
|
||||
```
|
||||
|
||||
## Google Gemini
|
||||
@@ -98,9 +104,9 @@ To start using Gemini, you must first get an API key from [Google AI Studio](htt
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: gemini
|
||||
api_key: "{FRIGATE_GEMINI_API_KEY}"
|
||||
model: gemini-2.0-flash
|
||||
- provider: gemini
|
||||
api_key: "{FRIGATE_GEMINI_API_KEY}"
|
||||
model: gemini-2.0-flash
|
||||
```
|
||||
|
||||
:::note
|
||||
@@ -125,9 +131,9 @@ To start using OpenAI, you must first [create an API key](https://platform.opena
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: openai
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
model: gpt-4o
|
||||
- provider: openai
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
model: gpt-4o
|
||||
```
|
||||
|
||||
:::note
|
||||
@@ -152,10 +158,10 @@ To start using Azure OpenAI, you must first [create a resource](https://learn.mi
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: azure_openai
|
||||
base_url: https://instance.cognitiveservices.azure.com/openai/responses?api-version=2025-04-01-preview
|
||||
model: gpt-5-mini
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
- provider: azure_openai
|
||||
base_url: https://instance.cognitiveservices.azure.com/openai/responses?api-version=2025-04-01-preview
|
||||
model: gpt-5-mini
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
```
|
||||
|
||||
## Usage and Best Practices
|
||||
@@ -171,10 +177,11 @@ Frigate provides an [MQTT topic](/integrations/mqtt), `frigate/tracked_object_up
|
||||
If looking to get notifications earlier than when an object ceases to be tracked, an additional send trigger can be configured of `after_significant_updates`.
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
send_triggers:
|
||||
tracked_object_end: true # default
|
||||
after_significant_updates: 3 # how many updates to a tracked object before we should send an image
|
||||
objects:
|
||||
genai:
|
||||
send_triggers:
|
||||
tracked_object_end: true # default
|
||||
after_significant_updates: 3 # how many updates to a tracked object before we should send an image
|
||||
```
|
||||
|
||||
## Custom Prompts
|
||||
|
||||
@@ -7,7 +7,7 @@ title: Configuring Generative AI
|
||||
|
||||
A Generative AI provider can be configured in the global config, which will make the Generative AI features available for use. There are currently 3 native providers available to integrate with Frigate. Other providers that support the OpenAI standard API can also be used. See the OpenAI section below.
|
||||
|
||||
To use Generative AI, you must define a single provider at the global level of your Frigate configuration. If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
|
||||
To use Generative AI, you must define one or more providers at the global level of your Frigate configuration. If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
|
||||
|
||||
## Ollama
|
||||
|
||||
@@ -58,13 +58,13 @@ You should have at least 8 GB of RAM available (or VRAM if running on GPU) to ru
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: minicpm-v:8b
|
||||
provider_options: # other Ollama client options can be defined
|
||||
keep_alive: -1
|
||||
options:
|
||||
num_ctx: 8192 # make sure the context matches other services that are using ollama
|
||||
- provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: minicpm-v:8b
|
||||
provider_options: # other Ollama client options can be defined
|
||||
keep_alive: -1
|
||||
options:
|
||||
num_ctx: 8192 # make sure the context matches other services that are using ollama
|
||||
```
|
||||
|
||||
## Google Gemini
|
||||
@@ -88,9 +88,9 @@ To start using Gemini, you must first get an API key from [Google AI Studio](htt
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: gemini
|
||||
api_key: "{FRIGATE_GEMINI_API_KEY}"
|
||||
model: gemini-1.5-flash
|
||||
- provider: gemini
|
||||
api_key: "{FRIGATE_GEMINI_API_KEY}"
|
||||
model: gemini-1.5-flash
|
||||
```
|
||||
|
||||
## OpenAI
|
||||
@@ -109,9 +109,9 @@ To start using OpenAI, you must first [create an API key](https://platform.opena
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: openai
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
model: gpt-4o
|
||||
- provider: openai
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
model: gpt-4o
|
||||
```
|
||||
|
||||
:::note
|
||||
@@ -136,7 +136,7 @@ To start using Azure OpenAI, you must first [create a resource](https://learn.mi
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: azure_openai
|
||||
base_url: https://example-endpoint.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
- provider: azure_openai
|
||||
base_url: https://example-endpoint.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview
|
||||
api_key: "{FRIGATE_OPENAI_API_KEY}"
|
||||
```
|
||||
|
||||
@@ -37,9 +37,9 @@ You are also able to define custom prompts in your configuration.
|
||||
|
||||
```yaml
|
||||
genai:
|
||||
provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: llava
|
||||
- provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: llava
|
||||
|
||||
objects:
|
||||
prompt: "Analyze the {label} in these images from the {camera} security camera. Focus on the actions, behavior, and potential intent of the {label}, rather than just describing its appearance."
|
||||
|
||||
@@ -686,16 +686,16 @@ lpr:
|
||||
# the camera level to enhance privacy for indoor cameras.
|
||||
genai:
|
||||
# Required: Provider must be one of ollama, gemini, or openai
|
||||
provider: ollama
|
||||
# Required if provider is ollama. May also be used for an OpenAI API compatible backend with the openai provider.
|
||||
base_url: http://localhost::11434
|
||||
# Required if gemini or openai
|
||||
api_key: "{FRIGATE_GENAI_API_KEY}"
|
||||
# Required: The model to use with the provider.
|
||||
model: gemini-1.5-flash
|
||||
# Optional additional args to pass to the GenAI Provider (default: None)
|
||||
provider_options:
|
||||
keep_alive: -1
|
||||
- provider: ollama
|
||||
# Required if provider is ollama. May also be used for an OpenAI API compatible backend with the openai provider.
|
||||
base_url: http://localhost::11434
|
||||
# Required if gemini or openai
|
||||
api_key: "{FRIGATE_GENAI_API_KEY}"
|
||||
# Required: The model to use with the provider.
|
||||
model: gemini-1.5-flash
|
||||
# Optional additional args to pass to the GenAI Provider (default: None)
|
||||
provider_options:
|
||||
keep_alive: -1
|
||||
|
||||
# Optional: Configuration for audio transcription
|
||||
# NOTE: only the enabled option can be overridden at the camera level
|
||||
|
||||
Reference in New Issue
Block a user