administrator/frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2026-08-12 13:51:12 +03:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

feat(genai): add TwelveLabs Marengo embeddings provider

Browse Source 
Adds an opt-in 'twelvelabs' GenAI provider implementing the embeddings
role using TwelveLabs' Marengo multimodal model. Text and image inputs
are embedded into a shared vector space, powering semantic search over
event thumbnails.

The provider only implements the embeddings role; descriptions/chat are
left to other providers since Marengo is an embedding model. Marengo
returns 512-dim vectors, which the existing GenAIEmbedding adapter pads
to Frigate's 768-dim search schema.

Includes no-network unit tests and a live test gated on
TWELVELABS_API_KEY.
This commit is contained in:
Mohit Varikuti
2026-06-25 00:43:04 -07:00
parent ec3fb00494
commit 85e16eb31b
5 changed files with 301 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/configuration/genai/config.md
+35 -1
View File
@@ -9,7 +9,7 @@ import NavPath from "@site/src/components/NavPath";
## Configuration
A Generative AI provider can be configured in the global config, which will make the Generative AI features available for use. There are currently 4 native providers available to integrate with Frigate. Other providers that support the OpenAI standard API can also be used. See the OpenAI-Compatible section below.
A Generative AI provider can be configured in the global config, which will make the Generative AI features available for use. There are currently several native providers available to integrate with Frigate. Other providers that support the OpenAI standard API can also be used. See the OpenAI-Compatible section below.
To use Generative AI, you must define a single provider at the global level of your Frigate configuration. If the provider you choose requires an API key, you may either directly paste it in your configuration, or store it in an environment variable prefixed with `FRIGATE_`.
@@ -386,3 +386,37 @@ genai:
</TabItem>
</ConfigTabs>
### TwelveLabs (Marengo embeddings)
[TwelveLabs](https://twelvelabs.io) provides the Marengo multimodal model, which embeds text and images into a shared vector space. This makes it usable as the `embeddings` provider for Frigate's [Semantic Search](/configuration/semantic_search), letting a natural-language query match stored event thumbnails directly.
This provider implements only the `embeddings` role. Use a separate provider (such as Ollama, Gemini, or OpenAI) for the `descriptions` and `chat` roles if you want generated descriptions as well.
#### Get API Key
Create an API key from the [TwelveLabs dashboard](https://playground.twelvelabs.io/dashboard/api-key). There is a generous free tier.
#### Configuration
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Enrichments > Generative AI" />.
- Set **Provider** to `twelvelabs`
- Set **API key** to your TwelveLabs API key (or use an environment variable such as `{FRIGATE_TWELVELABS_API_KEY}`)
- Optionally set **Model** to override the default Marengo model (`marengo3.0`)
</TabItem>
<TabItem value="yaml">
```yaml
genai:
provider: twelvelabs
api_key: "{FRIGATE_TWELVELABS_API_KEY}"
roles:
- embeddings
```
</TabItem>
</ConfigTabs>