Docs refactor (#22703)
CI / AMD64 Build (push) Waiting to run
CI / ARM Build (push) Waiting to run
CI / Jetson Jetpack 6 (push) Waiting to run
CI / AMD64 Extra Build (push) Blocked by required conditions
CI / ARM Extra Build (push) Blocked by required conditions
CI / Synaptics Build (push) Blocked by required conditions
CI / Assemble and push default build (push) Blocked by required conditions

* add generation script

a script to read yaml code blocks from docs markdown files and generate corresponding "Frigate UI" tab instructions based on the json schema, i18n, section configs (hidden fields), and nav mappings

* first pass

* components

* add to gitignore

* second pass

* fix broken anchors

* fixes

* clean up tabs

* version bump

* tweaks

* remove role mapping config from ui
This commit is contained in:
Josh Hawkins
2026-03-30 10:36:45 -06:00
committed by GitHub
parent a5e3dfd107
commit 5a5d23b503
53 changed files with 5310 additions and 276 deletions
+40 -4
View File
@@ -3,6 +3,10 @@ id: genai_objects
title: Object Descriptions
---
import ConfigTabs from "@site/src/components/ConfigTabs";
import TabItem from "@theme/TabItem";
import NavPath from "@site/src/components/NavPath";
Generative AI can be used to automatically generate descriptive text based on the thumbnails of your tracked objects. This helps with [Semantic Search](/configuration/semantic_search) in Frigate to provide more context about your tracked objects. Descriptions are accessed via the _Explore_ view in the Frigate UI by clicking on a tracked object's thumbnail.
Requests for a description are sent off automatically to your AI provider at the end of the tracked object's lifecycle, or can optionally be sent earlier after a number of significantly changed frames, for example in use in more real-time notifications. Descriptions can also be regenerated manually via the Frigate UI. Note that if you are manually entering a description for tracked objects prior to its end, this will be overwritten by the generated response.
@@ -15,9 +19,9 @@ Generative AI object descriptions can also be toggled dynamically for a camera v
## Usage and Best Practices
Frigate's thumbnail search excels at identifying specific details about tracked objects for example, using an "image caption" approach to find a "person wearing a yellow vest," "a white dog running across the lawn," or "a red car on a residential street." To enhance this further, Frigates default prompts are designed to ask your AI provider about the intent behind the object's actions, rather than just describing its appearance.
Frigate's thumbnail search excels at identifying specific details about tracked objects -- for example, using an "image caption" approach to find a "person wearing a yellow vest," "a white dog running across the lawn," or "a red car on a residential street." To enhance this further, Frigate's default prompts are designed to ask your AI provider about the intent behind the object's actions, rather than just describing its appearance.
While generating simple descriptions of detected objects is useful, understanding intent provides a deeper layer of insight. Instead of just recognizing "what" is in a scene, Frigates default prompts aim to infer "why" it might be there or "what" it could do next. Descriptions tell you whats happening, but intent gives context. For instance, a person walking toward a door might seem like a visitor, but if theyre moving quickly after hours, you can infer a potential break-in attempt. Detecting a person loitering near a door at night can trigger an alert sooner than simply noting "a person standing by the door," helping you respond based on the situations context.
While generating simple descriptions of detected objects is useful, understanding intent provides a deeper layer of insight. Instead of just recognizing "what" is in a scene, Frigate's default prompts aim to infer "why" it might be there or "what" it could do next. Descriptions tell you what's happening, but intent gives context. For instance, a person walking toward a door might seem like a visitor, but if they're moving quickly after hours, you can infer a potential break-in attempt. Detecting a person loitering near a door at night can trigger an alert sooner than simply noting "a person standing by the door," helping you respond based on the situation's context.
## Custom Prompts
@@ -33,7 +37,18 @@ Prompts can use variable replacements `{label}`, `{sub_label}`, and `{camera}` t
:::
You are also able to define custom prompts in your configuration.
You can define custom prompts at the global level and per-object type. To configure custom prompts:
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Global configuration > Objects" />.
- Expand the **GenAI object config** section
- Set **Caption prompt** to your custom prompt text
- Under **Object prompts**, add entries keyed by object type (e.g., `person`, `car`) with custom prompts for each
</TabItem>
<TabItem value="yaml">
```yaml
genai:
@@ -49,7 +64,25 @@ objects:
car: "Observe the primary vehicle in these images. Focus on its movement, direction, or purpose (e.g., parking, approaching, circling). If it's a delivery vehicle, mention the company."
```
Prompts can also be overridden at the camera level to provide a more detailed prompt to the model about your specific camera, if you desire.
</TabItem>
</ConfigTabs>
Prompts can also be overridden at the camera level to provide a more detailed prompt to the model about your specific camera. To configure camera-level overrides:
<ConfigTabs>
<TabItem value="ui">
1. Navigate to <NavPath path="Settings > Camera configuration > Objects" /> for the desired camera.
- Expand the **GenAI object config** section
- Set **Enable GenAI** to on
- Set **Use snapshots** to on if desired
- Set **Caption prompt** to a camera-specific prompt
- Under **Object prompts**, add entries keyed by object type with camera-specific prompts
- Set **GenAI objects** to the list of object types that should receive descriptions (e.g., `person`, `cat`)
- Set **Required zones** to limit descriptions to objects in specific zones (e.g., `steps`)
</TabItem>
<TabItem value="yaml">
```yaml
cameras:
@@ -69,6 +102,9 @@ cameras:
- steps
```
</TabItem>
</ConfigTabs>
### Experiment with prompts
Many providers also have a public facing chat interface for their models. Download a couple of different thumbnails or snapshots from Frigate and try new things in the playground to get descriptions to your liking before updating the prompt in Frigate.