Update docs for review descriptions

This commit is contained in:
Nicolas Mowen 2025-10-30 05:17:12 -06:00
parent fd1eb64921
commit 80197a4e40
2 changed files with 49 additions and 13 deletions

View File

@ -7,38 +7,70 @@ Generative AI can be used to automatically generate structured summaries of revi
Requests for a summary are requested automatically to your AI provider for alert review items when the activity has ended, they can also be optionally enabled for detections as well. Requests for a summary are requested automatically to your AI provider for alert review items when the activity has ended, they can also be optionally enabled for detections as well.
Generative AI review summaries can also be toggled dynamically for a camera via MQTT with the topic `frigate/<camera_name>/review_descriptions/set`. See the [MQTT documentation](/integrations/mqtt/#frigatecamera_namereviewdescriptionsset). Generative AI review summaries can also be toggled dynamically for a [camera via MQTT](/integrations/mqtt/#frigatecamera_namereviewdescriptionsset).
## Review Summary Usage and Best Practices ## Review Summary Usage and Best Practices
Review summaries provide structured JSON responses that are saved for each review item: Review summaries provide structured JSON responses that are saved for each review item:
``` ```
- `scene` (string): A full description including setting, entities, actions, and any plausible supported inferences. - `title` (string): A concise, one-sentence title that captures the main activity.
- `confidence` (float): 0-1 confidence in the analysis. - `scene` (string): A narrative description of what happens across the sequence from start to finish, including setting, detected objects, and their observable actions.
- `confidence` (float): 0-1 confidence in the analysis. Higher confidence when objects/actions are clearly visible and context is unambiguous.
- `other_concerns` (list): List of user-defined concerns that may need additional investigation. - `other_concerns` (list): List of user-defined concerns that may need additional investigation.
- `potential_threat_level` (integer): 0, 1, or 2 as defined below. - `potential_threat_level` (integer): 0, 1, or 2 as defined below.
Threat-level definitions: Threat-level definitions:
- 0 — Typical or expected activity for this location/time (includes residents, guests, or known animals engaged in normal activities, even if they glance around or scan surroundings). - 0 — **Normal activity**: The observable activity matches Normal Activity Indicators (brief vehicle access, deliveries, known people, pet activity, services). Very short sequences (under 15 seconds) during normal hours (6 AM - 11 PM) with apparent purpose (vehicle access, deliveries, passing through) are Level 0. Brief activities are generally normal.
- 1 — Unusual or suspicious activity: At least one security-relevant behavior is present **and not explainable by a normal residential activity**. - 1 — **Potentially suspicious**: The observable activity matches Suspicious Activity Indicators (testing access, stealing items, climbing barriers, lingering throughout most of sequence without task, unusual hours 11 PM - 5 AM with suspicious behavior). Requires clear suspicious actions visible in frames, not just ambiguity or brief presence.
- 2 — Active or immediate threat: Breaking in, vandalism, aggression, weapon display. - 2 — **Immediate threat**: Clear evidence of active criminal activity, forced entry, break-in, vandalism, aggression, weapons, theft in progress, or property damage.
``` ```
This will show in the UI as a list of concerns that each review item has along with the general description. This will show in the UI as a list of concerns that each review item has along with the general description.
### Defining Typical Activity ### Defining Typical Activity
Each installation and even camera can have different parameters for what is considered suspicious activity. Frigate allows the `activity_context_prompt` to be defined globally and at the camera level, which allows you to define more specifically what should be considered normal activity. It is important that this is not overly specific as it can sway the output of the response. The default `activity_context_prompt` is below: Each installation and even camera can have different parameters for what is considered suspicious activity. Frigate allows the `activity_context_prompt` to be defined globally and at the camera level, which allows you to define more specifically what should be considered normal activity. It is important that this is not overly specific as it can sway the output of the response.
<details>
<summary>Default Activity Context Prompt</summary>
``` ```
- **Zone context is critical**: Private enclosed spaces (back yards, back decks, fenced areas, inside garages) are resident territory where brief transient activity, routine tasks, and pet care are expected and normal. Front yards, driveways, and porches are semi-public but still resident spaces where deliveries, parking, and coming/going are routine. Consider whether the zone and activity align with normal residential use. ### Normal Activity Indicators (Level 0)
- **Person + Pet = Normal Activity**: When both "Person" and "Dog" (or "Cat") are detected together in residential zones, this is routine pet care activity (walking, letting out, playing, supervising). Assign Level 0 unless there are OTHER strong suspicious behaviors present (like testing doors, taking items, etc.). A person with their pet in a residential zone is baseline normal activity. - Known/verified people in any zone
- Brief appearances in private zones (back yards, garages) are normal residential patterns. - People with pets in residential areas
- Normal residential activity includes: residents, family members, guests, deliveries, services, maintenance workers, routine property use (parking, unloading, mail pickup, trash removal). - Brief activity near vehicles: approaching vehicles, brief standing, then leaving or entering vehicle (unloading, loading, checking something). Very short sequences (under 15 seconds) of vehicle access during typical hours (6 AM - 10 PM) are almost always normal.
- Brief movement with legitimate items (bags, packages, tools, equipment) in appropriate zones is routine. - Deliveries or services: brief approach to doors/porches, standing briefly, placing or retrieving items, then leaving
- Access to private areas: entering back yards, garages, or homes (with or without visible purpose in frame)
- Brief movement through semi-public areas (driveways, front yards) with items or approaching structure/vehicle
- Activity on public areas only (sidewalks, streets) without entering property
- Services/maintenance workers with tools, uniforms, or vehicles
### Suspicious Activity Indicators (Level 1)
- Testing or attempting to open doors/windows on vehicles or buildings
- Taking items that don't belong to them (stealing packages, objects from porches/driveways)
- Climbing or jumping fences/barriers to access property
- Attempting to conceal actions or items from view
- Prolonged presence without purpose: remaining in same area (near vehicles, private zones) throughout most/all of the sequence without clear activity or task. Brief stops (a few seconds of standing) are normal; sustained presence (most of the duration) without interaction is concerning.
- Activity at unusual hours (11 PM - 5 AM) combined with suspicious behavior patterns. Normal commute/daytime hours (6 AM - 6 PM) do not increase suspicion by themselves.
### Critical Threat Indicators (Level 2)
- Holding break-in tools (crowbars, pry bars, bolt cutters)
- Weapons visible (guns, knives, bats used aggressively)
- Forced entry in progress
- Physical aggression or violence
- Active property damage or theft
### Assessment Guidance
**Default to Level 0** for brief activity during normal hours. When evaluating, first check if it matches Normal Activity Indicators. Very short sequences (under 15 seconds) of vehicle access, deliveries, or movement through property during typical hours (6 AM - 11 PM) should be Level 0 unless there are clear suspicious actions visible (testing doors, stealing, climbing barriers).
Only assign Level 1 if the activity shows clear suspicious behaviors: testing access points, stealing items, lingering throughout most of the sequence without task, climbing barriers, or other explicit violations. Brief activity with apparent purpose (approaching vehicle, delivery, passing through) is Level 0.
Consider duration, time, zone, and actions holistically. Brief is normal; sustained suspicious behavior is concerning.
``` ```
</details>
### Image Source ### Image Source
By default, review summaries use preview images (cached preview frames) which have a lower resolution but use fewer tokens per image. For better image quality and more detailed analysis, you can configure Frigate to extract frames directly from recordings at a higher resolution: By default, review summaries use preview images (cached preview frames) which have a lower resolution but use fewer tokens per image. For better image quality and more detailed analysis, you can configure Frigate to extract frames directly from recordings at a higher resolution:

View File

@ -429,6 +429,10 @@ review:
alerts: True alerts: True
# Optional: Enable GenAI review summaries for detections (default: shown below) # Optional: Enable GenAI review summaries for detections (default: shown below)
detections: False detections: False
# Optional: Activity Context Prompt to give context to the GenAI what activity is and is not suspicious.
# It is important to be direct and detailed.
activity_context_prompt: """Define what is and is not suspicious
"""
# Optional: Image source for GenAI (default: preview) # Optional: Image source for GenAI (default: preview)
# Options: "preview" (uses cached preview frames at 180p) or "recordings" (extracts frames from recordings at 480p) # Options: "preview" (uses cached preview frames at 180p) or "recordings" (extracts frames from recordings at 480p)
# Using "recordings" provides better image quality but uses ~2-3x more tokens per image (~200-300 vs ~100 tokens) # Using "recordings" provides better image quality but uses ~2-3x more tokens per image (~200-300 vs ~100 tokens)