Review description updates (#20723)

* Update docs for review descriptions

* Add logging for context tokens used

* Incrase number of images due to lower than expected context usage

* Re-balance the suspicious activity checks

* Adjustments to context sizing

* optimize context usage

* Adjust context usage

* Make title more direct

* Update docs

docs/docs/configuration/genai/review_summaries.md

@@ -7,38 +7,67 @@ Generative AI can be used to automatically generate structured summaries of revi
 
 Requests for a summary are requested automatically to your AI provider for alert review items when the activity has ended, they can also be optionally enabled for detections as well.
 
-Generative AI review summaries can also be toggled dynamically for a camera via MQTT with the topic `frigate/<camera_name>/review_descriptions/set`. See the [MQTT documentation](/integrations/mqtt/#frigatecamera_namereviewdescriptionsset).
+Generative AI review summaries can also be toggled dynamically for a [camera via MQTT](/integrations/mqtt/#frigatecamera_namereviewdescriptionsset).
 
 ## Review Summary Usage and Best Practices
 
 Review summaries provide structured JSON responses that are saved for each review item:
 
 ```
-- `scene` (string): A full description including setting, entities, actions, and any plausible supported inferences.
-- `confidence` (float): 0-1 confidence in the analysis.
+- `title` (string): A concise, direct title that describes the purpose or overall action (e.g., "Person taking out trash", "Joe walking dog").
+- `scene` (string): A narrative description of what happens across the sequence from start to finish, including setting, detected objects, and their observable actions.
+- `confidence` (float): 0-1 confidence in the analysis. Higher confidence when objects/actions are clearly visible and context is unambiguous.
 - `other_concerns` (list): List of user-defined concerns that may need additional investigation.
 - `potential_threat_level` (integer): 0, 1, or 2 as defined below.
-
-Threat-level definitions:
-- 0 — Typical or expected activity for this location/time (includes residents, guests, or known animals engaged in normal activities, even if they glance around or scan surroundings).
-- 1 — Unusual or suspicious activity: At least one security-relevant behavior is present **and not explainable by a normal residential activity**.
-- 2 — Active or immediate threat: Breaking in, vandalism, aggression, weapon display.
 ```
 
-This will show in the UI as a list of concerns that each review item has along with the general description.
+This will show in multiple places in the UI to give additional context about each activity, and allow viewing more details when extra attention is required. Frigate's built in notifications will also automatically show the title and description when the data is available.
 
 ### Defining Typical Activity
 
-Each installation and even camera can have different parameters for what is considered suspicious activity. Frigate allows the `activity_context_prompt` to be defined globally and at the camera level, which allows you to define more specifically what should be considered normal activity. It is important that this is not overly specific as it can sway the output of the response. The default `activity_context_prompt` is below:
+Each installation and even camera can have different parameters for what is considered suspicious activity. Frigate allows the `activity_context_prompt` to be defined globally and at the camera level, which allows you to define more specifically what should be considered normal activity. It is important that this is not overly specific as it can sway the output of the response.
+
+<details>
+  <summary>Default Activity Context Prompt</summary>
 
 ```
-- **Zone context is critical**: Private enclosed spaces (back yards, back decks, fenced areas, inside garages) are resident territory where brief transient activity, routine tasks, and pet care are expected and normal. Front yards, driveways, and porches are semi-public but still resident spaces where deliveries, parking, and coming/going are routine. Consider whether the zone and activity align with normal residential use.
-- **Person + Pet = Normal Activity**: When both "Person" and "Dog" (or "Cat") are detected together in residential zones, this is routine pet care activity (walking, letting out, playing, supervising). Assign Level 0 unless there are OTHER strong suspicious behaviors present (like testing doors, taking items, etc.). A person with their pet in a residential zone is baseline normal activity.
-- Brief appearances in private zones (back yards, garages) are normal residential patterns.
-- Normal residential activity includes: residents, family members, guests, deliveries, services, maintenance workers, routine property use (parking, unloading, mail pickup, trash removal).
-- Brief movement with legitimate items (bags, packages, tools, equipment) in appropriate zones is routine.
+### Normal Activity Indicators (Level 0)
+- Known/verified people in any zone at any time
+- People with pets in residential areas
+- Deliveries or services during daytime/evening (6 AM - 10 PM): carrying packages to doors/porches, placing items, leaving
+- Services/maintenance workers with visible tools, uniforms, or service vehicles during daytime
+- Activity confined to public areas only (sidewalks, streets) without entering property at any time
+
+### Suspicious Activity Indicators (Level 1)
+- **Testing or attempting to open doors/windows/handles on vehicles or buildings** — ALWAYS Level 1 regardless of time or duration
+- **Unidentified person in private areas (driveways, near vehicles/buildings) during late night/early morning (11 PM - 5 AM)** — ALWAYS Level 1 regardless of activity or duration
+- Taking items that don't belong to them (packages, objects from porches/driveways)
+- Climbing or jumping fences/barriers to access property
+- Attempting to conceal actions or items from view
+- Prolonged loitering: remaining in same area without visible purpose throughout most of the sequence
+
+### Critical Threat Indicators (Level 2)
+- Holding break-in tools (crowbars, pry bars, bolt cutters)
+- Weapons visible (guns, knives, bats used aggressively)
+- Forced entry in progress
+- Physical aggression or violence
+- Active property damage or theft in progress
+
+### Assessment Guidance
+Evaluate in this order:
+
+1. **If person is verified/known** → Level 0 regardless of time or activity
+2. **If person is unidentified:**
+   - Check time: If late night/early morning (11 PM - 5 AM) AND in private areas (driveways, near vehicles/buildings) → Level 1
+   - Check actions: If testing doors/handles, taking items, climbing → Level 1
+   - Otherwise, if daytime/evening (6 AM - 10 PM) with clear legitimate purpose (delivery, service worker) → Level 0
+3. **Escalate to Level 2 if:** Weapons, break-in tools, forced entry in progress, violence, or active property damage visible (escalates from Level 0 or 1)
+
+The mere presence of an unidentified person in private areas during late night hours is inherently suspicious and warrants human review, regardless of what activity they appear to be doing or how brief the sequence is.
 ```
 
+</details>
+
 ### Image Source
 
 By default, review summaries use preview images (cached preview frames) which have a lower resolution but use fewer tokens per image. For better image quality and more detailed analysis, you can configure Frigate to extract frames directly from recordings at a higher resolution:
@@ -50,11 +79,19 @@ review:
     image_source: recordings # Options: "preview" (default) or "recordings"
 ```
 
-When using `recordings`, frames are extracted at 480p resolution (480px height), providing better detail for the LLM while being mindful of context window size. This is particularly useful for scenarios where fine details matter, such as identifying license plates, reading text, or analyzing distant objects. Note that using recordings will:
+When using `recordings`, frames are extracted at 480px height while maintaining the camera's original aspect ratio, providing better detail for the LLM while being mindful of context window size. This is particularly useful for scenarios where fine details matter, such as identifying license plates, reading text, or analyzing distant objects.
+
+The number of frames sent to the LLM is dynamically calculated based on:
+
+- Your LLM provider's context window size
+- The camera's resolution and aspect ratio (ultrawide cameras like 32:9 use more tokens per image)
+- The image source (recordings use more tokens than preview images)
+
+Frame counts are automatically optimized to use ~98% of the available context window while capping at 20 frames maximum to ensure reasonable inference times. Note that using recordings will:
 
 - Provide higher quality images to the LLM (480p vs 180p preview images)
-- Use more tokens per image (~200-300 tokens vs ~100 tokens for preview)
-- Result in fewer frames being sent to stay within context limits (typically 6-12 frames vs 8-20 frames)
+- Use more tokens per image due to higher resolution
+- Result in fewer frames being sent for ultrawide cameras due to larger image size
 - Require that recordings are enabled for the camera
 
 If recordings are not available for a given time period, the system will automatically fall back to using preview frames.

docs/docs/configuration/reference.md

@@ -429,9 +429,14 @@ review:
     alerts: True
     # Optional: Enable GenAI review summaries for detections (default: shown below)
     detections: False
+    # Optional: Activity Context Prompt to give context to the GenAI what activity is and is not suspicious.
+    # It is important to be direct and detailed. See documentation for the default prompt structure.
+    activity_context_prompt: """Define what is and is not suspicious
+"""
     # Optional: Image source for GenAI (default: preview)
-    # Options: "preview" (uses cached preview frames at 180p) or "recordings" (extracts frames from recordings at 480p)
-    # Using "recordings" provides better image quality but uses ~2-3x more tokens per image (~200-300 vs ~100 tokens)
+    # Options: "preview" (uses cached preview frames at ~180p) or "recordings" (extracts frames from recordings at 480p)
+    # Using "recordings" provides better image quality but uses more tokens per image.
+    # Frame count is automatically calculated based on context window size, aspect ratio, and image source (capped at 20 frames).
     image_source: preview
     # Optional: Additional concerns that the GenAI should make note of (default: None)
     additional_concerns:

frigate/config/camera/review.py

@@ -106,36 +106,38 @@ class GenAIReviewConfig(FrigateBaseModel):
     )
     activity_context_prompt: str = Field(
         default="""### Normal Activity Indicators (Level 0)
-- Known/verified people in any zone
+- Known/verified people in any zone at any time
 - People with pets in residential areas
-- Brief activity near vehicles: approaching vehicles, brief standing, then leaving or entering vehicle (unloading, loading, checking something). Very short sequences (under 15 seconds) of vehicle access during typical hours (6 AM - 10 PM) are almost always normal.
-- Deliveries or services: brief approach to doors/porches, standing briefly, placing or retrieving items, then leaving
-- Access to private areas: entering back yards, garages, or homes (with or without visible purpose in frame)
-- Brief movement through semi-public areas (driveways, front yards) with items or approaching structure/vehicle
-- Activity on public areas only (sidewalks, streets) without entering property
-- Services/maintenance workers with tools, uniforms, or vehicles
+- Deliveries or services during daytime/evening (6 AM - 10 PM): carrying packages to doors/porches, placing items, leaving
+- Services/maintenance workers with visible tools, uniforms, or service vehicles during daytime
+- Activity confined to public areas only (sidewalks, streets) without entering property at any time
 
 ### Suspicious Activity Indicators (Level 1)
-- Testing or attempting to open doors/windows on vehicles or buildings
-- Taking items that don't belong to them (stealing packages, objects from porches/driveways)
+- **Testing or attempting to open doors/windows/handles on vehicles or buildings** — ALWAYS Level 1 regardless of time or duration
+- **Unidentified person in private areas (driveways, near vehicles/buildings) during late night/early morning (11 PM - 5 AM)** — ALWAYS Level 1 regardless of activity or duration
+- Taking items that don't belong to them (packages, objects from porches/driveways)
 - Climbing or jumping fences/barriers to access property
 - Attempting to conceal actions or items from view
-- Prolonged presence without purpose: remaining in same area (near vehicles, private zones) throughout most/all of the sequence without clear activity or task. Brief stops (a few seconds of standing) are normal; sustained presence (most of the duration) without interaction is concerning.
-- Activity at unusual hours (11 PM - 5 AM) combined with suspicious behavior patterns. Normal commute/daytime hours (6 AM - 6 PM) do not increase suspicion by themselves.
+- Prolonged loitering: remaining in same area without visible purpose throughout most of the sequence
 
 ### Critical Threat Indicators (Level 2)
 - Holding break-in tools (crowbars, pry bars, bolt cutters)
 - Weapons visible (guns, knives, bats used aggressively)
 - Forced entry in progress
 - Physical aggression or violence
-- Active property damage or theft
+- Active property damage or theft in progress
 
 ### Assessment Guidance
-**Default to Level 0** for brief activity during normal hours. When evaluating, first check if it matches Normal Activity Indicators. Very short sequences (under 15 seconds) of vehicle access, deliveries, or movement through property during typical hours (6 AM - 11 PM) should be Level 0 unless there are clear suspicious actions visible (testing doors, stealing, climbing barriers).
+Evaluate in this order:
 
-Only assign Level 1 if the activity shows clear suspicious behaviors: testing access points, stealing items, lingering throughout most of the sequence without task, climbing barriers, or other explicit violations. Brief activity with apparent purpose (approaching vehicle, delivery, passing through) is Level 0.
+1. **If person is verified/known** → Level 0 regardless of time or activity
+2. **If person is unidentified:**
+   - Check time: If late night/early morning (11 PM - 5 AM) AND in private areas (driveways, near vehicles/buildings) → Level 1
+   - Check actions: If testing doors/handles, taking items, climbing → Level 1
+   - Otherwise, if daytime/evening (6 AM - 10 PM) with clear legitimate purpose (delivery, service worker) → Level 0
+3. **Escalate to Level 2 if:** Weapons, break-in tools, forced entry in progress, violence, or active property damage visible (escalates from Level 0 or 1)
 
-Consider duration, time, zone, and actions holistically. Brief is normal; sustained suspicious behavior is concerning.""",
+The mere presence of an unidentified person in private areas during late night hours is inherently suspicious and warrants human review, regardless of what activity they appear to be doing or how brief the sequence is.""",
         title="Custom activity context prompt defining normal and suspicious activity patterns for this property.",
     )
 

frigate/data_processing/post/review_descriptions.py

@@ -49,38 +49,51 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         self.review_descs_dps.start()
 
     def calculate_frame_count(
-        self, image_source: ImageSourceEnum = ImageSourceEnum.preview
+        self,
+        camera: str,
+        image_source: ImageSourceEnum = ImageSourceEnum.preview,
+        height: int = 480,
     ) -> int:
-        """Calculate optimal number of frames based on context size and image source."""
+        """Calculate optimal number of frames based on context size, image source, and resolution.
+
+        Token usage varies by resolution: larger images (ultrawide aspect ratios) use more tokens.
+        Estimates ~1 token per 1250 pixels. Targets 98% context utilization with safety margin.
+        Capped at 20 frames.
+        """
         context_size = self.genai_client.get_context_size()
+        camera_config = self.config.cameras[camera]
+
+        detect_width = camera_config.detect.width
+        detect_height = camera_config.detect.height
+        aspect_ratio = detect_width / detect_height
 
         if image_source == ImageSourceEnum.recordings:
-            # With recordings at 480p resolution (480px height), each image uses ~200-300 tokens
-            # This is ~2-3x more than preview images, so we reduce frame count accordingly
-            # to avoid exceeding context limits and maintain reasonable inference times
-            if context_size > 14000:
-                return 16
-            elif context_size > 12000:
-                return 14
-            elif context_size > 10000:
-                return 12
-            elif context_size > 6000:
-                return 10
-            elif context_size > 4000:
-                return 8
+            if aspect_ratio >= 1:
+                # Landscape or square: constrain height
+                width = int(height * aspect_ratio)
             else:
-                return 6
+                # Portrait: constrain width
+                width = height
+                height = int(width / aspect_ratio)
         else:
-            # With preview images (180px height), each image uses ~100 tokens
-            # We can send more frames since they're lower resolution
-            if context_size > 10000:
-                return 20
-            elif context_size > 6000:
-                return 16
-            elif context_size > 4000:
-                return 12
+            if aspect_ratio >= 1:
+                # Landscape or square: constrain height
+                target_height = 180
+                width = int(target_height * aspect_ratio)
+                height = target_height
             else:
-                return 8
+                # Portrait: constrain width
+                target_width = 180
+                width = target_width
+                height = int(target_width / aspect_ratio)
+
+        pixels_per_image = width * height
+        tokens_per_image = pixels_per_image / 1250
+        prompt_tokens = 3500
+        available_tokens = context_size * 0.98 - prompt_tokens
+        max_frames = int(available_tokens / tokens_per_image)
+
+        return min(max(max_frames, 3), 20)
 
     def process_data(self, data, data_type):
         self.metrics.review_desc_dps.value = self.review_descs_dps.eps()
@@ -258,7 +271,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
             all_frames.append(os.path.join(preview_dir, file))
 
         frame_count = len(all_frames)
-        desired_frame_count = self.calculate_frame_count()
+        desired_frame_count = self.calculate_frame_count(camera)
 
         if frame_count <= desired_frame_count:
             return all_frames
@@ -281,7 +294,9 @@ class ReviewDescriptionProcessor(PostProcessorApi):
     ) -> list[bytes]:
         """Get frames from recordings at specified timestamps."""
         duration = end_time - start_time
-        desired_frame_count = self.calculate_frame_count(ImageSourceEnum.recordings)
+        desired_frame_count = self.calculate_frame_count(
+            camera, ImageSourceEnum.recordings, height
+        )
 
         # Calculate evenly spaced timestamps throughout the duration
         if desired_frame_count == 1:

frigate/genai/__init__.py

@@ -81,7 +81,7 @@ Your task is to analyze the sequence of images ({len(thumbnails)} total) taken i
 Your task is to provide a clear, accurate description of the scene that:
 1. States exactly what is happening based on observable actions and movements.
 2. Evaluates the activity against the Normal and Suspicious Activity Indicators above.
-3. Assigns a potential_threat_level based on the definitions below, applying them consistently.
+3. Assigns a potential_threat_level (0, 1, or 2) based on the threat level indicators defined above, applying them consistently.
 
 **Use the activity patterns above as guidance to calibrate your assessment. Match the activity against both normal and suspicious indicators, then use your judgment based on the complete context.**
 
@@ -94,24 +94,18 @@ When forming your description:
 - Note visible details such as clothing, items being carried or placed, tools or equipment present, and how they interact with the property or objects.
 - Consider the full sequence chronologically: what happens from start to finish, how duration and actions relate to the location and objects involved.
 - **Use the actual timestamp provided in "Activity started at"** below for time of day context—do not infer time from image brightness or darkness. Unusual hours (late night/early morning) should increase suspicion when the observable behavior itself appears questionable. However, recognize that some legitimate activities can occur at any hour.
-- **Consider duration as a primary factor**: Very short sequences (under 15 seconds) during normal hours (6 AM - 11 PM) are almost always Level 0 unless explicit suspicious actions are visible (testing doors, stealing, climbing). Brief activity with apparent purpose (vehicle access, deliveries, passing through) is normal.
-- **Weigh all evidence holistically**: Match the activity against both the normal and suspicious patterns above, then evaluate based on the complete context (zone, objects, time, actions, duration). Activities matching normal patterns should be Level 0. Activities matching suspicious indicators should be Level 1. Use your judgment for edge cases.
+- **Consider duration as a primary factor**: Apply the duration thresholds defined in the activity patterns above. Brief sequences during normal hours with apparent purpose typically indicate normal activity unless explicit suspicious actions are visible.
+- **Weigh all evidence holistically**: Match the activity against the normal and suspicious patterns defined above, then evaluate based on the complete context (zone, objects, time, actions, duration). Apply the threat level indicators consistently. Use your judgment for edge cases.
 
 ## Response Format
 
 Your response MUST be a flat JSON object with:
-- `title` (string): A concise, one-sentence title that captures the main activity. Use names from "Objects in Scene" based on what you visually observe. If you see both a name and an unidentified object of the same type but visually observe only one person/object, use ONLY the name. Examples: "Joe walking dog in backyard", "Joe near vehicle in driveway", "Joe and a person on front porch".
+- `title` (string): A concise, direct title that describes the purpose or overall action, not just what you literally see. Use names from "Objects in Scene" based on what you visually observe. If you see both a name and an unidentified object of the same type but visually observe only one person/object, use ONLY the name. Examples: "Joe walking dog", "Person taking out trash", "Joe accessing vehicle", "Joe and person on front porch".
 - `scene` (string): A narrative description of what happens across the sequence from start to finish. **Only describe actions you can actually observe happening in the frames provided.** Do not infer or assume actions that aren't visible (e.g., if you see someone walking but never see them sit, don't say they sat down). Include setting, detected objects, and their observable actions. Avoid speculation or filling in assumed behaviors. Your description should align with and support the threat level you assign.
 - `confidence` (float): 0-1 confidence in your analysis. Higher confidence when objects/actions are clearly visible and context is unambiguous. Lower confidence when the sequence is unclear, objects are partially obscured, or context is ambiguous.
-- `potential_threat_level` (integer): 0, 1, or 2 as defined below. Your threat level must be consistent with your scene description and the guidance above.
+- `potential_threat_level` (integer): 0, 1, or 2 as defined in "Normal Activity Patterns for This Property" above. Your threat level must be consistent with your scene description and the guidance above.
 {get_concern_prompt()}
 
-## Threat Level Definitions
-
-- 0 — **Normal activity**: The observable activity matches Normal Activity Indicators (brief vehicle access, deliveries, known people, pet activity, services). **Very short sequences (under 15 seconds) during normal hours (6 AM - 11 PM) with apparent purpose (vehicle access, deliveries, passing through) are Level 0.** Brief activities are generally normal.
-- 1 — **Potentially suspicious**: The observable activity matches Suspicious Activity Indicators (testing access, stealing items, climbing barriers, lingering throughout most of sequence without task, unusual hours 11 PM - 5 AM with suspicious behavior). **Requires clear suspicious actions visible in frames, not just ambiguity or brief presence.**
-- 2 — **Immediate threat**: Clear evidence of active criminal activity, forced entry, break-in, vandalism, aggression, weapons, theft in progress, or property damage.
-
 ## Sequence Details
 
 - Frame 1 = earliest, Frame {len(thumbnails)} = latest

frigate/genai/ollama.py

@@ -64,8 +64,11 @@ class OllamaClient(GenAIClient):
                 images=images if images else None,
                 **self.provider_options,
             )
+            logger.debug(
+                f"Ollama tokens used: eval_count={result.get('eval_count')}, prompt_eval_count={result.get('prompt_eval_count')}"
+            )
             return result["response"].strip()
-        except (TimeoutException, ResponseError) as e:
+        except (TimeoutException, ResponseError, ConnectionError) as e:
             logger.warning("Ollama returned an error: %s", str(e))
             return None