Genai review summaries (#19473)

* Generate review item summaries with requests

* Adjust logic to only send important items

* Don't mention ladder

* Adjust prompt to be more specific

* Add more relaxed nature for normal activity

* Cleanup summary

* Update ollama client

* Add more directions to analyze the frames in order

* Remove environment from prompt

frigate/genai/__init__.py

@@ -1,5 +1,6 @@
 """Generative AI module for Frigate."""
 
+import datetime
 import importlib
 import logging
 import os
@@ -9,6 +10,7 @@ from typing import Any, Optional
 from playhouse.shortcuts import model_to_dict
 
 from frigate.config import CameraConfig, FrigateConfig, GenAIConfig, GenAIProviderEnum
+from frigate.const import CLIPS_DIR
 from frigate.data_processing.post.types import ReviewMetadata
 from frigate.models import Event
 
@@ -41,6 +43,7 @@ class GenAIClient:
         thumbnails: list[bytes],
         concerns: list[str],
         preferred_language: str | None,
+        debug_save: bool,
     ) -> ReviewMetadata | None:
         """Generate a description for the review item activity."""
         if concerns:
@@ -59,34 +62,39 @@ class GenAIClient:
             language_prompt = ""
 
         context_prompt = f"""
-Please analyze the image(s), which are in chronological order, strictly from the perspective of the {review_data["camera"].replace("_", " ")} security camera.
+Please analyze the sequence of images ({len(thumbnails)} total) taken in chronological order from the perspective of the {review_data["camera"].replace("_", " ")} security camera.
 
-Your task is to provide a **neutral, factual, and objective description** of the scene, while also:
-- Clearly stating **what is happening** based on observable actions and movements.
-- Including **reasonable, evidence-based inferences** about the likely activity or context, but only if directly supported by visible details.
+Your task is to provide a clear, security-focused description of the scene that:
+1. States exactly what is happening based on observable actions and movements.
+2. Identifies and emphasizes behaviors that match patterns of suspicious activity.
+3. Assigns a potential_threat_level based on the definitions below, applying them consistently.
+
+Facts come first, but identifying security risks is the primary goal.
 
 When forming your description:
-- **Facts first**: Describe the time, physical setting, people, and objects exactly as seen.
-- **Then context**: Briefly note plausible purposes or activities (e.g., “appears to be delivering a package” if carrying a box to a door).
-- Clearly separate certain facts (“A person is holding an object with horizontal rungs”) from reasonable inferences (“likely a ladder”).
-- Do not speculate beyond what is visible, and do not imply hostility, criminal intent, or other strong judgments unless there is unambiguous visual evidence.
+- Describe the time, people, and objects exactly as seen. Include any observable environmental changes (e.g., lighting changes triggered by activity).
+- Time of day should **increase suspicion only when paired with unusual or security-relevant behaviors**. Do not raise the threat level for common residential activities (e.g., residents walking pets, retrieving mail, gardening, playing with pets, supervising children) even at unusual hours, unless other suspicious indicators are present.
+- Focus on behaviors that are uncharacteristic of innocent activity: loitering without clear purpose, avoiding cameras, inspecting vehicles/doors, changing behavior when lights activate, scanning surroundings without an apparent benign reason.
+- **Benign context override**: If scanning or looking around is clearly part of an innocent activity (such as playing with a dog, gardening, supervising children, or watching for a pet), do not treat it as suspicious.
 
-Here is information already known:
-- Activity occurred at {review_data["timestamp"].strftime("%I:%M %p")}
-- Detected objects: {review_data["objects"]}
-- Recognized objects: {review_data["recognized_objects"]}
-- Zones involved: {review_data["zones"]}
-
-Your response **MUST** be a flat JSON object with:
+Your response MUST be a flat JSON object with:
 - `scene` (string): A full description including setting, entities, actions, and any plausible supported inferences.
-- `confidence` (float): A number 0-1 for overall confidence in the analysis.
-- `potential_threat_level` (integer, optional): Include only if there is a clear, observable security concern:
-    - 0 = Normal activity is occurring
-    - 1 = Unusual but not overtly threatening
-    - 2 = Suspicious or potentially harmful
-    - 3 = Clear and immediate threat
+- `confidence` (float): 0-1 confidence in the analysis.
+- `potential_threat_level` (integer): 0, 1, or 2 as defined below.
 {concern_prompt}
 
+Threat-level definitions:
+- 0 — Typical or expected activity for this location/time (includes residents, guests, or known animals engaged in normal activities, even if they glance around or scan surroundings).
+- 1 — Unusual or suspicious activity: At least one security-relevant behavior is present **and not explainable by a normal residential activity**.
+- 2 — Active or immediate threat: Breaking in, vandalism, aggression, weapon display.
+
+Sequence details:
+- Frame 1 = earliest, Frame 10 = latest
+- Activity occurred at {review_data["timestamp"].strftime("%I:%M %p")}
+- Detected objects: {list(set(review_data["objects"]))}
+- Recognized objects: {list(set(review_data["recognized_objects"])) or "None"}
+- Zones involved: {review_data["zones"]}
+
 **IMPORTANT:**
 - Values must be plain strings, floats, or integers — no nested objects, no extra commentary.
 {language_prompt}
@@ -94,6 +102,16 @@ Your response **MUST** be a flat JSON object with:
         logger.debug(
             f"Sending {len(thumbnails)} images to create review description on {review_data['camera']}"
         )
+
+        if debug_save:
+            with open(
+                os.path.join(
+                    CLIPS_DIR, "genai-requests", review_data["id"], "prompt.txt"
+                ),
+                "w",
+            ) as f:
+                f.write(context_prompt)
+
         response = self._send(context_prompt, thumbnails)
 
         if response:
@@ -112,6 +130,36 @@ Your response **MUST** be a flat JSON object with:
         else:
             return None
 
+    def generate_review_summary(
+        self, start_ts: float, end_ts: float, segments: list[dict[str, Any]]
+    ) -> str | None:
+        """Generate a summary of review item descriptions over a period of time."""
+        time_range = f"{datetime.datetime.fromtimestamp(start_ts).strftime('%I:%M %p')} to {datetime.datetime.fromtimestamp(end_ts).strftime('%I:%M %p')}"
+        timeline_summary_prompt = f"""
+You are a security officer. Time range: {time_range}.
+Input: JSON list with "scene", "confidence", "potential_threat_level" (1-2), "other_concerns".
+Write a report:
+
+Security Summary - {time_range}
+[One-sentence overview of activity]
+[Chronological bullet list of events with timestamps if in scene]
+[Final threat assessment]
+
+Rules:
+- List events in order.
+- Highlight potential_threat_level ≥ 1 with exact times.
+- Note any of the additional concerns which are present.
+- Note unusual activity even if not threats.
+- If no threats: "Final assessment: Only normal activity observed during this period."
+- No commentary, questions, or recommendations.
+- Output only the report.
+        """
+
+        for item in segments:
+            timeline_summary_prompt += f"\n{item}"
+
+        return self._send(timeline_summary_prompt, [])
+
     def generate_object_description(
         self,
         camera_config: CameraConfig,