Implement annotated frames for GenAI Review (#24379)

* Implement annotated frames mode for GenAI reviews to improve models with lacking temporal understanding

* Updates

* Improve debug sharing

* Do not number objects

* Fix assumptions

* Remove unhelpful content

* Improve object data sent as part of prompt

* Cleanup ollama dumbness

* Bind db

* Fixes

* Cleanup

docs/docs/configuration/genai/config.md

@@ -63,11 +63,11 @@ Running Generative AI models on CPU is not recommended, as high inference times
 
 You must use a vision-capable model with Frigate. The following models are recommended for local deployment of the `descriptions` and `chat` roles:
 
-| Model               | Notes                                                                                                                                                                |
-| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `qwen3-vl`          | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object.                                              |
-| `qwen3.6`/`qwen3.8` | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
-| `gemma4`            | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action.                                     |
+| Model               | Review [frame mode](/configuration/genai/genai_review#frame-mode) | Notes                                                                                                                                                                |
+| ------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `qwen3-vl`          | `frames`                                                              | Strong visual and situational understanding, enhanced ability to identify smaller objects and interactions with object. Follows a sequence of frames on its own.     |
+| `qwen3.6`/`qwen3.8` | `frames`                                                              | Strong situational understanding, but missing DeepStack from qwen3-vl leading to worse performance for identifying objects in people's hand and other small details. |
+| `gemma4`            | `annotated_frames`                                                    | Strong situational understanding, sometimes resorts to more vague terms like 'interacts' instead of assigning a specific action. Loses track of activity that repeats or reverses, so it benefits from annotated frames. |
 
 #### Embedding models
 

docs/docs/configuration/genai/review_summaries.md

@@ -192,6 +192,43 @@ review:
 </TabItem>
 </ConfigTabs>
 
+### Frame Mode
+
+Review items are sent to the model as a sequence of still frames. Some models follow that sequence well on their own; others lose track of activity that repeats or reverses, and describe a single trip when the subject actually made several. The `frame_mode` option controls how those frames are presented.
+
+- `frames` (default): the prompt followed by the frames, exactly as earlier versions of Frigate sent them.
+- `annotated_frames`: each frame is preceded by its frame number and elapsed time, along with notes describing what the object tracker recorded at that moment, such as an object being first detected, starting to move, turning around, stopping, or no longer being detected.
+
+The notes come from tracking data rather than from the images, so they describe activity the model may not have picked up on its own. In testing with a person carrying three waste bins to the curb one at a time, `gemma4` described a single trip on every attempt with `frames`, and consistently described multiple trips with `annotated_frames`. Models that already handle these sequences well, such as the `qwen3-vl` family, gain little and should stay on `frames`.
+
+Annotated mode also caps the number of frames, since the notes already establish the order of events and extra near-duplicate frames tend to crowd out the middle of a clip. Longer review items are sampled more sparsely as a result, and typically use fewer tokens than `frames` mode for the same item.
+
+:::note
+
+Annotated mode needs tracking data for the review item. If none is available, Frigate falls back to sending plain frames for that item.
+
+:::
+
+<ConfigTabs>
+<TabItem value="ui">
+
+Navigate to <NavPath path="Settings > Global configuration > Review" />.
+
+- Set **GenAI config > Frame mode** to the desired mode (e.g., `annotated_frames`)
+
+</TabItem>
+<TabItem value="yaml">
+
+```yaml {4}
+review:
+  genai:
+    enabled: true
+    frame_mode: annotated_frames
+```
+
+</TabItem>
+</ConfigTabs>
+
 ### Response Style
 
 Different models respond to the built-in prompt with very different writing styles: some produce natural narration while others sound short and mechanical. The `response_style` option selects a writing style preset that rewords the prompt's instructions for the user-facing fields (the title, short summary, and scene description). Presets replace those instructions rather than adding extra ones, so the model never receives competing style directions.

frigate/config/camera/review.py

@@ -9,6 +9,7 @@ __all__ = [
     "DetectionsConfig",
     "AlertsConfig",
     "ImageSourceEnum",
+    "ReviewFrameModeEnum",
     "ReviewResponseStyleEnum",
 ]
 
@@ -20,6 +21,13 @@ class ImageSourceEnum(str, Enum):
     recordings = "recordings"
 
 
+class ReviewFrameModeEnum(str, Enum):
+    """How review frames are presented to the GenAI provider."""
+
+    frames = "frames"
+    annotated_frames = "annotated_frames"
+
+
 class ReviewResponseStyleEnum(str, Enum):
     """Writing style presets for GenAI review descriptions."""
 
@@ -153,6 +161,11 @@ class GenAIReviewConfig(FrigateBaseModel):
         description="Preferred language to request from the GenAI provider for generated responses.",
         default=None,
     )
+    frame_mode: ReviewFrameModeEnum = Field(
+        default=ReviewFrameModeEnum.frames,
+        title="Frame mode",
+        description="How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses.",
+    )
     response_style: ReviewResponseStyleEnum = Field(
         default=ReviewResponseStyleEnum.default,
         title="Response style",

frigate/data_processing/post/review_annotations.py

@@ -0,0 +1,394 @@
+"""Frame annotations derived from object tracking data.
+
+Builds short notes describing what changed during a review item, keyed to the
+frames sampled from it. Everything here comes from tracked object data already
+in the database (each event's `path_data` trajectory and the timeline's
+stationary/active changes), so the notes can be stated to the model as fact
+rather than as something it must perceive.
+"""
+
+import logging
+import math
+from collections.abc import Sequence
+from typing import Any
+
+from frigate.models import Event, Timeline
+
+logger = logging.getLogger(__name__)
+
+# Movement smaller than this (normalized frame units) between two path points
+# is treated as the object holding still rather than travelling.
+STILL_THRESHOLD = 0.02
+
+# A heading change beyond this (dot product against the leg's own heading)
+# counts as the object turning back rather than curving.
+REVERSAL_DOT = -0.3
+
+# A run of travel shorter than this (normalized frame units) is treated as
+# milling about rather than going somewhere. Without it, a subject pacing in
+# one spot produces a burst of contradictory "turns around" notes on a single
+# frame.
+MIN_LEG_DISTANCE = 0.08
+
+# Movement that begins within this many seconds of detection is folded into
+# the detection note, so each arrival reads as one event instead of several.
+DETECT_MOVE_MERGE_SECONDS = 2.0
+
+STATE_CHANGE_PHRASES = {
+    "stationary": "has stopped moving",
+    "active": "starts moving again",
+}
+
+Point = tuple[float, float, float]
+Leg = tuple[int, int]
+
+
+def describe_position(x: float, y: float) -> str:
+    """Name a normalized frame position in plain terms."""
+    horizontal = "left" if x < 0.34 else ("right" if x > 0.66 else "center")
+    vertical = "top" if y < 0.34 else ("bottom" if y > 0.66 else "middle")
+
+    if horizontal == "center" and vertical == "middle":
+        return "the middle of the frame"
+
+    if horizontal == "center":
+        return f"the {vertical} of the frame"
+
+    if vertical == "middle":
+        return f"the {horizontal} of the frame"
+
+    return f"the {vertical} {horizontal} of the frame"
+
+
+def describe_heading(dx: float, dy: float) -> str:
+    """Name a direction of travel in frame terms.
+
+    y grows downward in normalized coordinates, so a falling y reads as moving
+    toward the top of the frame.
+    """
+    parts = []
+
+    if abs(dy) > abs(dx) * 0.4:
+        parts.append("down" if dy > 0 else "up")
+
+    if abs(dx) > abs(dy) * 0.4:
+        parts.append("right" if dx > 0 else "left")
+
+    return " and ".join(parts) if parts else "in place"
+
+
+def event_name(event: dict[str, Any]) -> str:
+    """Name an object for the notes, e.g. 'a person' or 'waste bin "Compost"'.
+
+    Objects are never numbered or given track identifiers. Frigate opens a new
+    tracked object whenever a subject is re-detected, so the tracking data
+    cannot say whether two entries are the same subject, and the notes stay
+    ambiguous rather than implying either answer.
+    """
+    label = str(event["label"]).replace("_", " ").replace("-verified", "")
+    sub_label = event.get("sub_label")
+
+    if sub_label:
+        return f'{label} "{sub_label}"'
+
+    article = "an" if label[:1].lower() in "aeiou" else "a"
+    return f"{article} {label}"
+
+
+def path_legs(points: list[Point]) -> list[Leg]:
+    """Split a trajectory into runs of travel in a consistent direction.
+
+    A leg ends when the subject starts moving back against the direction that
+    leg established, and only once the leg has covered MIN_LEG_DISTANCE, so
+    jitter around a standing subject does not register as a turn.
+
+    Returns (start, end) index pairs into `points`.
+    """
+    legs: list[Leg] = []
+    start = 0
+
+    for i in range(1, len(points)):
+        lx = points[i][0] - points[start][0]
+        ly = points[i][1] - points[start][1]
+        leg_distance = (lx * lx + ly * ly) ** 0.5
+
+        if leg_distance < MIN_LEG_DISTANCE:
+            continue
+
+        sx = points[i][0] - points[i - 1][0]
+        sy = points[i][1] - points[i - 1][1]
+        step = (sx * sx + sy * sy) ** 0.5
+
+        if step < STILL_THRESHOLD:
+            continue
+
+        dot = (lx / leg_distance) * (sx / step) + (ly / leg_distance) * (sy / step)
+
+        if dot < REVERSAL_DOT:
+            legs.append((start, i - 1))
+            start = i - 1
+
+    if start < len(points) - 1:
+        legs.append((start, len(points) - 1))
+
+    return [
+        (a, b)
+        for a, b in legs
+        if ((points[b][0] - points[a][0]) ** 2 + (points[b][1] - points[a][1]) ** 2)
+        ** 0.5
+        >= MIN_LEG_DISTANCE
+    ]
+
+
+def path_points(path_data: list[Any]) -> list[Point]:
+    """Flatten path_data into (x, y, timestamp) tuples, or [] if malformed."""
+    try:
+        return [(p[0][0], p[0][1], p[1]) for p in path_data or []]
+    except (IndexError, TypeError):
+        logger.debug("Malformed path_data, skipping trajectory notes")
+        return []
+
+
+def leg_start_time(points: list[Point], leg: Leg) -> float:
+    """When a leg's movement actually began.
+
+    path_data always keeps an object's first two samples, so a leg can open
+    with points recorded long before the object moved. The first sample that
+    has left the leg's origin is the earliest evidence of movement.
+    """
+    a, b = leg
+    x0, y0, t0 = points[a]
+
+    for x, y, t in points[a + 1 : b + 1]:
+        if math.hypot(x - x0, y - y0) >= STILL_THRESHOLD:
+            return t
+
+    return t0
+
+
+def leg_heading(points: list[Point], leg: Leg) -> str:
+    a, b = leg
+    return describe_heading(points[b][0] - points[a][0], points[b][1] - points[a][1])
+
+
+def leg_phrase(points: list[Point], leg: Leg, first: bool) -> str:
+    """Describe the start of a leg, e.g. 'turns around at ... and heads left'."""
+    heading = leg_heading(points, leg)
+    place = describe_position(points[leg[0]][0], points[leg[0]][1])
+
+    if first:
+        return f"starts moving {heading} from {place}"
+
+    return f"turns around at {place} and heads {heading}"
+
+
+def path_moments(path_data: list[Any]) -> list[tuple[float, str]]:
+    """Key moments in one trajectory as (timestamp, phrase).
+
+    Emits one note per leg of travel. Where the last leg ends is left out:
+    path_data only records significant movement, so its final point cannot
+    distinguish an object coming to rest from one leaving the frame.
+    """
+    points = path_points(path_data)
+
+    if len(points) < 2:
+        return []
+
+    return [
+        (leg_start_time(points, leg), leg_phrase(points, leg, index == 0))
+        for index, leg in enumerate(path_legs(points))
+    ]
+
+
+def build_timeline(
+    events: list[dict[str, Any]],
+    span_end: float,
+    state_changes: Sequence[dict[str, Any]] = (),
+) -> list[tuple[float, str]]:
+    """All annotated moments across every event, in time order.
+
+    Only changes are noted, since those are what sparse frames miss; an
+    object's state at the end of the clip is visible in the last frame.
+    `span_end` is the timestamp of the last sampled frame, and moments past it
+    describe nothing the model can see. A track ending means the object
+    stopped being detected, which may or may not mean it left the frame.
+
+    `state_changes` are timeline rows (timestamp, source_id, class_type); the
+    stationary and active ones become "has stopped moving" / "starts moving
+    again". Frigate only marks an object stationary after it has been still
+    for a while, which the past-tense wording reflects.
+
+    Each object keeps its own notes. Folding an object into the note of the
+    person moving it ("alongside ...") was tried and made models lose track of
+    where the object went.
+    """
+    changes_by_event: dict[str, list[tuple[float, str]]] = {}
+
+    for change in state_changes:
+        phrase = STATE_CHANGE_PHRASES.get(change["class_type"])
+
+        if phrase:
+            changes_by_event.setdefault(change["source_id"], []).append(
+                (change["timestamp"], phrase)
+            )
+
+    timeline: list[tuple[float, str]] = []
+
+    for event in sorted(events, key=lambda e: e["start_time"]):
+        # Frame extraction can come up short at the end of a clip, leaving
+        # objects that only appear after the last frame we actually have.
+        if event["start_time"] > span_end:
+            continue
+
+        points = path_points(event.get("path_data") or [])
+        legs = path_legs(points) if len(points) >= 2 else []
+        name = event_name(event)
+        detected_at = event["start_time"]
+        where = describe_position(points[0][0], points[0][1]) if points else "the frame"
+        merges = bool(legs) and leg_start_time(points, legs[0]) - detected_at <= (
+            DETECT_MOVE_MERGE_SECONDS
+        )
+        remaining = list(enumerate(legs))
+        moments: list[tuple[float, str]] = []
+
+        if merges:
+            heading = leg_heading(points, legs[0])
+            timeline.append(
+                (detected_at, f"{name} first detected at {where}, moving {heading}")
+            )
+            remaining = remaining[1:]
+        else:
+            timeline.append((detected_at, f"{name} first detected at {where}"))
+
+        for index, leg in remaining:
+            moments.append(
+                (
+                    leg_start_time(points, leg),
+                    leg_phrase(points, leg, index == 0),
+                )
+            )
+
+        moments.extend(
+            (timestamp, phrase)
+            for timestamp, phrase in changes_by_event.get(event["id"], [])
+            if timestamp >= detected_at
+        )
+
+        for timestamp, phrase in moments:
+            if timestamp <= span_end:
+                timeline.append((timestamp, f"{name} {phrase}"))
+
+        if event["end_time"] and event["end_time"] <= span_end:
+            timeline.append((event["end_time"], f"{name} is no longer detected"))
+
+    return sorted(timeline, key=lambda m: m[0])
+
+
+def annotations_by_frame(
+    timeline: list[tuple[float, str]], frame_times: list[float]
+) -> dict[int, list[str]]:
+    """Bucket timeline moments onto the frame that follows each one.
+
+    A moment is attached to the first frame at or after it happened, so the
+    note always precedes the image in which the change becomes visible.
+    """
+    buckets: dict[int, list[str]] = {}
+
+    if not frame_times:
+        return buckets
+
+    for timestamp, phrase in timeline:
+        index = next(
+            (i for i, ft in enumerate(frame_times) if ft >= timestamp),
+            len(frame_times) - 1,
+        )
+        buckets.setdefault(index, []).append(phrase)
+
+    return buckets
+
+
+def get_tracked_events(detection_ids: list[str]) -> list[dict[str, Any]]:
+    """Load the tracked objects behind a review item's detections."""
+    if not detection_ids:
+        return []
+
+    rows = list(
+        Event.select(
+            Event.id,
+            Event.label,
+            Event.sub_label,
+            Event.start_time,
+            Event.end_time,
+            Event.data,
+        )
+        .where(Event.id << detection_ids)
+        .dicts()
+        .iterator()
+    )
+
+    return [
+        {
+            "id": row["id"],
+            "label": row["label"],
+            "sub_label": row["sub_label"],
+            "start_time": row["start_time"],
+            "end_time": row["end_time"],
+            "path_data": (row["data"] or {}).get("path_data") or [],
+        }
+        for row in rows
+        if row["start_time"] is not None
+    ]
+
+
+def get_state_changes(detection_ids: list[str]) -> list[dict[str, Any]]:
+    """Stationary/active changes the timeline recorded for these objects."""
+    if not detection_ids:
+        return []
+
+    return list(
+        Timeline.select(Timeline.timestamp, Timeline.source_id, Timeline.class_type)
+        .where(
+            (Timeline.source_id << detection_ids)
+            & (Timeline.class_type << list(STATE_CHANGE_PHRASES))
+        )
+        .dicts()
+        .iterator()
+    )
+
+
+def build_frame_captions(
+    detection_ids: list[str],
+    frame_times: list[float],
+) -> list[str]:
+    """A caption for each sampled frame, in frame order.
+
+    Every frame gets its index and elapsed time so the model can tell them
+    apart; frames where something changed also carry the tracker notes for
+    that moment. Returns an empty list when there is nothing to say, which
+    callers treat as a reason to fall back to sending plain frames.
+    """
+    if not frame_times:
+        return []
+
+    events = get_tracked_events(detection_ids)
+
+    if not events:
+        logger.debug("No tracked events found for review item, skipping annotations")
+        return []
+
+    timeline = build_timeline(events, frame_times[-1], get_state_changes(detection_ids))
+    buckets = annotations_by_frame(timeline, frame_times)
+
+    if not buckets:
+        return []
+
+    total = len(frame_times)
+    origin = frame_times[0]
+    captions: list[str] = []
+
+    for index, timestamp in enumerate(frame_times):
+        lines = [f"Frame {index + 1} of {total} (+{timestamp - origin:.1f}s):"]
+        lines.extend(f"[tracker] {note}" for note in buckets.get(index, []))
+        captions.append("\n".join(lines))
+
+    return captions

frigate/data_processing/post/review_descriptions.py

@@ -19,7 +19,11 @@ from frigate.comms.embeddings_updater import EmbeddingsRequestEnum
 from frigate.comms.inter_process import InterProcessRequestor
 from frigate.config import FrigateConfig
 from frigate.config.camera import CameraConfig
-from frigate.config.camera.review import GenAIReviewConfig, ImageSourceEnum
+from frigate.config.camera.review import (
+    GenAIReviewConfig,
+    ImageSourceEnum,
+    ReviewFrameModeEnum,
+)
 from frigate.const import (
     ATTRIBUTE_LABEL_DISPLAY_MAP,
     CACHE_DIR,
@@ -36,6 +40,7 @@ from frigate.util.image import get_image_from_recording
 
 from ..post.api import PostProcessorApi
 from ..types import DataProcessorMetrics
+from .review_annotations import build_frame_captions
 
 logger = logging.getLogger(__name__)
 
@@ -43,6 +48,7 @@ RECORDING_BUFFER_EXTENSION_PERCENT = 0.10
 MIN_RECORDING_DURATION = 10
 MAX_IMAGE_TOKENS = 24000
 MAX_FRAMES_PER_SECOND = 1
+MAX_ANNOTATED_FRAMES = 28
 
 
 class ReviewDescriptionProcessor(PostProcessorApi):
@@ -67,6 +73,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         duration: float,
         image_source: ImageSourceEnum = ImageSourceEnum.preview,
         height: int = 480,
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
     ) -> int:
         """Calculate optimal number of frames based on event duration, context size,
         image source, and resolution.
@@ -80,6 +87,8 @@ class ReviewDescriptionProcessor(PostProcessorApi):
           - MAX_FRAMES_PER_SECOND x duration, to avoid drowning short events in
             near-duplicate frames where the model latches onto the redundant middle
             and skips the start/end action
+          - MAX_ANNOTATED_FRAMES in annotated mode, where the tracking notes
+            already carry the sequence
         """
         client = self.genai_manager.description_client
 
@@ -125,6 +134,10 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         max_frames_by_tokens = int(image_token_budget / tokens_per_image)
         max_frames_by_duration = int(duration * MAX_FRAMES_PER_SECOND)
         max_frames = min(max_frames_by_tokens, max_frames_by_duration)
+
+        if frame_mode == ReviewFrameModeEnum.annotated_frames:
+            max_frames = min(max_frames, MAX_ANNOTATED_FRAMES)
+
         return max(max_frames, 3)
 
     def process_data(
@@ -166,6 +179,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 return
 
             image_source = camera_config.review.genai.image_source
+            frame_mode = camera_config.review.genai.frame_mode
 
             if image_source == ImageSourceEnum.recordings:
                 buffer_extension = get_recording_buffer_extension(
@@ -174,40 +188,43 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 final_data["start_time"] -= buffer_extension
                 final_data["end_time"] += buffer_extension
 
-                thumbs = self.get_recording_frames(
+                frames = self.get_recording_frames(
                     camera,
                     final_data["start_time"],
                     final_data["end_time"],
                     height=480,  # Use 480p for good balance between quality and token usage
+                    frame_mode=frame_mode,
                 )
 
-                if not thumbs:
+                if not frames:
                     # Fallback to preview frames if no recordings available
                     logger.warning(
                         f"No recording frames found for {camera}, falling back to preview frames"
                     )
-                    thumbs = self.get_preview_frames_as_bytes(
+                    frames = self.get_preview_frames_as_bytes(
                         camera,
                         final_data["start_time"],
                         final_data["end_time"],
                         final_data["thumb_path"],
                         id,
                         camera_config.review.genai.debug_save_thumbnails,
+                        frame_mode,
                     )
                 elif camera_config.review.genai.debug_save_thumbnails:
-                    self.save_debug_recording_frames(id, thumbs)
+                    self.save_debug_recording_frames(id, frames)
             else:
                 # Use preview frames
-                thumbs = self.get_preview_frames_as_bytes(
+                frames = self.get_preview_frames_as_bytes(
                     camera,
                     final_data["start_time"],
                     final_data["end_time"],
                     final_data["thumb_path"],
                     id,
                     camera_config.review.genai.debug_save_thumbnails,
+                    frame_mode,
                 )
 
-            self.start_analysis(camera_config, final_data, thumbs)
+            self.start_analysis(camera_config, final_data, frames)
 
     def handle_request(self, topic: str, request_data: dict[str, Any]) -> str | None:
         if topic == EmbeddingsRequestEnum.regenerate_review_description.value:
@@ -386,31 +403,50 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         buffer_extension = get_recording_buffer_extension(
             final_data["end_time"] - final_data["start_time"]
         )
-        thumbs = self.get_recording_frames(
+        frames = self.get_recording_frames(
             str(review.camera),
             final_data["start_time"] - buffer_extension,
             final_data["end_time"] + buffer_extension,
             height=480,
+            frame_mode=camera_config.review.genai.frame_mode,
         )
 
-        if not thumbs:
+        if not frames:
             logger.error(
                 "No recording frames are available for review item %s", review_id
             )
             return
 
         if camera_config.review.genai.debug_save_thumbnails:
-            self.save_debug_recording_frames(review_id, thumbs)
+            self.save_debug_recording_frames(review_id, frames)
 
-        self.start_analysis(camera_config, final_data, thumbs)
+        self.start_analysis(camera_config, final_data, frames)
 
     def start_analysis(
         self,
         camera_config: CameraConfig,
         final_data: dict[str, Any],
-        thumbs: list[bytes],
+        frames: list[tuple[bytes, float]],
     ) -> None:
         """Kick off description generation for a review item in the background."""
+        thumbs = [frame for frame, _ in frames]
+        captions: list[str] = []
+
+        if (
+            camera_config.review.genai.frame_mode
+            == ReviewFrameModeEnum.annotated_frames
+        ):
+            captions = build_frame_captions(
+                final_data["data"].get("detections") or [],
+                [timestamp for _, timestamp in frames],
+            )
+
+            if not captions:
+                logger.debug(
+                    "No tracking annotations for review item %s, sending plain frames",
+                    final_data["id"],
+                )
+
         self.review_desc_dps.update()
         threading.Thread(
             target=run_analysis,
@@ -421,19 +457,22 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 camera_config,
                 final_data,
                 thumbs,
+                captions,
                 camera_config.review.genai,
                 sorted(self.config.all_labels),
                 self.config.all_attributes,
             ),
         ).start()
 
-    def save_debug_recording_frames(self, review_id: str, thumbs: list[bytes]) -> None:
+    def save_debug_recording_frames(
+        self, review_id: str, frames: list[tuple[bytes, float]]
+    ) -> None:
         """Write the recording frames sent to the provider out for debugging."""
         Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir(
             parents=True, exist_ok=True
         )
 
-        for idx, frame_bytes in enumerate(thumbs):
+        for idx, (frame_bytes, _) in enumerate(frames):
             with open(
                 os.path.join(CLIPS_DIR, f"genai-requests/{review_id}/{idx}.jpg"),
                 "wb",
@@ -445,7 +484,9 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         camera: str,
         start_time: float,
         end_time: float,
-    ) -> list[str]:
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
+    ) -> list[tuple[str, float]]:
+        """Preview frame paths paired with the time each one was captured."""
         preview_dir = os.path.join(CACHE_DIR, "preview_frames")
         file_start = f"preview_{camera}-"
         start_file = f"{file_start}{start_time}.webp"
@@ -477,18 +518,29 @@ class ReviewDescriptionProcessor(PostProcessorApi):
 
         frame_count = len(all_frames)
         desired_frame_count = self.calculate_frame_count(
-            camera, duration=end_time - start_time
+            camera,
+            duration=end_time - start_time,
+            frame_mode=frame_mode,
         )
 
+        def with_timestamp(path: str) -> tuple[str, float]:
+            # Preview frames are named preview_<camera>-<timestamp>.webp
+            stem = os.path.basename(path).removesuffix(".webp")
+
+            try:
+                return (path, float(stem.removeprefix(file_start)))
+            except ValueError:
+                return (path, start_time)
+
         if frame_count <= desired_frame_count:
-            return all_frames
+            return [with_timestamp(f) for f in all_frames]
 
         selected_frames = []
         step_size = (frame_count - 1) / (desired_frame_count - 1)
 
         for i in range(desired_frame_count):
             index = round(i * step_size)
-            selected_frames.append(all_frames[index])
+            selected_frames.append(with_timestamp(all_frames[index]))
 
         return selected_frames
 
@@ -498,11 +550,12 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         start_time: float,
         end_time: float,
         height: int = 480,
-    ) -> list[bytes]:
-        """Get frames from recordings at specified timestamps."""
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
+    ) -> list[tuple[bytes, float]]:
+        """Get frames from recordings paired with the time each was captured."""
         duration = end_time - start_time
         desired_frame_count = self.calculate_frame_count(
-            camera, duration, ImageSourceEnum.recordings, height
+            camera, duration, ImageSourceEnum.recordings, height, frame_mode
         )
 
         # Calculate evenly spaced timestamps throughout the duration
@@ -540,7 +593,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
             except DoesNotExist:
                 return None
 
-        frames = []
+        frames: list[tuple[bytes, float]] = []
 
         for timestamp in timestamps:
             try:
@@ -553,7 +606,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                     image_data = extract_frame_from_recording(rounded_timestamp)
 
                 if image_data:
-                    frames.append(image_data)
+                    frames.append((image_data, timestamp))
                 else:
                     logger.warning(
                         f"No recording found for {camera} at timestamp {timestamp}"
@@ -574,7 +627,8 @@ class ReviewDescriptionProcessor(PostProcessorApi):
         thumb_path_fallback: str,
         review_id: str,
         save_debug: bool,
-    ) -> list[bytes]:
+        frame_mode: ReviewFrameModeEnum = ReviewFrameModeEnum.frames,
+    ) -> list[tuple[bytes, float]]:
         """Get preview frames and convert them to JPEG bytes.
 
         Args:
@@ -586,14 +640,14 @@ class ReviewDescriptionProcessor(PostProcessorApi):
             save_debug: Whether to save debug thumbnails
 
         Returns:
-            List of JPEG image bytes
+            List of (JPEG image bytes, capture timestamp) pairs
         """
-        frame_paths = self.get_cache_frames(camera, start_time, end_time)
+        frame_paths = self.get_cache_frames(camera, start_time, end_time, frame_mode)
         if not frame_paths:
-            frame_paths = [thumb_path_fallback]
+            frame_paths = [(thumb_path_fallback, start_time)]
 
-        thumbs = []
-        for idx, thumb_path in enumerate(frame_paths):
+        thumbs: list[tuple[bytes, float]] = []
+        for idx, (thumb_path, timestamp) in enumerate(frame_paths):
             thumb_data = cv2.imread(thumb_path)
 
             if thumb_data is None:
@@ -606,7 +660,7 @@ class ReviewDescriptionProcessor(PostProcessorApi):
                 ".jpg", thumb_data, [int(cv2.IMWRITE_JPEG_QUALITY), 100]
             )
             if ret:
-                thumbs.append(jpg.tobytes())
+                thumbs.append((jpg.tobytes(), timestamp))
 
             if save_debug:
                 Path(os.path.join(CLIPS_DIR, "genai-requests", review_id)).mkdir(
@@ -641,6 +695,7 @@ def run_analysis(
     camera_config: CameraConfig,
     final_data: dict[str, Any],
     thumbs: list[bytes],
+    frame_captions: list[str],
     genai_config: GenAIReviewConfig,
     labelmap_objects: list[str],
     attribute_labels: list[str],
@@ -697,6 +752,7 @@ def run_analysis(
         genai_config.debug_save_thumbnails,
         genai_config.activity_context_prompt,
         genai_config.response_style,
+        frame_captions,
     )
     review_inference_speed.update(datetime.datetime.now().timestamp() - start)
 

frigate/embeddings/maintainer.py

@@ -68,7 +68,7 @@ from frigate.events.types import (
     RegenerateDescriptionEnum,
 )
 from frigate.genai import GenAIClientManager
-from frigate.models import Event, Recordings, ReviewSegment, Trigger
+from frigate.models import Event, Recordings, ReviewSegment, Timeline, Trigger
 from frigate.types import TrackedObjectUpdateTypesEnum
 from frigate.util.builtin import serialize
 from frigate.util.file import get_event_thumbnail_bytes
@@ -139,7 +139,7 @@ class EmbeddingMaintainer(threading.Thread):
             ),
             load_vec_extension=True,
         )
-        models = [Event, Recordings, ReviewSegment, Trigger]
+        models = [Event, Recordings, ReviewSegment, Timeline, Trigger]
         db.bind(models)
 
         self.genai_manager = GenAIClientManager(config)

frigate/genai/__init__.py

@@ -105,8 +105,21 @@ class GenAIClient:
         debug_save: bool,
         activity_context_prompt: str,
         response_style: str = "default",
+        frame_captions: list[str] | None = None,
     ) -> ReviewMetadata | None:
-        """Generate a description for the review item activity."""
+        """Generate a description for the review item activity.
+
+        `frame_captions` holds one caption per thumbnail for the annotated
+        frame mode; each is sent directly before its frame.
+        """
+        if frame_captions and len(frame_captions) != len(thumbnails):
+            logger.warning(
+                "Got %d frame captions for %d thumbnails, sending plain frames",
+                len(frame_captions),
+                len(thumbnails),
+            )
+            frame_captions = None
+
         context_prompt = build_review_description_prompt(
             review_data,
             thumbnails,
@@ -114,6 +127,7 @@ class GenAIClient:
             preferred_language,
             activity_context_prompt,
             response_style,
+            frame_captions,
         )
 
         logger.debug(
@@ -129,9 +143,30 @@ class GenAIClient:
             ) as f:
                 f.write(context_prompt)
 
+            if frame_captions:
+                # One file per frame, numbered to match the image it precedes
+                # (0.txt goes with 0.jpg), so the debug folder replays without
+                # having to re-derive the mapping.
+                for index, caption in enumerate(frame_captions):
+                    with open(
+                        os.path.join(
+                            CLIPS_DIR,
+                            "genai-requests",
+                            review_data["id"],
+                            f"{index}.txt",
+                        ),
+                        "w",
+                    ) as f:
+                        f.write(caption)
+
         response_format = build_review_description_response_format(concerns)
 
-        response = self._send(context_prompt, thumbnails, response_format)
+        response = self._send(
+            context_prompt,
+            thumbnails,
+            response_format,
+            image_captions=frame_captions,
+        )
 
         if debug_save and response:
             with open(
@@ -269,6 +304,7 @@ class GenAIClient:
         images: list[bytes],
         response_format: dict | None = None,
         enable_thinking: bool = False,
+        image_captions: list[str] | None = None,
     ) -> str | None:
         """Submit a request to the provider.
 
@@ -276,6 +312,10 @@ class GenAIClient:
         ``supports_toggleable_thinking``. Description-style callers leave it
         at the default (off) since synthesis tasks don't benefit from
         reasoning traces.
+
+        ``image_captions`` carries one caption per image, to be placed
+        immediately before its image so the model can tell the frames apart.
+        Providers build their request order with ``interleave_images``.
         """
         return None
 

frigate/genai/plugins/gemini.py

@@ -13,6 +13,7 @@ from google.genai.types import FunctionCallingConfigMode
 
 from frigate.config import GenAIProviderEnum
 from frigate.genai import GenAIClient, register_genai_provider
+from frigate.genai.utils import interleave_images
 
 logger = logging.getLogger(__name__)
 
@@ -118,11 +119,16 @@ class GeminiClient(GenAIClient):
         images: list[bytes],
         response_format: dict | None = None,
         enable_thinking: bool = False,
+        image_captions: list[str] | None = None,
     ) -> str | None:
         """Submit a request to Gemini."""
-        contents = [prompt] + [
-            types.Part.from_bytes(data=img, mime_type="image/jpeg") for img in images
+        contents: list[Any] = [
+            part
+            if isinstance(part, str)
+            else types.Part.from_bytes(data=part, mime_type="image/jpeg")
+            for part in interleave_images(prompt, images, image_captions)
         ]
+
         try:
             # Merge runtime_options into generation_config if provided
             generation_config_dict: dict[str, Any] = {"candidate_count": 1}
@@ -136,7 +142,7 @@ class GeminiClient(GenAIClient):
 
             response = self.provider.models.generate_content(
                 model=self.genai_config.model,
-                contents=contents,  # type: ignore[arg-type]
+                contents=contents,
                 config=types.GenerateContentConfig(
                     **generation_config_dict,
                 ),

frigate/genai/plugins/llama_cpp.py

@@ -14,7 +14,7 @@ from PIL import Image
 
 from frigate.config import GenAIProviderEnum
 from frigate.genai import GenAIClient, register_genai_provider
-from frigate.genai.utils import parse_tool_calls_from_message
+from frigate.genai.utils import interleave_images, parse_tool_calls_from_message
 
 logger = logging.getLogger(__name__)
 
@@ -333,6 +333,7 @@ class LlamaCppClient(GenAIClient):
         images: list[bytes],
         response_format: dict | None = None,
         enable_thinking: bool = False,
+        image_captions: list[str] | None = None,
     ) -> str | None:
         """Submit a request to llama.cpp server."""
         if self.provider is None:
@@ -342,18 +343,17 @@ class LlamaCppClient(GenAIClient):
             return None
 
         try:
-            content = [
-                {
-                    "type": "text",
-                    "text": prompt,
-                }
-            ]
-            for image in images:
-                encoded_image = base64.b64encode(image).decode("utf-8")
+            content: list[dict[str, Any]] = []
+            for part in interleave_images(prompt, images, image_captions):
+                if isinstance(part, str):
+                    content.append({"type": "text", "text": part})
+                    continue
+
+                encoded_image = base64.b64encode(part).decode("utf-8")
                 content.append(
                     {
                         "type": "image_url",
-                        "image_url": {  # type: ignore[dict-item]
+                        "image_url": {
                             "url": f"data:image/jpeg;base64,{encoded_image}",
                         },
                     }

frigate/genai/plugins/ollama.py

@@ -14,7 +14,7 @@ from ollama import ResponseError
 
 from frigate.config import GenAIProviderEnum
 from frigate.genai import GenAIClient, register_genai_provider
-from frigate.genai.utils import parse_tool_calls_from_message
+from frigate.genai.utils import interleave_images, parse_tool_calls_from_message
 
 logger = logging.getLogger(__name__)
 
@@ -50,6 +50,28 @@ def _extract_ollama_stats(response: Any) -> dict[str, Any] | None:
     return stats or None
 
 
+# Ollama replaces each occurrence of this marker in a message, in order, with
+# the next image from the message's images list. Without markers it puts every
+# image before the text.
+IMAGE_PLACEHOLDER = "[img]"
+
+
+def _flatten_parts(parts: list[str | bytes]) -> tuple[str, list[bytes] | None]:
+    """Collapse ordered text and image parts into Ollama's (content, images)
+    shape, marking where each image goes so the order survives."""
+    text: list[str] = []
+    images: list[bytes] = []
+
+    for part in parts:
+        if isinstance(part, bytes):
+            text.append(IMAGE_PLACEHOLDER)
+            images.append(part)
+        elif part:
+            text.append(part)
+
+    return "\n".join(text), (images or None)
+
+
 def _normalize_multimodal_content(
     content: Any,
 ) -> tuple[str | None, list[bytes] | None]:
@@ -58,13 +80,13 @@ def _normalize_multimodal_content(
     The chat API constructs user messages with content as a list of
     ``{"type": "text"}`` and ``{"type": "image_url"}`` parts when a tool
     returns a live frame. Ollama's SDK requires content to be a string and
-    images to be passed in a separate field, so we extract each.
+    images to be passed in a separate field, so images are pulled out and
+    their positions marked with placeholders.
     """
     if not isinstance(content, list):
         return content, None
 
-    text_parts: list[str] = []
-    images: list[bytes] = []
+    parts: list[str | bytes] = []
     for part in content:
         if not isinstance(part, dict):
             continue
@@ -72,17 +94,20 @@ def _normalize_multimodal_content(
         if part_type == "text":
             text = part.get("text")
             if text:
-                text_parts.append(str(text))
+                parts.append(str(text))
         elif part_type == "image_url":
             url = (part.get("image_url") or {}).get("url", "")
             if isinstance(url, str) and url.startswith("data:"):
                 try:
                     encoded = url.split(",", 1)[1]
-                    images.append(base64.b64decode(encoded, validate=True))
+                    parts.append(base64.b64decode(encoded, validate=True))
                 except (ValueError, IndexError, binascii.Error) as e:
                     logger.debug("Failed to decode multimodal image url: %s", e)
 
-    return ("\n".join(text_parts) if text_parts else None), (images or None)
+    if not parts:
+        return None, None
+
+    return _flatten_parts(parts)
 
 
 @register_genai_provider(GenAIProviderEnum.ollama)
@@ -196,58 +221,46 @@ class OllamaClient(GenAIClient):
         images: list[bytes],
         response_format: dict | None = None,
         enable_thinking: bool = False,
+        image_captions: list[str] | None = None,
     ) -> str | None:
-        """Submit a request to Ollama"""
+        """Submit a request to Ollama through the chat API, the same path the
+        tool-calling chat uses, with image placeholders keeping any captions
+        next to their frames."""
         if self.provider is None:
             logger.warning(
                 "Ollama provider has not been initialized, a description will not be generated. Check your Ollama configuration."
             )
             return None
+
+        content, message_images = _flatten_parts(
+            interleave_images(prompt, images, image_captions)
+        )
+        message: dict[str, Any] = {"role": "user", "content": content}
+
+        if message_images:
+            message["images"] = message_images
+
+        request_params = self._build_request_params(
+            [message], None, None, enable_thinking=enable_thinking
+        )
+
+        if response_format and response_format.get("type") == "json_schema":
+            schema = response_format.get("json_schema", {}).get("schema")
+            if schema:
+                request_params["format"] = self._clean_schema_for_ollama(schema)
+
+        logger.debug(
+            "Ollama chat request: model=%s, prompt_len=%s, image_count=%s, "
+            "has_format=%s, think=%s",
+            self.genai_config.model,
+            len(prompt),
+            len(images),
+            "format" in request_params,
+            request_params.get("think"),
+        )
+
         try:
-            ollama_options = {
-                **self.provider_options,
-                **self.genai_config.runtime_options,
-            }
-            if response_format and response_format.get("type") == "json_schema":
-                schema = response_format.get("json_schema", {}).get("schema")
-                if schema:
-                    ollama_options["format"] = self._clean_schema_for_ollama(schema)
-            if self.supports_toggleable_thinking:
-                ollama_options["think"] = enable_thinking
-            logger.debug(
-                "Ollama generate request: model=%s, prompt_len=%s, image_count=%s, "
-                "has_format=%s, options=%s",
-                self.genai_config.model,
-                len(prompt),
-                len(images) if images else 0,
-                "format" in ollama_options,
-                {k: v for k, v in ollama_options.items() if k != "format"},
-            )
-            result = self.provider.generate(
-                self.genai_config.model,
-                prompt,
-                images=images if images else None,
-                **ollama_options,
-            )
-            logger.debug(
-                "Ollama generate response: done=%s, done_reason=%s, eval_count=%s, "
-                "prompt_eval_count=%s, response_len=%s",
-                result.get("done"),
-                result.get("done_reason"),
-                result.get("eval_count"),
-                result.get("prompt_eval_count"),
-                len(result.get("response", "") or ""),
-            )
-            response_text = str(result["response"]).strip()
-            if not response_text:
-                logger.warning(
-                    "Ollama returned a blank response for model %s (done_reason=%s, "
-                    "eval_count=%s). Check model output, ensure thinking is disabled.",
-                    self.genai_config.model,
-                    result.get("done_reason"),
-                    result.get("eval_count"),
-                )
-            return response_text
+            response = self.provider.chat(**request_params)
         except (
             TimeoutException,
             ResponseError,
@@ -257,6 +270,27 @@ class OllamaClient(GenAIClient):
             logger.warning("Ollama returned an error: %s", str(e))
             return None
 
+        logger.debug(
+            "Ollama chat response: done=%s, done_reason=%s, eval_count=%s, "
+            "prompt_eval_count=%s",
+            response.get("done"),
+            response.get("done_reason"),
+            response.get("eval_count"),
+            response.get("prompt_eval_count"),
+        )
+        response_text = self._message_from_response(response)["content"] or ""
+
+        if not response_text:
+            logger.warning(
+                "Ollama returned a blank response for model %s (done_reason=%s, "
+                "eval_count=%s). Check model output, ensure thinking is disabled.",
+                self.genai_config.model,
+                response.get("done_reason"),
+                response.get("eval_count"),
+            )
+
+        return response_text
+
     def list_models(self) -> list[str]:
         """Return available model names from the Ollama server."""
         client = self.provider
@@ -306,6 +340,8 @@ class OllamaClient(GenAIClient):
             }
             if images:
                 msg_dict["images"] = images
+            elif msg.get("images"):
+                msg_dict["images"] = msg["images"]
             if msg.get("tool_call_id"):
                 msg_dict["tool_call_id"] = msg["tool_call_id"]
             if msg.get("name"):

frigate/genai/plugins/openai.py

@@ -11,6 +11,7 @@ from openai import OpenAI
 
 from frigate.config import GenAIProviderEnum
 from frigate.genai import GenAIClient, register_genai_provider
+from frigate.genai.utils import interleave_images
 
 logger = logging.getLogger(__name__)
 
@@ -63,21 +64,21 @@ class OpenAIClient(GenAIClient):
         images: list[bytes],
         response_format: dict | None = None,
         enable_thinking: bool = False,
+        image_captions: list[str] | None = None,
     ) -> str | None:
         """Submit a request to OpenAI."""
-        encoded_images = [base64.b64encode(image).decode("utf-8") for image in images]
-        messages_content: list[dict] = [
-            {
-                "type": "text",
-                "text": prompt,
-            }
-        ]
-        for image in encoded_images:
+        messages_content: list[dict] = []
+        for part in interleave_images(prompt, images, image_captions):
+            if isinstance(part, str):
+                messages_content.append({"type": "text", "text": part})
+                continue
+
+            encoded = base64.b64encode(part).decode("utf-8")
             messages_content.append(
                 {
                     "type": "image_url",
                     "image_url": {
-                        "url": f"data:image/jpeg;base64,{image}",
+                        "url": f"data:image/jpeg;base64,{encoded}",
                         "detail": "low",
                     },
                 }

frigate/genai/prompts.py

@@ -59,6 +59,13 @@ def get_review_field_guidelines(response_style: str = "default") -> dict[str, st
     }
 
 
+# Explains the per-frame labels and tracker notes used by the annotated frame
+# mode. Neither the notes nor this guidance say whether repeated detections are
+# the same subject, since the tracking data cannot tell.
+FRAME_ANNOTATION_GUIDANCE = """- Each image below is immediately preceded by a text label giving its frame number and how many seconds into the sequence it was captured. Use these labels to track the order of events and the time between them.
+- Some images below are preceded by notes from the camera's object tracker recording what changed at that point: an object being first detected, starting to move, reversing direction, stopping, or no longer being detected. These notes come from tracking data rather than from the images, and they are reliable. Use them to establish how many distinct activities occur and in what order, and describe every one of them."""
+
+
 def build_review_description_prompt(
     review_data: dict[str, Any],
     thumbnails: list[bytes],
@@ -66,8 +73,13 @@ def build_review_description_prompt(
     preferred_language: str | None,
     activity_context_prompt: str,
     response_style: str = "default",
+    frame_captions: list[str] | None = None,
 ) -> str:
-    """Build the prompt for review activity description generation."""
+    """Build the prompt for review activity description generation.
+
+    When `frame_captions` is set, each caption is sent directly before its
+    image, so the prompt explains that layout.
+    """
 
     def get_concern_prompt() -> str:
         if concerns:
@@ -93,6 +105,7 @@ def build_review_description_prompt(
             return "\n- (No objects detected)"
 
     fields = get_review_field_guidelines(response_style)
+    frame_guidance = f"\n{FRAME_ANNOTATION_GUIDANCE}" if frame_captions else ""
 
     return f"""
 Your task is to analyze a sequence of images taken in chronological order from a security camera.
@@ -130,7 +143,7 @@ Respond with a JSON object matching the provided schema. Field-specific guidance
 ## Sequence Details
 
 - Camera: {review_data["camera"]}
-- Total frames: {len(thumbnails)} (Frame 1 = earliest, Frame {len(thumbnails)} = latest)
+- Total frames: {len(thumbnails)} (Frame 1 = earliest, Frame {len(thumbnails)} = latest){frame_guidance}
 - Activity started at {review_data["start"]} and lasted {review_data["duration"]} seconds
 - Zones involved: {", ".join(review_data["zones"]) if review_data["zones"] else "None"}
 

frigate/genai/utils.py

@@ -7,6 +7,25 @@ from typing import Any
 logger = logging.getLogger(__name__)
 
 
+def interleave_images(
+    prompt: str, images: list[bytes], captions: list[str] | None = None
+) -> list[str | bytes]:
+    """The prompt, then each image preceded by its caption when one is given.
+
+    Providers map the text and image parts onto their own request format, so
+    every provider sends the same order.
+    """
+    parts: list[str | bytes] = [prompt]
+
+    for index, image in enumerate(images):
+        if captions and index < len(captions):
+            parts.append(captions[index])
+
+        parts.append(image)
+
+    return parts
+
+
 def parse_tool_calls_from_message(
     message: dict[str, Any],
 ) -> list[dict[str, Any]] | None:

frigate/test/test_genai_providers.py

@@ -360,9 +360,70 @@ class TestOllamaProvider(unittest.TestCase):
         from frigate.genai.plugins.ollama import _normalize_multimodal_content
 
         text, images = _normalize_multimodal_content(MULTIMODAL_MESSAGES[-1]["content"])
-        self.assertIn("live image", text)
-        self.assertEqual(len(images), 1)
-        self.assertEqual(images[0], b"\xff\xd8\xff\xd9")
+        self.assertEqual(
+            text, "Here is the current live image from camera 'front'.\n[img]"
+        )
+        self.assertEqual(images, [b"\xff\xd8\xff\xd9"])
+
+    def test_normalize_keeps_text_and_image_order(self):
+        from frigate.genai.plugins.ollama import _normalize_multimodal_content
+
+        text, images = _normalize_multimodal_content(
+            [
+                {"type": "text", "text": "intro"},
+                {"type": "text", "text": "Frame 1"},
+                {"type": "image_url", "image_url": {"url": _IMAGE_DATA_URI}},
+                {"type": "text", "text": "Frame 2"},
+                {"type": "image_url", "image_url": {"url": _IMAGE_DATA_URI}},
+            ]
+        )
+        self.assertEqual(text, "intro\nFrame 1\n[img]\nFrame 2\n[img]")
+        self.assertEqual(len(images), 2)
+
+    def test_send_uses_chat_with_captions_before_each_image(self):
+        client = self._client()
+        client.provider = MagicMock()
+        client.provider.chat.return_value = {
+            "message": {"content": '{"ok": true}'},
+            "done": True,
+            "done_reason": "stop",
+        }
+        client._supports_thinking_cache = False
+
+        result = client._send(
+            "prompt",
+            [b"a", b"b"],
+            {"type": "json_schema", "json_schema": {"schema": {"type": "object"}}},
+            image_captions=["Frame 1 of 2", "Frame 2 of 2"],
+        )
+
+        self.assertEqual(result, '{"ok": true}')
+        client.provider.generate.assert_not_called()
+        params = client.provider.chat.call_args.kwargs
+        self.assertEqual(
+            params["messages"],
+            [
+                {
+                    "role": "user",
+                    "content": "prompt\nFrame 1 of 2\n[img]\nFrame 2 of 2\n[img]",
+                    "images": [b"a", b"b"],
+                }
+            ],
+        )
+        self.assertEqual(params["format"], {"type": "object"})
+        self.assertNotIn("think", params)
+
+    def test_send_without_captions_puts_images_after_prompt(self):
+        client = self._client()
+        client.provider = MagicMock()
+        client.provider.chat.return_value = {"message": {"content": "ok"}, "done": True}
+        client._supports_thinking_cache = False
+
+        client._send("prompt", [b"a"])
+
+        message = client.provider.chat.call_args.kwargs["messages"][0]
+        self.assertEqual(message["content"], "prompt\n[img]")
+        self.assertEqual(message["images"], [b"a"])
 
 
 # ---------------------------------------------------------------------------

frigate/test/test_review_annotations.py

@@ -0,0 +1,316 @@
+"""Tests for tracker-derived review frame annotations."""
+
+import unittest
+
+from frigate.data_processing.post.review_annotations import (
+    annotations_by_frame,
+    build_timeline,
+    describe_heading,
+    describe_position,
+    event_name,
+    path_legs,
+    path_moments,
+)
+
+
+def straight_path(
+    start: tuple[float, float],
+    end: tuple[float, float],
+    steps: int,
+    t0: float,
+) -> list:
+    """A path_data-shaped trajectory travelling in a straight line."""
+    return [
+        [
+            [
+                start[0] + (end[0] - start[0]) * i / (steps - 1),
+                start[1] + (end[1] - start[1]) * i / (steps - 1),
+            ],
+            t0 + i,
+        ]
+        for i in range(steps)
+    ]
+
+
+class TestDescribers(unittest.TestCase):
+    def test_describes_frame_corners(self):
+        self.assertEqual(describe_position(0.9, 0.9), "the bottom right of the frame")
+        self.assertEqual(describe_position(0.1, 0.1), "the top left of the frame")
+        self.assertEqual(describe_position(0.5, 0.5), "the middle of the frame")
+
+    def test_heading_treats_falling_y_as_up(self):
+        self.assertEqual(describe_heading(0.0, -0.5), "up")
+        self.assertEqual(describe_heading(0.0, 0.5), "down")
+        self.assertEqual(describe_heading(-0.5, 0.0), "left")
+
+    def test_heading_combines_axes(self):
+        self.assertEqual(describe_heading(-0.5, -0.5), "up and left")
+
+
+class TestPathLegs(unittest.TestCase):
+    def test_straight_travel_is_one_leg(self):
+        points = [(0.9 - 0.05 * i, 0.6, float(i)) for i in range(10)]
+        self.assertEqual(len(path_legs(points)), 1)
+
+    def test_out_and_back_is_two_legs(self):
+        out = [(0.9 - 0.05 * i, 0.6, float(i)) for i in range(8)]
+        back = [(0.55 + 0.05 * i, 0.6, 8.0 + i) for i in range(8)]
+        self.assertEqual(len(path_legs(out + back)), 2)
+
+    def test_jitter_in_place_produces_no_legs(self):
+        # A subject standing still wobbles by a couple of percent; without the
+        # minimum leg distance this became a burst of contradictory turns.
+        points = [(0.5 + 0.01 * (i % 2), 0.5, float(i)) for i in range(20)]
+        self.assertEqual(path_legs(points), [])
+
+    def test_single_point_path_has_no_moments(self):
+        self.assertEqual(path_moments([[[0.5, 0.5], 1.0]]), [])
+
+    def test_malformed_path_is_ignored(self):
+        self.assertEqual(path_moments([[0.5, 1.0], [0.6, 2.0]]), [])
+
+
+class TestEventNames(unittest.TestCase):
+    def test_unnamed_objects_use_an_indefinite_article(self):
+        self.assertEqual(event_name({"label": "person"}), "a person")
+        self.assertEqual(event_name({"label": "animal"}), "an animal")
+
+    def test_sub_labeled_objects_use_their_name(self):
+        event = {"label": "waste_bin", "sub_label": "Compost"}
+        self.assertEqual(event_name(event), 'waste bin "Compost"')
+
+    def test_repeated_objects_are_never_numbered(self):
+        # Whether these are the same subject is unknown, so the notes must not
+        # imply either answer.
+        events = [
+            {
+                "id": f"1789481994.68448{i}-abcdef",
+                "label": "person",
+                "sub_label": None,
+                "start_time": float(i * 10),
+                "end_time": float(i * 10 + 50),
+                "zones": [],
+                "path_data": straight_path((0.9, 0.6), (0.4, 0.3), 10, i * 10.0),
+            }
+            for i in range(3)
+        ]
+        phrases = [p for _, p in build_timeline(events, span_end=100.0)]
+        self.assertTrue(all(p.startswith("a person ") for p in phrases))
+        self.assertFalse(any("#" in p for p in phrases))
+
+
+class TestTimeline(unittest.TestCase):
+    def setUp(self):
+        self.event = {
+            "id": "1789481994.684479-lpyc2z",
+            "label": "person",
+            "sub_label": None,
+            "start_time": 0.0,
+            "end_time": 20.0,
+            "zones": ["front_yard"],
+            "path_data": straight_path((0.9, 0.6), (0.4, 0.3), 10, 1.0),
+        }
+
+    def test_timeline_is_ordered_and_bounded(self):
+        timeline = build_timeline([self.event], span_end=30.0)
+        times = [t for t, _ in timeline]
+        self.assertEqual(times, sorted(times))
+        self.assertTrue(all(t <= 30.0 for t in times))
+
+    def test_track_identifiers_never_appear(self):
+        timeline = build_timeline([self.event], span_end=30.0)
+        joined = " ".join(phrase for _, phrase in timeline)
+        self.assertNotIn("track", joined.lower())
+        self.assertNotIn(self.event["id"], joined)
+
+    def test_object_still_tracked_at_end_gets_no_closing_note(self):
+        # Its state at the end is visible in the last frame, so nothing is said.
+        timeline = build_timeline([self.event], span_end=15.0)
+        self.assertEqual(
+            [p for _, p in timeline],
+            ["a person first detected at the right of the frame, moving up and left"],
+        )
+
+    def test_object_ending_inside_the_clip_is_no_longer_detected(self):
+        timeline = build_timeline([self.event], span_end=40.0)
+        phrases = [p for _, p in timeline]
+        self.assertTrue(any("no longer detected" in p for p in phrases))
+
+    def test_moments_past_the_last_frame_are_dropped(self):
+        # The subject keeps moving after the final sampled frame; those notes
+        # describe nothing the model can see.
+        late = dict(self.event)
+        out = straight_path((0.9, 0.6), (0.4, 0.6), 8, 100.0)
+        back = straight_path((0.4, 0.6), (0.9, 0.6), 8, 108.0)
+        late["path_data"] = out + back
+        late["start_time"] = 100.0
+        timeline = build_timeline([late], span_end=105.0)
+        self.assertTrue(any("moving left" in p for _, p in timeline))
+        self.assertFalse(any("turns around" in p for _, p in timeline))
+
+
+def track(event_id, label, start, path, sub_label=None, end=None):
+    return {
+        "id": event_id,
+        "label": label,
+        "sub_label": sub_label,
+        "start_time": start,
+        "end_time": end if end is not None else start + 500.0,
+        "zones": [],
+        "path_data": path,
+    }
+
+
+class TestArrival(unittest.TestCase):
+    def test_objects_arriving_together_keep_separate_notes(self):
+        person = track(
+            "1789481994.684479-lpyc2z",
+            "person",
+            0.0,
+            straight_path((0.9, 0.63), (0.58, 0.34), 10, 0.1),
+        )
+        bin_ = track(
+            "1789481995.063395-vlzd7q",
+            "waste_bin",
+            0.3,
+            straight_path((0.91, 0.64), (0.6, 0.33), 10, 0.4),
+            sub_label="Compost",
+        )
+        phrases = [p for _, p in build_timeline([person, bin_], 100.0)]
+        self.assertEqual(
+            phrases,
+            [
+                "a person first detected at the right of the frame, moving up and left",
+                'waste bin "Compost" first detected at the right of the frame, '
+                "moving up and left",
+            ],
+        )
+
+    def test_object_detected_well_before_it_moves_gets_separate_notes(self):
+        # path_data always keeps the first two samples, so a bin sitting in
+        # the yard opens its leg long before it is picked up.
+        path = [[[0.91, 0.64], 0.4]] + straight_path(
+            (0.91, 0.64), (0.6, 0.33), 10, 20.4
+        )
+        bin_ = track(
+            "1789481995.063395-vlzd7q", "waste_bin", 0.3, path, sub_label="Compost"
+        )
+        timeline = build_timeline([bin_], 100.0)
+        self.assertEqual(
+            timeline[0],
+            (0.3, 'waste bin "Compost" first detected at the right of the frame'),
+        )
+        self.assertAlmostEqual(timeline[1][0], 21.4)
+        self.assertEqual(
+            timeline[1][1],
+            'waste bin "Compost" starts moving up and left from the right of the frame',
+        )
+
+
+class TestStateChanges(unittest.TestCase):
+    def test_stationary_and_active_rows_become_notes(self):
+        bin_ = track(
+            "1789481995.063395-vlzd7q",
+            "waste_bin",
+            0.3,
+            straight_path((0.91, 0.64), (0.6, 0.33), 10, 0.4),
+            sub_label="Compost",
+        )
+        changes = [
+            {"timestamp": 20.0, "source_id": bin_["id"], "class_type": "stationary"},
+            {"timestamp": 30.0, "source_id": bin_["id"], "class_type": "active"},
+            {"timestamp": 35.0, "source_id": bin_["id"], "class_type": "entered_zone"},
+            {
+                "timestamp": 40.0,
+                "source_id": "someone-else",
+                "class_type": "stationary",
+            },
+        ]
+        timeline = build_timeline([bin_], 100.0, state_changes=changes)
+        self.assertEqual(
+            timeline[1:],
+            [
+                (20.0, 'waste bin "Compost" has stopped moving'),
+                (30.0, 'waste bin "Compost" starts moving again'),
+            ],
+        )
+
+    def test_state_changes_after_the_last_frame_are_dropped(self):
+        bin_ = track(
+            "1789481995.063395-vlzd7q",
+            "waste_bin",
+            0.3,
+            straight_path((0.91, 0.64), (0.6, 0.33), 10, 0.4),
+            sub_label="Compost",
+        )
+        changes = [
+            {"timestamp": 200.0, "source_id": bin_["id"], "class_type": "stationary"}
+        ]
+        timeline = build_timeline([bin_], 100.0, state_changes=changes)
+        self.assertFalse(any("stopped" in p for _, p in timeline))
+
+
+class TestNoAssumedState(unittest.TestCase):
+    def test_no_note_for_where_movement_ends(self):
+        # Ending a leftward walk still in the right third was read as a turn
+        # back to the right, and "stops moving" was read as standing still.
+        moments = path_moments(straight_path((0.95, 0.6), (0.7, 0.4), 6, 0.0))
+        self.assertEqual(
+            [p for _, p in moments],
+            ["starts moving up and left from the right of the frame"],
+        )
+
+    def test_notes_never_claim_an_object_is_stationary(self):
+        # A track still open at the last frame says nothing about motion.
+        event = {
+            "id": "1789482056.695307-3uhf47",
+            "label": "person",
+            "sub_label": None,
+            "start_time": 0.0,
+            "end_time": 500.0,
+            "zones": ["front_yard"],
+            "path_data": straight_path((0.9, 0.6), (0.4, 0.3), 10, 1.0),
+        }
+        joined = " ".join(p for _, p in build_timeline([event], span_end=20.0))
+        self.assertNotIn("stationary", joined)
+        self.assertNotIn("stops", joined)
+        self.assertNotIn("leaves", joined)
+        self.assertNotIn("still", joined)
+
+
+class TestLateEvents(unittest.TestCase):
+    def test_objects_first_detected_after_the_last_frame_are_skipped(self):
+        # Without this the arrival lands on the final frame, which was
+        # captured before the object appeared.
+        late = track(
+            "1789481999.000000-latear",
+            "person",
+            50.0,
+            straight_path((0.9, 0.6), (0.4, 0.3), 10, 50.1),
+        )
+        self.assertEqual(build_timeline([late], span_end=40.0), [])
+        self.assertEqual(
+            annotations_by_frame(build_timeline([late], 40.0), [0.0, 40.0]), {}
+        )
+
+
+class TestFrameBucketing(unittest.TestCase):
+    def test_moment_attaches_to_the_following_frame(self):
+        frame_times = [0.0, 10.0, 20.0, 30.0]
+        buckets = annotations_by_frame([(12.0, "something happened")], frame_times)
+        self.assertEqual(buckets, {2: ["something happened"]})
+
+    def test_moment_on_a_frame_boundary_uses_that_frame(self):
+        buckets = annotations_by_frame([(10.0, "x")], [0.0, 10.0, 20.0])
+        self.assertEqual(buckets, {1: ["x"]})
+
+    def test_moment_after_the_last_frame_falls_on_the_last_frame(self):
+        buckets = annotations_by_frame([(99.0, "x")], [0.0, 10.0])
+        self.assertEqual(buckets, {1: ["x"]})
+
+    def test_no_frames_yields_no_buckets(self):
+        self.assertEqual(annotations_by_frame([(1.0, "x")], []), {})
+
+
+if __name__ == "__main__":
+    unittest.main()

web/public/locales/en/config/cameras.json

@@ -657,6 +657,10 @@
         "label": "Preferred language",
         "description": "Preferred language to request from the GenAI provider for generated responses."
       },
+      "frame_mode": {
+        "label": "Frame mode",
+        "description": "How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses."
+      },
       "response_style": {
         "label": "Response style",
         "description": "Writing style preset for generated review descriptions. Presets adjust the tone and level of detail of the user-facing title, summary, and scene description; 'default' leaves the built-in prompt unchanged."

web/public/locales/en/config/global.json

@@ -1028,6 +1028,10 @@
         "label": "Preferred language",
         "description": "Preferred language to request from the GenAI provider for generated responses."
       },
+      "frame_mode": {
+        "label": "Frame mode",
+        "description": "How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses."
+      },
       "response_style": {
         "label": "Response style",
         "description": "Writing style preset for generated review descriptions. Presets adjust the tone and level of detail of the user-facing title, summary, and scene description; 'default' leaves the built-in prompt unchanged."

web/public/locales/en/views/settings.json

@@ -1912,6 +1912,10 @@
     "imageSource": {
       "recordings": "Recordings",
       "previews": "Previews"
+    },
+    "frameMode": {
+      "frames": "Frames",
+      "annotated_frames": "Annotated frames"
     }
   },
   "logger": {

web/src/components/config-form/section-configs/review.ts

@@ -82,6 +82,7 @@ const review: SectionConfigOverrides = {
       "detections.labels": "/configuration/review/#alerts-and-detections",
       genai: "/configuration/genai/genai_review",
       "genai.image_source": "/configuration/genai/genai_review#image-source",
+      "genai.frame_mode": "/configuration/genai/genai_review#frame-mode",
       "genai.additional_concerns":
         "/configuration/genai/genai_review#additional-concerns",
     },
@@ -138,6 +139,11 @@ const review: SectionConfigOverrides = {
             enumI18nPrefix: "review.imageSource",
           },
         },
+        frame_mode: {
+          "ui:options": {
+            enumI18nPrefix: "review.frameMode",
+          },
+        },
       },
     },
   },