Debug replay resolution (#23287)

* unlink shm frames when camera is removed

* drop stale shm cache refs when cached segment is too small for requested shape

* skip new-object frame cache write when current_frame is unavailable

* add tests

* use setdefault when adding a new camera

Multiple subscribers in the same process each unpickle the ZMQ payload independently and would otherwise write divergent Python objects to the shared cameras dict — leaving long-lived references (e.g. CameraState.camera_config) pointing at a copy that subsequent in-place mutations like apply_section_update can never reach. setdefault collapses everyone onto the first writer's object so attribute mutations propagate to every consumer in this process.

* rebuild ffmpeg commands on detect update

Rebuild the cached ffmpeg cmd so the next process spawn picks up new resolution/fps. Running cameras keep their existing cmd (ffmpeg_cmds is only read at process startup); replay cameras are recycled by CameraMaintainer to pick up the rebuilt cmd

* drop stale shm cache refs when cached segment size doesn't match requested shape

The cached SharedMemoryFrameManager reference can point at a segment whose
size no longer matches the requested shape — the segment was unlinked and
recreated at a different size in a camera add/remove cycle. This catches
both a resolution increase (cached too small) and a decrease (cached too
large, pointing at an orphaned inode whose stale bytes would otherwise be
misinterpreted at the new shape, producing distorted/miscolored YUV frames).

After reopening, if the OS-level segment still doesn't match the requested
shape we're in a transient mid-recreate state — either the maintainer
hasn't allocated the new segment yet (size too small) or we opened a
pre-recycle segment (size too big). Either way, skip the frame and don't
cache the mismatched ref.

* recycle replay camera on detect update

* discard tracked-object state when detect resolution changes mid-session

When detect resolution changes mid-session every tracked object we hold
was localized against the old pixel grid. Their boxes no longer
correspond to anything in the new frame, and the `end` callback that
fires when their IDs disappear from the new detect process's detections
publishes those stale boxes to consumers (LPR, snapshot crop) that slice
the new frame and crash on empty arrays. Drop the tracked-object state
on a shape change so no stale boxes ever cross the CameraState boundary.

Belt-and-suspenders: also drop any incoming batch whose boxes exceed the
current detect resolution. These are in-flight queue entries from the
pre-recycle detect process that beat the new detect process to the
queue; processing them would re-introduce stale-resolution tracked
objects we just dropped above. The per-camera detect process clamps
legitimate boxes to detect.width-1 / detect.height-1, so any coord
beyond that is unambiguously stale.

* rebuild motion and object filter masks on detect resolution change

Apply the detect update first so frame_shape reflects the new resolution
before we rebuild dependents.

Motion's rasterized_mask is sized to frame_shape at construction. When
detect resolution changes we must rebuild RuntimeMotionConfig so the
mask matches the new frame size; otherwise consumers like the LPR
processor and motion detector hit a shape mismatch when they index
frames with the stale mask.

Same story for per-object filter masks — rebuild RuntimeFilterConfig at
the new frame_shape so the merged global+per-object masks they hold
match what they'll be indexed against.

* republish motion and objects on in-memory detect resize

A detect resolution change also invalidates the rasterized masks on
motion and per-object filters. apply_section_update has rebuilt them at
the new frame_shape; publish them too so other processes replace their
old values.

* add test

* frontend

* add refresh topic for camera maintainer recycle action

The maintainer's recycle branch is doing an action (recycle the camera)
in response to a section-level signal. Introduce a
CameraConfigUpdateEnum.refresh case as an explicit action signal — the
maintainer subscribes to refresh instead of detect, parallel with add
and remove. Publishers fire refresh alongside detect when a recycle is
needed; section-level subscribers keep their existing topic.

Since no main-process subscriber listens for detect anymore, the
refresh handler calls recreate_ffmpeg_cmds() explicitly so the shared
CameraConfig's ffmpeg_cmds is rebuilt before the new subprocesses
spawn.

* factor stale-resolution state drop into a CameraState method

frigate/camera/maintainer.py

@@ -14,6 +14,7 @@ from frigate.config.camera.updater import (
     CameraConfigUpdateEnum,
     CameraConfigUpdateSubscriber,
 )
+from frigate.const import REPLAY_CAMERA_PREFIX
 from frigate.models import Regions
 from frigate.util.builtin import empty_and_close_queue
 from frigate.util.image import SharedMemoryFrameManager, UntrackedSharedMemory
@@ -50,6 +51,7 @@ class CameraMaintainer(threading.Thread):
             [
                 CameraConfigUpdateEnum.add,
                 CameraConfigUpdateEnum.remove,
+                CameraConfigUpdateEnum.refresh,
             ],
         )
         self.shm_count = self.__calculate_shm_frame_count()
@@ -202,6 +204,25 @@ class CameraMaintainer(threading.Thread):
                 capture_process.terminate()
                 capture_process.join()
 
+    def __unlink_camera_frame_slots(self, camera: str) -> None:
+        """Drop the camera's per-frame YUV SHM segments from this
+        process's frame_manager and unlink them at the OS level.
+
+        Safe to call after the camera's capture/processor subprocesses
+        have been joined — they no longer hold mappings, so unlink frees
+        the segments immediately. Other long-lived processes that opened
+        these slots will continue using their existing mappings until
+        they call frame_manager.get with a shape that no longer fits
+        (the get path drops and reopens stale refs).
+        """
+        prefix = f"{camera}_frame"
+        names = [n for n in list(self.frame_manager.shm_store) if n.startswith(prefix)]
+        for name in names:
+            try:
+                self.frame_manager.delete(name)
+            except Exception as exc:
+                logger.debug("Could not unlink SHM %s: %s", name, exc)
+
     def __stop_camera_process(self, camera: str) -> None:
         camera_process = self.camera_processes.get(camera)
         if camera_process is not None:
@@ -253,12 +274,45 @@ class CameraMaintainer(threading.Thread):
                     for camera in updated_cameras:
                         self.__stop_camera_capture_process(camera)
                         self.__stop_camera_process(camera)
+                        self.__unlink_camera_frame_slots(camera)
                         self.capture_processes.pop(camera, None)
                         self.camera_processes.pop(camera, None)
                         self.camera_stop_events.pop(camera, None)
                         self.region_grids.pop(camera, None)
                         self.camera_metrics.pop(camera, None)
                         self.ptz_metrics.pop(camera, None)
+                elif update_type == CameraConfigUpdateEnum.refresh.name:
+                    # Recycle replay cameras so detect width/height/fps
+                    # propagate through ffmpeg args, SHM sizing, and the
+                    # region grid. Regular cameras detect change still
+                    # requires a full restart.
+                    for camera in updated_cameras:
+                        if not camera.startswith(REPLAY_CAMERA_PREFIX):
+                            continue
+
+                        new_config = self.update_subscriber.camera_configs.get(camera)
+                        if new_config is None:
+                            # remove arrived in the same batch
+                            continue
+
+                        if (
+                            camera not in self.camera_processes
+                            and camera not in self.capture_processes
+                        ):
+                            continue
+
+                        # rebuild ffmpeg cmds on the shared config so the
+                        # new subprocesses spawn with current args
+                        new_config.recreate_ffmpeg_cmds()
+
+                        self.__stop_camera_capture_process(camera)
+                        self.__stop_camera_process(camera)
+                        self.__unlink_camera_frame_slots(camera)
+                        self.capture_processes.pop(camera, None)
+                        self.camera_processes.pop(camera, None)
+
+                        self.__start_camera_processor(camera, new_config, runtime=True)
+                        self.__start_camera_capture(camera, new_config, runtime=True)
 
         # ensure the capture processes are done
         for camera in self.capture_processes.keys():

frigate/camera/state.py

@@ -45,6 +45,7 @@ class CameraState:
         self.frame_cache: dict[float, dict[str, Any]] = {}
         self.zone_objects: defaultdict[str, list[Any]] = defaultdict(list)
         self._current_frame = np.zeros(self.camera_config.frame_shape_yuv, np.uint8)
+        self._last_frame_shape: tuple[int, int] = self.camera_config.frame_shape_yuv
         self.current_frame_lock = threading.Lock()
         self.current_frame_time = 0.0
         self.motion_boxes: list[tuple[int, int, int, int]] = []
@@ -303,6 +304,42 @@ class CameraState:
     def on(self, event_type: str, callback: Callable[..., Any]) -> None:
         self.callbacks[event_type].append(callback)
 
+    def _discard_stale_resolution_state(
+        self, current_detections: dict[str, dict[str, Any]]
+    ) -> bool:
+        """Drop tracked state when the camera's detect resolution has
+        changed, and signal the caller to skip this batch if it contains
+        out-of-bounds boxes from the pre-recycle detect process.
+
+        Returns True when the batch should be skipped entirely.
+        """
+        # detect resolution changed — drop tracked state so old-grid
+        # boxes don't leak through end-callbacks
+        current_shape = self.camera_config.frame_shape_yuv
+        if current_shape != self._last_frame_shape:
+            logger.debug(
+                f"{self.name}: detect resolution changed {self._last_frame_shape} -> {current_shape}, dropping tracked state"
+            )
+            with self.current_frame_lock:
+                self.tracked_objects.clear()
+                self.motion_boxes = []
+                self.regions = []
+            self._last_frame_shape = current_shape
+
+        # drop in-flight batches from the pre-recycle detect process
+        # whose boxes exceed the current detect resolution
+        detect = self.camera_config.detect
+        if detect.width is not None and detect.height is not None:
+            for obj in current_detections.values():
+                box = obj.get("box")
+                if box and (box[2] > detect.width or box[3] > detect.height):
+                    logger.debug(
+                        f"{self.name}: dropping stale-resolution detection batch (box {box} exceeds {detect.width}x{detect.height})"
+                    )
+                    return True
+
+        return False
+
     def update(
         self,
         frame_name: str,
@@ -311,6 +348,9 @@ class CameraState:
         motion_boxes: list[tuple[int, int, int, int]],
         regions: list[tuple[int, int, int, int]],
     ) -> None:
+        if self._discard_stale_resolution_state(current_detections):
+            return
+
         current_frame = self.frame_manager.get(
             frame_name, self.camera_config.frame_shape_yuv
         )
@@ -332,14 +372,18 @@ class CameraState:
                 current_detections[id],
             )
 
-            # add initial frame to frame cache
-            logger.debug(
-                f"{self.name}: New object, adding {frame_time} to frame cache for {id}"
-            )
-            self.frame_cache[frame_time] = {
-                "frame": np.copy(current_frame),  # type: ignore[arg-type]
-                "object_id": id,
-            }
+            # Skip caching when the frame buffer isn't readable — e.g.
+            # frame_manager.get returned None because the SHM segment was
+            # unlinked or hasn't been recreated yet during a camera
+            # add/remove cycle.
+            if current_frame is not None:
+                logger.debug(
+                    f"{self.name}: New object, adding {frame_time} to frame cache for {id}"
+                )
+                self.frame_cache[frame_time] = {
+                    "frame": np.copy(current_frame),
+                    "object_id": id,
+                }
 
             # save initial thumbnail data and best object
             thumbnail_data = {