Use record stream snapshots for detections

docs/docs/configuration/custom_classification/object_classification.md

@@ -83,6 +83,26 @@ classification:
 
 An optional config, `save_attempts`, can be set as a key under the model name. This defines the number of classification attempts to save in the Recent Classifications tab. For object classification models, the default is 200.
 
+### Recording Snapshot Fallback
+
+When using a low-resolution sub-stream for the `detect` role, distant or small objects may lack sufficient detail for accurate classification. The `use_recording_snapshot` option allows Frigate to fall back to extracting a high-resolution frame from the already-recorded main stream segments on disk when the classification score on the detect frame is below the configured threshold.
+
+```yaml
+classification:
+  custom:
+    dog:
+      threshold: 0.8
+      use_recording_snapshot: true
+      object_config:
+        objects: [dog]
+        classification_type: sub_label
+```
+
+- Default: `False`
+- Requires `record` to be enabled for the camera.
+- No additional ffmpeg decode processes are spawned — a single frame is extracted from the mp4 segment (~50–100ms CPU, no VRAM cost).
+- The existing multi-frame consensus scoring (60% agreement over 3+ attempts) handles any segment availability delays naturally.
+
 ## Training the model
 
 Creating and training the model is done within the Frigate UI using the `Classification` page. The process consists of two steps:

docs/docs/configuration/custom_classification/state_classification.md

@@ -50,6 +50,30 @@ classification:
 
 An optional config, `save_attempts`, can be set as a key under the model name. This defines the number of classification attempts to save in the Recent Classifications tab. For state classification models, the default is 100.
 
+### Recording Snapshot Fallback
+
+When using a low-resolution sub-stream for the `detect` role, the crop region may lack sufficient detail for accurate state classification. The `use_recording_snapshot` option allows Frigate to fall back to extracting a high-resolution frame from the already-recorded main stream segments on disk when the classification score on the detect frame is below the configured threshold.
+
+```yaml
+classification:
+  custom:
+    front_door:
+      threshold: 0.8
+      use_recording_snapshot: true
+      state_config:
+        motion: true
+        interval: 10
+        cameras:
+          front:
+            crop: [0, 180, 220, 400]
+```
+
+- Default: `False`
+- Requires `record` to be enabled for the camera.
+- No additional ffmpeg decode processes are spawned — a single frame is extracted from the mp4 segment (~50–100ms CPU, no VRAM cost).
+- The crop coordinates are scaled from detect resolution to recording resolution automatically.
+- The existing 3-frame state verification handles any segment availability delays naturally.
+
 ## Training the model
 
 Creating and training the model is done within the Frigate UI using the `Classification` page. The process consists of three steps:

docs/docs/configuration/face_recognition.md

@@ -49,7 +49,7 @@ Like the other real-time processors in Frigate, face recognition runs on the cam
 
 ## Advanced Configuration
 
-Fine-tune face recognition with these optional parameters at the global level of your config. The only optional parameters that can be set at the camera level are `enabled` and `min_area`.
+Fine-tune face recognition with these optional parameters at the global level of your config. The only optional parameters that can be set at the camera level are `enabled`, `min_area`, and `use_recording_snapshot`.
 
 ### Detection
 
@@ -77,6 +77,21 @@ Fine-tune face recognition with these optional parameters at the global level of
   - Default: `None`.
   - Note: This setting is only applicable when using the `large` model. See [onnxruntime's provider options](https://onnxruntime.ai/docs/execution-providers/)
 
+### Recording Snapshot Fallback
+
+- `use_recording_snapshot`: When enabled, if a detected face is too small on the detect stream (below `min_area`), Frigate will extract a high-resolution frame from the already-recorded main stream segments on disk and re-run face detection on that frame.
+  - Default: `False`
+  - Requires `record` to be enabled for the camera.
+  - No additional ffmpeg decode processes are spawned and no extra VRAM is consumed — the existing recording pipeline does all the heavy lifting.
+  - There may be a 2–5 second delay before the recording segment is available, but this is handled naturally by Frigate's multi-frame face history accumulator.
+  - This is especially useful when using a low-resolution sub-stream for the `detect` role (the recommended Frigate setup), where faces on distant persons are too small for recognition.
+
+```yaml
+face_recognition:
+  enabled: true
+  use_recording_snapshot: true
+```
+
 ## Usage
 
 Follow these steps to begin:
@@ -198,12 +213,16 @@ No, using another face recognition service will interfere with Frigate's built i
 
 ### Does face recognition run on the recording stream?
 
-Face recognition does not run on the recording stream, this would be suboptimal for many reasons:
+By default, face recognition runs only on the detect stream. However, with the `use_recording_snapshot` option enabled, Frigate can **fall back** to extracting a single high-resolution frame from the recording segments on disk when the face on the detect stream is too small. This is not the same as continuously decoding the main stream — it only extracts one frame per enrichment attempt (~50–100ms CPU, no VRAM cost).
+
+Without `use_recording_snapshot`, there are good reasons for not always using the recording stream:
 
 1. The latency of accessing the recordings means the notifications would not include the names of recognized people because recognition would not complete until after.
 2. The embedding models used run on a set image size, so larger images will be scaled down to match this anyway.
 3. Motion clarity is much more important than extra pixels, over-compression and motion blur are much more detrimental to results than resolution.
 
+The recording snapshot fallback is a middle ground: detect on the efficient sub-stream as usual, and only access the high-res recording when the detect stream resolution is genuinely insufficient.
+
 ### I get an unknown error when taking a photo directly with my iPhone
 
 By default iOS devices will use HEIC (High Efficiency Image Container) for images, but this format is not supported for uploads. Choosing `large` as the format instead of `original` will use JPG which will work correctly.

docs/docs/configuration/license_plate_recognition.md

@@ -57,7 +57,7 @@ Like the other real-time processors in Frigate, license plate recognition runs o
 
 ## Advanced Configuration
 
-Fine-tune the LPR feature using these optional parameters at the global level of your config. The only optional parameters that can be set at the camera level are `enabled`, `min_area`, and `enhancement`.
+Fine-tune the LPR feature using these optional parameters at the global level of your config. The only optional parameters that can be set at the camera level are `enabled`, `min_area`, `enhancement`, and `use_recording_snapshot`.
 
 ### Detection
 
@@ -131,6 +131,14 @@ lpr:
 - Any changes made by the rules are printed to the LPR debug log.
 - Tip: You can test patterns with tools like regex101.com.
 
+### Recording Snapshot Fallback
+
+- **`use_recording_snapshot`**: When enabled, if a detected license plate is too small on the detect stream (below `min_area`) or no plate is found at all, Frigate will extract a high-resolution frame from the already-recorded main stream segments on disk and re-run plate detection and OCR on that frame.
+  - Default: `False`
+  - Requires `record` to be enabled for the camera.
+  - No additional ffmpeg decode processes are spawned — a single frame is extracted from the mp4 segment on disk (~50–100ms CPU, no VRAM cost).
+  - This is especially useful when using a low-resolution sub-stream for the `detect` role, where plates on moving or distant cars are too small for recognition.
+
 ### Debugging
 
 - **`debug_save_plates`**: Set to `True` to save captured text on plates for debugging. These images are stored in `/media/frigate/clips/lpr`, organized into subdirectories by `<camera>/<event_id>`, and named based on the capture timestamp.
@@ -139,7 +147,7 @@ lpr:
 
 ## Configuration Examples
 
-These configuration parameters are available at the global level of your config. The only optional parameters that should be set at the camera level are `enabled`, `min_area`, and `enhancement`.
+These configuration parameters are available at the global level of your config. The only optional parameters that should be set at the camera level are `enabled`, `min_area`, `enhancement`, and `use_recording_snapshot`.
 
 ```yaml
 lpr:

docs/docs/configuration/reference.md

@@ -659,7 +659,7 @@ semantic_search:
   device: None
 
 # Optional: Configuration for face recognition capability
-# NOTE: enabled, min_area can be overridden at the camera level
+# NOTE: enabled, min_area, use_recording_snapshot can be overridden at the camera level
 face_recognition:
   # Optional: Enable face recognition (default: shown below)
   enabled: False
@@ -683,9 +683,12 @@ face_recognition:
   # Optional: Target a specific device to run the model (default: shown below)
   # NOTE: See https://onnxruntime.ai/docs/execution-providers/ for more information
   device: None
+  # Optional: Fall back to extracting a hi-res frame from recording segments when
+  # the detect stream resolution is insufficient for face recognition (default: shown below)
+  use_recording_snapshot: False
 
 # Optional: Configuration for license plate recognition capability
-# NOTE: enabled, min_area, and enhancement can be overridden at the camera level
+# NOTE: enabled, min_area, enhancement, and use_recording_snapshot can be overridden at the camera level
 lpr:
   # Optional: Enable license plate recognition (default: shown below)
   enabled: False
@@ -715,6 +718,9 @@ lpr:
   debug_save_plates: False
   # Optional: List of regex replacement rules to normalize detected plates (default: shown below)
   replace_rules: {}
+  # Optional: Fall back to extracting a hi-res frame from recording segments when
+  # the detect stream resolution is insufficient for license plate recognition (default: shown below)
+  use_recording_snapshot: False
 
 # Optional: Configuration for AI / LLM provider
 # WARNING: Depending on the provider, this will send thumbnails over the internet
@@ -769,6 +775,9 @@ classification:
       # Optional: Number of classification attempts to save in the recent classifications tab (default: shown below)
       # NOTE: Defaults to 200 for object classification and 100 for state classification if not specified
       save_attempts: None
+      # Optional: Fall back to extracting a hi-res frame from recording segments when
+      # the detect stream resolution is insufficient for classification (default: shown below)
+      use_recording_snapshot: False
       # Optional: Object classification configuration
       object_config:
         # Required: Object types to classify

frigate/config/classification.py

@@ -145,6 +145,11 @@ class CustomClassificationConfig(FrigateBaseModel):
         description="How many classification attempts to save for recent classifications UI.",
         ge=0,
     )
+    use_recording_snapshot: bool = Field(
+        default=False,
+        title="Use recording snapshot",
+        description="When enabled, fall back to extracting a high-resolution frame from recording segments when the detect stream resolution is insufficient for classification.",
+    )
     object_config: CustomClassificationObjectConfig | None = Field(default=None)
     state_config: CustomClassificationStateConfig | None = Field(default=None)
 
@@ -296,6 +301,11 @@ class FaceRecognitionConfig(FrigateBaseModel):
         title="Device",
         description="This is an override, to target a specific device. See https://onnxruntime.ai/docs/execution-providers/ for more information",
     )
+    use_recording_snapshot: bool = Field(
+        default=False,
+        title="Use recording snapshot",
+        description="When enabled, fall back to extracting a high-resolution frame from recording segments when the detect stream resolution is insufficient for face recognition.",
+    )
 
 
 class CameraFaceRecognitionConfig(FrigateBaseModel):
@@ -309,6 +319,11 @@ class CameraFaceRecognitionConfig(FrigateBaseModel):
         title="Minimum face area",
         description="Minimum area (pixels) of a detected face box required to attempt recognition.",
     )
+    use_recording_snapshot: bool = Field(
+        default=False,
+        title="Use recording snapshot",
+        description="When enabled, fall back to extracting a high-resolution frame from recording segments when the detect stream resolution is insufficient for face recognition.",
+    )
 
     model_config = ConfigDict(extra="forbid", protected_namespaces=())
 
@@ -391,6 +406,11 @@ class LicensePlateRecognitionConfig(FrigateBaseModel):
         title="Replacement rules",
         description="Regex replacement rules used to normalize detected plate strings before matching.",
     )
+    use_recording_snapshot: bool = Field(
+        default=False,
+        title="Use recording snapshot",
+        description="When enabled, fall back to extracting a high-resolution frame from recording segments when the detect stream resolution is insufficient for license plate recognition.",
+    )
 
 
 class CameraLicensePlateRecognitionConfig(FrigateBaseModel):
@@ -417,6 +437,11 @@ class CameraLicensePlateRecognitionConfig(FrigateBaseModel):
         ge=0,
         le=10,
     )
+    use_recording_snapshot: bool = Field(
+        default=False,
+        title="Use recording snapshot",
+        description="When enabled, fall back to extracting a high-resolution frame from recording segments when the detect stream resolution is insufficient for license plate recognition.",
+    )
 
     model_config = ConfigDict(extra="forbid", protected_namespaces=())
 

frigate/config/config.py

@@ -685,8 +685,14 @@ class FrigateConfig(FrigateBaseModel):
 
             # only populate some fields down to the camera level for specific keys
             allowed_fields_map = {
-                "face_recognition": ["enabled", "min_area"],
-                "lpr": ["enabled", "expire_time", "min_area", "enhancement"],
+                "face_recognition": ["enabled", "min_area", "use_recording_snapshot"],
+                "lpr": [
+                    "enabled",
+                    "expire_time",
+                    "min_area",
+                    "enhancement",
+                    "use_recording_snapshot",
+                ],
                 "audio_transcription": ["enabled", "live_enabled"],
             }
 

frigate/data_processing/common/license_plate/mixin.py

@@ -27,6 +27,7 @@ from frigate.embeddings.onnx.lpr_embedding import LPR_EMBEDDING_SIZE
 from frigate.types import TrackedObjectUpdateTypesEnum
 from frigate.util.builtin import EventsPerSecond, InferenceSpeed
 from frigate.util.image import area
+from frigate.util.recording_frame import get_recording_frame, scale_bounding_box
 
 logger = logging.getLogger(__name__)
 
@@ -1176,6 +1177,97 @@ class LicensePlateProcessingMixin:
         )
         return event_id
 
+    def _try_hires_lpr(
+        self,
+        camera: str,
+        obj_data: dict[str, Any],
+        frame_time: float,
+    ) -> Optional[np.ndarray]:
+        """Try to get a license plate frame from hi-res recording.
+
+        Extracts the vehicle region from a recording frame and runs
+        plate detection on it. Returns the plate crop ready for OCR,
+        or None if unsuccessful.
+        """
+        camera_config = self.config.cameras[camera]
+
+        if not camera_config.lpr.use_recording_snapshot:
+            return None
+
+        if not camera_config.record.enabled:
+            logger.debug(
+                f"{camera}: Recording not enabled, cannot use recording snapshot for LPR"
+            )
+            return None
+
+        car_box = obj_data.get("box")
+        if not car_box:
+            return None
+
+        logger.debug(
+            f"{camera}: Plate too small on detect stream, trying recording snapshot"
+        )
+
+        hires_frame = get_recording_frame(self.config.ffmpeg, camera, frame_time)
+        if hires_frame is None:
+            logger.debug(f"{camera}: Recording frame not available yet for LPR")
+            return None
+
+        detect_res = (camera_config.detect.width, camera_config.detect.height)
+        record_res = (hires_frame.shape[1], hires_frame.shape[0])
+
+        if record_res[0] <= detect_res[0] and record_res[1] <= detect_res[1]:
+            logger.debug(
+                f"{camera}: Recording resolution {record_res} not higher than detect {detect_res}, skipping LPR fallback"
+            )
+            return None
+
+        scaled_box = scale_bounding_box(car_box, detect_res, record_res, padding=0.20)
+        left, top, right, bottom = scaled_box
+        car_crop = hires_frame[top:bottom, left:right]
+
+        if car_crop.size == 0:
+            return None
+
+        # double the size for better box detection (same as detect-stream path)
+        car_crop = cv2.resize(
+            car_crop, (int(2 * car_crop.shape[1]), int(2 * car_crop.shape[0]))
+        )
+
+        license_plate = self._detect_license_plate(camera, car_crop)
+        if not license_plate:
+            logger.debug(f"{camera}: No plate found in hi-res car crop")
+            return None
+
+        license_plate_area = max(
+            0,
+            (license_plate[2] - license_plate[0])
+            * (license_plate[3] - license_plate[1]),
+        )
+
+        # doubled size, so compare against min_area * 2
+        if license_plate_area < camera_config.lpr.min_area * 2:
+            logger.debug(
+                f"{camera}: Plate still too small in hi-res frame: {license_plate_area} < {camera_config.lpr.min_area * 2}"
+            )
+            return None
+
+        plate_frame = car_crop[
+            license_plate[1] : license_plate[3],
+            license_plate[0] : license_plate[2],
+        ]
+
+        # double the size for better OCR (same as detect-stream path)
+        plate_frame = cv2.resize(
+            plate_frame,
+            (int(2 * plate_frame.shape[1]), int(2 * plate_frame.shape[0])),
+        )
+
+        logger.debug(
+            f"{camera}: Successfully extracted plate from hi-res recording frame (area={license_plate_area})"
+        )
+        return plate_frame
+
     def lpr_process(
         self, obj_data: dict[str, Any], frame: np.ndarray, dedicated_lpr: bool = False
     ):
@@ -1329,38 +1421,55 @@ class LicensePlateProcessingMixin:
                     logger.debug(
                         f"{camera}: Detected no license plates for car/motorcycle object."
                     )
-                    return
+                    # try hi-res fallback
+                    hires_plate = self._try_hires_lpr(
+                        camera, obj_data, obj_data.get("frame_time", current_time)
+                    )
+                    if hires_plate is None:
+                        return
+                    license_plate_frame = hires_plate
+                    plate_box = car_box
+                else:
+                    license_plate_area = max(
+                        0,
+                        (license_plate[2] - license_plate[0])
+                        * (license_plate[3] - license_plate[1]),
+                    )
 
-                license_plate_area = max(
-                    0,
-                    (license_plate[2] - license_plate[0])
-                    * (license_plate[3] - license_plate[1]),
-                )
+                    # check that license plate is valid
+                    # double the value because we've doubled the size of the car
+                    if (
+                        license_plate_area
+                        < self.config.cameras[camera].lpr.min_area * 2
+                    ):
+                        logger.debug(f"{camera}: License plate is less than min_area")
+                        # try hi-res fallback
+                        hires_plate = self._try_hires_lpr(
+                            camera, obj_data, obj_data.get("frame_time", current_time)
+                        )
+                        if hires_plate is None:
+                            return
+                        license_plate_frame = hires_plate
+                        plate_box = car_box
+                    else:
+                        # Scale back to original car coordinates and then to frame
+                        plate_box_in_car = (
+                            license_plate[0] // 2,
+                            license_plate[1] // 2,
+                            license_plate[2] // 2,
+                            license_plate[3] // 2,
+                        )
+                        plate_box = (
+                            left + plate_box_in_car[0],
+                            top + plate_box_in_car[1],
+                            left + plate_box_in_car[2],
+                            top + plate_box_in_car[3],
+                        )
 
-                # check that license plate is valid
-                # double the value because we've doubled the size of the car
-                if license_plate_area < self.config.cameras[camera].lpr.min_area * 2:
-                    logger.debug(f"{camera}: License plate is less than min_area")
-                    return
-
-                # Scale back to original car coordinates and then to frame
-                plate_box_in_car = (
-                    license_plate[0] // 2,
-                    license_plate[1] // 2,
-                    license_plate[2] // 2,
-                    license_plate[3] // 2,
-                )
-                plate_box = (
-                    left + plate_box_in_car[0],
-                    top + plate_box_in_car[1],
-                    left + plate_box_in_car[2],
-                    top + plate_box_in_car[3],
-                )
-
-                license_plate_frame = car[
-                    license_plate[1] : license_plate[3],
-                    license_plate[0] : license_plate[2],
-                ]
+                        license_plate_frame = car[
+                            license_plate[1] : license_plate[3],
+                            license_plate[0] : license_plate[2],
+                        ]
             else:
                 # don't run for object without attributes if this isn't dedicated lpr with frigate+
                 if (
@@ -1400,33 +1509,42 @@ class LicensePlateProcessingMixin:
                     < self.config.cameras[camera].lpr.min_area
                 ):
                     logger.debug(
-                        f"{camera}: Area for license plate box {area(license_plate_box)} is less than min_area {self.config.cameras[camera].lpr.min_area}"
+                        f"{camera}: Area for license plate box {area(license_plate_box) if license_plate_box else 0} is less than min_area {self.config.cameras[camera].lpr.min_area}"
                     )
-                    return
+                    # try hi-res fallback for attribute-based path
+                    hires_plate = self._try_hires_lpr(
+                        camera, obj_data, obj_data.get("frame_time", current_time)
+                    )
+                    if hires_plate is None:
+                        return
+                    license_plate_frame = hires_plate
+                    plate_box = license_plate_box or obj_data.get("box", (0, 0, 0, 0))
+                else:
+                    license_plate_frame = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_I420)
 
-                license_plate_frame = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_I420)
+                    # Expand the license_plate_box by 10%
+                    box_array = np.array(license_plate_box)
+                    expansion = (box_array[2:] - box_array[:2]) * 0.10
+                    expanded_box = np.array(
+                        [
+                            license_plate_box[0] - expansion[0],
+                            license_plate_box[1] - expansion[1],
+                            license_plate_box[2] + expansion[0],
+                            license_plate_box[3] + expansion[1],
+                        ]
+                    ).clip(
+                        0,
+                        [license_plate_frame.shape[1], license_plate_frame.shape[0]]
+                        * 2,
+                    )
 
-                # Expand the license_plate_box by 10%
-                box_array = np.array(license_plate_box)
-                expansion = (box_array[2:] - box_array[:2]) * 0.10
-                expanded_box = np.array(
-                    [
-                        license_plate_box[0] - expansion[0],
-                        license_plate_box[1] - expansion[1],
-                        license_plate_box[2] + expansion[0],
-                        license_plate_box[3] + expansion[1],
+                    plate_box = tuple(int(x) for x in expanded_box)
+
+                    # Crop using the expanded box
+                    license_plate_frame = license_plate_frame[
+                        int(expanded_box[1]) : int(expanded_box[3]),
+                        int(expanded_box[0]) : int(expanded_box[2]),
                     ]
-                ).clip(
-                    0, [license_plate_frame.shape[1], license_plate_frame.shape[0]] * 2
-                )
-
-                plate_box = tuple(int(x) for x in expanded_box)
-
-                # Crop using the expanded box
-                license_plate_frame = license_plate_frame[
-                    int(expanded_box[1]) : int(expanded_box[3]),
-                    int(expanded_box[0]) : int(expanded_box[2]),
-                ]
 
             # double the size of the license plate frame for better OCR
             license_plate_frame = cv2.resize(

frigate/data_processing/real_time/custom_classification.py

@@ -25,6 +25,7 @@ from frigate.log import suppress_stderr_during
 from frigate.types import TrackedObjectUpdateTypesEnum
 from frigate.util.builtin import EventsPerSecond, InferenceSpeed, load_labels
 from frigate.util.object import box_overlaps, calculate_region
+from frigate.util.recording_frame import get_recording_frame, scale_bounding_box
 
 from ..types import DataProcessorMetrics
 from .api import RealTimeProcessorApi
@@ -172,6 +173,60 @@ class CustomStateClassificationProcessor(RealTimeProcessorApi):
 
         return None
 
+    def _try_hires_state_classification(
+        self, camera: str, camera_config, frame_time: float
+    ) -> tuple[str, float] | None:
+        """Try state classification on a hi-res recording frame.
+
+        Returns (detected_state, score) or None.
+        """
+        logger.debug(
+            f"{camera}: State classification score below threshold, trying recording snapshot"
+        )
+
+        hires_frame = get_recording_frame(self.config.ffmpeg, camera, frame_time)
+        if hires_frame is None:
+            logger.debug(
+                f"{camera}: Recording frame not available for state classification"
+            )
+            return None
+
+        h, w = hires_frame.shape[:2]
+
+        # Scale normalized crop coordinates to hi-res resolution
+        x1 = max(0, min(int(camera_config.crop[0] * w), w))
+        y1 = max(0, min(int(camera_config.crop[1] * h), h))
+        x2 = max(0, min(int(camera_config.crop[2] * w), w))
+        y2 = max(0, min(int(camera_config.crop[3] * h), h))
+
+        if x2 <= x1 or y2 <= y1:
+            return None
+
+        crop = hires_frame[y1:y2, x1:x2]
+        # hires_frame is BGR from get_recording_frame, convert to RGB for model
+        crop_rgb = cv2.cvtColor(crop, cv2.COLOR_BGR2RGB)
+
+        try:
+            resized = cv2.resize(crop_rgb, (224, 224))
+        except Exception:
+            return None
+
+        input_data = np.expand_dims(resized, axis=0)
+        self.interpreter.set_tensor(self.tensor_input_details[0]["index"], input_data)
+        self.interpreter.invoke()
+        res: np.ndarray = self.interpreter.get_tensor(
+            self.tensor_output_details[0]["index"]
+        )[0]
+        probs = res / res.sum(axis=0)
+        best_id = np.argmax(probs)
+        score = round(probs[best_id], 2)
+        detected_state = self.labelmap[best_id]
+
+        logger.debug(
+            f"{camera}: Hi-res state classification: {detected_state} (score={score})"
+        )
+        return detected_state, score
+
     def process_frame(self, frame_data: dict[str, Any], frame: np.ndarray):
         if self.metrics and self.model_config.name in self.metrics.classification_cps:
             self.metrics.classification_cps[
@@ -309,7 +364,22 @@ class CustomStateClassificationProcessor(RealTimeProcessorApi):
             logger.debug(
                 f"Score {score} below threshold {self.model_config.threshold}, skipping verification"
             )
-            return
+            # try hi-res fallback for state classification
+            if (
+                self.model_config.use_recording_snapshot
+                and self.config.cameras[camera].record.enabled
+            ):
+                hires_result = self._try_hires_state_classification(
+                    camera, camera_config, now
+                )
+                if hires_result is not None:
+                    detected_state, score = hires_result
+                    if score < self.model_config.threshold:
+                        return
+                else:
+                    return
+            else:
+                return
 
         verified_state = self.verify_state_change(camera, detected_state)
 
@@ -470,6 +540,66 @@ class CustomObjectClassificationProcessor(RealTimeProcessorApi):
         )
         return best_label, avg_score
 
+    def _try_hires_object_classification(
+        self, camera: str, obj_data: dict[str, Any], frame_time: float
+    ) -> tuple[str, float] | None:
+        """Try object classification on a hi-res recording frame.
+
+        Returns (label, score) or None.
+        """
+        camera_config = self.config.cameras[camera]
+
+        logger.debug(
+            f"{camera}: Object classification score below threshold, trying recording snapshot"
+        )
+
+        hires_frame = get_recording_frame(self.config.ffmpeg, camera, frame_time)
+        if hires_frame is None:
+            logger.debug(
+                f"{camera}: Recording frame not available for object classification"
+            )
+            return None
+
+        detect_res = (camera_config.detect.width, camera_config.detect.height)
+        record_res = (hires_frame.shape[1], hires_frame.shape[0])
+
+        if record_res[0] <= detect_res[0] and record_res[1] <= detect_res[1]:
+            logger.debug(
+                f"{camera}: Recording resolution not higher than detect, skipping classification fallback"
+            )
+            return None
+
+        scaled_box = scale_bounding_box(
+            obj_data["box"], detect_res, record_res, padding=0.10
+        )
+        left, top, right, bottom = scaled_box
+        crop = hires_frame[top:bottom, left:right]
+
+        if crop.size == 0:
+            return None
+
+        # hires_frame is BGR, convert to RGB for the model
+        crop_rgb = cv2.cvtColor(crop, cv2.COLOR_BGR2RGB)
+
+        try:
+            resized = cv2.resize(crop_rgb, (224, 224))
+        except Exception:
+            return None
+
+        input_data = np.expand_dims(resized, axis=0)
+        self.interpreter.set_tensor(self.tensor_input_details[0]["index"], input_data)
+        self.interpreter.invoke()
+        res: np.ndarray = self.interpreter.get_tensor(
+            self.tensor_output_details[0]["index"]
+        )[0]
+        probs = res / res.sum(axis=0)
+        best_id = np.argmax(probs)
+        score = round(probs[best_id], 2)
+        label = self.labelmap[best_id]
+
+        logger.debug(f"{camera}: Hi-res object classification: {label} (score={score})")
+        return label, score
+
     def process_frame(self, obj_data, frame):
         if self.metrics and self.model_config.name in self.metrics.classification_cps:
             self.metrics.classification_cps[
@@ -578,9 +708,25 @@ class CustomObjectClassificationProcessor(RealTimeProcessorApi):
             logger.debug(
                 f"{self.model_config.name}: Score {score} < threshold {self.model_config.threshold} for {object_id}, skipping"
             )
-            return
-
-        sub_label = self.labelmap[best_id]
+            # try hi-res fallback for object classification
+            camera = obj_data["camera"]
+            if (
+                self.model_config.use_recording_snapshot
+                and self.config.cameras[camera].record.enabled
+            ):
+                hires_result = self._try_hires_object_classification(
+                    camera, obj_data, now
+                )
+                if hires_result is not None:
+                    sub_label, score = hires_result
+                    if score < self.model_config.threshold:
+                        return
+                else:
+                    return
+            else:
+                return
+        else:
+            sub_label = self.labelmap[best_id]
 
         logger.debug(
             f"{self.model_config.name}: Object {object_id} (label={obj_data['label']}) passed threshold with sub_label={sub_label}, score={score}"

frigate/data_processing/real_time/face.py

@@ -28,6 +28,7 @@ from frigate.data_processing.common.face.model import (
 from frigate.types import TrackedObjectUpdateTypesEnum
 from frigate.util.builtin import EventsPerSecond, InferenceSpeed
 from frigate.util.image import area
+from frigate.util.recording_frame import get_recording_frame, scale_bounding_box
 
 from ..types import DataProcessorMetrics
 from .api import RealTimeProcessorApi
@@ -177,6 +178,79 @@ class FaceRealTimeProcessor(RealTimeProcessorApi):
         self.faces_per_second.update()
         self.inference_speed.update(duration)
 
+    def _try_hires_face_detection(
+        self, camera: str, obj_data: dict[str, Any], frame_time: float
+    ) -> Optional[np.ndarray]:
+        """Attempt face detection on a hi-res frame from recordings.
+
+        Returns the face crop (BGR) if successful, or None.
+        """
+        camera_config = self.config.cameras[camera]
+
+        if not camera_config.face_recognition.use_recording_snapshot:
+            return None
+
+        if not camera_config.record.enabled:
+            logger.debug(
+                f"{camera}: Recording not enabled, cannot use recording snapshot for face"
+            )
+            return None
+
+        person_box = obj_data.get("box")
+        if not person_box:
+            return None
+
+        logger.debug(
+            f"{camera}: Face too small on detect stream, trying recording snapshot"
+        )
+
+        hires_frame = get_recording_frame(self.config.ffmpeg, camera, frame_time)
+        if hires_frame is None:
+            logger.debug(f"{camera}: Recording frame not available yet")
+            return None
+
+        detect_res = (camera_config.detect.width, camera_config.detect.height)
+        record_res = (hires_frame.shape[1], hires_frame.shape[0])
+
+        if record_res[0] <= detect_res[0] and record_res[1] <= detect_res[1]:
+            logger.debug(
+                f"{camera}: Recording resolution {record_res} not higher than detect {detect_res}, skipping"
+            )
+            return None
+
+        scaled_box = scale_bounding_box(
+            person_box, detect_res, record_res, padding=0.15
+        )
+        left, top, right, bottom = scaled_box
+        person_crop = hires_frame[top:bottom, left:right]
+
+        if person_crop.size == 0:
+            return None
+
+        face_box = self.__detect_face(person_crop, self.face_config.detection_threshold)
+        if not face_box:
+            logger.debug(f"{camera}: No face found in hi-res person crop")
+            return None
+
+        if area(face_box) < camera_config.face_recognition.min_area:
+            logger.debug(
+                f"{camera}: Face still too small in hi-res frame: {area(face_box)} < {camera_config.face_recognition.min_area}"
+            )
+            return None
+
+        face_crop = person_crop[
+            max(0, face_box[1]) : min(person_crop.shape[0], face_box[3]),
+            max(0, face_box[0]) : min(person_crop.shape[1], face_box[2]),
+        ]
+
+        if face_crop.size == 0:
+            return None
+
+        logger.debug(
+            f"{camera}: Successfully extracted face from hi-res recording frame (area={area(face_box)})"
+        )
+        return face_crop
+
     def process_frame(self, obj_data: dict[str, Any], frame: np.ndarray):
         """Look for faces in image."""
         self.metrics.face_rec_fps.value = self.faces_per_second.eps()
@@ -236,25 +310,41 @@ class FaceRealTimeProcessor(RealTimeProcessorApi):
 
             if not face_box:
                 logger.debug("Detected no faces for person object.")
-                return
-
-            face_frame = person[
-                max(0, face_box[1]) : min(frame.shape[0], face_box[3]),
-                max(0, face_box[0]) : min(frame.shape[1], face_box[2]),
-            ]
-
-            # check that face is correct size
-            if area(face_box) < self.config.cameras[camera].face_recognition.min_area:
-                logger.debug(
-                    f"Detected face that is smaller than the min_area {face} < {self.config.cameras[camera].face_recognition.min_area}"
+                # try hi-res fallback when no face detected at all
+                face_frame = self._try_hires_face_detection(
+                    camera, obj_data, obj_data.get("frame_time", start)
                 )
-                return
+                if face_frame is None:
+                    return
+            else:
+                face_frame = person[
+                    max(0, face_box[1]) : min(frame.shape[0], face_box[3]),
+                    max(0, face_box[0]) : min(frame.shape[1], face_box[2]),
+                ]
 
-            try:
-                face_frame = cv2.cvtColor(face_frame, cv2.COLOR_RGB2BGR)
-            except Exception as e:
-                logger.debug(f"Failed to convert face frame color for {id}: {e}")
-                return
+                # check that face is correct size
+                if (
+                    area(face_box)
+                    < self.config.cameras[camera].face_recognition.min_area
+                ):
+                    logger.debug(
+                        f"Detected face that is smaller than the min_area {face} < {self.config.cameras[camera].face_recognition.min_area}"
+                    )
+                    # try hi-res fallback
+                    hires_face = self._try_hires_face_detection(
+                        camera, obj_data, obj_data.get("frame_time", start)
+                    )
+                    if hires_face is None:
+                        return
+                    face_frame = hires_face
+                else:
+                    try:
+                        face_frame = cv2.cvtColor(face_frame, cv2.COLOR_RGB2BGR)
+                    except Exception as e:
+                        logger.debug(
+                            f"Failed to convert face frame color for {id}: {e}"
+                        )
+                        return
         else:
             # don't run for object without attributes
             if not obj_data.get("current_attributes"):
@@ -283,14 +373,20 @@ class FaceRealTimeProcessor(RealTimeProcessorApi):
                 < self.config.cameras[camera].face_recognition.min_area
             ):
                 logger.debug(f"Invalid face box {face}")
-                return
+                # try hi-res fallback for attribute-based path
+                hires_face = self._try_hires_face_detection(
+                    camera, obj_data, obj_data.get("frame_time", start)
+                )
+                if hires_face is None:
+                    return
+                face_frame = hires_face
+            else:
+                face_frame = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_I420)
 
-            face_frame = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_I420)
-
-            face_frame = face_frame[
-                max(0, face_box[1]) : min(frame.shape[0], face_box[3]),
-                max(0, face_box[0]) : min(frame.shape[1], face_box[2]),
-            ]
+                face_frame = face_frame[
+                    max(0, face_box[1]) : min(frame.shape[0], face_box[3]),
+                    max(0, face_box[0]) : min(frame.shape[1], face_box[2]),
+                ]
 
         res = self.recognizer.classify(face_frame)
 

frigate/util/recording_frame.py

@@ -0,0 +1,138 @@
+"""Utility for extracting high-resolution frames from recording segments."""
+
+import logging
+import math
+from typing import Optional
+
+import cv2
+import numpy as np
+from peewee import DoesNotExist
+
+from frigate.models import Recordings
+from frigate.util.image import get_image_from_recording
+
+logger = logging.getLogger(__name__)
+
+
+def get_recording_frame(
+    ffmpeg_config,
+    camera_name: str,
+    frame_time: float,
+) -> Optional[np.ndarray]:
+    """Extract a single full-resolution frame from recording segments.
+
+    Checks the Recordings database for a segment covering frame_time,
+    then uses ffmpeg to decode one frame. CPU only, ~50-100ms.
+
+    Args:
+        ffmpeg_config: FfmpegConfig with ffmpeg_path property.
+        camera_name: Name of the camera.
+        frame_time: Unix timestamp of the desired frame.
+
+    Returns:
+        BGR numpy array at full recording resolution, or None
+        if the segment is not available.
+    """
+    recording = None
+
+    try:
+        recording = (
+            Recordings.select(Recordings.path, Recordings.start_time)
+            .where(
+                (frame_time >= Recordings.start_time)
+                & (frame_time <= Recordings.end_time)
+            )
+            .where(Recordings.camera == camera_name)
+            .order_by(Recordings.start_time.desc())
+            .limit(1)
+            .get()
+        )
+    except DoesNotExist:
+        rounded = math.ceil(frame_time)
+        try:
+            recording = (
+                Recordings.select(Recordings.path, Recordings.start_time)
+                .where(
+                    (rounded >= Recordings.start_time)
+                    & (rounded <= Recordings.end_time)
+                )
+                .where(Recordings.camera == camera_name)
+                .order_by(Recordings.start_time.desc())
+                .limit(1)
+                .get()
+            )
+        except DoesNotExist:
+            pass
+
+    if recording is None:
+        logger.debug(f"No recording segment found for {camera_name} at {frame_time}")
+        return None
+
+    time_in_segment = frame_time - recording.start_time
+    image_data = get_image_from_recording(
+        ffmpeg_config, recording.path, time_in_segment, "png"
+    )
+
+    if not image_data:
+        logger.debug(
+            f"Failed to extract frame from recording for {camera_name} at {frame_time}"
+        )
+        return None
+
+    img_array = np.frombuffer(image_data, dtype=np.uint8)
+    frame = cv2.imdecode(img_array, cv2.IMREAD_COLOR)
+
+    if frame is None:
+        logger.debug(f"Failed to decode recording frame for {camera_name}")
+        return None
+
+    return frame
+
+
+def scale_bounding_box(
+    box: tuple[int, int, int, int],
+    from_res: tuple[int, int],
+    to_res: tuple[int, int],
+    padding: float = 0.15,
+) -> tuple[int, int, int, int]:
+    """Scale a bounding box from one resolution to another with padding.
+
+    Args:
+        box: (left, top, right, bottom) in source resolution.
+        from_res: (width, height) of source (detect stream).
+        to_res: (width, height) of target (recording stream).
+        padding: Fractional padding to add around the box (default 15%).
+
+    Returns:
+        (left, top, right, bottom) in target resolution, clipped to bounds.
+    """
+    from_w, from_h = from_res
+    to_w, to_h = to_res
+
+    scale_x = to_w / from_w
+    scale_y = to_h / from_h
+
+    left, top, right, bottom = box
+    # Scale to target resolution
+    left = left * scale_x
+    top = top * scale_y
+    right = right * scale_x
+    bottom = bottom * scale_y
+
+    # Apply padding
+    w = right - left
+    h = bottom - top
+    pad_x = w * padding
+    pad_y = h * padding
+    left -= pad_x
+    top -= pad_y
+    right += pad_x
+    bottom += pad_y
+
+    # Clip to frame bounds
+    left = max(0, int(left))
+    top = max(0, int(top))
+    right = min(to_w, int(right))
+    bottom = min(to_h, int(bottom))
+
+    return (left, top, right, bottom)