Use Fork-Server As Spawn Method (#18682)

* Set runtime

* Use count correctly

* Don't assume camera sizes

* Use separate zmq proxy for object detection

* Correct order

* Use forkserver

* Only store PID instead of entire process reference

* Cleanup

* Catch correct errors

* Fix typing

* Remove before_run from process util

The before_run never actually ran because:

You're right to suspect an issue with before_run not being called and a potential deadlock. The way you've implemented the run_wrapper using __getattribute__ for the run method of BaseProcess is a common pitfall in Python's multiprocessing, especially when combined with how multiprocessing.Process works internally.

Here's a breakdown of why before_run isn't being called and why you might be experiencing a deadlock:

The Problem: __getattribute__ and Process Serialization
When you create a multiprocessing.Process object and call start(), the multiprocessing module needs to serialize the process object (or at least enough of it to re-create the process in the new interpreter). It then pickles this serialized object and sends it to the newly spawned process.

The issue with your __getattribute__ implementation for run is that:

run is retrieved during serialization: When multiprocessing tries to pickle your Process object to send to the new process, it will likely access the run attribute. This triggers your __getattribute__ wrapper, which then tries to bind run_wrapper to self.
run_wrapper is bound to the parent process's self: The run_wrapper closure, when created in the parent process, captures the self (the Process instance) from the parent's memory space.
Deserialization creates a new object: In the child process, a new Process object is created by deserializing the pickled data. However, the run_wrapper method that was pickled still holds a reference to the self from the parent process. This is a subtle but critical distinction.
The child's run is not your wrapped run: When the child process starts, it internally calls its own run method. Because of the serialization and deserialization process, the run method that's ultimately executed in the child process is the original multiprocessing.Process.run or the Process.run if you had directly overridden it. Your __getattribute__ magic, which wraps run, isn't correctly applied to the Process object within the child's context.

* Cleanup

* Logging bugfix (#18465)

* use mp Manager to handle logging queues

A Python bug (https://github.com/python/cpython/issues/91555) was preventing logs from the embeddings maintainer process from printing. The bug is fixed in Python 3.14, but a viable workaround is to use the multiprocessing Manager, which better manages mp queues and causes the logging to work correctly.

* consolidate

* fix typing

* Fix typing

* Use global log queue

* Move to using process for logging

* Convert camera tracking to process

* Add more processes

* Finalize process

* Cleanup

* Cleanup typing

* Formatting

* Remove daemon

---------

Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com>

frigate/camera/__init__.py

@@ -1,7 +1,7 @@
 import multiprocessing as mp
+from multiprocessing.managers import SyncManager
 from multiprocessing.sharedctypes import Synchronized
 from multiprocessing.synchronize import Event
-from typing import Optional
 
 
 class CameraMetrics:
@@ -16,25 +16,25 @@ class CameraMetrics:
 
     frame_queue: mp.Queue
 
-    process: Optional[mp.Process]
-    capture_process: Optional[mp.Process]
+    process_pid: Synchronized
+    capture_process_pid: Synchronized
     ffmpeg_pid: Synchronized
 
-    def __init__(self):
-        self.camera_fps = mp.Value("d", 0)
-        self.detection_fps = mp.Value("d", 0)
-        self.detection_frame = mp.Value("d", 0)
-        self.process_fps = mp.Value("d", 0)
-        self.skipped_fps = mp.Value("d", 0)
-        self.read_start = mp.Value("d", 0)
-        self.audio_rms = mp.Value("d", 0)
-        self.audio_dBFS = mp.Value("d", 0)
+    def __init__(self, manager: SyncManager):
+        self.camera_fps = manager.Value("d", 0)
+        self.detection_fps = manager.Value("d", 0)
+        self.detection_frame = manager.Value("d", 0)
+        self.process_fps = manager.Value("d", 0)
+        self.skipped_fps = manager.Value("d", 0)
+        self.read_start = manager.Value("d", 0)
+        self.audio_rms = manager.Value("d", 0)
+        self.audio_dBFS = manager.Value("d", 0)
 
-        self.frame_queue = mp.Queue(maxsize=2)
+        self.frame_queue = manager.Queue(maxsize=2)
 
-        self.process = None
-        self.capture_process = None
-        self.ffmpeg_pid = mp.Value("i", 0)
+        self.process_pid = manager.Value("i", 0)
+        self.capture_process_pid = manager.Value("i", 0)
+        self.ffmpeg_pid = manager.Value("i", 0)
 
 
 class PTZMetrics:

frigate/camera/maintainer.py

@@ -1,10 +1,12 @@
 """Create and maintain camera processes / management."""
 
 import logging
+import multiprocessing as mp
 import os
 import shutil
 import threading
 from multiprocessing import Queue
+from multiprocessing.managers import DictProxy
 from multiprocessing.synchronize import Event as MpEvent
 
 from frigate.camera import CameraMetrics, PTZMetrics
@@ -16,11 +18,10 @@ from frigate.config.camera.updater import (
 )
 from frigate.const import SHM_FRAMES_VAR
 from frigate.models import Regions
-from frigate.util import Process as FrigateProcess
 from frigate.util.builtin import empty_and_close_queue
 from frigate.util.image import SharedMemoryFrameManager, UntrackedSharedMemory
 from frigate.util.object import get_camera_regions_grid
-from frigate.video import capture_camera, track_camera
+from frigate.video import CameraCapture, CameraTracker
 
 logger = logging.getLogger(__name__)
 
@@ -31,7 +32,7 @@ class CameraMaintainer(threading.Thread):
         config: FrigateConfig,
         detection_queue: Queue,
         detected_frames_queue: Queue,
-        camera_metrics: dict[str, CameraMetrics],
+        camera_metrics: DictProxy,
         ptz_metrics: dict[str, PTZMetrics],
         stop_event: MpEvent,
     ):
@@ -53,6 +54,8 @@ class CameraMaintainer(threading.Thread):
             ],
         )
         self.shm_count = self.__calculate_shm_frame_count()
+        self.camera_processes: dict[str, mp.Process] = {}
+        self.capture_processes: dict[str, mp.Process] = {}
 
     def __init_historical_regions(self) -> None:
         # delete region grids for removed or renamed cameras
@@ -94,7 +97,7 @@ class CameraMaintainer(threading.Thread):
 
         # leave room for 2 cameras that are added dynamically, if a user wants to add more cameras they may need to increase the SHM size and restart after adding them.
         cam_total_frame_size += 2 * round(
-            (camera.detect.width * camera.detect.height * 1.5 + 270480) / 1048576,
+            (1280 * 720 * 1.5 + 270480) / 1048576,
             1,
         )
 
@@ -151,24 +154,19 @@ class CameraMaintainer(threading.Thread):
             except FileExistsError:
                 pass
 
-        camera_process = FrigateProcess(
-            target=track_camera,
-            name=f"camera_processor:{name}",
-            args=(
-                config.name,
-                config,
-                self.config.model,
-                self.config.model.merged_labelmap,
-                self.detection_queue,
-                self.detected_frames_queue,
-                self.camera_metrics[name],
-                self.ptz_metrics[name],
-                self.region_grids[name],
-            ),
-            daemon=True,
+        camera_process = CameraTracker(
+            config,
+            self.config.model,
+            self.config.model.merged_labelmap,
+            self.detection_queue,
+            self.detected_frames_queue,
+            self.camera_metrics[name],
+            self.ptz_metrics[name],
+            self.region_grids[name],
         )
-        self.camera_metrics[config.name].process = camera_process
+        self.camera_processes[config.name] = camera_process
         camera_process.start()
+        self.camera_metrics[config.name].process_pid.value = camera_process.pid
         logger.info(f"Camera processor started for {config.name}: {camera_process.pid}")
 
     def __start_camera_capture(
@@ -179,36 +177,33 @@ class CameraMaintainer(threading.Thread):
             return
 
         # pre-create shms
-        for i in range(10 if runtime else self.shm_count):
+        count = 10 if runtime else self.shm_count
+        for i in range(count):
             frame_size = config.frame_shape_yuv[0] * config.frame_shape_yuv[1]
             self.frame_manager.create(f"{config.name}_frame{i}", frame_size)
 
-        capture_process = FrigateProcess(
-            target=capture_camera,
-            name=f"camera_capture:{name}",
-            args=(config, self.shm_count, self.camera_metrics[name]),
-        )
+        capture_process = CameraCapture(config, count, self.camera_metrics[name])
         capture_process.daemon = True
-        self.camera_metrics[name].capture_process = capture_process
+        self.capture_processes[name] = capture_process
         capture_process.start()
+        self.camera_metrics[name].capture_process_pid.value = capture_process.pid
         logger.info(f"Capture process started for {name}: {capture_process.pid}")
 
     def __stop_camera_capture_process(self, camera: str) -> None:
-        capture_process = self.camera_metrics[camera].capture_process
+        capture_process = self.capture_processes[camera]
         if capture_process is not None:
             logger.info(f"Waiting for capture process for {camera} to stop")
             capture_process.terminate()
             capture_process.join()
 
     def __stop_camera_process(self, camera: str) -> None:
-        metrics = self.camera_metrics[camera]
-        camera_process = metrics.process
+        camera_process = self.camera_processes[camera]
         if camera_process is not None:
             logger.info(f"Waiting for process for {camera} to stop")
             camera_process.terminate()
             camera_process.join()
             logger.info(f"Closing frame queue for {camera}")
-            empty_and_close_queue(metrics.frame_queue)
+            empty_and_close_queue(self.camera_metrics[camera].frame_queue)
 
     def run(self):
         self.__init_historical_regions()
@@ -230,18 +225,20 @@ class CameraMaintainer(threading.Thread):
                             runtime=True,
                         )
                         self.__start_camera_capture(
-                            camera, self.update_subscriber.camera_configs[camera]
+                            camera,
+                            self.update_subscriber.camera_configs[camera],
+                            runtime=True,
                         )
                 elif update_type == CameraConfigUpdateEnum.remove.name:
                     self.__stop_camera_capture_process(camera)
                     self.__stop_camera_process(camera)
 
         # ensure the capture processes are done
-        for camera in self.camera_metrics.keys():
+        for camera in self.camera_processes.keys():
             self.__stop_camera_capture_process(camera)
 
         # ensure the camera processors are done
-        for camera in self.camera_metrics.keys():
+        for camera in self.capture_processes.keys():
             self.__stop_camera_process(camera)
 
         self.update_subscriber.stop()