frigate/.github/copilot-instructions.md

Frigate AI Agent Instructions

Project Overview

Frigate is a local NVR with realtime AI object detection for IP cameras. It's a multiprocess Python service that combines:

• Video capture and processing from IP cameras
• Realtime object detection (TensorFlow, ONNX models on CPU/GPU/AI accelerators)
• Recording with retention policies
• Event tracking and review UI
• REST API + MQTT integration

Key architectural philosophy: Minimize resource use by only running expensive detection where/when motion is detected. Heavy use of multiprocessing for FPS and scalability.

Architecture Patterns

Multiprocess Communication

Frigate uses three primary IPC mechanisms (NOT shared state):

1. ZMQ pub/sub (for one-way broadcasts):

   • Config changes: frigate/comms/config_updater.py (PUB via ipc:///tmp/cache/config)
   • Object detection signals: frigate/comms/object_detector_signaler.py
   • Detection/event updates: detector signaler, event publishers

2. ZMQ req/rep (request-reply):

   • frigate/comms/inter_process.py - processes request data and get responses via ipc:///tmp/cache/comms
   • Used by InterProcessRequestor for sync queries

3. Multiprocessing Queues (frame data):

   • detection_queue: Camera frames → object detectors
   • tracked_objects_queue: Detected objects → event processor
   • timeline_queue: Events → timeline storage

Key pattern: Each service publishes to ZMQ topics, others subscribe. Config changes fan out via ZMQ pub/sub to all processes without central coordination.

Core Services (in frigate/app.py FrigateApp)

• CameraMaintainer (thread): Spawns camera capture/processing subprocess per camera
• ObjectDetectProcess (subprocess): Runs ML inference on queued frames
• TrackedObjectProcessor (thread): Receives detections, correlates into tracked objects, publishes events
• EventProcessor (thread): Manages event lifecycle, DB updates
• RecordProcess (subprocess): Manages video recording/retention
• OutputProcess (subprocess): Encodes/streams video
• ReviewProcess (subprocess): Processes review segments
• EmbeddingProcess (subprocess): Runs embeddings for semantic search/face/LPR

Logging pattern: Central log.py uses QueueListener to collect logs from all processes into one queue to avoid multiprocess logging chaos.

Config System

Configuration is Pydantic BaseModel hierarchy:

• Parsing: YAML → Pydantic models with validators in frigate/config/config.py
• Types: frigate/types.py has shared enums (EventType, ObjectType, etc.)
• Validation pattern: Use @field_validator with mode='before' to transform/validate before assignment
• Runtime values: RuntimeMotionConfig applies frame shape transforms to masks
• Key files:
  • config.py - main FrigateConfig entry point
  • camera/ - per-camera sub-configs (detect, record, snapshots, etc.)
  • classification.py - face/LPR/audio/semantic search configs

When adding config: Create Pydantic model → add to parent config → update migrations if DB schema changes.

Data Model & Persistence

Database: SQLite with custom SqliteVecQueueDatabase (vector support for embeddings)

• Models in frigate/models.py: Event, Timeline, Recordings, User, etc. (Peewee ORM)
• Key tables:
  • events - detected objects (car, person, etc.) with retention policies
  • timeline - events feed (entered_zone, audio, etc.)
  • recordings - video segments with metadata
  • review_segments - flagged clips for review

Event lifecycle:

1. TrackedObject detected → Event created with false_positive=False
2. EventProcessor updates Event (score, zones, clips, snapshots)
3. On object lost, Event gets end_time and is finalized

Migrations: Use Peewee migrations in migrations/ - run via peewee_migrate.Router.

Key Workflows

Adding a New Detector Type

1. Create detector class in frigate/detectors/plugins/ inheriting DetectionApi
2. Add config class in frigate/detectors/detector_config.py
3. Register in detector factory in frigate/detectors/__init__.py
4. Update DEFAULT_DETECTORS constant if it's the default

Object Detection Pipeline

Camera subprocess → capture frames → motion detect →
  queue frame to detection_queue →
  ObjectDetectProcess (inference) →
  TrackedObjectProcessor (correlate detections) →
  Event + tracking + DB updates

Recording Flow

• 24/7 recording: Segments written every frame duration
• Retention: Deleted if no events + retention time elapsed
• Cleanup: RecordingCleanup task deletes old segments based on retention config

Frontend Translation Pattern

• Rule: NEVER hardcode strings in .ts/.tsx files
• Pattern: Store strings in web/src/locales/en.json → import locale function → use in code
• See: .cursor/rules/frontend-always-use-translation-files.mdc

Common Code Patterns

Inter-process Config Updates

# In detector/processor:
self.config_subscriber = ConfigSubscriber(config, [ConfigUpdateEnum.cameras])

# In main app (FrigateApp):
publisher = ConfigPublisher()
publisher.publish("cameras", new_config)  # All subscribers notified

Event Publishing

from frigate.comms.events_updater import EventUpdatePublisher

publisher = EventUpdatePublisher()
publisher.publish({"camera": "cam1", "label": "person", ...})

Shared Memory Frames

from frigate.util.image import SharedMemoryFrameManager, UntrackedSharedMemory

frame_manager = SharedMemoryFrameManager()
shm = frame_manager.get(frame_id)  # returns np.ndarray view

Testing & Debugging

Test structure: frigate/test/test_*.py using Python unittest

• Run tests: make run_tests (builds Docker, runs in container)
• Key tests: config parsing, detector inference, frame processing

Build targets (Makefile):

• make local - builds Docker image locally with version
• make debug - builds with debug logging enabled
• make run - runs container with config volume mounted

Debugging multiprocess issues:

• Check log_queue output in frigate/log.py
• Enable DEBUG logging for specific modules in config
• Use faulthandler.enable() (already enabled in processes) for segfaults

Important Conventions

• Imports: Run Ruff with isort (extend-select = ["I"]) - enforces import sorting
• GPU/Acceleration: Hardware detection in frigate/util/services.py (NVIDIA, Intel VAAPI, AMD, etc.)
• Model paths: Stored in /config/model_cache/ (symlinked or volume mounted)
• Recording paths: /media/frigate/recordings/ (clips in clips/, exports in exports/)
• PID locking: Use setproctitle() to name processes for debugging via ps

Files to Know

File	Purpose
frigate/app.py	Main app startup, service orchestration
frigate/camera/	Camera subprocess, frame capture, motion
frigate/track/object_processing.py	Detection correlation, event publishing
frigate/events/maintainer.py	Event lifecycle management
frigate/config/config.py	Config parsing & validation
frigate/comms/	IPC (ZMQ pub/sub, req/rep)
frigate/api/fastapi_app.py	REST API setup
frigate/models.py	Database ORM models
frigate/const.py	Global constants (paths, defaults)

Gotchas & Common Mistakes

1. Pickle compatibility: Objects sent over multiprocess queues must be pickleable. Avoid lambdas, file handles.
2. Config subscriptions: Always check mode='before' in validators—Pydantic can be confusing.
3. Event state confusion: Events have transient state in TrackedObjectProcessor AND persistent state in DB—don't mix them.
4. Motion masks: Frame shape must be applied before creating RuntimeMotionConfig—validate in tests.
5. ZMQ timing: Topics must be subscribed BEFORE publisher sends; use small sleep if race condition suspected.
6. Frontend strings: Forgetting to use locale files breaks translations and fails linting.

External Integration

• MQTT: Via MqttClient in frigate/comms/mqtt.py - publishes detections, accepts commands
• Home Assistant: Native integration via custom component (separate repo)
• Frigate+: Paid cloud sync service - frigate/plus.py handles API calls
• Webhooks: Event-triggered POST requests configured per-camera

---

Last updated: Branch review-stream-tweaks | For architecture deep-dives, start with frigate/app.py::FrigateApp.__init__() to see all service wiring.