frigate/.cursor/architecture.md

# Frigate Architecture Deep Dive

## System Overview

Frigate is a sophisticated Network Video Recorder (NVR) with real-time AI object detection, designed specifically for Home Assistant integration. The system uses a multi-process architecture to achieve real-time performance while maintaining stability and resource efficiency.

## High-Level Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                          Frigate System                        │
├─────────────────┬─────────────────┬─────────────────────────────┤
│   Web Frontend  │   Python Core   │      Infrastructure        │
│   (React/TS)    │   (FastAPI)     │      (Docker/Nginx)        │
└─────────────────┴─────────────────┴─────────────────────────────┘
```

## Core Components

### 1. Python Backend (`frigate/`)

#### Main Application (`app.py`)

- **FrigateApp Class**: Central orchestrator managing all system processes
- **Process Management**: Spawns and manages detection, recording, cleanup processes
- **Resource Management**: Handles shared memory, queues, and inter-process communication
- **Service Lifecycle**: Manages startup, shutdown, and error recovery

#### API Layer (`api/`)

- **FastAPI Application**: Modern async Python web framework
- **Authentication**: JWT-based auth with user management
- **RESTful Endpoints**: Full CRUD operations for events, recordings, configuration
- **WebSocket Support**: Real-time communication for live feeds and notifications
- **OpenAPI Documentation**: Auto-generated API documentation

#### Configuration System (`config/`)

- **Pydantic Models**: Type-safe configuration with validation
- **YAML Support**: Human-readable configuration files
- **Runtime Validation**: Ensures configuration integrity
- **Hot Reloading**: Dynamic configuration updates without restart

#### Object Detection (`object_detection.py`, `detectors/`)

- **TensorFlow Integration**: Deep learning model execution
- **Hardware Acceleration**: Support for Coral TPU, NVIDIA GPU, Intel GPU
- **Multi-Model Support**: Different detectors for different use cases
- **Process Isolation**: Detection runs in separate processes for stability

#### Video Processing (`video/`, `motion/`)

- **Camera Management**: RTSP/HTTP stream handling
- **Motion Detection**: Efficient OpenCV-based motion detection
- **Frame Processing**: Shared memory frame management
- **Recording**: Segment-based video recording with retention policies

#### Event System (`events/`)

- **Event Processing**: Real-time event detection and classification
- **Timeline Management**: Chronological event organization
- **Cleanup Processes**: Automated old data removal
- **Embeddings**: Vector similarity search for event matching

#### Communication (`comms/`)

- **MQTT Integration**: Home Assistant and IoT device communication
- **WebSocket Server**: Real-time web interface updates
- **ZMQ Messaging**: Inter-process communication
- **Dispatcher Pattern**: Event distribution to multiple consumers

### 2. Web Frontend (`web/`)

#### React Application

- **Modern React 18+**: Functional components with hooks
- **TypeScript**: Full type safety throughout the frontend
- **Component Architecture**: Modular, reusable components
- **State Management**: Context providers and custom hooks

#### UI Framework

- **TailwindCSS**: Utility-first CSS framework
- **Shadcn/ui**: High-quality component library
- **Responsive Design**: Mobile-first responsive layout
- **Accessibility**: ARIA labels and keyboard navigation

#### Real-time Features

- **WebSocket Client**: Live feed streaming
- **Server-Sent Events**: Real-time notifications
- **Progressive Loading**: Efficient data fetching
- **Offline Support**: Graceful degradation

#### Key Pages

- **Live View**: Real-time camera feeds with detection overlays
- **Events**: Searchable event timeline with filtering
- **Recordings**: Video playback and export functionality
- **Settings**: Configuration management interface

### 3. Infrastructure

#### Docker Architecture

- **Multi-stage Builds**: Optimized container images
- **Multi-architecture**: Support for AMD64, ARM64, ARM32
- **Hardware-specific**: Specialized builds for Coral, NVIDIA, etc.
- **Development**: DevContainer support for consistent development

#### Process Management (S6 Overlay)

- **Service Supervision**: Automatic process restart on failure
- **Graceful Shutdown**: Proper cleanup on container stop
- **Logging**: Centralized log management
- **Health Monitoring**: Container health checks

#### Streaming (Go2RTC)

- **RTSP Server**: Re-streaming to reduce camera connections
- **WebRTC Support**: Low-latency browser streaming
- **Protocol Conversion**: Multiple streaming protocol support
- **Transcoding**: Hardware-accelerated video processing

## Data Flow

### Video Processing Pipeline

```
Camera → Go2RTC → Motion Detection → Object Detection → Event Creation → Storage
                                ↓
                           Live Stream → WebSocket → Frontend
```

### Event Processing Flow

```
Detection → Event Processor → Timeline → Database
                           ↓
                        MQTT/WebSocket → Home Assistant/Frontend
```

### Configuration Flow

```
YAML Config → Pydantic Validation → Runtime Config → Process Distribution
```

## Database Schema

### Core Tables

- **Events**: Object detection events with metadata
- **Recordings**: Video recording segments
- **ReviewSegment**: Motion-based review segments
- **Timeline**: Chronological event timeline
- **Previews**: Thumbnail previews for recordings

### Vector Database

- **Embeddings**: Semantic search capabilities
- **Similarity**: Find similar events/objects
- **Reindexing**: Background embedding updates

## Inter-Process Communication

### Message Queues

- **Detection Queue**: Frame processing requests
- **Event Queue**: Event notifications
- **Config Updates**: Runtime configuration changes

### Shared Memory

- **Frame Buffers**: Raw video frame data
- **Detection Results**: Object detection outputs
- **Statistics**: Real-time performance metrics

### ZeroMQ Patterns

- **Publisher/Subscriber**: Event distribution
- **Request/Reply**: Configuration queries
- **Push/Pull**: Work distribution

## Performance Optimizations

### CPU Optimization

- **Multiprocessing**: Separate processes for I/O vs CPU tasks
- **Process Affinity**: CPU core assignment for critical processes
- **Queue Management**: Efficient message passing
- **Memory Pools**: Reduced allocation overhead

### Hardware Acceleration

- **Coral TPU**: Edge TPU inference acceleration
- **NVIDIA GPU**: CUDA-based processing
- **Intel GPU**: Intel GPU acceleration
- **Hardware Decoding**: GPU-accelerated video decoding

### Memory Management

- **Shared Memory**: Zero-copy frame passing
- **Object Pools**: Reused detection objects
- **Garbage Collection**: Efficient memory cleanup
- **Memory Mapping**: Direct file access

## Security Architecture

### Authentication

- **JWT Tokens**: Stateless authentication
- **User Management**: Role-based access control
- **Session Management**: Secure session handling
- **API Security**: Endpoint protection

### Data Protection

- **Input Validation**: Pydantic model validation
- **SQL Injection Prevention**: ORM-based queries
- **File Path Sanitization**: Secure file access
- **CORS Configuration**: Cross-origin protection

## Monitoring and Observability

### Metrics Collection

- **System Stats**: CPU, memory, disk usage
- **Camera Stats**: FPS, detection rates
- **Process Health**: Process status monitoring
- **Performance Metrics**: Latency, throughput

### Logging

- **Structured Logging**: JSON-formatted logs
- **Log Levels**: Configurable verbosity
- **Log Rotation**: Automatic log management
- **Error Tracking**: Exception monitoring

## Scalability Considerations

### Horizontal Scaling

- **Process Distribution**: Multiple detection processes
- **Camera Distribution**: Cameras across processes
- **Load Balancing**: Request distribution

### Vertical Scaling

- **Multi-threading**: Thread-safe operations
- **Resource Allocation**: Dynamic resource assignment
- **Queue Sizing**: Configurable queue depths

## Integration Points

### Home Assistant

- **MQTT Discovery**: Automatic entity creation
- **State Updates**: Real-time state synchronization
- **Service Calls**: Action integration
- **Notification System**: Event notifications

### External APIs

- **Frigate+**: Cloud-based model training
- **GenAI Integration**: AI-powered descriptions
- **Webhooks**: External event notifications
- **Third-party Integrations**: Plugin architecture

This architecture prioritizes real-time performance, reliability, and ease of integration while maintaining modularity and extensibility for future enhancements.