I have a very repeatable reproduction of an issue where most of my
cameras show a "No frames have been received, check error logs" image in
the UI, but restreaming in HomeAssistant is working flawlessly. The only
errors in the logs I saw were some like this:
`OSError: [Errno 121] Remote I/O error`.
Doing a bit more debugging, it looked like Frigate was failing to create
the thumbnail directory for a camera because it already existed. This
error was a clue as to the class of error. I was surprised to learn that
`os.path.exists` [silently suppresses errors from
os.stat and returns False](https://github.com/python/cpython/blob/main/Lib/genericpath.py#L22).
This makes for a plausible series of events: a transient stat call
fails, so Frigate takes the creation path, which gets upset that the
directory already exists.
I found a few other possible cases to fix but did not make an exhaustive
search. It seems that this `exist_ok` flag is used elsewhere within
Frigate so I thought it would be a good solution.
AI disclosure: I used AI to diagnose my issue and asked it to translate
its init-time patches to the container source into this repo. I verified
that its patches solved the problem I was facing. Its theory fits the
facts - I am using a distributed file system and I saw the error in my
logs. I checked the upstream Python code to verify the error suppression
behavior, and read the corresponding Frigate code. I did not use AI to
author this commit message/PR description; all diction and typos here are my own.
* add log when probing detect stream on startup
when users don't explicitly set detect.width and detect.height, we probe for them. sometimes the probe hangs (camera doesn't support UDP, like some Reolinks), so this log message will make that clearer
* add faq about probing detect stream
* fix stuck activity ring when tracked object transitions to stationary
* drop cache segments past retain cutoff regardless of retention mode
* add maintainer test
* fix: operator precedence bug in detection type check
The condition:
topic == DetectionTypeEnum.api.value or DetectionTypeEnum.lpr.value
evaluates as:
(topic == DetectionTypeEnum.api.value) or (DetectionTypeEnum.lpr.value)
Since DetectionTypeEnum.lpr.value is a non-empty string (truthy), the
second operand is always True regardless of topic. The intended check
is whether topic matches either enum value:
topic == DetectionTypeEnum.api.value or topic == DetectionTypeEnum.lpr.value
* fix: apply same or operator fix to review/maintainer.py
Same issue as record/maintainer.py — the condition was always true
because the bare enum value is truthy.
* style: ruff format record/maintainer.py
* debug replay implementation
* fix masks after dev rebase
* fix squash merge issues
* fix
* fix
* fix
* no need to write debug replay camera to config
* camera and filter button and dropdown
* add filters
* add ability to edit motion and object config for debug replay
* add debug draw overlay to debug replay
* add guard to prevent crash when camera is no longer in camera_states
* fix overflow due to radix absolutely positioned elements
* increase number of messages
* ensure deep_merge replaces existing list values when override is true
* add back button
* add debug replay to explore and review menus
* clean up
* clean up
* update instructions to prevent exposing exception info
* fix typing
* refactor output logic
* refactor with helper function
* move init to function for consistency
* fix(recording): handle unexpected filenames in cache maintainer to prevent crash
* test(recording): add test for maintainer cache file parsing
* Prevent log spam from unexpected cache files
Addresses PR review feedback: Add deduplication to prevent warning
messages from being logged repeatedly for the same unexpected file
in the cache directory. Each unexpected filename is only logged once
per RecordingMaintainer instance lifecycle.
Also adds test to verify warning is only emitted once per filename.
* Fix code formatting for test_maintainer.py
* fixes + ruff
* refactor get_video_properties and use json output from ffprobe
* add zmq topic
* publish valid segment data in recording maintainer
* check for valid video data
- restart separate record ffmpeg process if no video data has been received in 120s
- refactor datetime import
* listen to correct topic in embeddings maintainer
* refactor to move get_latest_segment_datetime logic to recordings maintainer
* debug logging
* cleanup
* Add base class for global config updates
* Add or remove camera states
* Move camera process management to separate thread
* Move camera management fully to separate class
* Cleanup
* Stop camera processes when stop command is sent
* Start processes dynamically when needed
* Adjust
* Leave extra room in tracked object queue for two cameras
* Dynamically set extra config pieces
* Add some TODOs
* Fix type check
* Simplify config updates
* Improve typing
* Correctly handle indexed entries
* Cleanup
* Create out SHM
* Use ZMQ for signaling object detectoin is completed
* Get camera correctly created
* Cleanup for updating the cameras config
* Cleanup
* Don't enable audio if no cameras have audio transcription
* Use exact string so similar camera names don't interfere
* Add ability to update config via json body to config/set endpoint
Additionally, update the config in a single rather than multiple calls for each updated key
* fix autotracking calibration to support new config updater function
---------
Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com>
* Fix the `Any` typing hint treewide
There has been confusion between the Any type[1] and the any function[2]
in typing hints.
[1] https://docs.python.org/3/library/typing.html#typing.Any
[2] https://docs.python.org/3/library/functions.html#any
* Fix typing for various frame_shape members
Frame shapes are most likely defined by height and width, so a single int
cannot express that.
* Wrap gpu stats functions in Optional[]
These can return `None`, so they need to be `Type | None`, which is what
`Optional` expresses very nicely.
* Fix return type in get_latest_segment_datetime
Returns a datetime object, not an integer.
* Make the return type of FrameManager.write optional
This is necessary since the SharedMemoryFrameManager.write function can
return None.
* Fix total_seconds() return type in get_tz_modifiers
The function returns a float, not an int.
https://docs.python.org/3/library/datetime.html#datetime.timedelta.total_seconds
* Account for floating point results in to_relative_box
Because the function uses division the return types may either be int or
float.
* Resolve ruff deprecation warning
The config has been split into formatter and linter, and the global
options are deprecated.
On startup, the object processing pipeline would be delayed for 10ms every iteration through the loop while the end event subscriber blocked. For users with large numbers of cameras and a fair amount of detected objects, this would cause the detected objects queue length to rise and frames to be eventually dropped, leading to processing delays, incorrect frame times applied to thumbnail_data, and other seemingly unexplainable things.
* config options
* processing in maintainer
* detect and process dedicated lpr plates
* create camera type, add manual event and save snapshot
* use const
* ensure lpr events are always detections, typing fixes
* docs
* docs tweaks
* add preprocessing and penalization for low confidence chars
* recordings data pub/sub
* function to process recording stream frames
* model runner
* lpr model runner
* refactor to mixin class and use model runner
* separate out realtime and post processors
* move model and mixin folders
* basic postprocessor
* clean up
* docs
* postprocessing logic
* clean up
* return none if recordings are disabled
* run postprocessor handle_requests too
* tweak expansion
* add put endpoint
* postprocessor tweaks with endpoint
* Install multiple ffmpeg versions and add config to make it configurable
* Update docs
* Run ffprobe too
* Cleanup
* Apply config to go2rtc as well
* Fix ffmpeg bin
* Docs
* Restore path
* Cleanup env var
* Fix ffmpeg path for encoding
* Fix export
* Formatting
* Refactor recordings config to be based off of review items
* Update object processing logic for when an event is created
* Migrate to deciding recording retention based on review items
* Refactor recording expiration to be based off of review items
* Remove remainder of recording events access
* Handle migration automatically
* Update version and cleanup
* Update docs
* Clarify docs
* Cleanup
* Target camera config
* Safely access all fields
* Initial re-implementation of semantic search
* put docker-compose back and make reindex match docs
* remove debug code and fix import
* fix docs
* manually build pysqlite3 as binaries are only available for x86-64
* update comment in build_pysqlite3.sh
* only embed objects
* better error handling when genai fails
* ask ollama to pull requested model at startup
* update ollama docs
* address some PR review comments
* fix lint
* use IPC to write description, update docs for reindex
* remove gemini-pro-vision from docs as it will be unavailable soon
* fix OpenAI doc available models
* fix api error in gemini and metadata for embeddings
* Only update frame time if it is older
* Support manual detections as review items
* Don't handle api detections in recordings
* Store recordings for manual events
* Add config pub / sub pattern
* remove recording from feature metrics
* remove audio and feature metrics
* Check for updates from all cameras
* remove birdseye from camera metrics
* remove motion and detection camera metrics
* Ensure that all processes are stopped
* Stop communicators
* Detections
* Cleanup video output queue
* Use select for time sensitive polls
* Use ipc instead of tcp
* Use zmq for inter process communication
* Use localhost for reply and request
* Use pyobj instead of json and Need to use separate requestors for each audio listener
* Cleanup port defining
