The record watchdog treats a stale maintainer heartbeat as a dead
recorder. But the heartbeat is published by the recording maintainer,
so whenever the maintainer lags (e.g. "Unable to keep up with recording
segments in cache", #9661) every camera looks stale at once and all
record processes restart together - while recording was actually
healthy. The restart churn then produces more, shorter segments,
making the maintainer fall further behind.
Before restarting on staleness, check the camera's newest cache
segment on disk: if a segment is fresher than the staleness threshold,
the recorder is demonstrably writing - log a warning, adopt the disk
mtime as the heartbeat, and skip the restart. The invalid-segment path
is untouched.
Validated on a 26-camera production deployment (0.17.1 backport of
this change): synchronized mass restarts went from 52/hour to zero,
with heartbeat-stale events still occurring ~2/hour but now correctly
identified as maintainer lag instead of recording failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
