Tweaks and fixes (#24659)

* add last time to active notices

* Stop retrying the review lookup in the explore detail menu when it 404s

* don't let a slow websocket client block publishers

A websocket client that stopped reading its socket (for example a suspended workstation with a Frigate tab open) blocked ws4py's sendall on whatever thread was publishing, including the single inter-process responder thread, so the recording maintainer and stats hung until nginx dropped the client after 5 minutes. Each client now gets its own bounded send queue and writer thread, and a client that overflows the queue is shut down. ws4py writes close and pong frames directly on its shared manager thread, bypassing the queue, so a 10s socket send timeout keeps a stalled client from pinning that thread. The jsmpeg and birdseye outputs share the class, so their unreachable send error handling is removed.

* don't let a lost object enter a zone

When the detector lost an object, the tracker kept it at its last box and every held frame still counted toward the zone's inertia. A car detected in a zone once or twice as it drove out of view could reach an inertia of 8 and be marked as entering, which sent it to GenAI and any other required zones check even though Review never showed the zone. Inertia and loitering still count on frames without a detection, but an object can only newly enter a zone on a frame where it was actually detected.

* add test

* limit retry stop to 404 only

* add docstrings
This commit is contained in:
Josh Hawkins
2026-10-15 06:54:59 -06:00
committed by GitHub
parent 2b0a69c028
commit 64b07889af
9 changed files with 348 additions and 21 deletions
+82 -4
View File
@@ -3,6 +3,9 @@
import errno
import json
import logging
import queue
import socket
import struct
import threading
from collections.abc import Callable
from typing import Any
@@ -76,6 +79,12 @@ _WS_VIEWER_TOPICS = frozenset(
# Camera-scoped command topics a camera-authorized (non-admin) user may send.
_WS_CAMERA_COMMAND_TOPICS = frozenset({"ptz"})
# Max outbound messages waiting on a client's writer thread.
WS_MAX_PENDING_MESSAGES = 256
# Seconds a single send() may wait for a client to free socket buffer space.
WS_SEND_TIMEOUT = 10
def _check_ws_authorization(
topic: str,
@@ -464,6 +473,78 @@ def _materialize_for_ws(
class WebSocket(WebSocket_): # type: ignore[misc]
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, **kwargs)
self._send_queue: queue.Queue[tuple[Any, bool] | None] = queue.Queue(
maxsize=WS_MAX_PENDING_MESSAGES
)
self._writer: threading.Thread | None = None
self._aborted = False
def opened(self) -> None:
"""Set a send timeout on the socket and start this client's writer thread."""
# ws4py writes close and pong frames on the shared manager thread, so a
# stalled client must not block sendall forever. SO_SNDTIMEO leaves recv
# (and idle clients) untouched, unlike settimeout().
self.sock.setsockopt(
socket.SOL_SOCKET, socket.SO_SNDTIMEO, struct.pack("ll", WS_SEND_TIMEOUT, 0)
)
# every client gets its own writer so a client that stops reading only
# blocks itself, never the thread that called publish()
self._writer = threading.Thread(
target=self._drain_send_queue, name="ws_writer", daemon=True
)
self._writer.start()
def send(self, payload: Any, binary: bool = False) -> None:
"""Queue a message for this client's writer thread.
Returns without waiting for delivery. Disconnects the client if its queue
is full.
"""
try:
self._send_queue.put_nowait((payload, binary))
except queue.Full:
self._abort("Websocket client is not keeping up, disconnecting it")
def _drain_send_queue(self) -> None:
while True:
item = self._send_queue.get()
if item is None or self.terminated or self.sock is None:
return
try:
super().send(*item)
except Exception:
self._abort()
return
def _abort(self, reason: str | None = None) -> None:
# publish() keeps hitting a full queue until the manager thread removes
# the connection, so only act (and log) the first time
if self._aborted:
return
self._aborted = True
if reason:
logger.warning(reason)
# shutdown rather than close so the ws4py manager thread sees EOF and
# runs its normal unregister/terminate; this also unblocks a stuck sendall
sock = self.sock
if sock is not None:
try:
sock.shutdown(socket.SHUT_RDWR)
except OSError:
pass
def closed(self, code: int, reason: str | None = None) -> None:
"""Signal the writer thread to exit."""
try:
self._send_queue.put_nowait(None)
except queue.Full:
pass
def unhandled_error(self, error: Any) -> None:
"""
Handles the unfriendly socket closures on the server side
@@ -597,10 +678,7 @@ class WebSocketClient(Communicator):
)
if message is None:
continue
try:
ws.send(message)
except (ConnectionResetError, BrokenPipeError, ValueError):
pass
ws.send(message)
def stop(self) -> None:
if self.websocket_server is not None: