Previously relied on ORT's implicit default (True) for fixed-size models,
only flipping the flag in the variable-length branch. Set it explicitly
in both branches to be robust against ORT default changes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Targeted follow-ups to the embeddings_manager ORT leak fix that affect all
GPU-resident embedding models (Jina text+vision, PaddleOCR det+rec, ArcFace,
YOLOv9 license plate). Detection-side YOLO runners are unaffected since
CudaGraphRunner does not call compute_cuda_mem_limit.
- compute_cuda_mem_limit now checks the cudaMemGetInfo return code instead
of trusting that a non-throwing call populated the buffers. Previously a
non-zero rc left both pointers at 0, producing gpu_mem_limit=0 and
immediate session OOM rather than the documented 4 GB fallback.
- The limit also factors in currently-free VRAM (free * 0.9), not just
total. On a shared GPU where co-resident embedding sessions have already
consumed most of the device, capping at 80% of total still over-allocates.
- The CUDA graph fallback path now logs the underlying exception text so
failures (cudaErrorStreamCaptureUnsupported, missing libnvrtc, etc.)
stop being swallowed by the bare except.
Tests cover all three regression paths plus updated existing tests that
now require cudaMemGetInfo to return cudaSuccess explicitly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three independent ORT/glibc leak vectors identified and fixed:
1. **ORT CPU BFC arena** (`enable_cpu_mem_arena=False` for all sessions)
ORT's default CPU arena pools host-side GPU↔CPU staging buffers indefinitely.
Disabling it across every InferenceSession (detection + embedding) stops
hundreds-of-MB/h RSS growth seen on systems with CUDA EP sessions.
2. **ORT memory-pattern cache** (`enable_mem_pattern=False` for variable-length models)
For embedding models with variable-length inputs (Jina v1/v2, PaddleOCR),
ORT allocates one mmap-backed execution plan per unique sequence length and
never frees them. Disabling the pattern cache stops this unbounded anon-mmap
growth. Fixed-size models (YOLO) keep `enable_mem_pattern=True` to preserve
buffer aliasing and avoid CUDA graph capture failures.
3. **mallopt(M_ARENA_MAX)** called from `EmbeddingProcess.run()`
The forkserver start method exec()s a fresh Python interpreter that does not
inherit Docker env vars, so `MALLOC_ARENA_MAX` set in docker-compose never
reaches the child. Calling `mallopt(-8, os.cpu_count())` from `run()` caps
glibc malloc arenas in the child process.
Additional improvements:
- `compute_cuda_mem_limit()`: dynamically caps the ORT CUDA EP BFC arena for
embedding sessions to min(model_size × 7, 80% VRAM); prevents OOM on
multi-model systems while leaving headroom for detection sessions.
- CUDA graph capture is now wrapped in try/except so models with CPU-only ops
(e.g. attention, NMS) fall back to ONNXModelRunner instead of crashing.
- `ONNXModelRunner.has_variable_length_inputs()`: centralises the
Jina/PaddleOCR detection logic to keep SessionOptions creation consistent.
- 17 regression-guard unit tests in `frigate/test/test_detection_runners.py`
that will fail if any of these three fixes is accidentally reverted.
Fixes: #23007
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>