Audit of test_detection_runners.py against the actual fix changes
surfaced five cases where the test suite would silently pass after
a regression. Adds:
- test_fallback_warning_includes_developer_context: guards the
enriched CUDA-graph fallback warning fields (model_type, path,
device_id, providers) against revert to the bare form.
- test_default_sets_enable_all_optimization: guards the explicit
else branch that pins graph_optimization_level=ORT_ENABLE_ALL,
added to be robust against ORT default changes.
- test_gpu_mem_limit_key_present_when_cuda_query_succeeds: positive
counterpart to the existing omit-on-failure test.
- test_cuda_graph_doubles_peak_multiplier: locks in the 7 -> 14
multiplier relationship in compute_cuda_mem_limit.
- test_arcface_is_fixed / test_facenet_is_fixed /
test_yolov9_license_plate_is_fixed: explicit fixed-size guards
for the remaining enrichment models the fix targets.
28 tests pass (was 21).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When cudaMemGetInfo cannot be called or returns an error, compute_cuda_mem_limit
now returns None and the caller skips injecting gpu_mem_limit, leaving ORT to
manage its own arena (grow-as-needed up to device capacity).
Tradeoff documented in the docstring:
- Old behavior returned a hardcoded 4 GB. That was wrong for low-VRAM devices
(Jetson Nano 4 GB shared, Quadro K620 2 GB, GT 1030 2 GB) and broken
/dev/nvidia* container passthroughs, where requesting 4 GB caused
cudaErrorMemoryAllocation at session init. It was also wrong for big GPUs
(24 GB RTX 3090 with 20 GB free), needlessly starving the session.
- The leak vectors fixed elsewhere in this PR (mem_pattern + mallopt) are
independent of the BFC arena cap, so dropping the cap on the failure path
does not reintroduce the leak.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Same robustness pattern as the previous mem_pattern commit: previously
inherited ORT's implicit default (ORT_ENABLE_ALL) when is_complex_model
was False. Set explicitly in both branches so the behavior survives any
future ORT default change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously relied on ORT's implicit default (True) for fixed-size models,
only flipping the flag in the variable-length branch. Set it explicitly
in both branches to be robust against ORT default changes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Targeted follow-ups to the embeddings_manager ORT leak fix that affect all
GPU-resident embedding models (Jina text+vision, PaddleOCR det+rec, ArcFace,
YOLOv9 license plate). Detection-side YOLO runners are unaffected since
CudaGraphRunner does not call compute_cuda_mem_limit.
- compute_cuda_mem_limit now checks the cudaMemGetInfo return code instead
of trusting that a non-throwing call populated the buffers. Previously a
non-zero rc left both pointers at 0, producing gpu_mem_limit=0 and
immediate session OOM rather than the documented 4 GB fallback.
- The limit also factors in currently-free VRAM (free * 0.9), not just
total. On a shared GPU where co-resident embedding sessions have already
consumed most of the device, capping at 80% of total still over-allocates.
- The CUDA graph fallback path now logs the underlying exception text so
failures (cudaErrorStreamCaptureUnsupported, missing libnvrtc, etc.)
stop being swallowed by the bare except.
Tests cover all three regression paths plus updated existing tests that
now require cudaMemGetInfo to return cudaSuccess explicitly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three independent ORT/glibc leak vectors identified and fixed:
1. **ORT CPU BFC arena** (`enable_cpu_mem_arena=False` for all sessions)
ORT's default CPU arena pools host-side GPU↔CPU staging buffers indefinitely.
Disabling it across every InferenceSession (detection + embedding) stops
hundreds-of-MB/h RSS growth seen on systems with CUDA EP sessions.
2. **ORT memory-pattern cache** (`enable_mem_pattern=False` for variable-length models)
For embedding models with variable-length inputs (Jina v1/v2, PaddleOCR),
ORT allocates one mmap-backed execution plan per unique sequence length and
never frees them. Disabling the pattern cache stops this unbounded anon-mmap
growth. Fixed-size models (YOLO) keep `enable_mem_pattern=True` to preserve
buffer aliasing and avoid CUDA graph capture failures.
3. **mallopt(M_ARENA_MAX)** called from `EmbeddingProcess.run()`
The forkserver start method exec()s a fresh Python interpreter that does not
inherit Docker env vars, so `MALLOC_ARENA_MAX` set in docker-compose never
reaches the child. Calling `mallopt(-8, os.cpu_count())` from `run()` caps
glibc malloc arenas in the child process.
Additional improvements:
- `compute_cuda_mem_limit()`: dynamically caps the ORT CUDA EP BFC arena for
embedding sessions to min(model_size × 7, 80% VRAM); prevents OOM on
multi-model systems while leaving headroom for detection sessions.
- CUDA graph capture is now wrapped in try/except so models with CPU-only ops
(e.g. attention, NMS) fall back to ONNXModelRunner instead of crashing.
- `ONNXModelRunner.has_variable_length_inputs()`: centralises the
Jina/PaddleOCR detection logic to keep SessionOptions creation consistent.
- 17 regression-guard unit tests in `frigate/test/test_detection_runners.py`
that will fail if any of these three fixes is accidentally reverted.
Fixes: #23007
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>