administrator/frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2026-11-06 21:53:02 +03:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Measure Gemini image-token cost instead of using the pixel heuristic (#24469)

Browse Source 
* Measure Gemini image-token cost instead of using the pixel heuristic

Review descriptions size their frame budget from the provider's
estimate_image_tokens(). Gemini did not override it, so the base
~1-token-per-1250-pixels heuristic was used: about 46 tokens for a 320x180
preview frame. Gemini 3 models actually bill about 1100 tokens per image
regardless of size, so the 24000-token image budget let Frigate send one
frame per second of the review item (e.g. 90 frames for a 90 s alert)
instead of ~21, and review descriptions cost roughly 4x more than intended.

Probe the model with the free count_tokens endpoint, the same way the
llama.cpp provider measures its mmproj, and cache the result per image
size. Fall back to the base heuristic if the request fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound Gemini image-token probes and back off after failures

estimate_image_tokens() runs on the enrichment thread, and count_tokens was
using the client's defaults: a 120 s timeout and 3 attempts on 429/5xx. A
hung endpoint could stall review processing for minutes per review, and a
rate-limited key retried on every review of an unmeasured size.

Give the probe a 5 s timeout with no retries, and after a failure use the
heuristic without probing again for 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update docs to note that Gemini charges roughly-flat token cost regardless of video resolution

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
David Knowles
2026-11-06 07:47:44 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 52c6175255
commit 463942e01d
3 changed files with 136 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/configuration/genai/config.md
+8 -2
View File
@@ -344,7 +344,7 @@ To start using Gemini, you must first get an API key from [Google AI Studio](htt
1. Navigate to <NavPath path="Settings > Enrichments > Generative AI" />.
- Set **Provider** to `gemini`
- Set **API key** to your Gemini API key (or use an environment variable such as `{FRIGATE_GEMINI_API_KEY}`)
- Set **Model** to the desired model (e.g., `gemini-2.5-flash`)
- Set **Model** to the desired model (e.g., `gemini-3.5-flash-lite`)
</TabItem>
<TabItem value="yaml">
@@ -354,12 +354,18 @@ genai:
my_provider:
provider: gemini
api_key: "{FRIGATE_GEMINI_API_KEY}"
model: gemini-2.5-flash
model: gemini-3.5-flash-lite
```
</TabItem>
</ConfigTabs>
:::tip
Gemini 3.x models charge a (roughly) flat ~1,100 tokens per image regardless of resolution, so a 180p preview frame costs the same as a 480p recording frame. Set the review [image source](/configuration/genai/genai_review#image-source) to `recordings` to send higher quality frames at no extra cost, as long as recordings are enabled for the camera.
:::
:::note
To use a different Gemini-compatible API endpoint, set the `provider_options` with the `base_url` key to your provider's API URL. For example: