Implement annotated frames for GenAI Review (#24379)
CI / AMD64 Build (push) Canceled after 0s
CI / AMD64 Smoke Test (push) Canceled after 0s
CI / ARM Build (push) Canceled after 0s
CI / Jetson Jetpack 6 (push) Canceled after 0s
CI / AMD64 Extra Build (push) Canceled after 0s
CI / ARM Extra Build (push) Canceled after 0s
CI / Synaptics Build (push) Canceled after 0s
CI / Assemble and push default build (push) Canceled after 0s

* Implement annotated frames mode for GenAI reviews to improve models with lacking temporal understanding

* Updates

* Improve debug sharing

* Do not number objects

* Fix assumptions

* Remove unhelpful content

* Improve object data sent as part of prompt

* Cleanup ollama dumbness

* Bind db

* Fixes

* Cleanup
This commit is contained in:
Nicolas Mowen
2026-09-17 10:28:37 -06:00
committed by GitHub
parent 10a0d5ea37
commit eccd10cd94
19 changed files with 1128 additions and 118 deletions
@@ -657,6 +657,10 @@
"label": "Preferred language",
"description": "Preferred language to request from the GenAI provider for generated responses."
},
"frame_mode": {
"label": "Frame mode",
"description": "How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses."
},
"response_style": {
"label": "Response style",
"description": "Writing style preset for generated review descriptions. Presets adjust the tone and level of detail of the user-facing title, summary, and scene description; 'default' leaves the built-in prompt unchanged."
+4
View File
@@ -1028,6 +1028,10 @@
"label": "Preferred language",
"description": "Preferred language to request from the GenAI provider for generated responses."
},
"frame_mode": {
"label": "Frame mode",
"description": "How frames are presented to the model. 'frames' sends the prompt followed by the frames, which suits models that track a sequence well on their own. 'annotated_frames' labels each frame and interleaves notes derived from object tracking, which helps models that lose track of activity that repeats or reverses."
},
"response_style": {
"label": "Response style",
"description": "Writing style preset for generated review descriptions. Presets adjust the tone and level of detail of the user-facing title, summary, and scene description; 'default' leaves the built-in prompt unchanged."
@@ -1912,6 +1912,10 @@
"imageSource": {
"recordings": "Recordings",
"previews": "Previews"
},
"frameMode": {
"frames": "Frames",
"annotated_frames": "Annotated frames"
}
},
"logger": {
@@ -82,6 +82,7 @@ const review: SectionConfigOverrides = {
"detections.labels": "/configuration/review/#alerts-and-detections",
genai: "/configuration/genai/genai_review",
"genai.image_source": "/configuration/genai/genai_review#image-source",
"genai.frame_mode": "/configuration/genai/genai_review#frame-mode",
"genai.additional_concerns":
"/configuration/genai/genai_review#additional-concerns",
},
@@ -138,6 +139,11 @@ const review: SectionConfigOverrides = {
enumI18nPrefix: "review.imageSource",
},
},
frame_mode: {
"ui:options": {
enumI18nPrefix: "review.frameMode",
},
},
},
},
},