Compare commits

...
6 Commits
Author SHA1 Message Date
Josh HawkinsandGitHub db9543c24b Merge b5a360be39 into ec3fb00494 2026-06-23 23:42:36 +00:00
ec3fb00494 perf(track): use sum()/len() instead of np.mean in average_boxes (#23521)
CI / AMD64 Build (push) Has been cancelled
CI / ARM Build (push) Has been cancelled
CI / Jetson Jetpack 6 (push) Has been cancelled
CI / AMD64 Extra Build (push) Has been cancelled
CI / ARM Extra Build (push) Has been cancelled
CI / Synaptics Build (push) Has been cancelled
CI / Assemble and push default build (push) Has been cancelled
* perf(track): avoid numpy reductions on tiny box lists in position smoothing

update_position runs per tracked object per frame. While a position has
fewer than 10 samples it calls np.percentile four times, and average_boxes
(per stationary object per frame) calls np.mean four times - all on lists of
at most 10 ints, where numpy's per-call dispatch/validation overhead
dominates the actual work.

Replace them with pure-Python equivalents:
- average_boxes: sum()/len() instead of np.mean (bit-identical output)
- interpolated_percentile(): linear-interpolated percentile matching
  numpy.percentile (including its lerp branch at frac>=0.5) for the small
  lists used here, in place of np.percentile

Measured in the release image (numpy 1.26.4) on a 10-element list:
np.percentile 18735 ns -> 191 ns/call (98x); np.mean-based average_boxes
7480 ns -> 591 ns (12.7x); ~74 us saved per object-frame in update_position.
A live py-spy --gil profile of a camera process_frames worker showed
np.percentile (update_position) and np.mean (average_boxes) among the top
Frigate-owned on-CPU frames.

Output is unchanged: added tests assert both helpers are bit-identical to
numpy over randomized small inputs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Drop interpolated_percentile, keep only average_boxes

Per review: reimplementing np.percentile hurts readability and risks
divergence from numpy (e.g. numpy 2.x). Revert update_position to
np.percentile and remove the helper; keep only the average_boxes change
(sum()/len() instead of np.mean), which stays bit-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 15:47:04 -06:00
081d6f95ef perf(track): avoid per-frame allocations and list lookups in tracker (#23523)
Two small per-object-per-frame improvements in the tracker hot path
(match_and_update), both bit-identical:

- get_stationary_threshold returned a freshly constructed StationaryThresholds
  (a dataclass plus a list) on every call for any label not in the three
  known lists - i.e. for common labels like person/dog. The default thresholds
  are constant and never mutated, so return a shared module-level singleton,
  as the other three cases already do.
- untracked_object_boxes membership used `box not in [list of boxes]` (O(n));
  build a set of box tuples for O(1) membership. Boxes are hashable as tuples
  and output is unchanged.

get_stationary_threshold appeared in a live py-spy --gil profile of a camera
process_frames worker. Adds tests for the threshold lookups.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 15:24:05 -06:00
a2b46f5d84 perf(util): cut redundant work in per-frame detection consolidation (#23522)
video/detect.py runs these for every frame:

- get_cluster_candidates: used_boxes was a list with `in` membership tests
  inside the nested loop (O(n) per check). It is only ever membership-tested,
  so switching it to a set (O(1)) leaves output unchanged.
- get_consolidated_object_detections: area(current_box) was recomputed on
  every inner-loop iteration though it is loop-invariant; hoist it to one
  call per outer detection.

Both are bit-identical (verified against the previous implementations over
randomized inputs). Measured in the release image, get_cluster_candidates on
a frame of 30 detection boxes: 59.2 us -> 42.1 us (1.4x); the gain scales
with the number of boxes per frame.

Adds a partition-invariant test (every box index lands in exactly one
cluster).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 15:23:18 -06:00
Josh Hawkins b5a360be39 add test 2026-04-17 17:18:11 -05:00
Josh Hawkins 54a7c5015e fix birdseye layout calculation
replace the two pass layout with a single pass pixel space algorithm
2026-04-17 17:18:04 -05:00
8 changed files with 335 additions and 127 deletions
+83 -103
View File
@@ -595,112 +595,92 @@ class BirdsEyeFrameManager:
) -> Optional[list[list[Any]]]:
"""Calculate the optimal layout for 2+ cameras."""
def map_layout(
camera_layout: list[list[Any]], row_height: int
) -> tuple[int, int, Optional[list[list[Any]]]]:
"""Map the calculated layout."""
candidate_layout = []
starting_x = 0
x = 0
max_width = 0
y = 0
def find_available_x(
current_x: int,
width: int,
reserved_ranges: list[tuple[int, int]],
max_width: int,
) -> Optional[int]:
"""Find the first horizontal slot that does not collide with reservations."""
x = current_x
for row in camera_layout:
final_row = []
max_width = max(max_width, x)
x = starting_x
for cameras in row:
camera_dims = self.cameras[cameras[0]]["dimensions"].copy()
camera_aspect = cameras[1]
for reserved_start, reserved_end in sorted(reserved_ranges):
if x >= reserved_end:
continue
if camera_dims[1] > camera_dims[0]:
scaled_height = int(row_height * 2)
scaled_width = int(scaled_height * camera_aspect)
starting_x = scaled_width
else:
scaled_height = row_height
scaled_width = int(scaled_height * camera_aspect)
if x + width <= reserved_start:
return x
# layout is too large
if (
x + scaled_width > self.canvas.width
or y + scaled_height > self.canvas.height
):
return x + scaled_width, y + scaled_height, None
x = max(x, reserved_end)
final_row.append((cameras[0], (x, y, scaled_width, scaled_height)))
x += scaled_width
if x + width <= max_width:
return x
y += row_height
candidate_layout.append(final_row)
if max_width == 0:
max_width = x
return max_width, y, candidate_layout
canvas_aspect_x, canvas_aspect_y = self.canvas.get_aspect(coefficient)
camera_layout: list[list[Any]] = []
camera_layout.append([])
starting_x = 0
x = starting_x
y = 0
y_i = 0
max_y = 0
for camera in cameras_to_add:
camera_dims = self.cameras[camera]["dimensions"].copy()
camera_aspect_x, camera_aspect_y = self.canvas.get_camera_aspect(
camera, camera_dims[0], camera_dims[1]
)
if camera_dims[1] > camera_dims[0]:
portrait = True
else:
portrait = False
if (x + camera_aspect_x) <= canvas_aspect_x:
# insert if camera can fit on current row
camera_layout[y_i].append(
(
camera,
camera_aspect_x / camera_aspect_y,
)
)
if portrait:
starting_x = camera_aspect_x
else:
max_y = max(
max_y,
camera_aspect_y,
)
x += camera_aspect_x
else:
# move on to the next row and insert
y += max_y
y_i += 1
camera_layout.append([])
x = starting_x
if x + camera_aspect_x > canvas_aspect_x:
return None
camera_layout[y_i].append(
(
camera,
camera_aspect_x / camera_aspect_y,
)
)
x += camera_aspect_x
if y + max_y > canvas_aspect_y:
return None
row_height = int(self.canvas.height / coefficient)
total_width, total_height, standard_candidate_layout = map_layout(
camera_layout, row_height
)
def map_layout(row_height: int) -> tuple[int, int, Optional[list[list[Any]]]]:
"""Lay out cameras row by row while reserving portrait spans for the next row."""
candidate_layout: list[list[Any]] = []
reserved_ranges: dict[int, list[tuple[int, int]]] = {}
current_row: list[Any] = []
row_index = 0
row_y = 0
row_x = 0
max_width = 0
max_height = 0
for camera in cameras_to_add:
camera_dims = self.cameras[camera]["dimensions"].copy()
camera_aspect_x, camera_aspect_y = self.canvas.get_camera_aspect(
camera, camera_dims[0], camera_dims[1]
)
portrait = camera_dims[1] > camera_dims[0]
scaled_height = row_height * 2 if portrait else row_height
scaled_width = int(scaled_height * (camera_aspect_x / camera_aspect_y))
while True:
x = find_available_x(
row_x,
scaled_width,
reserved_ranges.get(row_index, []),
self.canvas.width,
)
if x is not None and row_y + scaled_height <= self.canvas.height:
current_row.append(
(camera, (x, row_y, scaled_width, scaled_height))
)
row_x = x + scaled_width
max_width = max(max_width, row_x)
max_height = max(max_height, row_y + scaled_height)
if portrait:
reserved_ranges.setdefault(row_index + 1, []).append(
(x, row_x)
)
break
if current_row:
candidate_layout.append(current_row)
current_row = []
row_index += 1
row_y = row_index * row_height
row_x = 0
if row_y + scaled_height > self.canvas.height:
overflow_width = max(max_width, scaled_width)
overflow_height = row_y + scaled_height
return overflow_width, overflow_height, None
if current_row:
candidate_layout.append(current_row)
return max_width, max_height, candidate_layout
row_height = max(1, int(self.canvas.height / coefficient))
total_width, total_height, standard_candidate_layout = map_layout(row_height)
if not standard_candidate_layout:
# if standard layout didn't work
@@ -709,9 +689,9 @@ class BirdsEyeFrameManager:
total_width / self.canvas.width,
total_height / self.canvas.height,
)
row_height = int(row_height / scale_down_percent)
row_height = max(1, int(row_height / scale_down_percent))
total_width, total_height, standard_candidate_layout = map_layout(
camera_layout, row_height
row_height
)
if not standard_candidate_layout:
@@ -725,8 +705,8 @@ class BirdsEyeFrameManager:
1 / (total_width / self.canvas.width),
1 / (total_height / self.canvas.height),
)
row_height = int(row_height * scale_up_percent)
_, _, scaled_layout = map_layout(camera_layout, row_height)
row_height = max(1, int(row_height * scale_up_percent))
_, _, scaled_layout = map_layout(row_height)
if scaled_layout:
return scaled_layout
+156 -2
View File
@@ -1,11 +1,64 @@
"""Test camera user and password cleanup."""
"""Tests for Birdseye canvas sizing and layout behavior."""
import unittest
from multiprocessing import Event
from frigate.output.birdseye import get_canvas_shape
from frigate.config import FrigateConfig
from frigate.output.birdseye import BirdsEyeFrameManager, get_canvas_shape
class TestBirdseye(unittest.TestCase):
def _build_manager(
self, camera_dimensions: dict[str, tuple[int, int]]
) -> BirdsEyeFrameManager:
config = {
"mqtt": {"host": "mqtt"},
"birdseye": {"width": 1280, "height": 720},
"cameras": {},
}
for order, (camera, dimensions) in enumerate(
camera_dimensions.items(), start=1
):
config["cameras"][camera] = {
"ffmpeg": {
"inputs": [
{
"path": f"rtsp://10.0.0.1:554/{camera}",
"roles": ["detect"],
}
]
},
"detect": {
"width": dimensions[0],
"height": dimensions[1],
"fps": 5,
},
"birdseye": {"order": order},
}
return BirdsEyeFrameManager(FrigateConfig(**config), Event())
def _assert_no_overlaps(
self, layout: list[list[tuple[str, tuple[int, int, int, int]]]]
):
rectangles = [position for row in layout for _, position in row]
for index, rect in enumerate(rectangles):
x1, y1, width1, height1 = rect
for other in rectangles[index + 1 :]:
x2, y2, width2, height2 = other
overlap = (
x1 < x2 + width2
and x2 < x1 + width1
and y1 < y2 + height2
and y2 < y1 + height1
)
self.assertFalse(
overlap,
msg=f"Overlapping rectangles found: {rect} and {other}",
)
def test_16x9(self):
"""Test 16x9 aspect ratio works as expected for birdseye."""
width = 1280
@@ -45,3 +98,104 @@ class TestBirdseye(unittest.TestCase):
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width # width will be the same
assert canvas_height != height
def test_portrait_camera_does_not_overlap_next_row(self):
"""Portrait cameras should reserve their real horizontal position on the next row."""
manager = self._build_manager(
{
"cam_a": (1280, 720),
"cam_p": (360, 640),
"cam_b": (1280, 720),
"cam_c": (640, 480),
}
)
layout = manager.calculate_layout(["cam_a", "cam_p", "cam_b", "cam_c"], 3)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
cam_c = [
position for row in layout for camera, position in row if camera == "cam_c"
][0]
self.assertEqual(cam_c[0], 0)
def test_portrait_reservation_only_applies_to_next_row(self):
"""Portrait reservations should not push later rows after the span ends."""
manager = self._build_manager(
{
"cam_a": (1280, 720),
"cam_p": (360, 640),
"cam_b": (1280, 720),
"cam_c": (1280, 720),
"cam_d": (1280, 720),
"cam_e": (1280, 720),
}
)
layout = manager.calculate_layout(
["cam_a", "cam_p", "cam_b", "cam_c", "cam_d", "cam_e"],
3,
)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
cam_e = [
position for row in layout for camera, position in row if camera == "cam_e"
][0]
self.assertEqual(cam_e[0], 0)
def test_multiple_portraits_reserve_distinct_ranges(self):
"""Multiple portrait cameras in one row should reserve separate spans below them."""
manager = self._build_manager(
{
"cam_a": (640, 480),
"cam_p1": (360, 640),
"cam_p2": (360, 640),
"cam_b": (640, 480),
"cam_c": (1280, 720),
"cam_d": (640, 480),
}
)
layout = manager.calculate_layout(
["cam_a", "cam_p1", "cam_p2", "cam_b", "cam_c", "cam_d"],
4,
)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
def test_two_landscapes_then_portrait_then_two_landscapes(self):
"""A portrait after two landscapes should reserve only its own tail span."""
manager = self._build_manager(
{
"cam_a": (1280, 720),
"cam_b": (1280, 720),
"cam_p": (360, 640),
"cam_c": (1280, 720),
"cam_d": (1280, 720),
}
)
layout = manager.calculate_layout(
["cam_a", "cam_b", "cam_p", "cam_c", "cam_d"],
3,
)
self.assertIsNotNone(layout)
assert layout is not None
self._assert_no_overlaps(layout)
cam_c = [
position for row in layout for camera, position in row if camera == "cam_c"
][0]
cam_d = [
position for row in layout for camera, position in row if camera == "cam_d"
][0]
self.assertEqual(cam_c[0], 0)
self.assertEqual(cam_d[0], cam_c[0] + cam_c[2])
+16
View File
@@ -1,6 +1,22 @@
import random
import unittest
import numpy as np
from frigate.track.tracked_object import TrackedObjectAttribute
from frigate.util.object import average_boxes
class TestBoxStatistics(unittest.TestCase):
def test_average_boxes_matches_numpy(self) -> None:
rng = random.Random(0)
for _ in range(5000):
boxes = [
[rng.randint(0, 4000) for _ in range(4)]
for _ in range(rng.randint(1, 10))
]
expected = [float(np.mean([b[i] for b in boxes])) for i in range(4)]
self.assertEqual(average_boxes(boxes), expected)
class TestAttribute(unittest.TestCase):
@@ -0,0 +1,39 @@
"""Tests for stationary object classification thresholds."""
import unittest
from frigate.track.stationary_classifier import (
DEFAULT_OBJECT_THRESHOLDS,
DYNAMIC_OBJECT_THRESHOLDS,
NON_STATIONARY_OBJECT_THRESHOLDS,
STATIONARY_OBJECT_THRESHOLDS,
StationaryThresholds,
get_stationary_threshold,
)
class TestStationaryThresholds(unittest.TestCase):
def test_known_labels_return_expected_singletons(self) -> None:
self.assertIs(get_stationary_threshold("package"), STATIONARY_OBJECT_THRESHOLDS)
self.assertIs(get_stationary_threshold("car"), DYNAMIC_OBJECT_THRESHOLDS)
self.assertIs(
get_stationary_threshold("license_plate"),
NON_STATIONARY_OBJECT_THRESHOLDS,
)
def test_unknown_label_returns_shared_default(self) -> None:
# an unknown label must reuse the shared default instance, not allocate
# a fresh one on every call (this runs per object per frame)
first = get_stationary_threshold("person")
second = get_stationary_threshold("dog")
self.assertIs(first, DEFAULT_OBJECT_THRESHOLDS)
self.assertIs(second, DEFAULT_OBJECT_THRESHOLDS)
def test_default_matches_a_fresh_instance(self) -> None:
# the shared default must be value-equivalent to the previous
# per-call StationaryThresholds()
self.assertEqual(DEFAULT_OBJECT_THRESHOLDS, StationaryThresholds())
if __name__ == "__main__":
unittest.main()
+18
View File
@@ -82,6 +82,24 @@ class TestRegion(unittest.TestCase):
assert len(cluster_candidates) == 2
def test_cluster_candidates_partition_boxes(self):
# every box index must appear in exactly one cluster (no box used twice,
# none dropped) - the invariant the used-box tracking enforces
boxes = [
(100, 100, 200, 200),
(202, 150, 252, 200),
(210, 160, 260, 210),
(900, 900, 950, 950),
(905, 905, 955, 955),
]
cluster_candidates = get_cluster_candidates(
self.frame_shape, self.min_region_size, boxes
)
assigned = [idx for cluster in cluster_candidates for idx in cluster]
self.assertEqual(sorted(assigned), list(range(len(boxes))))
def test_transliterate_to_latin(self):
self.assertEqual(transliterate_to_latin("frégate"), "fregate")
self.assertEqual(transliterate_to_latin("utilité"), "utilite")
+4 -2
View File
@@ -641,9 +641,11 @@ class NorfairTracker(ObjectTracker):
self.deregister(self.track_id_map[e_id], e_id)
# update list of object boxes that don't have a tracked object yet
tracked_object_boxes = [obj["box"] for obj in self.tracked_objects.values()]
tracked_object_boxes = {
tuple(obj["box"]) for obj in self.tracked_objects.values()
}
self.untracked_object_boxes = [
o[2] for o in detections if o[2] not in tracked_object_boxes
o[2] for o in detections if tuple(o[2]) not in tracked_object_boxes
]
def print_objects_as_table(self, tracked_objects: Sequence) -> None:
+4 -1
View File
@@ -63,6 +63,9 @@ NON_STATIONARY_OBJECT_THRESHOLDS = StationaryThresholds(
max_stationary_history=4,
)
# Default thresholds for any other object label
DEFAULT_OBJECT_THRESHOLDS = StationaryThresholds()
def get_stationary_threshold(label: str) -> StationaryThresholds:
"""Get the stationary thresholds for a given object label."""
@@ -76,7 +79,7 @@ def get_stationary_threshold(label: str) -> StationaryThresholds:
if label in NON_STATIONARY_OBJECT_THRESHOLDS.objects:
return NON_STATIONARY_OBJECT_THRESHOLDS
return StationaryThresholds()
return DEFAULT_OBJECT_THRESHOLDS
class StationaryMotionClassifier:
+15 -19
View File
@@ -339,18 +339,13 @@ def reduce_boxes(boxes, iou_threshold=0.0):
def average_boxes(boxes: list[list[int, int, int, int]]) -> list[int, int, int, int]:
"""Return a box that is the average of a list of boxes."""
x_mins = []
y_mins = []
x_max = []
y_max = []
for box in boxes:
x_mins.append(box[0])
y_mins.append(box[1])
x_max.append(box[2])
y_max.append(box[3])
return [np.mean(x_mins), np.mean(y_mins), np.mean(x_max), np.mean(y_max)]
n = len(boxes)
return [
sum(box[0] for box in boxes) / n,
sum(box[1] for box in boxes) / n,
sum(box[2] for box in boxes) / n,
sum(box[3] for box in boxes) / n,
]
def median_of_boxes(boxes: list[list[int, int, int, int]]) -> list[int, int, int, int]:
@@ -401,13 +396,13 @@ def get_cluster_candidates(frame_shape, min_region, boxes):
# determined by the max_region size minus half the box + 20%
# TODO: see if we can do this with numpy
cluster_candidates = []
used_boxes = []
used_boxes = set()
# loop over each box
for current_index, b in enumerate(boxes):
if current_index in used_boxes:
continue
cluster = [current_index]
used_boxes.append(current_index)
used_boxes.add(current_index)
cluster_boundary = get_cluster_boundary(b, min_region)
# find all other boxes that fit inside the boundary
for compare_index, compare_box in enumerate(boxes):
@@ -436,7 +431,7 @@ def get_cluster_candidates(frame_shape, min_region, boxes):
if should_cluster:
cluster.append(compare_index)
used_boxes.append(compare_index)
used_boxes.add(compare_index)
cluster_candidates.append(cluster)
# return the unique clusters only
@@ -558,6 +553,7 @@ def reduce_detections(
current_detection = sorted_by_area[current_detection_idx]
current_label = current_detection[0]
current_box = current_detection[2]
current_area = area(current_box)
overlap = 0
for to_check_idx in range(
min(current_detection_idx + 1, len(sorted_by_area)),
@@ -568,14 +564,14 @@ def reduce_detections(
# if area of current detection / area of check < 5% they should not be compared
# this covers cases where a large car parked in a driveway doesn't block detections
# of cars in the street behind it
if area(current_box) / area(to_check) < 0.05:
if current_area / area(to_check) < 0.05:
continue
intersect_box = intersection(current_box, to_check)
# if % of smaller detection is inside of another detection, consolidate
if intersect_box is not None and area(intersect_box) / area(
current_box
) > LABEL_CONSOLIDATION_MAP.get(
if intersect_box is not None and area(
intersect_box
) / current_area > LABEL_CONSOLIDATION_MAP.get(
current_label, LABEL_CONSOLIDATION_DEFAULT
):
overlap = 1