This commit is contained in:
tpjanssen 2023-10-06 09:26:27 +02:00
commit e4ba43ceef
12 changed files with 205 additions and 52 deletions

View File

@ -55,13 +55,20 @@ fi
# arch specific packages
if [[ "${TARGETARCH}" == "amd64" ]]; then
# Use debian testing repo only for hwaccel packages
# use debian bookworm for AMD hwaccel packages
echo 'deb https://deb.debian.org/debian bookworm main contrib' >/etc/apt/sources.list.d/debian-bookworm.list
apt-get -qq update
apt-get -qq install --no-install-recommends --no-install-suggests -y \
mesa-va-drivers radeontop
rm -f /etc/apt/sources.list.d/debian-bookworm.list
# Use debian testing repo only for intel hwaccel packages
echo 'deb http://deb.debian.org/debian testing main non-free' >/etc/apt/sources.list.d/debian-testing.list
apt-get -qq update
# intel-opencl-icd specifically for GPU support in OpenVino
apt-get -qq install --no-install-recommends --no-install-suggests -y \
intel-opencl-icd \
mesa-va-drivers libva-drm2 intel-media-va-driver-non-free i965-va-driver libmfx1 radeontop intel-gpu-tools
libva-drm2 intel-media-va-driver-non-free i965-va-driver libmfx1 intel-gpu-tools
# something about this dependency requires it to be installed in a separate call rather than in the line above
apt-get -qq install --no-install-recommends --no-install-suggests -y \
i965-va-driver-shaders

View File

@ -5,7 +5,7 @@ title: Camera Autotracking
An ONVIF-capable, PTZ (pan-tilt-zoom) camera that supports relative movement within the field of view (FOV) can be configured to automatically track moving objects and keep them in the center of the frame.
![Autotracking Example](/img/frigate-autotracking-example.gif)
![Autotracking example with zooming](/img/frigate-autotracking-example.gif)
## Autotracking behavior
@ -64,6 +64,11 @@ cameras:
# absolute - use absolute zooming (supported by most PTZ capable cameras)
# relative - use relative zooming (not supported on all PTZs, but makes concurrent pan/tilt/zoom movements)
zooming: disabled
# Optional: A value to change the behavior of zooming on autotracked objects. (default: shown below)
# A lower value will keep more of the scene in view around a tracked object.
# A higher value will zoom in more on a tracked object, but Frigate may lose tracking more quickly.
# The value should be between 0.1 and 0.75
zoom_factor: 0.3
# Optional: list of objects to track from labelmap.txt (default: shown below)
track:
- person
@ -100,16 +105,22 @@ The object tracker in Frigate estimates the motion of the PTZ so that tracked ob
A fast [detector](object_detectors.md) is recommended. CPU detectors will not perform well or won't work at all. You can watch Frigate's debug viewer for your camera to see a thicker colored box around the object currently being autotracked.
![Autotracking Debug View](/img/autotracking-debug.gif)
A full-frame zone in `required_zones` is not recommended, especially if you've calibrated your camera and there are `movement_weights` defined in the configuration file. Frigate will continue to autotrack an object that has entered one of the `required_zones`, even if it moves outside of that zone.
## Zooming
Zooming is an experimental feature and may use significantly more CPU when tracking objects than panning/tilting only. It may be helpful to tweak your camera's autofocus settings if you are noticing focus problems when using zooming.
Zooming is still a very experimental feature and may use significantly more CPU when tracking objects than panning/tilting only. It may be helpful to tweak your camera's autofocus settings if you are noticing focus problems when using zooming.
Absolute zooming makes zoom movements separate from pan/tilt movements. Most PTZ cameras will support absolute zooming.
Relative zooming attempts to make a zoom movement concurrently with any pan/tilt movements. It was tested to work with some Dahua and Amcrest PTZs. But the ONVIF specification indicates that there no assumption about how the generic zoom range is mapped to magnification, field of view or other physical zoom dimension when using relative zooming. So if relative zooming behavior is erratic or just doesn't work, use absolute zooming.
You can optionally adjust the `zoom_factor` for your camera in your configuration file. Lower values will leave more space from the scene around the tracked object while higher values will cause your camera to zoom in more on the object. However, keep in mind that Frigate needs a fair amount of pixels and scene details outside of the bounding box of the tracked object to estimate the motion of your camera. If the object is taking up too much of the frame, Frigate will not be able to track the motion of the camera and your object will be lost.
The range of this option is from 0.1 to 0.75. The default value of 0.3 should be sufficient for most users. If you have a powerful zoom lens on your PTZ or you find your autotracked objects are often lost, you may want to lower this value. Because every PTZ and scene is different, you should experiment to determine what works best for you.
## Usage applications
In security and surveillance, it's common to use "spotter" cameras in combination with your PTZ. When your fixed spotter camera detects an object, you could use an automation platform like Home Assistant to move the PTZ to a specific preset so that Frigate can begin automatically tracking the object. For example: a residence may have fixed cameras on the east and west side of the property, capturing views up and down a street. When the spotter camera on the west side detects a person, a Home Assistant automation could move the PTZ to a camera preset aimed toward the west. When the object enters the specified zone, Frigate's autotracker could then continue to track the person as it moves out of view of any of the fixed cameras.

View File

@ -11,11 +11,11 @@ A camera is enabled by default but can be temporarily disabled by using `enabled
Each role can only be assigned to one input per camera. The options for roles are as follows:
| Role | Description |
| ---------- | ---------------------------------------------------------------------------------------- |
| `detect` | Main feed for object detection |
| `record` | Saves segments of the video feed based on configuration settings. [docs](record.md) |
| `rtmp` | Deprecated: Broadcast as an RTMP feed for other services to consume. [docs](restream.md) |
| Role | Description |
| -------- | ---------------------------------------------------------------------------------------- |
| `detect` | Main feed for object detection |
| `record` | Saves segments of the video feed based on configuration settings. [docs](record.md) |
| `rtmp` | Deprecated: Broadcast as an RTMP feed for other services to consume. [docs](restream.md) |
```yaml
mqtt:
@ -51,13 +51,18 @@ For camera model specific settings check the [camera specific](camera_specific.m
## Setting up camera PTZ controls
Add onvif config to camera
:::caution
Not every PTZ supports ONVIF, which is the standard protocol Frigate uses to communicate with your camera. Check your camera documentation or manufacturer's website to ensure your camera supports ONVIF. If your camera supports ONVIF and you continue to have trouble, make sure your camera is running the latest firmware.
:::
Add the onvif section to your camera in your configuration file:
```yaml
cameras:
back:
ffmpeg:
...
ffmpeg: ...
onvif:
host: 10.0.10.10
port: 8000
@ -65,6 +70,20 @@ cameras:
password: password
```
then PTZ controls will be available in the cameras WebUI.
If the ONVIF connection is successful, PTZ controls will be available in the camera's WebUI.
An ONVIF-capable camera that supports relative movement within the field of view (FOV) can also be configured to automatically track moving objects and keep them in the center of the frame. For autotracking setup, see the [autotracking](autotracking.md) docs.
## ONVIF PTZ camera recommendations
This list of working and non-working PTZ cameras is based on user feedback.
| Brand or specific camera | PTZ Controls | Autotracking | Notes |
| ------------------------ | :----------: | :----------: | ------------------------------------------------------- |
| Amcrest | ✅ | ⛔️ | Some older models (IP2M-841) don't support autotracking |
| Amcrest ASH21 | ❌ | ❌ | No ONVIF support |
| Dahua | ✅ | ✅ |
| Reolink 511WA | ✅ | ❌ | Zoom only |
| Reolink E1 Zoom | ✅ | ❌ | |
| Tapo C210 | ❌ | ❌ | Incomplete ONVIF support |
| Vikylin PTZ-2804X-I2 | ❌ | ❌ | Incomplete ONVIF support |

View File

@ -64,11 +64,10 @@ ffmpeg:
### Configuring Intel GPU Stats in Docker
Additional configuration is needed for the Docker container to be able to access the `intel_gpu_top` command for GPU stats. Three possible changes can be made:
Additional configuration is needed for the Docker container to be able to access the `intel_gpu_top` command for GPU stats. There are two options:
1. Run the container as privileged.
2. Adding the `CAP_PERFMON` capability.
3. Setting the `perf_event_paranoid` low enough to allow access to the performance event system.
2. Add the `CAP_PERFMON` capability (note: you might need to set the `perf_event_paranoid` low enough to allow access to the performance event system.)
#### Run as privileged
@ -125,7 +124,7 @@ _Note: This setting must be changed for the entire system._
For more information on the various values across different distributions, see https://askubuntu.com/questions/1400874/what-does-perf-paranoia-level-four-do.
Depending on your OS and kernel configuration, you may need to change the `/proc/sys/kernel/perf_event_paranoid` kernel tunable. You can test the change by running `sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid'` which will persist until a reboot. Make it permanent by running `sudo sh -c 'echo kernel.perf_event_paranoid=1 >> /etc/sysctl.d/local.conf'`
Depending on your OS and kernel configuration, you may need to change the `/proc/sys/kernel/perf_event_paranoid` kernel tunable. You can test the change by running `sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid'` which will persist until a reboot. Make it permanent by running `sudo sh -c 'echo kernel.perf_event_paranoid=2 >> /etc/sysctl.d/local.conf'`
## AMD/ATI GPUs (Radeon HD 2000 and newer GPUs) via libva-mesa-driver

View File

@ -596,6 +596,11 @@ cameras:
# absolute - use absolute zooming (supported by most PTZ capable cameras)
# relative - use relative zooming (not supported on all PTZs, but makes concurrent pan/tilt/zoom movements)
zooming: disabled
# Optional: A value to change the behavior of zooming on autotracked objects. (default: shown below)
# A lower value will keep more of the scene in view around a tracked object.
# A higher value will zoom in more on a tracked object, but Frigate may lose tracking more quickly.
# The value should be between 0.1 and 0.75
zoom_factor: 0.3
# Optional: list of objects to track from labelmap.txt (default: shown below)
track:
- person

BIN
docs/static/img/autotracking-debug.gif vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.6 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.6 MiB

After

Width:  |  Height:  |  Size: 28 MiB

View File

@ -13,6 +13,7 @@ from pydantic import BaseModel, Extra, Field, parse_obj_as, validator
from pydantic.fields import PrivateAttr
from frigate.const import (
ALL_ATTRIBUTE_LABELS,
AUDIO_MIN_CONFIDENCE,
CACHE_DIR,
DEFAULT_DB_PATH,
@ -152,6 +153,12 @@ class PtzAutotrackConfig(FrigateBaseModel):
zooming: ZoomingModeEnum = Field(
default=ZoomingModeEnum.disabled, title="Autotracker zooming mode."
)
zoom_factor: float = Field(
default=0.3,
title="Zooming factor (0.1-0.75).",
ge=0.1,
le=0.75,
)
track: List[str] = Field(default=DEFAULT_TRACKED_OBJECTS, title="Objects to track.")
required_zones: List[str] = Field(
default_factory=list,
@ -467,7 +474,7 @@ class ZoneConfig(BaseModel):
class ObjectConfig(FrigateBaseModel):
track: List[str] = Field(default=DEFAULT_TRACKED_OBJECTS, title="Objects to track.")
filters: Optional[Dict[str, FilterConfig]] = Field(title="Object filters.")
filters: Dict[str, FilterConfig] = Field(default={}, title="Object filters.")
mask: Union[str, List[str]] = Field(default="", title="Object mask.")
@ -1071,6 +1078,13 @@ class FrigateConfig(FrigateBaseModel):
config.mqtt.user = config.mqtt.user.format(**FRIGATE_ENV_VARS)
config.mqtt.password = config.mqtt.password.format(**FRIGATE_ENV_VARS)
# set default min_score for object attributes
for attribute in ALL_ATTRIBUTE_LABELS:
if not config.objects.filters.get(attribute):
config.objects.filters[attribute] = FilterConfig(min_score=0.7)
elif config.objects.filters[attribute].min_score == 0.5:
config.objects.filters[attribute].min_score = 0.7
# Global config to propagate down to camera level
global_config = config.dict(
include={

View File

@ -33,7 +33,7 @@ from frigate.util.image import (
logger = logging.getLogger(__name__)
def get_standard_aspect_ratio(width, height) -> tuple[int, int]:
def get_standard_aspect_ratio(width: int, height: int) -> tuple[int, int]:
"""Ensure that only standard aspect ratios are used."""
known_aspects = [
(16, 9),
@ -52,6 +52,22 @@ def get_standard_aspect_ratio(width, height) -> tuple[int, int]:
return known_aspects[known_aspects_ratios.index(closest)]
def get_canvas_shape(width: int, height: int) -> tuple[int, int]:
"""Get birdseye canvas shape."""
canvas_width = width
canvas_height = height
a_w, a_h = get_standard_aspect_ratio(width, height)
if round(a_w / a_h, 2) != round(width / height, 2):
canvas_width = width
canvas_height = (canvas_width / a_w) * a_h
logger.warning(
f"The birdseye resolution is a non-standard aspect ratio, forcing birdseye resolution to {canvas_width} x {canvas_height}"
)
return (canvas_width, canvas_height)
class Canvas:
def __init__(self, canvas_width: int, canvas_height: int) -> None:
gcd = math.gcd(canvas_width, canvas_height)
@ -226,8 +242,7 @@ class BirdsEyeFrameManager:
self.config = config
self.mode = config.birdseye.mode
self.frame_manager = frame_manager
width = config.birdseye.width
height = config.birdseye.height
width, height = get_canvas_shape(config.birdseye.width, config.birdseye.height)
self.frame_shape = (height, width)
self.yuv_shape = (height * 3 // 2, width)
self.frame = np.ndarray(self.yuv_shape, dtype=np.uint8)

View File

@ -2,7 +2,6 @@
import copy
import logging
import math
import os
import queue
import threading
@ -105,16 +104,22 @@ class PtzMotionEstimator:
# Norfair estimator function needs color so it can convert it right back to gray
frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2BGRA)
self.coord_transformations = self.norfair_motion_estimator.update(
frame, mask
)
try:
self.coord_transformations = self.norfair_motion_estimator.update(
frame, mask
)
logger.debug(
f"Motion estimator transformation: {self.coord_transformations.rel_to_abs([[0,0]])}"
)
except Exception:
# sometimes opencv can't find enough features in the image to find homography, so catch this error
logger.warning(
f"Autotracker: motion estimator couldn't get transformations for {camera_name} at frame time {frame_time}"
)
self.coord_transformations = None
self.frame_manager.close(frame_id)
logger.debug(
f"Motion estimator transformation: {self.coord_transformations.rel_to_abs([[0,0]])}"
)
return self.coord_transformations
@ -165,12 +170,14 @@ class PtzAutoTracker:
self.object_types: dict[str, object] = {}
self.required_zones: dict[str, object] = {}
self.move_queues: dict[str, object] = {}
self.move_queue_locks: dict[str, object] = {}
self.move_threads: dict[str, object] = {}
self.autotracker_init: dict[str, object] = {}
self.move_metrics: dict[str, object] = {}
self.calibrating: dict[str, object] = {}
self.intercept: dict[str, object] = {}
self.move_coefficients: dict[str, object] = {}
self.zoom_factor: dict[str, object] = {}
# if cam is set to autotrack, onvif should be set up
for camera_name, cam in self.config.cameras.items():
@ -186,6 +193,7 @@ class PtzAutoTracker:
self.object_types[camera_name] = cam.onvif.autotracking.track
self.required_zones[camera_name] = cam.onvif.autotracking.required_zones
self.zoom_factor[camera_name] = cam.onvif.autotracking.zoom_factor
self.tracked_object[camera_name] = None
self.tracked_object_previous[camera_name] = None
@ -196,6 +204,7 @@ class PtzAutoTracker:
self.move_coefficients[camera_name] = []
self.move_queues[camera_name] = queue.Queue()
self.move_queue_locks[camera_name] = threading.Lock()
if not self.onvif.cams[camera_name]["init"]:
if not self.onvif._init_onvif(camera_name):
@ -371,8 +380,9 @@ class PtzAutoTracker:
def _process_move_queue(self, camera):
while True:
try:
move_data = self.move_queues[camera].get()
move_data = self.move_queues[camera].get()
with self.move_queue_locks[camera]:
frame_time, pan, tilt, zoom = move_data
# if we're receiving move requests during a PTZ move, ignore them
@ -435,9 +445,6 @@ class PtzAutoTracker:
# calculate new coefficients if we have enough data
self._calculate_move_coefficients(camera)
except queue.Empty:
continue
def _enqueue_move(self, camera, frame_time, pan, tilt, zoom):
def split_value(value):
clipped = np.clip(value, -1, 1)
@ -446,7 +453,7 @@ class PtzAutoTracker:
if (
frame_time > self.ptz_metrics[camera]["ptz_start_time"].value
and frame_time > self.ptz_metrics[camera]["ptz_stop_time"].value
and self.move_queues[camera].qsize() == 0
and not self.move_queue_locks[camera].locked()
):
# don't make small movements
if abs(pan) < 0.02:
@ -470,7 +477,7 @@ class PtzAutoTracker:
tilt = tilt_excess
zoom = zoom_excess
def _should_zoom_in(self, camera, box, area):
def _should_zoom_in(self, camera, box, area, average_velocity):
camera_config = self.config.cameras[camera]
camera_width = camera_config.frame_shape[1]
camera_height = camera_config.frame_shape[0]
@ -485,7 +492,15 @@ class PtzAutoTracker:
#
# TODO: Take into account the area changing when an object is moving out of frame
edge_threshold = 0.15
area_threshold = 0.7
area_threshold = self.zoom_factor[camera]
velocity_threshold = 0.1
# if we have a fast moving object, let's zoom out
# fast moving is defined as a velocity of more than 10% of the camera's width or height
# so an object with an x velocity of 15 pixels on a 1280x720 camera would trigger a zoom out
velocity_threshold = average_velocity[0] > (
camera_width * velocity_threshold
) or average_velocity[1] > (camera_height * velocity_threshold)
# returns True to zoom in, False to zoom out
return (
@ -494,10 +509,12 @@ class PtzAutoTracker:
and bb_top > edge_threshold * camera_height
and bb_bottom < (1 - edge_threshold) * camera_height
and area < area_threshold * camera_area
and not velocity_threshold
)
def _autotrack_move_ptz(self, camera, obj):
camera_config = self.config.cameras[camera]
average_velocity = (0,) * 4
# # frame width and height
camera_width = camera_config.frame_shape[1]
@ -525,12 +542,19 @@ class PtzAutoTracker:
(y1 + y2) / 2,
)
# this box could exceed the frame boundaries if velocity is high
# but we'll handle that in _enqueue_move() as two separate moves
predicted_box = [
round(x + camera_fps * predicted_movement_time * v)
for x, v in zip(obj.obj_data["box"], average_velocity)
]
# get euclidean distance of the two points, sometimes the estimate is way off
distance = np.linalg.norm([x2 - x1, y2 - y1])
if distance <= 5:
# this box could exceed the frame boundaries if velocity is high
# but we'll handle that in _enqueue_move() as two separate moves
predicted_box = [
round(x + camera_fps * predicted_movement_time * v)
for x, v in zip(obj.obj_data["box"], average_velocity)
]
else:
# estimate was bad
predicted_box = obj.obj_data["box"]
centroid_x = round((predicted_box[0] + predicted_box[2]) / 2)
centroid_y = round((predicted_box[1] + predicted_box[3]) / 2)
@ -545,7 +569,15 @@ class PtzAutoTracker:
if camera_config.onvif.autotracking.zooming == ZoomingModeEnum.relative:
# relative zooming concurrently with pan/tilt
zoom = obj.obj_data["area"] / (camera_width * camera_height)
zoom = min(
obj.obj_data["area"]
/ (camera_width * camera_height)
* 100
* self.zoom_factor[camera],
1,
)
logger.debug(f"Zoom value: {zoom}")
# test if we need to zoom out
if not self._should_zoom_in(
@ -554,15 +586,18 @@ class PtzAutoTracker:
if camera_config.onvif.autotracking.movement_weights
else obj.obj_data["box"],
obj.obj_data["area"],
average_velocity,
):
zoom = -(1 - zoom)
# don't make small movements if area hasn't changed significantly
# don't make small movements to zoom in if area hasn't changed significantly
# but always zoom out if necessary
if (
"area" in obj.previous
and abs(obj.obj_data["area"] - obj.previous["area"])
/ obj.obj_data["area"]
< 0.1
< 0.2
and zoom > 0
):
zoom = 0
else:
@ -579,7 +614,7 @@ class PtzAutoTracker:
if 0 < zoom_level <= 1:
if self._should_zoom_in(
camera, obj.obj_data["box"], obj.obj_data["area"]
camera, obj.obj_data["box"], obj.obj_data["area"], (0, 0, 0, 0)
):
zoom = min(1.0, zoom_level + 0.1)
else:
@ -637,10 +672,11 @@ class PtzAutoTracker:
# more often to keep the object in the center. Raising the percentage will cause less
# movement and will be more flexible with objects not quite being centered.
# TODO: there's probably a better way to approach this
distance = math.sqrt(
(obj.obj_data["centroid"][0] - camera_config.detect.width / 2) ** 2
+ (obj.obj_data["centroid"][1] - camera_config.detect.height / 2)
** 2
distance = np.linalg.norm(
[
obj.obj_data["centroid"][0] - camera_config.detect.width / 2,
obj.obj_data["centroid"][1] - camera_config.detect.height / 2,
]
)
obj_width = obj.obj_data["box"][2] - obj.obj_data["box"][0]

View File

@ -0,0 +1,47 @@
"""Test camera user and password cleanup."""
import unittest
from frigate.output import get_canvas_shape
class TestBirdseye(unittest.TestCase):
def test_16x9(self):
"""Test 16x9 aspect ratio works as expected for birdseye."""
width = 1280
height = 720
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width
assert canvas_height == height
def test_4x3(self):
"""Test 4x3 aspect ratio works as expected for birdseye."""
width = 1280
height = 960
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width
assert canvas_height == height
def test_32x9(self):
"""Test 32x9 aspect ratio works as expected for birdseye."""
width = 2560
height = 720
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width
assert canvas_height == height
def test_9x16(self):
"""Test 9x16 aspect ratio works as expected for birdseye."""
width = 720
height = 1280
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width
assert canvas_height == height
def test_non_16x9(self):
"""Test non 16x9 aspect ratio fails for birdseye."""
width = 1280
height = 840
canvas_width, canvas_height = get_canvas_shape(width, height)
assert canvas_width == width # width will be the same
assert canvas_height != height

View File

@ -21,7 +21,7 @@ export default function LargeDialog({ children, portalRootID = 'dialogs' }) {
>
<div
role="modal"
className={`absolute rounded shadow-2xl bg-white dark:bg-gray-700 w-4/5 max-w-7xl text-gray-900 dark:text-white transition-transform transition-opacity duration-75 transform scale-90 opacity-0 ${
className={`absolute rounded shadow-2xl bg-white dark:bg-gray-700 w-4/5 md:h-2/3 max-w-7xl text-gray-900 dark:text-white transition-transform transition-opacity duration-75 transform scale-90 opacity-0 ${
show ? 'scale-100 opacity-100' : ''
}`}
>