Merge 87f55be805 into efe585a920

* keep global camera config subscribers broad when only one camera exists at startup

* update glossary

docs/docs/configuration/object_detectors.md

@@ -495,6 +495,7 @@ detectors:
 | [MobileNet v2](#ssdlite-mobilenet-v2) | ✅  | ✅  | Fast and lightweight model, less accurate than larger models |
 | [YOLOX](#yolox)                       | ✅  | ?   |                                                              |
 | [D-FINE / DEIMv2](#d-fine--deimv2)    | ❌  | ❌  |                                                              |
+| [NanoDet-Plus](#nanodet-plus)         | ?   | ?   |                                                              |
 
 #### SSDLite MobileNet v2
 
@@ -791,6 +792,44 @@ Note that the labelmap uses a subset of the complete COCO label set that has onl
 
 </details>
 
+#### NanoDet-Plus
+[NanoDet-Plus](https://github.com/RangiLyu/nanodet) is a lightweight object detection model that achieves
+good accuracy on CPUs given its small footprint.
+
+Script to export an ONNX model for use in Frigate is provided in [the models section](#downloading-nanodet-plus-models).
+
+:::warning
+
+NanoDet-Plus has not been tested in GPU nor NPU modes.
+
+:::
+
+<details>
+  <summary>NanoDet-Plus Setup & Config</summary>
+
+After placing the exported onnx model in your config/model_cache folder, you can use the following configuration:
+
+```yaml
+detectors:
+  ov:
+    type: openvino
+    device: CPU
+
+model:
+  model_type: nanodet_plus
+  width: 320
+  height: 320
+  input_tensor: nchw
+  input_dtype: float
+  input_pixel_format: bgr
+  path: /config/model_cache/nanodet_plus.onnx
+  labelmap_path: /labelmap/coco-80.txt
+```
+
+Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
+
+</details>
+
 ## Apple Silicon detector
 
 The NPU in Apple Silicon can't be accessed from within a container, so the [Apple Silicon detector client](https://github.com/frigate-nvr/apple-silicon-detector) must first be setup. It is recommended to use the Frigate docker image with `-standard-arm64` suffix, for example `ghcr.io/blakeblackshear/frigate:stable-standard-arm64`.
@@ -1029,6 +1068,7 @@ detectors:
 | [YOLO-NAS](#yolo-nas-1)              | ⚠️         | ⚠️      | Not supported by CUDA Graphs                        |
 | [YOLOX](#yolox-1)                    | ✅         | ✅      | Supports CUDA Graphs for optimal Nvidia performance |
 | [D-FINE / DEIMv2](#d-fine--deimv2-1) | ⚠️         | ❌      | Not supported by CUDA Graphs                        |
+| [NanoDet-Plus](#nanodet-plus-1)      | ✅         | ?       | Supports CUDA Graphs for optimal Nvidia performance |
 
 There is no default model provided, the following formats are supported:
 
@@ -1311,6 +1351,42 @@ model:
 
 Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
 
+#### NanoDet-Plus
+[NanoDet-Plus](https://github.com/RangiLyu/nanodet) is a lightweight object detection model that achieves
+good accuracy on CPUs given its small footprint.
+
+Script to export an ONNX model for use in Frigate is provided in [the models section](#downloading-nanodet-plus-models).
+:::warning
+
+NanoDet-Plus has not been tested on AMD GPUs.
+
+:::
+
+<details>
+  <summary>NanoDet-Plus Setup & Config</summary>
+
+After placing the exported onnx model in your config/model_cache folder, you can use the following configuration:
+
+```yaml
+detectors:
+  onnx:
+    type: onnx
+
+model:
+  model_type: nanodet_plus
+  width: 320
+  height: 320
+  input_tensor: nchw
+  input_dtype: float
+  input_pixel_format: bgr
+  path: /config/model_cache/nanodet_plus.onnx
+  labelmap_path: /labelmap/coco-80.txt
+```
+
+Note that the labelmap uses a subset of the complete COCO label set that has only 80 objects.
+
+</details>
+
 ## CPU Detector (not recommended)
 
 The CPU detector type runs a TensorFlow Lite model utilizing the CPU without hardware acceleration. It is recommended to use a hardware accelerated detector type instead for better performance. To configure a CPU based detector, set the `"type"` attribute to `"cpu"`.
@@ -2460,3 +2536,39 @@ ARG IMG_SIZE
 COPY --from=build /yolov9/yolov9-${MODEL_SIZE}.onnx /yolov9-${MODEL_SIZE}-${IMG_SIZE}.onnx
 EOF
 ```
+
+### Downloading NanoDet-Plus models
+NanoDet-Plus can be downloaded using the command below. Copy and paste the complete command to your terminal to export
+the model as `nanodet_plus.onnx` in the current working directory. The command builds the NanoDet-Plus environment,
+downloads the specified model and converts it to ONNX.
+
+The below command is configured to use the smallest model provided by the authors, NanoDet-Plus-m-320. Other models
+can be specified by changing the `URL_WEIGHTS` link to the appropriate pretrained weights URL from
+[NanoDet-Plus Model Zoo](https://github.com/RangiLyu/nanodet#model-zoo). Remember to change the `IMG_HEIGHT`,
+`IMG_WIDTH` and `CFG_PATH` ([configuration files](https://github.com/RangiLyu/nanodet/tree/main/config)) parameters
+accordingly.
+
+Compatible with the `labelmap/coco-80.txt` labelmap
+
+```sh
+docker build . --build-arg URL_WEIGHTS=https://drive.google.com/file/d/1Dq0cTIdJDUhQxJe45z6rWncbZmOyh1Tv/view?usp=sharing --build-arg IMG_HEIGHT=320 --build-arg IMG_WIDTH=320 --build-arg CFG_PATH=config/nanodet-plus-m_320.yml --output . -f- <<'EOF'
+FROM python:3.9 AS build
+RUN apt-get update && apt-get install --no-install-recommends -y cmake libgl1 && rm -rf /var/lib/apt/lists/*
+COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
+WORKDIR /nanodet_plus
+ADD https://github.com/RangiLyu/nanodet.git .
+RUN uv pip install --system onnx==1.18.0 onnxruntime onnx-simplifier==0.4.* onnxscript
+RUN uv pip install --system "numpy<2"
+RUN uv pip install --system -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
+RUN uv pip install --system -e .
+ARG URL_WEIGHTS
+RUN uv pip install --system gdown
+RUN gdown --fuzzy ${URL_WEIGHTS} -O nanodet_plus.pth
+ARG IMG_HEIGHT
+ARG IMG_WIDTH
+ARG CFG_PATH
+RUN python tools/export_onnx.py --cfg_path=${CFG_PATH} --model_path=nanodet_plus.pth --input_shape=${IMG_HEIGHT},${IMG_WIDTH} --out_path=nanodet_plus.onnx
+FROM scratch
+COPY --from=build /nanodet_plus/nanodet_plus.onnx nanodet_plus.onnx
+EOF
+```

docs/docs/frigate/glossary.md

@@ -5,20 +5,40 @@ title: Glossary
 
 The glossary explains terms commonly used in Frigate's documentation.
 
+## Alert
+
+The higher-priority of the two [review item](#review-item) severities, the other being a [detection](#detection). By default a review item is an alert when it involves a `person` or `car`; the qualifying [labels](#label) and [zones](#zone) can be configured. [See the review docs for more info](/configuration/review)
+
+## Attribute
+
+A property detected on an [object](#object) that exists alongside its [label](#label). Unlike a [sub label](#sub-label), an object can carry several attributes at once. Some attributes come directly from the object detection [model](#model) — for example `face`, `license_plate`, or delivery carrier logos such as `amazon`, `ups`, and `fedex` — while others come from a [custom object classification model](/configuration/custom_classification/object_classification) configured with the `attribute` type. Attributes are visible in the Tracked Object Details pane in Explore, in `frigate/events` MQTT messages, and through the HTTP API.
+
 ## Bounding Box
 
-A box returned from the object detection model that outlines an object in the frame. These have multiple colors depending on object type in the debug live view.
+A box returned by the object detection [model](#model) that outlines a detected [object](#object) in the frame. In the Debug view, bounding boxes are colored by object [label](#label).
 
 ### Bounding Box Colors
 
 - At startup different colors will be assigned to each object label
 - A dark blue thin line indicates that object is not detected at this current point in time
 - A gray thin line indicates that object is detected as being stationary
-- A thick line indicates that object is the subject of autotracking (when enabled).
+- A thick line indicates that object is the subject of autotracking (when enabled)
+
+## Class
+
+The categories a classification [model](#model) is trained to distinguish between. Each class is a distinct visual category the model predicts, plus a `none` class for inputs that don't fit any category. For example, a custom object classification model for `person` objects might use the classes `delivery_person`, `resident`, and `none`. The predicted class is applied to the [object](#object) as either a [sub label](#sub-label) or an [attribute](#attribute), depending on the model's configuration. [See the object classification docs for more info](/configuration/custom_classification/object_classification)
+
+## Detection
+
+The lower-priority of the two [review item](#review-item) severities, the other being an [alert](#alert). By default, any review item that does not qualify as an alert is a detection; the qualifying [labels](#label) and [zones](#zone) can be configured. Despite the name, a detection is a category of review item — not the same as the object detection performed by the [model](#model). [See the review docs for more info](/configuration/review)
 
 ## False Positive
 
-An incorrect detection of an object type. For example a dog being detected as a person, a chair being detected as a dog, etc. A person being detected in an area you want to ignore is not a false positive.
+An incorrect result from the object detection [model](#model), where it assigns the wrong [label](#label) to something in the frame — for example a dog identified as a person, or a chair identified as a dog. A person correctly identified in an area you want to ignore is not a false positive.
+
+## Label
+
+The type assigned to a detected [object](#object) by the object detection [model](#model), drawn from the model's labelmap — for example `person`, `car`, or `dog`. Frigate tracks `person` by default; additional labels are tracked by adding them to the objects configuration. [See the available objects docs for the full list](/configuration/objects)
 
 ## Mask
 
@@ -26,44 +46,56 @@ There are two types of masks in Frigate. [See the mask docs for more info](/conf
 
 ### Motion Mask
 
-Motion masks prevent detection of [motion](#motion) in masked areas from triggering Frigate to run object detection, but do not prevent objects from being detected if object detection runs due to motion in nearby areas. For example: camera timestamps, skies, the tops of trees, etc.
+A motion mask stops [motion](#motion) in the masked area from triggering object detection. It does not stop an object from being detected when object detection runs because of motion in a nearby area. Use motion masks for parts of the frame that change constantly but never contain objects you care about — camera timestamps, the sky, the tops of trees, and so on.
 
 ### Object Mask
 
-Object filter masks drop any bounding boxes where the bottom center (overlap doesn't matter) is in the masked area. It forces them to be considered a [false positive](#false-positive) so that they are ignored.
+An object filter mask drops any [bounding box](#bounding-box) whose bottom center falls inside the masked area (overlap elsewhere doesn't matter). The object is forced to be treated as a [false positive](#false-positive) and ignored.
 
 ## Min Score
 
-The lowest score that an object can be detected with during tracking, any detection with a lower score will be assumed to be a false positive
+The lowest score a detected object can have to be kept during tracking. Anything scoring below the minimum is assumed to be a [false positive](#false-positive) and discarded.
+
+## Model
+
+A machine learning model that Frigate uses to detect or classify objects. The object detection model locates [objects](#object) in each frame and returns their [labels](#label) and [bounding boxes](#bounding-box). Additional enrichment models run on tracked objects to add detail: face recognition, license plate recognition, bird classification, custom object and state classification, and the embedding models used for semantic search. [See the object detectors docs for more info](/configuration/object_detectors)
 
 ## Motion
 
-When pixels in the current camera frame are different than previous frames. When many nearby pixels are different in the current frame they grouped together and indicated with a red motion box in the live debug view. [See the motion detection docs for more info](/configuration/motion_detection)
+A change in pixels between the current camera frame and previous frames. When many nearby pixels change together, they are grouped and shown as a red motion box in the debug live view. [See the motion detection docs for more info](/configuration/motion_detection)
+
+## Object
+
+Something Frigate can detect and follow in a camera frame, identified by its [label](#label) (for example a person or a car). The object types Frigate watches for are set in the `objects` configuration. Once an object is detected and followed across frames it becomes a [tracked object](#tracked-object-event-in-previous-versions), which may also carry a [sub label](#sub-label) and [attributes](#attribute). [See the available objects docs for more info](/configuration/objects)
 
 ## Region
 
-A portion of the camera frame that is sent to object detection, regions can be sent due to motion, active objects, or occasionally for stationary objects. These are represented by green boxes in the debug live view.
+A portion of the camera frame sent to the object detection [model](#model). Regions are selected because of [motion](#motion), active objects, or occasionally to recheck stationary objects, and are shown as green boxes in the debug live view.
 
 ## Review Item
 
-A review item is a time period where any number of events/tracked objects were active. [See the review docs for more info](/configuration/review)
+A period of time during which one or more [tracked objects](#tracked-object-event-in-previous-versions) were active, grouped together for review. Each review item is categorized as either an [alert](#alert) or a [detection](#detection). [See the review docs for more info](/configuration/review)
 
 ## Snapshot Score
 
-The score shown in a snapshot is the score of that object at that specific moment in time.
+The object's score at the specific moment the snapshot was captured.
+
+## Sub Label
+
+A more specific identity assigned to a [tracked object](#tracked-object-event-in-previous-versions) in addition to its [label](#label). A `person` may get the name of a recognized face, a `car` may get the name of a known license plate, and a `bird` may get its species. An object can have only one sub label at a time. Sub labels are produced by face recognition, license plate recognition, bird classification, custom object classification configured with the `sub label` type, and semantic search triggers.
 
 ## Threshold
 
-The threshold is the median score that an object must reach in order to be considered a true positive.
+The median score an object must reach to be considered a true positive.
 
 ## Top Score
 
-The top score for an object is the highest median score for an object.
+The highest median score an object reached over its lifetime.
 
 ## Tracked Object ("event" in previous versions)
 
-The time period starting when a tracked object entered the frame and ending when it left the frame, including any time that the object remained still. Tracked objects are saved when it is considered a [true positive](#threshold) and meets the requirements for a snapshot or recording to be saved.
+An [object](#object) followed from the moment it enters the frame until it leaves, including any time it stays still. A tracked object is saved once it is considered a [true positive](#threshold) and meets the requirements for a snapshot or recording.
 
 ## Zone
 
-Zones are areas of interest, zones can be used for notifications and for limiting the areas where Frigate will create a [review item](#review-item). [See the zone docs for more info](/configuration/zones)
+A user-defined area of interest within the camera frame. Zones can be used for notifications and to limit where Frigate creates a [review item](#review-item). [See the zone docs for more info](/configuration/zones)

frigate/config/camera/updater.py

@@ -73,7 +73,12 @@ class CameraConfigUpdateSubscriber:
 
         base_topic = "config/cameras"
 
-        if len(self.camera_configs) == 1:
+        # global subscribers must hear every camera; only narrow per-camera workers
+        is_global_subscriber = (
+            CameraConfigUpdateEnum.add in self.topics
+            or CameraConfigUpdateEnum.remove in self.topics
+        )
+        if not is_global_subscriber and len(self.camera_configs) == 1:
             base_topic += f"/{list(self.camera_configs.keys())[0]}"
 
         self.subscriber = ConfigSubscriber(

frigate/detectors/detector_config.py

@@ -42,6 +42,7 @@ class ModelTypeEnum(str, Enum):
     yolox = "yolox"
     yolonas = "yolonas"
     yologeneric = "yolo-generic"
+    nanodet_plus = "nanodet_plus"
 
 
 class ModelConfig(BaseModel):

frigate/detectors/plugins/onnx.py

@@ -14,6 +14,7 @@ from frigate.detectors.detector_config import (
 )
 from frigate.util.model import (
     post_process_dfine,
+    post_process_nanodet_plus,
     post_process_rfdetr,
     post_process_yolo,
     post_process_yolox,
@@ -137,6 +138,12 @@ class ONNXDetector(DetectionApi):
                 self.grids,
                 self.expanded_strides,
             )
+        elif self.onnx_model_type == ModelTypeEnum.nanodet_plus:
+            return post_process_nanodet_plus(
+                tensor_output[0],
+                self.width,
+                self.height,
+            )
         else:
             raise Exception(
                 f"{self.onnx_model_type} is currently not supported for onnx. See the docs for more info on supported models."

frigate/detectors/plugins/openvino.py

@@ -10,6 +10,7 @@ from frigate.detectors.detection_runners import OpenVINOModelRunner
 from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
 from frigate.util.model import (
     post_process_dfine,
+    post_process_nanodet_plus,
     post_process_rfdetr,
     post_process_yolo,
 )
@@ -43,6 +44,7 @@ class OvDetector(DetectionApi):
         ModelTypeEnum.yolonas,
         ModelTypeEnum.yologeneric,
         ModelTypeEnum.yolox,
+        ModelTypeEnum.nanodet_plus,
     ]
 
     def __init__(self, detector_config: OvDetectorConfig):
@@ -238,3 +240,9 @@ class OvDetector(DetectionApi):
                     object_detected[6], object_detected[5], object_detected[:4]
                 )
             return detections
+        elif self.ov_model_type == ModelTypeEnum.nanodet_plus:
+            return post_process_nanodet_plus(
+                outputs[0],
+                self.width,
+                self.height,
+            )

frigate/util/model.py

@@ -7,6 +7,7 @@ from typing import Any
 import cv2
 import numpy as np
 import onnxruntime as ort
+import scipy.special
 
 from frigate.const import MODEL_CACHE_DIR
 
@@ -16,6 +17,57 @@ logger = logging.getLogger(__name__)
 ### Post Processing
 
 
+def calculate_nanodet_center_priors(
+    input_height: int, input_width: int, strides: tuple, dtype: type
+):
+    """
+    Adapted from https://github.com/RangiLyu/nanodet/blob/be9b4a9001d7f9b6fc89c2df31ae8d428e35b4f0/nanodet/model/head/nanodet_plus_head.py
+    """
+
+    def get_single_level_center_priors(featmap_size, stride, dtype):
+        """Generate centers of a single stage feature map.
+        Args:
+            batch_size (int): Number of images in one batch.
+            featmap_size (tuple[int]): height and width of the feature map
+            stride (int): down sample stride of the feature map
+            dtype (obj:`torch.dtype`): data type of the tensors
+            device (obj:`torch.device`): device of the tensors
+        Return:
+            priors (Tensor): center priors of a single level feature map.
+        """
+        h, w = featmap_size
+        x_range = (np.arange(w, dtype=dtype)) * stride
+        y_range = (np.arange(h, dtype=dtype)) * stride
+        y, x = np.meshgrid(y_range, x_range, indexing="ij")
+        y = y.flatten()
+        x = x.flatten()
+        strides = np.full(x.shape[0], stride)
+        priors = np.stack([x, y, strides, strides], axis=-1)
+        return priors
+
+    featmap_sizes = [
+        (
+            int(np.ceil(input_height / stride)),
+            int(np.ceil(input_width) / stride),
+        )
+        for stride in strides
+    ]
+    mlvl_center_priors = [
+        get_single_level_center_priors(
+            featmap_sizes[i],
+            stride,
+            dtype,
+        )
+        for i, stride in enumerate(strides)
+    ]
+    center_priors = np.concatenate(mlvl_center_priors, axis=0)
+
+    return center_priors
+
+
+nanodet_center_priors: dict[(int, int, tuple, type), np.ndarray] = {}
+
+
 def post_process_dfine(
     tensor_output: np.ndarray, width: int, height: int
 ) -> np.ndarray:
@@ -280,6 +332,80 @@ def post_process_yolox(
     return detections
 
 
+def post_process_nanodet_plus(
+    predictions: np.ndarray,
+    width: int,
+    height: int,
+):
+    """
+    Adapted from https://github.com/RangiLyu/nanodet/blob/be9b4a9001d7f9b6fc89c2df31ae8d428e35b4f0/nanodet/model/head/nanodet_plus_head.py
+    """
+
+    def distance2bbox(points, distance, max_shape=None):
+        """Decode distance prediction to bounding box.
+
+        Args:
+            points (Tensor): Shape (n, 2), [x, y].
+            distance (Tensor): Distance from the given point to 4
+                boundaries (left, top, right, bottom).
+            max_shape (tuple): Shape of the image.
+
+        Returns:
+            Tensor: Decoded bboxes.
+        """
+        x1 = points[..., 0] - distance[..., 0]
+        y1 = points[..., 1] - distance[..., 1]
+        x2 = points[..., 0] + distance[..., 2]
+        y2 = points[..., 1] + distance[..., 3]
+        if max_shape is not None:
+            x1 = np.clip(x1, 0, max_shape[1])
+            y1 = np.clip(y1, 0, max_shape[0])
+            x2 = np.clip(x2, 0, max_shape[1])
+            y2 = np.clip(y2, 0, max_shape[0])
+        return np.stack([x1, y1, x2, y2], -1)
+
+    predictions = predictions[0]
+
+    # Below two parameters are consistent with all nanodet **plus** models
+    reg_max = 7
+    strides = (8, 16, 32, 64)
+
+    num_classes = predictions.shape[-1] - 4 * (reg_max + 1)
+    cls_scores, bbox_preds = predictions[:, :num_classes], predictions[:, num_classes:]
+
+    try:
+        center_priors = nanodet_center_priors[
+            (height, width, strides, predictions[0].dtype)
+        ]
+    except KeyError:
+        center_priors = calculate_nanodet_center_priors(
+            height, width, strides, predictions[0].dtype
+        )
+        nanodet_center_priors[(height, width, strides, predictions[0].dtype)] = (
+            center_priors
+        )
+
+    x = bbox_preds.reshape(bbox_preds.shape[0], 4, reg_max + 1)
+    x = scipy.special.softmax(x, axis=-1)
+    x = np.dot(x, np.linspace(0, reg_max, reg_max + 1))
+
+    dis_preds = x * center_priors[..., 2, None]
+    bboxes = distance2bbox(center_priors[..., :2], dis_preds, max_shape=(height, width))
+
+    class_ids = np.argmax(cls_scores, axis=1)
+    scores = np.max(cls_scores, axis=1)
+
+    detections = np.zeros((20, 6), dtype=np.float32)
+    for i, j in enumerate(np.argsort(scores)[::-1][:20]):
+        detections[i, 0] = class_ids[j]
+        detections[i, 1] = scores[j]
+        detections[i, 2] = bboxes[j, 1] / height
+        detections[i, 3] = bboxes[j, 0] / width
+        detections[i, 4] = bboxes[j, 3] / height
+        detections[i, 5] = bboxes[j, 2] / width
+    return detections
+
+
 ### ONNX Utilities