Container security hardening (phase 3, breaking) (#24081)

* Run the frigate service as the frigate user * Run go2rtc as its own restricted user * Run nginx as the frigate user with writable state in /tmp/nginx * Disable bandwidth stats gracefully when not running as root * Hand TensorRT model cache ownership to the runtime user * Document non-root operation and per-hardware device access * Create /media/frigate after the ownership sweep * Assert non-root services, JWT migration, and escape hatch in CI * only write the sweep sentinel when a media volume is mounted * tolerate homekit config chown failures in the go2rtc run script * chown the s6 log pipe so non-root nginx can reopen /dev/stdout * set HOME to /config for non-root services * run smoke nginx -t and the write probe as the runtime user * re-own the nginx shm cache on service restart * discard stdout for the unprivileged smoke nginx -t * unwrap hard-wrapped prose in the installation docs * report progress during the ownership sweep * document EXTRA_GROUPS as the only device access path for dropped services * expand the non-root device access docs with diagnosis steps and udev rules * document network storage ownership and the remaining detector hardware * skip lost+found during the ownership sweep * hand /tmp/cache to the runtime user before services start * make bundled models readable by the runtime user * reload nginx by signaling the master instead of parsing its config as root * harden root writes into unprivileged-owned paths Restrict the sweep sentinel to a mount at or below /media/frigate so a parent /media mount cannot bless a later-shadowed volume. Rebuild /tmp/nginx root-owned each start so root's cp and tempio writes cannot follow a symlink an unprivileged nginx planted in the previous run. * collapse the duplicated sentinel comment * add a service-runs-as-root helper for granular root services * validate FRIGATE_ROOT_SERVICES and fail fast on unknown names * let services listed in FRIGATE_ROOT_SERVICES skip the privilege drop * record the root-services mode in the sentinel and sweep small trees each boot * cache the runtime ids in the ownership helper * chown recordings, previews, and exports to the runtime user at create * chown the database files after init * recommend FRIGATE_ROOT_SERVICES in the bandwidth stats warning * assert granular root services in CI * document FRIGATE_ROOT_SERVICES * own every directory level created for a recording segment * clear the cached runtime ids when ownership tests finish * skip missing media paths in the per-boot ownership sweep * clarify granular root services docs * clean up * install acl for device access grants * grant runtime users access to mapped device nodes at boot * assert device access grants in CI * document automatic device access grants * stop telling users device access needs host side setup * clarify the non-root docs * link the migration script to the repo * group the manual device setup under one section * harden against symlink attacks /config is owned by the unprivileged runtime user after the ownership sweep, so root operations on files there could be redirected by a planted symlink. - go2rtc HomeKit setup: replace the root yq/jq normalization and chown with an O_NOFOLLOW helper (prepare_homekit.py), so a symlink at go2rtc_homekit.yml can't redirect a root write or chown onto another file - go2rtc binary override: ignore /config/go2rtc whenever the service runs as root, so a planted binary can't exec as root under FRIGATE_ROOT_SERVICES - sweep sentinel: read and write it through safe-sentinel, which trusts only a root-owned regular file and never follows a symlink, so it can't be forged to skip the migration or symlinked to clobber a root file - ownership sweep: chown with -execdir so a parent directory swapped for a symlink mid-walk can't redirect the chown out of the volume - validate inputs: restrict DEVICE_ACL_PATHS to /dev, require nonzero numeric EXTRA_GROUPS, and reject PUID/PGID that collide with the go2rtc ids - docs: correct the TLS key ownership note to match what actually happens * tweak docs * stop the ownership sweep chasing entries other mechanisms own * keep custom binaries out of root services only under granular root
2026-09-26 23:28:59 +03:00 · 2026-09-12 07:30:04 -06:00
parent 1693415375
commit 3be59c9c18
38 changed files with 1427 additions and 83 deletions
@@ -1,15 +1,17 @@
 #!/bin/bash
 # Single source of truth for aligning volume ownership with the runtime user.
 #
-# Usage: fix-ownership [--dry-run] [--sentinel FILE] UID GID PATH [PATH...]
+# Usage: fix-ownership [--dry-run] [--sentinel FILE] [--mode STRING] UID GID PATH [PATH...]
 #
 #   --dry-run    report what would change, touch nothing
 #   --sentinel   skip entirely when FILE already records "SCHEMA:UID:GID";
 #                write it after a successful run (used by the boot path so
 #                multi-TB volumes are swept once per UID/schema change, not
 #                on every boot)
+#   --mode       append STRING to the sentinel, so changing it re-sweeps once
 #
 # Only files whose uid OR gid differs are touched, so re-runs are cheap.
+# lost+found is skipped: fsck fills it with root-only recovered fragments.
 # Top-level /config additionally grants group frigate-data TRAVERSE ONLY
 # (g+rx) so the separate go2rtc user can reach its pre-created HomeKit file
 # on hosts where /config is mounted 0700. Never g+w: directory write means
@@ -22,10 +24,11 @@ set -o errexit -o nounset -o pipefail
 # Permissions-layout epoch. Bump to force a one-time re-sweep on upgrade
 # (e.g. when the privilege-drop release must capture files created as root
 # since the previous sweep).
-schema=1
+schema=2

 dry_run=0
 sentinel=""
+mode=""

 while [[ "${1:-}" == --* ]]; do
    case "$1" in
@@ -36,12 +39,18 @@ while [[ "${1:-}" == --* ]]; do
                exit 2
            fi
            sentinel="$2"; shift 2 ;;
+        --mode)
+            if [[ -z "${2:-}" ]]; then
+                echo "[ERROR] fix-ownership: --mode requires a value" >&2
+                exit 2
+            fi
+            mode="$2"; shift 2 ;;
        *) echo "[ERROR] fix-ownership: unknown option $1" >&2; exit 2 ;;
    esac
 done

 if [[ $# -lt 3 ]]; then
-    echo "Usage: fix-ownership [--dry-run] [--sentinel FILE] UID GID PATH..." >&2
+    echo "Usage: fix-ownership [--dry-run] [--sentinel FILE] [--mode STRING] UID GID PATH..." >&2
    exit 2
 fi

@@ -54,11 +63,21 @@ if [[ "$(id -u)" -ne 0 ]]; then
    exit 0
 fi

-# A dry run always inspects: the sentinel records what a past sweep did, not
-# what the volume looks like now, and reporting from it would hide later drift.
-if [[ "$dry_run" -eq 0 && -n "$sentinel" && -f "$sentinel" && "$(cat "$sentinel")" == "${schema}:${target_uid}:${target_gid}" ]]; then
-    echo "[INFO] fix-ownership: ${target_uid}:${target_gid} (schema ${schema}) already applied, skipping"
-    exit 0
+# The list folds into the sentinel so entering or leaving a granular root mode
+# re-sweeps once, catching whatever the other ownership mechanisms missed.
+sentinel_content="${schema}:${target_uid}:${target_gid}"
+if [[ -n "$mode" ]]; then
+    sentinel_content="${sentinel_content}:${mode}"
+fi
+
+# safe-sentinel reports only a root-owned regular file, so a forged or
+# symlinked sentinel in the runtime-user-owned /config can't suppress the sweep
+if [[ "$dry_run" -eq 0 && -n "$sentinel" ]]; then
+    if existing=$(/usr/local/bin/safe-sentinel read "$sentinel" 2>/dev/null) && \
+        [[ "$existing" == "$sentinel_content" ]]; then
+        echo "[INFO] fix-ownership: ${target_uid}:${target_gid} (schema ${schema}) already applied, skipping"
+        exit 0
+    fi
 fi

 # A sweep that could not chown everything must not be recorded as complete:
@@ -66,6 +85,26 @@ fi
 # unreachable once services run unprivileged.
 swept_clean=1

+# Entries another mechanism deliberately owns. Chowning them undoes that work
+# and leaves the same "mismatch" waiting for the next boot, so /config could
+# never report itself clean: /config is chgrp'd to frigate-data below so go2rtc
+# can traverse it, and the HomeKit file is handed to the go2rtc user by the
+# go2rtc service. Only the GROUP on /config is exempt; a root-owned /config
+# must still be chowned or the runtime user cannot write there at all.
+# Shared by the counting and the chowning walk so the two cannot disagree.
+mismatch_expr=(
+    "(" -not -uid "$target_uid"
+        -o "(" -not -gid "$target_gid" -a ! -path /config ")"
+    ")"
+    -a ! -path /config/go2rtc_homekit.yml
+)
+if [[ -n "$sentinel" ]]; then
+    # safe-sentinel keeps the sentinel root-owned on purpose and rejects one
+    # owned by anybody else, so chowning it here would suppress the skip and
+    # make every boot re-sweep. Only the trailing write puts it back today.
+    mismatch_expr+=(-a ! -path "$sentinel")
+fi
+
 for path in "$@"; do
    # An absent root is an incomplete sweep, not a finished one: /media/frigate
    # is not in the image, so a boot before the volume is mounted would
@@ -76,11 +115,13 @@ for path in "$@"; do
        continue
    fi

+    echo "[INFO] fix-ownership: scanning ${path} for ownership mismatches; this may take a while on large filesystems"
+
    # find may fail mid-walk on a live volume (file deleted under it) or on a
    # stale mount. Tolerate it rather than aborting under errexit, but never
    # read a failed scan as "nothing to do": that would record the sweep as
    # complete without having looked.
-    if ! count=$(find "$path" \( -not -uid "$target_uid" -o -not -gid "$target_gid" \) -printf '.' 2>/dev/null | wc -c); then
+    if ! count=$(find "$path" -name lost+found -prune -o "${mismatch_expr[@]}" -printf '.' 2>/dev/null | wc -c); then
        swept_clean=0
        echo "[WARN] fix-ownership: could not scan ${path}; will retry on next boot"
        continue
@@ -98,17 +139,41 @@ for path in "$@"; do
        echo "[WARN] fix-ownership: ${path} contains symlinked directories; ownership behind them is not managed and must be aligned by hand"
    fi

-    echo "[WARN] fix-ownership: adjusting ownership of ${count} entries under ${path}; on large recordings volumes this can take a long time"
+    echo "[WARN] fix-ownership: adjusting ownership of ${count} entries under ${path}"
    if [[ "$dry_run" -eq 1 ]]; then
        echo "[INFO] fix-ownership: dry run, not changing ${path}"
        continue
    fi

-    find "$path" \( -not -uid "$target_uid" -o -not -gid "$target_gid" \) \
-        -exec chown -h "${target_uid}:${target_gid}" {} + || {
+    # -execdir chowns from the entry's own directory, so a parent swapped for a
+    # symlink mid-walk can't redirect the chown out of the volume
+    started=$SECONDS
+    if find "$path" -name lost+found -prune -o "${mismatch_expr[@]}" \
+        -print -execdir chown -h "${target_uid}:${target_gid}" {} + \
+        | awk -v total="$count" -v path="$path" '
+            BEGIN { next_pct = 5 }
+            {
+                pct = int(NR * 100 / total)
+                if (pct > 100) pct = 100
+                if (pct >= next_pct) {
+                    printf "[INFO] fix-ownership: %s %d%% (%d/%d entries)\n", path, pct, NR, total
+                    # mawk block-buffers to a pipe; without fflush the whole
+                    # progress log arrives at once
+                    fflush()
+                    while (next_pct <= pct) next_pct += 5
+                }
+            }'; then
+        elapsed=$((SECONDS - started))
+        if [[ "$elapsed" -ge 60 ]]; then
+            elapsed="$((elapsed / 60))m $((elapsed % 60))s"
+        else
+            elapsed="${elapsed}s"
+        fi
+        echo "[INFO] fix-ownership: finished ${path} in ${elapsed}"
+    else
        swept_clean=0
        echo "[WARN] fix-ownership: some entries under ${path} could not be updated (deleted mid-sweep or chown denied); will retry on next mismatch"
-    }
+    fi
 done

 # go2rtc (separate user) must be able to REACH its HomeKit state in /config.
@@ -124,6 +189,6 @@ if [[ "$dry_run" -eq 0 && -d /config ]]; then
 fi

 if [[ "$dry_run" -eq 0 && -n "$sentinel" && "$swept_clean" -eq 1 ]]; then
-    echo "${schema}:${target_uid}:${target_gid}" > "$sentinel" || \
+    /usr/local/bin/safe-sentinel write "$sentinel" "$sentinel_content" || \
        echo "[WARN] fix-ownership: could not write ${sentinel}; the sweep will run again on next boot"
 fi
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""Read or write the ownership sweep sentinel without following symlinks.
+
+The sentinel lives in /config, which the unprivileged runtime user owns, so it
+can be swapped for a symlink. read trusts only a root-owned regular file; write
+never follows a symlink or fifo onto another file.
+
+Usage:
+  safe-sentinel read PATH            print content, exit 0 only if root-owned regular file
+  safe-sentinel write PATH CONTENT   write CONTENT to a regular file at PATH
+"""
+
+import errno
+import os
+import stat
+import sys
+
+MODE = 0o644
+
+
+def do_read(path: str) -> int:
+    try:
+        fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
+    except OSError:
+        return 1
+    try:
+        st = os.fstat(fd)
+        if not stat.S_ISREG(st.st_mode) or st.st_uid != 0:
+            return 1
+        sys.stdout.buffer.write(os.read(fd, 4096))
+    finally:
+        os.close(fd)
+    return 0
+
+
+def do_write(path: str, content: str) -> int:
+    # O_NONBLOCK so a fifo fails fast (ENXIO) instead of blocking the open.
+    flags = os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW | os.O_NONBLOCK
+    replace = (errno.ELOOP, errno.ENXIO)
+    try:
+        fd = os.open(path, flags, MODE)
+        if not stat.S_ISREG(os.fstat(fd).st_mode):
+            os.close(fd)
+            raise OSError(errno.ELOOP, "not a regular file")
+    except OSError as err:
+        if err.errno not in replace:
+            raise
+        os.unlink(path)
+        fd = os.open(path, flags | os.O_EXCL, MODE)
+    try:
+        os.ftruncate(fd, 0)
+        os.write(fd, content.encode())
+        # keep it root-owned so a later sweep that chowned the old sentinel to
+        # the runtime user can't make the next read reject and re-sweep
+        os.fchown(fd, 0, 0)
+    finally:
+        os.close(fd)
+    return 0
+
+
+def main(argv: list[str]) -> int:
+    if len(argv) == 3 and argv[1] == "read":
+        return do_read(argv[2])
+    if len(argv) == 4 and argv[1] == "write":
+        try:
+            return do_write(argv[2], argv[3])
+        except OSError:
+            return 1
+    print("usage: safe-sentinel read PATH | write PATH CONTENT", file=sys.stderr)
+    return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))
@@ -0,0 +1,18 @@
+#!/bin/bash
+# Exit 0 when FRIGATE_ROOT_SERVICES names the given service. Membership only:
+# the euid and FRIGATE_RUN_AS_ROOT checks stay in the callers.
+#
+# Usage: service-runs-as-root SERVICE
+
+set -o nounset
+
+service="${1:?usage: service-runs-as-root SERVICE}"
+
+IFS=',' read -ra entries <<< "${FRIGATE_ROOT_SERVICES:-}"
+for entry in "${entries[@]}"; do
+    entry="${entry//[[:space:]]/}"
+    if [[ "$entry" == "$service" ]]; then
+        exit 0
+    fi
+done
+exit 1
@@ -0,0 +1,97 @@
+"""Normalize the go2rtc HomeKit file and hand it to go2rtc, as root.
+
+Runs before the drop. The file is in the runtime-user-owned /config, so a
+planted symlink could redirect the root write or chown onto another file;
+every operation goes through an O_NOFOLLOW fd to prevent that.
+
+Usage: prepare_homekit.py PATH [--chown]
+"""
+
+import errno
+import grp
+import io
+import os
+import pwd
+import stat
+import sys
+
+from ruamel.yaml import YAML
+
+RUNTIME_OWNER = "go2rtc"
+SHARED_GROUP = "frigate-data"
+MODE = 0o664
+MAX_BYTES = 10 * 1024 * 1024
+
+
+def open_nofollow(path: str) -> int:
+    """Return an fd to a regular file at path, never following a symlink."""
+    flags = os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW
+    try:
+        fd = os.open(path, flags, MODE)
+    except OSError as err:
+        if err.errno != errno.ELOOP:
+            raise
+        os.unlink(path)
+        return os.open(path, flags | os.O_EXCL, MODE)
+
+    # A fifo or other non-regular file would hang or misbehave on read; replace it.
+    if not stat.S_ISREG(os.fstat(fd).st_mode):
+        os.close(fd)
+        os.unlink(path)
+        return os.open(path, flags | os.O_EXCL, MODE)
+    return fd
+
+
+def normalize(content: str) -> str:
+    """Keep only the homekit section, matching the previous yq/jq behavior."""
+    yaml = YAML(typ="safe")
+    try:
+        data = yaml.load(content)
+    except Exception:
+        return ""
+
+    if not isinstance(data, dict) or "homekit" not in data:
+        return ""
+
+    buf = io.StringIO()
+    yaml.dump({"homekit": data["homekit"]}, buf)
+    return buf.getvalue()
+
+
+def main() -> int:
+    if len(sys.argv) < 2:
+        print("[ERROR] prepare_homekit: PATH is required", file=sys.stderr)
+        return 2
+
+    path = sys.argv[1]
+    do_chown = "--chown" in sys.argv[2:]
+
+    fd = open_nofollow(path)
+    try:
+        content = os.read(fd, MAX_BYTES).decode("utf-8", "replace")
+        normalized = normalize(content)
+        os.ftruncate(fd, 0)
+        os.lseek(fd, 0, os.SEEK_SET)
+        os.write(fd, normalized.encode("utf-8"))
+
+        if do_chown:
+            # tolerate a chown-refusing mount (NFS root_squash): pairing
+            # persistence degrades, the service does not
+            try:
+                uid = pwd.getpwnam(RUNTIME_OWNER).pw_uid
+                gid = grp.getgrnam(SHARED_GROUP).gr_gid
+                os.fchown(fd, uid, gid)
+                os.fchmod(fd, MODE)
+            except (KeyError, OSError):
+                print(
+                    f"[WARN] Could not hand {path} to the go2rtc user; "
+                    "HomeKit pairing changes may not persist"
+                )
+    finally:
+        os.close(fd)
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -1,9 +1,13 @@
+# Loaded with -c from the /tmp/nginx/conf copy: relative includes follow the -c
+# file, all other path directives follow --prefix and must stay absolute.
+
 daemon off;
+# ignored by a non-root master; keeps workers root under FRIGATE_RUN_AS_ROOT
 user root;
 worker_processes auto;

 error_log /dev/stdout warn;
-pid /var/run/nginx.pid;
+pid /tmp/nginx/nginx.pid;

 events {
    worker_connections 1024;
@@ -13,6 +17,12 @@ http {
    map_hash_bucket_size 256;
    server_tokens off;

+    client_body_temp_path /tmp/nginx/client_body;
+    proxy_temp_path /tmp/nginx/proxy;
+    fastcgi_temp_path /tmp/nginx/fastcgi;
+    uwsgi_temp_path /tmp/nginx/uwsgi;
+    scgi_temp_path /tmp/nginx/scgi;
+
    include mime.types;
    default_type application/octet-stream;