mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-08-01 16:42:18 +03:00
Early 0.18 work (#22138)
* Update version * Create scaffolding for case management (#21293) * implement case management for export apis (#21295) * refactor vainfo to search for first GPU (#21296) use existing LibvaGpuSelector to pick appropritate libva device * Case management UI (#21299) * Refactor export cards to match existing cards in other UI pages * Show cases separately from exports * Add proper filtering and display of cases * Add ability to edit and select cases for exports * Cleanup typing * Hide if no unassigned * Cleanup hiding logic * fix scrolling * Improve layout * Camera connection quality indicator (#21297) * add camera connection quality metrics and indicator * formatting * move stall calcs to watchdog * clean up * change watchdog to 1s and separately track time for ffmpeg retry_interval * implement status caching to reduce message volume * Export filter UI (#21322) * Get started on export filters * implement basic filter * Implement filtering and adjust api * Improve filter handling * Improve navigation * Cleanup * handle scrolling * Refactor temperature reporting for detectors and implement Hailo temp reading (#21395) * Add Hailo temperature retrieval * Refactor `get_hailo_temps()` to use ctxmanager * Show Hailo temps in system UI * Move hailo_platform import to get_hailo_temps * Refactor temperatures calculations to use within detector block * Adjust webUI to handle new location --------- Co-authored-by: tigattack <10629864+tigattack@users.noreply.github.com> * Camera-specific hwaccel settings for timelapse exports (correct base) (#21386) * added hwaccel_args to camera.record.export config struct * populate camera.record.export.hwaccel_args with a cascade up to camera then global if 'auto' * use new hwaccel args in export * added documentation for camera-specific hwaccel export * fix c/p error * missed an import * fleshed out the docs and comments a bit * ruff lint * separated out the tips in the doc * fix documentation * fix and simplify reference config doc * Add support for GPU and NPU temperatures (#21495) * Add rockchip temps * Add support for GPU and NPU temperatures in the frontend * Add support for Nvidia temperature * Improve separation * Adjust graph scaling * Exports Improvements (#21521) * Add images to case folder view * Add ability to select case in export dialog * Add to mobile review too * Add API to handle deleting recordings (#21520) * Add recording delete API * Re-organize recordings apis * Fix import * Consolidate query types * Add media sync API endpoint (#21526) * add media cleanup functions * add endpoint * remove scheduled sync recordings from cleanup * move to utils dir * tweak import * remove sync_recordings and add config migrator * remove sync_recordings * docs * remove key * clean up docs * docs fix * docs tweak * Media sync API refactor and UI (#21542) * generic job infrastructure * types and dispatcher changes for jobs * save data in memory only for completed jobs * implement media sync job and endpoints * change logs to debug * websocket hook and types * frontend * i18n * docs tweaks * endpoint descriptions * tweak docs * use same logging pattern in sync_recordings as the other sync functions (#21625) * Fix incorrect counting in sync_recordings (#21626) * Update go2rtc to v1.9.13 (#21648) Co-authored-by: Eugeny Tulupov <eugeny.tulupov@spirent.com> * Refactor Time-Lapse Export (#21668) * refactor time lapse creation to be a separate API call with ability to pass arbitrary ffmpeg args * Add CPU fallback * Optimize empty directory cleanup for recordings (#21695) The previous empty directory cleanup did a full recursive directory walk, which can be extremely slow. This new implementation only removes directories which have a chance of being empty due to a recent file deletion. * Implement llama.cpp GenAI Provider (#21690) * Implement llama.cpp GenAI Provider * Add docs * Update links * Fix broken mqtt links * Fix more broken anchors * Remove parents in remove_empty_directories (#21726) The original implementation did a full directory tree walk to find and remove empty directories, so this implementation should remove the parents as well, like the original did. * Implement LLM Chat API with tool calling support (#21731) * Implement initial tools definiton APIs * Add initial chat completion API with tool support * Implement other providers * Cleanup * Offline preview image (#21752) * use latest preview frame for latest image when camera is offline * remove frame extraction logic * tests * frontend * add description to api endpoint * Update to ROCm 7.2.0 (#21753) * Update to ROCm 7.2.0 * ROCm now works properly with JinaV1 * Arcface has compilation error * Add live context tool to LLM (#21754) * Add live context tool * Improve handling of images in request * Improve prompt caching * Add networking options for configuring listening ports (#21779) * feat: add X-Frame-Time when returning snapshot (#21932) Co-authored-by: Florent MORICONI <170678386+fmcloudconsulting@users.noreply.github.com> * Improve jsmpeg player websocket handling (#21943) * improve jsmpeg player websocket handling prevent websocket console messages from appearing when player is destroyed * reformat files after ruff upgrade * Allow API Events to be Detections or Alerts, depending on the Event Label (#21923) * - API created events will be alerts OR detections, depending on the event label, defaulting to alerts - Indefinite API events will extend the recording segment until those events are ended - API event start time is the actual start time, instead of having a pre-buffer of record.event_pre_capture * Instead of checking for indefinite events on a camera before deciding if we should end the segment, only update last_detection_time and last_alert_time if frame_time is greater, which should have the same effect * Add the ability to set a pre_capture number of seconds when creating a manual event via the API. Default behavior unchanged * Remove unnecessary _publish_segment_start() call * Formatting * handle last_alert_time or last_detection_time being None when checking them against the frame_time * comment manual_info["label"].split(": ")[0] for clarity * ffmpeg Preview Segment Optimization for "high" and "very_high" (#21996) * Introduce qmax parameter for ffmpeg preview encoding Added PREVIEW_QMAX_PARAM to control ffmpeg encoding quality. * formatting * Fix spacing in qmax parameters for preview quality * Adapt to new Gemini format * Fix frame time access * Remove exceptions * Cleanup --------- Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com> Co-authored-by: tigattack <10629864+tigattack@users.noreply.github.com> Co-authored-by: Andrew Roberts <adroberts@gmail.com> Co-authored-by: Eugeny Tulupov <zhekka3@gmail.com> Co-authored-by: Eugeny Tulupov <eugeny.tulupov@spirent.com> Co-authored-by: John Shaw <1753078+johnshaw@users.noreply.github.com> Co-authored-by: Eric Work <work.eric@gmail.com> Co-authored-by: FL42 <46161216+fl42@users.noreply.github.com> Co-authored-by: Florent MORICONI <170678386+fmcloudconsulting@users.noreply.github.com> Co-authored-by: nulledy <254504350+nulledy@users.noreply.github.com>
This commit is contained in:
co-authored by
Josh Hawkins
tigattack
Andrew Roberts
Eugeny Tulupov
Eugeny Tulupov
John Shaw
Eric Work
FL42
Florent MORICONI
nulledy
parent
7df3622243
commit
d24b96d3bb
+31
-26
@@ -13,9 +13,8 @@ from playhouse.sqlite_ext import SqliteExtDatabase
|
||||
from frigate.config import CameraConfig, FrigateConfig, RetainModeEnum
|
||||
from frigate.const import CACHE_DIR, CLIPS_DIR, MAX_WAL_SIZE, RECORD_DIR
|
||||
from frigate.models import Previews, Recordings, ReviewSegment, UserReviewStatus
|
||||
from frigate.record.util import remove_empty_directories, sync_recordings
|
||||
from frigate.util.builtin import clear_and_unlink
|
||||
from frigate.util.time import get_tomorrow_at_time
|
||||
from frigate.util.media import remove_empty_directories
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -61,7 +60,7 @@ class RecordingCleanup(threading.Thread):
|
||||
db.execute_sql("PRAGMA wal_checkpoint(TRUNCATE);")
|
||||
db.close()
|
||||
|
||||
def expire_review_segments(self, config: CameraConfig, now: datetime) -> None:
|
||||
def expire_review_segments(self, config: CameraConfig, now: datetime) -> set[Path]:
|
||||
"""Delete review segments that are expired"""
|
||||
alert_expire_date = (
|
||||
now - datetime.timedelta(days=config.record.alerts.retain.days)
|
||||
@@ -85,9 +84,12 @@ class RecordingCleanup(threading.Thread):
|
||||
.namedtuples()
|
||||
)
|
||||
|
||||
maybe_empty_dirs = set()
|
||||
thumbs_to_delete = list(map(lambda x: x[1], expired_reviews))
|
||||
for thumb_path in thumbs_to_delete:
|
||||
Path(thumb_path).unlink(missing_ok=True)
|
||||
thumb_path = Path(thumb_path)
|
||||
thumb_path.unlink(missing_ok=True)
|
||||
maybe_empty_dirs.add(thumb_path.parent)
|
||||
|
||||
max_deletes = 100000
|
||||
deleted_reviews_list = list(map(lambda x: x[0], expired_reviews))
|
||||
@@ -100,13 +102,15 @@ class RecordingCleanup(threading.Thread):
|
||||
<< deleted_reviews_list[i : i + max_deletes]
|
||||
).execute()
|
||||
|
||||
return maybe_empty_dirs
|
||||
|
||||
def expire_existing_camera_recordings(
|
||||
self,
|
||||
continuous_expire_date: float,
|
||||
motion_expire_date: float,
|
||||
config: CameraConfig,
|
||||
reviews: ReviewSegment,
|
||||
) -> None:
|
||||
) -> set[Path]:
|
||||
"""Delete recordings for existing camera based on retention config."""
|
||||
# Get the timestamp for cutoff of retained days
|
||||
|
||||
@@ -137,6 +141,8 @@ class RecordingCleanup(threading.Thread):
|
||||
.iterator()
|
||||
)
|
||||
|
||||
maybe_empty_dirs = set()
|
||||
|
||||
# loop over recordings and see if they overlap with any non-expired reviews
|
||||
# TODO: expire segments based on segment stats according to config
|
||||
review_start = 0
|
||||
@@ -191,8 +197,10 @@ class RecordingCleanup(threading.Thread):
|
||||
)
|
||||
or (mode == RetainModeEnum.active_objects and recording.objects == 0)
|
||||
):
|
||||
Path(recording.path).unlink(missing_ok=True)
|
||||
recording_path = Path(recording.path)
|
||||
recording_path.unlink(missing_ok=True)
|
||||
deleted_recordings.add(recording.id)
|
||||
maybe_empty_dirs.add(recording_path.parent)
|
||||
else:
|
||||
kept_recordings.append((recording.start_time, recording.end_time))
|
||||
|
||||
@@ -253,8 +261,10 @@ class RecordingCleanup(threading.Thread):
|
||||
|
||||
# Delete previews without any relevant recordings
|
||||
if not keep:
|
||||
Path(preview.path).unlink(missing_ok=True)
|
||||
preview_path = Path(preview.path)
|
||||
preview_path.unlink(missing_ok=True)
|
||||
deleted_previews.add(preview.id)
|
||||
maybe_empty_dirs.add(preview_path.parent)
|
||||
|
||||
# expire previews
|
||||
logger.debug(f"Expiring {len(deleted_previews)} previews")
|
||||
@@ -266,7 +276,9 @@ class RecordingCleanup(threading.Thread):
|
||||
Previews.id << deleted_previews_list[i : i + max_deletes]
|
||||
).execute()
|
||||
|
||||
def expire_recordings(self) -> None:
|
||||
return maybe_empty_dirs
|
||||
|
||||
def expire_recordings(self) -> set[Path]:
|
||||
"""Delete recordings based on retention config."""
|
||||
logger.debug("Start expire recordings.")
|
||||
logger.debug("Start deleted cameras.")
|
||||
@@ -291,10 +303,14 @@ class RecordingCleanup(threading.Thread):
|
||||
.iterator()
|
||||
)
|
||||
|
||||
maybe_empty_dirs = set()
|
||||
|
||||
deleted_recordings = set()
|
||||
for recording in no_camera_recordings:
|
||||
Path(recording.path).unlink(missing_ok=True)
|
||||
recording_path = Path(recording.path)
|
||||
recording_path.unlink(missing_ok=True)
|
||||
deleted_recordings.add(recording.id)
|
||||
maybe_empty_dirs.add(recording_path.parent)
|
||||
|
||||
logger.debug(f"Expiring {len(deleted_recordings)} recordings")
|
||||
# delete up to 100,000 at a time
|
||||
@@ -311,7 +327,7 @@ class RecordingCleanup(threading.Thread):
|
||||
logger.debug(f"Start camera: {camera}.")
|
||||
now = datetime.datetime.now()
|
||||
|
||||
self.expire_review_segments(config, now)
|
||||
maybe_empty_dirs |= self.expire_review_segments(config, now)
|
||||
continuous_expire_date = (
|
||||
now - datetime.timedelta(days=config.record.continuous.days)
|
||||
).timestamp()
|
||||
@@ -341,7 +357,7 @@ class RecordingCleanup(threading.Thread):
|
||||
.namedtuples()
|
||||
)
|
||||
|
||||
self.expire_existing_camera_recordings(
|
||||
maybe_empty_dirs |= self.expire_existing_camera_recordings(
|
||||
continuous_expire_date, motion_expire_date, config, reviews
|
||||
)
|
||||
logger.debug(f"End camera: {camera}.")
|
||||
@@ -349,12 +365,9 @@ class RecordingCleanup(threading.Thread):
|
||||
logger.debug("End all cameras.")
|
||||
logger.debug("End expire recordings.")
|
||||
|
||||
def run(self) -> None:
|
||||
# on startup sync recordings with disk if enabled
|
||||
if self.config.record.sync_recordings:
|
||||
sync_recordings(limited=False)
|
||||
next_sync = get_tomorrow_at_time(3)
|
||||
return maybe_empty_dirs
|
||||
|
||||
def run(self) -> None:
|
||||
# Expire tmp clips every minute, recordings and clean directories every hour.
|
||||
for counter in itertools.cycle(range(self.config.record.expire_interval)):
|
||||
if self.stop_event.wait(60):
|
||||
@@ -363,16 +376,8 @@ class RecordingCleanup(threading.Thread):
|
||||
|
||||
self.clean_tmp_previews()
|
||||
|
||||
if (
|
||||
self.config.record.sync_recordings
|
||||
and datetime.datetime.now().astimezone(datetime.timezone.utc)
|
||||
> next_sync
|
||||
):
|
||||
sync_recordings(limited=True)
|
||||
next_sync = get_tomorrow_at_time(3)
|
||||
|
||||
if counter == 0:
|
||||
self.clean_tmp_clips()
|
||||
self.expire_recordings()
|
||||
remove_empty_directories(RECORD_DIR)
|
||||
maybe_empty_dirs = self.expire_recordings()
|
||||
remove_empty_directories(Path(RECORD_DIR), maybe_empty_dirs)
|
||||
self.truncate_wal()
|
||||
|
||||
+90
-38
@@ -33,6 +33,7 @@ from frigate.util.time import is_current_hour
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
DEFAULT_TIME_LAPSE_FFMPEG_ARGS = "-vf setpts=0.04*PTS -r 30"
|
||||
TIMELAPSE_DATA_INPUT_ARGS = "-an -skip_frame nokey"
|
||||
|
||||
|
||||
@@ -40,11 +41,6 @@ def lower_priority():
|
||||
os.nice(PROCESS_PRIORITY_LOW)
|
||||
|
||||
|
||||
class PlaybackFactorEnum(str, Enum):
|
||||
realtime = "realtime"
|
||||
timelapse_25x = "timelapse_25x"
|
||||
|
||||
|
||||
class PlaybackSourceEnum(str, Enum):
|
||||
recordings = "recordings"
|
||||
preview = "preview"
|
||||
@@ -62,8 +58,11 @@ class RecordingExporter(threading.Thread):
|
||||
image: Optional[str],
|
||||
start_time: int,
|
||||
end_time: int,
|
||||
playback_factor: PlaybackFactorEnum,
|
||||
playback_source: PlaybackSourceEnum,
|
||||
export_case_id: Optional[str] = None,
|
||||
ffmpeg_input_args: Optional[str] = None,
|
||||
ffmpeg_output_args: Optional[str] = None,
|
||||
cpu_fallback: bool = False,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self.config = config
|
||||
@@ -73,8 +72,11 @@ class RecordingExporter(threading.Thread):
|
||||
self.user_provided_image = image
|
||||
self.start_time = start_time
|
||||
self.end_time = end_time
|
||||
self.playback_factor = playback_factor
|
||||
self.playback_source = playback_source
|
||||
self.export_case_id = export_case_id
|
||||
self.ffmpeg_input_args = ffmpeg_input_args
|
||||
self.ffmpeg_output_args = ffmpeg_output_args
|
||||
self.cpu_fallback = cpu_fallback
|
||||
|
||||
# ensure export thumb dir
|
||||
Path(os.path.join(CLIPS_DIR, "export")).mkdir(exist_ok=True)
|
||||
@@ -179,9 +181,16 @@ class RecordingExporter(threading.Thread):
|
||||
|
||||
return thumb_path
|
||||
|
||||
def get_record_export_command(self, video_path: str) -> list[str]:
|
||||
def get_record_export_command(
|
||||
self, video_path: str, use_hwaccel: bool = True
|
||||
) -> list[str]:
|
||||
# handle case where internal port is a string with ip:port
|
||||
internal_port = self.config.networking.listen.internal
|
||||
if type(internal_port) is str:
|
||||
internal_port = int(internal_port.split(":")[-1])
|
||||
|
||||
if (self.end_time - self.start_time) <= MAX_PLAYLIST_SECONDS:
|
||||
playlist_lines = f"http://127.0.0.1:5000/vod/{self.camera}/start/{self.start_time}/end/{self.end_time}/index.m3u8"
|
||||
playlist_lines = f"http://127.0.0.1:{internal_port}/vod/{self.camera}/start/{self.start_time}/end/{self.end_time}/index.m3u8"
|
||||
ffmpeg_input = (
|
||||
f"-y -protocol_whitelist pipe,file,http,tcp -i {playlist_lines}"
|
||||
)
|
||||
@@ -213,25 +222,30 @@ class RecordingExporter(threading.Thread):
|
||||
for page in range(1, num_pages + 1):
|
||||
playlist = export_recordings.paginate(page, page_size)
|
||||
playlist_lines.append(
|
||||
f"file 'http://127.0.0.1:5000/vod/{self.camera}/start/{float(playlist[0].start_time)}/end/{float(playlist[-1].end_time)}/index.m3u8'"
|
||||
f"file 'http://127.0.0.1:{internal_port}/vod/{self.camera}/start/{float(playlist[0].start_time)}/end/{float(playlist[-1].end_time)}/index.m3u8'"
|
||||
)
|
||||
|
||||
ffmpeg_input = "-y -protocol_whitelist pipe,file,http,tcp -f concat -safe 0 -i /dev/stdin"
|
||||
|
||||
if self.playback_factor == PlaybackFactorEnum.realtime:
|
||||
ffmpeg_cmd = (
|
||||
f"{self.config.ffmpeg.ffmpeg_path} -hide_banner {ffmpeg_input} -c copy -movflags +faststart"
|
||||
).split(" ")
|
||||
elif self.playback_factor == PlaybackFactorEnum.timelapse_25x:
|
||||
if self.ffmpeg_input_args is not None and self.ffmpeg_output_args is not None:
|
||||
hwaccel_args = (
|
||||
self.config.cameras[self.camera].record.export.hwaccel_args
|
||||
if use_hwaccel
|
||||
else None
|
||||
)
|
||||
ffmpeg_cmd = (
|
||||
parse_preset_hardware_acceleration_encode(
|
||||
self.config.ffmpeg.ffmpeg_path,
|
||||
self.config.ffmpeg.hwaccel_args,
|
||||
f"-an {ffmpeg_input}",
|
||||
f"{self.config.cameras[self.camera].record.export.timelapse_args} -movflags +faststart",
|
||||
hwaccel_args,
|
||||
f"{self.ffmpeg_input_args} -an {ffmpeg_input}".strip(),
|
||||
f"{self.ffmpeg_output_args} -movflags +faststart".strip(),
|
||||
EncodeTypeEnum.timelapse,
|
||||
)
|
||||
).split(" ")
|
||||
else:
|
||||
ffmpeg_cmd = (
|
||||
f"{self.config.ffmpeg.ffmpeg_path} -hide_banner {ffmpeg_input} -c copy -movflags +faststart"
|
||||
).split(" ")
|
||||
|
||||
# add metadata
|
||||
title = f"Frigate Recording for {self.camera}, {self.get_datetime_from_timestamp(self.start_time)} - {self.get_datetime_from_timestamp(self.end_time)}"
|
||||
@@ -241,7 +255,9 @@ class RecordingExporter(threading.Thread):
|
||||
|
||||
return ffmpeg_cmd, playlist_lines
|
||||
|
||||
def get_preview_export_command(self, video_path: str) -> list[str]:
|
||||
def get_preview_export_command(
|
||||
self, video_path: str, use_hwaccel: bool = True
|
||||
) -> list[str]:
|
||||
playlist_lines = []
|
||||
codec = "-c copy"
|
||||
|
||||
@@ -309,20 +325,25 @@ class RecordingExporter(threading.Thread):
|
||||
"-y -protocol_whitelist pipe,file,tcp -f concat -safe 0 -i /dev/stdin"
|
||||
)
|
||||
|
||||
if self.playback_factor == PlaybackFactorEnum.realtime:
|
||||
ffmpeg_cmd = (
|
||||
f"{self.config.ffmpeg.ffmpeg_path} -hide_banner {ffmpeg_input} {codec} -movflags +faststart {video_path}"
|
||||
).split(" ")
|
||||
elif self.playback_factor == PlaybackFactorEnum.timelapse_25x:
|
||||
if self.ffmpeg_input_args is not None and self.ffmpeg_output_args is not None:
|
||||
hwaccel_args = (
|
||||
self.config.cameras[self.camera].record.export.hwaccel_args
|
||||
if use_hwaccel
|
||||
else None
|
||||
)
|
||||
ffmpeg_cmd = (
|
||||
parse_preset_hardware_acceleration_encode(
|
||||
self.config.ffmpeg.ffmpeg_path,
|
||||
self.config.ffmpeg.hwaccel_args,
|
||||
f"{TIMELAPSE_DATA_INPUT_ARGS} {ffmpeg_input}",
|
||||
f"{self.config.cameras[self.camera].record.export.timelapse_args} -movflags +faststart {video_path}",
|
||||
hwaccel_args,
|
||||
f"{self.ffmpeg_input_args} {TIMELAPSE_DATA_INPUT_ARGS} {ffmpeg_input}".strip(),
|
||||
f"{self.ffmpeg_output_args} -movflags +faststart {video_path}".strip(),
|
||||
EncodeTypeEnum.timelapse,
|
||||
)
|
||||
).split(" ")
|
||||
else:
|
||||
ffmpeg_cmd = (
|
||||
f"{self.config.ffmpeg.ffmpeg_path} -hide_banner {ffmpeg_input} {codec} -movflags +faststart {video_path}"
|
||||
).split(" ")
|
||||
|
||||
# add metadata
|
||||
title = f"Frigate Preview for {self.camera}, {self.get_datetime_from_timestamp(self.start_time)} - {self.get_datetime_from_timestamp(self.end_time)}"
|
||||
@@ -348,17 +369,20 @@ class RecordingExporter(threading.Thread):
|
||||
video_path = f"{EXPORT_DIR}/{self.camera}_{filename_start_datetime}-{filename_end_datetime}_{cleaned_export_id}.mp4"
|
||||
thumb_path = self.save_thumbnail(self.export_id)
|
||||
|
||||
Export.insert(
|
||||
{
|
||||
Export.id: self.export_id,
|
||||
Export.camera: self.camera,
|
||||
Export.name: export_name,
|
||||
Export.date: self.start_time,
|
||||
Export.video_path: video_path,
|
||||
Export.thumb_path: thumb_path,
|
||||
Export.in_progress: True,
|
||||
}
|
||||
).execute()
|
||||
export_values = {
|
||||
Export.id: self.export_id,
|
||||
Export.camera: self.camera,
|
||||
Export.name: export_name,
|
||||
Export.date: self.start_time,
|
||||
Export.video_path: video_path,
|
||||
Export.thumb_path: thumb_path,
|
||||
Export.in_progress: True,
|
||||
}
|
||||
|
||||
if self.export_case_id is not None:
|
||||
export_values[Export.export_case] = self.export_case_id
|
||||
|
||||
Export.insert(export_values).execute()
|
||||
|
||||
try:
|
||||
if self.playback_source == PlaybackSourceEnum.recordings:
|
||||
@@ -376,6 +400,34 @@ class RecordingExporter(threading.Thread):
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
# If export failed and cpu_fallback is enabled, retry without hwaccel
|
||||
if (
|
||||
p.returncode != 0
|
||||
and self.cpu_fallback
|
||||
and self.ffmpeg_input_args is not None
|
||||
and self.ffmpeg_output_args is not None
|
||||
):
|
||||
logger.warning(
|
||||
f"Export with hardware acceleration failed, retrying without hwaccel for {self.export_id}"
|
||||
)
|
||||
|
||||
if self.playback_source == PlaybackSourceEnum.recordings:
|
||||
ffmpeg_cmd, playlist_lines = self.get_record_export_command(
|
||||
video_path, use_hwaccel=False
|
||||
)
|
||||
else:
|
||||
ffmpeg_cmd, playlist_lines = self.get_preview_export_command(
|
||||
video_path, use_hwaccel=False
|
||||
)
|
||||
|
||||
p = sp.run(
|
||||
ffmpeg_cmd,
|
||||
input="\n".join(playlist_lines),
|
||||
encoding="ascii",
|
||||
preexec_fn=lower_priority,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
if p.returncode != 0:
|
||||
logger.error(
|
||||
f"Failed to export {self.playback_source.value} for command {' '.join(ffmpeg_cmd)}"
|
||||
|
||||
@@ -1,147 +0,0 @@
|
||||
"""Recordings Utilities."""
|
||||
|
||||
import datetime
|
||||
import logging
|
||||
import os
|
||||
|
||||
from peewee import DatabaseError, chunked
|
||||
|
||||
from frigate.const import RECORD_DIR
|
||||
from frigate.models import Recordings, RecordingsToDelete
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def remove_empty_directories(directory: str) -> None:
|
||||
# list all directories recursively and sort them by path,
|
||||
# longest first
|
||||
paths = sorted(
|
||||
[x[0] for x in os.walk(directory)],
|
||||
key=lambda p: len(str(p)),
|
||||
reverse=True,
|
||||
)
|
||||
for path in paths:
|
||||
# don't delete the parent
|
||||
if path == directory:
|
||||
continue
|
||||
if len(os.listdir(path)) == 0:
|
||||
os.rmdir(path)
|
||||
|
||||
|
||||
def sync_recordings(limited: bool) -> None:
|
||||
"""Check the db for stale recordings entries that don't exist in the filesystem."""
|
||||
|
||||
def delete_db_entries_without_file(check_timestamp: float) -> bool:
|
||||
"""Delete db entries where file was deleted outside of frigate."""
|
||||
|
||||
if limited:
|
||||
recordings = Recordings.select(Recordings.id, Recordings.path).where(
|
||||
Recordings.start_time >= check_timestamp
|
||||
)
|
||||
else:
|
||||
# get all recordings in the db
|
||||
recordings = Recordings.select(Recordings.id, Recordings.path)
|
||||
|
||||
# Use pagination to process records in chunks
|
||||
page_size = 1000
|
||||
num_pages = (recordings.count() + page_size - 1) // page_size
|
||||
recordings_to_delete = set()
|
||||
|
||||
for page in range(num_pages):
|
||||
for recording in recordings.paginate(page, page_size):
|
||||
if not os.path.exists(recording.path):
|
||||
recordings_to_delete.add(recording.id)
|
||||
|
||||
if len(recordings_to_delete) == 0:
|
||||
return True
|
||||
|
||||
logger.info(
|
||||
f"Deleting {len(recordings_to_delete)} recording DB entries with missing files"
|
||||
)
|
||||
|
||||
# convert back to list of dictionaries for insertion
|
||||
recordings_to_delete = [
|
||||
{"id": recording_id} for recording_id in recordings_to_delete
|
||||
]
|
||||
|
||||
if float(len(recordings_to_delete)) / max(1, recordings.count()) > 0.5:
|
||||
logger.warning(
|
||||
f"Deleting {(len(recordings_to_delete) / max(1, recordings.count()) * 100):.2f}% of recordings DB entries, could be due to configuration error. Aborting..."
|
||||
)
|
||||
return False
|
||||
|
||||
# create a temporary table for deletion
|
||||
RecordingsToDelete.create_table(temporary=True)
|
||||
|
||||
# insert ids to the temporary table
|
||||
max_inserts = 1000
|
||||
for batch in chunked(recordings_to_delete, max_inserts):
|
||||
RecordingsToDelete.insert_many(batch).execute()
|
||||
|
||||
try:
|
||||
# delete records in the main table that exist in the temporary table
|
||||
query = Recordings.delete().where(
|
||||
Recordings.id.in_(RecordingsToDelete.select(RecordingsToDelete.id))
|
||||
)
|
||||
query.execute()
|
||||
except DatabaseError as e:
|
||||
logger.error(f"Database error during recordings db cleanup: {e}")
|
||||
|
||||
return True
|
||||
|
||||
def delete_files_without_db_entry(files_on_disk: list[str]):
|
||||
"""Delete files where file is not inside frigate db."""
|
||||
files_to_delete = []
|
||||
|
||||
for file in files_on_disk:
|
||||
if not Recordings.select().where(Recordings.path == file).exists():
|
||||
files_to_delete.append(file)
|
||||
|
||||
if len(files_to_delete) == 0:
|
||||
return True
|
||||
|
||||
logger.info(
|
||||
f"Deleting {len(files_to_delete)} recordings files with missing DB entries"
|
||||
)
|
||||
|
||||
if float(len(files_to_delete)) / max(1, len(files_on_disk)) > 0.5:
|
||||
logger.debug(
|
||||
f"Deleting {(len(files_to_delete) / max(1, len(files_on_disk)) * 100):.2f}% of recordings DB entries, could be due to configuration error. Aborting..."
|
||||
)
|
||||
return False
|
||||
|
||||
for file in files_to_delete:
|
||||
os.unlink(file)
|
||||
|
||||
return True
|
||||
|
||||
logger.debug("Start sync recordings.")
|
||||
|
||||
# start checking on the hour 36 hours ago
|
||||
check_point = datetime.datetime.now().replace(
|
||||
minute=0, second=0, microsecond=0
|
||||
).astimezone(datetime.timezone.utc) - datetime.timedelta(hours=36)
|
||||
db_success = delete_db_entries_without_file(check_point.timestamp())
|
||||
|
||||
# only try to cleanup files if db cleanup was successful
|
||||
if db_success:
|
||||
if limited:
|
||||
# get recording files from last 36 hours
|
||||
hour_check = f"{RECORD_DIR}/{check_point.strftime('%Y-%m-%d/%H')}"
|
||||
files_on_disk = {
|
||||
os.path.join(root, file)
|
||||
for root, _, files in os.walk(RECORD_DIR)
|
||||
for file in files
|
||||
if root > hour_check
|
||||
}
|
||||
else:
|
||||
# get all recordings files on disk and put them in a set
|
||||
files_on_disk = {
|
||||
os.path.join(root, file)
|
||||
for root, _, files in os.walk(RECORD_DIR)
|
||||
for file in files
|
||||
}
|
||||
|
||||
delete_files_without_db_entry(files_on_disk)
|
||||
|
||||
logger.debug("End sync recordings.")
|
||||
Reference in New Issue
Block a user