Skip to content

recording-daemon: lifecycle notification events#2142

Open
goolanceman wants to merge 2 commits into
sipwise:masterfrom
goolanceman:recording-notify-lifecycle
Open

recording-daemon: lifecycle notification events#2142
goolanceman wants to merge 2 commits into
sipwise:masterfrom
goolanceman:recording-notify-lifecycle

Conversation

@goolanceman

@goolanceman goolanceman commented Jul 18, 2026

Copy link
Copy Markdown

Summary

Extends rtpengine-recording notifications from the historical finished-only path to a configurable call/stream lifecycle event set, while remaining backward compatible.

Events (notify-events CSV)

Token Event name
opened recording_file_opened
started recording_started
finished (default) recording_finished
discarded recording_discarded
failed recording_failed
call-started call_recording_started
call-finished call_recording_finished
call-discarded call_recording_discarded
all all of the above

Default remains finished only so existing deployments are unchanged.

New options

  • notify-events — CSV list (default finished)
  • notify-json — POST JSON body via json-glib
  • notify-no-metadata — omit call metadata from payloads
  • notify-command-formatlegacy | extended | json-env
  • notify-queue-limit — bound non-terminal queue depth (default 1000; 0 = unlimited). Terminal events are always accepted.

Design

  • Keeps upstream action-based notify model (notif_action_t / notif_req_t) and lib/http.c helpers
  • Pure event helpers in notify_events.c (shared with unit tests)
  • Stream hooks in output.c; call hooks in metafile.c / packet.c
  • Snapshots strings into notif_req before queueing (workers never touch live objects)
  • Metrics logged on shutdown: enqueued / success / retry / giveup / dropped

Tests

  • Unit: t/test-notify-events.c — PASS
  • E2E smoke: t/notify_e2e/ temp Python HTTP receiver + metafile lifecycle — E2E_NOTIFY_OK
    • Built/ran in Docker
    • Received call_recording_finished with JSON body + lifecycle headers

Docs

  • docs/rtpengine-recording.md
  • etc/rtpengine-recording.conf

Example

notify-uri = https://example.com/rec/events
notify-events = opened,started,finished,discarded,failed,call-started,call-finished,call-discarded
notify-json = true
notify-concurrency = 5
notify-retries = 10

Compatibility

  • Default config behavior is identical to previous finished-only notifications
  • Legacy notify-command argv (path, db_id) preserved when notify-command-format=legacy

Reviewer notes / BC details

Scope

  • Code changes are limited to the recording daemon notify stack, its hooks, docs/conf, and tests.
  • S3/GCS/DB storage code (s3.c, gcs.c, db.c) is not modified; these still use the shared notif_action_t pool.

Thread pool / storage safety

  • notify_setup() now only gates on notify_threads > 0 and always starts the shared thread pool when enabled.
  • This matches current master semantics and ensures S3/GCS/DB notif_action_t users continue to work even when notify-uri / notify-command are not configured.

Memory safety

  • req_free_fields() now only frees lifecycle snapshot strings (call/output/tag/error/JSON), leaving action-owned fields (headers, argv/envp, content, object_name, etc.) to the relevant action->cleanup().
  • This avoids a potential double-free of req->object_name on the S3/GCS paths, which already free it in their own cleanup.

Finished-only backward compatibility

  • With default settings, only the historical finished event is emitted (via the new event mask helpers).
  • Finished notifications keep the legacy notify-command argv layout in legacy mode.
  • HTTP notifications now add a few extra headers (X-Recording-Event, X-Recording-Status, X-Recording-Event-Time, plus media/size/duration where available) and slightly richer X-Recording-Full-File-Name semantics (prefers the actual on-disk filename where available).
  • These are additive and intended to be safe for existing consumers that ignore unknown headers, but are called out here for reviewers in case any downstream integration relies on a byte-for-byte header set or reconstructs paths from Full-File-Name.

@goolanceman
goolanceman marked this pull request as ready for review July 18, 2026 17:53
Extend rtpengine-recording HTTP/command notifications beyond the
historical finished-only path so operators can track call and stream
recording lifecycle.

Events (notify-events CSV, default: finished):
  opened, started, finished, discarded, failed,
  call-started, call-finished, call-discarded

New options: notify-events, notify-json, notify-no-metadata,
notify-command-format, notify-queue-limit.

Docs/conf only document the new notify options; upstream storage
(S3/GCS/output-storage) docs are left unchanged.

Backward compatible: default mask is finished only.
Includes unit tests and Docker e2e smoke harness.
@goolanceman
goolanceman force-pushed the recording-notify-lifecycle branch from a4ccba2 to fd097da Compare July 18, 2026 18:10
notify_setup must not require notify-uri/notify-command: the same
thread pool backs S3, GCS and DB notif_action_t work. Also leave
object_name to action cleanup to avoid double-free on those paths.

@rfuchs rfuchs left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all AI generated, isn't it

Comment thread recording-daemon/notify.c
Comment on lines -150 to 234

static void *notify_timer(void *p) {
pthread_mutex_lock(&timer_lock);

// notify_timers being NULL acts as our shutdown flag
while (notify_timers) {
ilog(LOG_DEBUG, "Notification timer thread looping");

// grab first entry in list, check retry time, sleep if it's in the future

notif_req_t *first = rtpe_g_tree_first(notify_timers);
if (!first) {
ilog(LOG_DEBUG, "No scheduled notification retries, sleeping");
pthread_cond_wait(&timer_cond, &timer_lock);
continue;
}
int64_t now = now_us();
if (now < first->retry_time) {
ilog(LOG_DEBUG, "Sleeping until next scheduled notification retry in %" PRId64 " seconds",
(first->retry_time - now) / 1000000L);
cond_timedwait(&timer_cond, &timer_lock, first->retry_time);
continue;
}

// first entry is ready to run

g_tree_remove(notify_timers, first);
ilog(LOG_DEBUG, "Notification retry for '%s%s%s' is scheduled now", FMT_M(first->name));
g_thread_pool_push(notify_threadpool, first, NULL);
}

// clean up

pthread_mutex_unlock(&timer_lock);
pthread_mutex_destroy(&timer_lock);
pthread_cond_destroy(&timer_cond);

return NULL;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the debug log messages and the existing comments?

Comment thread recording-daemon/notify.c
Comment on lines -193 to 242

if (a->retry_time < b->retry_time)
return -1;
if (a->retry_time > b->retry_time)
return 1;
if (a < b)
return -1;
if (a > b)
return 1;
if (a->retry_time < b->retry_time) return -1;
if (a->retry_time > b->retry_time) return 1;
if (a < b) return -1;
if (a > b) return 1;
return 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't lump in reformats in the same commit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants