feat(hang): define segment-addressed recordings - #2574
Conversation
moq-archive specified a generic, arrival-ordered chunk format at the moq-lite layer, with its own per-track index mapping group runs to byte ranges. It predated the timeline rework, which already publishes a per-segment index of group ranges per track. Addressing a recording by segment instead of by flush interval collapses the format: the segment number names the object, so the timeline is the only index and the separate index log, chunk numbering, byte offsets, partial spans, and overlap resolution all disappear. It also gives a reader one whole-object GET per segment per track, which byte-range splicing across chunks could not. That makes the format a hang concept rather than a moq-lite one, since segments are defined by the timeline and the timeline is hang. Specify it in the hang draft and delete the standalone draft, which was never submitted to the datatracker. Segment objects carry their own group boundaries so they parse without the timeline, and frames stay byte-identical to moq-lite FRAME so a recorded group is a FETCH response body unchanged. Known trade: a group arriving after its segment object is written is not recorded. Writers wait for completeness and bound that wait. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Cxx2ebpiGmWaD8AvJkRdi
The segmenter required every enrolled track to vote on boundaries and to report past a segment's end before its record could flush. That works for audio and video, whose groups arrive continuously, and stalls the timeline permanently for anything else: boundary() returns None the moment an enrolled track has no group past the threshold, and a catalog publishes a group only when the renditions change. So a broadcast could only segment tracks it could also pace, which left catalogs, metadata, and application tracks like a chat log with no way to be indexed at all. Add Producer::passive alongside Producer::track. A passive track's groups are recorded into whichever segment is open when they arrive, but it never votes on a boundary, never gates completeness, and never anchors the first segment (a catalog published while the encoder warms up would otherwise stretch segment 0 across the whole startup gap). It also does not create the timeline track, so segmentation stays opt-in by pacing track. Placement is by arrival rather than content time, since nothing waits for these groups: one that shows up after its segment flushed is recorded in the next. The frames still carry their own timestamps. A passive track whose group never closes is recorded once, in the segment its group opened in. Rolling the group at segment boundaries is the fix and is left to the publisher. Recording drops its special case for the catalog as a result: every track the timeline lists is stored the same way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Cxx2ebpiGmWaD8AvJkRdi
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ed2fe6e74
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ~~~ | ||
| <prefix>/.timeline | ||
| <prefix>/<track>/<segment> | ||
| ~~~ |
There was a problem hiding this comment.
Persisting only .timeline and per-track segment objects makes a recording impossible to decode or replay after the live broadcast disappears. The stored FRAME timestamps are deltas in the track's negotiated timescale, and moq-lite requires TRACK_INFO before parsing them (drafts/draft-lcurley-moq-lite.md:1180-1182 and 1050-1054), but this layout stores neither the timescale nor the other immutable properties needed to reconstruct TRACK_INFO. Add per-track metadata to the recording layout so readers can interpret frames and serve the recorded tracks. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
| pub fn passive(&self, name: &str) -> crate::Result<Recorder> { | ||
| self.enroll(name, true) |
There was a problem hiding this comment.
This only marks the track passive for boundary and completeness calculations. flush_segment still stops draining it when group_pts >= end, and the terminal duration still takes the maximum frontier across all tracks; the JS implementation mirrors both behaviors. When passive metadata uses a future or different timestamp basis, such as catalog snapshots stamped with Timestamp::now() alongside file media starting at PTS 0, an update that arrived during an open segment remains pending until the final flush and can inflate or fail the final segment. Drain all pending passive groups at each flush and exclude passive frontiers from terminal duration. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
| Segment objects are immutable and SHOULD be served with long-lived caching. | ||
| `.timeline` and `.catalog` change while the recording is live and SHOULD be served with short lifetimes; all three become immutable once the recording has ended. |
There was a problem hiding this comment.
.catalog object
The layout defines only .timeline and immutable <track>/<segment> objects, with catalog generations stored as ordinary segment objects under the encoded catalog track name. Consequently there is no .catalog object that changes during a live recording, and catalog segment objects should receive the long-lived caching described immediately above. This instruction can make implementations request or publish an undefined resource and apply the wrong cache policy. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Persist immutable track metadata alongside per-track segment objects, and specify archive and duration-bounded DVR retention with the recording timeline as the authoritative durable index. Fix passive timeline tracks to drain by arrival and exclude their timestamps from final media duration. Add matching Rust and TypeScript regression coverage. Co-Authored-By: OpenAI Codex <noreply@openai.com>
kixelated
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c8dec4b58d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pub fn passive(&self, name: &str) -> crate::Result<Recorder> { | ||
| self.enroll(name, true) |
There was a problem hiding this comment.
The new recording layout relies on catalog groups appearing in timeline records, but the production catalog::Producer never calls this method or reports its snapshot groups through the returned recorder. A repo-wide search at this commit finds passive() used only in timeline tests, while rs/moq-mux/src/catalog/producer.rs continues to emit catalog.json directly. Consequently, recordings produced through the normal Rust catalog/media path have no catalog segment objects and cannot recover the rendition and decoder configuration after the live catalog disappears. Wire the built-in catalog publisher to a passive recorder rather than only exposing the primitive. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
| If any object for a segment cannot be made durable, the writer MUST record an atomic gap for that segment rather than publish a partial set of tracks. | ||
| The gap is the same timeline record with `tracks` absent or empty, so it preserves the segment number and timing but references no objects. |
There was a problem hiding this comment.
When an object write fails in an unbounded archive, this rule requires replacing the source record with a modified gap record, contradicting the earlier requirement that .timeline contain the source frames verbatim. More importantly, later source frames were DEFLATE-compressed against the original record through the shared window, so appending them after a rewritten gap makes readers decompress against a different dictionary and corrupts the remaining timeline. Specify that the writer must re-encode the gap and every subsequent record, or encode the failure out of band. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
| `<track>` is the track's name with every byte outside `A-Z a-z 0-9 _ -` percent-encoded. | ||
| An encoded name therefore never contains `/` and never begins with `.`, so a track can neither collide with the reserved `.timeline` name nor address anything outside the prefix. |
There was a problem hiding this comment.
This does not specify the hexadecimal letter case for percent encoding, so two compliant implementations can derive different object keys for the same track. For example, the built-in catalog.json name can become either catalog%2Ejson or catalog%2ejson, which are distinct on case-sensitive filesystems and object stores; an independently implemented reader can therefore return 404 for every object written by another implementation. Require one canonical hex case for encoded bytes. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
Enroll catalog.json as a passive timeline track and report each published snapshot group so durable recordings include the catalog needed to interpret media segments. Specify uppercase percent escapes and require recording-owned timeline re-encoding so atomic gaps do not corrupt the shared DEFLATE dictionary. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Make ordinary timeline track enrollment non-pacing so sparse catalog or metadata tracks cannot accidentally stall segment publication. Add explicit pacing_track and pacingTrack methods for continuous media that votes on boundaries and gates completeness. BREAKING CHANGE: Rust timeline Producer::track is now infallible and non-pacing, and TypeScript Producer.track is now non-pacing. Use pacing_track or pacingTrack for media tracks that drive segmentation. Co-Authored-By: OpenAI Codex <noreply@openai.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ddaf604060
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| A writer SHOULD wait for the segment to be complete rather than write early, and MAY bound that wait so a track that has stopped without closing cannot stall the recording indefinitely. | ||
| This is a deliberate trade: addressing content by segment is what makes a segment retrievable in one request, and it costs the ability to append a late group to an object already written. | ||
|
|
||
| A writer SHOULD record the broadcast's final state on a clean end, so a reader can distinguish an ended recording from one whose writer died. |
There was a problem hiding this comment.
Define an on-disk representation for this final state. After either a clean shutdown or a writer crash, the recording currently contains the same last sync-flushed timeline record and immutable segment objects; merely observing that .timeline has stopped changing cannot distinguish those cases. Without a terminal marker or status object, readers cannot implement the stated distinction or know when the archive is safe to cache as immutable. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L96-L100
Useful? React with 👍 / 👎.
2 participants
Summary
.timeline, immutable per-track.trackmetadata, and one object per track and segment.The root causes were three mismatches in the original proposal. A recorded segment retained FRAME timestamp deltas without persisting the track timescale and ordering needed to reconstruct
TRACK_INFO. Non-pacing tracks were documented as arrival-ordered but flushed using media timestamps, so future-based metadata could be left behind and could inflate the final segment duration. The low-level enrollment primitive was only exercised directly by tests, so the normal catalog producer never reported its published groups to the timeline.This follows the durable Recording/DVR direction in #2275 and #2281, plus moq-dev/moq.pro#807. The old standalone archive format and relay-RAM-derived retention are not used.
Public API changes
moq_mux::timeline::Producer::trackis now infallible and enrolls a non-pacing track.moq_mux::timeline::Producer::pacing_trackfor continuous media that votes on boundaries and gates completeness.@moq/hangtimelineProducer.trackis now non-pacing.Producer.pacingTrackfor continuous media.This is a semantic break to the existing low-level timeline API, so the existing
devtarget is appropriate. High-level catalog and media producers select the correct mode internally.Test plan
nix develop --command just fixnix develop --command just drafts checknix develop --command just checknix develop --command just testThe Rust/TypeScript timeline implementations and regression tests are kept in sync. No other cross-package sync row applies.
(Written by GPT-5)