Add timestamp-paced replay (replay_dbn / replay_records)#43
Conversation
Re-emit DBN records in time order, paced by their timestamps to simulate a live feed (backtesting, demos, driving real-time consumers). - replay_dbn(f, filename; ...) streams from a file (Zstd-aware, skips unknown rtypes like DBNStream) and invokes f(record) paced in real time. - replay_records(f, records; ...) does the same over an in-memory collection. - Wall-clock-anchored pacing absorbs callback execution time instead of accumulating drift; the stream re-synchronizes after slow callbacks. Options: speed (time-compression multiplier; Inf = no waiting), timestamp (:ts_event or :ts_recv with fallback), max_sleep (cap large gaps, re-anchors), and precise (busy-wait sub-millisecond gaps for dense-burst fidelity, since Base.sleep only resolves ~1ms on Unix / ~15ms on Windows). clock/sleep_fn are injectable for deterministic testing. Adds test/test_replay.jl (35 tests, fake-clock deterministic pacing checks + real-clock precise-mode checks) and documents the timing-resolution limit in the docstrings and README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump version 0.1.5 -> 0.1.6 (additive, non-breaking: new replay_dbn / replay_records exports only). Version the changelog's Unreleased section as 0.1.6 and document the replay feature under Added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 60f7885c63
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| deadline = time() + s | ||
| if s > _SPIN_THRESHOLD | ||
| sleep(s - _SPIN_THRESHOLD) # yield the bulk to the scheduler | ||
| end | ||
| while time() < deadline # spin the residual (≤ _SPIN_THRESHOLD) |
There was a problem hiding this comment.
Use a monotonic clock for precise sleeps
When precise = true, this deadline is based on Base.time(), which is wall-clock time rather than a monotonic clock. If the system clock is adjusted while replay is spinning (for example by NTP or manual correction), _precise_sleep can return early or busy-wait far longer than requested, distorting replay timing and potentially pinning a CPU core; use a monotonic source such as time_ns() for the deadline instead.
Useful? React with 👍 / 👎.
_precise_sleep measured its busy-wait deadline against wall-clock time(); a system-clock adjustment (NTP, manual) mid-spin could make it return early or busy-wait far longer than requested (pinning a core). Measure the deadline with time_ns() (monotonic) instead. Addresses Codex PR review (P2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Adds "replay" functionality: re-emit a DBN file's records in time order, paced by their timestamps to simulate a live feed. Useful for backtesting, demos, and driving consumers that expect a real-time stream. This was previously absent (the package only had unpaced streaming via
DBNStream/foreach_record).API
replay_dbn(f, filename; ...)— streams from a file (Zstd-aware, skips unknown rtypes exactly likeDBNStream) and invokesf(record)paced in real time. Returns the record count.replay_records(f, records; ...)— same pacing over an already-loaded collection (e.g. fromread_dbn).Options
speed— time-compression multiplier (2.0= twice as fast;Inf= no waiting, i.e. plain streaming).timestamp—:ts_event(default; present on every record) or:ts_recv(falls back tots_eventfor record types without it).max_sleep— cap on any single wait so overnight session gaps don't stall replay; re-anchors after a clamp.precise— busy-wait sub-millisecond gaps for dense-burst fidelity (see below).clock/sleep_fn— injectable for deterministic testing.Design notes
anchor + (ts − anchor)/speed) rather than sleeping the raw inter-record gap. Time spent insidefis absorbed instead of accumulating as drift — a slow callback just shortens the next wait, and the stream re-synchronizes.Base.sleepresolves to ~1 ms on Unix and as coarse as the ~15 ms system timer tick on Windows. Records spaced more tightly than that arrive clumped, but because pacing is anchored, the clumping is local and non-cumulative, and same-timestamp records are delivered back-to-back.precise = truebusy-waits small gaps for sub-millisecond accuracy at the cost of pinning a CPU core.Tests
test/test_replay.jl(35 tests), wired intoruntests.jl.speed,speed = Inf, out-of-order timestamps,max_sleepclamping + re-anchor,:ts_recvpacing, empty input, argument validation, and a file round-trip viareplay_dbn.precisemode covered by_resolve_sleep_fndispatch checks, an_precise_sleepno-early-return check, and a real-clock sub-millisecond burst replay.Pkg.test().Docs
README gains a "Replaying DBN Files" section + a feature bullet, and both functions have full docstrings including the timing-resolution caveat.
🤖 Generated with Claude Code