Skip to content

Add timestamp-paced replay (replay_dbn / replay_records)#43

Merged
tbeason merged 3 commits into
mainfrom
feature/replay-dbn
Jun 24, 2026
Merged

Add timestamp-paced replay (replay_dbn / replay_records)#43
tbeason merged 3 commits into
mainfrom
feature/replay-dbn

Conversation

@tbeason

@tbeason tbeason commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Summary

Adds "replay" functionality: re-emit a DBN file's records in time order, paced by their timestamps to simulate a live feed. Useful for backtesting, demos, and driving consumers that expect a real-time stream. This was previously absent (the package only had unpaced streaming via DBNStream / foreach_record).

API

  • replay_dbn(f, filename; ...) — streams from a file (Zstd-aware, skips unknown rtypes exactly like DBNStream) and invokes f(record) paced in real time. Returns the record count.
  • replay_records(f, records; ...) — same pacing over an already-loaded collection (e.g. from read_dbn).
# Real-time
replay_dbn("trades.dbn") do rec
    println(price_to_float(rec.price))
end

# 100x faster, paced by receive time, overnight gaps capped at 1s
replay_dbn("mbo.dbn.zst"; speed = 100, timestamp = :ts_recv, max_sleep = 1.0) do rec
    handle(rec)
end

Options

  • speed — time-compression multiplier (2.0 = twice as fast; Inf = no waiting, i.e. plain streaming).
  • timestamp:ts_event (default; present on every record) or :ts_recv (falls back to ts_event for record types without it).
  • max_sleep — cap on any single wait so overnight session gaps don't stall replay; re-anchors after a clamp.
  • precise — busy-wait sub-millisecond gaps for dense-burst fidelity (see below).
  • clock / sleep_fn — injectable for deterministic testing.

Design notes

  • Wall-clock-anchored pacing. Each wait targets an absolute wall-clock time (anchor + (ts − anchor)/speed) rather than sleeping the raw inter-record gap. Time spent inside f is absorbed instead of accumulating as drift — a slow callback just shortens the next wait, and the stream re-synchronizes.
  • Timing resolution. DBN timestamps are nanosecond precision, but pacing is bounded by the sleep function: Base.sleep resolves to ~1 ms on Unix and as coarse as the ~15 ms system timer tick on Windows. Records spaced more tightly than that arrive clumped, but because pacing is anchored, the clumping is local and non-cumulative, and same-timestamp records are delivered back-to-back. precise = true busy-waits small gaps for sub-millisecond accuracy at the cost of pinning a CPU core.

Tests

  • New test/test_replay.jl (35 tests), wired into runtests.jl.
  • Deterministic pacing verified with an injected fake clock + sleep function (no real waiting): gap reproduction, speed, speed = Inf, out-of-order timestamps, max_sleep clamping + re-anchor, :ts_recv pacing, empty input, argument validation, and a file round-trip via replay_dbn.
  • precise mode covered by _resolve_sleep_fn dispatch checks, an _precise_sleep no-early-return check, and a real-clock sub-millisecond burst replay.
  • Full suite green: 3783/3783 via Pkg.test().

Docs

README gains a "Replaying DBN Files" section + a feature bullet, and both functions have full docstrings including the timing-resolution caveat.

🤖 Generated with Claude Code

tbeason and others added 2 commits June 24, 2026 13:18
Re-emit DBN records in time order, paced by their timestamps to simulate a
live feed (backtesting, demos, driving real-time consumers).

- replay_dbn(f, filename; ...) streams from a file (Zstd-aware, skips unknown
  rtypes like DBNStream) and invokes f(record) paced in real time.
- replay_records(f, records; ...) does the same over an in-memory collection.
- Wall-clock-anchored pacing absorbs callback execution time instead of
  accumulating drift; the stream re-synchronizes after slow callbacks.

Options: speed (time-compression multiplier; Inf = no waiting), timestamp
(:ts_event or :ts_recv with fallback), max_sleep (cap large gaps, re-anchors),
and precise (busy-wait sub-millisecond gaps for dense-burst fidelity, since
Base.sleep only resolves ~1ms on Unix / ~15ms on Windows). clock/sleep_fn are
injectable for deterministic testing.

Adds test/test_replay.jl (35 tests, fake-clock deterministic pacing checks +
real-clock precise-mode checks) and documents the timing-resolution limit in
the docstrings and README.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump version 0.1.5 -> 0.1.6 (additive, non-breaking: new replay_dbn /
replay_records exports only). Version the changelog's Unreleased section as
0.1.6 and document the replay feature under Added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60f7885c63

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/replay.jl Outdated
Comment on lines +137 to +141
deadline = time() + s
if s > _SPIN_THRESHOLD
sleep(s - _SPIN_THRESHOLD) # yield the bulk to the scheduler
end
while time() < deadline # spin the residual (≤ _SPIN_THRESHOLD)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a monotonic clock for precise sleeps

When precise = true, this deadline is based on Base.time(), which is wall-clock time rather than a monotonic clock. If the system clock is adjusted while replay is spinning (for example by NTP or manual correction), _precise_sleep can return early or busy-wait far longer than requested, distorting replay timing and potentially pinning a CPU core; use a monotonic source such as time_ns() for the deadline instead.

Useful? React with 👍 / 👎.

_precise_sleep measured its busy-wait deadline against wall-clock time(); a
system-clock adjustment (NTP, manual) mid-spin could make it return early or
busy-wait far longer than requested (pinning a core). Measure the deadline with
time_ns() (monotonic) instead. Addresses Codex PR review (P2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tbeason tbeason merged commit c284156 into main Jun 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant