Skip to content

feat: add fill feature logging for ML dataset building#159

Open
keitaj wants to merge 1 commit into
mainfrom
feat/fill-feature-logging
Open

feat: add fill feature logging for ML dataset building#159
keitaj wants to merge 1 commit into
mainfrom
feat/fill-feature-logging

Conversation

@keitaj

@keitaj keitaj commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • Persist per-fill order book features (spread, book imbalance, micro-price skew, top-of-book sizes) plus the adverse selection tracker's 5s/30s/60s markout samples as one JSON line per fill, in a daily-rotated JSONL file — a ready-made supervised dataset for offline adverse-selection modelling.
  • Feature definitions live in a single pure function (fill_features.compute_fill_features) shared by the logger and any future in-bot inference, structurally preventing train/serve skew.
  • Observation-only and fail-silent: the WebSocket thread performs no file IO (in-memory buffer, main-loop flush after a 65s maturity window so markout labels land in the same line); IO errors, a daily size cap, and a buffer cap are all counted and never affect trading. Default OFF preserves existing behaviour exactly.

Changes

  • fill_features.py (new): shared pure feature computation + FEATURE_SCHEMA_VERSION.
  • ws/fill_feature_writer.py (new): FillFeatureWriter — buffered, maturity-gated, daily-rotated JSONL writer with size/buffer caps and rate-limited error logging.
  • ws/adverse_selection_tracker.py: optional raw_fill kwarg on on_fill (backward compatible), feature record composition (tid/oid/hash join keys + features), set_feature_writer() hook, record field on FillSnapshot.
  • ws/fill_feed.py: pass the raw userFills object to the tracker (raw_fill=fill).
  • bot.py: writer wiring (warns and disables when the adverse tracker is unavailable), main-loop maybe_flush(), shutdown flush_all(), config keys + defaults, argparse flags.
  • ws/__init__.py: export FillFeatureWriter.
  • README.md: human-facing description + AI YAML parameter reference entries.
  • Tests: test_fill_features.py, test_fill_feature_writer.py (new), plus additions to tracker/fill-feed tests and a stub-attribute fix in test_risk_check_throttle.py.

Test plan

  • Unit tests added/updated (feature math, maturity gating, rotation, size/buffer caps, fail-silent paths, join-key passthrough, backward compatibility of legacy on_fill calls)
  • flake8 passes
  • pytest passes (1,149 tests)
  • No regressions in existing tests (default OFF path unchanged)

🤖 Generated with Claude Code

Persist per-fill order book features (spread, book imbalance,
micro-price skew, top-of-book sizes) together with the adverse
selection tracker's 5s/30s/60s markout samples as one JSON line per
fill in a daily-rotated JSONL file.

- fill_features.compute_fill_features is the single shared feature
  definition (guards against train/serve skew for future inference)
- FillFeatureWriter buffers records in memory on the WS thread and
  flushes from the main loop after a 65s maturity window so markout
  labels can be embedded in the same line
- Fail-silent by design: IO errors, size caps and buffer overflows
  are counted and never affect trading
- Default OFF; requires --enable-ws and --enable-adverse-selection-log

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant