Tolerant typed decode: skip interleaved non-schema records (#30)#36
Merged
Conversation
The Historical guide showed `foreach_record(client, DBN.TradeMsg; ...)`, a method that does not exist — the call raises a MethodError. The record type is inferred from `schema` (overridable via the `record_type` kwarg). Corrected the example and the surrounding prose. Fixes #32 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Long-range get_range queries spend well over 100s in server-side assembly before the first byte streams, so the old 100s default killed every multi-year pull. Read timeouts are deterministic for a given query shape, so they are no longer retried; HTTP.jl TimeoutError is mapped to a new BentoTimeoutError whose message names the remedy. Fixes #31 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Historical responses can legitimately interleave gateway ErrorMsg, SystemMsg, and SymbolMappingMsg records with the data; the typed decode path threw on the first one, forcing callers back to typed=false. The new _foreach_typed_tolerant loop keeps the zero-allocation typed hot path, @Warns ErrorMsg contents (they explain missing data), quietly skips other control records, and skips mismatched/unknown rtypes with one summary warning. Fixes #30 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guide only imports DatabentoAPI and Dates, so the record_type override examples referenced an undefined DBN name (review feedback). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If the foreach_record callback itself throws HTTP.Exceptions.TimeoutError (e.g. it makes its own HTTP request), open_stream unconditionally mapped it to BentoTimeoutError, blaming the Databento stream and reporting the wrong timeout value (review feedback). Track the exact exception object that escapes the consumer and map only stream-originated timeouts; a mid-body stall of our own stream still maps correctly because HTTP.jl raises the TimeoutError from the HTTP.open frame, a different object than whatever escaped the consumer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tbeason
added a commit
that referenced
this pull request
Jun 10, 2026
This was referenced Jun 17, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #35 (which is stacked on #34) — merge in order; this PR auto-retargets as base branches are deleted.
The typed
get_range/foreach_recordpath called DBN''s_foreach_record_impl, which throwsExpected ... but got rtype=...on the first record that isn''t the schema''s type. But historical responses can legitimately interleave gatewayErrorMsg(e.g. partial continuous-symbol resolution),SystemMsg, andSymbolMappingMsgrecords with the data — so real-world pulls forcedtyped=false, losing the ~10× fast path and the error record''s explanation.New private
_foreach_typed_tolerant(f, decoder, ::Type{T})insrc/store.jl, used by both the buffered (get_range) and streaming (foreach_record) typed paths. It keeps the zero-allocation hot path (peek header → read into a reusedRef{T}) and handles everything else by category:ErrorMsg→ per-record@warnwith the message text (rare, and it explains why data is missing — the core ask of Typed get_range/foreach_record throw on interleaved non-schema records (gateway ErrorMsg/SystemMsg) #30)SystemMsg/SymbolMappingMsg→ decoded,@debugonly@warn(never per-record; keeps data-loss visible without flooding logs)Implemented DatabentoAPI-side using existing DBN primitives (
read_record_header,_read_typed_record_stream,read_record_dispatch) — same precedent as the live typed reader. DBN''s exportedforeach_record_with_controlwas considered but rejected: it silently drops mismatched data records and carries file-closingfinallysemantics this layer doesn''t own.Behavior note (in CHANGELOG): an explicitly wrong
record_typeoverride onforeach_recordnow yields zero callbacks + summary warning instead of throwing.Tests
Full offline suite passes (1847 tests, Julia 1.12.6). New coverage: mixed trades+ErrorMsg+SystemMsg payloads through both
get_rangeandforeach_record, unknown-rtype skip with summary warn, wrongrecord_typeoverride, and a no-warnings regression test on pure streams.Fixes #30
🤖 Generated with Claude Code