Migrate Parquet I/O from Parquet2.jl to DuckDB.jl#42
Conversation
Parquet2.jl output was not reliably readable by all Parquet consumers
(DuckDB, pandas/pyarrow), so back Parquet read/write with DuckDB.jl
instead. Both dbn_to_parquet and parquet_to_dbn now go through DuckDB,
and Parquet2 is dropped as a dependency (replaced by DuckDB + DBInterface).
- dbn_to_parquet writes a single file at the given path and now compresses
with ZSTD by default, with a `compression` keyword
("zstd"/"snappy"/"gzip"/"uncompressed").
- Fix pre-existing FieldErrors in the DataFrame converters that crashed
export for OHLCV, STATUS, IMBALANCE, and DEFINITION records (they
referenced struct fields that don't exist). They now read the real
fields.
- Handle the empty-record edge case: a zero-column DataFrame is written as
a valid, empty header-only Parquet file instead of erroring in DuckDB.
All 3748 tests pass; every record-type fixture exports and round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41a62b9c4b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Fall back to a minimal header-only schema so we still write a valid, empty | ||
| # Parquet file that reads back as zero records. | ||
| if ncol(df) == 0 | ||
| df = DataFrame(ts_event = Int64[], instrument_id = UInt32[], publisher_id = UInt16[]) |
There was a problem hiding this comment.
When dbn_to_parquet sees an empty DBN, records_to_dataframe has lost the file's metadata schema, so this fallback always writes the same three generic columns. Empty trades/OHLCV exports therefore have a Parquet schema that is missing their real fields, and feeding this file back through parquet_to_dbn(..., schema=...) still trips minimum/maximum on the zero-row ts_event column in create_metadata_from_dataframe. Please use the DBN metadata/schema to build the correct zero-row DataFrame and handle empty metadata rather than a generic stub.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in ac3706e. Both points addressed:
- Schema preserved.
dbn_to_parquetnow rebuilds an empty record set fromMetadata.schemavia a newempty_dataframe_for_schema(schema), so an empty trades/OHLCV/etc. export keeps its real columns instead of the generic 3-column stub. (Also added anisemptyguard tombp10_to_dataframe, whose row-builder otherwise collapsed to a 0-column frame.) Emptyohlcv-1dnow exports as0x8with the real OHLCV schema. - Empty round-trip no longer crashes.
create_metadata_from_dataframenow guardsminimum/maximumagainst a zero-row column (start_ts = end_ts = 0), so a fully-empty Parquet/CSV round-trips back to DBN.
Verified: empty ohlcv-1d → Parquet (real schema) → DBN yields 0 records with no error; empty_dataframe_for_schema returns typed columns for every schema incl. MBP-10; all 3748 tests pass.
Addresses Codex review on #42. The empty-record fallback wrote a generic 3-column stub, so an empty trades/OHLCV export lost its real schema, and feeding it back through parquet_to_dbn crashed on min/max over the zero-row ts_event column. - dbn_to_parquet rebuilds an empty record set from Metadata.schema via the new empty_dataframe_for_schema, so the exported Parquet keeps the real columns and still round-trips to zero records. - mbp10_to_dataframe returns a typed zero-row frame on empty input (its row-builder otherwise collapsed to a 0-column frame). - create_metadata_from_dataframe guards minimum/maximum on a zero-row DataFrame (start_ts = end_ts = 0), so a fully-empty Parquet/CSV round-trips back to DBN. Empty ohlcv-1d now exports as 0x8 (real OHLCV schema); all 3748 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Why
Parquet2.jl's output is not reliably readable by all Parquet consumers (e.g. DuckDB, pandas/pyarrow), and the Parquet2.jl project itself recommends DuckDB.jl for Parquet workloads. Since this package exists for Databento-ecosystem interop, broken cross-reader compatibility undermines the Parquet export.
This PR replaces Parquet2.jl with DuckDB.jl for both Parquet write (
dbn_to_parquet) and read (parquet_to_dbn), and dropsParquet2as a dependency. Targeted as patch release 0.1.5.What changed
Parquet engine swap
Parquet2removed from deps;DuckDB(1.5.2) +DBInterface(2.6.1) added.src/export.jl(_duckdb_sql_path,_write_parquet,_read_parquet) own the DuckDB connection lifecycle and SQL path handling for both directions.compressionkeyword.ZSTD compression by default
dbn_to_parquet(input, output; compression="zstd"). Accepts"zstd"(default),"snappy","gzip","uncompressed".dbn_to_parquetwrites a single Parquet file at the exactoutput_filepath (pass a*.parquetpath, not a directory). Docs/examples updated accordingly.Bug fixes surfaced while testing the migration
FieldErrors in the DataFrame converters that crashed export for OHLCV, STATUS, IMBALANCE, and DEFINITION records — the converters referenced struct fields that don't exist. They now read the real fields (e.g. STATUS emitsreason/trading_event/is_trading/is_quoting/is_short_sell_restricted; IMBALANCE emitsauction_time/ssr_filling_price/ind_match_price/upper_collar/lower_collar; DEFINITION emitstick_rule).Verification
test/dataexports cleanly; independent DuckDB readback; ZSTD-default +compressionoverride confirmed; full DBN → Parquet → DBN round-trip; empty-file round-trip.🤖 Generated with Claude Code