Migrate Parquet I/O from Parquet2.jl to DuckDB.jl

CHANGELOG.md

@@ -7,8 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Parquet read/write is now backed by **DuckDB.jl** instead of **Parquet2.jl**. The
+  previous Parquet2-written files were not reliably readable by all Parquet consumers
+  (e.g. DuckDB, pandas/pyarrow); DuckDB-written output is broadly compatible. `Parquet2`
+  has been dropped as a dependency (replaced by `DuckDB` and `DBInterface`). Both
+  `dbn_to_parquet` and `parquet_to_dbn` are affected. `dbn_to_parquet` writes a single
+  Parquet file at the exact `output_file` path (pass a `*.parquet` path, not a directory).
+
 ### Added
 
+- `dbn_to_parquet` now compresses with **ZSTD by default** and accepts a `compression`
+  keyword — one of `"zstd"` (default), `"snappy"`, `"gzip"`, or `"uncompressed"`.
 - Symbol resolution helpers that join the human-readable `raw_symbol` back onto
   records using `Metadata.mappings`: `symbol_map(metadata)` builds an
   `instrument_id -> [(start_date, end_date, raw_symbol)]` lookup,
@@ -22,6 +33,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `records_to_dataframe` (and therefore `dbn_to_csv` / `dbn_to_parquet`) no longer throws
+  `FieldError` on **OHLCV**, **STATUS**, **IMBALANCE**, and **DEFINITION** records. The
+  converters referenced struct fields that do not exist: `OHLCVMsg.ts_recv`;
+  `StatusMsg.ts_in_delta`/`sequence` (and a bogus `Action.T` cast of the status action
+  code); `ImbalanceMsg.auction_price`/`ts_in_delta`/`sequence`; and
+  `InstrumentDefMsg.multiplier`/`min_price_increment_portfolio_type`/`ts_in_delta`/
+  `sequence`. They now read the real fields (e.g. STATUS emits `reason`,
+  `trading_event`, `is_trading`/`is_quoting`/`is_short_sell_restricted`; IMBALANCE emits
+  `auction_time`, `ssr_filling_price`, `ind_match_price`, `upper_collar`, `lower_collar`;
+  DEFINITION emits `tick_rule`).
+- `dbn_to_parquet` no longer errors on an **empty record set** (e.g. an OHLCV-1d window
+  with no bars). The exported Parquet preserves the file's real schema (built from
+  `Metadata.schema` via `empty_dataframe_for_schema`) as a valid, zero-row file that reads
+  back as zero records, instead of raising a DuckDB "Table function must return at least
+  one column" error. `mbp10_to_dataframe` likewise returns a typed zero-row frame on empty
+  input rather than a column-less one.
+- `create_metadata_from_dataframe` (used by `parquet_to_dbn` / `csv_to_dbn`) no longer
+  throws from `minimum`/`maximum` on a zero-row DataFrame; an empty frame yields
+  `start_ts = end_ts = 0`, so a fully-empty Parquet/CSV round-trips back to DBN.
 - `records_to_dataframe` (and therefore `to_dataframe` / `dbn_to_parquet`) no
   longer throws `FieldError` on **TBBO / MBP-1** and **MBP-10**. The converters
   read non-existent flat `bid_px_00…` fields; the bid/ask data actually lives in

Project.toml

@@ -1,39 +1,41 @@
 name = "DatabentoBinaryEncoding"
 uuid = "90689371-c8cb-40d1-831f-18033db90f74"
+version = "0.1.5"
 authors = ["Tyler Beason <tbeas12@gmail.com>"]
-version = "0.1.4"
 
 [deps]
 CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
 CodecZstd = "6b39b394-51ab-5f42-8807-6242bab2b4c2"
+DBInterface = "a10d1c49-ce27-4219-8d33-6db1a4562965"
 DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
+DuckDB = "d2f5444f-75bc-4fdf-ac35-56f514c445e1"
 EnumX = "4e289a0a-7415-4d19-859d-a7e5c4648b56"
 JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
-Parquet2 = "98572fba-bba0-415d-956f-fa77e587d26d"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 StructTypes = "856f2bd8-1eba-4b0a-8007-ebc267875bd4"
 TranscodingStreams = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
 
 [compat]
-julia = "1.12"
 BenchmarkTools = "1.6.0"
 CSV = "0.10.15"
 CodecZstd = "0.8.6"
+DBInterface = "2.6.1"
 DataFrames = "1.7.0"
 Dates = "1.11.0"
+DuckDB = "1.5.2"
 EnumX = "1.0.5"
 JSON3 = "1.14.3"
-Parquet2 = "0.2.31"
 Statistics = "1.11.0"
 StructTypes = "1.11.0"
 TranscodingStreams = "0.11.3"
+julia = "1.12"
 
 [extras]
 BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "BenchmarkTools"]
 docs = ["Documenter"]
+test = ["Test", "BenchmarkTools"]

README.md

@@ -149,7 +149,7 @@ close_writer!(writer)
 # Convert to different formats
 dbn_to_csv("trades.dbn", "trades.csv")
 dbn_to_json("trades.dbn", "trades.json")
-dbn_to_parquet("trades.dbn", "output_dir/")
+dbn_to_parquet("trades.dbn", "trades.parquet")  # ZSTD-compressed by default
 
 # Convert to DataFrame for analysis
 df = records_to_dataframe(records)

docs/src/api/conversion.md

@@ -41,8 +41,8 @@ dbn_to_csv("trades.dbn", "trades.csv")
 # Convert to JSON
 dbn_to_json("trades.dbn", "trades.json")
 
-# Convert to Parquet
-dbn_to_parquet("trades.dbn", "output_dir/")
+# Convert to Parquet (ZSTD-compressed by default)
+dbn_to_parquet("trades.dbn", "trades.parquet")
 
 # Convert to DataFrame for analysis
 records = read_trades("trades.dbn")

docs/src/guide/conversion.md

@@ -86,17 +86,18 @@ dbn_to_json("trades.dbn", "trades.jsonl")
 ```julia
 using DatabentoBinaryEncoding
 
-# Convert to Parquet
-dbn_to_parquet("trades.dbn", "output_directory/")
+# Convert to Parquet (ZSTD-compressed by default)
+dbn_to_parquet("trades.dbn", "trades.parquet")
 
-# Output creates trades.parquet in the directory
+# Choose a different compression codec
+dbn_to_parquet("trades.dbn", "trades.parquet", compression="snappy")
 ```
 
 **Parquet Output:**
 - Columnar format (efficient for analytics)
 - Preserves data types
-- Automatic compression
-- Compatible with Arrow, DuckDB, pandas, etc.
+- ZSTD compression by default (`compression` keyword: `"zstd"`, `"snappy"`, `"gzip"`, `"uncompressed"`)
+- Written via DuckDB, so it is reliably readable by Arrow, DuckDB, pandas/pyarrow, etc.
 
 ### DBN to DataFrame
 
@@ -274,7 +275,7 @@ rm("latest.dbn")
 using DatabentoBinaryEncoding
 
 # Convert historical data to Parquet
-dbn_to_parquet("2024_trades.dbn.zst", "data_lake/")
+dbn_to_parquet("2024_trades.dbn.zst", "data_lake/trades.parquet")
 
 # Now you can query with DuckDB, pandas, etc.
 # SELECT AVG(price) FROM 'data_lake/trades.parquet'

docs/src/index.md

@@ -43,7 +43,7 @@ end
 
 # Convert to other formats
 dbn_to_csv("trades.dbn", "trades.csv")
-dbn_to_parquet("trades.dbn", "output_dir/")
+dbn_to_parquet("trades.dbn", "trades.parquet")
 ```
 
 ## Performance Characteristics

docs/src/installation.md

@@ -56,7 +56,7 @@ DatabentoBinaryEncoding.jl has the following dependencies (automatically install
 - **CodecZstd** - Zstd compression support
 - **CSV** - CSV file conversion
 - **JSON3** - JSON file conversion
-- **Parquet2** - Parquet file conversion
+- **DuckDB** - Parquet file conversion (read/write)
 - **DataFrames** - DataFrame conversion
 - **EnumX** - Enhanced enum support
 - **StructTypes** - Type serialization

docs/src/quickstart.md

@@ -153,8 +153,8 @@ dbn_to_csv("trades.dbn", "trades.csv")
 # DBN → JSON
 dbn_to_json("trades.dbn", "trades.json")
 
-# DBN → Parquet
-dbn_to_parquet("trades.dbn", "output_dir/")
+# DBN → Parquet (ZSTD-compressed by default)
+dbn_to_parquet("trades.dbn", "trades.parquet")
 
 # DBN → DataFrame
 df = records_to_dataframe(records)

docs/src/troubleshooting.md

@@ -352,8 +352,10 @@ end
 **Solution**: Ensure Parquet has correct schema:
 ```julia
 # Check column names
-using Parquet2
-df = Parquet2.Dataset("file.parquet") |> DataFrame
+using DuckDB, DBInterface, DataFrames
+con = DBInterface.connect(DuckDB.DB)
+df = DBInterface.execute(con, "SELECT * FROM read_parquet('file.parquet')") |> DataFrame
+DBInterface.close!(con)
 names(df)  # Should match DBN field names
 
 # Ensure correct types (especially timestamps, prices)

src/DatabentoBinaryEncoding.jl

@@ -85,7 +85,8 @@ using TranscodingStreams
 using EnumX
 using DataFrames
 using CSV
-using Parquet2
+using DuckDB
+using DBInterface
 using JSON3
 using StructTypes