Feat tape

.cargo/config.toml.example

@@ -0,0 +1,8 @@
+# Copy this file to `.cargo/config.toml` to tune the Rust code for your CPU.
+# It is gitignored so machine-specific flags never get committed or published.
+#
+#   cp .cargo/config.toml.example .cargo/config.toml
+#
+# For the C++ kernels, also pass `--features native` (see Cargo.toml).
+[build]
+rustflags = ["-C", "target-cpu=native"]

.gitignore

@@ -6,4 +6,10 @@ Cargo.lock
 .vscode/
 .idea/
 .cache/
-build/
+build/
+
+compile_commands.json
+
+# Local-only cargo config that enables `-C target-cpu=native` for benching.
+# Copy from `.cargo/config.toml.example`; never commit machine-specific flags.
+/.cargo/config.toml

CHANGELOG.md

@@ -9,6 +9,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Bindings for [simdjson] v0.3.1.
 - Use [cxx] instead of [bindgen] because of too many generics.
+- **Breaking:** restructured `tape::Error` and `serde::Error` into typed
+  variants carrying positions/lengths (`OutOfBounds`, `UnknownTapeType`,
+  `UnexpectedTapeType`, `StringOutOfBounds`, `MissingValueWord`,
+  `InvalidUtf8`) instead of a single `TapeCorrupted { message: String }`.
+  `serde::Error::TapeCorrupted` is removed; tape failures now surface as
+  `serde::Error::Tape { source: tape::Error }` (transparent).
+- Build tuning: `lto = "fat"` + `codegen-units = 1` on release/bench
+  profiles. New opt-in `native` feature compiles the bundled simdjson C++
+  with `-march=native`; copy `.cargo/config.toml.example` to
+  `.cargo/config.toml` for the matching Rust `-C target-cpu=native`.
 
 
 ## [0.1.0] - 2019-03-13

Cargo.toml

@@ -1,39 +1,58 @@
 [package]
 name = "simdjson-rust"
 version = { workspace = true }
-authors = ["SunDoge <384813529@qq.com>"]
-edition = "2021"
-license = "Apache-2.0"
+edition = "2024"
 description = "Rust bindings for the simdjson project."
-homepage = "https://crates.io/crates/simdjson-rust"
 documentation = "https://docs.rs/simdjson-rust"
-repository = "https://github.com/SunDoge/simdjson-rust"
 readme = "README.md"
+homepage = "https://crates.io/crates/simdjson-rust"
+repository = "https://github.com/SunDoge/simdjson-rust"
+license = "Apache-2.0"
 exclude = [".github/", "examples/"]
 
-
 [workspace]
 resolver = "2"
 members = ["simdjson-sys"]
 
 [workspace.package]
-version = "0.3.0-alpha.3"
+version = "0.4.0-alpha.1"
 
 [workspace.dependencies]
-simdjson-sys = { path = "simdjson-sys", version = "0.1.0-alpha.2" }
+simdjson-sys = { path = "simdjson-sys", version = "0.2.0-alpha.1" }
 
+[[bench]]
+harness = false
+name = "parser_bench"
 
 [dependencies]
-thiserror = "1.0"
+cxx = "1.0"
 simdjson-sys = { workspace = true }
 
 # serde compatibilty
 serde = { version = "1", features = ["derive"], optional = true }
-serde_json = { version = "1", optional = true }
+snafu = "0.9.1"
 
+[dev-dependencies]
+criterion = { version = "0.5", features = ["html_reports"] }
+serde = { version = "1", features = ["derive"] }
+serde_json = "1.0"
+simd-json = "0.14"
 
 [features]
-default = []
+default = ["serde"]
 
 # serde compatibility
-serde_impl = ["serde", "serde_json"]
+serde = ["dep:serde"]
+
+# Compile the bundled simdjson C++ with `-march=native` for the building CPU.
+# Off by default for portability. Pair with `RUSTFLAGS="-C target-cpu=native"`
+# (or copy `.cargo/config.toml.example`) to also tune the Rust code.
+native = ["simdjson-sys/native"]
+
+[profile.bench]
+lto = "fat"
+codegen-units = 1
+
+[profile.release]
+lto = "fat"
+codegen-units = 1

README.md

@@ -4,115 +4,167 @@
 [![Crates.io](https://img.shields.io/crates/v/simdjson-rust?style=for-the-badge)](https://crates.io/crates/simdjson-rust)
 [![docs.rs](https://img.shields.io/docsrs/simdjson-rust/latest?style=for-the-badge)](https://docs.rs/simdjson-rust)
 
-This crate currently uses [`simdjson 3.2.3`][simdjson]. You can have a try and give feedback.
+Rust bindings for the [simdjson][simdjson] C++ library (currently **v4.6.4**),
+exposing its DOM parser as a zero-copy tape and integrating with `serde` for
+direct deserialization into Rust structs.
 
-If you
+## What this crate does
 
-- find certain APIs are missing
-- encounter memory errors
-- experience performance degradation
+simdjson parses a JSON document in two stages: **stage1** scans the input with
+SIMD to find structural characters, **stage2** builds a flat _tape_ — an array
+of 64-bit words encoding every value with its type tag and payload, plus a
+side string buffer holding all (already-unescaped) string bytes.
 
-Please submit an issue.
+This crate hands that tape to Rust as two borrowed slices (`&[u64]` tape +
+`&[u8]` string buffer) and lets you walk it with zero copy:
 
-## Benchmark
+- `dom::Parser` — parse a JSON string/bytes into a `TapeRef` cursor or a
+  `tape::Value` DOM tree.
+- `tape::Value` — a zero-copy `enum` (`Null`/`Bool`/`Int64`/`Uint64`/`Double`/
+  `String(&str)`/`Array`/`Object`) borrowing the parser's memory.
+- `serde::from_str` / `from_tape` — deserialize any `serde::Deserialize` type
+  directly off the tape, with zero-copy `&'de str` for string fields.
 
-Check [SunDoge/json-benchmark](https://github.com/SunDoge/json-benchmark/tree/simdjson-rust)
-
-```
-                                DOM                  STRUCT
-======= serde_json ======= parse|stringify ===== parse|stringify ====
-data/canada.json         400 MB/s   460 MB/s   500 MB/s   390 MB/s
-data/citm_catalog.json   540 MB/s   790 MB/s  1000 MB/s  1070 MB/s
-data/twitter.json        390 MB/s  1020 MB/s   680 MB/s  1090 MB/s
-
-======= simd-json ======== parse|stringify ===== parse|stringify ====
-data/canada.json         400 MB/s   540 MB/s   530 MB/s
-data/citm_catalog.json  1080 MB/s  1000 MB/s  1570 MB/s
-data/twitter.json       1100 MB/s  1460 MB/s  1180 MB/s
-
-===== simdjson-rust ====== parse|stringify ===== parse|stringify ====
-data/canada.json         960 MB/s
-data/citm_catalog.json  3110 MB/s
-data/twitter.json       3160 MB/s
-```
-
-## Usage
-
-Add this to your `Cargo.toml`
+## Quick start
 
 ```toml
-# In the `[dependencies]` section
-simdjson-rust = "0.3.0"
+[dependencies]
+simdjson-rust = "0.4.0-alpha.1"
 ```
 
-Then, get started.
-
 ```rust
-use simdjson_rust::prelude::*;
-use simdjson_rust::{dom, ondemand};
-
-fn main() -> simdjson_rust::Result<()> {
-    let ps = "[0,1,2,3]".to_padded_string();
-
-    // ondemand api.
-    {
-        let mut parser = ondemand::Parser::default();
-        let mut doc = parser.iterate(&ps)?;
-        let mut array = doc.get_array()?;
-        for (index, value) in array.iter()?.enumerate() {
-            assert_eq!(index as u64, value?.get_uint64()?);
-        }
-    }
-
-    // dom api.
-    {
-        let mut parser = dom::Parser::default();
-        let elem = parser.parse(&ps)?;
-        let arr = elem.get_array()?;
-        for (index, value) in arr.iter().enumerate() {
-            assert_eq!(index as u64, value.get_uint64()?);
-        }
-    }
-
-    Ok(())
-}
-```
+use simdjson_rust::dom::Parser;
 
-### `dom` and `ondemand` 
-
-[`simdjson`][simdjson] now offer two kinds of API, `dom` and `ondemand`.
-`dom` will parsed the whole string while `ondemand` only parse what you request.
-Due to `ffi`, the overhead of `ondemand` API is relatively high. I have tested `lto` but it only improves a little :(
+let mut parser = Parser::default();
+let value = parser.parse_to_value(r#"{"hello": "world", "n": 42}"#).unwrap();
+assert_eq!(value.get("n").unwrap().as_i64(), Some(42));
+```
 
-Thus it is suggestted that
+With the default `serde` feature, deserialize straight into your own types:
 
-- use `ondemand` if you only want to access a specific part of a large json,
-- use `dom` if you want to parse the whole json.
+```rust
+use serde::Deserialize;
+use simdjson_rust::serde::from_str;
 
+#[derive(Deserialize, Debug, PartialEq)]
+struct Point { x: f64, y: f64 }
 
-### `padded_string`
+let p: Point = from_str(r#"{"x": 1.0, "y": 2.0}"#).unwrap();
+assert_eq!(p, Point { x: 1.0, y: 2.0 });
+```
 
-[`simdjson`][simdjson] requires the input string to be padded. We must provide a string with `capacity = len + SIMDJSON_PADDING`.
-We provide utils to do so.
+For high-throughput work, reuse the `Parser` across documents and deserialize
+from the tape cursor — this avoids re-allocating the parser's internal buffers
+on every call:
 
 ```rust
-use simdjson_rust::prelude::*;
-
-fn main() -> simdjson_rust::Result<()> {
-    let ps = make_padded_string("[0,1,2,3]");
-    let ps = "[0,1,2,3]".to_padded_string();
-    // or reuse a buffer.
-    let unpadded = String::from("[1,2,3,4]");
-    let ps = unpadded.into_padded_string();
-    // or load from file.
-    let ps = load_padded_string("test.json")?;
-    Ok(())
+use simdjson_rust::dom::Parser;
+use simdjson_rust::serde::from_tape;
+
+let mut parser = Parser::default();
+for json in inputs {
+    let tape = parser.parse_str(json)?;
+    let item: MyStruct = from_tape(tape)?;
+    // ...
 }
 ```
 
-## Other interesting things
-
-There are also pure Rust port of [`simdjson`][simdjson] available here [`simd-json`](https://github.com/simd-lite/simd-json).
-
+## Performance
+
+Numbers below are from `cargo bench --bench parser_bench --features native` on
+the author's machine (criterion, median). They compare three serde struct
+deserialization paths — `serde_json`, the pure-Rust [`simd-json`][simd-json]
+crate, and this crate — across three input sizes. Treat them as relative
+indicators, not absolute claims; always benchmark on your own hardware and
+your own data shape.
+
+| Input           | `serde_json` | `simd-json` | `simdjson-rust` |
+| --------------- | ------------ | ----------- | --------------- |
+| Small (~100 B)  | 84 ns        | 286 ns      | **110 ns**      |
+| Medium (~2 KB)  | ~810 ns      | 1210 ns     | **638 ns**      |
+| Large (~100 KB) | 187 µs       | 176 µs      | **137 µs**      |
+
+Where `simdjson-rust` wins:
+
+- **Large inputs**: the C++ stage1+stage2 pass is highly tuned, and the Rust
+  tape-walk that follows is pure sequential `&[u64]` reads. On 100 KB the
+  crate beats both `serde_json` and `simd-json`.
+- **Repeated small parses**: `Parser` is reusable, so the per-call fixed cost
+  is low; `serde_json` is still a touch faster on a single 100 B document, but
+  `simdjson-rust` pulls ahead once the parser is reused and on medium inputs.
+- **Zero-copy strings**: simdjson unescapes strings into `string_buf` during
+  stage2, so `&'de str` fields borrow directly with no re-validation or
+  allocation.
+
+Where it does _not_ win:
+
+- A single tiny JSON parsed once — `serde_json`'s scalar parser has the lowest
+  fixed cost.
+- Anything that needs the C++ toolchain at build time (see below).
+
+### How the benchmark is run
+
+The bench lives in `benches/parser_bench.rs` and uses [criterion][criterion].
+Each of the three size groups deserializes the same struct shape via all three
+parsers. To run it locally:
+
+```sh
+# Copy the local tuning config (tunes the Rust code for your CPU).
+cp .cargo/config.toml.example .cargo/config.toml
+# Run with the C++ side tuned for your CPU too.
+cargo bench --bench parser_bench --features native
+# Or, portable build (no native tuning):
+cargo bench --bench parser_bench
+```
 
-[simdjson]: https://github.com/simdjson/simdjson
+Note: `simd-json`'s serde entry point rewrites its `&mut str` input in place,
+so each iteration clones the input string for it — a constant tax specific to
+that bench function, not the parser. Read its column as "parse + one
+`String` clone".
+
+## Build requirements
+
+This crate builds simdjson from source via CMake, so the build host needs a
+C++17 compiler and CMake. simdjson is fetched through CMake's `FetchContent`
+on first build (no manual download), then cached. There is an opt-in `native`
+cargo feature that compiles the C++ kernels with `-march=native` for the
+building CPU; it is **off by default** so published binaries stay portable.
+See `.cargo/config.toml.example` for the matching Rust `-C target-cpu=native`.
+
+## Why only DOM, not ondemand?
+
+simdjson offers two APIs: **DOM** (parse the whole document into a tape up
+front) and **ondemand** (lazily parse only the fields you touch). This crate
+currently binds only DOM.
+
+Ondemand would be attractive for "large document, few fields" workloads, but
+binding it cleanly is hard:
+
+- simdjson's ondemand layer keeps its structural-index array and stage1 state
+  behind `internal::` and private members; there is no stable public surface
+  to hand that state to Rust without per-field FFI calls (which dominate the
+  cost for small reads).
+- Doing it efficiently would mean exposing simdjson's internal stage1
+  structural indexes to Rust and re-implementing stage2 (scope walking,
+  number parsing, string unescaping) in Rust — a large, internal-API-dependent
+  undertaking that would carry a real maintenance burden across simdjson
+  upgrades.
+
+For full-document deserialization — the common case and what `serde` is built
+for — DOM is both simpler and faster (one C++ pass, then a pure-Rust tape
+walk). Ondemand only wins when you deliberately skip most of a large document,
+which is a narrower use case.
+
+This is not a closed door. If ondemand support would unblock a real workload
+of yours, please open an issue describing your access pattern (document size,
+how many fields you read). Concrete demand will re-prioritize the work, and
+the likely path is a small patch to simdjson that exposes a stage1-only entry
+point — feasible, but not worth building speculatively.
+
+## License
+
+Apache-2.0, same as [simdjson][simdjson].
+
+[simdjson]: https://github.com/simdjson/simdjson
+[simd-json]: https://github.com/simd-lite/simd-json
+[criterion]: https://github.com/bheisler/criterion.rs