chore: update docs

Author: pi0

AGENTS.md

@@ -13,36 +13,60 @@
 - `build.zig.zon` — minimal manifest (name, version, fingerprint).
 - `.zig-cache/cargo-target/<triple>/` — per-target cargo target directory, isolated so parallel cross builds don't clobber each other's fingerprints.
 - `dist/rg-<triple>[.exe|.wasm]` — install outputs. The published package only ships the wasm one.
-- `build.ts` — inlines `dist/rg-wasm32-wasip1.wasm` into `lib/_rg.wasm.mjs` (brotli + z85). Run with `node build.ts`.
+- `build.ts` — inlines `dist/rg-wasm32-wasip1.wasm` into `lib/_rg.wasm.mjs` (brotli + z85) and stamps the wasm hash into `lib/_rg.mjs` for disk-cache invalidation. Run with `node build.ts`.
 
 ### npm package (`lib/`)
 
-- `lib/index.mjs` — ESM entry. Exports `ripgrep(args, options)` and doubles as the `rg` bin (`bin: { rg: "./lib/index.mjs" }`). Compiles the wasm module lazily once via `WebAssembly.compile(getRgWasmBytes())` and caches the `Promise<WebAssembly.Module>`; instances are still created per-call since they own WASI state.
+- `lib/index.mjs` — ESM entry. Exports `ripgrep(args, options)` and `rgPath`. Delegates wasm loading and WASI runtime creation to `_rg.mjs`.
+- `lib/rg.mjs` — CLI bin entry (`bin: { rg, ripgrep }`). Calls `enableCompileCache()` from `node:module` for faster repeated runs, then forwards `process.argv` to `ripgrep()` and exits with its code.
+- `lib/_rg.mjs` — orchestration layer between the wasm blob and WASI runtimes:
+  - `getRgWasmModule()` — memoizes the `Promise<WebAssembly.Module>`. Decompresses the blob via `brotliDecompressSync` and caches the raw `.wasm` bytes to disk (`os.tmpdir()/ripgrep-wasm-<hash>.wasm`) so subsequent runs skip decompression entirely.
+  - `createWasiRuntime(options)` — picks the WASI backend: uses `node:wasi` by default on Node (suppresses `ExperimentalWarning`), falls back to the custom shim if `node:wasi` import fails or if custom `stdout`/`stderr` streams without a numeric `fd` are provided. On Bun and Deno the custom shim is always used.
 - `lib/_wasi.mjs` — minimal WASI preview1 shim. Implements only the ~23 syscalls ripgrep actually imports (`fd_read`, `fd_write`, `fd_readdir`, `fd_seek`, `fd_tell`, `fd_close`, `fd_fdstat_get`, `fd_fdstat_set_flags`, `fd_filestat_get`, `fd_prestat_*`, `path_open`, `path_filestat_get`, `path_readlink`, `args_*`, `environ_*`, `clock_time_get`, `random_get`, `proc_exit`, `sched_yield`, `poll_oneoff` stub). Backed by `node:fs` sync APIs so it works on Node, Bun, and Deno uniformly. `proc_exit` throws a `WASIExit` that `start()` catches to return the exit code.
-- `lib/_rg.wasm.mjs` — auto-generated by `build.ts`. Exports `getRgWasmBytes()` which z85-decodes + `brotliDecompressSync`s the blob into a `Uint8Array` on demand. The encoded blob lives inside a `getEncoded = () => "…"` function so V8 lazy-parses the large string literal and only allocates it on first call, not at import time.
-- `lib/index.d.mts` — hand-written types, including an `RgFlag` union of known ripgrep long/short flags for autocomplete on `ripgrep` args (still accepts arbitrary strings via `RgArg = RgFlag | (string & {})`).
-- `package.json` — `name: "ripgrep"`, `type: module`, `files: ["lib"]`, `bin.rg`, exports only `./lib/index.mjs`.
+- `lib/_rg.wasm.mjs` — auto-generated by `build.ts`. Exports `getCompressedBytes()` which z85-decodes the blob into raw brotli-compressed bytes. The encoded blob lives inside a `getEncoded = () => "…"` function so V8 lazy-parses the large string literal and only allocates it on first call, not at import time.
+- `lib/index.d.mts` — hand-written types. Includes an `RgFlag` union of known ripgrep long/short flags for autocomplete on `ripgrep` args (still accepts arbitrary strings via `RgArg = RgFlag | (string & {})`). Overloads `ripgrep()` for `{ buffer: true }` → `RipgrepBufferedResult` (with `stdout` and `stderr` strings).
+- `package.json` — `name: "ripgrep"`, `type: module`, `files: ["lib"]`, `bin: { rg, ripgrep }` → `./lib/rg.mjs`, exports only `./lib/index.mjs`.
 
 ### Runtime behavior
 
 - Default preopens map `.` → `process.cwd()`; absolute paths passed as args are auto-added as preopens so they work without extra configuration.
-- ripgrep's TTY auto-detection doesn't work through WASI preview1 (it always sees a non-TTY), so `ripgrep` auto-injects `--color=ansi` when `process.stdout.isTTY` and the caller hasn't specified a color flag. Detection checks `--color`, `--color=…`, and `--no-color`.
-- `nodeWasi: true` (or `RIPGREP_NODE_WASI=1`) swaps the custom shim for Node's built-in `node:wasi`. Same `{ imports, start }` shape either way. Node's version prints an `ExperimentalWarning` on every run — that's why the custom shim is the default.
+- ripgrep's TTY auto-detection doesn't work through WASI preview1 (it always sees a non-TTY), so `ripgrep` auto-injects `--color=ansi` when `process.stdout.isTTY` and the caller hasn't specified a color flag and no custom `stdout` stream is provided. Detection checks `--color`, `--color=…`, and `--no-color`.
+- **WASI backend selection:** On Node, `node:wasi` is used by default (the `ExperimentalWarning` is silently suppressed). On Bun and Deno, the custom shim is used. Override with `nodeWasi: true/false` or `RIPGREP_NODE_WASI=1/0`. If `node:wasi` import fails at runtime, it falls back to the custom shim automatically. When custom `stdout`/`stderr` streams without a numeric `.fd` property are provided, the custom shim is forced regardless.
+- **Wasm disk cache:** The decompressed wasm bytes are cached in `os.tmpdir()/ripgrep-wasm-<hash>.wasm`. On subsequent runs, the cached file is read directly via `readFileSync`, skipping z85 decode + brotli decompression. The hash in the filename changes when the wasm binary is rebuilt.
+- **Buffered mode:** `ripgrep(args, { buffer: true })` captures stdout/stderr into strings returned as `result.stdout` / `result.stderr`. Custom streams take precedence over buffering for their respective fd.
 - `start()` returns `0` on clean exit or the exit code from `WASIExit`; `node:wasi`'s `start()` returns `undefined` on success, so the adapter coerces with `?? 0`. The public `ripgrep()` function wraps this into a `{ code }` result object.
 
+### Testing
+
+- `test/ripgrep.test.mjs` — vitest tests covering the programmatic API (`ripgrep()` with buffered/non-buffered/custom streams), `rgPath` export, and CLI execution via `spawn`.
+- `test/fixture/` — test data: `hello.txt`, `subdir/nested.txt`, `link.txt` → symlink to `hello.txt`.
+- Run: `pnpm vitest run test/ripgrep.test.mjs`.
+
+### Benchmarks
+
+- `bench/cli.mjs` — cold-start comparison: native `rg` binary vs `node lib/rg.mjs` (both via `execFileSync`). Uses [`mitata`](https://github.com/evanwashere/mitata).
+- `bench/api.mjs` — warm comparison: native `exec(rg)` vs `ripgrep()` API (wasm module pre-warmed, measures per-call overhead). Also uses `mitata`.
+- Both benchmark against `vendor/ripgrep/crates` searching for `fn main`.
+- Run: `node bench/cli.mjs` or `node bench/api.mjs`.
+
 ## Build flavors
 
 There is only one cargo profile: cross-compile with the smallest-size settings. No mode option, no native install, no sub-steps for tweaking.
 
 The cargo profile is `release-lto` (defined in `vendor/ripgrep/Cargo.toml`: fat LTO, `codegen-units=1`, `panic="abort"`), with size tuning layered on via `cargo --config`: `opt-level="z"`, `debug=false`, `strip="symbols"`.
 
+For wasm targets, SIMD is enabled via `RUSTFLAGS="-C target-feature=+simd128"` — this unlocks memchr's simd128 vectorized search paths in the wasm binary.
+
 ## Commands
 
+- `pnpm build` — build wasm + inline into JS (equivalent to `zig build wasi && node build.ts`)
 - `zig build wasi` — build `wasm32-wasip1` → `dist/rg-wasm32-wasip1.wasm` (default step)
-- `zig build native` — cross-compile all native targets → `dist/rg-<triple>[.exe]` (not shipped to npm; kept for local use / future flavors)
+- `zig build native` — cross-compile all native targets → `dist/rg-<triple>[.exe]` (not shipped to npm; kept for local use / benchmarks)
 - `zig build` — same as `zig build wasi`
-- `node build.ts` — inline the built wasm into `lib/_rg.wasm.mjs`. Must be re-run any time the wasm changes.
-- `node lib/index.mjs <rg args…>` — run the packaged CLI directly from source. Pass `--node-wasi` as the first arg to use Node's built-in WASI.
+- `node build.ts` — inline the built wasm into `lib/_rg.wasm.mjs` and stamp hash into `lib/_rg.mjs`. Must be re-run any time the wasm changes.
+- `node lib/rg.mjs <rg args…>` — run the packaged CLI directly from source.
+- `pnpm test` — run vitest tests
+- `pnpm fmt` — format with oxfmt
 
 ## Cross-compile targets
 
@@ -77,21 +101,34 @@ The install loop uses `.{ .custom = "../dist" }` as the install directory — th
 ## Gotchas / design notes
 
 - **Wasm is inlined, not a separate file.** `lib/_rg.wasm.mjs` ships the compressed bytes inside an ESM module so the npm package is pure JS — no `.wasm` asset resolution, no postinstall step. The tradeoff is a larger JS file and a one-time decode on first call.
+- **Two-layer decompression.** `_rg.wasm.mjs` exports `getCompressedBytes()` (z85 decode only), and `_rg.mjs` handles brotli decompression + disk caching. This separation keeps the generated blob module simple and the caching logic in hand-written code.
+- **Disk cache in tmpdir.** The decompressed wasm is written to `os.tmpdir()/ripgrep-wasm-<hash>.wasm` on first run. Subsequent runs load the cached file directly via `readFileSync`, skipping z85 decode and brotli decompression. The `<hash>` is the first 16 hex chars of the wasm's SHA-256, stamped by `build.ts`.
 - **Lazy string literal.** The z85-encoded blob is wrapped in `const getEncoded = () => "…"` specifically so V8 doesn't eagerly parse/allocate it at import time. Don't inline it back into a top-level `const`.
 - **Wasm module is cached, instances aren't.** `getRgWasmModule()` memoizes the `Promise<WebAssembly.Module>`; each `ripgrep` call still creates a fresh `WebAssembly.Instance` because WASI state (memory, fds) is per-instance.
+- **node:wasi ExperimentalWarning suppression.** `createNodeWasi` temporarily replaces `process.emitWarning` with a no-op while constructing the WASI instance, then restores it. This avoids the warning spam without `--no-warnings`.
+- **Stream-based stdio forces custom shim.** `node:wasi` only accepts numeric fd values for stdout/stderr. If a custom stream without a `.fd` property is passed, `createWasiRuntime` automatically uses the custom shim instead.
 - **Custom shim grants all rights in `fd_fdstat_get`.** ripgrep only reads, so over-granting `fs_rights_base` / `fs_rights_inheriting` (`~0n`) is harmless and avoids tracking precise capability bits.
 - **`poll_oneoff` is stubbed to `NOTSUP`.** ripgrep only uses it for stdin-driven modes, which the shim doesn't support anyway (stdin reads return 0 / EOF).
 - **`path_open` ENOENT handling.** Only ENOENT is swallowed when `O_CREAT` is set; everything else propagates, so bad paths surface as real errors instead of silent creates.
+- **WASM SIMD.** `build.zig` sets `RUSTFLAGS="-C target-feature=+simd128"` for wasm targets, enabling memchr's simd128 vectorized paths for faster search.
 - **TOML-quoted `--config` values.** `cargo --config` parses values as TOML, so string values must include the quotes: `opt-level="z"` not `opt-level=z`.
 - **Env-var form doesn't work for hyphenated profile names.** `CARGO_PROFILE_RELEASE_LTO_OPT_LEVEL` is ambiguous — cargo parses it as profile `release` with key `lto_opt_level` and fails. Use `--config` instead.
 - **Per-target `CARGO_TARGET_DIR`.** Sharing one target dir across triples causes cargo to constantly rebuild dependencies. Keeping them separate under `.zig-cache/cargo-target/<triple>/` gives proper caching.
 - **Cargo's output path is `$CARGO_TARGET_DIR/<triple>/release-lto/rg[.exe|.wasm]`** — for custom profiles the profile dir equals the profile name.
 - **ripgrep is still a Rust project.** `rustc` does all the code generation; Zig only acts as the C compiler and linker. A pure `zig build` of ripgrep would require rewriting it.
+- **`enableCompileCache`.** `rg.mjs` calls `node:module`'s `enableCompileCache()` for faster cold starts on Node. The call is wrapped in try/catch since some Node-compatible runtimes don't support it.
 
 ## Dependencies
 
 - `zig` (tested with 0.15.2)
 - `rustc` + `cargo` (tested with 1.94.1)
 - `cargo-zigbuild` (`cargo install cargo-zigbuild`)
 - Rust std for `wasm32-wasip1` (plus the native targets if building those)
-- Node.js 18+ / Bun / Deno at runtime (for `WebAssembly.compileStreaming`, web streams, and `node:fs` sync APIs)
+- Node.js 18+ / Bun / Deno at runtime (for `WebAssembly.compile`, `node:fs` sync APIs, and `node:zlib` brotli)
+
+### Dev dependencies
+
+- `vitest` — test runner
+- `mitata` — benchmarking
+- `oxfmt` — code formatter
+- `@vitest/coverage-v8` — coverage

README.md

@@ -61,7 +61,7 @@ Absolute filesystem path to a JS shim that runs ripgrep via `ripgrep`. Drop-in r
 Requirements:
 
 - `zig` (tested with 0.15.2)
-- `rustc` + `cargo` (tested with 1.90.0)
+- `rustc` + `cargo` (tested with 1.94.1)
 - [`cargo-zigbuild`](https://github.com/rust-cross/cargo-zigbuild): `cargo install cargo-zigbuild`
 - `rustup target add wasm32-wasip1`