Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/documentation-site.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"schema-stream": patch
---

Link the package to schema.stream and add the generated documentation site, executable integration
guides, synchronized canonical Markdown, browser playground, and a reproducible product demo video.
18 changes: 18 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
FROM oven/bun:1.3.14 AS bun-runtime

FROM node:24.18.0-bookworm-slim

ARG BUN_VERSION=1.3.14

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

COPY --from=bun-runtime /usr/local/bin/bun /usr/local/bin/bun

RUN apt-get update \
&& apt-get install --yes --no-install-recommends ca-certificates git \
&& rm -rf /var/lib/apt/lists/* \
&& ln --symbolic /usr/local/bin/bun /usr/local/bin/bunx \
&& test "$(bun --version)" = "${BUN_VERSION}" \
&& test "$(node --version)" = "v24.18.0"

USER node
41 changes: 41 additions & 0 deletions .devcontainer/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Schema Stream development container

This container pins Node.js 24.18.0 and Bun 1.3.14, installs the lockfile with
`bun install --frozen-lockfile`, and forwards the WebSocket example on port 3400. It works with
GitHub Codespaces or a local editor that supports the Development Container specification.

Create an editor and terminal workspace at
[codespaces.new/hack-dance/schema-stream](https://codespaces.new/hack-dance/schema-stream), or open
the repository locally with your editor's **Reopen in Container** command. Codespaces availability
and included usage depend on the developer's GitHub plan; the local container path has no hosted
workspace charge.

The safe default is credential-free: live E2E is disabled and the WebSocket UI opens in Fixture
mode. Run the deterministic checks from the repository root:

```sh
bun run examples
bun test tests/sdk-runtime.test.ts
```

Run the browser example locally:

```sh
bun run example:websocket
```

Then open <http://127.0.0.1:3400>. In Codespaces, start it with the exact private forwarded origin:

```sh
SCHEMA_STREAM_EXAMPLE_ORIGIN="https://${CODESPACE_NAME}-3400.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}" \
bun run example:websocket
```

Open the forwarded port from the **Ports** panel. Keep it private. The example validates this exact
origin rather than accepting arbitrary proxy hosts.

`OPENAI_API_KEY` is an optional [recommended Codespaces
secret](https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/configuring-dev-containers/specifying-recommended-secrets-for-a-repository).
Associate a personal secret with the repository before creating the codespace to enable the
explicit OpenAI mode. Never paste a key into the documentation website, browser UI, source files,
or terminal history.
41 changes: 41 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{
"name": "Schema Stream",
"build": {
"context": "..",
"dockerfile": "Dockerfile",
"args": {
"BUN_VERSION": "1.3.14"
}
},
"containerEnv": {
"SCHEMA_STREAM_EXAMPLE_PORT": "3400",
"SCHEMA_STREAM_LIVE_E2E": "0"
},
"forwardPorts": [3400],
"hostRequirements": {
"cpus": 2,
"memory": "4gb",
"storage": "16gb"
},
"otherPortsAttributes": {
"onAutoForward": "silent"
},
"portsAttributes": {
"3400": {
"label": "Schema Stream WebSocket example",
"onAutoForward": "notify",
"protocol": "http"
}
},
"postCreateCommand": "bun install --frozen-lockfile",
"remoteUser": "node",
"runArgs": ["--init"],
"secrets": {
"OPENAI_API_KEY": {
"description": "Optional personal OpenAI API key for explicitly selected live examples. Fixture examples and the default test suite do not need it.",
"documentationUrl": "https://platform.openai.com/api-keys"
}
},
"updateRemoteUserUID": true,
"waitFor": "postCreateCommand"
}
14 changes: 12 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ jobs:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 0
persist-credentials: false

- name: Setup Bun
Expand All @@ -39,11 +40,20 @@ jobs:
- name: Type-check
run: bun run type-check

- name: Verify release plan
run: bun run release:check --since origin/main

- name: Test with coverage
run: bun run test:coverage

- name: Build
run: bun run build
- name: Verify documentation sources
run: bun run docs:check

- name: Verify product demo composition
run: bun run demo:check

- name: Build package and documentation site
run: bun run docs:build

- name: Verify packed consumers
run: bun run test:packed
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,18 @@ node_modules
# Build and test output
coverage
dist
.source
site/.next
site/.source
site/content/docs
site/public/generated

# Local configuration
.env
.env.*
!.env.example
.mise.local.toml
.vercel

# Local credentials and signing material
*.key
Expand Down
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ bun run check
- Avoid narration-style inline comments. Use an inline comment only for a subtle invariant, protocol detail, compatibility workaround, or actionable TODO with context.
- Update README or focused files under `docs/` when public behavior, configuration, performance tradeoffs, or compatibility changes.
- Keep examples executable, type-correct, and representative of the published API.
- Treat `README.md`, public guides under `docs/`, and `CHANGELOG.md` as canonical sources for both GitHub and schema.stream. `CONTRIBUTING.md`, `docs/benchmarks/`, and `archive/` remain repository-only. Never edit ignored `site/content/docs/`; `bun run docs:prepare` rebuilds it.
- Run `bun run docs:generate` after changing public exports, exported TSDoc, or checked-in benchmark evidence. Commit the generated `docs/reference/api.md` and `docs/benchmarks/latest.md` updates.
- Run `bun run docs:check` for generated drift and Markdown link coverage, and `bun run docs:build` for the complete production documentation build.
- Keep browser examples credential-free. Live-provider examples must use user-owned server or Codespaces secrets and must never ask for an API key in the deployed docs UI.

## Testing expectations

Expand Down Expand Up @@ -83,5 +87,6 @@ Performance tests must be deterministic enough to detect major regressions witho
- `src/`: library source and public entry points
- `tests/`: unit, integration, benchmark, and packed-consumer verification
- `docs/`: focused design and behavior documentation
- `media/`: reproducible HyperFrames source for public product media
- `.changeset/`: release notes and version intent
- `.github/workflows/`: CI and release automation
39 changes: 39 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,45 @@ The opt-in live provider matrix fails closed when explicitly enabled without its
credential variables. See [`docs/integration-testing.md`](./docs/integration-testing.md) for secure
runtime injection, provider selection, and the exact verification contract.

## Documentation

Edit `README.md`, public guides under `docs/`, and `CHANGELOG.md` as the canonical sources for both
GitHub and [schema.stream](https://schema.stream/). Do not edit the ignored `site/content/docs/`
staging directory directly. `CONTRIBUTING.md`, `docs/benchmarks/`, and `archive/` are intentionally
repository-only material.

Regenerate checked-in API and benchmark reference pages after changing exported TSDoc, public
exports, or benchmark evidence:

```sh
bun run docs:generate
```

Verify generated-file drift and Markdown links, then build the package and production docs site:

```sh
bun run docs:check
bun run docs:build
```

For local documentation work, `bun run docs:dev` prepares the canonical Markdown, serves the site at
`http://127.0.0.1:3401`, and keeps staged pages synchronized as canonical Markdown, exported TSDoc,
benchmark evidence, or documentation assets change. Browser examples must remain credential-free;
live-provider examples read user-owned secrets only from the server or Codespaces environment.

### Product demo media

The editable HyperFrames composition for the README and documentation demo lives in
`media/schema-stream-demo/`. Validate it without rendering before committing composition changes:

```sh
bun run demo:check
```

Run `bun run demo:render` to validate the composition and regenerate the canonical MP4, GIF, and
poster under `docs/assets/`. The GIF is used by GitHub's README, while the documentation homepage
uses the MP4 with the poster as its loading frame.

## Benchmarking

Run the Bun and Node snapshot benchmark after changes to parser hot paths, snapshot materialization,
Expand Down
5 changes: 3 additions & 2 deletions MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,6 @@ older release was developed.

## Source monorepo follow-up

The exact island-ai cleanup is documented in [docs/island-ai-follow-up.md](docs/island-ai-follow-up.md).
It is deliberately not part of this repository's branch.
The historical island-ai cleanup plan is retained in
[archive/island-ai-follow-up.md](archive/island-ai-follow-up.md). It is repository history rather
than published package documentation.
65 changes: 15 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
[![CI](https://github.com/hack-dance/schema-stream/actions/workflows/ci.yml/badge.svg)](https://github.com/hack-dance/schema-stream/actions/workflows/ci.yml)
[![Follow @dimitrikennedy](https://img.shields.io/twitter/follow/dimitrikennedy?style=social&labelColor=000000)](https://twitter.com/dimitrikennedy)

[Documentation](https://schema.stream/docs) · [Examples](https://schema.stream/examples) ·
[Approach](https://schema.stream/approach) · [Playground](https://schema.stream/playground)

Parse JSON while it is still arriving. `schema-stream` turns a Web Stream or async iterable into
typed, schema-shaped snapshots, so a UI can render partial strings, nested objects, and arrays
before the response is complete.
Expand All @@ -19,7 +22,7 @@ Snapshots are progressive parser output, not validation results. `schema-stream`
types and initial placeholders, but it does not validate partial or final values. Validate the
settled value with your schema or use the producing SDK's validated result.

![Terminal demo of schema-stream filling a typed object as JSON arrives](https://raw.githubusercontent.com/hack-dance/schema-stream/main/docs/assets/schema-stream-demo.gif)
![Schema Stream progressing from raw JSON chunks to completion events, snapshot policies, and a materialized dashboard](https://raw.githubusercontent.com/hack-dance/schema-stream/main/docs/assets/schema-stream-demo.gif)

## One stream, many snapshots

Expand Down Expand Up @@ -135,53 +138,6 @@ See the [transport guide](./docs/transports.md) and the repository's
[Bun WebSocket UI](./examples/websocket-ui/) for an interactive localhost visualization of the
message protocol and snapshot-policy controls.

## Performance

Run the repeatable Bun and Node benchmark from a repository checkout:

```bash
bun run benchmark
```

The benchmark compares three feature-aligned paths for each snapshot policy: serialized byte
snapshots from `parse()`, those same snapshots materialized through UTF-8 decoding and `JSON.parse`,
and direct object snapshots from `iterate()`. The terminal summary reports the speedup of
`iterate()` over the serialized round-trip baseline and the cumulative serialization it avoids.

It also reports normalized `JSON.stringify`, UTF-8 encoding, and `JSON.parse` costs as native
runtime references. Those isolated operations are not presented as equivalent alternatives to
progressive schema-shaped parsing. Results are local synthetic measurements; the runtime versions,
fixture size, chunk size, warmups, and sample count printed by the command define each result.

Representative local results from an Apple M5 Max arm64 host on July 11, 2026 are shown below.
These are medians from the default 2 MiB fixtures, 64 KiB chunks, one warmup, and five measured
runs. The [machine-readable evidence](./docs/benchmarks/2026-07-11-apple-m5-max.json) retains the raw
samples, host, baseline revision, validation method, and benchmark configuration.

| Runtime | Fixture / policy | Roundtrip | `iterate()` | Speedup | Serialization avoided |
| --- | --- | ---: | ---: | ---: | ---: |
| Bun 1.3.14 | long string / chunk | 11.94 ms | 1.53 ms | 7.80x | 33.00 MiB |
| Bun 1.3.14 | long string / final | 1.95 ms | 1.29 ms | 1.52x | 2.00 MiB |
| Bun 1.3.14 | object heavy / chunk | 310.36 ms | 298.24 ms | 1.04x | 35.00 MiB |
| Bun 1.3.14 | object heavy / final | 93.33 ms | 87.88 ms | 1.06x | 2.00 MiB |
| Node 24.18.0 | long string / chunk | 100.60 ms | 2.73 ms | 36.83x | 33.00 MiB |
| Node 24.18.0 | long string / final | 7.64 ms | 2.63 ms | 2.91x | 2.00 MiB |
| Node 24.18.0 | object heavy / chunk | 339.35 ms | 245.88 ms | 1.38x | 35.00 MiB |
| Node 24.18.0 | object heavy / final | 55.63 ms | 52.18 ms | 1.07x | 2.00 MiB |

Use `bun run benchmark --help` for filters, verbose ranges, and machine-readable JSON. See the
[benchmark methodology](./docs/snapshot-policies.md#benchmark) for the fixtures and timing
boundaries.

Completion callback scaling is measured separately so the normal suite stays quick:

```bash
bun run benchmark --completion-scaling
```

That mode doubles nested record counts and compares parsing with no callback, delta-based
`onValueComplete`, and cumulative-history `onKeyComplete`.

## OpenAI Agents SDK

Pass the Agents SDK text stream directly to `iterate()`:
Expand Down Expand Up @@ -371,7 +327,9 @@ bun install
bun run format
bun run lint
bun run examples
bun run benchmark
bun run docs:dev
bun run docs:check
bun run docs:build
bun run check
```

Expand All @@ -381,7 +339,14 @@ declarations with TypeScript 7.0.2. TypeScript is a development-only dependency,

Ultracite and Biome own formatting and linting. `bun run format` applies safe fixes, `bun run lint`
checks the repository without writing, and `bun run check` includes linting before type checks,
tests, and packed-consumer verification.
tests, packed-consumer verification, generated-doc drift checks, and the production docs build.

The checked-in Markdown in `README.md`, public guides under `docs/`, and `CHANGELOG.md` is the
documentation source of truth for both GitHub and [schema.stream](https://schema.stream). Public API
reference Markdown is generated from exported TSDoc, and `site/content/docs` is an ignored
build-stage copy. Run `bun run docs:generate` after changing exports or public TSDoc; never edit the
staged site content directly. Contributor guidance and engineering evidence remain repository-only
material.

`test:packed` installs the generated tarball into clean consumers and verifies ESM, CommonJS,
Zod 4/Mini, Zod 3, OpenAI Agents SDK, Mastra, and Vercel AI SDK compatibility with TypeScript 5.9
Expand Down
File renamed without changes.
4 changes: 4 additions & 0 deletions biome.jsonc
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "./node_modules/@biomejs/biome/configuration_schema.json",
"extends": ["ultracite/biome/core", "ultracite/biome/vitest"],
"files": {
// HyperFrames compositions use their framework-specific validator via `bun run demo:check`.
"includes": ["!media/schema-stream-demo"]
},
"formatter": {
"lineWidth": 100
},
Expand Down
Loading