ci: add performance regression gate with criterion benchmarks

[membphis]

Background

qjson positions itself as a high-performance JSON parser, but currently lacks automated performance regression detection. Benchmarks exist (benches/lua_bench.lua) but run manually without CI enforcement.

Goal

Prevent unintentional performance degradation by adding automated benchmark comparison to CI.

Scope

In Scope

• Add Rust criterion benchmarks for core parse paths (eager/lazy, field access)
• Integrate benchmark comparison into PR CI using benchmark-action/github-action-benchmark
• Set regression threshold: >5% degradation triggers warning (not blocking)
• Store baseline results in gh-pages branch

Out of Scope

• Lua-level benchmarks in CI (keep manual due to OpenResty setup complexity)
• Micro-benchmarks for every function (focus on user-facing parse/access paths)

Design

1. Criterion Benchmarks

File: benches/rust_bench.rs

Benchmark groups:

• parse_eager - eager mode parsing (full validation)
• parse_lazy - lazy mode parsing (structural scan only)
• field_access - field access operations (get_str, get_f64, nested paths)

Fixtures (based on manifest.json ci tags):

• PR level: rest_api_small (2KB), wide_object (3KB), deep_nesting (0.5KB)
• Full level: additionally rest_api_medium (60KB), unicode_heavy_twitter (631KB)

Cargo.toml changes:

[dev-dependencies]
criterion = { version = "0.5", features = ["html_reports"] }

[[bench]]
name = "rust_bench"
harness = false

2. CI Workflow

New file: .github/workflows/bench.yml

Triggers:

• push to main → run benchmarks, update gh-pages baseline
• pull_request → run benchmarks, compare against main baseline, post PR comment

Job structure:

jobs:
  benchmark:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - checkout (with submodules for fixtures)
      - install stable rust
      - cargo bench --bench rust_bench -- --output-format bencher
      - benchmark-action/github-action-benchmark:
          tool: cargo
          output-file-path: target/criterion/output.txt
          github-token: ${{ secrets.GITHUB_TOKEN }}
          auto-push: true (main only)
          comment-on-alert: true
          alert-threshold: "105%"  # >5% regression triggers warning
          fail-on-alert: false     # non-blocking, reviewer decides
          gh-pages-branch: gh-pages
          benchmark-data-dir-path: dev/bench

3. Makefile Integration

bench-rust:  ## Run Rust criterion benchmarks
	cargo bench --bench rust_bench

4. Initialization

• First merge to main auto-creates baseline in gh-pages
• PR comments show comparison table with regression highlighting

• 5% regression marked yellow/red but does not block merge

Acceptance Criteria

• cargo bench runs criterion benchmarks covering parse_eager, parse_lazy, field access patterns
• CI job compares PR benchmarks against main branch baseline
• PRs with >5% regression on any benchmark are flagged (not auto-blocked, but visible in PR comment)
• Baseline artifacts stored in gh-pages branch

References

• benchmark-action: https://github.com/benchmark-action/github-action-benchmark
• criterion book: https://bheisler.github.io/criterion.rs/book/

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• ci: automated performance regression detection #69

🔗 Related PRs

#43 - test: add cjson and simdjson fixture coverage [merged]
#51 - perf: skip redundant validation in eager decode, inline depth check [closed]
#54 - perf: Lua encode + cache + scanner + validation stack optimizations [merged]
#99 - ci: test qjson against OpenResty runtimes [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[membphis]

Design updated after brainstorming session:

• Chose criterion + benchmark-action approach
• Baseline storage: gh-pages branch
• Core benchmarks: parse_eager, parse_lazy, field_access
• PR comment reporting with 5% regression threshold (non-blocking)

[membphis]

Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. The recommended flow is: implement tasks with fresh subagents, open a draft PR so CI starts, run consolidated reviews while CI runs, fix findings and CI failures, then mark the PR ready. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add criterion benchmarks for core parse/access paths with CI regression detection via benchmark-action.

Architecture: Expose minimal internal API via __bench_api module for benchmark use. Criterion benchmarks cover parse_eager, parse_lazy, and field_access groups. GitHub Actions workflow runs benchmarks on PR and main, stores baseline in gh-pages, posts comparison comments on PRs.

Tech Stack: criterion 0.5, benchmark-action/github-action-benchmark, gh-pages branch for baseline storage.

---

File Structure

File	Purpose
src/lib.rs	Add __bench_api module exposing Document, Options, QJSON_MODE_LAZY
Cargo.toml	Add criterion dev-dependency and bench target
benches/rust_bench.rs	Criterion benchmarks for parse_eager, parse_lazy, field_access
.github/workflows/bench.yml	CI workflow for benchmark comparison
Makefile	Add bench-rust target

---

Task 1: Expose Benchmark API

Files:

• Modify: src/lib.rs:14-21

• Step 1: Inspect existing test API pattern

Read src/lib.rs lines 14-21 to see how __test_api exposes internals.

• Step 2: Add __bench_api module

#[doc(hidden)]
pub mod __bench_api {
    pub use crate::doc::Document;
    pub use crate::options::{Options, QJSON_MODE_LAZY};
}

Add this after the existing __test_api block (around line 21).

• Step 3: Verify compilation

Run: cargo build --release
Expected: BUILD SUCCEEDED

• Step 4: Commit

git add src/lib.rs
git commit -m "feat(bench): expose internal API for criterion benchmarks"

---

Task 2: Add Criterion Dependency

Files:

• Modify: Cargo.toml:24-27

• Step 1: Add criterion to dev-dependencies

After line 27 in Cargo.toml, add:

criterion = { version = "0.5", features = ["html_reports"] }

• Step 2: Add bench target configuration

At the end of Cargo.toml, add:

[[bench]]
name = "rust_bench"
harness = false

• Step 3: Verify dependency resolution

Run: cargo check --benches
Expected: CHECK SUCCEEDED

• Step 4: Commit

git add Cargo.toml
git commit -m "build: add criterion benchmark dependency"

---

Task 3: Implement Criterion Benchmarks

Files:

• Create: benches/rust_bench.rs

• Step 1: Create benchmark file with fixture loader

use criterion::{black_box, criterion_group, criterion_main, Criterion, BenchmarkId};
use qjson::__bench_api::{Document, Options, QJSON_MODE_LAZY};
use std::fs;

fn read_fixture(path: &str) -> Vec<u8> {
    fs::read(path).unwrap_or_else(|e| panic!("failed to read {}: {}", path, e))
}

struct Fixture {
    name: &'static str,
    path: &'static str,
    data: Vec<u8>,
}

fn load_fixtures() -> Vec<Fixture> {
    vec![
        Fixture {
            name: "small_api",
            path: "benches/fixtures/small_api.json",
            data: read_fixture("benches/fixtures/small_api.json"),
        },
        Fixture {
            name: "wide_object",
            path: "tests/fixtures/data/wide_object.json",
            data: read_fixture("tests/fixtures/data/wide_object.json"),
        },
        Fixture {
            name: "deep_nesting",
            path: "tests/fixtures/data/deep_nesting.json",
            data: read_fixture("tests/fixtures/data/deep_nesting.json"),
        },
    ]
}

• Step 2: Add parse_eager benchmark group

fn bench_parse_eager(c: &mut Criterion) {
    let fixtures = load_fixtures();
    let mut group = c.benchmark_group("parse_eager");

    for f in &fixtures {
        group.bench_with_input(BenchmarkId::new("parse", f.name), &f.data, |b, data| {
            b.iter(|| Document::parse(black_box(data)).unwrap())
        });
    }

    group.finish();
}

• Step 3: Add parse_lazy benchmark group

fn bench_parse_lazy(c: &mut Criterion) {
    let fixtures = load_fixtures();
    let opts = Options { mode: QJSON_MODE_LAZY, max_depth: 0 };
    let mut group = c.benchmark_group("parse_lazy");

    for f in &fixtures {
        group.bench_with_input(BenchmarkId::new("parse", f.name), &f.data, |b, data| {
            b.iter(|| Document::parse_with_options(black_box(data), &opts).unwrap())
        });
    }

    group.finish();
}

• Step 4: Add field_access benchmark group

fn bench_field_access(c: &mut Criterion) {
    let small = read_fixture("benches/fixtures/small_api.json");
    let doc = Document::parse(&small).unwrap();
    let mut group = c.benchmark_group("field_access");

    group.bench_function("get_str/model", |b| {
        b.iter(|| {
            let mut out_ptr = std::ptr::null();
            let mut out_len = 0usize;
            unsafe {
                qjson::ffi::qjson_get_str(
                    &doc as *const _ as *mut _,
                    b"model".as_ptr() as *const _,
                    5,
                    &mut out_ptr,
                    &mut out_len,
                )
            }
        })
    });

    group.bench_function("get_f64/max_tokens", |b| {
        b.iter(|| {
            let mut out = 0f64;
            unsafe {
                qjson::ffi::qjson_get_f64(
                    &doc as *const _ as *mut _,
                    b"max_tokens".as_ptr() as *const _,
                    10,
                    &mut out,
                )
            }
        })
    });

    group.bench_function("get_str/nested", |b| {
        b.iter(|| {
            let mut out_ptr = std::ptr::null();
            let mut out_len = 0usize;
            unsafe {
                qjson::ffi::qjson_get_str(
                    &doc as *const _ as *mut _,
                    b"messages[0].role".as_ptr() as *const _,
                    16,
                    &mut out_ptr,
                    &mut out_len,
                )
            }
        })
    });

    group.finish();
}

• Step 5: Add criterion main

criterion_group!(benches, bench_parse_eager, bench_parse_lazy, bench_field_access);
criterion_main!(benches);

• Step 6: Run benchmarks locally

Run: cargo bench --bench rust_bench
Expected: Benchmarks run successfully with timing output

• Step 7: Commit

git add benches/rust_bench.rs
git commit -m "feat(bench): add criterion benchmarks for parse and field access"

---

Task 4: Fix Benchmark API (if needed)

Files:

• Modify: src/lib.rs
• Modify: src/ffi.rs

Note: Task 3 step 4 uses FFI functions which expect *mut qjson_doc, not &Document. We need to adjust the approach.

• Step 1: Expose qjson_doc wrapper in __bench_api

In src/lib.rs, update __bench_api:

#[doc(hidden)]
pub mod __bench_api {
    pub use crate::doc::Document;
    pub use crate::options::{Options, QJSON_MODE_LAZY};
    pub use crate::ffi::qjson_doc;
}

• Step 2: Add helper function for benchmark Document creation

In src/ffi.rs, add after the qjson_doc struct definition (around line 48):

impl qjson_doc {
    /// Create a qjson_doc from a Document for benchmark use.
    /// # Safety
    /// The input slice must outlive the returned qjson_doc.
    #[doc(hidden)]
    pub unsafe fn from_slice(buf: &[u8]) -> Result<Self, crate::error::qjson_err> {
        let slice: &'static [u8] = std::mem::transmute(buf);
        Document::parse(slice).map(qjson_doc)
    }

    #[doc(hidden)]
    pub unsafe fn from_slice_with_options(
        buf: &[u8],
        opts: &crate::options::Options,
    ) -> Result<Self, crate::error::qjson_err> {
        let slice: &'static [u8] = std::mem::transmute(buf);
        Document::parse_with_options(slice, opts).map(qjson_doc)
    }

    #[doc(hidden)]
    pub fn as_ptr(&mut self) -> *mut qjson_doc {
        self as *mut _
    }
}

• Step 3: Verify compilation

Run: cargo build --release
Expected: BUILD SUCCEEDED

• Step 4: Commit

git add src/lib.rs src/ffi.rs
git commit -m "feat(bench): expose qjson_doc helpers for benchmark use"

---

Task 5: Update Benchmarks to Use Safe API

Files:

• Modify: benches/rust_bench.rs

• Step 1: Update field_access benchmark

Replace the field_access benchmark with:

fn bench_field_access(c: &mut Criterion) {
    let small = read_fixture("benches/fixtures/small_api.json");
    let mut group = c.benchmark_group("field_access");

    group.bench_function("get_str/model", |b| {
        let mut doc = unsafe { qjson::__bench_api::qjson_doc::from_slice(&small).unwrap() };
        b.iter(|| {
            let mut out_ptr = std::ptr::null();
            let mut out_len = 0usize;
            unsafe {
                qjson::ffi::qjson_get_str(
                    doc.as_ptr(),
                    b"model".as_ptr() as *const _,
                    5,
                    &mut out_ptr,
                    &mut out_len,
                )
            }
        })
    });

    group.bench_function("get_f64/max_tokens", |b| {
        let mut doc = unsafe { qjson::__bench_api::qjson_doc::from_slice(&small).unwrap() };
        b.iter(|| {
            let mut out = 0f64;
            unsafe {
                qjson::ffi::qjson_get_f64(
                    doc.as_ptr(),
                    b"max_tokens".as_ptr() as *const _,
                    10,
                    &mut out,
                )
            }
        })
    });

    group.bench_function("get_str/nested", |b| {
        let mut doc = unsafe { qjson::__bench_api::qjson_doc::from_slice(&small).unwrap() };
        b.iter(|| {
            let mut out_ptr = std::ptr::null();
            let mut out_len = 0usize;
            unsafe {
                qjson::ffi::qjson_get_str(
                    doc.as_ptr(),
                    b"messages[0].role".as_ptr() as *const _,
                    16,
                    &mut out_ptr,
                    &mut out_len,
                )
            }
        })
    });

    group.finish();
}

• Step 2: Update parse benchmarks to also use from_slice

fn bench_parse_eager(c: &mut Criterion) {
    let fixtures = load_fixtures();
    let mut group = c.benchmark_group("parse_eager");

    for f in &fixtures {
        group.bench_with_input(BenchmarkId::new("parse", f.name), &f.data, |b, data| {
            b.iter(|| unsafe {
                qjson::__bench_api::qjson_doc::from_slice(black_box(data)).unwrap()
            })
        });
    }

    group.finish();
}

fn bench_parse_lazy(c: &mut Criterion) {
    let fixtures = load_fixtures();
    let opts = Options { mode: QJSON_MODE_LAZY, max_depth: 0 };
    let mut group = c.benchmark_group("parse_lazy");

    for f in &fixtures {
        group.bench_with_input(BenchmarkId::new("parse", f.name), &f.data, |b, data| {
            b.iter(|| unsafe {
                qjson::__bench_api::qjson_doc::from_slice_with_options(black_box(data), &opts).unwrap()
            })
        });
    }

    group.finish();
}

• Step 3: Verify benchmarks run

Run: cargo bench --bench rust_bench
Expected: All benchmarks run successfully

• Step 4: Commit

git add benches/rust_bench.rs
git commit -m "fix(bench): use safe qjson_doc wrapper in benchmarks"

---

Task 6: Create CI Workflow

Files:

• Create: .github/workflows/bench.yml

• Step 1: Create benchmark workflow file

name: Benchmark

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: write
  pull-requests: write

concurrency:
  group: bench-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

env:
  CARGO_TERM_COLOR: always

jobs:
  benchmark:
    name: Performance regression check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive

      - name: Install Rust (stable)
        run: |
          rustup toolchain install stable --profile minimal --no-self-update
          rustup default stable

      - name: Cache cargo registry & target
        uses: actions/cache@v4
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: bench-${{ runner.os }}-${{ hashFiles('Cargo.toml', 'Cargo.lock') }}
          restore-keys: |
            bench-${{ runner.os }}-

      - name: Run benchmarks
        run: cargo bench --bench rust_bench -- --output-format bencher | tee output.txt

      - name: Store benchmark result
        uses: benchmark-action/github-action-benchmark@v1
        with:
          tool: cargo
          output-file-path: output.txt
          github-token: ${{ secrets.GITHUB_TOKEN }}
          auto-push: ${{ github.ref == 'refs/heads/main' }}
          comment-on-alert: true
          alert-threshold: '105%'
          fail-on-alert: false
          gh-pages-branch: gh-pages
          benchmark-data-dir-path: dev/bench

• Step 2: Verify YAML syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.github/workflows/bench.yml'))"
Expected: No error output

• Step 3: Commit

git add .github/workflows/bench.yml
git commit -m "ci: add benchmark regression detection workflow"

---

Task 7: Add Makefile Target

Files:

• Modify: Makefile:25

• Step 1: Add bench-rust target

After line 25 (the .PHONY declaration), update it to include bench-rust:

.PHONY: help build test lua-property-test lua-mutation-property-test lint lua-lint bench-smoke bench bench-rust clean

Then add the target after the bench target (around line 66):

bench-rust: build ## Run Rust criterion benchmarks
	cargo bench --bench rust_bench

• Step 2: Verify make target

Run: make help | grep bench-rust
Expected: bench-rust — Run Rust criterion benchmarks

• Step 3: Commit

git add Makefile
git commit -m "build: add bench-rust Makefile target"

---

Task 8: Final Verification

Files: None (verification only)

• Step 1: Run full benchmark suite

Run: make bench-rust
Expected: All benchmarks complete with timing output

• Step 2: Verify CI workflow syntax

Run: gh workflow view bench.yml 2>/dev/null || echo "Workflow file created, will validate on push"
Expected: Either workflow details or acknowledgment message

• Step 3: Check for uncommitted changes

Run: git status
Expected: Clean working tree or only untracked files

---

Notes

• The gh-pages branch will be auto-created by benchmark-action on the first merge to main
• PR comments will show regression warnings for >5% slowdowns but won't block merge
• Local make bench-rust produces HTML reports in target/criterion/