Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ claude-output
.mcpregistry_*
/.testmondata
.benchmarks/
/.scratch/

# Consolidated package build artifacts
/integrations/openclaw/node_modules/
Expand Down
49 changes: 49 additions & 0 deletions benchmarks/docs/read-load-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Read-path load benchmark

This benchmark measures repeated, direct-permalink `read_note` calls through the real
`bm mcp` stdio server. It compares an authoritative warm baseline with the same workload after
the standalone Redis read cache is warmed.

## Workload

- deterministic 1, 16, and 64 KiB Markdown notes;
- 32 distinct notes per size;
- 128 measured reads at concurrency 1, 8, 32, and 64;
- corpus materialization, indexing, connection setup, and cache warmup outside measurement;
- isolated Basic Memory config, database, project, and home directories for every run;
- JSONL output with p50, p95, p99, throughput, response bandwidth, errors, and workload metadata;
- a `manifest.json` beside each result with benchmark and Basic Memory SHAs, provider versions,
dirty-worktree state, synthetic-corpus checksum, and runtime configuration.

Every inherited `BASIC_MEMORY_*` setting is removed before the harness adds its explicit isolated
configuration. The cached run sets `BASIC_MEMORY_REDIS_URL` only in the spawned MCP process. Output
records whether Redis was enabled without recording the URL, because URLs may contain credentials.

The Basic Memory SHA comes from the `basic_memory` module imported by the Python environment behind
`--bm-command`, not from the console script's parent directory. Publishable per-ref environments
must therefore use an editable install linked to the source checkout; the harness fails rather
than attributing a wheel or unrelated enclosing repository to the wrong SHA.

## Run a paired comparison

Use the same Redis server for the cached repetitions, but use a distinct scratch directory for
every run. The server must be ready before starting the benchmark; its startup time is not part
of the measurement.

```bash
just bench-read-cache redis://127.0.0.1:6379/0 run-01
```

The recipe writes each side's `results.jsonl` and `manifest.json` under
`.scratch/read-load-{authoritative,redis-warm}-run-01/`, then prints the Markdown comparison. Give
each repetition a distinct run ID so its corpus, results, and provenance remain available. Use
`just bench-read-load <label> [redis_url]` when running one side independently.

Use `just bench-read-smoke [redis_url]` for a tiny real-MCP run that verifies result and manifest
generation before committing to the full matrix.

Latency is evidence, not a CI threshold. Use at least six paired repetitions before making a
performance claim, alternate run order, and discard runs with material host contention. The
manifest must report clean benchmark and Basic Memory worktrees for a publishable comparison. The
real-Redis integration suite remains the correctness gate for cache identity and invalidation.
Run that gate with `just test-read-cache`.
Loading
Loading