WIP - feat(storage): add snapshot/fork benchmark mode#145
Open
kisernl wants to merge 5 commits into
Open
Conversation
Add a new `snapshot-fork` benchmark mode that measures snapshot creation, fork-from-snapshot, and fork-from-live timings against the storage providers, with read-back verification and best-effort teardown of every created object/snapshot/fork. - src/storage/snapshot-fork-benchmark.ts: per-iteration seed -> snapshot -> fork(snapshot) -> fork(live) -> verify -> cleanup - src/storage/snapshot-fork-types.ts: small/wide/deep dataset presets to separate per-object overhead from bytes-copied cost - src/storage/stats.ts: extract shared median/p95/p99 helpers out of benchmark.ts and reuse in both storage benchmarks - src/run.ts: wire up --mode snapshot-fork and --dataset, results output - package.json: bench:snapshot-fork scripts - CI: PR-only snapshot-fork job (single iteration, small dataset) across aws-s3, cloudflare-r2, tigris; skips gracefully without secrets NOTE: untested — the snapshot/fork paths have not been run against any live provider yet. Needs validation before relying on results.
Contributor License AgreementAll contributors are covered by a CLA. |
kisernl
changed the title
Contributor
Browser Benchmark Results
View full run · SVG available as build artifact |
Contributor
Sandbox Benchmark ResultsSequential
Staggered
Burst
View full run · SVGs available as build artifacts |
superagent-security
Bot
added
the
pr:flagged
![superagent-security[bot]](https://avatars.githubusercontent.com/in/3287076?s=60&v=4){kind=small}
| # smallest dataset to keep cost and leak risk minimal. Fork PRs without | ||
| # secrets skip gracefully (missing-creds path) rather than failing. | ||
| if: github.event_name == 'pull_request' | ||
| runs-on: namespace-profile-default |
There was a problem hiding this comment.
P0: New workflow job runs on self-hosted runner for pull_request events
New PR job runs on a self-hosted runner, which fork PRs can reach.
Restrict the runner group to disallow public fork jobs, or use GitHub-hosted runners for PRs.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/storage-benchmarks.yml">
<violation number="1" location=".github/workflows/storage-benchmarks.yml:108">
<priority>P0</priority>
<title>New workflow job runs on self-hosted runner for pull_request events</title>
<evidence>The snapshot-fork job uses runs-on: namespace-profile-default with if: github.event_name == 'pull_request'. Self-hosted runners are reachable by fork PRs under the pull_request trigger, and runner state is shared across jobs. This has led to critical compromises (e.g., PyTorch).</evidence>
<recommendation>Verify in GitHub Settings that the namespace-profile-default runner group is restricted to not accept jobs from public forks. If fork PRs must run benchmarks, consider using GitHub-hosted runners (ubuntu-latest) for the fork-PR path, or gate the job behind an approval requirement.</recommendation>
</violation>
</file>
Contributor
Browser Throughput Benchmark Results
View full run · SVG available as build artifact |
Contributor
Storage Benchmark Results1MB Files
4MB Files
10MB Files
16MB Files
View full run · SVGs available as build artifacts |
Wire the snapshot/fork benchmark into the collect job so results show up on PRs and get committed to results/ on scheduled runs, matching how the storage benchmark is handled. - collect now depends on the snapshot-fork job and downloads its per-provider artifacts into a separate sf-artifacts/ dir (kept apart so merge-results.ts doesn't try to parse them as storage results) - post a dedicated "Snapshot/Fork Benchmark Results" PR comment (its own marker, find-or-update) rendered from those artifacts - merge-results.ts: add --mode snapshot-fork to combine per-provider artifacts into results/snapshot-fork/<dataset>/, deduped by provider with scores recomputed (absolute latency ceiling, so no cross-provider normalization needed) - run snapshot-fork on schedule/dispatch (not just PRs) with low, event-scaled iteration counts, clear stale checked-out results, and stage results/snapshot-fork/ in the existing commit-and-push step
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
superagent-security
Bot
removed
the
pr:flagged
Contributor
Snapshot/Fork Benchmark Resultssmall dataset
|
The Tigris adapter wraps unmapped failures in a StorageError whose message is just the code (e.g. "Provider"), with the real error in `.cause`. The benchmark only recorded `err.message`, so CI logged the uninformative "FAILED: Provider" and discarded the underlying cause. Add a `formatError` helper that walks the cause chain and prepends each error's code, and use it at the error-capture sites in the snapshot/fork and storage benchmarks so failures are self-diagnosing.
…hot-fork Tigris snapshots require a Standard-tier, snapshot-enabled bucket, which the default upload/download bucket is not. Add an optional per-provider snapshotFork override (bucket/credentials/requiredEnvVars) applied only in snapshot-fork mode, and point Tigris at TIGRIS_SNAPSHOT_* env vars. Wire the new secrets into the snapshot-fork CI job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new
snapshot-forkbenchmark mode that measures snapshot creation, fork-from-snapshot, and fork-from-live timings against the storage providers, with read-back verification and best-effort teardown of every created object/snapshot/fork.