Skip to content

bench: add cold skill load macrobenchmark#31295

Draft
anp-oai wants to merge 1 commit into
codex/test-app-server-rpc-delayfrom
codex/durable-executor-skill-benchmark
Draft

bench: add cold skill load macrobenchmark#31295
anp-oai wants to merge 1 commit into
codex/test-app-server-rpc-delayfrom
codex/durable-executor-skill-benchmark

Conversation

@anp-oai

@anp-oai anp-oai commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Why

We need a durable end-to-end measurement for cold remote skill discovery under realistic same-region RPC latency.

What

  • add a Bazel-only Divan macrobenchmark for cold skill loading
  • seed 1, 8, or 64 short project skills outside the timed thread/start request
  • trust synthetic roots and disable unrelated plugin startup work
  • run app-server against the fixed-delay remote-executor fixture through its public RPC API

Validation

  • bazel test --compilation_mode=fastbuild --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=no --@rules_rust//rust/settings:extra_exec_rustc_flag=-Cdebug-assertions=no --cache_test_results=no --test_output=streamed --test_arg=--test //codex-rs/app-server:cold-skill-load-bench

Stack

  1. #31425 test: add TestAppServer builder
  2. #31427 test: add delayed exec-server transport
  3. #31295 bench: add cold skill load macrobenchmark
  4. #31428 bench: add e2e benchmark entrypoints
  5. #31429 ci: smoke Bazel e2e benchmarks

anp-oai added a commit that referenced this pull request Jul 6, 2026
@anp-oai
codex: fix CI failure on PR #31295
04b3763
anp-oai added a commit that referenced this pull request Jul 6, 2026
@anp-oai
codex: fix CI failure on PR #31295
dae98b8
anp-oai added a commit that referenced this pull request Jul 6, 2026
@anp-oai
codex: fix CI failure on PR #31295
44e6a99
anp-oai added a commit that referenced this pull request Jul 6, 2026
@anp-oai
codex: fix CI failure on PR #31295
e357ff1
anp-oai added a commit that referenced this pull request Jul 6, 2026
@anp-oai
codex: fix CI failure on PR #31295
557b7c9
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from e357ff1 to 557b7c9 Compare July 6, 2026 23:23
anp-oai added a commit that referenced this pull request Jul 6, 2026
@anp-oai
codex: fix CI failure on PR #31295
8875647
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 557b7c9 to 8875647 Compare July 6, 2026 23:56
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
aef0c66
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 8875647 to aef0c66 Compare July 7, 2026 00:41
@github-actions github-actions Bot mentioned this pull request Jul 7, 2026
Open
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
790039d
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from aef0c66 to 790039d Compare July 7, 2026 01:02
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
9cca043
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 790039d to 9cca043 Compare July 7, 2026 02:05
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
3e6ae1a
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 9cca043 to 3e6ae1a Compare July 7, 2026 02:38
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
1e6a517
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 3e6ae1a to 1e6a517 Compare July 7, 2026 03:10
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
a2119d2
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 1e6a517 to a2119d2 Compare July 7, 2026 03:19
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
47f0e8c
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from a2119d2 to 47f0e8c Compare July 7, 2026 03:39
@anp-oai anp-oai mentioned this pull request Jul 7, 2026
Merged
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
633e020
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 47f0e8c to 633e020 Compare July 7, 2026 04:09
@github-actions github-actions Bot mentioned this pull request Jul 7, 2026
Open
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
65775a0
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 633e020 to 65775a0 Compare July 7, 2026 04:41
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
782e981
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from ae49886 to 0f8b670 Compare July 7, 2026 05:59
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
f6d7cc0
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 0f8b670 to f6d7cc0 Compare July 7, 2026 06:25
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
b9b0dfd
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from f6d7cc0 to b9b0dfd Compare July 7, 2026 06:39
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
codex: fix CI failure on PR #31295
9a9b320
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch 2 times, most recently from 9a9b320 to 38360b6 Compare July 7, 2026 16:20
@anp-oai anp-oai changed the title bench: add delayed cold thread start benchmark bench: add cold skill load macrobenchmark Jul 7, 2026
@anp-oai anp-oai changed the base branch from main to codex/test-app-server-rpc-delay July 7, 2026 16:20
This was referenced Jul 7, 2026
Draft
Draft
Merged
Merged
Draft
@anp-oai anp-oai force-pushed the codex/test-app-server-rpc-delay branch from a9701fb to 97c4357 Compare July 7, 2026 17:31
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch 2 times, most recently from 726589b to e6f4113 Compare July 7, 2026 17:35
@anp-oai anp-oai force-pushed the codex/test-app-server-rpc-delay branch from 97c4357 to ede6cf5 Compare July 7, 2026 18:17
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from e6f4113 to ecf0c9c Compare July 7, 2026 18:18
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
test: generalize exec-server fixture (#31422)
f158b31 
## Why

Remote-executor integration tests need one host-agnostic exec-server
fixture target instead of a Windows-only wrapper.

## What

- rename the testing binary target to exec-server
- make the fixture source and target host-agnostic
- update Windows remote-executor test wiring to use the shared target

## Validation

- bazel build //codex-rs/exec-server/testing:exec-server
- bazel cquery --config=ci-windows-cross
'set(//codex-rs/exec-server/testing:exec-server
//codex-rs/core/tests/remote_env_windows:smoke-test)'

## Stack

1. [#31422 test: generalize exec-server
fixture](#31422)
2. [#31425 test: add TestAppServer
builder](#31425)
3. [#31427 test: add delayed exec-server
transport](#31427)
4. [#31295 bench: add cold skill load
macrobenchmark](#31295)
5. [#31428 bench: add e2e benchmark
entrypoints](#31428)
6. [#31429 ci: smoke Bazel e2e
benchmarks](#31429)
@anp-oai anp-oai force-pushed the codex/test-app-server-rpc-delay branch from ede6cf5 to 36ec068 Compare July 7, 2026 18:37
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from ecf0c9c to 8ac3c16 Compare July 7, 2026 18:38
@anp-oai anp-oai force-pushed the codex/test-app-server-rpc-delay branch from 36ec068 to ec6b04b Compare July 7, 2026 19:05
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 8ac3c16 to 0079a34 Compare July 7, 2026 19:05
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
test: add TestAppServer builder (#31425)
9222493 
## Why

Test callers need one composable way to create app-server fixtures
instead of a growing family of overlapping constructor implementations.

## What

- add a feature-complete TestAppServer::builder()
- make the default builder own a temporary CODEX_HOME and select the
automatic test environment
- expose builder knobs for no automatic environment, explicit
CODEX_HOME, program, arguments, plugin startup tasks, environment
overrides, managed config, and JSON logging
- keep the existing public constructor surface, but route every
constructor through the builder so the new path is exercised immediately
- remove the redundant private constructor ladders; caller migration and
public constructor removal live in the optional cleanup stack

## Validation

- just test -p codex-app-server (940/941 before updating the expected
builder error wording)
- just test -p codex-app-server
auto_env_rejects_explicit_environment_config
- just fix -p codex-app-server
- just fmt

## Follow-up stacks

Cleanup, optional for the benchmark work:

1. [#31451 test: migrate TestAppServer callers to
builder](#31451)
2. [#31452 test: remove TestAppServer
constructors](#31452)

Benchmark infrastructure:

1. [#31427 test: add delayed exec-server
transport](#31427)
2. [#31295 bench: add cold skill load
macrobenchmark](#31295)
3. [#31428 bench: add e2e benchmark
entrypoints](#31428)
4. [#31429 ci: smoke Bazel e2e
benchmarks](#31429)
@anp-oai anp-oai force-pushed the codex/test-app-server-rpc-delay branch from ec6b04b to 0db9962 Compare July 7, 2026 22:55
@anp-oai anp-oai force-pushed the codex/durable-executor-skill-benchmark branch from 0079a34 to ce11104 Compare July 7, 2026 22:55
anp-oai added a commit that referenced this pull request Jul 7, 2026
@anp-oai
perf(skills): resolve plugin namespaces per root (#31348)
58b66d3 
## Why

Loading skills from a remote executor can add a lot to thread start time
when there are many skills. Previous changes added some concurrency for
the file reads themselves, but we're still bottlenecked on the initial
root path discovery.

Using the benchmark from
[#31295](#31295), this change
reduces the measured mean of loading 66 skills about 71%. Behavioral
coverage lands separately in
[#31369](#31369).

## What

- resolve the scanned root's inherited namespace once
- resolve discovered nested plugin roots once, retaining
nearest-valid-ancestor behavior
- pass an explicit resolved Plain / Plugin namespace into skill parsing
instead of probing per skill
- preserve explicitly provided plugin namespaces as the highest-priority
source
- reuse the same resolver for environment skills, deleting its duplicate
root-probe and ancestor-selection path

## Validation

- just test -p codex-core-skills namespace: 12 passed on both the parent
and optimized branches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@anp-oai