Skip to content

ci: measure Windows Bazel test bottlenecks#31339

Draft
anp-oai wants to merge 6 commits into
mainfrom
codex/windows-bazel-bottleneck
Draft

ci: measure Windows Bazel test bottlenecks#31339
anp-oai wants to merge 6 commits into
mainfrom
codex/windows-bazel-bottleneck

Conversation

@anp-oai

@anp-oai anp-oai commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Why

Measure what makes the Windows Bazel test shards slow under forced test execution, without treating shard assignment as the variable.

What

  • bust selected test-action caches per uploaded commit with CODEX_BAZEL_TEST_CACHE_BUST=${GITHUB_SHA}
  • collect 15-second CPU, processor-queue, disk, network, and memory counters
  • upload full Bazel profiles
  • compare the existing 4 local test jobs, RUST_TEST_THREADS=2, 8 local test jobs, and 12 local test jobs
  • latest head is the terminal 12-way experiment, not a merge-ready configuration

Findings

  • 4-way baseline: slow shard Bazel elapsed 835.5s; average CPU 25.3%
  • two Rust harness threads: small and inconsistent movement
  • 8 local test jobs: all shards improved; slow shard 609.3s (-27%), with average CPU 40.4%
  • 12 local test jobs: regressed; slow shard 727.4s and processor queues reached the hundreds
  • network throughput, disk latency, and available memory did not show sustained saturation; the avoidable bottleneck is the 4-way local test-action cap, with 8 near the observed knee

Manual validation

  • yq '.' .github/workflows/bazel.yml
  • git diff --check
  • just test-github-scripts
  • just fmt

anp-oai added a commit that referenced this pull request Jul 7, 2026
## Why

Windows Bazel test shards currently cap local test actions at 4.
Controlled forced-test-execution measurements in #31339 found that 8
local jobs reduced Bazel elapsed time on every shard by 19–29%,
including the slow shard from 835.5s to 609.3s, while 12 jobs regressed.

## What

Set `common:ci-windows-cross --local_test_jobs=8` in `.bazelrc`.

## Manual validation

- `git diff --check`
- `just fmt`
- [8-job CI
experiment](https://github.com/openai/codex/actions/runs/28838387129)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant