bench: pilot-bench overlay throughput benchmark harness - #445
Draft
TeoSlayer wants to merge 1 commit into
Draft
Conversation
Repeatable overlay throughput measurement to gate the transport-efficiency work: 'run --label baseline' records reference numbers, 'run' + 'compare' diff any later run against them, per node and fleet-median. Per node it captures ping RTT, round-trip echo goodput and completion % (pilotctl bench --json), and live transport counters (cwnd min/max, fast_retx, srtt) sampled from pilotctl info --json during the transfer, so a slow run also records why it was slow. Results are timestamped JSON under ~/.pilot/bench-results, stdlib-only Python. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TeoSlayer
pushed a commit
that referenced
this pull request
Aug 2, 2026
…ery crawl) Fleet benchmarks showed ~0.2 MB/s best-case round-trip goodput with multi-minute stalls on 1 MB echo transfers (fleet median 0.068 MB/s, PR #445 baseline). Live conn-stats sampling traced it to four compounding defects, fixed here: 1. Loss amplification: MaxOOOBuf (128 segs) < MaxCongWin (256 segs), so one lost segment with a full window in flight made the receiver silently drop every later in-window segment. MaxOOOBuf is now 512 (= RecvBufSize, covers the whole 2 MB receive window). 2. Recovery crawl: timeout-based recovery had no partial-ACK retransmit path (RFC 6582 step 6a only ran in fast recovery), leaving multi- segment losses to the one-per-RTO timer with RTO backed off to 10 s (~1.3 KB/s observed). Partial ACKs in timeout recovery now retransmit ACK-clocked out of slow start, and both recovery modes use the new retransmitLost: up to a cwnd of un-SACKed segments below the SACK frontier per ACK event, capped at maxRetxBurst=32. A 180-segment hole now drains in ~15 RTTs (~3 s) instead of up to 30 min. 3. Slow-start overshoot: initial SSThresh of MaxCongWin/2 (512 KB) let slow start double straight into a path-collapsing ~550 KB burst. InitialSSThresh is now 128 KB; congestion avoidance probes beyond it. 4. Echo service died silently on backpressure: handleEchoConn treated ErrSendBufFull (routine when data arrives faster than cwnd drains) as fatal, so bulk echoes returned 0 bytes while inbound kept ACKing (observed against pilot-mom). Echo now writes through connAdapter, which blocks-and-retries per its v1.9.1 semantics. All changes are endpoint-local heuristics — no wire format change, safe in a mixed-version fleet. pkg/daemon suite green incl. all RFC congestion tests; in-repo CC simulation improves 20-35% at 0-1% loss; new zz tests pin each fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Repeatable throughput benchmark for the overlay, built to gate the upcoming transport-efficiency work: record a baseline once, then
run+compareafter each change to see per-node and fleet-median deltas.Per node it captures ping RTT, round-trip echo goodput + completion % (
pilotctl bench --json), and live transport counters (cwnd min/max, fast_retx, srtt) sampled frompilotctl info --jsonduring the transfer — so a slow run also records why it was slow. Results are timestamped JSON under~/.pilot/bench-results/. Python stdlib only.Baseline (2026-08-02, laptop → fleet, 1 MB echo, 2 trials/node)
Fleet median 0.068 MB/s · best 0.206 MB/s. The bimodal pattern confirms the diagnosis: trials where cwnd stays ≤ ~64 KB complete at ~0.2 MB/s (the window/RTT limit at 214 ms SRTT); trials where slow start overshoots to ~500 KB blow past MaxOOOBuf (128 segs) and collapse into the 1-per-RTO recovery crawl. Theoretical ceiling with a safe 512 KB window: ~2.4 MB/s (~10×).
Fix list these numbers gate
🤖 Generated with Claude Code