daemon: fix transport throughput collapse (loss amplification + recovery crawl)

[TeoSlayer]

Problem

Fleet benchmarks (baseline in #445) showed round-trip goodput of 0.068 MB/s fleet median / 0.206 MB/s best case on 1 MB echo transfers, with multi-minute stalls and transfers that never complete — on paths whose window/RTT ceiling is ~2.4 MB/s. Live conn_list sampling during transfers traced the collapse to four compounding defects.

Fixes

1. Loss amplification — MaxOOOBuf 128 → 512 (ports.go)
The receiver's out-of-order buffer (128 segs) was half the max congestion window (256 segs). One lost segment with a full window in flight ⇒ every later in-window segment silently dropped (DeliverInOrder bound), turning 1 loss into ~130. Now 512 = RecvBufSize, covering the whole 2 MB advertised receive window. New static test pins MaxOOOBuf×MSS ≥ MaxCongWin.

2. Recovery crawl — ACK-clocked retransmission in timeout recovery (ports.go)
Timeout-based recovery had no partial-ACK retransmit path (RFC 6582 step 6a only ran under FastRecovery), so after a window collapse the only driver was the 1-segment-per-RTO timer with RTO backed off to 10 s — observed live draining ~180 lost segments at ~1.3 KB/s (segs_sent frozen, unacked −1 per ~3 s). Partial ACKs in both recovery modes now call the new retransmitLost: up to a cwnd of un-SACKed segments below the SACK frontier (known-lost, RFC 6675 IsLost rationale) per ACK event, capped at maxRetxBurst=32. A 180-segment hole drains in ~15 RTTs (~3 s at 214 ms) instead of up to tens of minutes — pinned by TestTimeoutRecoveryHoleDrainsInBoundedRounds.

3. Slow-start overshoot — initial SSThresh 512 KB → 128 KB (ports.go)
Initial ssthresh of MaxCongWin/2 let slow start double straight into a ~550 KB single-RTT burst (observed: cwnd 40 KB → 525 KB → mass loss → collapse to 8 KB). Slow start now hands over to congestion avoidance at 128 KB and probes beyond it at ~1 MSS/RTT.

4. Echo service silently died on backpressure (services.go)
handleEchoConn called SendData raw and treated ErrSendBufFull — routine transient backpressure during any bulk transfer — as fatal, killing the echo loop while the transport kept ACKing inbound. This is why pilot-mom echoed 0 bytes in every bench. Echo now writes through connAdapter.Write, which blocks-and-retries with the v1.9.1 semantics net.Conn callers already get, and still fails on real errors (conn closed, 30 s deadline).

Compatibility

All changes are endpoint-local sender/receiver heuristics — no wire format change, safe to roll out incrementally across a mixed-version fleet. Benefits are per-endpoint: an upgraded sender recovers fast against an old receiver; an upgraded receiver stops amplifying loss for old senders; full gains need both ends.

Verification

• pkg/daemon suite green (incl. all RFC 5681/6582/3465 congestion tests) + go vet
• In-repo CC simulation (TestThroughputReport): 20–35% faster at 0–1% loss, no regression at 5%
• 7 new regression tests pin each fix
• tests/ has 3 pre-existing failures on pristine origin/main in this env (registry/hostname, unrelated)
• Real-fleet numbers require deployed daemons on both ends — verify post-deploy with bench/pilot-bench.py compare (bench: pilot-bench overlay throughput benchmark harness #445) against the recorded baseline

🤖 Generated with Claude Code

[TeoSlayer]

Local A/B verification — built daemons, emulated path

Built cmd/daemon from this branch and from origin/main, ran isolated pairs (own registry+beacon, hermetic $HOME, throwaway identities) with a lossy/delayed UDP proxy in the data path (50 ms one-way delay ≈ real fleet RTT), 1 MB echo bench, 3 trials per config, 90 s deadline:

Path loss	main (old)	this branch (new)
0%	0.08–1.7 MB/s, one 12 s send	1.0 MB/s ×2, one slow cold-start — all 100%
2%	1/3 dead at 90 s (72% echoed), others 0.6 MB/s	0.25–1.04 MB/s — all 100%
5%	2/3 dead at 90 s (31% / 84% echoed)	0.23–0.67 MB/s — all 100%

Old binary: 3/9 trials hit the deadline incomplete at ~0.005 MB/s — the exact fleet pathology (stalled echo, partial completion) reproduced locally. New binary: 9/9 trials complete, goodput up to 1.05 MB/s through a single-threaded Python proxy.

The rig also caught a real regression in the first commit, fixed in 90a8ed5: ACK-clocked retransmissions were inflating the attempts counter that the RTO tick uses as the give-up RST budget, so a spurious RTO + partial ACKs could RST a live connection in ~1.6 s. The budget now counts only RTO-timer retransmissions (rtoAttempts), reset on any ACK progress; two new tests pin it.

Full daemon suite green after both commits. Rig scripts: ~/.claude/jobs/ef0e14ef/tmp/{run_ab.sh,udpproxy.py} + untracked cmd/testnet in the worktree.

[TeoSlayer]

Round 2: window scaling (f05ecc7) — sustained throughput 4.07 MB/s

1 MB/s wasn't great — it was the window math: 128 KB slow-start cap ÷ 100 ms rig RTT ≈ 1.3 MB/s, exactly what we measured. With recovery now robust, the windows scale safely:

MaxCongWin 1→4 MB, recv window + OOO buffer to match (4 MB), InitialSSThresh 512 KB, send queue covers a full cwnd.

8 MB echo through the 50 ms proxy (zero loss):

binary	goodput	completion
main	0.005 MB/s	gave up at 1.6%
this branch	4.07 MB/s round-trip	100% ×2 (steady-state trials)

That's window-limited (8 MB in ~1.96 s both directions) — ~4× the previous branch ceiling, ~800× main on this transfer size. 1 MB transfers stay ~1 MB/s because they live entirely in connection setup + slow start.

Known residual + roadmap (not this PR):

1. First transfer on a fresh tunnel rides the relay until the direct path flips (direct path silent, flipping to relay) and the relay leg is slow — relay-side pacing/buffers in rendezvous/beacon are the next server-side bottleneck.
2. Fleet RTT (~214 ms) is the other half of the equation — direct-path/NAT-punch success rate improvements multiply throughput for free.
3. Static SSThresh should eventually be replaced by pacing + HyStart-style slow-start exit, and Reno CA by CUBIC-style growth, to find per-path capacity instead of constants.
4. 4 KB virtual MSS = high per-segment overhead; batching segments per datagram would cut crypto+syscall cost.

pkg/daemon suite green after the change.

[TeoSlayer]

Round 3: HyStart-style adaptive slow-start exit (b4ee407) — 11.9 MB/s

Empirically confirmed the round-2 ceiling: 8 MB transfers ran at exactly InitialSSThresh ÷ RTT — the window sat pinned at the static 512 KB exit because Reno CA only adds 1 MSS/RTT beyond it. The constant was the ceiling.

Now slow start exits on measured RTT rise (RFC 9406 simplified: round-over-round min-RTT + eta with ≥8 samples) — the window rides to whatever the path actually supports, and the static threshold is just a backstop at MaxCongWin/2.

Rig progression (steady-state round-trip goodput, 50 ms one-way proxy):

stage	8 MB	16 MB
main (stock)	dead at 1.6%	—
+ recovery fixes	~1 MB/s ceiling	—
+ 4× windows	4.07 MB/s	—
+ HyStart exit	8.4 MB/s	11.9 MB/s (16 MB echoed in 1.35 s)

~60× the original 0.2 MB/s fleet best-case, ~2400× the collapse mode. Next ceilings in order: the 4 MB window itself (~20 MB/s echo at 100 ms RTT), then per-segment overhead (4 KB MSS × per-packet AES-GCM/syscall — loopback tops at ~50-60 MB/s), and on the real fleet the relay leg + RTT (still the dominant fleet factor — first trials on a fresh tunnel ride the relay at <0.5 MB/s until the direct path flips, reproduced every run).

pkg/daemon suite green; TestHyStartExitsSlowStartOnRTTRise pins the behavior.