feat(relay): serve QUIC from pinned per-core workers (M1, part 1)

[kixelated]

Milestone 1 of #2875, first of two PRs. Targets dev because moq-tokio only exists there (M2's rename landed in #2896) and because bind::udp changes signature.

What this does

runtime.workers = N moves QUIC off the shared work-stealing runtime and onto N threads, each pinned to a core, each running a current_thread runtime and owning one socket in a SO_REUSEPORT group on the listen address. A connection lands on whichever worker the kernel steers its first packet to and stays there for its whole life: its QUIC driver, its session, and its tasks never leave that thread.

Everything that is not QUIC stays on the shared runtime (web, internal, tcp/unix, clustering, signals, cert reload), matching decision 4 in the epic. The workers reach the rest of the relay through the shared Cluster, exactly as sessions on one runtime already do, so 1:N fanout across worker threads goes through the existing origin model rather than anything new.

Off by default; the shared runtime is unchanged when runtime.workers is unset.

Why the steering is split into a second PR

The kernel picks a member of a reuseport group by hashing the packet's 4-tuple. That pins a connection to one worker for as long as the client keeps its address, but a client that migrates (NAT rebinding, network change) hashes somewhere else and its packets reach a worker that has never seen that connection ID, so the connection is lost rather than migrated.

Fixing that means server CIDs that encode the owning member plus a reuseport filter that reads them, which needs a with_cid_generator addition to web-transport-quiche upstream. Splitting it out keeps this PR reviewable and isolates the number M1 actually asks for: how much does pinning alone buy, before any steering work? Until that lands, this mode is a measurement knob, which is the other reason it is not the default.

Per the discussion on mechanism: the filter in PR 2 is planned as classic BPF (SO_ATTACH_REUSEPORT_CBPF) rather than the SK_REUSEPORT + BPF_MAP_TYPE_REUSEPORT_SOCKARRAY eBPF program named in the epic. Our steering rule fits in about eight cBPF instructions, and cBPF needs no CAP_BPF, no clang or aya in the build, no map lifecycle, and no precompiled object in the repo. The trade is that the kernel selects by index % socks rather than by socket, so group membership has to be fixed at startup, which it is. eBPF stays available later if dynamic membership or richer steering turns out to matter; the steering module's API will hide which one is in use.

API

moq-tokio gains the two primitives the relay drives, rather than the relay reaching into sockets itself:

• bind::Udp, an options struct for bind::udp (the repo's "options struct, not positional parameters" rule), carrying with_reuse_port. It is a signature change, so existing bind::udp(addr) calls become bind::udp(Udp::new(addr)). I first tried impl Into<Udp> to keep those compiling, and backed it out: it breaks inference at every bind::udp("...".parse().unwrap()) call site with an E0283, which is a papercut every downstream caller would hit.
• listen::Shard, a validated slot in a reuseport group (Shard::new returns None for a slot that does not exist, so an out-of-range index is unrepresentable). Its index is unused by the kernel today but is what PR 2's CID codec encodes, so the config surface lands once rather than twice.
• listen::Listeners, selecting which of the configured listeners a Server opens. This is what lets one process build a QUIC-only Server per shard while a stream-only Server keeps tcp/unix on the main runtime. It also makes "no QUIC" expressible: previously that was only implied by configuring a stream listener, so a stream-only server with no tcp/unix bind would silently open UDP.

moq_relay::Workers is bound during Relay::load (so a port conflict is a startup error, not a worker that quietly died) and released to serve in Relay::run, because the cluster it serves into does not exist yet at bind time.

Constraints and caveats

• Linux only. bind::udp fails with Unsupported elsewhere rather than binding a group the platform will not balance. macOS and the BSDs accept SO_REUSEPORT and then deliver a unicast flow to a single member, so the workers would come up looking healthy with one of them serving everything.
• --listen-tls-generate is rejected with workers, since each worker would generate and serve a different self-signed certificate and the fingerprint endpoint publishes only one of them. Real certificate files are required. Every worker loads the same files, so worker 0's Certificates handle answers for the fingerprint endpoint, and each worker keeps its own hot-reload watcher.
• The shared runtime still sizes its pool to the machine, so --runtime-workers N on an N-core box gives N pinned threads plus a full stealing pool. Documented: set TOKIO_WORKER_THREADS to bound it. Worth revisiting if the M1 profile shows it as noise.
• Backend-agnostic. The reuseport bind goes through bind::udp, so quinn (the default), quiche, and noq all get it; PR 2's CID work is quiche-first.

Testing

just check and just test pass on macOS (3223 tests). Neither reuseport nor the integration test compiles there, so the Linux-only paths ran in a container on kernel 6.19 aarch64:

• bind::tests::udp_reuse_port_shares_a_port / udp_without_reuse_port_keeps_the_port — the group forms, and a member that forgets the option loses the port, which is what makes a missed with_reuse_port a startup failure rather than a silently lopsided group.
• bind::tests::udp_reuse_port_spreads_datagrams — 64 senders on distinct source ports across a 4-member group; asserts every datagram reached exactly one member and more than one member was fed. This is the kernel behavior the whole mode rests on, tested without paying for a QUIC handshake.
• runtime_workers::workers_serve_quic_and_share_one_origin — a real relay with 4 workers, a publisher and 4 subscribers over QUIC, each subscriber reading the frame. The subscribers are spread over threads the publisher is probably not on, so the frame has to cross through the shared origin; that crossing is the part that could deadlock or drop.

All pass. One gap: I meant to confirm the integration test is not vacuous by mutating runtime.workers to 0 (nothing else in the test serves QUIC, so the connect should time out), and the container ran out of disk before that run finished. The argument holds from the code — Workers::serve pends forever when disabled and the test spawns nothing else — but it is reasoned, not observed.

Cross-package sync

moq-relay config change → doc/bin/relay/config.md (new [runtime] section). No wire format change, so no drafts/ update. No moq-ffi surface change, so no binding rows apply. moq-bench is unaffected; the M0 suite runs against this mode unchanged.

Closes nothing on its own; #2875 stays open through M1 part 2.

🤖 Generated with Claude Code

(written by Opus 5)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a85de2680

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Shut down workers when the serve future is dropped

If an embedder cancels Workers::serve, or Relay::run returns because the cluster, web, internal, or shared server branch fails, these detached threads retain their servers and continue accepting on the QUIC sockets indefinitely. The discarded JoinHandle and lack of a cancellation path also prevent the caller from reclaiming the port or reliably restarting the relay in-process. Keep an owned shutdown/join guard whose Drop stops every started worker. rs/CLAUDE.mdL128-L128 (Written by GPT-5.6 Sol)

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Bind all shards to the first allocated port

When listen.bind uses port 0, each call to shard_config preserves :0, so Linux allocates a different ephemeral port for every worker instead of forming one reuseport group. Only the first address is stored and exposed through Relay::addr, leaving the remaining workers unreachable at that address and defeating the configured pool; bind the first shard, then use its selected port for subsequent shards. (Written by GPT-5.6 Sol)

Useful? React with 👍 / 👎.