ci: run the IPv6-only e2e job against the full IPv6 stack [DO NOT MERGE — signal only] - #1065
Draft
Yuan Gao (ygao-g) wants to merge 21 commits into
Draft
ci: run the IPv6-only e2e job against the full IPv6 stack [DO NOT MERGE — signal only]#1065Yuan Gao (ygao-g) wants to merge 21 commits into
Yuan Gao (ygao-g) wants to merge 21 commits into
Conversation
On a fresh IP_FAMILY=ipv6 cluster nothing resolves from inside a pod and no actor boots: CoreDNS inherits the node's IPv4 resolver, which a v6-only pod cannot reach, and "kind-registry" NXDOMAINs in atelet's own netns. Point the forward at an IPv6 upstream, overridable with IPV6_DNS_UPSTREAM, and give the registry its own server block, so it is asked for nothing but its own name. IPv4 and dual-stack clusters are unchanged, and atenet-egress still crashloops on v6-only for an unrelated Envoy bind bug. Asking once was not enough to prove that: about half of fresh clusters do not answer the first query, and a pod that goes unanswered stays unanswered, so the check re-asks with a new pod and prints what the pod saw when it gives up. It lives in hack/verify-ipv6-dns.sh rather than inline, because the registry block records an address the registry can move off and there was no way to re-check a cluster without rebuilding it.
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and on nothing at all for IPv6. Each primary socket now carries an additional "::" address on the same port. Ipv4Compat stays false on the additional address: clearing IPV6_V6ONLY would collide with the primary already bound to that port. Leaving the primary alone is what keeps an IPv4-only cluster unchanged -- with the caveat that a node lacking AF_INET6 entirely could not bind "::" and the listener would not come up. First of three commits binding atenet's gateways dual-stack. (cherry picked from commit 501991d)
The Envoy admin socket bound 0.0.0.0, and the atenet-router Service carried no ipFamilyPolicy -- which the API server defaults to SingleStack, one IPv4 ClusterIP and nothing else. Between them the router had no IPv6 address to answer on. The socket now binds "::" with ipv4_compat, one socket for both families, and the Service asks for PreferDualStack. bootstrap.v3.Admin takes a single address and has no additional_addresses, so the ingress listeners' shape is not available here; ipv4_compat is what makes the one socket serve both families. It is load-bearing: dataplane.go health-checks the admin listener over http://127.0.0.1:9901/ready, so a bare "::" would report the dataplane component of /statusz unhealthy. Prefer, not Require, keeps the Service valid on a single-stack cluster; spec.ipFamilies is left alone because the primary family is immutable and the API server appends the secondary itself. (cherry picked from commit 2a21292a3262d4a642c723550881d253737e410d)
The gateway's admin and :443 sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet probed the pod on its only address and atenet-egress crashlooped -- Envoy started fine and logged "admin address: 0.0.0.0:15000" -- while an actor's CONNECT had no v6 path in. Both sockets now bind "::" with ipv4_compat, and the Service asks for PreferDualStack so a dual-stack cluster hands out an IPv6 ClusterIP to reach them on. One socket here rather than the ingress listeners' pair: IPv4 peers then arrive as ::ffff: addresses, and nothing on this path reads the peer -- actor identity comes from the client certificate and the access log records the cert SAN. ipv4_compat also has to stay on the admin socket, because the ext-proc sidecar's drainer dials 127.0.0.1:15000 and envoydrain.go reads a refusal there as "Envoy already exited", skipping the drain silently. Last of three. (cherry picked from commit 2549657)
Both gateway admin sockets bind "::" with ipv4_compat, and the flag is what keeps their in-pod callers working: dataplane.go health-checks the router's over IPv4 loopback, and envoydrain.go dials the egress one the same way and reads a refusal as "Envoy already exited", skipping the drain without reporting an error. No Go test, golden file, or verify script read either manifest, so dropping the flag would have failed silently. make verify now rejects an admin socket that binds "::" without it. (cherry picked from commit 4b478a0ead243a0f8f77af9e2ed969996874d2cd)
The CONNECT-terminating listeners landed after the first commit of this series, so they kept a bare 0.0.0.0 socket while ingress HTTP and HTTPS gained their "::" pair. Give them the same additional address, so all four of the router's socket listeners answer on both families. Both are port-gated and no e2e suite configures them yet, which is why nothing caught this; the internal main_internal listener has no socket and needs nothing. (cherry picked from commit 54c727d22843556dfcbd06a4910087efe2114ba2)
TCPOriginalDestination read only the IPv4 SOL_IP/SO_ORIGINAL_DST, so an actor's IPv6 connection redirected into the transparent egress listener had no destination to dial and the proxy failed it. Read IP6T_SO_ORIGINAL_DST too, falling back to it only when the IPv4 lookup returns ENOENT, so unrelated IPv4 failures keep their own error. One step towards dual-stack actor networking; the actor veth and its nftables rules are still IPv4-only. Co-authored-by: Yuan Gao <ypgao@google.com> (cherry picked from commit d8527b5e8ea24e588f694da726ee76784d11c41b)
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 19, 2026 18:34
95624a2 to
bbc44ca
Compare
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
2 times, most recently
from
August 21, 2026 01:00
efaf5ee to
4da3798
Compare
Both ateom herders defaulted the actor ingress flags to "0.0.0.0:443" and "0.0.0.0:444", which reads as IPv4-only. It never was: Go treats an unspecified address as a wildcard and binds it dual-stack, so the sockets already served both families. Spell the defaults ":443" and ":444" so the flag says what it does, and note why in a comment. Part of the dual-stack actor networking series; no behavior change. (cherry picked from commit 51b2cbe)
EnableIPv4Forwarding now also writes /proc/sys/net/ipv6/conf/all/forwarding so actor IPv6 traffic (including DNS queries) is routed between the actor veth and pod eth0 instead of being dropped by ip6_forward() on dual-stack / IPv6-only clusters. Factor the sysctl write into writeSysctlIfUnset preserving the original read-only remount/restore behavior, and add unit coverage for its fast paths. Fixes: agent-substrate#945
IPv6 sysctls are absent on kernels with IPv6 disabled (e.g. some containers set net.ipv6.conf.* only when IPv6 is enabled). Treat a missing path as 'nothing to enable' instead of forcing a remount and failing, matching the documented behavior.
The helper has enabled both address families since IPv6 forwarding was added; the name now says so. The single call site in SetupActorNetwork is updated along with the doc comment.
The os.Stat/IsNotExist fallback was never executed by the unit tests: every existing subtest's temp path could be created, so each returned at the os.WriteFile fast path. Point the new subtest at a node under a directory that does not exist — what procfs always does in production — and assert the file stays absent.
The egress Envoy pinned dns_lookup_family to V4_ONLY, so it asked only for A records. On an IPv6-only cluster no upstream name resolves and no actor can reach the internet. AUTO tries AAAA and falls back to A, so IPv4-only clusters behave as before. One step of the IPv6 egress work, and not the one that unblocks it -- actor egress still stops earlier, in atunnel's original-destination lookup. (cherry picked from commit de81578)
Runs the full install plus the demo and networking e2e suites against a single-stack IPv6-only kind cluster, and asserts the cluster really is v6-only so a green run cannot quietly become a second IPv4 run. It stays out of the e2e-test merge gate, so it reports IPv6 status without being able to block a PR, and it runs on every PR for now so the results are visible; the TODO on the trigger records the intended ci/ipv6 label gate. ubuntu-latest has no IPv6 egress, so the job stands up tayga for NAT64 and points CoreDNS at an upstream resolver through the well-known prefix. DNS64 is scoped to a catch-all server block: synthesizing AAAA over the cluster zones destroys the v6-only ClusterIP answers and the control plane never comes up. (cherry picked from commit 748e841)
The probe attached to the pod to collect its markers, and an attach can end before the last write arrives. A CI run lost the registry marker that way, so the check reported a registry it could not reach -- and then refused to re-probe, because only the resolve leg was treated as a settling race. Wait for the pod to terminate and read its log instead, and close the probe with a PROBE_DONE marker so a short read is re-probed rather than read as a failed fetch. A registry that really is down still fails on the first attempt.
The actor's NAT and filter rules lived in an ip table, which can only ever carry IPv4. They are now in an inet table, so one table can hold both address families when the actor veth becomes dual-stack. A bare payload match is ambiguous in an inet table, so every match now opens with an NFPROTO comparison and behaves exactly as it did before. NAT in the inet family needs Linux 4.18 or later. Teardown sweeps ip as well as inet. A table name is unique per family, so the ip table an earlier ateom left behind is invisible to an inet-only cleanup: the dump comes back empty, the "already clean" path reports success, and the stale table keeps redirecting alongside the new one. Part of agent-substrate#246
Actor networking was IPv4-only, so an actor on a dual-stack worker pod could not reach an IPv6-only destination at all. SetupActorNetwork now assigns the fd00:169:254::/126 counterparts of the existing point-to-point pair to both ends of the actor veth, installs an IPv6 default route in the interior netns, and adds the matching rules to the inet-family actor table. Whether the actor gets IPv6 is decided once in the worker pod netns and carried into the interior one, which is created fresh and so always reports IPv6 available whatever the cluster's families are. Both halves have to hold: the pod needs a global IPv6 address of its own, and the veth has to accept an IPv6 address -- IPv4-only GKE sets disable_ipv6 and netlink then rejects the assignment with EPERM. Addresses carry IFA_F_NODAD rather than the accept_dad sysctl, which the unprivileged ateom container cannot write. Part of agent-substrate#246
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 21, 2026 15:14
4da3798 to
dc1bb16
Compare
The IPv6 lookup ran only as a fallback, after an IPv4 lookup that a pure-IPv6 socket can never satisfy: its inet addresses are zeroed, so the conntrack tuple is all zeros and the query always misses with ENOENT. That made every IPv6 egress connection pay a guaranteed-to-fail syscall whose result was discarded, and left the code depending on that kernel detail holding. The connection's local address already says which family the flow is, so ask for that family's option directly. The failure message now names the family it asked for. atunnel logs it at warn level and it is the only signal an operator gets when a redirected connection cannot be resolved.
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 21, 2026 16:04
79d08df to
edb2348
Compare
The redirect tests built their veth, their nftables NAT table and both listeners in the host network namespace, so a machine with a default-deny INPUT policy dropped the redirected SYN and the test failed rather than skipped. Both sides now live in throwaway namespaces, which also removes the PID-derived addresses and interface names that could truncate or collide, and leaves no nftables state behind on the host. Coverage grew three ways: the IPv4 and IPv6 cases are one table built from the same ateomnet matchers production uses; that table gained the dual-stack socket the worker actually listens on, where an IPv4 actor arrives with a v4-mapped local address and is still only readable through the IPv4 socket option; and the address formatting, which was reachable only from root-gated tests, now has an ordinary one. Both lookups report a miss as ENOENT, so the failure-path cases assert the family named in the error rather than the errno alone.
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 21, 2026 16:56
edb2348 to
561cd0e
Compare
|
|
||
| // IP6T_SO_ORIGINAL_DST is not generated by golang.org/x/sys/unix. It is | ||
| // defined as 80 in linux/netfilter_ipv6/ip6_tables.h. | ||
| // IP6T_SO_ORIGINAL_DST is not generated by golang.org/x/sys/unix. It is defined |
Collaborator
Author
There was a problem hiding this comment.
Nit. make comment concise.
| // The IPv6 option is only meaningful on an AF_INET6 socket: on AF_INET the | ||
| // kernel returns EOPNOTSUPP, which would mask the real IPv4 error. A | ||
| // v4-mapped local address still means an IPv4 flow, so To4 is the test. | ||
| // Each family keeps its original destination under its own socket option |
Collaborator
Author
There was a problem hiding this comment.
Nit. make comment concise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #246.
The IPv6-only kind e2e job gets a cluster up and passes the demo suite on
main, but TestActorEgress and TestActorEgressHTTPS fail. This branch stacks
the commits that close that gap and runs the job against them, so the result
is visible before any of them merge.
Where each commit comes from, in branch order:
5a9f010274ad449f9f6bcdf1908499f8ac9ee2dcf047e7315ad28ad0cb7d66e2583a7b87deb103ff05f0be0f47ca7484a08ed2e844b82b3042432593e2531ca77249571b1157ed1adc1bb16b13a1e4b7561cd0eaEvery cherry-picked row is that PR's commit unchanged, with its original
author. #979 has taken the EnableIPv4Forwarding to EnableForwarding rework
upstream, so this branch carries it verbatim like the rest, and picks up its
missing-path test along the way.
The last two rows are not cherry-picks. They are review fixes for #753, stacked
here because that PR's workflows have never been approved, so nothing it
contains has ever been through CI. The first collapses the original-destination
lookup to one syscall chosen by family; the second moves the redirect tests into
throwaway network namespaces, where a default-deny INPUT policy can no longer
drop the redirected SYN, and adds the dual-stack socket production actually
listens on.
The last two rows replace the three #1057 commits this branch used to carry.
#1057 has been restructured into #1116, which moves the actor nftables table
from the
ipfamily toinetwith no behaviour change on IPv4, plus a singlecommit for the IPv6 address and the family gate. That gate is what keeps the
IPv4
e2e-testgreen — without it the actor gets an IPv6 address on a clusterwith no route for it and the two egress tests fail. The restructure also brings
a teardown fix this branch was missing: a table name is unique per family, so
an inet-only cleanup leaves a stale
iptable redirecting alongside the newone.
Signal only; it will never be merged. This tip is a fresh run on the latest
#753 and #979. What is left to file is the takeover of #753 and #979.
🤖 Generated with Claude Code