From 8b68a23cc126bed5230526c7b5d850bf91e529ef Mon Sep 17 00:00:00 2001 From: Peter Sprygada Date: Fri, 21 Aug 2026 11:17:16 -0400 Subject: [PATCH] refactor: Split node roles into independent capability labels Every DaemonSet's node-affinity used to key off one label, galactic.datumapis.com/node, mixing genuinely exclusive primary roles (edge vs. route-reflector vs. gateway) with capabilities that can and do coexist on the same node. That conflation was already live and wrong: fabric-router enumerated roles by hand and silently never included nat66, so NAT66 shard nodes never got the underlay BGP session their own SID advertisement depends on; nat66 and the route-reflector role each required their own node enum value, which made it structurally impossible for a node to be both that role and the tenant-serving role at once -- the one configuration those two actually use in every real deployment. This replaces the single overloaded label with independent labels per concern, matching how galactic.datumapis.com/fabric already worked: - galactic.datumapis.com/node is now a two-value enum (compute, edge) for the roles that really are mutually exclusive. "edge" now means the actual network-edge/gateway role (galactic-gateway) rather than the old per-node tenant role, resolving a long-standing collision with the pre-existing "edge XDP NAT+LB gateway" terminology used everywhere else in this repo; the tenant/compute role that used to be called "edge" is now "compute". - galactic.datumapis.com/galactic-route-reflector and galactic.datumapis.com/fabric are independent boolean flags, so a route-reflector node can also be a compute node, and any role that needs the underlay just adds the fabric label rather than needing every consumer's affinity list kept in sync. - nat66 is no longer its own node value at all -- every compute node now runs galactic-nat66 unconditionally, since that's the only configuration ever used. - galactic.datumapis.com/fabric-route-reflector is reserved for a future fabric-router-rr and documented as not yet implemented. config/galactic-router's overlay directories are renamed to match (edge -> default, tenant-control -> rr) and flattened to mirror plain Kustomize base/overlay conventions instead of bespoke naming. The containerlab lab's node labels, taints, and per-site fabric-router narrowing overlays are updated to the new scheme, including two overlays whose stale values: lists this same work caught live via a real crash-loop (fabric-router/base and fabric-control/iad, both still referencing retired label values) -- fixing that also surfaced a latent deploy-fabric.sh bug where a redeploy silently kept serving stale copied manifests, now fixed with the same rm -rf guard the other deploy scripts already use. See docs/node-labels.md for the full label reference and the reasoning behind which concerns got their own label versus stayed on the shared enum. Co-Authored-By: Claude Sonnet 5 --- AGENTS.md | 29 ++- README.md | 14 +- cmd/galactic-gateway/root.go | 2 +- config/fabric-router/daemonset.yaml | 39 ++-- config/galactic-cni/daemonset.yaml | 2 +- config/galactic-gateway/base/daemonset.yaml | 21 ++- config/galactic-nat66/base/daemonset.yaml | 37 ++-- config/galactic-router/base/daemonset.yaml | 36 ++-- config/galactic-router/kustomization.yaml | 16 +- .../overlays/default/daemonset-patch.yaml | 44 +++++ .../default}/kustomization.yaml | 2 +- .../overlays/rr/daemonset-patch.yaml | 62 +++++++ .../overlays/rr/kustomization.yaml | 10 + .../tenant-control/daemonset-patch.yaml | 54 ------ .../tenant-control/kustomization.yaml | 10 - .../tenant/daemonset-patch.yaml | 41 ---- deploy/containerlab/AGENTS.md | 17 +- deploy/containerlab/README.md | 98 +++++----- deploy/containerlab/Taskfile.yaml | 60 +++--- deploy/containerlab/docs/tenants.md | 46 ++--- deploy/containerlab/gvpc.clab.yaml | 8 +- .../containerlab/node_files/dfw/config.yaml | 3 +- .../containerlab/node_files/iad/config.yaml | 24 ++- .../containerlab/node_files/sjc/config.yaml | 3 +- .../fabric-control/iad/fabric-lab-patch.yaml | 21 ++- .../fabric-control/iad/kustomization.yaml | 2 +- .../fabric-router/base/fabric-lab-patch.yaml | 29 +-- .../galactic-control/iad/bgprouter.yaml | 2 +- .../galactic-control/iad/kustomization.yaml | 2 +- .../iad-gateway1/node-patch.yaml | 4 +- .../iad-gateway2/node-patch.yaml | 4 +- .../iad/servicevipbinding-ns60.yaml | 6 +- .../resources/galactic-nat66/README.md | 2 +- .../galactic-nat66/dfw/node-patch.yaml | 39 ++-- .../galactic-nat66/iad/node-patch.yaml | 55 ++---- .../galactic-nat66/sjc/node-patch.yaml | 39 ++-- .../galactic-router/base/kustomization.yaml | 2 +- deploy/containerlab/scripts/deploy-fabric.sh | 33 +++- .../scripts/deploy-galactic-router.sh | 61 +++--- deploy/containerlab/scripts/verify-ns40.sh | 2 +- docs/agents/ARCHITECTURE-GATEWAY.md | 4 +- docs/agents/ARCHITECTURE-ROUTER.md | 23 ++- docs/cni/gc-cmd-sequence.md | 2 +- docs/node-labels.md | 175 ++++++++++++++++++ internal/config/nat66.go | 13 +- internal/gc/gc.go | 2 +- .../plumbing/ebpf/nat66prog/nat66_test.go | 2 +- internal/plumbing/srv6/egress.go | 2 +- 48 files changed, 719 insertions(+), 485 deletions(-) create mode 100644 config/galactic-router/overlays/default/daemonset-patch.yaml rename config/galactic-router/{tenant => overlays/default}/kustomization.yaml (88%) create mode 100644 config/galactic-router/overlays/rr/daemonset-patch.yaml create mode 100644 config/galactic-router/overlays/rr/kustomization.yaml delete mode 100644 config/galactic-router/tenant-control/daemonset-patch.yaml delete mode 100644 config/galactic-router/tenant-control/kustomization.yaml delete mode 100644 config/galactic-router/tenant/daemonset-patch.yaml create mode 100644 docs/node-labels.md diff --git a/AGENTS.md b/AGENTS.md index 5392f8b6..da0c41f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,7 +35,7 @@ VPC and VPCAttachment CRD management lives in a separate companion operator; Gal - **Go 1.26** — every binary - **controller-runtime** — BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance reconcilers (`galactic-router`) and NetworkGateway/NetworkRule reconcilers (`galactic-gateway`); a bare `pkg/client` (no manager) in `galactic-bgp`/`galactic-cni` - **BGP API** (`go.datum.net/network`) — BGPRouter, BGPPeer, BGPAdvertisement, BGPPolicy, BGPVRFInstance, NetworkGateway, NetworkRule CRDs -- **GoBGP v4** — embedded BGP server (tenant role) +- **GoBGP v4** — embedded BGP server (default role) - **SRv6 + netlink** — kernel-level routing; `github.com/vishvananda/netlink` - **eBPF** (`github.com/cilium/ebpf`) — TC-BPF SRv6 uSID decap datapath (CNI side) and XDP edge NAT+LB datapath (`galactic-gateway`) - **Multus CNI** — multi-network for pods; NAD generation handled by the external operator @@ -68,16 +68,28 @@ Summary: `config/` is Kustomize-composed: `kubectl apply -k config/` deploys everything (namespace, both DaemonSets' RBAC/ServiceAccounts, and all three DaemonSets) in one command — `kubectl` sorts by kind before applying, so the namespace and RBAC/ServiceAccounts always land before anything namespace-scoped needs them. Each component also has its own `kustomization.yaml` and can be applied independently: +**Node label strategy.** Every DaemonSet's node affinity keys off one or more of these labels — deliberately independent labels rather than one enum, since Kubernetes labels are single-valued per key and some of these must be able to coexist on the same node (e.g. a route reflector that's also a compute node): + +| Label | Deploys | +| ----- | ------- | +| `galactic.datumapis.com/fabric=true` | `fabric-router` | +| `galactic.datumapis.com/node=compute` | `galactic-router` (default role), `galactic-nat66`, `galactic-cni` | +| `galactic.datumapis.com/node=edge` | `galactic-gateway` (the actual network-edge/ingress boundary — not to be confused with `compute`) | +| `galactic.datumapis.com/galactic-route-reflector=true` | `galactic-router-rr` | +| `galactic.datumapis.com/fabric-route-reflector=true` | `fabric-router-rr` (**future** — not yet implemented; `fabric-router` has no route-reflector variant today) | + +See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why these are independent labels rather than one enum, the naming collision between `node=edge` and `galactic-gateway`'s pre-existing "edge XDP" terminology, and the specific bugs this scheme replaced. + - **`config/galactic-system/`** — Creates the `galactic-system` namespace both components deploy into. Apply with `kubectl apply -k config/galactic-system/`. - **`config/galactic-cni/`** — Production manifests for the CNI installer DaemonSet, ConfigMap, RBAC, and ServiceAccount. Apply with `kubectl apply -k config/galactic-cni/`. - **`config/galactic-router/`** — Shared RBAC/ServiceAccount plus DaemonSet roles: - - **`config/galactic-router/tenant/`** — the per-node role (`galactic-router`); runs on every node except Kubernetes control-plane nodes and nodes labeled for the route-reflector or gateway roles. - - **`config/galactic-router/tenant-control/`** — the BGP route-reflector role (`galactic-router-control`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires nodes labeled `galactic.datumapis.com/node: control` (stays at zero replicas otherwise). `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it. - - **`config/galactic-router/base/`** — the DaemonSet spec shared by both roles above; not applied directly. - - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/tenant/`. `galactic-router` no longer has a gateway role of its own — see `config/galactic-gateway/` below. -- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a separate `galactic-gateway` binary rather than a `galactic-router` role, so a crash on either side no longer takes the other down with it. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (the two-container `galactic-router` + `galactic-gateway` pod, requiring nodes labeled `galactic.datumapis.com/node: gateway`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/controller/networkgateway_controller.go`'s `publishSelfAddress` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` for a worked two-node example. Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. -- **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `tenant`/`tenant-control`-style role split — its affinity matches nodes labeled `galactic.datumapis.com/node` `In` `[edge, control, gateway]` directly, since (unlike galactic-router's route-reflector role and the gateway role above) there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so `frr-init` gets the pod's node name via a `NODE_NAME` downward-API env var and selects a per-node `frr.conf.` key from the ConfigMap, rather than assuming one shared `frr.conf` for the whole DaemonSet. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author a `fabric-config` ConfigMap with one `frr.conf.` key per matching node (`daemons`/`vtysh.conf` are baked into the `fabric-router` image and only need to be in the ConfigMap if overriding those defaults) before applying `kubectl apply -k config/fabric-router/`. -- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an IPv6 SRv6 transit mesh. FRR runs as a hostNetwork DaemonSet on each worker for eBGP underlay; `galactic-router` (tenant role) handles EVPN path distribution over iBGP, the iad route reflector builds on `config/galactic-router/tenant-control/`, and iad additionally has two dedicated gateway-role nodes (`iad-gateway1`/`iad-gateway2`, `config/galactic-gateway/base/`) as a canary for the edge XDP NAT+LB gateway. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. + - **`config/galactic-router/base/`** — the role-agnostic DaemonSet spec both roles below patch; not applied directly (no affinity, no BGP listen port of its own). + - **`config/galactic-router/overlays/default/`** — the default per-node role (`galactic-router`); runs on every node except Kubernetes control-plane nodes, opt-in via `galactic.datumapis.com/node: compute` (the same label `galactic-cni` and `galactic-nat66` key off of — see `config/galactic-cni/daemonset.yaml`, `config/galactic-nat66/base/daemonset.yaml`). This is the default role, so it carries no role-specific name/label of its own — only `rr` below needs one, to coexist as a second DaemonSet. + - **`config/galactic-router/overlays/rr/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic-route-reflector` set (stays at zero replicas otherwise). This is a dedicated boolean-style label, not a `galactic.datumapis.com/node` value — that key is single-valued, so a route-reflector node can still independently be `galactic.datumapis.com/node: compute` (or any other value) at the same time. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it. + - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/default/`. `galactic-router` no longer has a gateway role of its own — see `config/galactic-gateway/` below. +- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a separate `galactic-gateway` binary rather than a `galactic-router` role, so a crash on either side no longer takes the other down with it. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (the two-container `galactic-router` + `galactic-gateway` pod, requiring nodes labeled `galactic.datumapis.com/node: edge` — the actual network-edge/ingress boundary, not to be confused with `galactic-router`'s own `compute` role) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/controller/networkgateway_controller.go`'s `publishSelfAddress` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` for a worked two-node example. Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. +- **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric` (a dedicated boolean-style label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic-route-reflector` flag that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so `frr-init` gets the pod's node name via a `NODE_NAME` downward-API env var and selects a per-node `frr.conf.` key from the ConfigMap, rather than assuming one shared `frr.conf` for the whole DaemonSet. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author a `fabric-config` ConfigMap with one `frr.conf.` key per matching node (`daemons`/`vtysh.conf` are baked into the `fabric-router` image and only need to be in the ConfigMap if overriding those defaults) before applying `kubectl apply -k config/fabric-router/`. +- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an IPv6 SRv6 transit mesh. FRR runs as a hostNetwork DaemonSet on each worker for eBGP underlay; `galactic-router` (default role) handles EVPN path distribution over iBGP, the iad route reflector builds on `config/galactic-router/overlays/rr/`, and iad additionally has two dedicated edge-role nodes (`iad-gateway1`/`iad-gateway2`, `config/galactic-gateway/base/`) as a canary for the edge XDP NAT+LB gateway. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. ## New Developer Entry Points @@ -89,3 +101,4 @@ Summary: 6. Explore `internal/plumbing/` for shared kernel and network primitives (VRF, sysctl, interface naming, SRv6). 7. Read `internal/gateway/engine.go` and `internal/controller/networkgateway_controller.go` to understand the edge XDP NAT+LB gateway's convergence loop and Active-Active BGP placement model. See [ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md). 8. See `docs/cni/cni-cmd-sequence.md`, `docs/cni/gc-cmd-sequence.md`, and `docs/agent-startup.md` for Mermaid sequence diagrams of the CNI attach path, garbage collection, and router startup, respectively. `docs/cni/configuration.md` and `docs/router/configuration.md` document CNI config fields and router environment variables. +9. Read [docs/node-labels.md](docs/node-labels.md) to understand which `galactic.datumapis.com/*` labels put which DaemonSets on which nodes, and why it's five independent labels rather than one enum — this cuts across all five binaries and isn't owned by any single component's architecture doc. diff --git a/README.md b/README.md index a4b0db16..5e0b70a9 100644 --- a/README.md +++ b/README.md @@ -32,18 +32,18 @@ See the [galactic DevContainer](./.devcontainer/galactic/) for development envir ### Production Deployment -Manifests for a real cluster live under [`config/`](./config/), composed with [Kustomize](https://kustomize.io). One command deploys the `galactic-system` namespace (labeled `pod-security.kubernetes.io/enforce: privileged` — every DaemonSet here needs it, for hostPath volumes, hostNetwork, and elevated capabilities), the `galactic-cni` DaemonSet, and both `galactic-router` roles — `tenant` (per-node, runs everywhere except control-plane nodes) and `tenant-control` (BGP route reflector, opt-in — stays at zero replicas until nodes are labeled `galactic.datumapis.com/node: control`): +Manifests for a real cluster live under [`config/`](./config/), composed with [Kustomize](https://kustomize.io). One command deploys the `galactic-system` namespace (labeled `pod-security.kubernetes.io/enforce: privileged` — every DaemonSet here needs it, for hostPath volumes, hostNetwork, and elevated capabilities), the `galactic-cni` DaemonSet, and both `galactic-router` roles — the default per-node role (`galactic-router`, runs everywhere except control-plane nodes) and `rr` (`galactic-router-rr`, the BGP route reflector, opt-in — stays at zero replicas until nodes are labeled `galactic.datumapis.com/galactic-route-reflector: "true"`): ```bash kubectl apply -k config/ ``` -Each component can also be applied on its own, e.g. `kubectl apply -k config/galactic-router` for just the router (both roles) or `kubectl apply -k config/galactic-router/tenant` for just the per-node role. +Each component can also be applied on its own, e.g. `kubectl apply -k config/galactic-router` for just the router (both roles) or `kubectl apply -k config/galactic-router/overlays/default` for just the default per-node role. Two components are **not** part of `kubectl apply -k config/` and must be applied separately, each with its own per-node prerequisite: - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet `galactic-router` depends on. -- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane (`galactic-router` + `galactic-gateway` running together on dedicated gateway-role nodes). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` for a worked example) — apply that overlay per node instead of `base/` directly. +- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane (`galactic-router` + `galactic-gateway` running together on dedicated `galactic.datumapis.com/node: edge` nodes). `kubectl apply -k config/galactic-gateway/` only installs the shared, cluster-safe ServiceAccount/RBAC; `config/galactic-gateway/base/` itself is a template meant to be instantiated once per gateway node by a further overlay (see `deploy/containerlab/resources/galactic-gateway/` for a worked example) — apply that overlay per node instead of `base/` directly. ```bash kubectl apply -k config/fabric-router/ @@ -61,7 +61,7 @@ kubectl apply -k config/galactic-gateway/ cd config/fabric-router && kustomize edit set image ghcr.io/datum-cloud/fabric-router=ghcr.io/datum-cloud/fabric-router: ``` -- **`config/fabric-router/`: per-node `frr.conf`.** Unlike every other component under `config/`, `config/fabric-router/daemonset.yaml` has no generic default config — the underlay eBGP session (interface addresses, remote-AS, etc.) is different for every physical node, and this DaemonSet's `nodeAffinity` (`galactic.datumapis.com/node` `In` `[edge, control]`) can legitimately match more than one node per cluster. Before applying `config/fabric-router/`, create a `fabric-config` ConfigMap in the `galactic-system` namespace with one `frr.conf.` key per matching node (`` is the Kubernetes node name, e.g. `frr.conf.worker-1`) — `frr-init` picks the right key at pod start via the pod's `NODE_NAME` downward-API env var. The other two files FRR needs, `daemons` and `vtysh.conf`, are already baked into the `fabric-router` image (see `containers/fabric-router/Dockerfile`); include them in the ConfigMap too only if you need to override the image defaults. `deploy/containerlab/resources/fabric/{dfw,iad,sjc}/frr.conf` are worked examples from the lab, not something you can apply as-is. +- **`config/fabric-router/`: per-node `frr.conf`.** Unlike every other component under `config/`, `config/fabric-router/daemonset.yaml` has no generic default config — the underlay eBGP session (interface addresses, remote-AS, etc.) is different for every physical node, and this DaemonSet's `nodeAffinity` (`galactic.datumapis.com/fabric` `Exists`) can legitimately match more than one node per cluster. Before applying `config/fabric-router/`, create a `fabric-config` ConfigMap in the `galactic-system` namespace with one `frr.conf.` key per matching node (`` is the Kubernetes node name, e.g. `frr.conf.worker-1`) — `frr-init` picks the right key at pod start via the pod's `NODE_NAME` downward-API env var. The other two files FRR needs, `daemons` and `vtysh.conf`, are already baked into the `fabric-router` image (see `containers/fabric-router/Dockerfile`); include them in the ConfigMap too only if you need to override the image defaults. `deploy/containerlab/resources/fabric/{dfw,iad,sjc}/frr.conf` are worked examples from the lab, not something you can apply as-is. - **`config/fabric-router/` and `config/galactic-router/`: rolling out updates is manual.** Both `config/fabric-router/daemonset.yaml` and `config/galactic-router/base/daemonset.yaml` use `updateStrategy: OnDelete` — a `kubectl apply` (new image tag, or a spec change) will not restart any pod on its own. This is deliberate for both: each is a BGP speaker whose liveness/health probe only reflects "the process is up," not "the BGP session has reconverged," so `RollingUpdate` would advance to the next node on exactly the wrong signal — see the comment above `updateStrategy` in each manifest for the full reasoning. To actually roll out a change, delete pods one at a time and confirm the new pod's session(s) have reconverged before moving to the next node, e.g. for `fabric-router`: @@ -75,14 +75,14 @@ kubectl apply -k config/galactic-gateway/ A ConfigMap edit to `fabric-config` behaves the same way today regardless of `updateStrategy` — there's no checksum annotation wiring pod restarts to ConfigMap changes, so a config change also requires this same manual, node-by-node pod bounce to take effect. -- **`config/galactic-router/`: rollout order — reflector last, one tenant at a time.** For `tenant-control` (the route reflector), every `tenant` node's iBGP session pivots through that one pod, so bouncing it is a fleet-wide route flap, not a single-node one — roll it only after all `tenant` nodes are already on the new version, and confirm every client has re-peered before considering the rollout done. For `tenant`, a bounce only withdraws that one node's own advertised prefixes, so it's safe to go node-by-node as with `fabric-router`. Check session state via the `BGPPeer` CRD's `STATE` column rather than `vtysh` (there's no `vtysh` in this binary — `galactic-router` reports session state itself): +- **`config/galactic-router/`: rollout order — `rr` last, one compute node at a time.** For `rr`, every compute node's iBGP session pivots through that one pod, so bouncing it is a fleet-wide route flap, not a single-node one — roll it only after all compute nodes are already on the new version, and confirm every client has re-peered before considering the rollout done. For the default role, a bounce only withdraws that one node's own advertised prefixes, so it's safe to go node-by-node as with `fabric-router`. Check session state via the `BGPPeer` CRD's `STATE` column rather than `vtysh` (there's no `vtysh` in this binary — `galactic-router` reports session state itself): ```bash - kubectl -n galactic-system get pods -l app.kubernetes.io/name=galactic-router-tenant -o wide + kubectl -n galactic-system get pods -l app.kubernetes.io/name=galactic-router -o wide kubectl -n galactic-system delete pod # wait for the new pod to be Ready, then confirm its BGPPeer CRDs are back to STATE=Established kubectl get bgppeer - # repeat for the next tenant node, then only last roll galactic-router-control the same way + # repeat for the next compute node, then only last roll galactic-router-rr the same way ``` - **`config/galactic-gateway/`: per-node public interface and SRv6 address.** `config/galactic-gateway/base/daemonset.yaml`'s `galactic-gateway` container requires `GALACTIC_GATEWAY_PUBLIC_INTERFACE` and `GALACTIC_GATEWAY_SRV6_ADDRESS` — the latter must be unique per gateway node and has no generic default (there's no in-cluster mechanism yet that derives it automatically; see `publishSelfAddress`'s doc comment in `internal/controller/networkgateway_controller.go`). Applying `base/` as shipped, without pinning both per node, produces a crash-looping container. Instantiate `base/` via a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's values — see `deploy/containerlab/resources/galactic-gateway/` for a worked two-node example. diff --git a/cmd/galactic-gateway/root.go b/cmd/galactic-gateway/root.go index 623449c9..6012fa76 100644 --- a/cmd/galactic-gateway/root.go +++ b/cmd/galactic-gateway/root.go @@ -45,7 +45,7 @@ const ( // BGP-family reconciler here at all: NetworkGatewayReconciler/ // NetworkRuleReconciler need no BGP client of their own — they only // create/update/delete BGPAdvertisement CRDs, which the co-located -// galactic-router (tenant role) picks up via its own +// galactic-router (default role) picks up via its own // BGPAdvertisementReconciler. func runCmd(cfg *config.GatewayConfig) error { nodeName := cfg.NodeName diff --git a/config/fabric-router/daemonset.yaml b/config/fabric-router/daemonset.yaml index 7829ccd0..43e8131a 100644 --- a/config/fabric-router/daemonset.yaml +++ b/config/fabric-router/daemonset.yaml @@ -14,7 +14,7 @@ spec: # physical fabric switch, not an in-cluster control-plane relationship. # Bouncing FRR here drops the node's only underlay path, so every overlay # SRv6/EVPN path riding on it blackholes until the eBGP session - # re-establishes and reconverges -- and galactic-router-control/ + # re-establishes and reconverges -- and galactic-router-rr/ # galactic-gateway crash-loop per the toleration comment below if it # doesn't come back. The startupProbe/livenessProbe below only check that # zebra/bgpd answer vtysh (the daemon is alive), not that the eBGP @@ -48,26 +48,33 @@ spec: # Blanket-tolerate all taints, same as every other DaemonSet in # config/ (galactic-cni, galactic-router, galactic-gateway). # Placement is controlled entirely by the node affinity - # below (edge/control/gateway roles) — tolerations don't need to be + # below (the galactic.datumapis.com/fabric label) — tolerations don't need to be # scoped, and scoping them risks silently blocking fabric-router if # an unenumerated taint (a new role taint, a maintenance taint, etc.) # ever lands on an eligible node: the route reflector's lo address # (and BGP_LOCAL_ADDRESS auto-detection) or the gateway node's - # underlay session would never come up and galactic-router-control/ + # underlay session would never come up and galactic-router-rr/ # galactic-gateway would crashloop forever per their dependency on # this DaemonSet. tolerations: - operator: Exists # Opt-in only, same as galactic-cni and galactic-router: runs on every - # node labeled for regular tenant traffic, the galactic-router - # route-reflector role, or the edge XDP NAT+LB gateway role - # (config/galactic-gateway/base/, a separate galactic-gateway binary), since - # all three need underlay connectivity -- the gateway role's own XDP - # datapath attaches to - # this same interface (see internal/gateway's doc.go) and depends on - # FRR having already brought it up and established the eBGP - # underlay session. New node types (GPU, monitoring, etc.) don't - # automatically get fabric. + # node explicitly labeled galactic.datumapis.com/fabric, a dedicated + # boolean-style label independent of galactic.datumapis.com/node's + # role value (compute, edge) and of the separate + # galactic.datumapis.com/galactic-route-reflector flag -- rather than + # enumerating every role that happens to need underlay connectivity + # today, this lets any *current or future* role opt in by adding + # one label, + # with no matching edit ever required here again. (This replaced an + # earlier version that did enumerate roles via `In` -- which meant a + # role like nat66 was never added to the list and so silently never + # got fabric, even though NAT66 shards need underlay BGP to + # advertise their own SID just as much as any other role.) The + # gateway role's own XDP datapath attaches to this same interface + # (see internal/gateway's doc.go) and depends on FRR having already + # brought it up and established the eBGP underlay session. New node + # types (GPU, monitoring, etc.) don't automatically get fabric. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -75,12 +82,8 @@ spec: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - edge - - control - - gateway + - key: galactic.datumapis.com/fabric + operator: Exists initContainers: - name: frr-init # ":latest" is a placeholder only: no such tag is ever pushed to diff --git a/config/galactic-cni/daemonset.yaml b/config/galactic-cni/daemonset.yaml index 3e6235ac..9530c8ee 100644 --- a/config/galactic-cni/daemonset.yaml +++ b/config/galactic-cni/daemonset.yaml @@ -43,7 +43,7 @@ spec: - key: galactic.datumapis.com/node operator: In values: - - edge + - compute initContainers: - name: install-cni # ":latest" is a placeholder only: no such tag is ever pushed to diff --git a/config/galactic-gateway/base/daemonset.yaml b/config/galactic-gateway/base/daemonset.yaml index 7eafd40c..745a45a1 100644 --- a/config/galactic-gateway/base/daemonset.yaml +++ b/config/galactic-gateway/base/daemonset.yaml @@ -19,10 +19,15 @@ spec: app.kubernetes.io/name: galactic-gateway spec: # Dedicated gateway-role nodes only, opt-in, matching every other - # role's pattern (config/galactic-router/tenant, config/galactic-router/tenant-control). - # Excludes control-plane outright for the same reason those do. See - # ../kustomization.yaml for why this base is meant to be instantiated - # once per gateway node rather than applied as-is. + # role's pattern (config/galactic-router/overlays/default, config/galactic-router/overlays/rr). + # "edge" here means the actual network edge -- the ingress/egress + # boundary this gateway sits on -- not galactic-router's per-node + # role, which is galactic.datumapis.com/node=compute (see + # config/galactic-router/overlays/default/daemonset-patch.yaml's own + # comment on that naming). Excludes control-plane outright for the + # same reason those do. See ../kustomization.yaml for why this base + # is meant to be instantiated once per gateway node rather than + # applied as-is. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -33,7 +38,7 @@ spec: - key: galactic.datumapis.com/node operator: In values: - - gateway + - edge # See ../rbac.yaml's header comment: one ServiceAccount for both # containers is a Kubernetes constraint (a Pod has exactly one # ServiceAccount identity), not a design choice — its ClusterRole @@ -52,11 +57,11 @@ spec: - operator: Exists containers: # Tenant BGP only: galactic-router has no gateway-role flags/env. - # Identical to config/galactic-router/tenant's + # Identical to config/galactic-router/base's # own container spec; duplicated here (rather than patched from # config/galactic-router/base) because this pod additionally needs a second # container, a distinct grpc-health port, and a scoped-down - # securityContext the shared tenant/tenant-control roles don't need + # securityContext the shared default/rr roles don't need # to think about. - name: galactic-router # ":latest" is a placeholder only: no such tag is ever pushed to @@ -77,7 +82,7 @@ spec: - name: GALACTIC_ROUTER_GC_NAMESPACE value: galactic-system - name: GALACTIC_ROUTER_BGP_LISTEN_PORT - # Same as config/galactic-router/tenant/daemonset-patch.yaml: -1 + # Same as config/galactic-router/overlays/default/daemonset-patch.yaml: -1 # disables the inbound BGP listener, so there's no BGP # containerPort to declare on this container either -- this # role only dials out to iBGP peers. diff --git a/config/galactic-nat66/base/daemonset.yaml b/config/galactic-nat66/base/daemonset.yaml index dde83dbb..4f22eb48 100644 --- a/config/galactic-nat66/base/daemonset.yaml +++ b/config/galactic-nat66/base/daemonset.yaml @@ -18,20 +18,27 @@ spec: labels: app.kubernetes.io/name: galactic-nat66 spec: - # Dedicated NAT66-shard-role nodes only, opt-in, matching every - # other role's pattern (config/galactic-router/tenant, - # config/galactic-router/tenant-control, - # config/galactic-gateway/base). "nat66" is a new - # galactic.datumapis.com/node value, introduced here alongside - # "edge"/"control"/"gateway" for the same reason "gateway" was: this - # is its own dedicated, opt-in node role with no generic default, - # not a variant of any existing one -- a NAT66 shard's own SID/public - # address must be unique per node (see ../kustomization.yaml), the - # same uniqueness property that already justifies "gateway" being - # its own label value rather than reusing "edge". Excludes - # control-plane outright for the same reason those do. See - # ../kustomization.yaml for why this base is meant to be instantiated - # once per shard node rather than applied as-is. + # galactic-nat66 rides on the same galactic.datumapis.com/node=compute + # label as galactic-router's default role and galactic-cni (see + # config/galactic-router/overlays/default, config/galactic-cni/ + # daemonset.yaml) -- every compute node runs all three, NAT66 + # sharding included, unconditionally. No separate opt-in capability + # label: unlike galactic-route-reflector (config/galactic-router/ + # overlays/rr/daemonset-patch.yaml's own comment) NAT66 isn't a + # thing some compute nodes opt out of -- it's simply part of what + # "compute" means. A NAT66 shard's own SID/public address must + # still be unique per node (see ../kustomization.yaml), which is + # why this base is meant to be instantiated once per shard node + # rather than applied as-is -- that per-node uniqueness is handled + # by the per-node overlay's env vars, not by the scheduling gate + # here. Excludes control-plane outright for the same reason the + # other roles do. + # + # A NAT66 shard advertises its own SID over BGP just like any other + # role, so a real shard node also needs galactic.datumapis.com/fabric + # set (config/fabric-router/'s own affinity keys off that label, + # independently of the label here) -- otherwise it never gets the + # underlay eBGP session and its SID has no fabric to advertise onto. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -42,7 +49,7 @@ spec: - key: galactic.datumapis.com/node operator: In values: - - nat66 + - compute serviceAccountName: galactic-nat66 hostNetwork: true dnsPolicy: ClusterFirstWithHostNet diff --git a/config/galactic-router/base/daemonset.yaml b/config/galactic-router/base/daemonset.yaml index 70925f18..c26ea2c7 100644 --- a/config/galactic-router/base/daemonset.yaml +++ b/config/galactic-router/base/daemonset.yaml @@ -15,12 +15,11 @@ spec: # the server starts, never tied to iBGP session/route state -- so # RollingUpdate would advance to the next node on exactly the wrong # signal. A restart implicitly withdraws routes (no BGP graceful restart - # support in internal/runtime/gobgp), and for the tenant-control - # (route-reflector) role every tenant node's iBGP session pivots through - # this one pod, so an automatic rollout there is a fleet-wide route flap, - # not a single-node one -- worse than fabric-router's per-node blast - # radius. OnDelete puts pod recreation under explicit, node-by-node - # control instead. + # support in internal/runtime/gobgp), and for the rr role every compute + # node's iBGP session pivots through this one pod, so an automatic + # rollout there is a fleet-wide route flap, not a single-node one -- + # worse than fabric-router's per-node blast radius. OnDelete puts pod + # recreation under explicit, node-by-node control instead. updateStrategy: type: OnDelete template: @@ -31,6 +30,18 @@ spec: serviceAccountName: galactic-router hostNetwork: true dnsPolicy: ClusterFirstWithHostNet + # This base is role-agnostic on purpose -- no affinity, no BGP + # listen port -- so it's not meant to be applied on its own (its + # blanket toleration below plus no affinity would schedule it onto + # every node, control-plane included). ../overlays/default/ patches in + # the default per-node role's affinity + GALACTIC_ROUTER_BGP_LISTEN_PORT + # (see ../overlays/default/daemonset-patch.yaml); ../overlays/rr/ + # independently patches this same, unmodified base into the separate + # route-reflector role. + # Neither overlay derives from the other -- both patch this file + # independently, each in its own isolated kustomize build, rather + # than via a shared patches: entry one level up -- see + # ../kustomization.yaml's own comment for why that's unsafe here. # Loss of galactic-router blackholes every VPC route through this # node, so the scheduler/kubelet must never preempt it for ordinary # workloads under node pressure. @@ -71,13 +82,12 @@ spec: - name: grpc-health containerPort: 5179 protocol: TCP - # No BGP containerPort here: GALACTIC_ROUTER_BGP_LISTEN_PORT - # isn't set at this (never-applied-directly) base level at all, - # and its real value is role-dependent -- see - # ../tenant-control/daemonset-patch.yaml (1790, the one role - # that actually listens) and ../tenant/daemonset-patch.yaml - # (-1, disabled). Declaring one here would just be wrong for - # whichever role doesn't match it. + # No BGP containerPort or GALACTIC_ROUTER_BGP_LISTEN_PORT here: + # this base is role-agnostic, and the right value is role-dependent + # -- see ../overlays/default/daemonset-patch.yaml (-1, disabled) and + # ../overlays/rr/daemonset-patch.yaml (1790, the one role that actually + # listens). Declaring one here would just be wrong for whichever + # role doesn't match it. # GoBGP's cold start (peer session setup, RIB population) can # outrun readinessProbe/livenessProbe's short initialDelaySeconds # on a slow node, risking a liveness-triggered restart loop before diff --git a/config/galactic-router/kustomization.yaml b/config/galactic-router/kustomization.yaml index b50b9ae1..7a6ff765 100644 --- a/config/galactic-router/kustomization.yaml +++ b/config/galactic-router/kustomization.yaml @@ -1,5 +1,17 @@ +# base/ is a role-agnostic template, not meant to be applied on its own -- +# overlays/default/ and overlays/rr/ each independently build their own +# isolated kustomize nested build on top of base/ and patch it into their +# own role. A top-level patches: entry here (rather than a dedicated +# overlay) was tried and reverted: Kustomize's patches[].target.name is +# matched unanchored-regex-style against the fully-accumulated resource +# set, so even an anchored "^galactic-router$" here still cross-contaminated +# rr's already-nameSuffix'd "galactic-router-rr" DaemonSet, silently +# overwriting its own overrides. Scoping each role's patch inside its own +# nested kustomization build (matching before any nameSuffix is applied, +# and before the parent ever sees more than one DaemonSet) is the only +# reliable way to keep the two roles' patches from colliding. resources: - serviceaccount.yaml - rbac.yaml - - tenant - - tenant-control + - overlays/default + - overlays/rr diff --git a/config/galactic-router/overlays/default/daemonset-patch.yaml b/config/galactic-router/overlays/default/daemonset-patch.yaml new file mode 100644 index 00000000..4303a033 --- /dev/null +++ b/config/galactic-router/overlays/default/daemonset-patch.yaml @@ -0,0 +1,44 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-router +spec: + template: + spec: + # This is the default, unqualified per-node role -- it needs no + # name/label of its own (../../base/daemonset.yaml's plain "galactic-router" + # identity is all it gets) since nothing else needs to select it + # apart from the rr (route reflector) role, which independently + # patches the same ../../base/ with its own distinct name/label -- see + # ../rr/daemonset-patch.yaml. Opt-in via the + # galactic.datumapis.com/node=compute label so new node types (GPU, + # monitoring, etc.) don't accidentally pick up the role, and + # excludes Kubernetes control-plane nodes outright — the base + # DaemonSet's blanket toleration would otherwise let it land there + # if one ever carried the compute label. "compute" (not "edge") -- + # galactic-cni and galactic-nat66 opt in via the same label value, + # since all three run together on every ordinary tenant-serving + # node; "edge" is reserved for galactic-gateway's node label, the + # actual network-edge/ingress role (see + # config/galactic-gateway/base/daemonset.yaml). + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: DoesNotExist + - key: galactic.datumapis.com/node + operator: In + values: + - compute + containers: + - name: galactic-router + env: + - name: GALACTIC_ROUTER_BGP_LISTEN_PORT + # This role only dials *out* to iBGP peers (the reflector, + # see ../rr/daemonset-patch.yaml, still a sibling under + # overlays/); -1 disables the inbound + # listener entirely, so no BGP containerPort belongs on this + # patch -- there's no port here to document. + value: "-1" diff --git a/config/galactic-router/tenant/kustomization.yaml b/config/galactic-router/overlays/default/kustomization.yaml similarity index 88% rename from config/galactic-router/tenant/kustomization.yaml rename to config/galactic-router/overlays/default/kustomization.yaml index 7fde98c8..7a237d4d 100644 --- a/config/galactic-router/tenant/kustomization.yaml +++ b/config/galactic-router/overlays/default/kustomization.yaml @@ -1,5 +1,5 @@ resources: - - ../base + - ../../base patches: - path: daemonset-patch.yaml target: diff --git a/config/galactic-router/overlays/rr/daemonset-patch.yaml b/config/galactic-router/overlays/rr/daemonset-patch.yaml new file mode 100644 index 00000000..b712ac95 --- /dev/null +++ b/config/galactic-router/overlays/rr/daemonset-patch.yaml @@ -0,0 +1,62 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: galactic-router + labels: + app.kubernetes.io/name: galactic-router-rr +spec: + selector: + matchLabels: + app.kubernetes.io/name: galactic-router-rr + template: + metadata: + labels: + app.kubernetes.io/name: galactic-router-rr + spec: + # Route-reflector role: opt-in only, requires + # galactic.datumapis.com/galactic-route-reflector explicitly set, so + # it never double-schedules with the per-worker default DaemonSet + # (see ../../base/daemonset.yaml) just because a node happens to + # also be galactic.datumapis.com/node=compute. Deliberately *not* a + # galactic.datumapis.com/node enum value: that key is single-valued, + # so encoding this role there would make it impossible for a node + # to be both compute and a route reflector at once -- the same + # class of bug fixed for galactic.datumapis.com/nat66 (see + # config/galactic-nat66/base/daemonset.yaml's own affinity comment). + # A dedicated boolean-style label, independent of whatever + # galactic.datumapis.com/node value a node also carries, matches + # galactic.datumapis.com/fabric's pattern instead. Also excludes + # Kubernetes control-plane nodes outright — the base DaemonSet's + # blanket toleration would otherwise let it land there if one ever + # carried the galactic-route-reflector label. + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: DoesNotExist + - key: galactic.datumapis.com/galactic-route-reflector + operator: Exists + containers: + - name: galactic-router + env: + - name: GALACTIC_ROUTER_REFLECTOR + value: "true" + - name: GALACTIC_ROUTER_BGP_LISTEN_PORT + value: "1790" + # GALACTIC_ROUTER_BGP_LOCAL_ADDRESS is auto-detected from the + # host's lo interface by default (the stable peering address + # other compute routers reflect through). Set it explicitly here + # only if lo doesn't carry the desired address on this cluster. + # Documentation only (hostNetwork: true, so this doesn't change + # behavior): the rr role is the one place galactic-router + # actually listens for inbound BGP -- the default role (see + # ../../base/daemonset.yaml) and the router container in + # config/galactic-gateway/base/daemonset.yaml both set + # GALACTIC_ROUTER_BGP_LISTEN_PORT=-1 and open no BGP port at all, + # so this entry only belongs on this patch, not the shared base. + ports: + - name: bgp + containerPort: 1790 + protocol: TCP diff --git a/config/galactic-router/overlays/rr/kustomization.yaml b/config/galactic-router/overlays/rr/kustomization.yaml new file mode 100644 index 00000000..eba62b8a --- /dev/null +++ b/config/galactic-router/overlays/rr/kustomization.yaml @@ -0,0 +1,10 @@ +resources: + - ../../base +# Only resource pulled in here is the DaemonSet from ../../base, so this +# can't accidentally rename the shared RBAC/ServiceAccount in ../../.. . +nameSuffix: -rr +patches: + - path: daemonset-patch.yaml + target: + kind: DaemonSet + name: galactic-router diff --git a/config/galactic-router/tenant-control/daemonset-patch.yaml b/config/galactic-router/tenant-control/daemonset-patch.yaml deleted file mode 100644 index 29fb584c..00000000 --- a/config/galactic-router/tenant-control/daemonset-patch.yaml +++ /dev/null @@ -1,54 +0,0 @@ -apiVersion: apps/v1 -kind: DaemonSet -metadata: - name: galactic-router - labels: - app.kubernetes.io/name: galactic-router-control -spec: - selector: - matchLabels: - app.kubernetes.io/name: galactic-router-control - template: - metadata: - labels: - app.kubernetes.io/name: galactic-router-control - spec: - # Route-reflector role: opt-in only, requires nodes explicitly labeled - # for it, so it never double-schedules with the per-worker tenant - # DaemonSet (see ../tenant/daemonset-patch.yaml). Also excludes - # Kubernetes control-plane nodes outright — the base DaemonSet's - # blanket toleration would otherwise let it land there if one ever - # carried the galactic.datumapis.com/node=control label. - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/control-plane - operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - control - containers: - - name: galactic-router - env: - - name: GALACTIC_ROUTER_REFLECTOR - value: "true" - - name: GALACTIC_ROUTER_BGP_LISTEN_PORT - value: "1790" - # GALACTIC_ROUTER_BGP_LOCAL_ADDRESS is auto-detected from the - # host's lo interface by default (the stable peering address - # other tenant routers reflect through). Set it explicitly here - # only if lo doesn't carry the desired address on this cluster. - # Documentation only (hostNetwork: true, so this doesn't change - # behavior): the reflector role is the one place galactic-router - # actually listens for inbound BGP -- tenant nodes (see - # ../tenant/daemonset-patch.yaml) and the router container in - # config/galactic-gateway/base/daemonset.yaml both set - # GALACTIC_ROUTER_BGP_LISTEN_PORT=-1 and open no BGP port at all, - # so this entry only belongs on this patch, not the shared base. - ports: - - name: bgp - containerPort: 1790 - protocol: TCP diff --git a/config/galactic-router/tenant-control/kustomization.yaml b/config/galactic-router/tenant-control/kustomization.yaml deleted file mode 100644 index 80e675a9..00000000 --- a/config/galactic-router/tenant-control/kustomization.yaml +++ /dev/null @@ -1,10 +0,0 @@ -resources: - - ../base -# Only resource pulled in here is the DaemonSet from ../base, so this can't -# accidentally rename the shared RBAC/ServiceAccount in ../.. . -nameSuffix: -control -patches: - - path: daemonset-patch.yaml - target: - kind: DaemonSet - name: galactic-router diff --git a/config/galactic-router/tenant/daemonset-patch.yaml b/config/galactic-router/tenant/daemonset-patch.yaml deleted file mode 100644 index 77cd6f2e..00000000 --- a/config/galactic-router/tenant/daemonset-patch.yaml +++ /dev/null @@ -1,41 +0,0 @@ -apiVersion: apps/v1 -kind: DaemonSet -metadata: - name: galactic-router - labels: - app.kubernetes.io/name: galactic-router-tenant -spec: - selector: - matchLabels: - app.kubernetes.io/name: galactic-router-tenant - template: - metadata: - labels: - app.kubernetes.io/name: galactic-router-tenant - spec: - # Tenant pods run on worker nodes explicitly labeled for it; - # opt-in only so new node types (GPU, monitoring, etc.) don't - # accidentally pick up the tenant. Also excludes Kubernetes - # control-plane nodes outright — the base DaemonSet's blanket - # toleration would otherwise let it land there if one ever - # carried the galactic.datumapis.com/node=edge label. - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/control-plane - operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - edge - containers: - - name: galactic-router - env: - - name: GALACTIC_ROUTER_BGP_LISTEN_PORT - # Tenant nodes only dial *out* to iBGP peers (the reflector, - # see ../tenant-control/daemonset-patch.yaml); -1 disables - # the inbound listener entirely, so no BGP containerPort - # belongs on this patch -- there's no port here to document. - value: "-1" diff --git a/deploy/containerlab/AGENTS.md b/deploy/containerlab/AGENTS.md index 691ed9bf..3d3c9186 100644 --- a/deploy/containerlab/AGENTS.md +++ b/deploy/containerlab/AGENTS.md @@ -3,18 +3,23 @@ ## Important Conventions - **YAML extensions**: Always use `.yaml`, never `.yml`. -- **fabric-router image**: `containers/fabric-router/Dockerfile` layers `tcpdump`, `mtr-tiny`, `telnet`, `dnsutils`, and `curl` onto the upstream `quay.io/frrouting/frr` image (version pinned via the Taskfile's `FRR_VERSION`, currently `10.2.0`). Built and tagged `fabric-router:latest` by `task build:fabric-router`, then used by both fabric DaemonSets (`resources/fabric-router/base/` and `resources/fabric-control/iad/` both build on `config/fabric-router/` — see the "Shared manifests" bullet below — loaded into the Kind clusters via `deploy:images`) and the transit routers (`gvpc.clab.yaml`'s `kinds.linux.image`, picked up directly from the local Docker image cache). All reference sites set/require `imagePullPolicy: Never` where applicable — there's no registry to pull `fabric-router:latest` from. The image also bakes in the two files common to every fabric role, `containers/fabric-router/{daemons,vtysh.conf}`, to `/etc/frr-defaults/` (not `/etc/frr` directly, since that's an `emptyDir` the `frr-init` container populates at pod start so a per-node `frr.conf` can still be layered in from the `fabric-config` ConfigMap — see `config/fabric-router/daemonset.yaml`'s init command). `frr-init` picks the right `frr.conf` via a `frr.conf.` ConfigMap key selected using the pod's `NODE_NAME` downward-API env var, since a DaemonSet's `nodeAffinity` can match more than one node and BGP underlay config differs per node. The lab's single-node-per-site clusters (dfw, sjc, and iad's `edge` role) each still only need one such key (`resources/fabric-router/{dfw,iad,sjc}/kustomization.yaml` key their `frr.conf` file as `frr.conf.`, e.g. `frr.conf.dfw-worker`); `resources/fabric-control/iad/kustomization.yaml` does the same for `iad-worker-control` (`resources/fabric-control/iad/frr.conf`) — no raw manifests remain under `resources/fabric-router/` or `resources/fabric-control/`. +- **fabric-router image**: `containers/fabric-router/Dockerfile` layers `tcpdump`, `mtr-tiny`, `telnet`, `dnsutils`, and `curl` onto the upstream `quay.io/frrouting/frr` image (version pinned via the Taskfile's `FRR_VERSION`, currently `10.2.0`). Built and tagged `fabric-router:latest` by `task build:fabric-router`, then used by both fabric DaemonSets (`resources/fabric-router/base/` and `resources/fabric-control/iad/` both build on `config/fabric-router/` — see the "Shared manifests" bullet below — loaded into the Kind clusters via `deploy:images`) and the transit routers (`gvpc.clab.yaml`'s `kinds.linux.image`, picked up directly from the local Docker image cache). All reference sites set/require `imagePullPolicy: Never` where applicable — there's no registry to pull `fabric-router:latest` from. The image also bakes in the two files common to every fabric role, `containers/fabric-router/{daemons,vtysh.conf}`, to `/etc/frr-defaults/` (not `/etc/frr` directly, since that's an `emptyDir` the `frr-init` container populates at pod start so a per-node `frr.conf` can still be layered in from the `fabric-config` ConfigMap — see `config/fabric-router/daemonset.yaml`'s init command). `frr-init` picks the right `frr.conf` via a `frr.conf.` ConfigMap key selected using the pod's `NODE_NAME` downward-API env var, since a DaemonSet's `nodeAffinity` can match more than one node and BGP underlay config differs per node. The lab's single-node-per-site clusters (dfw, sjc, and iad's `compute` role) each still only need one such key (`resources/fabric-router/{dfw,iad,sjc}/kustomization.yaml` key their `frr.conf` file as `frr.conf.`, e.g. `frr.conf.dfw-worker`); `resources/fabric-control/iad/kustomization.yaml` does the same for `iad-worker-rr` (`resources/fabric-control/iad/frr.conf`) — no raw manifests remain under `resources/fabric-router/` or `resources/fabric-control/`. - **Image loading**: Uses `ctr --namespace k8s.io images import` (not `kind load docker-image`) due to containerd v2 incompatibility. -- **iad-worker-control**: Created as `iad-worker2` in the topology, renamed post-deploy via `deploy:rename-control`. The Kind config sets the hostname via `kubeadmConfigPatches`. Runs the galactic-router route reflector (RR) for the iad region. +- **iad-worker-rr**: Created as `iad-worker2` in the topology, renamed post-deploy via `deploy:rename-control`. The Kind config sets the hostname via `kubeadmConfigPatches`. Runs the galactic-router route reflector (RR) for the iad region. - **galactic-router image**: Uses `galactic-router:latest` with `imagePullPolicy: Never` — stale images persist across rebuilds. - **Kind serviceSubnet**: All clusters use `/108` service subnet (non-standard; may cause issues with some services). -- **BGP listen port**: Tenant worker DaemonSets run with `GALACTIC_ROUTER_BGP_LISTEN_PORT=-1` (outbound-only). The RR role listens on `1790` (set in `config/galactic-router/tenant-control/daemonset-patch.yaml`). +- **BGP listen port**: Compute worker DaemonSets run with `GALACTIC_ROUTER_BGP_LISTEN_PORT=-1` (outbound-only). The RR role listens on `1790` (set in `config/galactic-router/overlays/rr/daemonset-patch.yaml`). - **BGP remote port**: BGPPeer CRDs inside `resources/galactic-router/{dfw,iad,sjc}/` explicitly set `remotePort: 1790` to connect to the RR. The control/RR-side peers in `resources/galactic-control/iad/` rely on galactic-router's default of `1790`. - **Worker SRv6 blackhole route**: A single standalone blackhole route covering the node's whole `/56` locator block (`metric 2048`, no interface — `ip -6 route add blackhole` takes no `dev`) prevents the default route from matching any USID this node could compute, set via `exec` commands in the ContainerLab topology. One aggregate route rather than one per VPC: IPv6 FIB lookup is longest-prefix-match first, so a real seg6local `/128` decap route always wins over the covering blackhole regardless of metric — the aggregate protects every current and future VPC's SID with no per-VPC config needed. The USID is **not** configured as an address on any interface. The seg6local route (`metric 1024`) in the main table handles SRv6 decapsulation. If the USID were an interface address, the kernel's local table route (metric 0) would shadow the seg6local route and break decapsulation. (A dedicated `lo-galactic` dummy interface previously hosted this — removed since it wasn't referenced by any Go code or by the blackhole route itself; `install.sh`'s `all`/`default` sysctl entries already cover the SRv6 kernel knobs it existed for.) -- **Node labels**: Workers use `galactic.datumapis.com/node: edge` (not `galactic.io/role: pop`). The control node uses `galactic.datumapis.com/node: control` with a matching `NoSchedule` taint. -- **GC namespace**: The tenant DaemonSet sets `GALACTIC_ROUTER_GC_NAMESPACE=galactic-system` for namespace-scoped garbage collection. +- **Node labels**: this lab follows the same label strategy as production (see the root `AGENTS.md`'s "Node label strategy" table) — independent labels per concern, not one enum, since some of these must coexist on the same node: + - `galactic.datumapis.com/node: compute` — the three site workers (`dfw-worker`, `sjc-worker`, `iad-worker`); deploys `galactic-router` (default role), `galactic-nat66`, and `galactic-cni`. NAT66 is *not* a separate opt-in label here — every compute node runs it unconditionally, so all three workers double as this lab's NAT66 shards. + - `galactic.datumapis.com/node: edge` — `iad-gateway1`/`iad-gateway2`; deploys `galactic-gateway`. This is the actual network-edge/ingress role, unrelated to `compute` despite the historical `edge` name collision this lab used to have before the label strategy split them apart. + - `galactic.datumapis.com/galactic-route-reflector: "true"` — `iad-worker-rr`; deploys `galactic-router-rr`. Not a `galactic.datumapis.com/node` value (that key is single-valued, so a route-reflector node could in principle also be `compute` at once) — a matching `NoSchedule` taint uses the same key/value. + - `galactic.datumapis.com/fabric: "true"` — every worker in the lab (currently all six); deploys `fabric-router`, independently of whatever other labels a node carries. + - See `node_files/{dfw,iad,sjc}/config.yaml` for the actual assignments, and `config/galactic-router/overlays/rr/daemonset-patch.yaml` / `config/galactic-nat66/base/daemonset.yaml` for the full reasoning behind each split. +- **GC namespace**: The default-role DaemonSet sets `GALACTIC_ROUTER_GC_NAMESPACE=galactic-system` for namespace-scoped garbage collection. - **FRR config**: Transit router configs omit the `frr version` directive (managed by the FRR image, not the config). -- **Shared manifests**: `resources/galactic-cni/kustomization.yaml` builds on `config/galactic-cni/`, `resources/galactic-router/base/kustomization.yaml` builds on `config/galactic-router/tenant/`, `resources/galactic-control/iad/kustomization.yaml` builds on `config/galactic-router/tenant-control/`, and both `resources/fabric-router/base/kustomization.yaml` and `resources/fabric-control/iad/kustomization.yaml` build on `config/fabric-router/` (each in turn pulls in its own `base/` where one exists: `config/galactic-router/base/`; `config/fabric-router/` has no role split, it's a single flat DaemonSet) rather than forking them, patching in only what the lab needs to differ (image, env — for both fabric overlays, just `image: fabric-router:latest` + `imagePullPolicy: Never` via each one's own `fabric-lab-patch.yaml`). `config/fabric-router/`'s own affinity already covers both node labels (`galactic.datumapis.com/node` `In` `[edge, control]`), but the *lab* still needs the router/control split back apart — iad's two nodes need different `frr.conf`, and a single shared ConfigMap can't serve both — so `resources/fabric-router/base/fabric-lab-patch.yaml` and `resources/fabric-control/iad/fabric-lab-patch.yaml` each narrow that affinity locally to just `edge` or just `control` respectively (on top of the image override), a lab-only concern with no production equivalent. Because both of those overlays land on the *same* iad cluster, they also can't share a DaemonSet name — but a strategic-merge `patches:` entry can't rename a resource (Kustomize keeps the `target`'s original identity regardless of what `metadata.name` the patch body says), so `resources/fabric-control/iad/kustomization.yaml` renames it to `fabric-control` via a second, JSON6902-style `patches:` entry (`op: replace, path: /metadata/name`) instead — `nameSuffix`/`namePrefix` transformers are the only other sanctioned way to rename, and neither fit here (nameSuffix would produce `fabric-router-control`, not `fabric-control`). `kubectl apply -k` refuses to load resource files from outside a kustomization's own root, so `deploy-cni.sh`/`deploy-galactic-router.sh`/`deploy-fabric.sh` `docker cp` those directories into a local `base/` subdirectory on the node at deploy time (`resources/galactic-cni/base/`, `resources/galactic-router/base/{base,tenant}/`, `resources/galactic-control/iad/{base,tenant-control}/`, `resources/fabric-router/base/fabric/`, `resources/fabric-control/iad/fabric/`) instead of referencing `config/` across that boundary. The `galactic-system` namespace and RBAC/ServiceAccount are applied straight from `config/galactic-system/namespace.yaml` and `config/*/{rbac,serviceaccount}.yaml` by `deploy-system.sh` via `lib.sh`'s `copy_config` (which copies all of `config/` to `/galactic/config/`) — not part of any kustomize build, so per-site `namePrefix` never touches the shared cluster-scoped RBAC. `config/galactic-system/namespace.yaml` is also the only place `galactic-system` gets created for production — there is no other bootstrap step in this repo, so if it's ever missing on a real cluster, deploying `config/galactic-router/` or `config/galactic-cni/` fails outright. +- **Shared manifests**: `resources/galactic-cni/kustomization.yaml` builds on `config/galactic-cni/`, `resources/galactic-router/base/kustomization.yaml` builds on `config/galactic-router/overlays/default/`, `resources/galactic-control/iad/kustomization.yaml` builds on `config/galactic-router/overlays/rr/`, and both `resources/fabric-router/base/kustomization.yaml` and `resources/fabric-control/iad/kustomization.yaml` build on `config/fabric-router/` (each in turn pulls in its own `base/` where one exists: `config/galactic-router/base/`; `config/fabric-router/` has no role split, it's a single flat DaemonSet) rather than forking them, patching in only what the lab needs to differ (image, env — for both fabric overlays, just `image: fabric-router:latest` + `imagePullPolicy: Never` via each one's own `fabric-lab-patch.yaml`). `config/fabric-router/`'s own affinity already covers every fabric-needing node via `galactic.datumapis.com/fabric` `Exists` (every worker in iad carries it, regardless of its other labels), but the *lab* still needs the router/route-reflector split back apart — iad's two nodes need different `frr.conf`, and a single shared ConfigMap can't serve both — so `resources/fabric-router/base/fabric-lab-patch.yaml` and `resources/fabric-control/iad/fabric-lab-patch.yaml` each narrow that affinity locally to just `compute`/`edge` or just `galactic-route-reflector` respectively (on top of the image override), a lab-only concern with no production equivalent. Because both of those overlays land on the *same* iad cluster, they also can't share a DaemonSet name — but a strategic-merge `patches:` entry can't rename a resource (Kustomize keeps the `target`'s original identity regardless of what `metadata.name` the patch body says), so `resources/fabric-control/iad/kustomization.yaml` renames it to `fabric-control` via a second, JSON6902-style `patches:` entry (`op: replace, path: /metadata/name`) instead — `nameSuffix`/`namePrefix` transformers are the only other sanctioned way to rename, and neither fit here (nameSuffix would produce `fabric-router-control`, not `fabric-control`). `kubectl apply -k` refuses to load resource files from outside a kustomization's own root, so `deploy-cni.sh`/`deploy-galactic-router.sh`/`deploy-fabric.sh` `docker cp` those directories into a local `base/` subdirectory on the node at deploy time (`resources/galactic-cni/base/`, `resources/galactic-router/base/{base,overlays/default}/`, `resources/galactic-control/iad/{base,overlays/rr}/` — each nested at the same relative depth as `config/galactic-router/` itself, since the copied `overlays/default`/`overlays/rr` kustomization.yamls' own `../../base` references are copied verbatim, `resources/fabric-router/base/fabric/`, `resources/fabric-control/iad/fabric/`) instead of referencing `config/` across that boundary. The `galactic-system` namespace and RBAC/ServiceAccount are applied straight from `config/galactic-system/namespace.yaml` and `config/*/{rbac,serviceaccount}.yaml` by `deploy-system.sh` via `lib.sh`'s `copy_config` (which copies all of `config/` to `/galactic/config/`) — not part of any kustomize build, so per-site `namePrefix` never touches the shared cluster-scoped RBAC. `config/galactic-system/namespace.yaml` is also the only place `galactic-system` gets created for production — there is no other bootstrap step in this repo, so if it's ever missing on a real cluster, deploying `config/galactic-router/` or `config/galactic-cni/` fails outright. ## Naming Layers diff --git a/deploy/containerlab/README.md b/deploy/containerlab/README.md index 5b84a5cf..ebf7700a 100644 --- a/deploy/containerlab/README.md +++ b/deploy/containerlab/README.md @@ -14,31 +14,31 @@ over iBGP to the route reflector on iad-control. │ ╱ ╲ │ (mesh) (mesh) tr3 ──eth5── iad-worker - tr3 ──eth4── iad-worker-control + tr3 ──eth4── iad-worker-rr tr3 ──eth6── iad-gateway1 tr3 ──eth7── iad-gateway2 ``` ### Node roles -| Node | Kind | Role | -|----------------------------------------------------------|---------------|-------------------------------------------------------| -| `dfw-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | -| `dfw-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE | -| `iad-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | -| `iad-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE | -| `iad-worker2` (renamed `iad-worker-control` post-deploy) | ext-container | Kind worker; runs FRR PE + galactic-router RR | -| `iad-worker3` (renamed `iad-gateway1` post-deploy) | ext-container | Kind worker; edge XDP NAT+LB gateway canary (Phase D) | -| `iad-worker4` (renamed `iad-gateway2` post-deploy) | ext-container | Kind worker; edge XDP NAT+LB gateway canary (Phase D) | -| `sjc-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | -| `sjc-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE | -| `tr1`–`tr4` | linux (FRR) | iBGP full mesh, AS 65100 | - -`iad-gateway1`/`iad-gateway2` are tainted (`galactic.datumapis.com/node=gateway:NoSchedule`) -dedicated nodes, same idea as `iad-worker-control`'s taint: no tenant pods land there, only +| Node | Kind | Role | +|-----------------------------------------------------|---------------|-------------------------------------------------------| +| `dfw-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | +| `dfw-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE | +| `iad-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | +| `iad-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE | +| `iad-worker2` (renamed `iad-worker-rr` post-deploy) | ext-container | Kind worker; runs FRR PE + galactic-router RR | +| `iad-worker3` (renamed `iad-gateway1` post-deploy) | ext-container | Kind worker; edge XDP NAT+LB gateway canary (Phase D) | +| `iad-worker4` (renamed `iad-gateway2` post-deploy) | ext-container | Kind worker; edge XDP NAT+LB gateway canary (Phase D) | +| `sjc-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus | +| `sjc-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE | +| `tr1`–`tr4` | linux (FRR) | iBGP full mesh, AS 65100 | + +`iad-gateway1`/`iad-gateway2` are tainted (`galactic.datumapis.com/node=edge:NoSchedule`) +dedicated nodes, same idea as `iad-worker-rr`'s taint: no tenant pods land there, only DaemonSets with a blanket toleration (`fabric-router`, `galactic-gateway1`/`-gateway2` — each a two-container pod, `galactic-router` + `galactic-gateway`). -They never run `galactic-cni` (config/galactic-cni's affinity is edge-only) or a route-reflector. +They never run `galactic-cni` (config/galactic-cni's affinity is compute-only) or a route-reflector. **Underlay BGP peering on their `tr3` uplinks is wired** (`node_files/tr3/frr.conf`, plus two `BGPPeer` objects in `resources/galactic-control/iad/` for the route reflector side) and the full fabric converges. Real end-to-end ingress traffic through the datapath is now @@ -86,7 +86,7 @@ AS 65000 (sjc-tenant / galactic-router) ──iBGP── iad-control-tenant - The transit mesh carries IPv6 unicast (SRv6 locator prefixes and loopbacks) via iBGP within AS 65100. - FRR PE nodes originate their per-node SRv6 locator block (`2001:db8:ffXX:100::/56`) and BGP peering loopback (`fc00:0:X::1/128`) toward the transit layer via eBGP over numbered IPv6 links — never the site's full `/48` uSID Block or loopback pool, which would create an anycast ambiguity once a second worker joins a site. - `allowas-in 1` is configured on all cluster FRR instances so each site accepts prefixes that carry AS 65000 in the path — necessary because the transit reflects routes from one AS 65000 site to another. -- galactic-router instances on dfw/iad/sjc workers peer with iad-worker-control over iBGP (AS 65000) for `l2vpn-evpn` routes. GoBGP runs with outbound-only mode (`listenPort=-1`); all BGP sessions are initiated outbound. +- galactic-router instances on dfw/iad/sjc workers peer with iad-worker-rr over iBGP (AS 65000) for `l2vpn-evpn` routes. GoBGP runs with outbound-only mode (`listenPort=-1`); all BGP sessions are initiated outbound. ## Addressing @@ -101,25 +101,25 @@ AS 65000 (sjc-tenant / galactic-router) ──iBGP── iad-control-tenant ### TR–TR point-to-point links (numbered) -| Link | Subnet | -|---------|---------------------| -| tr1–tr2 | 2001:db8:0:12::/64 | -| tr1–tr3 | 2001:db8:0:13::/64 | -| tr1–tr4 | 2001:db8:0:14::/64 | -| tr2–tr3 | 2001:db8:0:23::/64 | -| tr2–tr4 | 2001:db8:0:24::/64 | -| tr3–tr4 | 2001:db8:0:34::/64 | +| Link | Subnet | +|---------|--------------------| +| tr1–tr2 | 2001:db8:0:12::/64 | +| tr1–tr3 | 2001:db8:0:13::/64 | +| tr1–tr4 | 2001:db8:0:14::/64 | +| tr2–tr3 | 2001:db8:0:23::/64 | +| tr2–tr4 | 2001:db8:0:24::/64 | +| tr3–tr4 | 2001:db8:0:34::/64 | ### Worker–TR links (numbered, eBGP) -| Link | Subnet | TR address | Worker address | -|--------------------------|--------------------|------------------|------------------| -| dfw-worker – tr1 | 2001:db8:1:10::/64 | 2001:db8:1:10::1 | 2001:db8:1:10::2 | -| sjc-worker – tr2 | 2001:db8:1:20::/64 | 2001:db8:1:20::1 | 2001:db8:1:20::2 | -| iad-worker – tr3 | 2001:db8:1:30::/64 | 2001:db8:1:30::1 | 2001:db8:1:30::2 | -| iad-worker-control – tr3 | 2001:db8:1:31::/64 | 2001:db8:1:31::1 | 2001:db8:1:31::2 | -| iad-gateway1 – tr3 | 2001:db8:1:32::/64 | 2001:db8:1:32::1 | 2001:db8:1:32::2 | -| iad-gateway2 – tr3 | 2001:db8:1:33::/64 | 2001:db8:1:33::1 | 2001:db8:1:33::2 | +| Link | Subnet | TR address | Worker address | +|---------------------|--------------------|------------------|------------------| +| dfw-worker – tr1 | 2001:db8:1:10::/64 | 2001:db8:1:10::1 | 2001:db8:1:10::2 | +| sjc-worker – tr2 | 2001:db8:1:20::/64 | 2001:db8:1:20::1 | 2001:db8:1:20::2 | +| iad-worker – tr3 | 2001:db8:1:30::/64 | 2001:db8:1:30::1 | 2001:db8:1:30::2 | +| iad-worker-rr – tr3 | 2001:db8:1:31::/64 | 2001:db8:1:31::1 | 2001:db8:1:31::2 | +| iad-gateway1 – tr3 | 2001:db8:1:32::/64 | 2001:db8:1:32::1 | 2001:db8:1:32::2 | +| iad-gateway2 – tr3 | 2001:db8:1:33::/64 | 2001:db8:1:33::1 | 2001:db8:1:33::2 | ### Cluster SRv6 addressing @@ -132,9 +132,9 @@ a matching seg6local route installed, for any current or future VPC — not just the ones with a pod running today. The FRR fabric DaemonSet advertises the same `/56` into the transit mesh via a static Null0 route + BGP `network` statement. -Each site's tenant node advertises its own `/56` SRv6 locator block into the +Each site's compute node advertises its own `/56` SRv6 locator block into the fabric — never the site's full `/48` uSID Block, which would create an -anycast ambiguity the instant a second tenant node joins a site. The test VPC +anycast ambiguity the instant a second compute node joins a site. The test VPC `ns10` (see [docs/tenants.md](docs/tenants.md)) gets a host address within its node's block (illustrative only — the exact hextet depends on allocation order; see docs/tenants.md's [SRv6 USID Argument allocation](docs/tenants.md#srv6-usid-argument-allocation)): @@ -169,17 +169,17 @@ supplied statically through `GALACTIC_GATEWAY_SRV6_ADDRESS` (originally ### Management network (fc00:10::/64) -| Node | Address | -|--------------------------------------------|--------------| -| dfw-control-plane | fc00:10::102 | -| dfw-worker | fc00:10::103 | -| sjc-control-plane | fc00:10::122 | -| sjc-worker | fc00:10::123 | -| iad-control-plane | fc00:10::112 | -| iad-worker | fc00:10::113 | -| iad-worker2 (renamed `iad-worker-control`) | fc00:10::114 | -| iad-worker3 (renamed `iad-gateway1`) | fc00:10::115 | -| iad-worker4 (renamed `iad-gateway2`) | fc00:10::116 | +| Node | Address | +|---------------------------------------|--------------| +| dfw-control-plane | fc00:10::102 | +| dfw-worker | fc00:10::103 | +| sjc-control-plane | fc00:10::122 | +| sjc-worker | fc00:10::123 | +| iad-control-plane | fc00:10::112 | +| iad-worker | fc00:10::113 | +| iad-worker2 (renamed `iad-worker-rr`) | fc00:10::114 | +| iad-worker3 (renamed `iad-gateway1`) | fc00:10::115 | +| iad-worker4 (renamed `iad-gateway2`) | fc00:10::116 | ## Lab layout @@ -260,7 +260,7 @@ task deploy | `deploy` | Build images, apply host sysctls, and deploy the lab | | `deploy:topology` | Deploy the ContainerLab topology (transit routers) | | `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | -| `deploy:rename-control` | Rename `iad-worker2`→`iad-worker-control`, `iad-worker3/4`→`iad-gateway1/2` | +| `deploy:rename-control` | Rename `iad-worker2`→`iad-worker-rr`, `iad-worker3/4`→`iad-gateway1/2` | | `deploy:images` | Load container images into Kind clusters | | `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC | | `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | @@ -313,4 +313,4 @@ task verify # automated: bgp-transit, bgp-fabric, bgp-peers, srv6, evpn - Worker–TR links use numbered IPv6 subnets (/64) with eBGP peering. - Cilium's iptables rules block BGP by default; the worker bootstrap script (`install.sh`) inserts `ip6tables -I INPUT` rules for TCP/179 before Cilium starts. -- iad-worker-control peers with tr3 as AS 65000, the same AS used by all three clusters. +- iad-worker-rr peers with tr3 as AS 65000, the same AS used by all three clusters. diff --git a/deploy/containerlab/Taskfile.yaml b/deploy/containerlab/Taskfile.yaml index b2342a76..be09cdb5 100644 --- a/deploy/containerlab/Taskfile.yaml +++ b/deploy/containerlab/Taskfile.yaml @@ -30,7 +30,7 @@ tasks: - docker build --network=host -t kindest/node:galactic -f containers/kindest-node-galactic/Dockerfile ../.. "build:fabric-router": - desc: Build the fabric-router image (FRR v{{.FRR_VERSION}} + debug tooling) + desc: Build the fabric-router image (FRR v{{.FRR_VERSION}}) cmds: - docker build --network=host --build-arg FRR_VERSION={{.FRR_VERSION}} -t {{.FABRIC_ROUTER_IMAGE}} -f ../../containers/fabric-router/Dockerfile ../.. @@ -60,12 +60,12 @@ tasks: - docker build --network=host --build-context network=../../../network -t galactic-cni:latest -f ../../containers/galactic-cni/Dockerfile ../.. "build:galactic-nat66": - desc: Build the galactic-nat66 sharded egress datapath image + desc: Build the galactic-nat66 egress datapath image cmds: - docker build --network=host --build-context network=../../../network -t galactic-nat66:latest -f ../../containers/galactic-nat66/Dockerfile ../.. deploy: - desc: Build images and deploy the full lab end-to-end + desc: Build images and deploy the full lab deps: [build, host-setup] cmds: - task: "deploy:clusters" @@ -98,7 +98,7 @@ tasks: "deploy:rename-control": desc: Rename iad worker nodes cmds: - - docker rename iad-worker2 iad-worker-control + - docker rename iad-worker2 iad-worker-rr - docker rename iad-worker3 iad-gateway1 - docker rename iad-worker4 iad-gateway2 @@ -119,18 +119,18 @@ tasks: fi "build:lab-xdp-passthrough": - desc: Compile the lab-only pass-through XDP program (see node_files/common/xdp-passthrough.c) + desc: Compile the lab-only pass-through XDP program deps: [require-clang] cmds: - clang -O2 -g -target bpf -c node_files/common/xdp-passthrough.c -o node_files/common/xdp-passthrough.o + # Loads the pass-through XDP program on tr3's gateway-facing links + # (eth6/eth7) -- makes those veth links behave like a real gateway's + # physical NIC uplink for edgedsr.c's XDP_TX redirect (see + # xdp-passthrough.c's own doc comment). Idempotent: `ip link set ... + # xdp obj` replaces whatever program (if any) is already attached. "deploy:lab-xdp-passthrough": - desc: >- - Load the pass-through XDP program on tr3's gateway-facing links - (eth6/eth7) -- makes those veth links behave like a real gateway's - physical NIC uplink for edgedsr.c's XDP_TX redirect (see - xdp-passthrough.c's own doc comment). Idempotent: `ip link set ... - xdp obj` replaces whatever program (if any) is already attached. + desc: Load the pass-through XDP program on tr3 deps: [build:lab-xdp-passthrough] vars: # tr3 is a real containerlab-managed node (kind: linux), unlike the @@ -146,21 +146,23 @@ tasks: - docker exec {{.TR3_CONTAINER}} ip -force link set dev eth6 xdp obj /xdp-passthrough.o sec xdp - docker exec {{.TR3_CONTAINER}} ip -force link set dev eth7 xdp obj /xdp-passthrough.o sec xdp + # Uses ctr directly instead of `kind load docker-image`, which is + # incompatible with containerd v2. "load-image": internal: true - desc: Load a docker image into a kind node via ctr (workaround for containerd v2 incompatibility) + desc: Load a docker image into a kind node via ctr requires: vars: [IMAGE, NODE] cmds: - docker save {{.IMAGE}} | docker exec -i {{.NODE}} ctr --namespace k8s.io images import - "deploy:images": - desc: Load all container images into the Kind clusters + desc: Load all container images into Kind cmds: - task: load-image vars: {IMAGE: "{{.FABRIC_ROUTER_IMAGE}}", NODE: iad-worker} - task: load-image - vars: {IMAGE: "{{.FABRIC_ROUTER_IMAGE}}", NODE: iad-worker-control} + vars: {IMAGE: "{{.FABRIC_ROUTER_IMAGE}}", NODE: iad-worker-rr} # iad-gateway1/2 need fabric-router (the underlay eBGP session the # gateway XDP datapath's uplink depends on -- see config/fabric-router/'s # affinity, extended to the gateway role), galactic-router (tenant @@ -168,7 +170,7 @@ tasks: # pod), and galactic-gateway (the edge XDP NAT+LB engine, the other # container in that pod), but NOT galactic-cni: they never run the # CNI DaemonSet at all (it only - # targets the edge role). + # targets the compute role). - task: load-image vars: {IMAGE: "{{.FABRIC_ROUTER_IMAGE}}", NODE: iad-gateway1} - task: load-image @@ -188,7 +190,7 @@ tasks: - task: load-image vars: {IMAGE: galactic-router:latest, NODE: iad-worker} - task: load-image - vars: {IMAGE: galactic-router:latest, NODE: iad-worker-control} + vars: {IMAGE: galactic-router:latest, NODE: iad-worker-rr} - task: load-image vars: {IMAGE: galactic-router:latest, NODE: sjc-worker} - task: load-image @@ -196,7 +198,7 @@ tasks: - task: load-image vars: {IMAGE: galactic-cni:latest, NODE: iad-worker} - task: load-image - vars: {IMAGE: galactic-cni:latest, NODE: iad-worker-control} + vars: {IMAGE: galactic-cni:latest, NODE: iad-worker-rr} - task: load-image vars: {IMAGE: galactic-cni:latest, NODE: sjc-worker} - task: load-image @@ -214,7 +216,7 @@ tasks: - ./scripts/deploy-system.sh "deploy:cni": - desc: Install Cilium and Multus, then the galactic-cni DaemonSet + desc: Install Cilium, Multus, and galactic-cni cmds: - ./scripts/deploy-cni.sh @@ -227,12 +229,12 @@ tasks: - kind delete cluster --name iad || true inspect: - desc: Inspect deployed nodes and management addresses + desc: List deployed nodes and mgmt addresses cmds: - sudo containerlab inspect -t {{.TOPO}} graph: - desc: Generate a draw.io diagram for the current topology + desc: Generate a draw.io diagram of the topology cmds: - sudo containerlab graph -t {{.TOPO}} --drawio --drawio-dir . @@ -247,12 +249,12 @@ tasks: - ./scripts/deploy-fabric.sh "deploy:galactic-router": - desc: Install galactic-router DaemonSets and BGP resources + desc: Install galactic-router and BGP resources cmds: - ./scripts/deploy-galactic-router.sh "deploy:galactic-nat66": - desc: Install the sharded NAT66 egress DaemonSet and NAT66Shard objects + desc: Install the NAT66 egress DaemonSet and shards cmds: - ./scripts/deploy-galactic-nat66.sh @@ -271,7 +273,7 @@ tasks: - ./scripts/deploy-ns.sh ns10 dfw sjc iad "deploy:ns20": - desc: Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) + desc: Deploy ns20 test VPC (dual-stack) cmds: - ./scripts/deploy-ns.sh ns20 dfw sjc iad @@ -286,7 +288,7 @@ tasks: - ./scripts/deploy-ns.sh ns40 iad "deploy:ns60": - desc: Deploy ns60 test VPC, nginx backend for the gateway canary + desc: Deploy ns60 test VPC (gateway canary backend) cmds: - ./scripts/deploy-ns.sh ns60 iad @@ -303,7 +305,7 @@ tasks: - task: "verify:scenarios" "verify:bgp-transit": - desc: Verify transit router BGP sessions (iBGP full mesh) + desc: Verify transit router BGP sessions cmds: - | for r in tr1 tr2 tr3 tr4; do @@ -312,7 +314,7 @@ tasks: done "verify:bgp-fabric": - desc: Verify fabric BGP sessions on dfw, iad, and sjc workers + desc: Verify fabric BGP sessions on all sites cmds: - | docker exec dfw-control-plane \ @@ -332,14 +334,14 @@ tasks: -- vtysh -c "show bgp ipv6 unicast summary" "verify:bgp-peers": - desc: Verify galactic-router BGP peer sessions (GoBGP) + desc: Verify galactic-router BGP peer sessions cmds: - docker exec dfw-control-plane kubectl get bgppeers -n galactic-system - docker exec sjc-control-plane kubectl get bgppeers -n galactic-system - docker exec iad-control-plane kubectl get bgppeers -n galactic-system "verify:srv6": - desc: Verify each site's SRv6 locator prefix on tr1 + desc: Verify each site's SRv6 locator prefix cmds: - docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff01::/48" - docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff02::/48" @@ -359,7 +361,7 @@ tasks: - docker exec iad-control-plane kubectl get daemonset -n galactic-system -l app.kubernetes.io/name=galactic-gateway "verify:nat66-sharding": - desc: Verify each site's NAT66Shard is Ready and its shard SID is advertised + desc: Verify each site's NAT66Shard and shard SID cmds: - | for site in dfw sjc iad; do diff --git a/deploy/containerlab/docs/tenants.md b/deploy/containerlab/docs/tenants.md index cd17218b..b3c16b01 100644 --- a/deploy/containerlab/docs/tenants.md +++ b/deploy/containerlab/docs/tenants.md @@ -19,12 +19,12 @@ reachability across sites. They differ only in scope and addressing: -| VPC | Sites | Address families | VRF interface | Notes | -|--------|------------------------------|-------------------|-------------------|-------| -| `ns10` | dfw, sjc, iad (3-site) | IPv6-only (fd20 ULA) | `G000000010V` | No `ipv4_subnet` at all. | -| `ns20` | dfw, sjc, iad (3-site) | Dual-stack (fd20 ULA + IPv4) | `G000000020V` | Both families active; exercises the dual-stack IPAM path. | -| `ns30` | dfw only, 2 attachments | IPv6-only (fd20 ULA) | `G000000030V` | Two distinct attachments (`private`/`private-b`, distinct `vpcattachment` values, same `vpc`), each its own single-replica Deployment, both land on `dfw-worker` and share one VRF — same-node connectivity, no cross-site hop. `verify:ns30` asserts the two pods share a node. | -| `ns40` | iad only, 2 attachments | IPv4-only | `G000000040V` | Two distinct attachments (`private`/`private-b`), each its own single-replica Deployment, both land on `iad-worker` (not `iad-worker-control`, which is tainted for the route-reflector role) and share one VRF. `verify:ns40` asserts the two pods share a node. | +| VPC | Sites | Address families | VRF interface | Notes | +|--------|-------------------------|------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `ns10` | dfw, sjc, iad (3-site) | IPv6-only (fd20 ULA) | `G000000010V` | No `ipv4_subnet` at all. | +| `ns20` | dfw, sjc, iad (3-site) | Dual-stack (fd20 ULA + IPv4) | `G000000020V` | Both families active; exercises the dual-stack IPAM path. | +| `ns30` | dfw only, 2 attachments | IPv6-only (fd20 ULA) | `G000000030V` | Two distinct attachments (`private`/`private-b`, distinct `vpcattachment` values, same `vpc`), each its own single-replica Deployment, both land on `dfw-worker` and share one VRF — same-node connectivity, no cross-site hop. `verify:ns30` asserts the two pods share a node. | +| `ns40` | iad only, 2 attachments | IPv4-only | `G000000040V` | Two distinct attachments (`private`/`private-b`), each its own single-replica Deployment, both land on `iad-worker` (not `iad-worker-rr`, which is tainted for the route-reflector role) and share one VRF. `verify:ns40` asserts the two pods share a node. | The VRF interface name follows `GV` on every worker — e.g. `ns20` (`vpc="20"`) is `G000000020V`. Unlike the host/guest veth @@ -50,7 +50,7 @@ collision while still landing both pods in the one shared VRF their common ### SRv6 USID Argument allocation -Each site's tenant node advertises its own `/56` SRv6 locator block into the +Each site's compute node advertises its own `/56` SRv6 locator block into the fabric. The low hextet of a pod's USID is `(Function << 12) | Argument` (`uFMT 48+16`, `internal/plumbing/ebpf/uformat`): `Function` is the constant `0xE` (`FunctionEndDT46`) for every plain L3 VRF attachment, and `Argument` is @@ -73,18 +73,18 @@ docker exec dfw-control-plane kubectl get bgpvrfinstances -A ## Addressing reference -| Site | VPC | IPv6 pool (fd20 ULA) | IPv4 subnet | -|------|--------|------------------------|--------------------| -| dfw | `ns10` | `fd20:10:ff01::/48` | none | -| sjc | `ns10` | `fd20:10:ff02::/48` | none | -| iad | `ns10` | `fd20:10:ff03::/48` | none | -| dfw | `ns20` | `fd20:20:ff01::/48` | `172.21.1.0/24` | -| sjc | `ns20` | `fd20:20:ff02::/48` | `172.21.20.0/24` | -| iad | `ns20` | `fd20:20:ff03::/48` | `172.21.10.0/24` | -| dfw | `ns30` (`private`) | `fd20:30:ff01::/48` | none | -| dfw | `ns30` (`private-b`) | `fd20:30:ff02::/48` | none | -| iad | `ns40` (`private`) | none | `172.40.10.0/24` | -| iad | `ns40` (`private-b`) | none | `172.40.20.0/24` | +| Site | VPC | IPv6 pool (fd20 ULA) | IPv4 subnet | +|------|----------------------|----------------------|------------------| +| dfw | `ns10` | `fd20:10:ff01::/48` | none | +| sjc | `ns10` | `fd20:10:ff02::/48` | none | +| iad | `ns10` | `fd20:10:ff03::/48` | none | +| dfw | `ns20` | `fd20:20:ff01::/48` | `172.21.1.0/24` | +| sjc | `ns20` | `fd20:20:ff02::/48` | `172.21.20.0/24` | +| iad | `ns20` | `fd20:20:ff03::/48` | `172.21.10.0/24` | +| dfw | `ns30` (`private`) | `fd20:30:ff01::/48` | none | +| dfw | `ns30` (`private-b`) | `fd20:30:ff02::/48` | none | +| iad | `ns40` (`private`) | none | `172.40.10.0/24` | +| iad | `ns40` (`private-b`) | none | `172.40.20.0/24` | `ns20`'s IPv4 pool (`172.21.0.0/16`) and `ns40`'s (`172.40.0.0/16`) are deliberately from distinct `/16` blocks so the two VPCs' addressing never @@ -144,7 +144,7 @@ For the 3-site VPCs (`ns10`, `ns20`), expect one `Running` pod per site. For the single-site VPCs, expect **two** `Running` pods — one per attachment (`private`/`private-b`, distinct Deployments) — both on the one site's worker (`dfw-worker` for `ns30`, `iad-worker` for `ns40` — not -`iad-worker-control`, which carries the route-reflector taint). +`iad-worker-rr`, which carries the route-reflector taint). ### Inspect a pod's VPC interface @@ -273,16 +273,16 @@ docker exec dfw-worker dmesg | grep galactic VPCs (`ns30`/`ns40`), a healthy ping also depends only on this VRF's local routes/neighbor table — there's no cross-site EVPN dependency to check. -### Pod scheduled on iad-worker-control instead of iad-worker (ns40) +### Pod scheduled on iad-worker-rr instead of iad-worker (ns40) Every test VPC's Deployment uses a `node-role.kubernetes.io/control-plane -DoesNotExist` affinity, but that doesn't exclude `iad-worker-control` — it's +DoesNotExist` affinity, but that doesn't exclude `iad-worker-rr` — it's a tainted *worker*, not a Kubernetes control-plane node. If an `ns40` pod ends up `Pending`, confirm the route-reflector taint is still in place rather than assuming the affinity alone keeps pods off it: ```bash -docker exec iad-control-plane kubectl describe node iad-worker-control | grep -A2 Taints +docker exec iad-control-plane kubectl describe node iad-worker-rr | grep -A2 Taints ``` If the taint is gone the pods schedule fine but may split across the two diff --git a/deploy/containerlab/gvpc.clab.yaml b/deploy/containerlab/gvpc.clab.yaml index 4252caec..84564d55 100644 --- a/deploy/containerlab/gvpc.clab.yaml +++ b/deploy/containerlab/gvpc.clab.yaml @@ -149,12 +149,12 @@ topology: # matching declaration order in node_files/iad/config.yaml) even though # their Kubernetes node names are iad-gateway1/iad-gateway2 (set via # that file's kubeadmConfigPatches, same indirection iad-worker2/ - # iad-worker-control already uses) -- Taskfile.yaml's deploy:rename-* + # iad-worker-rr already uses) -- Taskfile.yaml's deploy:rename-* # step renames both to match after cluster creation, purely cosmetic # (doesn't affect the links below, which are wired to the container # before the rename happens). No blackhole route in `exec`: unlike # iad-worker, these nodes never run galactic-veth's usid.c decap - # program at all (config/galactic-cni's DaemonSet only targets the edge role), + # program at all (config/galactic-cni's DaemonSet only targets the compute role), # so the blackhole-route workaround it needs doesn't apply here. iad-worker3: kind: ext-container @@ -250,7 +250,7 @@ topology: # node links below already document in detail: galactic-nat66's own # XDP program (nat66attach.go, also native/driver-mode-only) attaches # to this exact interface (eth1, shared with the TC-BPF SRv6 uSID - # datapath, which has no such restriction), and every tenant worker is + # datapath, which has no such restriction), and every compute worker is # now also a NAT66 shard (design plan §8's "reuse the three site # workers as shards"). Confirmed live: galactic-nat66 crash-looped on # dfw-worker/sjc-worker/iad-worker with exactly this ERANGE @@ -303,7 +303,7 @@ topology: # # mtu: 1500 on both ends, unlike every other link in this topology # (which gets containerlab's jumbo-frame veth default, ~9500) -- - # required, not cosmetic: galactic-router's edge NAT+LB datapath + # required, not cosmetic: galactic-gateway's edge NAT+LB datapath # attaches its XDP program to this link's gateway-side interface in # native/driver mode (edgeattach.go, XDPDriverMode; generic/SKB mode # is deliberately not attempted, see that package's doc comment), and diff --git a/deploy/containerlab/node_files/dfw/config.yaml b/deploy/containerlab/node_files/dfw/config.yaml index 2233fe67..3ee8e227 100644 --- a/deploy/containerlab/node_files/dfw/config.yaml +++ b/deploy/containerlab/node_files/dfw/config.yaml @@ -5,7 +5,8 @@ nodes: - role: worker labels: topology.kubernetes.io/region: dfw - galactic.datumapis.com/node: edge + galactic.datumapis.com/node: compute + galactic.datumapis.com/fabric: "true" networking: ipFamily: ipv6 disableDefaultCNI: true diff --git a/deploy/containerlab/node_files/iad/config.yaml b/deploy/containerlab/node_files/iad/config.yaml index b72fb86d..81ca5713 100644 --- a/deploy/containerlab/node_files/iad/config.yaml +++ b/deploy/containerlab/node_files/iad/config.yaml @@ -5,26 +5,28 @@ nodes: - role: worker labels: topology.kubernetes.io/region: iad - galactic.datumapis.com/node: edge + galactic.datumapis.com/node: compute + galactic.datumapis.com/fabric: "true" - role: worker labels: topology.kubernetes.io/region: iad - galactic.datumapis.com/node: control + galactic.datumapis.com/galactic-route-reflector: "true" + galactic.datumapis.com/fabric: "true" kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: - name: iad-worker-control + name: iad-worker-rr taints: - - key: galactic.datumapis.com/node - value: control + - key: galactic.datumapis.com/galactic-route-reflector + value: "true" effect: NoSchedule # Dedicated edge XDP NAT+LB gateway nodes -- canary for # config/galactic-gateway/base/. Two nodes, active-active, matching the design's # own topology requirement (see # internal/gateway/doc.go): a single gateway node would still work but # would never actually exercise the primary/secondary BGP local-pref - # failover path. Tainted the same way the control role is above, to + # failover path. Tainted the same way the route-reflector node is above, to # keep ordinary tenant pods (resources/tenants/*/base/pod.yaml's # netshoot Deployments) off these dedicated nodes -- only DaemonSets # with a blanket toleration (config/galactic-router/base/daemonset.yaml, @@ -32,7 +34,8 @@ nodes: - role: worker labels: topology.kubernetes.io/region: iad - galactic.datumapis.com/node: gateway + galactic.datumapis.com/node: edge + galactic.datumapis.com/fabric: "true" kubeadmConfigPatches: - | kind: JoinConfiguration @@ -40,12 +43,13 @@ nodes: name: iad-gateway1 taints: - key: galactic.datumapis.com/node - value: gateway + value: edge effect: NoSchedule - role: worker labels: topology.kubernetes.io/region: iad - galactic.datumapis.com/node: gateway + galactic.datumapis.com/node: edge + galactic.datumapis.com/fabric: "true" kubeadmConfigPatches: - | kind: JoinConfiguration @@ -53,7 +57,7 @@ nodes: name: iad-gateway2 taints: - key: galactic.datumapis.com/node - value: gateway + value: edge effect: NoSchedule networking: ipFamily: ipv6 diff --git a/deploy/containerlab/node_files/sjc/config.yaml b/deploy/containerlab/node_files/sjc/config.yaml index cbcb0676..b330492e 100644 --- a/deploy/containerlab/node_files/sjc/config.yaml +++ b/deploy/containerlab/node_files/sjc/config.yaml @@ -5,7 +5,8 @@ nodes: - role: worker labels: topology.kubernetes.io/region: sjc - galactic.datumapis.com/node: edge + galactic.datumapis.com/node: compute + galactic.datumapis.com/fabric: "true" networking: ipFamily: ipv6 disableDefaultCNI: true diff --git a/deploy/containerlab/resources/fabric-control/iad/fabric-lab-patch.yaml b/deploy/containerlab/resources/fabric-control/iad/fabric-lab-patch.yaml index b2f4076c..76863cc8 100644 --- a/deploy/containerlab/resources/fabric-control/iad/fabric-lab-patch.yaml +++ b/deploy/containerlab/resources/fabric-control/iad/fabric-lab-patch.yaml @@ -5,11 +5,16 @@ metadata: spec: template: spec: - # Narrower than config/fabric-router/'s default (edge or control): this is - # the control-node-only fabric pod (iad-worker-control), which needs - # its own frr.conf distinct from iad's regular per-worker fabric - # (resources/fabric-router/iad/) — so this overlay sticks to control - # only, or the two would double-schedule onto the same node. + # Narrower than config/fabric-router/'s default (galactic.datumapis.com/fabric + # Exists, matching any role): this is the route-reflector-node-only + # fabric pod (iad-worker-rr), which needs its own frr.conf distinct + # from iad's regular per-worker fabric (resources/fabric-router/iad/) + # — so this overlay sticks to galactic.datumapis.com/galactic-route-reflector, + # or the two would double-schedule onto the same node. Not + # galactic.datumapis.com/node: route-reflector -- that's no longer a + # node value at all (see config/galactic-router/overlays/rr/ + # daemonset-patch.yaml's own affinity comment for why route-reflector + # is an independent flag instead of a node enum value). affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -17,10 +22,8 @@ spec: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - control + - key: galactic.datumapis.com/galactic-route-reflector + operator: Exists initContainers: - name: frr-init image: fabric-router:latest diff --git a/deploy/containerlab/resources/fabric-control/iad/kustomization.yaml b/deploy/containerlab/resources/fabric-control/iad/kustomization.yaml index 2245b131..e36f5063 100644 --- a/deploy/containerlab/resources/fabric-control/iad/kustomization.yaml +++ b/deploy/containerlab/resources/fabric-control/iad/kustomization.yaml @@ -25,4 +25,4 @@ patches: configMapGenerator: - name: fabric-config files: - - frr.conf.iad-worker-control=frr.conf + - frr.conf.iad-worker-rr=frr.conf diff --git a/deploy/containerlab/resources/fabric-router/base/fabric-lab-patch.yaml b/deploy/containerlab/resources/fabric-router/base/fabric-lab-patch.yaml index 4df7df2a..e7c2c299 100644 --- a/deploy/containerlab/resources/fabric-router/base/fabric-lab-patch.yaml +++ b/deploy/containerlab/resources/fabric-router/base/fabric-lab-patch.yaml @@ -5,19 +5,20 @@ metadata: spec: template: spec: - # Narrower than config/fabric-router/'s default (edge, control, or gateway): - # iad's control-node fabric pod is deployed separately by - # resources/fabric-control/iad/, with its own frr.conf for that - # node's distinct peering — so this per-site overlay sticks to edge - # and gateway, or the control node would get two competing fabric - # pods. edge and gateway share this one DaemonSet because both just - # need an underlay eBGP session brought up before galactic-router - # starts (config/fabric-router/daemonset.yaml's own doc comment) — nothing - # about FRR itself differs by role, only the per-node frr.conf key - # each pod selects via NODE_NAME (../iad/kustomization.yaml's - # configMapGenerator carries both iad-worker's and iad-gateway1/2's - # keys in the same fabric-config ConfigMap). dfw/sjc have no - # gateway-labeled nodes, so this widening is a no-op there. + # Narrower than config/fabric-router/'s default (galactic.datumapis.com/fabric + # Exists, matching any role): iad's route-reflector-node fabric pod is + # deployed separately by resources/fabric-control/iad/, with its own + # frr.conf for that node's distinct peering — so this per-site overlay + # sticks to compute and edge, or the route-reflector node would get two + # competing fabric pods. compute and edge share this one DaemonSet + # because both just need an underlay eBGP session brought up before + # galactic-router starts (config/fabric-router/daemonset.yaml's own doc + # comment) — nothing about FRR itself differs by role, only the + # per-node frr.conf key each pod selects via NODE_NAME + # (../iad/kustomization.yaml's configMapGenerator carries both + # iad-worker's and iad-gateway1/2's keys in the same fabric-config + # ConfigMap). dfw/sjc have no edge-labeled nodes, so this widening is a + # no-op there. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -28,8 +29,8 @@ spec: - key: galactic.datumapis.com/node operator: In values: + - compute - edge - - gateway initContainers: - name: frr-init image: fabric-router:latest diff --git a/deploy/containerlab/resources/galactic-control/iad/bgprouter.yaml b/deploy/containerlab/resources/galactic-control/iad/bgprouter.yaml index 8653002e..369ec948 100644 --- a/deploy/containerlab/resources/galactic-control/iad/bgprouter.yaml +++ b/deploy/containerlab/resources/galactic-control/iad/bgprouter.yaml @@ -6,7 +6,7 @@ metadata: spec: targetRef: kind: Node - name: iad-worker-control + name: iad-worker-rr roles: - tenant localASN: 65000 diff --git a/deploy/containerlab/resources/galactic-control/iad/kustomization.yaml b/deploy/containerlab/resources/galactic-control/iad/kustomization.yaml index 66df9221..0a725301 100644 --- a/deploy/containerlab/resources/galactic-control/iad/kustomization.yaml +++ b/deploy/containerlab/resources/galactic-control/iad/kustomization.yaml @@ -1,6 +1,6 @@ namespace: galactic-system resources: - - tenant-control + - overlays/rr - bgprouter.yaml - bgppeer-dfw.yaml - bgppeer-iad.yaml diff --git a/deploy/containerlab/resources/galactic-gateway/iad-gateway1/node-patch.yaml b/deploy/containerlab/resources/galactic-gateway/iad-gateway1/node-patch.yaml index 1d7cf96a..c6a236f1 100644 --- a/deploy/containerlab/resources/galactic-gateway/iad-gateway1/node-patch.yaml +++ b/deploy/containerlab/resources/galactic-gateway/iad-gateway1/node-patch.yaml @@ -9,7 +9,7 @@ spec: # node: GALACTIC_GATEWAY_SRV6_ADDRESS below must be unique per # gateway node (see config/galactic-gateway/base/kustomization.yaml's doc # comment), so galactic-gateway1/galactic-gateway2 are two separate - # DaemonSets rather than one DaemonSet matching both gateway-labeled + # DaemonSets rather than one DaemonSet matching both edge-labeled # nodes with identical env. affinity: nodeAffinity: @@ -21,7 +21,7 @@ spec: - key: galactic.datumapis.com/node operator: In values: - - gateway + - edge - key: kubernetes.io/hostname operator: In values: diff --git a/deploy/containerlab/resources/galactic-gateway/iad-gateway2/node-patch.yaml b/deploy/containerlab/resources/galactic-gateway/iad-gateway2/node-patch.yaml index 8ed7f057..324d2b83 100644 --- a/deploy/containerlab/resources/galactic-gateway/iad-gateway2/node-patch.yaml +++ b/deploy/containerlab/resources/galactic-gateway/iad-gateway2/node-patch.yaml @@ -9,7 +9,7 @@ spec: # node: GALACTIC_GATEWAY_SRV6_ADDRESS below must be unique per # gateway node (see config/galactic-gateway/base/kustomization.yaml's doc # comment), so galactic-gateway1/galactic-gateway2 are two separate - # DaemonSets rather than one DaemonSet matching both gateway-labeled + # DaemonSets rather than one DaemonSet matching both edge-labeled # nodes with identical env. affinity: nodeAffinity: @@ -21,7 +21,7 @@ spec: - key: galactic.datumapis.com/node operator: In values: - - gateway + - edge - key: kubernetes.io/hostname operator: In values: diff --git a/deploy/containerlab/resources/galactic-gateway/iad/servicevipbinding-ns60.yaml b/deploy/containerlab/resources/galactic-gateway/iad/servicevipbinding-ns60.yaml index b8211b4b..26a3bb5f 100644 --- a/deploy/containerlab/resources/galactic-gateway/iad/servicevipbinding-ns60.yaml +++ b/deploy/containerlab/resources/galactic-gateway/iad/servicevipbinding-ns60.yaml @@ -3,9 +3,9 @@ # provenance). ns60's nginx Deployment (resources/tenants/ns60/) has no # node pin of its own beyond excluding control-plane nodes, and lands on # iad-worker in practice (the only untainted, non-route-reflector, -# non-gateway-role worker in the iad cluster -- iad-worker2/ -# iad-worker-control is the route reflector and iad-gateway1/iad-gateway2 -# are tainted gateway-role nodes; confirm the live pod's actual node with +# non-edge-role worker in the iad cluster -- iad-worker2/ +# iad-worker-rr is the route reflector and iad-gateway1/iad-gateway2 +# are tainted edge-role (galactic.datumapis.com/node=edge) nodes; confirm the live pod's actual node with # `kubectl get pod -n ns60 -o wide` the same way that file's comment # already documents for the backend address itself). # diff --git a/deploy/containerlab/resources/galactic-nat66/README.md b/deploy/containerlab/resources/galactic-nat66/README.md index a39dff76..0b9c589d 100644 --- a/deploy/containerlab/resources/galactic-nat66/README.md +++ b/deploy/containerlab/resources/galactic-nat66/README.md @@ -56,7 +56,7 @@ RT-less-EVPN main-table import path (`internal/runtime/gobgp/monitor.go`'s `matchTableID`/`RouteMainAdd`). `GALACTIC_CNI_NAT66_SHARD_SIDS` (set identically on every site's `galactic-cni` DaemonSet, `resources/galactic-cni/daemonset-patch.yaml`) -carries the fabric-wide membership list every tenant node needs to build +carries the fabric-wide membership list every compute node needs to build its own default route — operator-supplied in this phase, not learned in-cluster; see that env var's own doc comment (`internal/config/cni.go`) for why. diff --git a/deploy/containerlab/resources/galactic-nat66/dfw/node-patch.yaml b/deploy/containerlab/resources/galactic-nat66/dfw/node-patch.yaml index 0e27d68c..bedd0dea 100644 --- a/deploy/containerlab/resources/galactic-nat66/dfw/node-patch.yaml +++ b/deploy/containerlab/resources/galactic-nat66/dfw/node-patch.yaml @@ -6,37 +6,20 @@ spec: template: spec: # config/galactic-nat66/base/daemonset.yaml's own affinity requires - # galactic.datumapis.com/node: nat66 -- a dedicated node role this - # lab has never introduced (see AGENTS.md's "Node labels" note: - # every site worker here is labeled galactic.datumapis.com/node: - # edge). Per the redesign plan's own §8 suggestion, this lab reuses - # dfw-worker itself as a shard rather than adding a new dedicated - # nat66-labeled node, so this patch replaces the whole affinity - # block to match dfw-worker's real "edge" label plus a - # kubernetes.io/hostname pin -- otherwise the base's own affinity - # would never match any node in this cluster and the DaemonSet - # would sit at zero replicas forever. GALACTIC_NAT66_SHARD_SID/ + # galactic.datumapis.com/node: compute -- dfw-worker + # (node_files/dfw/config.yaml) carries it, reusing that existing + # compute worker as this site's shard rather than introducing a + # dedicated node, per the redesign plan's own §8 suggestion. No + # affinity override needed here: dfw-worker is the only node in + # this cluster carrying that label, so the base's own affinity + # already resolves to exactly it -- unlike + # resources/galactic-gateway/iad-gateway1/, which needs a + # kubernetes.io/hostname pin because iad has *two* edge-labeled + # nodes needing different per-node env. GALACTIC_NAT66_SHARD_SID/ # _SHARD_PUB_ADDR below must be unique per shard node (see # config/galactic-nat66/kustomization.yaml's doc comment), so # dfw/iad/sjc are three separate per-node overlays rather than one - # overlay matching all three site workers with identical env -- - # mirrors resources/galactic-gateway/iad-gateway1/'s identical - # per-node-overlay rationale. - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/control-plane - operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - edge - - key: kubernetes.io/hostname - operator: In - values: - - dfw-worker + # overlay matching all three site workers with identical env. containers: - name: galactic-nat66 env: diff --git a/deploy/containerlab/resources/galactic-nat66/iad/node-patch.yaml b/deploy/containerlab/resources/galactic-nat66/iad/node-patch.yaml index e1d9717b..86b2a7f5 100644 --- a/deploy/containerlab/resources/galactic-nat66/iad/node-patch.yaml +++ b/deploy/containerlab/resources/galactic-nat66/iad/node-patch.yaml @@ -6,40 +6,25 @@ spec: template: spec: # config/galactic-nat66/base/daemonset.yaml's own affinity requires - # galactic.datumapis.com/node: nat66 -- a dedicated node role this - # lab has never introduced (see AGENTS.md's "Node labels" note: - # every site worker here is labeled galactic.datumapis.com/node: - # edge). Per the redesign plan's own §8 suggestion, this lab reuses - # iad-worker itself as a shard rather than adding a new dedicated - # nat66-labeled node, so this patch replaces the whole affinity - # block to match iad-worker's real "edge" label plus a - # kubernetes.io/hostname pin -- otherwise the base's own affinity - # would never match any node in this cluster and the DaemonSet - # would sit at zero replicas forever. iad-worker (worker-index 1) - # is the plain tenant worker, not iad-worker2/iad-worker-control - # (the route reflector) or iad-gateway1/iad-gateway2 (the tainted - # gateway-role nodes) -- see gvpc.clab.yaml and README.md's node - # table. GALACTIC_NAT66_SHARD_SID/_SHARD_PUB_ADDR below must be - # unique per shard node (see config/galactic-nat66/kustomization. - # yaml's doc comment), so dfw/iad/sjc are three separate per-node - # overlays rather than one overlay matching all three site workers - # with identical env -- mirrors resources/galactic-gateway/ - # iad-gateway1/'s identical per-node-overlay rationale. - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/control-plane - operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - edge - - key: kubernetes.io/hostname - operator: In - values: - - iad-worker + # galactic.datumapis.com/node: compute -- iad-worker + # (node_files/iad/config.yaml, worker-index 1) carries it, reusing + # that existing compute worker as this site's shard rather than + # introducing a dedicated node, per the redesign plan's own §8 + # suggestion. iad-worker2/iad-worker-rr (the route reflector, + # galactic.datumapis.com/galactic-route-reflector, not a + # galactic.datumapis.com/node value at all) and iad-gateway1/ + # iad-gateway2 (the tainted galactic.datumapis.com/node=edge nodes, + # see gvpc.clab.yaml and README.md's node table) never carry the + # compute label, so no affinity override or hostname pin is needed + # here: iad-worker is the only node in this cluster carrying it, + # and the base's own affinity already resolves to exactly it -- + # unlike resources/galactic-gateway/iad-gateway1/, which needs a + # kubernetes.io/hostname pin because iad has *two* edge-labeled + # nodes needing different per-node env. GALACTIC_NAT66_SHARD_SID/ + # _SHARD_PUB_ADDR below must be unique per shard node (see + # config/galactic-nat66/kustomization.yaml's doc comment), so + # dfw/iad/sjc are three separate per-node overlays rather than one + # overlay matching all three site workers with identical env. containers: - name: galactic-nat66 env: @@ -67,7 +52,7 @@ spec: # them. Physical co-location (which node this pod is # scheduled to) and uSID identity (which Node-ID a shard's own # address claims) are independent; nodeID=9 is free at this - # site (iad-worker=1, iad-worker-control=1, iad-gateway1=2, + # site (iad-worker=1, iad-worker-rr=1, iad-gateway1=2, # iad-gateway2=3 -- see resources/galactic-router/iad/, # resources/galactic-control/iad/, and resources/ # galactic-gateway/iad-gateway{1,2}/'s own bgprouter.yaml diff --git a/deploy/containerlab/resources/galactic-nat66/sjc/node-patch.yaml b/deploy/containerlab/resources/galactic-nat66/sjc/node-patch.yaml index 508c620d..3735aace 100644 --- a/deploy/containerlab/resources/galactic-nat66/sjc/node-patch.yaml +++ b/deploy/containerlab/resources/galactic-nat66/sjc/node-patch.yaml @@ -6,37 +6,20 @@ spec: template: spec: # config/galactic-nat66/base/daemonset.yaml's own affinity requires - # galactic.datumapis.com/node: nat66 -- a dedicated node role this - # lab has never introduced (see AGENTS.md's "Node labels" note: - # every site worker here is labeled galactic.datumapis.com/node: - # edge). Per the redesign plan's own §8 suggestion, this lab reuses - # sjc-worker itself as a shard rather than adding a new dedicated - # nat66-labeled node, so this patch replaces the whole affinity - # block to match sjc-worker's real "edge" label plus a - # kubernetes.io/hostname pin -- otherwise the base's own affinity - # would never match any node in this cluster and the DaemonSet - # would sit at zero replicas forever. GALACTIC_NAT66_SHARD_SID/ + # galactic.datumapis.com/node: compute -- sjc-worker + # (node_files/sjc/config.yaml) carries it, reusing that existing + # compute worker as this site's shard rather than introducing a + # dedicated node, per the redesign plan's own §8 suggestion. No + # affinity override needed here: sjc-worker is the only node in + # this cluster carrying that label, so the base's own affinity + # already resolves to exactly it -- unlike + # resources/galactic-gateway/iad-gateway1/, which needs a + # kubernetes.io/hostname pin because iad has *two* edge-labeled + # nodes needing different per-node env. GALACTIC_NAT66_SHARD_SID/ # _SHARD_PUB_ADDR below must be unique per shard node (see # config/galactic-nat66/kustomization.yaml's doc comment), so # dfw/iad/sjc are three separate per-node overlays rather than one - # overlay matching all three site workers with identical env -- - # mirrors resources/galactic-gateway/iad-gateway1/'s identical - # per-node-overlay rationale. - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/control-plane - operator: DoesNotExist - - key: galactic.datumapis.com/node - operator: In - values: - - edge - - key: kubernetes.io/hostname - operator: In - values: - - sjc-worker + # overlay matching all three site workers with identical env. containers: - name: galactic-nat66 env: diff --git a/deploy/containerlab/resources/galactic-router/base/kustomization.yaml b/deploy/containerlab/resources/galactic-router/base/kustomization.yaml index 6e2689dc..999fd752 100644 --- a/deploy/containerlab/resources/galactic-router/base/kustomization.yaml +++ b/deploy/containerlab/resources/galactic-router/base/kustomization.yaml @@ -1,5 +1,5 @@ resources: - - tenant + - overlays/default patches: - path: router-lab-patch.yaml target: diff --git a/deploy/containerlab/scripts/deploy-fabric.sh b/deploy/containerlab/scripts/deploy-fabric.sh index dd51c7b4..e0b42dd2 100755 --- a/deploy/containerlab/scripts/deploy-fabric.sh +++ b/deploy/containerlab/scripts/deploy-fabric.sh @@ -1,19 +1,21 @@ #!/bin/bash # deploy-fabric.sh — Install the FRR fabric DaemonSets on every cluster. -# iad additionally receives the control-node fabric overlay. +# iad additionally receives the route-reflector-node fabric overlay. set -euo pipefail SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) source "${SCRIPT_DIR}/lib.sh" # config/fabric-router/ (shared with production) is a single DaemonSet whose -# affinity allows the edge, control, and gateway node labels. +# affinity allows any node carrying galactic.datumapis.com/fabric, +# regardless of its galactic.datumapis.com/node value or +# galactic-route-reflector flag. # resources/fabric-router/base/ and resources/fabric-control/iad/ each # build on a copy of it and patch in the lab-only image/imagePullPolicy -# plus a narrower affinity (edge+gateway / control-only respectively) — -# iad needs the control node split back out because it needs its own -# frr.conf and would otherwise get two competing fabric pods; edge and -# gateway share one DaemonSet since neither needs anything but a +# plus a narrower affinity (compute+edge / route-reflector-only respectively) — +# iad needs the route-reflector node split back out because it needs its own +# frr.conf and would otherwise get two competing fabric pods; compute and +# edge share one DaemonSet since neither needs anything but a # per-node frr.conf key (fabric-router/iad/kustomization.yaml's # configMapGenerator carries both). Copied onto the node at deploy time # nested under each consuming overlay's own root so its "fabric" resource @@ -23,23 +25,35 @@ FABRIC_DIR=$(cd "${SCRIPT_DIR}/../../../config/fabric-router" && pwd) # copy_fabric_config NODE copies config/fabric-router/ onto NODE, nested under # resources/fabric-router/base/ so the base overlay's "fabric" resource -# reference resolves. +# reference resolves. rm -rf first: like deploy-galactic-router.sh's +# copy_router_config, docker cp nests SRC inside an already-existing DEST +# dir instead of overwriting it, so a rerun against an already-provisioned +# node would silently keep serving the prior copy from underneath the new +# one -- kubectl would then report the DaemonSet "unchanged" even after a +# real manifest edit (found live: a fabric-lab-patch.yaml affinity fix +# never took effect on a redeploy until this guard was added). copy_fabric_config() { local node="$1" + docker exec "${node}" rm -rf /galactic/resources/fabric-router/base/fabric docker cp "${FABRIC_DIR}" "${node}:/galactic/resources/fabric-router/base/fabric" } # copy_fabric_control_config NODE copies config/fabric-router/ onto NODE, nested # under resources/fabric-control/iad/ so that overlay's "fabric" -# resource reference resolves. +# resource reference resolves. rm -rf first -- see copy_fabric_config's +# comment. copy_fabric_control_config() { local node="$1" + docker exec "${node}" rm -rf /galactic/resources/fabric-control/iad/fabric docker cp "${FABRIC_DIR}" "${node}:/galactic/resources/fabric-control/iad/fabric" } -# dfw and sjc only need the fabric overlay. +# dfw and sjc only need the fabric overlay. rm -rf first -- see +# copy_fabric_config's comment; copy_to (lib.sh) doesn't overwrite an +# already-provisioned node's copy on its own either. for site in dfw sjc; do node=$(control_plane "${site}") + docker exec "${node}" rm -rf /galactic/resources/fabric-router copy_to "${node}" fabric-router copy_fabric_config "${node}" apply_k "${node}" "/galactic/resources/fabric-router/${site}/" @@ -51,6 +65,7 @@ done # frr.conf. keys; ../base/'s affinity matches both roles). node=$(control_plane iad) echo "Copying resources to ${node}..." +docker exec "${node}" rm -rf /galactic/resources/fabric-router /galactic/resources/fabric-control copy_to "${node}" fabric-router copy_fabric_config "${node}" copy_to "${node}" fabric-control diff --git a/deploy/containerlab/scripts/deploy-galactic-router.sh b/deploy/containerlab/scripts/deploy-galactic-router.sh index 44531c78..f8a173cb 100755 --- a/deploy/containerlab/scripts/deploy-galactic-router.sh +++ b/deploy/containerlab/scripts/deploy-galactic-router.sh @@ -6,51 +6,60 @@ set -euo pipefail SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) source "${SCRIPT_DIR}/lib.sh" -# The router DaemonSet base and the production tenant/tenant-control -# overlays (node affinity keeping each role on the right nodes) live in -# config/galactic-router/{base,tenant,tenant-control}/ (shared with production; -# the shared RBAC/ServiceAccount aren't needed here). +# The router DaemonSet base (config/galactic-router/base/ -- role-agnostic, +# not applied directly, shared with production) and the production +# default/rr overlays (config/galactic-router/overlays/{default,rr}/, each +# independently patching ../../base into its own role) live under +# config/galactic-router/ (the shared RBAC/ServiceAccount aren't needed +# here). # resources/galactic-router/base/ and resources/galactic-control/iad/ -# build on the copied tenant/tenant-control overlays and patch in only the +# build on the copied base/default or base/rr dirs and patch in only the # lab-only image and env vars. Dirs are copied onto the node at deploy time # nested under each consuming overlay's own root (kustomize requires resources # in or below the overlay root) rather than duplicated in the repo. GALACTIC_ROUTER_BASE_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-router/base" && pwd) -GALACTIC_ROUTER_TENANT_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-router/tenant" && pwd) -GALACTIC_ROUTER_TENANT_CONTROL_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-router/tenant-control" && pwd) +GALACTIC_ROUTER_DEFAULT_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-router/overlays/default" && pwd) +GALACTIC_ROUTER_RR_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-router/overlays/rr" && pwd) # config/galactic-gateway/base is the edge XDP NAT+LB gateway's own two-container # pod base (galactic-router + galactic-gateway) -- self-contained, unlike -# config/galactic-router/{tenant,tenant-control}, so no matching +# config/galactic-router/{base,overlays/default,overlays/rr}, so no matching # config/galactic-router/base copy is needed alongside it the way copy_router_config/ # copy_router_control_config need one. GALACTIC_GATEWAY_BASE_DIR=$(cd "${SCRIPT_DIR}/../../../config/galactic-gateway/base" && pwd) -# copy_router_config NODE copies config/galactic-router/{base,tenant} onto NODE, -# nested under resources/galactic-router/base/ so the overlay's "../base" -# resource reference resolves. rm -rf first: like deploy-cni.sh's -# GALACTIC_CNI_DIR copy, docker cp nests SRC inside an already-existing -# DEST dir instead of overwriting it, so a rerun against an -# already-provisioned node would silently keep serving the prior copy -# from underneath the new one -- kubectl would then report the DaemonSet -# "unchanged" even after a real manifest edit. +# copy_router_config NODE copies config/galactic-router/base and +# config/galactic-router/overlays/default onto NODE, nested under +# resources/galactic-router/base/ at the *same relative depth* as in +# config/ (base/base/, base/overlays/default/) -- required because +# overlays/default/kustomization.yaml's own "../../base" reference is copied +# verbatim, unmodified, so the local copy has to resolve at the same two +# levels up or that reference points outside the tree entirely. rm -rf +# first: like deploy-cni.sh's GALACTIC_CNI_DIR copy, docker cp nests SRC +# inside an already-existing DEST dir instead of overwriting it, so a +# rerun against an already-provisioned node would silently keep serving +# the prior copy from underneath the new one -- kubectl would then report +# the DaemonSet "unchanged" even after a real manifest edit. copy_router_config() { local node="$1" - docker exec "${node}" rm -rf /galactic/resources/galactic-router/base/base /galactic/resources/galactic-router/base/tenant + docker exec "${node}" rm -rf /galactic/resources/galactic-router/base/base /galactic/resources/galactic-router/base/overlays + docker exec "${node}" mkdir -p /galactic/resources/galactic-router/base/overlays docker cp "${GALACTIC_ROUTER_BASE_DIR}" "${node}:/galactic/resources/galactic-router/base/base" - docker cp "${GALACTIC_ROUTER_TENANT_DIR}" "${node}:/galactic/resources/galactic-router/base/tenant" + docker cp "${GALACTIC_ROUTER_DEFAULT_DIR}" "${node}:/galactic/resources/galactic-router/base/overlays/default" } -# copy_router_control_config NODE copies config/galactic-router/{base,tenant-control} -# onto NODE, nested under resources/galactic-control/iad/ so the -# tenant-control overlay's "../base" resource reference resolves. Its node -# affinity (route-reflector role, control node only) applies as-is; the -# lab only needs to patch in the image and BGP address/port. rm -rf first -# -- see copy_router_config's comment. +# copy_router_control_config NODE copies config/galactic-router/base and +# config/galactic-router/overlays/rr onto NODE, nested under +# resources/galactic-control/iad/ at the same relative depth as in +# config/, for the same reason copy_router_config's comment explains. +# Its node affinity (route-reflector role, control node only) applies +# as-is; the lab only needs to patch in the image and BGP address/port. +# rm -rf first -- see copy_router_config's comment. copy_router_control_config() { local node="$1" - docker exec "${node}" rm -rf /galactic/resources/galactic-control/iad/base /galactic/resources/galactic-control/iad/tenant-control + docker exec "${node}" rm -rf /galactic/resources/galactic-control/iad/base /galactic/resources/galactic-control/iad/overlays + docker exec "${node}" mkdir -p /galactic/resources/galactic-control/iad/overlays docker cp "${GALACTIC_ROUTER_BASE_DIR}" "${node}:/galactic/resources/galactic-control/iad/base" - docker cp "${GALACTIC_ROUTER_TENANT_CONTROL_DIR}" "${node}:/galactic/resources/galactic-control/iad/tenant-control" + docker cp "${GALACTIC_ROUTER_RR_DIR}" "${node}:/galactic/resources/galactic-control/iad/overlays/rr" } # copy_router_gateway_config NODE copies config/galactic-gateway/base onto NODE, diff --git a/deploy/containerlab/scripts/verify-ns40.sh b/deploy/containerlab/scripts/verify-ns40.sh index e97a93a2..49d85899 100755 --- a/deploy/containerlab/scripts/verify-ns40.sh +++ b/deploy/containerlab/scripts/verify-ns40.sh @@ -29,7 +29,7 @@ done # one worker, so both pods landing on the same node is the subject of the test, # not an incidental detail. Nothing in the fixture pins them together — iad has # a second worker, and only the route-reflector NoSchedule taint on -# iad-worker-control keeps the pair on iad-worker — so if that taint goes away +# iad-worker-rr keeps the pair on iad-worker — so if that taint goes away # the pings below would still pass over the cross-node path and quietly stop # testing what ns40 is named for. Assert it instead. SCHED0=$(pod_scheduling_node "${NODE}" "${NS}" "${PODS[0]}") diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index be7ac536..db9209b3 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -36,7 +36,7 @@ load-balances into a tenant VPC's backend Pods, without a VRF or tunnel dependency and without a per-tenant Geneve device. It is a **separate binary from `galactic-router`**, deployed as a second container in the same `hostNetwork: true` pod on dedicated gateway-role nodes only -(`galactic.datumapis.com/node: gateway`), specifically so a crash on either +(`galactic.datumapis.com/node: edge`), specifically so a crash on either side — tenant BGP vs. the XDP-holding gateway engine — no longer takes the other down with it. Tenant BGP itself (the embedded GoBGP server, the `BGPRouter`/`BGPPeer`/`BGPAdvertisement`/`BGPPolicy`/`BGPVRFInstance` @@ -424,7 +424,7 @@ binary's own, see [RBAC](#rbac) below). | Container | Capabilities | Why | | ------------------ | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `galactic-router` | `NET_ADMIN` | Same as the plain `tenant`/`tenant-control` roles — no BPF/PERFMON, since gateway-specific eBPF is confined to the other container now | +| `galactic-router` | `NET_ADMIN` | Same as the plain `default`/`rr` roles — no BPF/PERFMON, since gateway-specific eBPF is confined to the other container now | | `galactic-gateway` | `NET_ADMIN`, `BPF`, `PERFMON` | `BPF` for the `bpf()` syscalls (program/map creation); `PERFMON` because the verifier only allows pointer+scalar arithmetic on packet data/data_end when the loading process is `perfmon_capable()` — without it, even a `root` container gets "pointer arithmetic ... prohibited for !root" | Both containers mount host paths: `galactic-router` mounts diff --git a/docs/agents/ARCHITECTURE-ROUTER.md b/docs/agents/ARCHITECTURE-ROUTER.md index d877dd70..1fe44ac1 100644 --- a/docs/agents/ARCHITECTURE-ROUTER.md +++ b/docs/agents/ARCHITECTURE-ROUTER.md @@ -89,11 +89,16 @@ galactic/ │ └── vrf/ ├── config/ │ └── router/ # Shared RBAC/ServiceAccount, plus: -│ ├── base/ # common DaemonSet spec -│ ├── tenant/ # per-node role: base + node affinity excluding control-plane -│ │ # and tenant-control nodes -│ └── tenant-control/ # route-reflector role: base + GALACTIC_ROUTER_REFLECTOR=true, -│ # opt-in via the galactic.datumapis.com/node=control node label +│ ├── base/ # role-agnostic DaemonSet spec; not applied directly +│ └── overlays/ +│ ├── default/ # the default per-node role: patches ../../base with node +│ │ # affinity excluding control-plane nodes, opt-in via +│ │ # galactic.datumapis.com/node=compute -- no +│ │ # role-specific name/label of its own +│ └── rr/ # route-reflector role: independently patches ../../base with +│ # GALACTIC_ROUTER_REFLECTOR=true, nameSuffix -rr, opt-in via +│ # the galactic.datumapis.com/galactic-route-reflector label +│ # (not a galactic.datumapis.com/node value) └── containers/ └── galactic-router/ # galactic-router production image ``` @@ -152,9 +157,9 @@ lives in `root.go`'s `runCmd`: that waits for cache sync, then runs on `--gc-interval`, default 5m). 8. `mgr.Start(ctx)` — blocks until the signal-handler context is cancelled. -This is identical whether the pod is running in the plain `tenant` role -(`config/galactic-router/tenant/`), the route-reflector role -(`config/galactic-router/tenant-control/`), or as the tenant-BGP container co-located +This is identical whether the pod is running in the plain default role +(`config/galactic-router/overlays/default/`), the route-reflector role +(`config/galactic-router/overlays/rr/`), or as the tenant-BGP container co-located with `galactic-gateway` on a gateway-role node (`config/galactic-gateway/base/daemonset.yaml`) — `galactic-router` carries no gateway-role code path of its own at all; see @@ -182,7 +187,7 @@ See [docs/router/configuration.md](../router/configuration.md) for the full refe On a gateway-role node, `GALACTIC_ROUTER_BGP_LISTEN_PORT=-1` is set by `config/galactic-gateway/base/daemonset.yaml` — the tenant-BGP container only dials -out to iBGP peers there, same as the plain tenant role. +out to iBGP peers there, same as the plain default role. `GALACTIC_ROUTER_GRPC_HEALTH_PORT=5179` is set explicitly too, matching the binary's own default, so it can't silently start colliding with the co-located `galactic-gateway` container's own health port on the same diff --git a/docs/cni/gc-cmd-sequence.md b/docs/cni/gc-cmd-sequence.md index a1f56a60..78aa1497 100644 --- a/docs/cni/gc-cmd-sequence.md +++ b/docs/cni/gc-cmd-sequence.md @@ -196,7 +196,7 @@ sequenceDiagram [docs/cni/cni-cmd-sequence.md](cni-cmd-sequence.md#notes-on-del). - **Node scoping matters because CRDs are namespace-, not node-, scoped.** `BGPAdvertisement`/`BGPVRFInstance` objects in a namespace can belong to - routers on other nodes (e.g. the tenant and tenant-control roles watching + routers on other nodes (e.g. the default and rr roles watching the same namespace). Every collection pass first filters to only the `BGPRouter`s whose `TargetRef.Name` matches this node, and this node's own kernel/filesystem state can only ever confirm or deny liveness for diff --git a/docs/node-labels.md b/docs/node-labels.md new file mode 100644 index 00000000..beb20980 --- /dev/null +++ b/docs/node-labels.md @@ -0,0 +1,175 @@ +# Node Labeling Strategy + +> How Galactic decides which DaemonSets run on which nodes. + +_Last updated: 2026-08-21_ + +This document is cross-cutting — it covers the node-selection contract shared +by `galactic-cni`, `galactic-router`, `galactic-gateway`, `galactic-nat66`, +and `fabric-router`, none of which owns it individually. See +[AGENTS.md](../AGENTS.md) for which per-component architecture doc to read +for everything else about a given binary. + +--- + +## The five labels + +| Label | Deploys | Kind | +|--------------------------------------------------------|--------------------------------------------------------------------|------------------------------------------------------------| +| `galactic.datumapis.com/fabric=true` | `fabric-router` | independent boolean flag | +| `galactic.datumapis.com/node=compute` | `galactic-router` (default role), `galactic-nat66`, `galactic-cni` | primary role (enum value) | +| `galactic.datumapis.com/node=edge` | `galactic-gateway` | primary role (enum value) | +| `galactic.datumapis.com/galactic-route-reflector=true` | `galactic-router-rr` | independent boolean flag | +| `galactic.datumapis.com/fabric-route-reflector=true` | `fabric-router-rr` | independent boolean flag — **future**, not yet implemented | + +Every affinity also excludes Kubernetes control-plane nodes +(`node-role.kubernetes.io/control-plane: DoesNotExist`), independently of +which of the above a node carries. + +--- + +## Two different kinds of label, on purpose + +**`galactic.datumapis.com/node` is a primary-role enum.** A node has exactly +one value for it — `compute` or `edge` — because a Kubernetes label key is +single-valued, and these two are genuinely mutually exclusive by design: +`edge` nodes are tainted specifically to keep tenant workloads and the +`compute` role off them (see +`deploy/containerlab/node_files/iad/config.yaml`'s gateway-node taints), so a +node is never both at once. + +**Everything else is an independent boolean-style flag**, not a fifth/sixth +enum value, because the thing it gates is *not* mutually exclusive with a +node's primary role: + +- `fabric` applies to every role that needs underlay BGP connectivity — + which today is all of them. +- `galactic-route-reflector` can, in principle, coexist with `node=compute` + on the same node (a route reflector doesn't have to be a dedicated node); + encoding it as `node=route-reflector` would have made that impossible, + since the node would have had to give up being `compute` to become a + reflector. +- `fabric-route-reflector` (future) follows the same reasoning for the + underlay's own route reflector, independently of the EVPN one. + +**The rule of thumb**: if two roles can never legitimately coexist on one +node, encode the difference as `galactic.datumapis.com/node`'s value. If +they can — or if you're not sure yet — give the capability its own +independent label instead. Getting this wrong is a real, previously-shipped +bug, not a hypothetical: + +- `fabric-router`'s affinity used to enumerate roles via `In: [edge, + route-reflector, gateway]`. `nat66` was never added to that list, so a + NAT66 shard node silently never got the underlay BGP session its own SID + advertisement depends on. Fixed by making `fabric` its own label that any + role opts into, rather than a list `fabric-router` has to keep in sync + with every other component's roles. +- `galactic-nat66` used to require `galactic.datumapis.com/node: nat66` as a + dedicated enum value. Since a node can only have one `node` value, that + made it impossible for a node to be both `edge` (i.e. `compute`, in + today's naming — see below) and a NAT66 shard at once, which is the only + configuration that's ever actually used. Fixed by dropping `nat66` from + the enum and folding shard duty into `node=compute` directly — every + compute node now runs `galactic-nat66` unconditionally. +- `galactic-router-rr` used to require `galactic.datumapis.com/node: + route-reflector`. Fixed the same way as `nat66`, for the same underlying + reason: a route reflector shouldn't have to stop being anything else it + might also need to be. + +--- + +## Naming collision, read this before anything else + +**`galactic.datumapis.com/node=edge` deploys `galactic-gateway` — it does +*not* mean "the per-node tenant/compute role."** That's `node=compute`. + +This is a deliberate rename, not an inconsistency to double-check: "edge" +now means the actual network edge — the ingress/egress boundary +`galactic-gateway`'s XDP NAT+LB datapath sits on — which is also why +`galactic-gateway` has always been described elsewhere in this repo as "the +edge XDP NAT+LB gateway," independently of and predating this label scheme. +Before this rename, the *tenant-serving* role was called `edge` and the +*gateway* role was called `gateway`, which collided with that existing +"edge XDP" terminology. Calling the tenant role `compute` and reserving +`edge` for the actual network-edge role resolves that collision instead of +perpetuating it. + +If you're reading an older comment, commit, or diagram that says +`galactic.datumapis.com/node: edge` and shows `galactic-router`/ +`galactic-cni` attached to it, it predates this rename and means what +`compute` means today. + +--- + +## Per-role reference + +### `galactic.datumapis.com/node=compute` + +Runs `galactic-router` (default role), `galactic-nat66`, and `galactic-cni` +together, unconditionally — this is the ordinary tenant-serving node. + +- `config/galactic-router/overlays/default/daemonset-patch.yaml` +- `config/galactic-nat66/base/daemonset.yaml` +- `config/galactic-cni/daemonset.yaml` + +### `galactic.datumapis.com/node=edge` + +Runs `galactic-gateway`'s two-container pod (`galactic-gateway` + +`galactic-router`, the latter carrying no gateway-specific code of its own — +see [ARCHITECTURE-GATEWAY.md](agents/ARCHITECTURE-GATEWAY.md)). Dedicated, +opt-in, tainted to keep ordinary tenant pods off — see +`config/galactic-gateway/base/daemonset.yaml`. + +### `galactic.datumapis.com/fabric=true` + +Runs `fabric-router`, the FRR underlay eBGP DaemonSet, independently of +whatever `node` value or route-reflector flag a node also carries. See +`config/fabric-router/daemonset.yaml`'s own affinity comment for the full +history of why this is a dedicated label rather than an enumerated list. + +### `galactic.datumapis.com/galactic-route-reflector=true` + +Runs `galactic-router-rr` (`GALACTIC_ROUTER_REFLECTOR=true`), the EVPN route +reflector every compute node's `galactic-router` peers into over iBGP. See +`config/galactic-router/overlays/rr/daemonset-patch.yaml`. + +### `galactic.datumapis.com/fabric-route-reflector=true` (future) + +Reserved for `fabric-router-rr` — the underlay's own iBGP route reflector, +distinct from the EVPN one above (see the two separate listener ports, 1179 +vs. 2179, in the production POP architecture this mirrors). Not implemented: +`fabric-router` today has no route-reflector variant, and `config/` has no +`fabric-router-rr` overlay. + +--- + +## Worked example: the containerlab lab + +| Node | `node` | `galactic-route-reflector` | `fabric` | Runs | +|------------------------------------------|-----------|----------------------------|----------|--------------------------------------------------------------------------------| +| `dfw-worker`, `sjc-worker`, `iad-worker` | `compute` | — | `true` | `galactic-router` (default), `galactic-nat66`, `galactic-cni`, `fabric-router` | +| `iad-worker-rr` | — | `true` | `true` | `galactic-router-rr`, `fabric-router` | +| `iad-gateway1`, `iad-gateway2` | `edge` | — | `true` | `galactic-gateway` + `galactic-router`, `fabric-router` | + +Every worker in the lab carries `fabric=true` today (see +`deploy/containerlab/node_files/{dfw,iad,sjc}/config.yaml`), since every +role in this topology needs the underlay. That's a property of this +particular lab's topology, not a rule the label scheme enforces — a real +deployment is free to have nodes with no galactic role at all (GPU, +monitoring, etc.), which correctly get none of these labels and none of +these DaemonSets. + +--- + +## Adding a new role + +1. Decide whether it's mutually exclusive with `compute`/`edge` (→ a new + `galactic.datumapis.com/node` value) or can coexist with an existing + role (→ its own independent label, `galactic.datumapis.com/=true`). + Default to the independent label unless you're certain the exclusivity + is real and permanent — see the bugs list above for what happens when + that assumption turns out to be wrong later. +2. If it needs the underlay, it still needs `galactic.datumapis.com/fabric` + set explicitly — nothing infers this from the new label automatically. +3. Update the reference table at the top of this document and in + [AGENTS.md](../AGENTS.md)'s "Node label strategy" section. diff --git a/internal/config/nat66.go b/internal/config/nat66.go index 3e50d896..b95441be 100644 --- a/internal/config/nat66.go +++ b/internal/config/nat66.go @@ -23,12 +23,13 @@ const ( // (5180/9180), and galactic-gateway's (5181/8081 -- see // internal/config/gateway.go's own doc comment for why that pair // isn't a clean continuation of the 517x/917x pattern). galactic-nat66 - // is deployed on its own dedicated, hostNetwork: true shard nodes - // (config/galactic-nat66/base/daemonset.yaml) -- disjoint from - // galactic-gateway's own gateway-role nodes -- but a node's role - // labels are not mutually exclusive by construction, so these still - // avoid every value already claimed above rather than assuming no - // overlap will ever happen. + // runs hostNetwork: true on every galactic.datumapis.com/node=compute + // node (config/galactic-nat66/base/daemonset.yaml) -- the same nodes + // as galactic-router's default role and galactic-cni, disjoint from + // galactic-gateway's own edge-role nodes -- but a node's role labels + // are not mutually exclusive by construction, so these still avoid + // every value already claimed above rather than assuming no overlap + // will ever happen. DefaultNAT66MetricsPort = 9182 DefaultNAT66GRPCHealthPort = 5182 ) diff --git a/internal/gc/gc.go b/internal/gc/gc.go index e72d6afc..7fc3f920 100644 --- a/internal/gc/gc.go +++ b/internal/gc/gc.go @@ -79,7 +79,7 @@ var legacyVRFNameRegex = regexp.MustCompile(`^G([A-Za-z0-9]{9})[A-Za-z0-9]{3}V$` // routerNamesForNode returns the names of every BGPRouter in the namespace // whose TargetRef points at nodeName. BGPAdvertisement/BGPVRFInstance CRDs // are namespace-scoped, not node-scoped — a namespace can hold CRDs created -// by routers on other nodes (e.g. the tenant and tenant-control roles both +// by routers on other nodes (e.g. the default and rr roles both // watch the same namespace in the containerlab lab), and this node's local // kernel/filesystem state (VRFs, /var/run/netns) can only ever confirm or // deny liveness for containers that actually ran here. Callers must use this diff --git a/internal/plumbing/ebpf/nat66prog/nat66_test.go b/internal/plumbing/ebpf/nat66prog/nat66_test.go index 1e86c7dd..2cab14e3 100644 --- a/internal/plumbing/ebpf/nat66prog/nat66_test.go +++ b/internal/plumbing/ebpf/nat66prog/nat66_test.go @@ -296,7 +296,7 @@ func TestNat66Ingress_ForwardSNATsAndPreservesChecksum(t *testing.T) { // regardless of what the low 64 bits (Function+Argument) happen to // contain. This is the actual boundary a real deployment must keep // disjoint -- see locator_matches' own doc comment for the live -// incident (a shard reusing its co-located tenant node's own Node-ID) +// incident (a shard reusing its co-located compute node's own Node-ID) // this same 64-bit match cannot, by itself, detect or prevent; this test // only proves the match's own stated granularity is what's implemented, // not a fix for that allocation-level constraint. diff --git a/internal/plumbing/srv6/egress.go b/internal/plumbing/srv6/egress.go index 7c74622e..6296f427 100644 --- a/internal/plumbing/srv6/egress.go +++ b/internal/plumbing/srv6/egress.go @@ -202,7 +202,7 @@ func RouteMainDel(prefix *net.IPNet, tableID uint32) error { // // EgressDefaultRouteAdd still needs to skip a shard SID it cannot yet // resolve a route+neighbor for, and only fail if every one does: a real, -// necessary case found live -- a tenant node that is *also* one of the +// necessary case found live -- a compute node that is *also* one of the // configured shards itself (this lab's own "reuse the site workers as // shards" layout) can never resolve a route to its *own* advertised SID, // since GoBGP, like every BGP implementation, never reflects a