Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ VPC and VPCAttachment CRD management lives in a separate companion operator; Gal
- **Go 1.26** — every binary
- **controller-runtime** — BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance reconcilers (`galactic-router`) and NetworkGateway/NetworkRule reconcilers (`galactic-gateway`); a bare `pkg/client` (no manager) in `galactic-bgp`/`galactic-cni`
- **BGP API** (`go.datum.net/network`) — BGPRouter, BGPPeer, BGPAdvertisement, BGPPolicy, BGPVRFInstance, NetworkGateway, NetworkRule CRDs
- **GoBGP v4** — embedded BGP server (tenant role)
- **GoBGP v4** — embedded BGP server (default role)
- **SRv6 + netlink** — kernel-level routing; `github.com/vishvananda/netlink`
- **eBPF** (`github.com/cilium/ebpf`) — TC-BPF SRv6 uSID decap datapath (CNI side) and XDP edge NAT+LB datapath (`galactic-gateway`)
- **Multus CNI** — multi-network for pods; NAD generation handled by the external operator
Expand Down Expand Up @@ -68,16 +68,28 @@ Summary:

`config/` is Kustomize-composed: `kubectl apply -k config/` deploys everything (namespace, both DaemonSets' RBAC/ServiceAccounts, and all three DaemonSets) in one command — `kubectl` sorts by kind before applying, so the namespace and RBAC/ServiceAccounts always land before anything namespace-scoped needs them. Each component also has its own `kustomization.yaml` and can be applied independently:

**Node label strategy.** Every DaemonSet's node affinity keys off one or more of these labels — deliberately independent labels rather than one enum, since Kubernetes labels are single-valued per key and some of these must be able to coexist on the same node (e.g. a route reflector that's also a compute node):

| Label | Deploys |
| ----- | ------- |
| `galactic.datumapis.com/fabric=true` | `fabric-router` |
| `galactic.datumapis.com/node=compute` | `galactic-router` (default role), `galactic-nat66`, `galactic-cni` |
| `galactic.datumapis.com/node=edge` | `galactic-gateway` (the actual network-edge/ingress boundary — not to be confused with `compute`) |
| `galactic.datumapis.com/galactic-route-reflector=true` | `galactic-router-rr` |
| `galactic.datumapis.com/fabric-route-reflector=true` | `fabric-router-rr` (**future** — not yet implemented; `fabric-router` has no route-reflector variant today) |

See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why these are independent labels rather than one enum, the naming collision between `node=edge` and `galactic-gateway`'s pre-existing "edge XDP" terminology, and the specific bugs this scheme replaced.

- **`config/galactic-system/`** — Creates the `galactic-system` namespace both components deploy into. Apply with `kubectl apply -k config/galactic-system/`.
- **`config/galactic-cni/`** — Production manifests for the CNI installer DaemonSet, ConfigMap, RBAC, and ServiceAccount. Apply with `kubectl apply -k config/galactic-cni/`.
- **`config/galactic-router/`** — Shared RBAC/ServiceAccount plus DaemonSet roles:
- **`config/galactic-router/tenant/`** — the per-node role (`galactic-router`); runs on every node except Kubernetes control-plane nodes and nodes labeled for the route-reflector or gateway roles.
- **`config/galactic-router/tenant-control/`** — the BGP route-reflector role (`galactic-router-control`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires nodes labeled `galactic.datumapis.com/node: control` (stays at zero replicas otherwise). `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it.
- **`config/galactic-router/base/`** — the DaemonSet spec shared by both roles above; not applied directly.
- Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/tenant/`. `galactic-router` no longer has a gateway role of its own — see `config/galactic-gateway/` below.
- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a separate `galactic-gateway` binary rather than a `galactic-router` role, so a crash on either side no longer takes the other down with it. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (the two-container `galactic-router` + `galactic-gateway` pod, requiring nodes labeled `galactic.datumapis.com/node: gateway`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/controller/networkgateway_controller.go`'s `publishSelfAddress` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` for a worked two-node example. Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption.
- **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `tenant`/`tenant-control`-style role split — its affinity matches nodes labeled `galactic.datumapis.com/node` `In` `[edge, control, gateway]` directly, since (unlike galactic-router's route-reflector role and the gateway role above) there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so `frr-init` gets the pod's node name via a `NODE_NAME` downward-API env var and selects a per-node `frr.conf.<nodename>` key from the ConfigMap, rather than assuming one shared `frr.conf` for the whole DaemonSet. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author a `fabric-config` ConfigMap with one `frr.conf.<nodename>` key per matching node (`daemons`/`vtysh.conf` are baked into the `fabric-router` image and only need to be in the ConfigMap if overriding those defaults) before applying `kubectl apply -k config/fabric-router/`.
- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an IPv6 SRv6 transit mesh. FRR runs as a hostNetwork DaemonSet on each worker for eBGP underlay; `galactic-router` (tenant role) handles EVPN path distribution over iBGP, the iad route reflector builds on `config/galactic-router/tenant-control/`, and iad additionally has two dedicated gateway-role nodes (`iad-gateway1`/`iad-gateway2`, `config/galactic-gateway/base/`) as a canary for the edge XDP NAT+LB gateway. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
- **`config/galactic-router/base/`** — the role-agnostic DaemonSet spec both roles below patch; not applied directly (no affinity, no BGP listen port of its own).
- **`config/galactic-router/overlays/default/`** — the default per-node role (`galactic-router`); runs on every node except Kubernetes control-plane nodes, opt-in via `galactic.datumapis.com/node: compute` (the same label `galactic-cni` and `galactic-nat66` key off of — see `config/galactic-cni/daemonset.yaml`, `config/galactic-nat66/base/daemonset.yaml`). This is the default role, so it carries no role-specific name/label of its own — only `rr` below needs one, to coexist as a second DaemonSet.
- **`config/galactic-router/overlays/rr/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic-route-reflector` set (stays at zero replicas otherwise). This is a dedicated boolean-style label, not a `galactic.datumapis.com/node` value — that key is single-valued, so a route-reflector node can still independently be `galactic.datumapis.com/node: compute` (or any other value) at the same time. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it.
- Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/default/`. `galactic-router` no longer has a gateway role of its own — see `config/galactic-gateway/` below.
- **`config/galactic-gateway/`** — the edge XDP NAT+LB gateway control plane, a separate `galactic-gateway` binary rather than a `galactic-router` role, so a crash on either side no longer takes the other down with it. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (the two-container `galactic-router` + `galactic-gateway` pod, requiring nodes labeled `galactic.datumapis.com/node: edge` — the actual network-edge/ingress boundary, not to be confused with `galactic-router`'s own `compute` role) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_SRV6_ADDRESS` must be unique per gateway node and has no generic default (no in-cluster mechanism yet derives it automatically — see `internal/controller/networkgateway_controller.go`'s `publishSelfAddress` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public-interface/SRv6-address values; see `deploy/containerlab/resources/galactic-gateway/` for a worked two-node example. Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption.
- **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric` (a dedicated boolean-style label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic-route-reflector` flag that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so `frr-init` gets the pod's node name via a `NODE_NAME` downward-API env var and selects a per-node `frr.conf.<nodename>` key from the ConfigMap, rather than assuming one shared `frr.conf` for the whole DaemonSet. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author a `fabric-config` ConfigMap with one `frr.conf.<nodename>` key per matching node (`daemons`/`vtysh.conf` are baked into the `fabric-router` image and only need to be in the ConfigMap if overriding those defaults) before applying `kubectl apply -k config/fabric-router/`.
- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an IPv6 SRv6 transit mesh. FRR runs as a hostNetwork DaemonSet on each worker for eBGP underlay; `galactic-router` (default role) handles EVPN path distribution over iBGP, the iad route reflector builds on `config/galactic-router/overlays/rr/`, and iad additionally has two dedicated edge-role nodes (`iad-gateway1`/`iad-gateway2`, `config/galactic-gateway/base/`) as a canary for the edge XDP NAT+LB gateway. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.

## New Developer Entry Points

Expand All @@ -89,3 +101,4 @@ Summary:
6. Explore `internal/plumbing/` for shared kernel and network primitives (VRF, sysctl, interface naming, SRv6).
7. Read `internal/gateway/engine.go` and `internal/controller/networkgateway_controller.go` to understand the edge XDP NAT+LB gateway's convergence loop and Active-Active BGP placement model. See [ARCHITECTURE-GATEWAY.md](docs/agents/ARCHITECTURE-GATEWAY.md).
8. See `docs/cni/cni-cmd-sequence.md`, `docs/cni/gc-cmd-sequence.md`, and `docs/agent-startup.md` for Mermaid sequence diagrams of the CNI attach path, garbage collection, and router startup, respectively. `docs/cni/configuration.md` and `docs/router/configuration.md` document CNI config fields and router environment variables.
9. Read [docs/node-labels.md](docs/node-labels.md) to understand which `galactic.datumapis.com/*` labels put which DaemonSets on which nodes, and why it's five independent labels rather than one enum — this cuts across all five binaries and isn't owned by any single component's architecture doc.
Loading