Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions config/galactic-cni/rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,22 @@ rules:
resources:
- nodes
verbs: ["get"]
# discovery.k8s.io/endpointslices: galactic-bgp's per-pod EndpointSlice
# publish (ADD), delete (DEL), and CHECK — see internal/cnibgp/
# endpointslice.go and docs/plans/854-vpc-http-ingress-endpointslice.md.
- apiGroups: ["discovery.k8s.io"]
resources:
- endpointslices
verbs: ["get", "list", "create", "update", "patch", "delete"]
# pods (get only): to look up the owning Pod's UID and set it as the
# EndpointSlice's ownerReference, so the k8s garbage collector reclaims
# a force-deleted/never-DEL'd pod's EndpointSlice as a backstop (Phase 8
# of the #854 plan) -- no create/update/delete needed, this SA never
# writes a Pod.
- apiGroups: [""]
resources:
- pods
verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
Expand Down
16 changes: 11 additions & 5 deletions docs/agents/ARCHITECTURE-CNI.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
> (VRF, veth/tap, SRv6 uSID datapath registration) and writes
> `BGPAdvertisement`/`BGPVRFInstance` CRDs for `galactic-router` to pick up.

_Last updated: 2026-08-13_
_Last updated: 2026-08-18_

This document covers the CNI side of Galactic only. See
[ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md) for the BGP/EVPN control
Expand All @@ -23,7 +23,13 @@ When a pod or VM is attached to a VPC, a chain of CNI plugins creates the
required kernel state (VRF, veth pair or tap device, host-side routes) and
writes a `BGPAdvertisement` CRD. `galactic-router` (see
[ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md)) watches that CRD and
injects the EVPN path into the node-local GoBGP server.
injects the EVPN path into the node-local GoBGP server. `galactic-bgp` also
publishes a per-pod `discoveryv1.EndpointSlice` (when the attachment has an
IPv6 address to carry) — the mechanism the HTTP-ingress extension server
discovers VPC backends through; see
[docs/plans/854-vpc-http-ingress-endpointslice.md](../plans/854-vpc-http-ingress-endpointslice.md)
and [docs/cni/configuration.md](../cni/configuration.md)'s "EndpointSlice
publish" section.

The CNI attach side is a **chain of small binaries**, not one monolithic
plugin: a master plugin (`galactic-veth` for containers, `galactic-tap` for
Expand Down Expand Up @@ -414,15 +420,15 @@ any shared, per-attachment kernel/CRD state — see the `cmdDel` note in
| `internal/cni` | galactic-veth | Veth master plugin: `cmdAdd`/`cmdDel`/`cmdCheck`/`cmdStatus`; PluginConf parsing; NAD annotation; host-device delegation; delegates kernel work to plumbing | No |
| `internal/hostconf` | every CNI-chain binary | Shared `HostConf` schema + static-conflist loader, plus API-based node-name auto-detect | No |
| `internal/hostgw` | galactic-veth, galactic-tap | Host-side gateway address/route configuration for a VPC attachment's allocated IPAM addresses | No |
| `internal/crdnames` | galactic-veth, galactic-bgp | Deterministic `BGPVRFInstance`/`BGPAdvertisement` CRD name + annotation-key derivation (also read by `galactic-router`'s GC — see [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md)) | No |
| `internal/nadpatch` | galactic-veth, galactic-tap | NAD annotation patch (host interface name) + pod-namespace parsing from `CNI_ARGS` | No |
| `internal/crdnames` | galactic-veth, galactic-bgp | Deterministic `BGPVRFInstance`/`BGPAdvertisement`/EndpointSlice CRD name + annotation/label-key derivation (also read by `galactic-router`'s GC — see [ARCHITECTURE-ROUTER.md](ARCHITECTURE-ROUTER.md)) | No |
| `internal/nadpatch` | galactic-veth, galactic-tap, galactic-bgp | NAD annotation patch (host interface name) + pod-name/pod-namespace parsing from `CNI_ARGS` | No |
| `internal/cni/ipam` | galactic-ipam | IPv6/IPv4 pool allocators + static IP allocator; on-disk marker-file persistence (flock-guarded, keyed by containerID) | Yes (pool allocations + marker files) |
| `internal/cni/route` | galactic-route | Host-side static route add/delete via netlink | No |
| `internal/cni/tap` | galactic-tap | Tap interface create/delete for VM workloads (Kata, Firecracker, kraftlet/Unikraft) | No |
| `internal/cni/veth` | galactic-veth | veth pair create/delete | No |
| `internal/cnitap` | galactic-tap | Tap master plugin (mirrors `internal/cni`; no host-device delegation, no guest netns) | No |
| `internal/cniipam` | galactic-ipam | CNI IPAM delegation protocol (`cmdAdd`/`cmdDel`/`cmdCheck`/`cmdStatus`); explicit `"ipam"`-block contract; no k8s dependency | No |
| `internal/cnibgp` | galactic-bgp | BGP/SRv6/eBPF publish: SID/Argument allocation + collision detection, `registerEBPFDatapath`/`unregisterEBPFDatapath`, `BGPVRFInstance`/`BGPAdvertisement` CRUD with retry; learns everything from `prevResult` | No |
| `internal/cnibgp` | galactic-bgp | BGP/SRv6/eBPF publish: SID/Argument allocation + collision detection, `registerEBPFDatapath`/`unregisterEBPFDatapath`, `BGPVRFInstance`/`BGPAdvertisement` CRUD with retry, per-pod EndpointSlice publish/delete/CHECK for HTTP-ingress backend discovery (`endpointslice.go`); learns everything from `prevResult` | No |
| `internal/cniroute` | galactic-route | Termination-route plugin: installs/rolls-back VRF-table routes; no k8s dependency | No |
| `internal/vmtap` | vmtap-cni | Patches Cilium's own chain conflist to add a tap-interface stage for VM workloads | No |
| `internal/installer` | galactic-cni | DaemonSet `init`/`run` support: binary staging (every chain binary), node-identity check, conflist/kubeconfig templating, credential refresh ticker, log rotation, eBPF datapath lifecycle, gRPC health server | No |
Expand Down
41 changes: 41 additions & 0 deletions docs/cni/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,47 @@ what addresses were allocated entirely from `prevResult` (the accumulated
result of every preceding plugin in the chain), never from its own config or
a kernel call.

### EndpointSlice publish (HTTP ingress backend discovery)

Alongside the `BGPVRFInstance`/`BGPAdvertisement` CRDs, `galactic-bgp`
publishes one `discoveryv1.EndpointSlice` per pod — named after the pod, in
the pod's own namespace — whenever the attachment has an IPv6 address to
carry (i.e. `ipam` is configured on the master plugin's stanza; a tap/VM
attachment with no `ipam` block, same as a veth attachment with none, has
nothing to publish and is skipped, not an error). This is the mechanism the
HTTP ingress extension server (datum-cloud/enhancements#854/#796) discovers
VPC backends through — no backing `Service` object exists to key a
Service-generated `EndpointSlice` off of.

IPv6-only: a dual-stack pod's IPv4 address is not published. The
EndpointSlice carries:

- `spec.endpoints[].addresses`: the pod's allocated IPv6 address.
- Label `galactic.datum.net/tenant-id`: `TenantIdentifier(vpc, vpcattachment)`
— the discovery mechanism (annotations aren't selectable in a k8s
`List`/`Watch`).
- Annotation `galactic.datum.net/tenant-id`: the same value, for
human-readable detail.
- Annotation `galactic.datum.net/srv6-sid`: the computed SRv6 uSID
(`internal/plumbing/srv6.ComputeSID`) routing to this pod's VRF — absent
when this node's `BGPRouter` has no `srv6Locator`/`nodeID` configured.
- `metadata.ownerReferences`: the owning Pod, so the Kubernetes garbage
collector reclaims it if this plugin's own DEL is never run (a
force-deleted pod). CNI DEL deletes it explicitly and unconditionally on
the normal path — unlike the BGP CRDs, an EndpointSlice is 1:1 with
exactly one pod and is never shared with a sibling attachment, so DEL
deleting it immediately is safe.

CHECK verifies the EndpointSlice still exists with the expected address and
annotations.

Both the pod's name and its namespace are parsed from `K8S_POD_NAME`/
`K8S_POD_NAMESPACE` in `CNI_ARGS` (`internal/nadpatch.ParsePodName`/
`ParsePodNamespace`) — Multus always sets both for a real pod-scoped
invocation, but a standalone/manual invocation (e.g. one that skips the CNI
runtime, as some `tests/e2e` cases do) must set `CNI_ARGS` itself or ADD
fails outright and CHECK reports an error.

## Example Configurations

Every example below is a full conflist (a `NetworkAttachmentDefinition`'s
Expand Down
4 changes: 3 additions & 1 deletion docs/plans/854-vpc-http-ingress-endpointslice.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
- **Issue:** [datum-cloud/enhancements#854](https://github.com/datum-cloud/enhancements/issues/854)
- **Parent:** [datum-cloud/enhancements#796](https://github.com/datum-cloud/enhancements/issues/796) — HTTP Ingress for VPC Networks
- **Design doc:** [HTTP Ingress for VPC Networks](https://github.com/datum-cloud/enhancements/blob/main/enhancements/networking/http-ingress-for-vpc-networks.md) (PR [#851](https://github.com/datum-cloud/enhancements/pull/851), resolves [#853](https://github.com/datum-cloud/enhancements/issues/853))
- **Status:** planning only — no implementation started. Revised 2026-08-13 after a review pass caught drift against the current repo (see the "Revision note" callouts throughout) — most of it stale paths, but two are real design gaps (Phases 4 and 8) worth reading before starting. Revised again 2026-08-17: Open Decision 5 resolved — VM/tap-attached workloads are in scope and are this issue's primary use case, not an implicitly-excluded edge case; Phase 4's nil-`ipamResult` skip already handles this correctly (it's an address-existence check, not a VM exclusion) once confirmed against the current `internal/cnitap`/`internal/cnibgp` code, so no VM-specific implementation work is added by this revision. Open Decision 4 (Phase 4's rollback-risk callout) also resolved — fix #1, narrowing `advertisementCreated` to create-only gating, confirmed safe and adopted. Open Decision 6 (Phase 8's GC mechanism) also resolved — the recommended `ownerReference`-to-Pod approach is adopted as-is; `internal/gc/gc.go`/`config/galactic-router/rbac.yaml` are untouched by this issue.
- **Status:** implemented 2026-08-18, per this plan's sequencing (§4) — Phases 1–2 (`crdnames`/`nadpatch.ParsePodName`), 3 (SID computation), 4 (EndpointSlice publish on ADD + the `advertisementCreated` rollback-risk fix), 5 (DEL), 6 (CHECK), 7 (RBAC), 8 (ownerReference-to-Pod GC backstop, folded into Phase 4's publish step), 9 (docs), and unit tests (10) all landed together. See `internal/cnibgp/endpointslice.go`, the `internal/cnibgp/bgp.go`/`ops_add.go`/`ops_del.go`/`ops_check.go` changes, `internal/crdnames`/`internal/nadpatch`, and `config/galactic-cni/rbac.yaml`. Not yet exercised in `tests/e2e` (§3's suggested e2e case is still open). Previously revised 2026-08-13 after a review pass caught drift against the current repo (see the "Revision note" callouts throughout) — most of it stale paths, but two were real design gaps (Phases 4 and 8). Revised again 2026-08-17: Open Decision 5 resolved — VM/tap-attached workloads are in scope and are this issue's primary use case, not an implicitly-excluded edge case; Phase 4's nil-`ipamResult` skip already handles this correctly (it's an address-existence check, not a VM exclusion), confirmed against `internal/cnitap`/`internal/cnibgp`. Open Decision 4 (Phase 4's rollback-risk callout) also resolved — fix #1, narrowing `advertisementCreated` to create-only gating, confirmed safe and adopted. Open Decision 6 (Phase 8's GC mechanism) also resolved — the recommended `ownerReference`-to-Pod approach is adopted as-is; `internal/gc/gc.go`/`config/galactic-router/rbac.yaml` are untouched by this issue.

> **Revision note — CI fix 2026-08-19: EndpointSlice namespace-wiring bug.** Adding the suggested `tests/e2e` case (previous revision's "not yet exercised" callout) surfaced that `ops_add.go`/`ops_check.go`/`ops_del.go` were passing `pluginConf.Namespace` (where the BGP CRDs live, defaulting to `galactic-system`) as the EndpointSlice's namespace, instead of the pod's own namespace parsed from `CNI_ARGS` via `nadpatch.ParsePodNamespace` — contradicting this plan's own Phase 4 description and `docs/cni/configuration.md`'s "in the pod's own namespace." It went undetected because every existing unit test's fixtures set `pluginConf.Namespace` equal to the pod's test namespace, and the e2e cluster's own default `kubectl` namespace happens to be `galactic-system` too (`scripts/ci.sh`'s `kubectl config set-context --current --namespace=galactic-system`) — masking the mismatch everywhere it would otherwise have surfaced. Fixed by parsing `podNamespace` from `CNI_ARGS` at all three call sites (mirroring the existing `podName` parsing) and threading it through instead; `pluginConf.Namespace` is now used only for the BGP CRD/`BGPRouter` lookups it was always meant for. The e2e test's manually-chained `galactic-bgp` invocation (`tests/e2e/e2e_test.go`) also had to start setting `CNI_ARGS` itself (`K8S_POD_NAME`/`K8S_POD_NAMESPACE`), since it invokes the binary directly rather than through the CNI runtime/Multus, which sets these for every real pod-scoped invocation.

## Correction to #854's framing

Expand Down
37 changes: 34 additions & 3 deletions internal/cnibgp/bgp.go
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,14 @@ type publishResult struct {
// for why that distinction, not "CreateOrUpdate succeeded" alone, is
// what makes rollback-deletion safe.
vrfInstanceCreated bool
// sid is the computed SRv6 uSID for this attachment (see
// internal/plumbing/srv6.ComputeSID), valid (netip.Addr.IsValid()) only
// when this node's BGPRouter has SRv6Locator/nodeID configured — the
// same condition registerEBPFDatapath's own skip case checks. Consumed
// by the EndpointSlice publish step (endpointslice.go), which runs as
// its own step after publishBGPState returns, not folded into its retry
// closure — see Phase 4's rollback-risk note in the #854 plan for why.
sid netip.Addr
}

// isTransientError reports whether err is a transient failure that may
Expand Down Expand Up @@ -467,6 +475,19 @@ func publishBGPState(
// registerEBPFDatapath's own doc comment for why.
prefixes, ipv6Subnet, ipv4Addr := ipamAdvertisementPrefixes(ipamResult)

// Reuses registerEBPFDatapath's own "SRv6 not configured, skip
// silently" sentinel: if this node's router has no
// srv6Locator/nodeID configured, there's nothing to publish, so
// leave result.sid at its zero value (IsValid() == false) rather
// than computing a SID for an attachment that has no SRv6 endpoint.
if bgp.srv6Locator != "" && bgp.nodeID != 0 {
sid, err := srv6.ComputeSID(bgp.srv6Locator, bgp.nodeID, vrfID, bgpv1alpha1.SRv6FunctionEndDT46)
if err != nil {
return fmt.Errorf("compute SRv6 uSID: %w", err)
}
result.sid = sid
}

// The return values aren't tracked for rollback: the vrf_table entry
// they'd describe is shared by every attachment on this VPC/node,
// same as the BGPVRFInstance above — see resourceTracker.cleanup's
Expand All @@ -484,7 +505,7 @@ func publishBGPState(
},
}
var mergedPrefixes []string
_, err = controllerutil.CreateOrUpdate(ctx, k8s, adv, func() error {
advOp, err := controllerutil.CreateOrUpdate(ctx, k8s, adv, func() error {
if adv.Annotations == nil {
adv.Annotations = make(map[string]string)
}
Expand Down Expand Up @@ -519,9 +540,19 @@ func publishBGPState(
if err != nil {
return fmt.Errorf("apply BGPAdvertisement: %w", err)
}
result.advertisementCreated = true
// Gated on OperationResultCreated, mirroring vrfInstanceCreated's
// existing pattern exactly: a BGPAdvertisement is reused (updated,
// not created) across pod churn on the same vpcAttachment, so
// marking it created on every successful write — including a mere
// update of an already-live sibling's CRD — would let
// resourceTracker.cleanup delete a BGPAdvertisement still backing a
// different, live container's route if a later ADD step fails. See
// the #854 plan's Phase 4 rollback-risk note.
if advOp == controllerutil.OperationResultCreated {
result.advertisementCreated = true
}
slog.Debug("BGP: BGPAdvertisement applied", "name", adv.Name, "namespace", namespace,
"prefixes", mergedPrefixes, "addedPrefixes", prefixes, "containerID", args.ContainerID)
"prefixes", mergedPrefixes, "addedPrefixes", prefixes, "containerID", args.ContainerID, "operation", advOp)

slog.Info("ADD: BGP state published", "containerID", args.ContainerID,
"vpc", cfg.vpc, "vpcAttachment", cfg.vpcAttachment)
Expand Down
Loading