Skip to content

[olcedar] d8 platform olcedar node add for static immutable nodes - #450

Draft
borg-z wants to merge 4 commits into
mainfrom
olcedar-node-add
Draft

[olcedar] d8 platform olcedar node add for static immutable nodes#450
borg-z wants to merge 4 commits into
mainfrom
olcedar-node-add

Conversation

@borg-z

@borg-z borg-z commented Aug 20, 2026

Copy link
Copy Markdown
Member

What

Adds d8 platform olcedar node add <address> --group <nodegroup> — one command in place of the five-step manual flow for introducing a static node running the immutable OS (olcedar):

# before
d8 k get nodeconfigtemplate worker -o yaml > template.yaml
curl http://<machine>:50000/inventory > machine.yaml
# uncomment the disk in machine.yaml
yq eval-all 'select(fi==0) * select(fi==1)' template.yaml machine.yaml > worker-0.yaml
yq -i '.metadata.name = "worker-0" | .spec.nodeName = "worker-0"' worker-0.yaml
curl -X PUT --data-binary @worker-0.yaml http://<machine>:50000/config

# after
d8 platform olcedar node add 10.12.4.55 --group worker

Neither side can do this alone: the cluster does not know the machine's disks, the machine does not know the cluster's bootstrap token. The CLI sees both — and, unlike a file, it can ask.

Flow

  1. GET /whoami — a machine already held by the node agent is refused by node name, resolved from its InternalIP.
  2. Read nodeconfigtemplates.templates.internal.deckhouse.io/v1alpha1 for the group. A 404 is explained against the NodeGroup itself (missing / wrong nodeType / wrong systemType), never shown as "not found".
  3. GET /inventory.json. An image too old to serve it is refused — there is nothing to pick a disk out of.
  4. Ask the disk, the network and the node name.
  5. Build internal.deckhouse.io/v1alpha1 NodeConfig, PUT /config, wait for the node to register.

Adopting a disk, and never erasing one by accident

init decides for itself whether anything is installed. resolveSystemDisk (images/init/src/0.1/disk.go) finds one disk carrying BOOT/CONFIG/DATA and answers install=false; initmode.go then computes provision := install || cfg.Spec.Storage.Wipe. So wipe: true is the one thing that can force a provision where none is needed — and that provision is destructive first and sourced second: provisionDisk erases, and only afterwards copy-boot-assets looks for the UKI and the rootfs under /run/media. A machine booted from its own disk (every disk cloned from an image, so every DVP node) has no such media: the erase succeeds, the copy fails, and nothing is left to boot from.

This command therefore:

  • adopts a disk that already carries the layout — the document names no storage at all, because init identifies that disk itself, and a selector disagreeing with the pin writeDiskPin recorded at install is itself a reason for it to reinstall;
  • writes wipe only from an explicit --wipe, which reads as "reinstall, from installation media" and says what it costs without one;
  • calls out a second disk carrying the layout: the node identifies its disk by that layout before it reads any selector, so it would take that one and leave the named disk alone.

What those notes warn about cannot be settled from an inventory — diskState spells installer media and an installed system the same way (inventory.go:101), and only the machine knows which disk it booted from. Teaching the inventory that fact belongs to the initramfs repository, where bootedDisk() already computes it.

The three documents this produces:

# blank disk, no --wipe                 # layout already there (adopt)      # --wipe
storage:                                storage: {}                         storage:
  diskSelector:                                                               diskSelector:
    serial: S3Z8NB0K700002                                                      serial: ZA20ABCD
                                                                              wipe: true

The rest of what is never decided silently

  • The single blank disk is the default but still confirmed. Two blank disks and no answer → refusal naming --disk-selector.
  • The disk is named by the most stable attribute it reports (wwidserialbusPathname), so the document survives sda becoming sdb.
  • The interface the CLI reached the machine on is the default, proven by the connection itself. An address belonging to no interface (a port forward, a NAT) is asked about, or named with --network-interface.
  • DHCP unless the operator pins the current address; --network dhcp|static decides it up front.
  • Without a TTY and without --yes the command refuses instead of choosing.

Secrets

The template carries a live bootstrap token, registry.auth and the registry-packages-proxy token on every read. The document is assembled in memory, is never logged or written to a file, and --dry-run prints it with those three fields redacted. There is a test asserting the fixture token value does not appear in the redacted output.

Non-interactive

d8 platform olcedar node add 10.12.4.55 --group worker \
  --name worker-1 --disk-selector serial=S3Z8NB0K700002 --network dhcp --yes

--group completes to Static+Immutable NodeGroups only — a group that cannot take a machine is never offered.

On not importing dhctl

The machine-facing HTTP contract (/whoami, /inventory.json, /config) and the inventory types mirror dhctl/pkg/immutable of the deckhouse repository rather than importing it. CheckDocumentAgainstInventory there reaches dhctl/pkg/config and dhctl/pkg/global, so importing it means pulling dhctl's whole dependency tree into the CLI for some 250 lines of net/http. The copies carry a mirror comment naming the original.

Its disk-vs-inventory check is not ported wholesale either: TemplateStorage.render blanks spec.storage and spec.network before serving, so both halves are filled in by this command out of the inventory it just read — checking them back against it checks the CLI against itself. What is ported is the part that catches a live mistake: a --disk-selector from a human must resolve to exactly one disk.

Tests

go test ./internal/olcedar/... ./internal/system/...222 passed in 44 packages. Among them:

  • refusal on agent,
  • refusal without a TTY and without --yes,
  • worker-0 taken → worker-1 offered (and hole-filling, and other groups ignored),
  • a system-layout disk adopted with no storage in the document,
  • wipe present only when --wipe was passed,
  • the missing-media and second-layout-disk warnings,
  • the fixture token absent from --dry-run output.

Still to do

A live run on a stand — machine in installer → command → node registers, d8 k get nodeconfig <name> showing the chosen network and disk — is not part of this PR; that is what the draft state is for.

🤖 Generated with Claude Code

Adding a static node running the immutable OS took five steps by hand: read
the NodeGroup template, curl the machine inventory, uncomment a disk, merge
the two documents with yq, and PUT the result back. Neither side could do it
alone — the cluster does not know the machine's disks, the machine does not
know the cluster's bootstrap token — but the CLI sees both, and unlike a file
it can ask.

The command reads the aggregated NodeConfigTemplate of the group and the
machine's /inventory.json, asks which disk, which network and which node name,
and pushes the assembled NodeConfig to the machine's maintenance port.

Nothing is decided silently. The single blank disk is offered as a default but
still confirmed; a disk that already holds a system is refused until the
operator confirms the erase (or passes --wipe); without a terminal and without
--yes the command refuses rather than choosing. A machine whose port is held by
the node agent is refused by node name: a second configuration would replace
the one a working node runs on.

The template carries a live bootstrap token, the registry auth and the
registry-packages-proxy token on every read, so the document is built in
memory, never logged, and --dry-run prints it with those three fields redacted.

The machine-facing HTTP contract (/whoami, /inventory.json, /config) mirrors
dhctl/pkg/immutable of the deckhouse repository rather than importing it: the
dhctl module would pull its whole dependency tree in for some 250 lines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
borg-z and others added 3 commits August 21, 2026 21:28
…sing it

The first cut read `system-layout` as "an OS is here, the install will erase
it" and wrote `spec.storage.wipe: true` once the operator confirmed. On a
machine whose disk was cloned from an image — every DVP node — that is not a
confirmation, it is a self-destruct.

init decides for itself whether anything is installed:
`resolveSystemDisk` (images/init/src/0.1/disk.go) finds one disk carrying
BOOT/CONFIG/DATA and answers `install=false`, and initmode.go computes
`provision := install || cfg.Spec.Storage.Wipe`. So `wipe: true` is the one
thing that can force a provision where none is needed — and the provision is
destructive first and sourced second: `provisionDisk` erases, and only then
`copy-boot-assets` looks for the UKI and the rootfs under /run/media. A machine
booted from its own disk has no such media, so the copy fails after the erase
and there is nothing left to boot from.

So:

  - a disk that already carries the layout is adopted, not installed onto, and
    the document names no storage at all: init identifies that disk itself, and
    a selector disagreeing with the pin recorded at install (writeDiskPin) is
    itself a reason for it to reinstall;
  - `wipe` reaches the document only from an explicit --wipe, which now reads as
    "reinstall, from installation media", and says what it costs without it;
  - a second disk carrying the layout is called out: the node identifies its
    disk by that layout before it reads any selector, so it would take that one
    and leave the named disk alone.

What the notes warn about cannot be settled from an inventory: `diskState`
spells installer media and an installed system the same way (inventory.go:101),
and only the machine knows which disk it booted from. Adding that fact to the
inventory belongs to the initramfs repository, where bootedDisk() already
computes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
A live run on zykov-st read as one paragraph: the answer to a question, the
note about the disk and the next question all ran together, and the wait that
followed said nothing for minutes.

So the prompt sets its blocks apart itself — a blank line after a list, before
a question, and around a note — rather than leaving each caller to remember a
newline. A machine with one disk now offers it as the default: there is no
choice to make, and the list still shows what the disk is.

The wait says it is still waiting every 30 seconds, and the command closes with
what the addition cost:

  Timings
    read the cluster and the machine      1.2s
    pushed the configuration              0.3s
    node registered                      1m24s
    machine time in total                1m24s

The operator's own thinking time sits between the read and the push and is
deliberately not counted: the number worth knowing is how long the machine
takes, not how long someone stared at a disk list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
A run against 10.12.0.40 hung with nothing on the screen at all. The machine
was fine — it had already answered /whoami — and the command was sitting in the
template read: nodeconfigtemplates is served by an aggregated API, so the
kube-apiserver proxies it to node-controller, and client-go carries no timeout
of its own. A backend that never answers hangs the command forever.

So the rest config now bounds every cluster read at 30 seconds, and each
network step says what it is about to do before it does it. Two silent minutes
become two lines and, at worst, one bounded failure.

That failure names where the answer comes from, because the kube-apiserver is
almost never the part that is broken:

  read the node configuration template of worker: <err>. It is served by an
  aggregated API, so this read is proxied by the kube-apiserver to
  node-controller of node-manager: a node-controller that is down, unreachable
  or unregistered fails it. Check it with:
  d8 k get apiservice v1alpha1.templates.internal.deckhouse.io

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant