Skip to content

Bond local NVMe drives together via software RAID. Run kubelet root there. #7929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
keith-ms merged 7 commits into from
Mar 18, 2026
Merged

Bond local NVMe drives together via software RAID. Run kubelet root there. #7929

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
086c20e
Add format and mount for NVMe root for GB200 image. Move kubelet root…
abenn135 Feb 20, 2026
4b47a11
copy over format-mount-nvme-root script and service to packer vm in g…
abenn135 Feb 23, 2026
b532d27
Use mdadm to bond the four NVMe drives in software RAID0. Put kubelet…
abenn135 Feb 23, 2026
97456dc
Change from using a script and service and put the functionality into…
keith-ms Mar 5, 2026
1332ee7
chore: update trivy v0.69.2 (#7994)
chmill-zz Mar 2, 2026
6995394
Revert to changes abenn made
keith-ms Mar 17, 2026
aade4f2
Extract configuration from cse_main.sh, reorganize files
keith-ms Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions parts/linux/cloud-init/artifacts/ubuntu/format-mount-kubelet.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[Unit]
Requires=format-mount-nvme-root.service
After=format-mount-nvme-root.service
13 changes: 13 additions & 0 deletions parts/linux/cloud-init/artifacts/ubuntu/format-mount-nvme-root.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[Unit]
Description=Format NVMe local disk and mount Kubelet there
Requires=mnt.mount
After=mnt.mount

[Service]
Restart=on-failure
RemainAfterExit=yes
Type=oneshot
ExecStart=/bin/bash /opt/azure/containers/format-mount-nvme-root.sh

[Install]
WantedBy=multi-user.target
35 changes: 35 additions & 0 deletions parts/linux/cloud-init/artifacts/ubuntu/format-mount-nvme-root.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
set -x

# Bind mount kubelet to local NVMe storage specifically on startup.
MOUNT_POINT="/mnt/aks"


KUBELET_MOUNT_POINT="${MOUNT_POINT}/kubelet"
KUBELET_DIR="/var/lib/kubelet"

mkdir -p "${MOUNT_POINT}"

SENTINEL_FILE="/opt/azure/containers/bind-sentinel"
if [ ! -e "${SENTINEL_FILE}" ]; then
# Bond (via software RAID) and format the NVMe disks if that's not already done.
if [ -e /dev/disk/azure/local/by-index/1 ] && [ ! -e /dev/md0 ]; then
mdadm --create --verbose /dev/md0 --level=0 --raid-devices=4 /dev/disk/azure/local/by-index/1 /dev/disk/azure/local/by-index/2 /dev/disk/azure/local/by-index/3 /dev/disk/azure/local/by-index/4
mkfs.ext4 -F /dev/md0
fi
mount /dev/md0 "${MOUNT_POINT}"
mv "${KUBELET_DIR}" "${KUBELET_MOUNT_POINT}"
touch "${SENTINEL_FILE}"
else
# On subsequent boots, the disk should already be partitioned and formatted, so just mount it.
mount /dev/md0 "${MOUNT_POINT}"
fi

# on every boot, bind mount the kubelet directory back to the expected
# location before kubelet itself may start.
mkdir -p "${KUBELET_DIR}"
mount --bind "${KUBELET_MOUNT_POINT}" "${KUBELET_DIR}"
chmod a+w "${KUBELET_DIR}"
12 changes: 12 additions & 0 deletions vhdbuilder/packer/packer_source.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,18 @@ copyPackerFiles() {
CLOUD_INIT_STATUS_CHECK_DEST=/opt/azure/containers/cloud-init-status-check.sh
cpAndMode $CLOUD_INIT_STATUS_CHECK_SRC $CLOUD_INIT_STATUS_CHECK_DEST 0744

if grep -q "GB200" <<< "$FEATURE_FLAGS"; then
FMT_SH_SRC=/home/packer/format-mount-nvme-root.sh
FMT_SH_DEST=/opt/azure/containers/format-mount-nvme-root.sh
cpAndMode $FMT_SH_SRC $FMT_SH_DEST 0544
FMT_SVC_SRC=/home/packer/format-mount-nvme-root.service
FMT_SVC_DEST=/etc/systemd/system/format-mount-nvme-root.service
cpAndMode $FMT_SVC_SRC $FMT_SVC_DEST 600
FMT_SVC_SRC=/home/packer/format-mount-kubelet.conf
FMT_SVC_DEST=/etc/systemd/system/kubelet.service.d/11-fmtmount.conf
cpAndMode $FMT_SVC_SRC $FMT_SVC_DEST 600
fi

NOTICE_SRC=/home/packer/NOTICE.txt
NOTICE_DEST=/NOTICE.txt

Expand Down
2 changes: 1 addition & 1 deletion vhdbuilder/packer/trivy-scan.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ CVE_DIFF_QUERY_OUTPUT_PATH=${TRIVY_REPORT_DIRNAME}/cve-diff.txt
CVE_LIST_QUERY_OUTPUT_PATH=${TRIVY_REPORT_DIRNAME}/cve-list.txt
TRIVY_DB_REPOSITORIES="mcr.microsoft.com/mirror/ghcr/aquasecurity/trivy-db:2,ghcr.io/aquasecurity/trivy-db:2,public.ecr.aws/aquasecurity/trivy-db"

TRIVY_VERSION="0.57.0"
TRIVY_VERSION="0.69.2"
TRIVY_ARCH=""

MODULE_NAME="vuln-to-kusto-vhd"
Expand Down
15 changes: 15 additions & 0 deletions vhdbuilder/packer/vhd-image-builder-arm64-gb200.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,21 @@
"source": "parts/linux/cloud-init/artifacts/cloud-init-status-check.sh",
"destination": "/home/packer/cloud-init-status-check.sh"
},
{
"type": "file",
"source": "parts/linux/cloud-init/artifacts/ubuntu/format-mount-nvme-root.sh",
"destination": "/home/packer/format-mount-nvme-root.sh"
},
{
"type": "file",
"source": "parts/linux/cloud-init/artifacts/ubuntu/format-mount-nvme-root.service",
"destination": "/home/packer/format-mount-nvme-root.service"
},
{
"type": "file",
"source": "parts/linux/cloud-init/artifacts/ubuntu/format-mount-kubelet.conf",
"destination": "/home/packer/format-mount-kubelet.conf"
},
{
"type": "file",
"source": "vhdbuilder/packer/prefetch.sh",
Expand Down
Loading