feat(localdns): add systemd-based Prometheus-format metrics exporter

[saewoni]

Adds CPU and memory metrics for localdns.service using systemd accounting and socket activation for efficient, zero-overhead monitoring.

Also export IP addresses configured for forward plugins for kubednsoverrides and vnetdnsoverrides

Implementation:

• localdns_exporter.sh: Scrapes systemd CPUUsageNSec and MemoryCurrent
• localdns-exporter.socket: Socket activation on port 9353
• localdns-exporter@.service: Instantiated service per connection
• Integrated into all VHD builders (Ubuntu, Mariner, Flatcar, all arches)

Metrics exposed:

• localdns_cpu_usage_seconds_total (counter)
• localdns_memory_usage_mb (gauge)

Test coverage:

• e2e/test-localdns-exporter.sh validates exporter functionality

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

[Copilot]

Pull request overview

This PR adds a lightweight Prometheus-compatible metrics exporter for the localdns.service using systemd socket activation. The exporter exposes CPU and memory metrics on port 9353 with zero overhead when not being scraped, making it suitable for production monitoring.

Changes:

• Added localdns_exporter.sh bash script that queries systemd accounting metrics and formats them as Prometheus metrics
• Added systemd socket (localdns-exporter.socket) and service (localdns-exporter@.service) units for on-demand activation
• Integrated the exporter into all Linux VHD builds (Ubuntu, Mariner, Flatcar, ARM64/x64)
• Added VHD content validation tests and a standalone test script

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
parts/linux/cloud-init/artifacts/localdns_exporter.sh	Bash script that scrapes systemd metrics (CPUUsageNSec, MemoryCurrent) and outputs Prometheus-formatted HTTP response
parts/linux/cloud-init/artifacts/localdns-exporter@.service	Systemd template service for per-connection worker instances with security hardening
parts/linux/cloud-init/artifacts/localdns-exporter.socket	Systemd socket unit listening on port 9353 with Accept=yes for on-demand activation
vhdbuilder/packer/vhd-image-builder-*.json	Added file provisioners to copy exporter artifacts to all Linux VHD variants
vhdbuilder/packer/packer_source.sh	Added file copying logic and systemctl enable command for the socket
vhdbuilder/packer/imagecustomizer/azlosguard/azlosguard.yml	Added exporter files to OSGuard VHD build but missing socket enablement
vhdbuilder/packer/test/linux-vhd-content-test.sh	Added validation for exporter files and permissions
e2e/test-localdns-exporter.sh	Standalone test script for manual validation

[yewmsft]

LGTM in general. Let's make sure we have proper test coverages

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

[Copilot]

This change updates parts/linux/cloud-init/artifacts/cse_config.sh, which is snapshot-tested via pkg/agent/testdata/** (CustomData/CSECommand). The PR currently doesn’t include regenerated testdata artifacts; make generate should be run and the updated snapshot files committed, otherwise unit tests comparing generated CustomData/CSE output are likely to fail in CI.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 89 changed files in this pull request and generated 4 comments.

[Copilot]

The security-directive assertions are brittle across OS/systemd versions: systemctl show --property=... may omit unsupported properties (and the unit may ignore unknown directives), but the script hard-fails expecting all 16 keys (e.g., PrivateMounts=yes). To avoid false negatives across Ubuntu/Azure Linux/Flatcar variants, detect whether each property is supported before asserting, or validate by parsing the unit file instead of requiring systemctl show to return every property.

Suggested change

	# Fetch all security-related properties in batches (systemctl has limits)
	SECURITY_PROPS_1=$(systemctl show localdns-exporter@.service \
	--property=DynamicUser,PrivateTmp,ProtectSystem,ProtectHome,ReadOnlyPaths,NoNewPrivileges \
	2>/dev/null || true)
	SECURITY_PROPS_2=$(systemctl show localdns-exporter@.service \
	--property=ProtectKernelTunables,ProtectKernelModules,ProtectControlGroups,RestrictAddressFamilies \
	2>/dev/null || true)
	SECURITY_PROPS_3=$(systemctl show localdns-exporter@.service \
	--property=RestrictNamespaces,LockPersonality,RestrictRealtime,RestrictSUIDSGID,RemoveIPC,PrivateMounts \
	2>/dev/null || true)
	SECURITY_PROPS="$SECURITY_PROPS_1
	$SECURITY_PROPS_2
	$SECURITY_PROPS_3"
	echo " Retrieved security properties:"
	echo "$SECURITY_PROPS" | sed 's/^/ /'
	echo ""
	# Check all 16 security directives
	# Fetch all security-related properties individually so we can detect unsupported ones
	SECURITY_PROPERTIES=(
	DynamicUser
	PrivateTmp
	ProtectSystem
	ProtectHome
	ReadOnlyPaths
	NoNewPrivileges
	ProtectKernelTunables
	ProtectKernelModules
	ProtectControlGroups
	RestrictAddressFamilies
	RestrictNamespaces
	LockPersonality
	RestrictRealtime
	RestrictSUIDSGID
	RemoveIPC
	PrivateMounts
	)
	SECURITY_PROPS=""
	SUPPORTED_SECURITY_PROPS=()
	UNSUPPORTED_SECURITY_PROPS=()
	for prop in "${SECURITY_PROPERTIES[@]}"; do
	# systemctl show prints "<Prop>=<value>" when the property is known; for
	# unknown/unsupported properties it typically prints nothing.
	value="$(systemctl show localdns-exporter@.service --property="$prop" 2>/dev/null || true)"
	# Normalize whitespace and ignore empty output (treated as unsupported).
	if [[ -n "${value//[[:space:]]/}" ]]; then
	SECURITY_PROPS+="${value}"$'\n'
	SUPPORTED_SECURITY_PROPS+=("$prop")
	else
	UNSUPPORTED_SECURITY_PROPS+=("$prop")
	fi
	done
	echo " Retrieved security properties (supported on this systemd version):"
	echo "$SECURITY_PROPS" | sed 's/^/ /'
	if ((${#UNSUPPORTED_SECURITY_PROPS[@]} > 0)); then
	echo ""
	echo " Note: the following security directives are not supported by this systemd version and will be skipped:"
	for prop in "${UNSUPPORTED_SECURITY_PROPS[@]}"; do
	echo " - $prop"
	done
	fi
	echo ""
	# Check all 16 security directives (only those supported will have entries in SECURITY_PROPS)

[Copilot]

Pull request overview

Copilot reviewed 40 out of 89 changed files in this pull request and generated 3 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

The awk range end regex ^} only matches a closing brace at column 1. If the Corefile closes blocks with indentation (e.g., }), the range won’t terminate and may capture subsequent server blocks, producing incorrect forward IP metrics. Use an end pattern that tolerates whitespace (e.g., ^[[:space:]]*}) for both VnetDNS and KubeDNS ranges.

Suggested change

	vnetdns_ips=($(awk '/bind 169.254.10.10/,/^}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {for(i=3; i<=NF; i++) if ($i ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/) print $i}'))
	kubedns_ips=($(awk '/bind 169.254.10.11/,/^}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {for(i=3; i<=NF; i++) if ($i ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/) print $i}'))
	vnetdns_ips=($(awk '/bind 169.254.10.10/,/^[[:space:]]*}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {for(i=3; i<=NF; i++) if ($i ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/) print $i}'))
	kubedns_ips=($(awk '/bind 169.254.10.11/,/^[[:space:]]*}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {for(i=3; i<=NF; i++) if ($i ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/) print $i}'))

[Copilot]

Pull request overview

Copilot reviewed 40 out of 89 changed files in this pull request and generated 1 comment.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

The runtime check that the exporter worker has no TCP/UDP sockets will fail if the service instance is still running during the scrape: the accepted connection from localdns-exporter.socket is an AF_INET TCP socket that will appear in ss -tupn and is expected to be held open as stdin/stdout. Consider adjusting this assertion to allow the single inherited scrape socket (or dropping this process-level check and relying on the unit's RestrictAddressFamilies= configuration validation).

[Copilot]

Pull request overview

Copilot reviewed 41 out of 89 changed files in this pull request and generated no new comments.

---

You can also share your feedback on Copilot code review. Take the survey.