Skip to content

Add authenticated host capabilities endpoint - #338

Open
rgarcia wants to merge 11 commits into
mainfrom
oss/host-capabilities-safe-diagnostics
Open

Add authenticated host capabilities endpoint#338
rgarcia wants to merge 11 commits into
mainfrom
oss/host-capabilities-safe-diagnostics

Conversation

@rgarcia

@rgarcia rgarcia commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What

Adds an authenticated, machine-readable capability contract so clients (e.g. deployment tooling) can discover what a Hypeman host can do without hard-coding hypervisor knowledge.

GET /capabilities (bearer-auth, resource:read scope)

Reports:

  • server: build version (git revision) and API contract version (from the embedded OpenAPI document)
  • host: OS and architecture
  • runtimes: every runtime this server build supports on this host platform, each with an available boolean (launch prerequisites verified — e.g. qemu requires a runnable system-installed QEMU binary and the host /dev/vhost-vsock device, neither of which hypeman requires at startup) and its own stable per-runtime feature IDs (snapshots, standby, fork, pause, hotplug-memory, balloon-control, vsock, gpu-passthrough, disk-io-limit, disk-resize). Hosts commonly support several at once — e.g. cloud-hypervisor, firecracker, qemu, and qemu-microvm on linux/amd64.
  • default_runtime: the configured default's identity plus an available boolean matching its runtimes[] entry, so clients can verify ordinary (runtime-unspecified) launches are backed by a launchable runtime
  • network: model (bridge/nat), guest-visible host gateway and subnet (omitted — never empty strings — when no default network has resolved), and guest-to-guest reachability
  • images: runnable image platforms (incl. Rosetta-emulated linux/amd64 on Apple Silicon macOS only when Rosetta is currently installed, probed via the same Virtualization.framework availability check the vz-shim enforces at launch) and the default platform
  • features: server-level feature IDs (instances, images, builds, volumes, ingress, exec, logs, plus devices on Linux and rosetta-emulation on Apple Silicon hosts with Rosetta currently installed), kept distinct from per-runtime features

Design: the capability registry is the single source of truth

There is no hand-maintained "supported runtimes" switch and no handler-owned feature mapping:

  • Platform-gated registration, launch-checked availability. Each backend registers via hypervisor.RegisterRuntime only where it can genuinely launch VMs: cloud-hypervisor/firecracker/qemu register from //go:build linux files (they require KVM / kernel AF_VSOCK), vz from its darwin-only package, and qemu-microvm only where its x86 board resolves (linux/amd64) — using the same machine-type resolver that gates launches. Registration means supported by this build; an optional per-runtime LaunchCheck determines available: QEMU resolves the binary with the same lookup launches use, verifies it actually executes and reports a parseable version (the same --version probe ResolveVersion persists on every cold start), and checks the host /dev/vhost-vsock device that every instance launch needs (each instance gets a nonzero vsock CID, so QEMU always attaches a vhost-vsock device); firecracker validates an active hypervisor.firecracker_binary_path override (which takes precedence over the embedded binaries on every launch) with the same executable check the launch path applies. The registry's keys therefore are the host-supported runtime set, and new runtimes appear in the endpoint automatically.
  • Effective, not init-frozen, capabilities. Registrations carry a capability resolver evaluated on every registry read, so configuration applied after package init is reflected: pinning cloud_hypervisor_default_version: v49.0 stops advertising the v51.1-only disk-resize feature, and installing a missing QEMU binary flips availability without a restart.
  • Deterministic enumeration. New hypervisor.RegisteredRuntimes() returns value copies sorted by type name, resolved at read time.
  • Feature derivation lives beside the struct. hypervisor.Capabilities.FeatureIDs() (with SupportsStandby() = snapshot ∧ pause) means adding a capability touches one package plus the OpenAPI schema — the HTTP handler never changes. Fork is an explicit SupportsFork capability — independent of snapshot support in both directions: a snapshot-capable backend may reject PrepareFork with ErrNotSupported, and a backend can fork a stopped source (a disk clone, no machine-state snapshot involved) without snapshot support at all. The fork feature promises the stopped-source fork; forking a standby or running source restores/creates snapshots and additionally requires the standby feature — the documented client gating contract.
  • Concurrency-safe registry. RegisterRuntime is public API, so custom backends can register while capability requests enumerate the registry. All registry access is guarded by an RWMutex; readers snapshot the map under the lock and resolve capability/launch-check callbacks only after releasing it, so callbacks may be slow or re-enter the registry (both pinned by tests, including a -race register/enumerate reproducer).
  • Bounded, coalesced QEMU probing. qemu --version runs under a 5s context in its own process group, and the whole group is SIGKILLed on every completion path — not only context cancellation — so neither a hung binary (Cancel path) nor a wrapper that exits early while a background descendant holds the output pipe (WaitDelay path, where Cancel never fires) can wedge requests or leak subprocesses across cache refreshes. Both QEMU board registrations share one short-TTL (1s) single-flight cache: a registry read and concurrent capability requests execute at most one probe per window, while installing QEMU or loading vhost_vsock still flips availability without a restart.
  • One canonical representation. Per-runtime features are reported only as feature-ID lists; there are no parallel booleans to drift out of sync.

Runtime-derived macOS capabilities

vz snapshot/standby support was a static runtime.GOARCH == "arm64" check, which overstates support on macOS 13 (VZ save/restore requires macOS 14+). It is now probed at runtime (kern.osproductversion), and a failed probe reports no support rather than guessing. On macOS 13, vz truthfully omits snapshots/standby while still advertising fork: a stopped-source fork clones disks with no machine-state save/restore involved (Starter.PrepareFork succeeds without a snapshot), so gating it on the probe would hide a valid operation. Hot-source forks (standby/running) require the standby feature per the fork contract above — and macOS 13 hosts cannot have standby sources in the first place.

Acceptance criteria mapping

  • Authenticated machine-readable capability endpoint: version, host OS/arch, per-runtime availability + features for every supported runtime, default-runtime identity/availability, network model/gateway, image platforms, stable feature IDs
  • Runtime-derived capabilities do not overstate macOS snapshot/standby support, QEMU launchability, or configured Cloud Hypervisor version features
  • Tests cover Linux/macOS registration boundaries, launch-check availability, config-aware capability resolution (v49/v51 boundary), deterministic enumeration, per-runtime independence, unavailable defaults, gateway absence/failure, version serialization
  • Generic to Hypeman — no downstream-product logic, no installer work

Tests

  • cmd/api/api/capabilities_test.go — handler (version serialization, host identity, registry-driven runtimes with per-runtime availability, per-runtime feature independence incl. qemu-microvm, default availability tracks the launch check, default-not-available, gateway omission, network failure → 500, typed network-model mapping, server features)
  • cmd/api/api/capabilities_{linux,darwin}_test.go — build-tagged registration boundary pins (exact registry contents per platform, incl. qemu-microvm only on linux/amd64 and vz-only on macOS)
  • lib/hypervisor/features_test.go — FeatureIDs matrix (zero/full/partial capability sets, internal hints excluded, standby semantics, snapshot-without-fork), enumeration semantics (per-read capability resolution, launch checks gating availability and re-evaluated per read) tested against a local map so the global registry is never mutated
  • lib/hypervisor/cloudhypervisor/register_linux_test.go — registered capabilities track the configured default version across the v49/v51 disk-resize boundary; fork advertised on every supported version
  • lib/hypervisor/qemu/register_linux_test.go — qemu-microvm registration tracks the launch-gating board resolver; qemu/qemu-microvm availability tracks the full launch-prerequisite check, with unavailable-case coverage (missing/non-executable/broken/unparseable-version binary, missing vhost-vsock device) via fake binaries and device paths
  • lib/hypervisor/vz/save_restore_support_test.go — macOS 13/14/15/26 boundary matrix; darwin-only test proves advertised snapshot/standby capabilities track the probed OS version while fork stays advertised on macOS 13 (stopped-source forks need no save/restore)
  • lib/hypervisor/registry_concurrency_test.go — concurrent register/enumerate under -race (crashes with "concurrent map iteration and map write" before the RWMutex; unique test-only types so production registrations are never mutated) and a reentrancy test proving callbacks run outside the registry lock
  • lib/hypervisor/qemu/launch_check_cache_test.go — deterministic single-flight/TTL cache tests on private instances (never the package global): concurrent-caller coalescing onto one probe, expiry with a fake clock (cached failures re-probed after TTL so repaired prerequisites surface), explicit Invalidate() seam; register_linux_test.go adds the hung-binary deadline case and pins both boards sharing the production cache; process_test.go adds TestVersionFromBinaryKillsProcessGroupOnTimeout, a wrapper binary that spawns a descendant (sleep 60 & wait) and asserts both the wrapper and the descendant are killed when the probe's context is cancelled — the test cancels only once the script has demonstrably hung with both pids recorded, so slow script startup (e.g. macOS first-exec scanning) can never race the kill (fails without the process-group kill) — and TestVersionFromBinaryKillsDescendantsWhenWrapperExitsFirst, where the wrapper prints a valid version, backgrounds sleep 60, and exits before the deadline: the probe completes via WaitDelay/ErrWaitDelay and the descendant must still be reaped (fails without the completion-path group kill). All tests that exec just-written scripts retry the well-known fork/exec ETXTBSY race so parallel test subprocess spawns can't flake them
  • lib/network/default_network_test.go — typed model, guest-to-guest rules

Ran locally (Linux/KVM): go build ./..., go vet ./..., gofmt clean; make oapi-generate drift-free; full go test ./lib/... ./cmd/... — all packages pass except pre-existing lib/instances network-integration tests and one env-dependent TestBuildEnv, all verified to fail identically on origin/main in this sandbox (no NET_ADMIN). GOOS=darwin GOARCH=arm64 go build/go vet clean for cmd/api/api, lib/network, lib/instances, lib/hypervisor/..., lib/scopes.

Risks

  • Additive endpoint, and the public Go API stays source-compatible: hypervisor.RegisterCapabilities remains as a deprecated wrapper over RegisterRuntime (static set, no launch check — the old semantics), and instances.Manager is unchanged; the handler type-asserts a narrow DefaultHypervisor() accessor on the concrete manager instead; when a wrapper hides that method, it falls back to the configured hypervisor.default (the value the wrapped manager was constructed from), normalizing only an empty value to the same cloud-hypervisor default lib/instances applies. Capability registration moving to platform-gated files means hypervisor.CapabilitiesForType now reports ok=false for runtimes that cannot run on the host — the internal callers consult it only for instances that exist locally, which can only be of launchable types. Capability resolvers and launch checks run per registry read (cheap: struct construction; QEMU additionally runs a context-bounded qemu --version behind a shared 1s single-flight cache and stats /dev/vhost-vsock; the probe runs in its own process group and the group is SIGKILLed on every completion path — cancellation and normal/WaitDelay completion alike — so neither a wedged binary nor a wrapper that exits leaving background descendants can orphan processes). Registry reads snapshot under an RWMutex and never hold the lock across callbacks, so late RegisterRuntime calls are safe.
  • The generated Go client gains GetCapabilities types/client method; stainless.yaml maps the capabilities resource and all Capabilities* models so Stainless SDKs include the endpoint. CapabilitiesRuntime gains a required available field (added before any release ships the endpoint).
  • Cloud Hypervisor's client-side Capabilities() now also resolves from the configured default version instead of the compile-time default; the only version-varying flag (SupportsDiskResize) is not consumed by any runtime gate today, so behavior is unchanged.

Review

Please run Cursor Bugbot review on this PR. (@cursor)


Note

Medium Risk
Large cross-cutting change (registry, QEMU subprocess probes, macOS capability gating) but the endpoint is additive; main risk is misreported availability if launch checks drift from actual launch paths.

Overview
Adds GET /capabilities (bearer auth, resource:read) so clients can discover what a host can run without hard-coding hypervisor behavior. The handler reports server/API versions, host OS/arch, every platform-registered runtime with available (launch prerequisites) and stable feature IDs, default runtime identity/availability, network model/gateway/subnet/guest-to-guest, image platforms (including Rosetta-gated linux/amd64), and server-level features.

Hypervisor layer: static init-time capability maps are replaced by a mutex-protected RegisterRuntime registry with per-read capability resolvers and optional LaunchCheck (QEMU binary + /dev/vhost-vsock, Firecracker custom binary path, etc.). Capabilities.FeatureIDs() and explicit SupportsFork drive the API contract; vz snapshot/standby now follow macOS 14+ probing instead of arm64-only; QEMU version probes use process-group kills and a short-TTL coalesced cache.

Supporting changes: typed network.Model for bridge vs NAT, optional DefaultHypervisor() on the concrete instance manager (type-asserted by the handler), OpenAPI/oapi/Stainless/scopes updates, and broad tests for registration boundaries, concurrency, and launch-check truthfulness.

Reviewed by Cursor Bugbot for commit d7fba22. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

✱ Stainless preview builds for hypeman

This PR will update the hypeman SDKs with the following commit message.

feat: Add host capabilities endpoint and secret-safe diagnostics

Edit this comment to update it. It will appear in the SDK's changelogs.

hypeman-openapi studio · code · diff

Your SDK build had at least one new note diagnostic, which is a regression from the base state.
generate ✅

New diagnostics (1 note)
💡 Endpoint/NotConfigured: Skipped endpoint because it's not in your Stainless config: `get /capabilities`
hypeman-typescript studio · code · diff

Your SDK build had at least one new note diagnostic, which is a regression from the base state.
generate ✅build ⏭️ (prev: build ✅) → lint ⏭️ (prev: lint ❗) → test ✅

New diagnostics (1 note)
💡 Endpoint/NotConfigured: Skipped endpoint because it's not in your Stainless config: `get /capabilities`
hypeman-go studio · code · diff

Your SDK build had at least one new note diagnostic, which is a regression from the base state.
generate ✅build ⏭️ (prev: build ✅) → lint ✅test ✅

go get github.com/stainless-sdks/hypeman-go@1a7b83c08d26cea2ee9e97a19c959de911ebb196
New diagnostics (1 note)
💡 Endpoint/NotConfigured: Skipped endpoint because it's not in your Stainless config: `get /capabilities`

This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
If you push custom code to the preview branch, re-run this workflow to update the comment.
Last updated: 2026-08-04 01:21:02 UTC

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Build config perms not enforced
    • writeBuildConfig now explicitly chmods config.json to 0600 after every write so rewrites of legacy 0644 files tighten permissions.
  • ✅ Fixed: Capabilities ignore platform support
    • GetCapabilities now zeros runtime-derived booleans/features when the default runtime is not in the host-supported runtime list, preventing unsupported runtime overreporting.

Create PR

Or push these changes by commenting:

@cursor push 2ff8424299
Preview (2ff8424299)
diff --git a/cmd/api/api/capabilities.go b/cmd/api/api/capabilities.go
--- a/cmd/api/api/capabilities.go
+++ b/cmd/api/api/capabilities.go
@@ -55,12 +55,14 @@
 	if s.InstanceManager != nil {
 		defaultRuntime = s.InstanceManager.DefaultHypervisor()
 	}
-	caps, capsKnown := hypervisor.CapabilitiesForType(defaultRuntime)
+	supported := supportedRuntimes(runtime.GOOS)
+	caps, capsKnown := capabilitiesForDefaultRuntime(defaultRuntime, supported)
 	if !capsKnown {
-		// The configured default runtime is not available on this platform;
+		// The configured default runtime is not usable on this host;
 		// report zeroed features rather than guessing.
-		log.WarnContext(ctx, "default runtime has no registered capabilities on this host",
-			"runtime", string(defaultRuntime))
+		log.WarnContext(ctx, "default runtime has no usable capabilities on this host",
+			"runtime", string(defaultRuntime),
+			"supported", supported)
 	}
 
 	emulation := emulationSupported(runtime.GOOS, runtime.GOARCH, defaultRuntime)
@@ -85,7 +87,7 @@
 		},
 		Runtime: oapi.CapabilitiesRuntime{
 			Default:        string(defaultRuntime),
-			Supported:      supportedRuntimes(runtime.GOOS),
+			Supported:      supported,
 			Snapshot:       caps.SupportsSnapshot,
 			Standby:        standbySupported(caps),
 			Pause:          caps.SupportsPause,
@@ -157,6 +159,23 @@
 	}
 }
 
+func capabilitiesForDefaultRuntime(defaultRuntime hypervisor.Type, supported []string) (hypervisor.Capabilities, bool) {
+	if !runtimeSupported(defaultRuntime, supported) {
+		return hypervisor.Capabilities{}, false
+	}
+	return hypervisor.CapabilitiesForType(defaultRuntime)
+}
+
+func runtimeSupported(defaultRuntime hypervisor.Type, supported []string) bool {
+	defaultRuntimeName := string(defaultRuntime)
+	for _, runtimeName := range supported {
+		if runtimeName == defaultRuntimeName {
+			return true
+		}
+	}
+	return false
+}
+
 // emulationSupported reports whether the host can boot images built for the
 // other CPU architecture. This mirrors the create-path rule for attaching
 // the Rosetta share: vz on Apple Silicon macOS.

diff --git a/cmd/api/api/capabilities_test.go b/cmd/api/api/capabilities_test.go
--- a/cmd/api/api/capabilities_test.go
+++ b/cmd/api/api/capabilities_test.go
@@ -111,6 +111,19 @@
 	require.Equal(t, []string{"vz"}, supportedRuntimes("darwin"))
 }
 
+func TestRuntimeSupported(t *testing.T) {
+	t.Parallel()
+	require.True(t, runtimeSupported(hypervisor.TypeVZ, supportedRuntimes("darwin")))
+	require.False(t, runtimeSupported(hypervisor.TypeCloudHypervisor, supportedRuntimes("darwin")))
+}
+
+func TestCapabilitiesForDefaultRuntime_IgnoresUnsupportedRuntime(t *testing.T) {
+	t.Parallel()
+	caps, ok := capabilitiesForDefaultRuntime(hypervisor.TypeCloudHypervisor, supportedRuntimes("darwin"))
+	require.False(t, ok)
+	require.Equal(t, hypervisor.Capabilities{}, caps)
+}
+
 func TestEmulationSupported(t *testing.T) {
 	t.Parallel()
 	require.True(t, emulationSupported("darwin", "arm64", hypervisor.TypeVZ))

diff --git a/lib/builds/storage.go b/lib/builds/storage.go
--- a/lib/builds/storage.go
+++ b/lib/builds/storage.go
@@ -237,6 +237,9 @@
 	if err := os.WriteFile(configPath, data, 0600); err != nil {
 		return fmt.Errorf("write build config: %w", err)
 	}
+	if err := os.Chmod(configPath, 0600); err != nil {
+		return fmt.Errorf("chmod build config: %w", err)
+	}
 
 	return nil
 }

diff --git a/lib/builds/storage_test.go b/lib/builds/storage_test.go
--- a/lib/builds/storage_test.go
+++ b/lib/builds/storage_test.go
@@ -36,3 +36,34 @@
 	loaded.Tags["team"] = "mutated"
 	require.Equal(t, "backend", build.Tags["team"])
 }
+
+func TestWriteBuildConfig_UsesOwnerOnlyPermissions(t *testing.T) {
+	tempDir := t.TempDir()
+	p := paths.New(tempDir)
+	id := "build-config-1"
+
+	cfg := &BuildConfig{RegistryToken: "secret-token"}
+	require.NoError(t, writeBuildConfig(p, id, cfg))
+
+	info, err := os.Stat(p.BuildConfig(id))
+	require.NoError(t, err)
+	require.Equal(t, os.FileMode(0600), info.Mode().Perm())
+}
+
+func TestWriteBuildConfig_TightensLegacyPermissions(t *testing.T) {
+	tempDir := t.TempDir()
+	p := paths.New(tempDir)
+	id := "build-config-legacy"
+
+	require.NoError(t, os.MkdirAll(p.BuildDir(id), 0755))
+	configPath := p.BuildConfig(id)
+	require.NoError(t, os.WriteFile(configPath, []byte(`{"registry_token":"old-token"}`), 0644))
+	require.NoError(t, os.Chmod(configPath, 0644))
+
+	cfg := &BuildConfig{RegistryToken: "new-token"}
+	require.NoError(t, writeBuildConfig(p, id, cfg))
+
+	info, err := os.Stat(configPath)
+	require.NoError(t, err)
+	require.Equal(t, os.FileMode(0600), info.Mode().Perm())
+}

You can send follow-ups to the cloud agent here.

Comment thread lib/builds/storage.go Outdated
Comment thread cmd/api/api/capabilities.go Outdated
rgarcia added a commit that referenced this pull request Aug 3, 2026
os.WriteFile does not change permissions of an existing file, so token
refresh rewrites of build config.json (which carries the registry push
token) would leave legacy 0644 files world-readable. Chmod after every
write and tighten legacy build config/metadata files at manager startup.

Addresses Cursor Bugbot feedback on PR #338.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Sentinel env still counts as update
    • The update path now strips redaction sentinels from req.Env before validation and control-flow checks, and a regression test confirms sentinel-only env maps no longer trigger env-update restrictions on stopped instances.

Create PR

Or push these changes by commenting:

@cursor push 40e329ae56
Preview (40e329ae56)
diff --git a/lib/instances/update.go b/lib/instances/update.go
--- a/lib/instances/update.go
+++ b/lib/instances/update.go
@@ -49,6 +49,7 @@
 		}
 		req.RestartPolicy = normalizedRestartPolicy
 	}
+	req.Env = mergeEnvUpdate(nil, req.Env)
 
 	if err := validateUpdateInstanceRequest(meta, req); err != nil {
 		return nil, err

diff --git a/lib/instances/update_test.go b/lib/instances/update_test.go
--- a/lib/instances/update_test.go
+++ b/lib/instances/update_test.go
@@ -9,6 +9,7 @@
 	"github.com/kernel/hypeman/lib/autostandby"
 	"github.com/kernel/hypeman/lib/egressproxy"
 	"github.com/kernel/hypeman/lib/healthcheck"
+	"github.com/kernel/hypeman/lib/redact"
 	snapshotstore "github.com/kernel/hypeman/lib/snapshot"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -401,3 +402,56 @@
 		t.Fatal("timed out waiting for lifecycle update event")
 	}
 }
+
+func TestManagerUpdateInstanceIgnoresSentinelOnlyEnvUpdateOnStoppedInstance(t *testing.T) {
+	t.Parallel()
+
+	manager, _ := setupTestManager(t)
+	id := "inst-update-sentinel-noop"
+	require.NoError(t, manager.ensureDirectories(id))
+	meta := &metadata{
+		StoredMetadata: StoredMetadata{
+			Id:         id,
+			Name:       id,
+			CreatedAt:  time.Now(),
+			DataDir:    manager.paths.InstanceDir(id),
+			SocketPath: manager.paths.InstanceSocket(id, "cloud-hypervisor.sock"),
+			NetworkEgress: &NetworkEgressPolicy{
+				Enabled: true,
+			},
+			Credentials: map[string]CredentialPolicy{
+				"OUTBOUND_OPENAI_KEY": {
+					Source: CredentialSource{Env: "OUTBOUND_OPENAI_KEY"},
+				},
+			},
+			Env: map[string]string{
+				"OUTBOUND_OPENAI_KEY": "real-secret",
+			},
+			AutoStandby: &autostandby.Policy{
+				Enabled:     false,
+				IdleTimeout: "5m0s",
+			},
+		},
+	}
+	require.NoError(t, manager.saveMetadata(meta))
+
+	updated, err := manager.UpdateInstance(context.Background(), id, UpdateInstanceRequest{
+		Env: map[string]string{
+			"OUTBOUND_OPENAI_KEY": redact.Sentinel,
+		},
+		AutoStandby: &autostandby.Policy{
+			Enabled:     true,
+			IdleTimeout: "10m",
+		},
+	})
+	require.NoError(t, err)
+	require.NotNil(t, updated)
+	require.NotNil(t, updated.AutoStandby)
+	assert.True(t, updated.AutoStandby.Enabled)
+	assert.Equal(t, "10m0s", updated.AutoStandby.IdleTimeout)
+
+	saved, err := manager.loadMetadata(id)
+	require.NoError(t, err)
+	require.NotNil(t, saved)
+	assert.Equal(t, "real-secret", saved.Env["OUTBOUND_OPENAI_KEY"])
+}

You can send follow-ups to the cloud agent here.

Comment thread lib/instances/update.go
rgarcia added a commit that referenced this pull request Aug 3, 2026
Strip redaction sentinels at the top of updateInstance so a
read-modify-write that round-trips only redacted env values does not
count as an env mutation: it no longer requires the instance to be
running and no longer routes through the egress-proxy credential path.

Addresses Cursor Bugbot feedback on PR #338.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Legacy config disks stay world-readable
    • Added a startup permission sweep for existing config.ext4 files and wired it into manager initialization so legacy disks are tightened to 0600 immediately after upgrade.

Create PR

Or push these changes by commenting:

@cursor push cf40507f85
Preview (cf40507f85)
diff --git a/lib/instances/manager.go b/lib/instances/manager.go
--- a/lib/instances/manager.go
+++ b/lib/instances/manager.go
@@ -311,6 +311,9 @@
 	// Restrict permissions on metadata written by older versions (may be 0644
 	// and contains env values / credential bindings).
 	m.tightenMetadataPermissions()
+	// Restrict permissions on guest config disks written by older versions
+	// (may be 0644 and embed config.json with env values).
+	m.tightenConfigDiskPermissions()
 
 	return m, nil
 }

diff --git a/lib/instances/metadata_permissions_test.go b/lib/instances/metadata_permissions_test.go
--- a/lib/instances/metadata_permissions_test.go
+++ b/lib/instances/metadata_permissions_test.go
@@ -86,6 +86,26 @@
 		"legacy 0644 metadata must be tightened to 0600 at startup")
 }
 
+// TestManagerTightensLegacyConfigDiskPermissions proves the startup sweep
+// upgrades config disks written by older versions (mode 0644) to 0600.
+func TestManagerTightensLegacyConfigDiskPermissions(t *testing.T) {
+	t.Parallel()
+	dataDir := t.TempDir()
+	p := paths.New(dataDir)
+	id := "inst-config-disk-legacy"
+
+	// Simulate a legacy config disk written with world-readable permissions.
+	require.NoError(t, os.MkdirAll(p.InstanceDir(id), 0755))
+	require.NoError(t, os.WriteFile(p.InstanceConfigDisk(id), []byte("ext4-bytes-placeholder"), 0644))
+
+	newPermTestManager(t, dataDir)
+
+	info, err := os.Stat(p.InstanceConfigDisk(id))
+	require.NoError(t, err)
+	require.Equal(t, os.FileMode(0600), info.Mode().Perm(),
+		"legacy 0644 config disks must be tightened to 0600 at startup")
+}
+
 // TestMergeEnvUpdateSkipsRedactionSentinel proves a redacted read response
 // round-tripped into an env update cannot clobber real secret values.
 func TestMergeEnvUpdateSkipsRedactionSentinel(t *testing.T) {

diff --git a/lib/instances/storage.go b/lib/instances/storage.go
--- a/lib/instances/storage.go
+++ b/lib/instances/storage.go
@@ -154,6 +154,33 @@
 	}
 }
 
+// tightenConfigDiskPermissions restricts existing guest config disk files to
+// owner-only access. Disks written before restrictive permissions were
+// introduced may be mode 0644; they embed config.json with environment values.
+// Best-effort: individual failures are logged, not fatal.
+func (m *manager) tightenConfigDiskPermissions() {
+	log := logger.FromContext(context.Background())
+	entries, err := os.ReadDir(m.paths.GuestsDir())
+	if err != nil {
+		return // no guests directory yet
+	}
+	for _, entry := range entries {
+		if !entry.IsDir() {
+			continue
+		}
+		configDiskPath := m.paths.InstanceConfigDisk(entry.Name())
+		info, err := os.Stat(configDiskPath)
+		if err != nil {
+			continue
+		}
+		if info.Mode().Perm() != 0600 {
+			if err := os.Chmod(configDiskPath, 0600); err != nil {
+				log.Warn("failed to tighten instance config disk permissions", "path", configDiskPath, "error", err)
+			}
+		}
+	}
+}
+
 // createOverlayDisk creates a sparse overlay disk for the instance
 func (m *manager) createOverlayDisk(id string, sizeBytes int64) error {
 	overlayPath := m.paths.InstanceOverlay(id)

You can send follow-ups to the cloud agent here.

Comment thread lib/instances/configdisk.go Outdated
rgarcia added a commit that referenced this pull request Aug 3, 2026
Legacy config.ext4 files embed env values and could remain 0644 after
upgrade until the instance is recreated. Extend the startup permission
sweep to cover them alongside metadata.json.

Addresses Cursor Bugbot feedback on PR #338.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Wrong macOS capabilities gateway
    • DefaultNetwork now always resolves live NAT network state on macOS so /capabilities reports the guest-visible gateway and subnet instead of cached Linux config values.

Create PR

Or push these changes by commenting:

@cursor push 511c951880
Preview (511c951880)
diff --git a/lib/network/default_network_test.go b/lib/network/default_network_test.go
--- a/lib/network/default_network_test.go
+++ b/lib/network/default_network_test.go
@@ -9,10 +9,9 @@
 	"github.com/stretchr/testify/require"
 )
 
-// TestDefaultNetworkPrefersInitializedNetwork proves DefaultNetwork returns
-// the effective default network established at Initialize time, including the
-// guest-visible gateway, without touching host kernel state.
-func TestDefaultNetworkPrefersInitializedNetwork(t *testing.T) {
+// TestDefaultNetworkReturnsGuestVisibleValues proves DefaultNetwork reports
+// guest-visible gateway/subnet details for the active host networking model.
+func TestDefaultNetworkReturnsGuestVisibleValues(t *testing.T) {
 	t.Parallel()
 	cfg := &config.Config{}
 	m := NewManager(paths.New(t.TempDir()), cfg, nil).(*manager)
@@ -28,14 +27,23 @@
 
 	nw, err := m.DefaultNetwork(context.Background())
 	require.NoError(t, err)
-	require.Equal(t, "10.100.0.1", nw.Gateway)
-	require.Equal(t, "10.100.0.0/16", nw.Subnet)
+	if NetworkModel() == "nat" {
+		require.Equal(t, "192.168.64.1", nw.Gateway)
+		require.Equal(t, "192.168.64.0/24", nw.Subnet)
+	} else {
+		require.Equal(t, "10.100.0.1", nw.Gateway)
+		require.Equal(t, "10.100.0.0/16", nw.Subnet)
+	}
 
 	// Mutating the returned copy must not affect the cached network.
 	nw.Gateway = " mutated "
 	again, err := m.DefaultNetwork(context.Background())
 	require.NoError(t, err)
-	require.Equal(t, "10.100.0.1", again.Gateway)
+	if NetworkModel() == "nat" {
+		require.Equal(t, "192.168.64.1", again.Gateway)
+	} else {
+		require.Equal(t, "10.100.0.1", again.Gateway)
+	}
 }
 
 func TestGuestToGuestEnabled(t *testing.T) {

diff --git a/lib/network/manager.go b/lib/network/manager.go
--- a/lib/network/manager.go
+++ b/lib/network/manager.go
@@ -167,10 +167,13 @@
 	m.defaultNetwork = cloneNetwork(network)
 }
 
-// DefaultNetwork returns the effective default network. It prefers the
-// network established during Initialize and falls back to querying live
-// host state (kernel bridge on Linux, the vz NAT stub on macOS).
+// DefaultNetwork returns the effective default network. Linux hosts prefer
+// the network established during Initialize, while macOS hosts always query
+// live NAT state because config subnet/gateway settings are not guest-visible.
 func (m *manager) DefaultNetwork(ctx context.Context) (*Network, error) {
+	if NetworkModel() == "nat" {
+		return m.getDefaultNetwork(ctx)
+	}
 	if network := m.cachedDefaultNetwork(); network != nil {
 		return network, nil
 	}

You can send follow-ups to the cloud agent here.

Comment thread lib/network/manager.go
rgarcia added a commit that referenced this pull request Aug 3, 2026
On macOS the configured subnet/gateway are ignored (guests use vz NAT),
so the Initialize-cached network is never guest-visible. DefaultNetwork
now prefers live host state per platform: the config-mirroring cache on
Linux, the vz NAT stub (192.168.64.1, 192.168.64.0/24) on macOS. This is
reporting-only; allocation behavior is unchanged.

Addresses Cursor Bugbot feedback on PR #338.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Build metadata perms not forced
    • Added an explicit os.Chmod(tempPath, 0600) in writeMetadata so pre-existing temp files cannot retain broader permissions before rename.

Create PR

Or push these changes by commenting:

@cursor push 7afceaf739
Preview (7afceaf739)
diff --git a/lib/builds/storage.go b/lib/builds/storage.go
--- a/lib/builds/storage.go
+++ b/lib/builds/storage.go
@@ -64,6 +64,10 @@
 	if err := os.WriteFile(tempPath, data, 0600); err != nil {
 		return fmt.Errorf("write temp metadata: %w", err)
 	}
+	// WriteFile does not change permissions of an existing file.
+	if err := os.Chmod(tempPath, 0600); err != nil {
+		return fmt.Errorf("chmod temp metadata: %w", err)
+	}
 
 	finalPath := p.BuildMetadata(meta.ID)
 	if err := os.Rename(tempPath, finalPath); err != nil {

You can send follow-ups to the cloud agent here.

Comment thread lib/builds/storage.go Outdated
rgarcia added a commit that referenced this pull request Aug 3, 2026
A leftover 0644 temp file from an older write would otherwise keep its
permissions through WriteFile and be renamed into place world-readable.

Addresses Cursor Bugbot feedback on PR #338.
@rgarcia

rgarcia commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Independent review — round 1 (head 63ca479)

Verdict: satisfied — no blocking or important findings.

Checked against ticket 02 (capabilities endpoint + safe diagnostics) and the planning spec:

  • GET /capabilities is bearer-authenticated (security: bearerAuth in the spec, resource:read scope in lib/scopes), reports server/API version, host OS/arch, effective default-runtime feature booleans, network model + guest-visible gateway/subnet, image platforms (incl. Rosetta emulation), and stable feature IDs — no hypervisor selection required by clients.
  • ✅ macOS snapshot/standby is now runtime-derived (kern.osproductversion, macOS 14+ floor, failed probe ⇒ unsupported), replacing the arch-only check; boundary matrix tests run on any GOOS and a darwin-only test tracks the probed version. capabilities() feeds both the endpoint and the existing standby path.
  • /health (still unauthenticated) and /resources are unchanged and pinned by backward-compat tests.
  • ✅ Env redaction on list/get with include_env=true opt-in, sentinel-safe read-modify-write (withoutRedactionSentinels strips sentinels before validation and merge), Authorization-header canary test on the access logger.
  • ✅ 0600 on instance metadata, config disks, build config/metadata, with startup sweeps for legacy 0644 files; operator config files only warn. Bugbot's five earlier findings (build-config chmod, unusable-default-runtime overstatement, sentinel control-flow, legacy config-disk sweep, macOS NAT gateway, build-metadata temp chmod) were each fixed in follow-up commits and all threads are resolved; its check passed on the latest head.
  • ✅ CI green (test, test-darwin, e2e-install, semgrep, Stainless preview). Change is generic to Hypeman — no downstream-product logic or installer work.

Minor, non-blocking nits (no action required for this round):

  1. Lifecycle mutation responses (create/start/stop/standby/restore/fork/update) still echo plaintext env; only list/get redact. Since include_env=true needs no extra scope this isn't a privilege boundary, but redacting there too (with the same opt-in) would make the default uniform.
  2. An env-only PATCH whose values are all sentinels now returns 400 ("request must include env, …") rather than a no-op. That matches empty-PATCH behavior, but a naive redacted read-modify-write client will see an error; worth a note in the API docs if it bites anyone.

Structured verdict for the workflow: approved.

@rgarcia rgarcia changed the title Add host capabilities endpoint and secret-safe diagnostics Add authenticated host capabilities endpoint Aug 3, 2026
@rgarcia

rgarcia commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Independent narrowing review — round 1 (head 7632089)

Verdict: approved — scope is narrow, no blocking or important findings.

Scope verification (origin/main..7632089, 18 files):

  • ✅ The final diff contains only the capability endpoint and its truthful prerequisites: GET /capabilities (handler, scope mapping, OpenAPI schema + regenerated lib/oapi), runtime-derived vz snapshot/standby gating (kern.osproductversion probe, macOS 14+ floor, failed probe → unsupported), and guest-visible network reporting (DefaultNetwork with preferCachedDefaultNetwork() so the macOS vz NAT stub is authoritative).
  • ✅ Grepped the full diff for include_env, redaction, sentinels, canaries, chmod/permission changes — none present. All tabled hardening (ticket 05) is fully removed: lib/builds/storage.go, lib/instances/update.go, lib/instances/configdisk.go, lib/redact no longer appear in the diff.
  • GET /instances / GET /instances/{id} untouched; the lib/oapi/oapi.go diff has zero non-generated deletions and no instance-type changes — purely additive Capabilities* types + re-encoded embedded spec. Backward-compat tests pin /health and /resources.
  • ✅ Interface additions (instances.Manager.DefaultHypervisor, network.Manager.DefaultNetwork) are minimal accessors; test doubles updated accordingly.

Correctness spot-checks:

  • supportedRuntimes("darwin") == ["vz"] is correct as a runnability floor: qemu/cloud-hypervisor are registered in darwin platformStarters but hypeman's qemu config is accel=kvm-only, so listing only vz avoids overstating.
  • Zeroing runtime features when the configured default runtime can't run on the host (with a warning log) is the right "don't overstate" behavior; standby correctly requires snapshot and pause, and this now also gates the existing standby path via the shared Capabilities().
  • DefaultNetwork returns a clone (mutation-safe, verified by test); Linux falls back to live bridge state when the cache is unset; macOS always queries the NAT stub (192.168.64.1), fixing the earlier Bugbot-flagged wrong-gateway issue.
  • vz version-parse boundary matrix (13/14/14.x/15/26, empty, unparsable) runs on any GOOS; darwin-only test proves advertised caps track the stubbed probe.

CI / Bugbot at head 7632089: test, test-darwin, e2e-install, preview, scan, Socket — all pass. Cursor Bugbot: completed, no issues found. All six earlier Bugbot findings targeted code removed by the narrowing or were fixed (platform-support gating, macOS gateway) and do not apply to the final diff.

Minor (non-blocking):

  • The Stainless preview comment's SDK changelog message still reads "Add host capabilities endpoint and secret-safe diagnostics" — stale from the pre-narrowed scope. Worth editing that comment before merge so SDK changelogs don't advertise removed functionality.

@rgarcia
rgarcia force-pushed the oss/host-capabilities-safe-diagnostics branch from 328e8a7 to a4892cd Compare August 14, 2026 17:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Missing Stainless SDK mapping
    • Added a new capabilities resource in stainless.yaml with get /capabilities plus Capabilities* model mappings so SDK generation includes this endpoint and types.
  • ✅ Fixed: Rosetta gated on default runtime
    • Changed emulation detection to require Apple Silicon macOS with vz in the supported runtime list rather than requiring vz as the configured default runtime, and updated tests accordingly.

Create PR

Or push these changes by commenting:

@cursor push f2fd2aa197
Preview (f2fd2aa197)
diff --git a/cmd/api/api/capabilities.go b/cmd/api/api/capabilities.go
--- a/cmd/api/api/capabilities.go
+++ b/cmd/api/api/capabilities.go
@@ -70,7 +70,7 @@
 		caps = hypervisor.Capabilities{}
 	}
 
-	emulation := emulationSupported(runtime.GOOS, runtime.GOARCH, defaultRuntime)
+	emulation := emulationSupported(runtime.GOOS, runtime.GOARCH, supported)
 
 	networkCaps, err := s.networkCapabilities(ctx)
 	if err != nil {
@@ -167,8 +167,10 @@
 // emulationSupported reports whether the host can boot images built for the
 // other CPU architecture. This mirrors the create-path rule for attaching
 // the Rosetta share: vz on Apple Silicon macOS.
-func emulationSupported(goos, goarch string, defaultRuntime hypervisor.Type) bool {
-	return defaultRuntime == hypervisor.TypeVZ && goos == "darwin" && goarch == "arm64"
+func emulationSupported(goos, goarch string, supported []string) bool {
+	return goos == "darwin" &&
+		goarch == "arm64" &&
+		slices.Contains(supported, string(hypervisor.TypeVZ))
 }
 
 // imagePlatforms returns the image platforms (os/arch) the host can run:

diff --git a/cmd/api/api/capabilities_test.go b/cmd/api/api/capabilities_test.go
--- a/cmd/api/api/capabilities_test.go
+++ b/cmd/api/api/capabilities_test.go
@@ -122,11 +122,11 @@
 
 func TestEmulationSupported(t *testing.T) {
 	t.Parallel()
-	require.True(t, emulationSupported("darwin", "arm64", hypervisor.TypeVZ))
-	require.False(t, emulationSupported("darwin", "amd64", hypervisor.TypeVZ))
-	require.False(t, emulationSupported("darwin", "arm64", hypervisor.TypeCloudHypervisor))
-	require.False(t, emulationSupported("linux", "arm64", hypervisor.TypeVZ))
-	require.False(t, emulationSupported("linux", "amd64", hypervisor.TypeCloudHypervisor))
+	require.True(t, emulationSupported("darwin", "arm64", []string{"vz"}))
+	require.False(t, emulationSupported("darwin", "arm64", []string{"cloud-hypervisor"}))
+	require.False(t, emulationSupported("darwin", "amd64", []string{"vz"}))
+	require.False(t, emulationSupported("linux", "arm64", []string{"vz"}))
+	require.False(t, emulationSupported("linux", "amd64", []string{"cloud-hypervisor"}))
 }
 
 func TestImagePlatforms(t *testing.T) {

diff --git a/stainless.yaml b/stainless.yaml
--- a/stainless.yaml
+++ b/stainless.yaml
@@ -65,6 +65,17 @@
     methods:
       check: get /health
 
+  capabilities:
+    models:
+      capabilities: "#/components/schemas/Capabilities"
+      capabilities_server: "#/components/schemas/CapabilitiesServer"
+      capabilities_host: "#/components/schemas/CapabilitiesHost"
+      capabilities_runtime: "#/components/schemas/CapabilitiesRuntime"
+      capabilities_network: "#/components/schemas/CapabilitiesNetwork"
+      capabilities_images: "#/components/schemas/CapabilitiesImages"
+    methods:
+      get: get /capabilities
+
   images:
     # Configure the models--named types--defined in the resource. Each key in the
     # object is the name of the model and the value is either the name of a schema in

You can send follow-ups to the cloud agent here.

Comment thread openapi.yaml
Comment thread cmd/api/api/capabilities.go
@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

116 files generated at a5737ca (pushed)

go get github.com/kernel/hypeman-go-staging@a5737ca92ce7dbecf7dcd4fc448aae51669f5fed
typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

138 files generated at 5eab05a (pushed)

Diagnostics: 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_768d9nD2-saline-latch
Timestamp2026-08-16T00:26:30.896Z
stlc8413509
Spec hash3958e4a2d109
Config hash162c7c01f833

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-16 00:26:47 UTC

@rgarcia
rgarcia force-pushed the oss/host-capabilities-safe-diagnostics branch from a4892cd to bd2c6e8 Compare August 14, 2026 18:16

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: MicroVM omitted from supported runtimes
    • Added qemu-microvm to Linux supportedRuntimes (and updated the unit test expectation) so default runtime capability checks no longer incorrectly zero MicroVM features.

Create PR

Or push these changes by commenting:

@cursor push c3e22ab24d
Preview (c3e22ab24d)
diff --git a/cmd/api/api/capabilities.go b/cmd/api/api/capabilities.go
--- a/cmd/api/api/capabilities.go
+++ b/cmd/api/api/capabilities.go
@@ -160,6 +160,7 @@
 			string(hypervisor.TypeCloudHypervisor),
 			string(hypervisor.TypeFirecracker),
 			string(hypervisor.TypeQEMU),
+			string(hypervisor.TypeQEMUMicroVM),
 		}
 	}
 }

diff --git a/cmd/api/api/capabilities_test.go b/cmd/api/api/capabilities_test.go
--- a/cmd/api/api/capabilities_test.go
+++ b/cmd/api/api/capabilities_test.go
@@ -115,7 +115,7 @@
 func TestSupportedRuntimes(t *testing.T) {
 	t.Parallel()
 	require.Equal(t,
-		[]string{"cloud-hypervisor", "firecracker", "qemu"},
+		[]string{"cloud-hypervisor", "firecracker", "qemu", "qemu-microvm"},
 		supportedRuntimes("linux"))
 	require.Equal(t, []string{"vz"}, supportedRuntimes("darwin"))
 }

You can send follow-ups to the cloud agent here.

Reviewed by Cursor Bugbot for commit bd2c6e8. Configure here.

Comment thread cmd/api/api/capabilities.go Outdated
@rgarcia
rgarcia requested a review from sjmiller609 August 14, 2026 18:22

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

primary issue: capabilities reported for only one runtime, even though a host can support multiple at once

explore this: how to avoid always updating this code each time capabilities are changed or added?

each hypervisor already registers its Capabilities struct at init via RegisterCapabilities. two fixes make the endpoint self-updating: (1) build-tag the linux hypervisor registrations (vz already is darwin-only), so the registry's keys are the host-supported runtime set — delete the hardcoded supportedRuntimes() switch, and new runtimes appear automatically. (2) move feature-ID derivation onto the struct as caps.FeatureIDs() in lib/hypervisor, next to the field definitions. adding a capability then touches one package: the struct field, its feature ID, and the openapi schema — the endpoint handler never changes.

nits from AI review

  • fork appears in features but has no boolean in runtime — the two surfaces are inconsistent.
  • gateway is schema-required but can be empty string when no default network resolves.
  • linux GuestToGuestEnabled comment says guests can't reach each other; code returns !n.Isolated. contradictory.
  • imagePlatforms's if goarch == "" fallback is dead defensive code — delete it.
  • emulationSupported's supported param is always-true on darwin; simplify to goos/goarch check.
  • NetworkModel() returns bare strings then casts to the oapi enum; return a typed constant.
  • standby description says "on this host" while siblings say "default runtime" — align wording.

- Report every runtime available on the host (registry-driven), each with
  its own feature IDs; retain configured default identity + availability
- Platform-gate backend capability registration (linux backends via
  build-tagged files, vz already darwin-only); qemu-microvm registers only
  where its board resolves, so it appears automatically on linux/amd64
- Add hypervisor.RegisteredRuntimes deterministic enumeration and
  Capabilities.FeatureIDs/SupportsStandby/SupportsFork beside the struct
- One canonical per-runtime representation (feature IDs, no boolean twins)
- Optional network gateway/subnet (never empty-string), typed
  network.Model contract, truthful guest-to-guest docs
- Simplify emulation detection to host os/arch; drop dead imagePlatforms
  fallback; align OpenAPI descriptions; map CapabilitiesDefaultRuntime in
  stainless.yaml
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks Steven — redesigned in f8a1ca5 along the lines you sketched. Point-by-point mapping:

Primary: capabilities for only one runtime. The endpoint now reports runtimes[] — every runtime genuinely available on the host, each with its own features list — plus default_runtime: {name, available} so clients can still verify that ordinary launches are backed by an available runtime. On linux/amd64 that's cloud-hypervisor, firecracker, qemu, and qemu-microvm, each with independent features.

Self-updating, no per-capability handler edits. Exactly as you proposed:

  1. Backend capability registration is now build-tagged (register_linux.go in cloud-hypervisor/firecracker/qemu; vz was already darwin-only), so the registry's keys are the host-supported set. The hardcoded supportedRuntimes() switch is deleted, and qemu-microvm registers automatically where its board resolves (linux/amd64, via the same resolveMachineTypeForPlatform that gates launches) — no handler entry. Enumeration is a new deterministic hypervisor.RegisteredRuntimes() (sorted, value copies).
  2. Feature-ID derivation moved beside the struct: Capabilities.FeatureIDs() in lib/hypervisor, next to the field definitions. Adding a capability touches the struct field, its feature ID, and the OpenAPI schema — the handler never changes.

Nits:

  • fork/boolean inconsistency → the duplicate surface is gone entirely: per-runtime capabilities are reported only as feature-ID lists (no boolean twins to drift). fork derives from Capabilities.SupportsFork() (tracks snapshot support, since fork restores a snapshot of the source) and standby from SupportsStandby() (snapshot ∧ pause — the standby path pauses then snapshots), rather than being assumed from names.
  • required gateway can be empty stringgateway (and subnet) are now optional and omitted when no default network has resolved; they never serialize as empty strings. New test pins this.
  • linux GuestToGuestEnabled comment vs !n.Isolated → comment rewritten to match the code: default networks are provisioned with per-TAP isolation, and the flag is read from the network rather than assumed so a non-isolated network reports truthfully.
  • dead imagePlatforms goarch == "" fallback → deleted; the emulated-arch switch collapsed too (emulation implies arm64→amd64).
  • always-true supported param in emulationSupported → simplified to goos == "darwin" && goarch == "arm64"; the OpenAPI description now also states Rosetta installation itself is verified at instance start, so the platform-eligibility claim isn't overstated.
  • bare-string NetworkModel() castlib/network now returns a typed network.Model (ModelBridge/ModelNAT); the handler maps it onto the oapi enum via an explicit switch, keeping lib/network free of oapi dependencies.
  • "on this host" vs "default runtime" wording → the per-boolean descriptions were replaced wholesale by the uniform per-runtime features description; all values are documented as host-truthful.

Boundary coverage: capabilities_{linux,darwin}_test.go pin the exact registry contents per platform (incl. qemu-microvm only on linux/amd64, vz-only on macOS); enumeration semantics are tested against a local map so tests never mutate the global registry.

@rgarcia rgarcia left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GPT-5.6 capabilities redesign review — round 1 (f8a1ca5)

Verdict: changes requested. The redesign does resolve Steven's primary shape concern: all registered runtimes are emitted deterministically, feature-ID derivation is beside Capabilities, the handler has no runtime list, the OpenAPI/Stainless/generated surfaces are coherent, and Steven's listed nits are addressed. Two important truthfulness gaps remain:

  1. “Available” currently means “registered for this GOOS,” not “this host can launch it.” qemu/register_linux.go registers QEMU unconditionally, while cmd/api/main.go:318-321 explicitly permits startup without a QEMU binary and says that runtime will not work. /capabilities still includes qemu (and qemu-microvm on amd64), and a configured QEMU default reports available: true; the tests pin that overstatement. Availability/default availability need to incorporate launch prerequisites (or expose a distinct supported-vs-available/readiness model) without reintroducing a handler list.

  2. The static init-time capability value is not always the effective runtime capability. Cloud Hypervisor registration snapshots CapabilitiesForVersion(vmm.DefaultVersion) before config is applied, but cloud_hypervisor_default_version can select v49.0; the endpoint then advertises disk-resize from v51.1 even though ordinary v49 launches do not have it. The registry needs a config/runtime-aware capability provider (or equivalent effective resolution), not a value frozen at package init.

Also please make fork explicit in the capability contract rather than defining SupportsFork() as SupportsSnapshot. VMStarter.PrepareFork explicitly allows a snapshot-capable backend to return ErrNotSupported, so the current derivation can advertise fork when the actual fork gate rejects it. That reintroduces the exact boolean/ID drift this redesign is meant to prevent for the next backend.

Checks: complete 95db3fb...f8a1ca5 diff inspected; focused Linux tests + race checks pass; Darwin arm64 API/vz cross-compilation passes; make oapi-generate is drift-free; all real CI, Stainless generation, and current Bugbot pass with no unresolved Bugbot thread.

Address GPT-5.6 review round 1 (three important findings):

1. Runtime availability now reflects launch prerequisites, not just
   platform registration. The registry accepts a per-runtime LaunchCheck;
   QEMU (standard and microvm) probes the same system-binary lookup that
   launches use, since hypeman deliberately starts without it. The
   endpoint gains a required per-runtime "available" field (supported =
   listed; available = launchable now), and default_runtime.available
   uses the same verdict.

2. Capabilities are resolved per registry read instead of being frozen at
   package init. Cloud Hypervisor resolves from the configured default
   version (SetDefaultVersion), so pinning v49.0 no longer advertises the
   v51.1-only disk-resize feature. Covered by a v49/v51 boundary test.

3. Fork support is an explicit Capabilities.SupportsFork field instead of
   being inferred from SupportsSnapshot, keeping each backend's
   PrepareFork implementation and its advertised feature aligned. A
   snapshot-without-fork test pins that snapshots alone never advertise
   fork; vz gates fork on the same save/restore probe as snapshots.

Regenerated lib/oapi via make oapi-generate.
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Response to GPT-5.6 capabilities review — round 1 (fixes in cbc919c)

All three findings adjudicated valid and addressed:

1. Availability now means launch prerequisites, not just registration. hypervisor.RegisterRuntime replaces RegisterCapabilities and accepts an optional per-runtime LaunchCheck, evaluated on every registry read. QEMU (standard and microvm) probes the exact binary lookup that StartVM uses (Starter.GetBinaryPath — the same call cmd/api/main.go warns on), so a host without a system QEMU lists qemu as supported but available: false, and a configured QEMU default reports default_runtime.available: false. This exposes the distinct supported-vs-available semantics the review suggested: runtimes[] gains a required available boolean (OpenAPI + regenerated client), and the handler still owns no runtime list. Embedded-binary backends (cloud-hypervisor, firecracker, vz) register no check — registration implies launchability. Because checks re-run per read, installing QEMU flips availability without a restart. Pinned by TestQEMUAvailabilityTracksSystemBinary, the enumeration launch-check subtests, and TestGetCapabilitiesDefaultAvailabilityTracksLaunchCheck.

2. Capabilities are resolved per registry read, not frozen at init. Registrations carry a func() Capabilities resolver. Cloud Hypervisor resolves CapabilitiesForVersion(GetDefaultVersion()), so cloud_hypervisor_default_version: v49.0 stops advertising the v51.1-only disk-resize feature. The v49/v51 boundary is pinned in TestRegisteredCapabilitiesTrackConfiguredDefaultVersion (both CapabilitiesForType and the enumeration used by the endpoint). The CH client's Capabilities() was aligned to the same resolver for consistency; the only version-varying flag (SupportsDiskResize) has no runtime consumer, so no behavior change. (True per-instance version capabilities would need the version from instance metadata plumbed into the client — out of scope here and unchanged from main.)

3. Fork is now an explicit capability. Capabilities.SupportsFork replaces the SupportsFork() == SupportsSnapshot derivation. Each backend sets it beside a comment pointing at its PrepareFork implementation: cloud-hypervisor/firecracker/qemu(+microvm) true; vz gates it on the same save/restore probe as snapshots (fork restores a snapshot of the source). The requested snapshot-without-fork regression test is in features_test.go ("snapshot support alone must not advertise fork"), and the vz darwin test now asserts fork tracks the probed OS version. A future snapshot-only backend simply leaves SupportsFork false and its feature list stays truthful.

Checks: go build/vet/gofmt clean (linux + darwin/arm64 cross for touched packages), make oapi-generate drift-free and idempotent, lib/hypervisor/... + cmd/api/api pass with -race and -count=1 (the pre-existing TestRegistryLayerCaching test-helper race reproduces identically at f8a1ca5 without these changes), and the two lib/instances fork tests that touch the registry pass. PR description updated for the new available field semantics.

@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

GPT-5.6 capabilities redesign review — round 2 (cbc919c)

Verdict: changes requested. Steven's primary architecture concern is resolved: runtime output is registry-driven and deterministic, per-runtime IDs derive beside Capabilities, fork is explicit, and the handler has neither a runtime list nor capability mapping. The earlier QEMU/default and configured-CH findings are fixed. Three important issues remain:

  1. Rosetta is reported as usable when it is unavailable. emulationSupported is true for every darwin/arm64 host, so macOS 11/12 and macOS 13+ without Rosetta advertise both linux/amd64 and rosetta-emulation; cmd/vz-shim/rosetta_arm64.go then rejects launch using LinuxRosettaDirectoryShareAvailability. Probe that same availability, or explicitly model supported-vs-currently-available image platforms instead of saying the host “can run” them.

  2. Firecracker availability ignores its effective configured binary. SetCustomBinaryPath overrides the embedded binary, but Firecracker registers without a LaunchCheck. A missing/non-executable hypervisor.firecracker_binary_path therefore reports Firecracker—and potentially default_runtime—as available: true even though every launch fails. Validate an active override in the registry check and cover it alongside QEMU.

  3. The additive endpoint introduces avoidable public Go source breaks. This public module removes exported hypervisor.RegisterCapabilities and adds DefaultHypervisor to exported instances.Manager, breaking downstream custom backends and manager mocks. Keep a deprecated static-registration wrapper, and let the API depend on/type-assert a narrow default-runtime accessor rather than widening the large provider interface.

Checks: complete origin/main...cbc919c diff and launch paths inspected; focused hypervisor/network tests pass; make oapi-generate is byte-for-byte drift-free; all Linux/Darwin/e2e/Stainless/security CI passes; current Bugbot completed with no findings.

… override, restore public API compatibility

- Rosetta emulation reporting now uses the same Virtualization.framework
  availability probe the vz-shim enforces at launch
  (LinuxRosettaDirectoryShareAvailability): hosts without Rosetta installed
  or on macOS < 13 no longer advertise the linux/amd64 image platform or the
  rosetta-emulation feature. cmd/api already links the framework on macOS,
  so this adds no build or runtime requirement; off darwin/arm64 the probe
  is a constant false.
- Firecracker registers a LaunchCheck validating the active
  hypervisor.firecracker_binary_path override with the same executable check
  the launch path applies, since the override takes precedence over the
  embedded binaries on every launch. Regression test covers no-override,
  missing, non-executable, and valid overrides, including per-read
  re-evaluation.
- Public Go source compatibility restored: hypervisor.RegisterCapabilities
  is back as a deprecated wrapper over RegisterRuntime, and
  instances.Manager no longer gains DefaultHypervisor \u2014 the capabilities
  handler type-asserts a narrow defaultHypervisorProvider accessor instead,
  falling back to the same cloud-hypervisor default lib/instances applies.
  Reverted the mock widening in lib/builds and lib/instances tests.
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Round-2 findings adjudication (fable-repair-pr338-r2)

All three round-2 findings were adjudicated as valid and are addressed in c7ed2fe.

1. Rosetta image capability overstated current host availability — valid, fixed.
The round-1 wording ("platform eligibility, verified at instance start") still let a macOS 11/12 host or an Apple Silicon host without Rosetta advertise linux/amd64 and rosetta-emulation that cmd/vz-shim/rosetta_arm64.go would reject. The endpoint's design goal is launch-truthful reporting, so eligibility-only reporting was the wrong call. Capability reporting now uses the same probe launches enforce: vz.LinuxRosettaDirectoryShareAvailability() == Installed (cmd/api/api/capabilities_rosetta_darwin_arm64.go), evaluated per request so softwareupdate --install-rosetta is reflected without a restart. NotSupported (macOS < 13) and NotInstalled both suppress the platform/feature, exactly matching shim admission. No new build/runtime requirement: cmd/api already links Virtualization.framework on macOS (checkHypervisorAccess); off darwin/arm64 the probe is a constant false. Tests: TestEmulationAvailable (pure gate matrix) and TestGetCapabilitiesRosettaTracksProbe (darwin/arm64, pins handler output to the live probe). OpenAPI descriptions updated and regenerated via make oapi-generate (drift-free).

2. Firecracker availability ignored the configured binary override — valid, fixed.
resolveBinaryPath gives hypervisor.firecracker_binary_path unconditional precedence over the embedded binaries, so "no LaunchCheck because binaries are embedded" was wrong whenever an override is set: every launch would fail while the registry reported available=true (and default_runtime.available=true for firecracker defaults). Firecracker now registers a LaunchCheck (checkLaunchPrerequisites) that validates the active override with the same validateExecutable check the launch path applies, and returns nil when no override is set (embedded binaries genuinely imply launchability). Regression test TestFirecrackerAvailabilityTracksBinaryOverride covers no-override, missing, non-executable, and valid overrides, including per-registry-read re-evaluation (availability flips back without re-registration).

3. Public Go source incompatibilities — valid, fixed.
This module is public; silently deleting hypervisor.RegisterCapabilities and widening instances.Manager were compile-time breaks for downstream custom backends and alternate Manager implementations/mocks — and neither break was necessary for the feature.

  • hypervisor.RegisterCapabilities is restored as a deprecated wrapper over RegisterRuntime (static capability set, no launch check — the old semantics exactly). Pinned by TestRegisterCapabilitiesCompat.
  • instances.Manager is back to its main shape. DefaultHypervisor() stays on the concrete manager only, and the capabilities handler type-asserts a narrow defaultHypervisorProvider accessor, falling back to the same cloud-hypervisor default lib/instances applies when the accessor is absent (TestGetCapabilitiesDefaultWithoutAccessor). The mock-widening in lib/builds and lib/instances tests is reverted, which doubles as proof that pre-existing implementations compile unchanged.

Verification: go build ./..., go vet ./..., gofmt clean; make oapi-generate drift-free; go test ./cmd/api/api ./lib/hypervisor/... ./lib/builds ./lib/network ./lib/scopes ./lib/instances(focused) pass — the only lib/instances failures are the known network-integration tests requiring NET_ADMIN, verified to fail identically on origin/main in this sandbox. PR description updated for the changed public semantics (Rosetta gating, launch checks, compatibility guarantees).

@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

GPT-5.6 capabilities redesign review — round 3 (c7ed2fe)

Verdict: changes requested. Steven’s primary architecture concern is resolved: built-in runtimes are enumerated deterministically from platform-gated registry entries; per-runtime feature IDs derive beside Capabilities; fork is explicit; and the handler owns neither a runtime list nor a capability mapping. Rosetta, Firecracker override, public manager-interface, gateway/network, macOS-version, OpenAPI/Stainless, and generated-output fixes check out. Two important truthfulness gaps remain:

  1. A valid manager wrapper can make default_runtime lie. ApiService accepts instances.Manager, and adapters commonly embed that interface. Embedding the interface does not expose the concrete manager’s extra DefaultHypervisor method, so a wrapped manager configured for Firecracker/QEMU falls into the hardcoded cloud-hypervisor fallback even though launches still use the wrapped manager’s real default. stubOpaqueInstanceManager currently pins this incorrect result. Seed the fallback from s.Config.Hypervisor.Default (normalizing only empty to cloud-hypervisor), then let the optional accessor override it.

  2. QEMU available=true does not establish the prerequisites every launch needs. Its LaunchCheck only calls GetBinaryPath; the fixed /usr/bin and /usr/local/bin candidates use os.Stat, so even a non-executable/broken QEMU is accepted. It also does not verify the required host vhost-vsock device, while every created instance gets a nonzero vsock CID and QEMU always adds vhost-vsock; API startup checks KVM only. Reuse a side-effect-free launch prerequisite check that at least validates the executable/version and required host device(s), and cover unavailable cases.

Proof: complete origin/main...c7ed2fe diff inspected; focused hypervisor/network/scopes/API/instance tests and vet pass; generated lib/oapi is byte-for-byte drift-free; Linux, Darwin, e2e, Semgrep, and current Stainless SDK generation pass. Latest real Cursor Bugbot completed successfully with no findings.

…nch prerequisites

Round-3 finding 1: a valid manager wrapper embedding instances.Manager hides
the concrete manager's DefaultHypervisor method, so the handler's hardcoded
cloud-hypervisor fallback could misreport a configured Firecracker/QEMU
default while launches still used it. The fallback is now seeded from
config.Hypervisor.Default (the value the wrapped manager was constructed
from), normalizing only an empty value to cloud-hypervisor — mirroring
lib/instances.NewManagerWithConfigE — with the optional accessor still
authoritative when exposed. The opaque-manager test now pins a configured
non-default runtime, plus unconfigured normalization and accessor precedence.

Round-3 finding 2: QEMU's LaunchCheck only called GetBinaryPath, whose fixed
/usr/bin and /usr/local/bin candidates are accepted on a bare os.Stat even if
non-executable or broken, and it never verified the host vhost-vsock device —
required because every created instance gets a nonzero vsock CID and
buildArgs unconditionally attaches a vhost-vsock device (API startup
validates KVM only). checkLaunchPrerequisites now validates the resolved
binary's execute bit, runs the same --version probe ResolveVersion persists
on every cold start, and stats /dev/vhost-vsock; all evaluated per registry
read and side-effect-free. Unavailable cases (missing/non-executable/broken/
unparseable binary, missing device) are covered with fake binaries and device
paths, and the availability test tracks the full prerequisite check. OpenAPI
description updated and regenerated via make oapi-generate.
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Round-3 findings adjudication (fable-repair-pr338-r3)

Both round-3 findings were adjudicated as valid and are addressed in 610529c.

1. Wrapped managers could misreport the default runtime — valid, fixed.
ApiService accepts the instances.Manager interface, and adapters commonly embed it; embedding hides the concrete manager's extra DefaultHypervisor method, so the previous hardcoded cloud-hypervisor fallback could report the wrong default (and the wrong default_runtime.available) while launches still routed through the wrapped manager's real configured default. The handler now seeds the fallback from s.Config.Hypervisor.Default — the exact value lib/providers.ProvideInstanceManager constructs the manager from — normalizing only an empty value to cloud-hypervisor (mirroring instances.NewManagerWithConfigE), and the optional accessor still overrides it when exposed. TestGetCapabilitiesDefaultWithoutAccessor now pins three contracts: a configured non-default runtime (firecracker) is reported through an opaque wrapper, an unconfigured default normalizes to cloud-hypervisor, and the accessor remains authoritative over config when present.

2. QEMU availability did not verify the prerequisites every launch needs — valid, fixed.
The LaunchCheck only called GetBinaryPath, whose fixed /usr/bin//usr/local/bin candidates pass on a bare os.Stat even when non-executable or broken, and it never checked the host vhost-vsock device — required because lib/instances assigns every instance a nonzero vsock CID and buildArgs unconditionally attaches a vhost-vsock device for it, while API startup validates KVM only. checkLaunchPrerequisites (shared by qemu and qemu-microvm) now, side-effect-free and per registry read:

  • resolves the binary with the same lookup launches use;
  • validates its execute bit (validateExecutable);
  • runs the same --version probe that backs ResolveVersion — which every cold start persists and treats as fatal on failure — so a broken or fake binary reports available=false;
  • stats /dev/vhost-vsock, with an actionable error (load the vhost_vsock kernel module).

Unavailable cases are covered in TestCheckLaunchPrerequisitesFor (missing, non-executable, broken, unparseable-version binaries; missing device) using fake binaries/device paths so the tests are host-QEMU-agnostic, and TestQEMUAvailabilityTracksLaunchPrerequisites pins registry availability to the full check. The CapabilitiesRuntime.available OpenAPI description was updated accordingly and regenerated via make oapi-generate (drift-free); the PR description's availability/fallback wording was updated to match.

Checks: go build ./..., go vet, gofmt clean; go test ./lib/hypervisor/... ./cmd/api/api/ ./lib/network/ ./lib/scopes/ pass (qemu package additionally -race -count=5); darwin cross-build of cmd/api/api + lib/hypervisor/... clean; lib/instances integration failures are the pre-existing no-NET_ADMIN sandbox failures, verified identical at c7ed2fef.

@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

GPT-5.6 capabilities redesign review — round 4 (610529c1)

Verdict: changes requested. Steven’s primary architecture concern is resolved: built-in runtimes now come from deterministic platform-gated registry enumeration, feature IDs live beside Capabilities, fork is explicit, and the handler owns neither a runtime list nor per-runtime feature mapping. Default-unavailable, gateway/error, typed network, Rosetta/image, macOS version, OpenAPI/Stainless, registration-tag, and generated-output changes otherwise check out. Three important issues remain:

  1. VZ underreports fork on macOS 13. lib/hypervisor/vz/client.go:89-93 gates SupportsFork on save/restore availability, but stopped-source VZ forks do not need machine-state save/restore: Starter.PrepareFork succeeds when SnapshotConfigPath is empty, and TestForkInstance_VZStoppedSourceSupported pins the operation. A client gating the generic fork API on this ID will hide a valid macOS 13 operation. Model stopped fork independently from running/standby prerequisites, or advertise fork and require snapshots/standby additionally for those source states.

  2. The public runtime registry can race with capability requests and crash the server. RegisterRuntime/the compatibility wrapper write runtimeRegistrations at lib/hypervisor/hypervisor.go:111-130, while every request iterates it at :167-180, with no synchronization or freeze. A concurrent register/enumerate reproducer under -race reports races and terminates with fatal error: concurrent map iteration and map write. Protect and snapshot the map (resolve callbacks after releasing the lock), or explicitly freeze registration before serving and reject/document late registration.

  3. The QEMU availability probe is unbounded and runs twice per request on linux/amd64. Both QEMU registrations call the same check, and each GET /capabilities executes qemu --version via context-free exec.Command (qemu/process.go:146-148). A hung/broken binary leaves subprocesses and request goroutines indefinitely; concurrent authenticated reads amplify this. Bound the probe with a context/timeout and coalesce the shared host prerequisite result per read (or with a short cache).

Proof: complete origin/main...610529c1 diff and launch/fork/network paths inspected; focused hypervisor/QEMU/Firecracker/VZ/network/scopes tests pass with -race; make oapi-generate is byte-for-byte clean; all current Linux, Darwin, e2e, generation, and security CI passes; latest real Cursor Bugbot completed successfully with no new actionable finding. The closeout autoreview independently confirmed finding 1.

…ounded+coalesced QEMU probe

- vz: advertise SupportsFork unconditionally. Stopped-source forks clone
  disks with no machine-state save/restore, so gating fork on the macOS 14
  save/restore probe hid a valid macOS 13 operation. The fork feature now
  documents its contract precisely: it promises the stopped-source fork;
  forking a standby or running source restores/creates snapshots and
  additionally requires the standby feature (which vz still derives from
  the save/restore probe). Feature IDs are unchanged, so the OpenAPI/
  Stainless surface only gains documentation.

- hypervisor: guard the public runtime registry with an RWMutex.
  RegisterRuntime is public API and can run while capability requests
  enumerate the registry; the unsynchronized map crashed under concurrent
  register/enumerate (-race reproducer added). Readers snapshot the map
  under the lock and resolve Capabilities/LaunchCheck callbacks only after
  releasing it, so callbacks may safely re-enter the registry (pinned by a
  reentrancy test). The deprecated RegisterCapabilities wrapper is
  preserved unchanged.

- qemu: bound and coalesce the availability probe. qemu --version now runs
  under a 5s context (WaitDelay backstops descendants holding the pipe),
  so a hung binary fails the check instead of wedging requests and leaking
  subprocesses. Both board registrations share one launchCheckCache: a
  short-TTL (1s) single-flight cache so a registry read and concurrent
  capability requests execute at most one probe per window, while
  installing QEMU or loading vhost_vsock still flips availability without
  a restart. Invalidate() is the explicit invalidation seam. Deterministic
  tests cover coalescing, expiry, error caching, invalidation, and the
  hung-binary deadline; tests use private cache instances, never the
  package global.
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Round-4 findings adjudication (fable-final-repair-r1) — f5a2d44

All three round-4 findings (review) adjudicated valid and fixed in f5a2d44. Point-by-point:

1. VZ underreported fork on macOS 13 — valid, fixed with a state-aware contract (no new IDs).
vz now advertises SupportsFork unconditionally (lib/hypervisor/vz/client.go): Starter.PrepareFork is implemented for every source state, and the stopped-source fork — a disk clone — needs no machine-state save/restore, so it genuinely works on macOS 13. Rather than adding per-state feature IDs, the existing small vocabulary now carries a precise gating contract, documented identically in Capabilities.SupportsFork, FeatureFork, and the OpenAPI CapabilitiesRuntime.features description: fork promises the stopped-source fork; forking a standby or running source restores/creates snapshots and additionally requires standby. That contract is complete and truthful on macOS 13: hot-source forks need a standby-capable host, and a macOS 13 host can't hold a standby source at all (SupportsStandby() = snapshot ∧ pause = false there). Public feature IDs are unchanged, so stainless.yaml and the generated SDK mapping needed no structural change — lib/oapi/oapi.go regenerated for the description (byte-stable across repeated make oapi-generate). Tests updated: save_restore_support_darwin_test.go now pins fork advertised on macOS 13 while snapshots/standby are not (and on a failed version probe), and features_test.go pins that Capabilities{SupportsFork: true} alone yields ["fork"] — the exact macOS 13 vz shape.

2. Registry race — valid, fixed with an RWMutex + snapshot-outside-lock reads.
runtimeRegistrations is now guarded by runtimeRegistrationsMu (lib/hypervisor/hypervisor.go): RegisterRuntime (and therefore the preserved deprecated RegisterCapabilities wrapper, unchanged) writes under the lock; RegisteredRuntimes clones the map under RLock and runs enumerateRuntimes on the snapshot; CapabilitiesForType reads the entry under RLock and resolves the callback after releasing it. No capability or launch-check callback ever executes while the registry lock is held. New lib/hypervisor/registry_concurrency_test.go:

  • TestRegistryConcurrentRegisterAndEnumerate — 8 writers × 8 enumerators under -race, using unique test-only type names so production registrations shared with unrelated parallel tests are never mutated. Verified it reproduces the crash on the pre-fix tree (fatal error/race report at 610529c) and passes post-fix.
  • TestRegistryCallbacksRunOutsideLock — a Capabilities resolver that itself calls RegisterRuntime (a write) and a LaunchCheck that re-enters CapabilitiesForType; this deadlocks if callbacks ran under even a read lock.

3. Unbounded, duplicated QEMU probe — valid, fixed with a bounded probe + shared single-flight TTL cache.

  • Bounded: versionFromBinary now takes a context and runs exec.CommandContext with WaitDelay = 1s (qemu/process.go); both the launch check and detectVersion (GetVersion/ResolveVersion) pass a versionProbeTimeout = 5s context, so a hung binary fails promptly instead of leaking subprocesses and request goroutines.
  • Coalesced: both the standard and microvm registrations resolve availability through one shared launchPrereqCache (qemu/launch_check_cache.go), a single-flight cache with a 1s TTL: one registry read (which checks both boards) and any burst of concurrent capability requests execute at most one qemu --version per window; concurrent callers needing a fresh result wait on the in-flight probe rather than spawning duplicates. Successes and failures cache alike and expire after 1s, so installing QEMU or loading vhost_vsock flips availability on the next read — no restart. Invalidate() is the explicit invalidation seam.
  • Tests (deterministic, no package-global leaks): launch_check_cache_test.go builds private cache instances with a fake clock — concurrent-coalescing (probe body panics if entered twice), TTL expiry incl. error-then-repair, explicit invalidation; register_linux_test.go adds the hung-binary case (exec sleep 60 fake binary fails at a 100ms deadline) and pins both registrations sharing the production cache/TTL. The shared launchPrereqCache itself is never mutated by tests.

Evidence (Linux/KVM sandbox):

  • go test -race -count=1 ./lib/hypervisor/... ./lib/network/... ./lib/scopes/... — all pass; go test -race ./cmd/api/api/ -run 'TestGetCapabilities|TestRegisteredRuntimes|…' — all pass. Full ./cmd/api/api passes without -race; with -race the only failure is the pre-existing TestRegistryLayerCaching loggingTransport race in registry_test.go, a file untouched by this PR (verified identical at 610529c).
  • go build ./..., go vet ./..., gofmt clean; make oapi-generate byte-stable (md5-identical across runs).
  • GOOS=darwin GOARCH=arm64 go build clean for lib/hypervisor, lib/hypervisor/qemu (incl. the new untagged cache file), lib/network, lib/oapi, lib/scopes; the cgo vz package needs a macOS toolchain (as on main) and is covered by the test-darwin CI job.
  • lib/instances full-package failures are the known no-NET_ADMIN iptables environment issue, identical at 610529c; fork-path tests (-run 'TestForkInstance|TestValidateForkSupport') pass.

Scope kept narrow: no security-hardening work, no public Go API breaks (only additive/unexported changes plus doc-precision on SupportsFork), no changes to the multi-runtime registry-driven design.

@cursor please review the new commit.

@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

GPT-5.6 capabilities continuation review — round 1 (f5a2d44)

Verdict: changes requested. Fork semantics now read coherently across stopped, standby/snapshot, and running-source flows (including VZ macOS 13 vs 14+); registry synchronization/snapshotting, callback lock boundaries, deterministic output, compatibility wrappers, multi-runtime/default/config behavior, network/Rosetta/Firecracker/CH/OpenAPI/Stainless/scope changes, and Steven’s nits check out. One important QEMU issue remains:

  1. The bounded version probe can still leak descendant processes. versionFromBinary uses exec.CommandContext plus WaitDelay (lib/hypervisor/qemu/process.go:157-160). Context cancellation kills only the direct child; WaitDelay closes inherited pipes but does not kill descendants. A fake QEMU wrapper containing sleep 60 & wait returned after the deadline/WaitDelay while leaving sleep alive under PID 1. The new test avoids this by using exec sleep 60, so it proves bounded return but not the claimed no-process-leak property. Run the probe in a dedicated process group and kill/reap the group on cancellation, then test a wrapper that spawns a child and assert both processes are gone.

Proof: complete 95db3fb...f5a2d44 diff and prior reviews inspected; focused registry/QEMU/VZ/network/scopes/API/fork tests pass (including race); build/vet pass with CI-provided embed placeholders; OpenAPI regeneration is byte-identical. All current CI and real Cursor Bugbot are green with no actionable Bugbot finding.

The bounded probe used exec.CommandContext's default cancel, which
signals only the direct child: a wrapper binary that spawned a
descendant (sleep 60 & wait) returned at the deadline but left the
descendant orphaned under PID 1, and WaitDelay only unblocked the
pipe read. Run the probe in a dedicated process group (Setpgid) and
SIGKILL the group (-pid) on cancellation so descendants die and are
reaped by init; keep WaitDelay as a backstop for a descendant that
setsid'd out of the group while holding the output pipe.

TestVersionFromBinaryKillsProcessGroupOnTimeout pins the property
with a wrapper that records both PIDs before hanging, asserting the
wrapper and its descendant are both gone after the probe's context
deadline (fails without the group kill).
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Continuation round-1 finding adjudication (fable-final-repair-r2)

Finding 1 — bounded version probe can orphan descendant processes: valid, fixed in 2260f64.

Confirmed exactly as described: exec.CommandContext's default cancel signals only the direct child, and WaitDelay merely stops waiting on the inherited output pipe — a fake QEMU wrapper running sleep 60 & wait returned at the deadline while leaving sleep alive under PID 1. The prior hung-probe test used exec sleep 60 (no descendant), so it proved bounded return but not the no-leak property.

Fix (lib/hypervisor/qemu/process.go): versionFromBinary now starts the probe in a dedicated process group (SysProcAttr{Setpgid: true}) and sets cmd.Cancel to SIGKILL the whole group (kill(-pid), mapping ESRCH to os.ErrProcessDone), so cancellation kills and lets init reap every descendant. WaitDelay stays as a backstop for the only remaining escape (a descendant that setsids out of the group while holding the pipe). This covers all probe callers — GetVersion/ResolveVersion on launches/cold starts and the shared launch-prerequisite cache behind both QEMU capability registrations.

Evidence:

  • New TestVersionFromBinaryKillsProcessGroupOnTimeout (process_test.go): a wrapper that records its own PID and a background sleep 60 child's PID before hanging in wait; asserts the probe errors at a 250ms deadline and that both PIDs are gone (kill(pid, 0)ESRCH) shortly after. Verified red/green: with the group kill reverted, the test fails with the descendant left alive; with the fix it passes under -race.
  • The existing exec sleep 60 deadline case is kept (direct-child coverage) with its comment corrected to point at the new group-kill test for descendant cleanup.
  • go build ./..., go vet ./..., gofmt clean; go test -race ./lib/hypervisor/... and the capabilities handler tests in cmd/api/api pass; registry concurrency tests pass with -count=2 -race; GOOS=darwin GOARCH=arm64 go build ./lib/hypervisor/qemu/... ./lib/hypervisor/ clean (Setpgid/syscall.Kill are portable to the darwin build, matching the existing daemonize path); make oapi-generate byte-identical (no API surface change this round).

Notes on unrelated observations while validating:

  • go test -race ./cmd/api/api/ intermittently reports a data race in TestRegistryLayerCaching's test-local loggingTransport (unsynchronized slice append during go-containerregistry's concurrent layer push). That file is untouched by this PR (git diff origin/main...HEAD -- cmd/api/api/registry_test.go is empty) — pre-existing test-only race on main, left out of scope.
  • lib/instances network-integration failures remain the known no-NET_ADMIN sandbox limitation, identical on origin/main.

@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

GPT-5.6 capabilities continuation review — round 2 (2260f64)

Verdict: changes requested. The full diff now checks out for VZ/macOS 13 vs 14+ fork semantics, every backend’s feature mapping, registry synchronization/callback lock boundaries/determinism/compatibility, multi-runtime/default/config behavior, network/Rosetta/Firecracker/CH/OpenAPI/Stainless/scopes, and Steven’s nits. One important QEMU leak remains:

  1. The version probe still leaks descendants when the direct wrapper exits first. versionFromBinary only kills the process group from cmd.Cancel (lib/hypervisor/qemu/process.go:164-174), and Cancel is not called when the direct child exits before the context deadline. A wrapper containing sleep 60 &; echo 'QEMU emulator version 8.2.0'; exit 0 leaves sleep alive; cmd.Output returns after the 1s WaitDelay with exec: WaitDelay expired before I/O complete, and each 1s cache refresh can leak another process. I reproduced this at the expected head. Clean up the probe’s process group on every completion path, not only context cancellation, and add a behavioral test where the wrapper exits while its background child retains the output pipe.

Proof: complete 95db3fb...2260f64 diff and all Steven/prior GPT comments inspected; focused hypervisor/network/scopes/API/fork race tests, build, vet, gofmt/diff checks, and byte-stable OpenAPI regeneration pass. All current CI and the real Cursor Bugbot are green with no actionable Bugbot finding.

cmd.Cancel only fires on context cancellation, so a wrapper binary that
exits before the deadline while leaving a background descendant holding
the stdout pipe leaked that descendant on every probe: Output returned
after WaitDelay with ErrWaitDelay, but nothing reaped the group. Kill the
probe's process group unconditionally after Output returns (no-op ESRCH
when the group is already empty) and pin the behavior with a regression
test where the wrapper prints a valid version, backgrounds sleep 60, and
exits immediately.
Parallel tests fork subprocesses, and a concurrent fork can transiently
inherit a just-written script's write descriptor between fork and exec,
failing the probe with 'text file busy' before the script ever runs.
Retry the probe (with a fresh timeout context per attempt) so the two
process-group-kill regression tests assert probe behavior, not the
well-known exec race.
@rgarcia

rgarcia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Continuation round-2 finding adjudication (fable-final-repair-r3)

Finding 1 — probe still leaks descendants when the direct wrapper exits first: valid, fixed in 66a7fa1 (+ test-hardening in 993d31e).

Confirmed exactly as reproduced in the round-2 review: cmd.Cancel only fires on context cancellation, so a wrapper containing echo 'QEMU emulator version 8.2.0'; sleep 60 &; exit 0 never triggered the group kill — cmd.Output returned after the 1s WaitDelay with exec: WaitDelay expired before I/O complete while sleep 60 stayed alive, leaking one process per cache refresh.

Fix (lib/hypervisor/qemu/process.go): versionFromBinary now SIGKILLs the probe's process group unconditionally after cmd.Output returns, on every completion path — success, exec failure, ErrWaitDelay, and cancellation alike. Setpgid keeps the direct child's pid pinned as the group's pgid while any descendant lives, so the post-Output kill(-pid, SIGKILL) reaps stragglers; ESRCH (group already empty, the common healthy-binary case) is a no-op. cmd.Cancel retains the group kill so a hung direct child still dies at the deadline rather than after WaitDelay.

Regression test: TestVersionFromBinaryKillsDescendantsWhenWrapperExitsFirst builds exactly the reproduction wrapper (valid version output, background sleep 60 inheriting the stdout pipe, immediate exit), asserts the probe completes via WaitDelay with ErrWaitDelay (not the context deadline), and asserts the recorded descendant pid is reaped. Verified it fails on the prior head 2260f64 ("Condition never satisfied: background descendant must be killed") and passes with the fix. 993d31e additionally hardens both process-group tests against the well-known fork/exec ETXTBSY race (a concurrently forked test subprocess can transiently inherit the just-written script's write fd), observed once in ~10 full-package -race -count=2 runs; with the retry, 30 consecutive -race -count=2 runs (60 package executions) pass clean.

Evidence at 993d31e0:

  • go test ./lib/hypervisor/... -race -count=1 — all pass; qemu package additionally clean over 30× -race -count=2 runs
  • go test ./cmd/... -count=1 — pass (incl. cmd/api/api capabilities suite)
  • go build ./..., go vet ./..., gofmt clean; GOOS=darwin GOARCH=arm64 go build/go vet clean for lib/hypervisor + lib/hypervisor/qemu
  • make oapi-generate — zero drift (no public surface changed this round: the fix is entirely inside the probe; no FeatureID/OpenAPI/Stainless changes)
  • Pre-existing lib/instances network-integration failures reproduced identically with the fix stashed (sandbox lacks NET_ADMIN/iptables) — unrelated

Scope stayed narrow: 2 files of production+test change in lib/hypervisor/qemu, no registry/capability/API surface touched.

…Y everywhere

The timeout regression test cancelled the probe on a fixed 250ms deadline,
which raced slow script startup — macOS CI runners can take longer than
that to first-exec a fresh script (XProtect scan), killing the group before
the pid files existed. Cancel the context only once the script has
demonstrably hung with both pids recorded, with a generous backstop that
still fails fast on regression. Extend the fork/exec ETXTBSY retry (a
concurrent fork transiently inheriting a just-written script's write fd) to
the launch-prerequisite subtests that exec fake binaries; observed ~1/30
full-package -race runs, 0/50 after. No production code changes.
@rgarcia

rgarcia commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Round-3 closeout — final head 47b6e51c, all CI green

Follow-up to the round-3 adjudication: after pushing the completion-path group kill (66a7fa1) and the ETXTBSY retry (993d31e), Darwin CI exposed a latent timing race in the pre-existing TestVersionFromBinaryKillsProcessGroupOnTimeout: the fixed 250ms probe deadline could kill the group before a slow-starting script (macOS first-exec scanning) recorded its pid files. 47b6e51c (test-only) makes both process-group tests deterministic:

  • the timeout test now cancels the probe's context itself, only once the script has demonstrably hung with both pids recorded non-empty — no fixed deadline to race — with a 30s backstop that still fails fast on regression (the test now completes in ~30ms locally);
  • the fork/exec ETXTBSY retry is shared and extended to the launch-prerequisite subtests that exec fake binaries (register_linux_test.go), where the same race surfaced ~1/30 full-package -race runs.

No production code changed after 66a7fa1. Verified at 47b6e51c: 50 consecutive go test ./lib/hypervisor/qemu/ -race -count=2 runs clean (100 package executions), go test ./lib/hypervisor/... -race, go vet, gofmt, Darwin go vet ./lib/hypervisor/qemu/ clean, and the descendant-leak regression test still fails when run against the pre-fix probe. All CI checks green at this head — including test-darwin — and Cursor Bugbot passed with no new findings.

@rgarcia

rgarcia commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

GPT-5.6 capabilities continuation review — round 3 (47b6e51)

Verdict: changes requested. The full 95db3fb...47b6e51 diff and Steven/all prior GPT comments check out for fork state semantics (including VZ macOS 13 vs 14+), registry locking/callback boundaries/determinism/compatibility, bounded/coalesced QEMU process cleanup, default/runtime enumeration, network/Rosetta/Firecracker/OpenAPI/Stainless/scopes, and Steven’s nits. Two capability-truthfulness issues remain:

  1. QEMU can report available with an unusable vhost-vsock path. checkLaunchPrerequisitesFor only os.Stats the path (lib/hypervisor/qemu/register_linux.go:88-90). A regular file, directory, or character device the service user/cgroup cannot open still passes, although every launch requires QEMU to open /dev/vhost-vsock read/write. The “working” test currently uses a regular file, so it pins this false positive. Validate the device type and actual O_RDWR access (then close it), with behavioral wrong-type/permission coverage.

  2. A connected Cloud Hypervisor client now reports the mutable default version’s capabilities, not its VMM’s version. CloudHypervisor.Capabilities() delegates to GetDefaultVersion() (lib/hypervisor/cloudhypervisor/cloudhypervisor.go:47-55). Mixed-version hosts are supported: existing/restored v49 and v51.1 instances remain on their stored versions when the configured default changes. They can therefore gain or lose disk-resize incorrectly. Keep the registry resolver default-version-aware, but derive client capabilities from the connected/started VMM version (or its ping feature/version data).

Proof: focused hypervisor/QEMU/API/network/scopes/fork race tests, 20× QEMU -race -count=2, build, vet, Darwin cross-vet, and byte-stable OpenAPI regeneration pass. All expected-head CI and real Cursor Bugbot checks are green with no Bugbot finding.

@rgarcia

rgarcia commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Round-3 finding adjudication (d7fba22)

Both findings were valid to examine; one required a production fix, while the other exposed a pre-existing client limitation that this PR had accidentally made mutable.

  1. QEMU vhost-vsock prerequisite — fixed. The launch check now requires the path to be a character-device node and verifies that the service process can open it O_RDWR, then closes it. A regular file, directory, missing node, or inaccessible character device reports the runtime unavailable. Tests now use /dev/null as a harmless character-device stand-in and cover wrong node types plus propagated open denial. This remains a diagnostic readiness check; it intentionally does not mutate the device with vhost ioctls.

  2. Connected Cloud Hypervisor client version — regression removed, broader redesign deferred. The PR had changed CloudHypervisor.Capabilities() from its existing compile-time-version behavior to the mutable configured default. That was wrong for already-running mixed-version clients, so d7fba22 restores the exact origin/main client behavior. The new registry resolver remains correctly default-version-aware for /capabilities, whose contract is the runtime used for new launches. Making an existing client instance-version-aware requires carrying stored HypervisorVersion through the reconnect factory (or adding a bounded VMM ping/version path); that is a separate lifecycle API change and is not necessary for this additive host-capabilities endpoint. This PR no longer introduces or worsens that pre-existing limitation.

Validation: focused QEMU, Cloud Hypervisor, and registry suites pass under -race; capabilities handler tests pass; go vet and git diff --check are clean. CI/Bugbot are running on the pushed head.

@rgarcia

rgarcia commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Final head review — d7fba22

Approved: no diff-caused blocking or important findings remain.

Verified from the code and checks:

  • QEMU rejects missing, non-character, and O_RDWR-inaccessible vhost-vsock nodes; the probe only opens/closes the node (no reads, writes, or ioctls), and its 1s cache coalesces bounded, process-group-cleaned refreshes.
  • The Cloud Hypervisor registry resolves GetDefaultVersion() for new-launch capabilities, while CloudHypervisor.Capabilities() again uses vmm.DefaultVersion, preserving origin/main client behavior. Instance-version-aware connected-client reporting remains a non-blocking pre-existing follow-up.
  • Prior multi-runtime/registry/default/fork/network/Rosetta/Firecracker/OpenAPI/Stainless findings remain fixed.
  • Focused race tests, capabilities/fork tests, build, vet, gofmt, and diff checks pass. Exact-head Linux, Darwin, e2e, SDK generation, Semgrep, Socket, and real Cursor Bugbot checks are green. Final autoreview was clean.

@rgarcia
rgarcia requested a review from sjmiller609 August 16, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants