fix: model ready state check

README.md

@@ -7,6 +7,7 @@
 [![Go Reference](https://pkg.go.dev/badge/github.com/InftyAI/Nebula.svg)](https://pkg.go.dev/github.com/InftyAI/Nebula)
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 ![Go Version](https://img.shields.io/badge/go-1.24-00ADD8?logo=go&logoColor=white)
+[![Discord](https://img.shields.io/badge/Discord-Join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/7WTUuFqyS6)
 
 Run GPU workloads on any NeoCloud or hyperscaler through one Kubernetes API.
 
@@ -96,6 +97,8 @@ placement controller owns those.
 - See [config/samples](config/samples) for example NodePools and a runnable workload.
 - See [docs/add-a-provider.md](docs/add-a-provider.md) to add a provider backend.
 - See [docs/architecture.md](docs/architecture.md) for design details.
+- See [docs/status.md](docs/status.md) for how instance lifecycle becomes Pod and
+  NodeClaim status, per provider.
 
 ## License
 

api/v1alpha1/groupversion_info.go

@@ -150,3 +150,48 @@ const (
 	// VK liveness (see docs/architecture.md §3).
 	TerminateInstanceFinalizer = "nebula.inftyai.com/terminate-instance"
 )
+
+// Pod status reasons the virtual kubelet stamps on the Pods it reports, projecting
+// the external instance's lifecycle onto standard Pod status (pkg/vnode/status.go
+// is the only writer).
+//
+// They live here, not privately in pkg/vnode, for two reasons. They are a CONTRACT
+// between packages: the Pod phase is lossy — Provisioning and "booting" both
+// surface as PodPending — so the reason is the only thing separating "no instance
+// exists yet" from "an instance exists and is coming up", and the NodeClaim
+// controller keys its teardown guard off exactly that distinction (see
+// desiredPhase). A rename on the writing side that the reading side did not follow
+// would still compile, still pass tests, and silently leak paid instances: every
+// booting instance would read as Provisioning, so a Pod that vanished mid-boot
+// would be left running behind the cache-lag grace window. And they are user-facing
+// — operators match on status.reason in jsonpath and alerts — so every value is
+// public API whether or not Nebula's own code currently reads it. That is why the
+// whole set is here rather than the subset with in-tree readers: these are the
+// values status.reason can take, and a reader should find them in one place.
+const (
+	// PodReasonProvisioning: a provider Provision call has been issued but the
+	// instance does not yet exist — we are still allocating it (e.g. EC2
+	// RunInstances in flight). Set on CreatePod, before the first poll observes
+	// the instance.
+	PodReasonProvisioning = "Provisioning"
+	// PodReasonInitializing: the instance EXISTS at the provider but is not yet
+	// reachable — it is booting (EC2 "pending"), running-but-not-yet-passing its
+	// reachability checks (running, <2/2, EC2's own "Initializing" status), or a Modal
+	// sandbox whose readiness probe has not passed. It mirrors that EC2 status-check
+	// term. Provisioning is done; the instance is coming up. Distinct from
+	// Provisioning so a Pod stuck here points at a slow boot / failing status checks,
+	// not a stuck allocation — and so the NodeClaim controller can tell that an
+	// instance exists. The virtual kubelet stamps it only for an instance it observed
+	// in the provider's List, which is what makes it trustworthy as that evidence.
+	PodReasonInitializing = "Initializing"
+	// PodReasonRunning: the provider reports the instance running.
+	PodReasonRunning = "Running"
+	// PodReasonProvisionFailed: the provider rejected or failed the Provision call.
+	PodReasonProvisionFailed = "ProvisionFailed"
+	// PodReasonFailed: the provider reports the instance in a failed state.
+	PodReasonFailed = "Failed"
+	// PodReasonTerminated: the instance is gone from the provider (torn down,
+	// reclaimed, or exited). Disappearance alone does not say WHY, so this is the
+	// neutral term rather than "Preempted".
+	PodReasonTerminated = "Terminated"
+)

api/v1alpha1/nodeclaim_types.go

@@ -72,40 +72,47 @@ type PodReference struct {
 // NodeClaimPhase is the coarse, user-facing lifecycle state.
 //
 // The NodeClaim is a passive teardown ledger, not a status mirror: it does NOT
-// track finer workload runtime status (CPU/logs/restarts) — the Pod is the
-// source of truth for that (see pkg/vnode/status.go). It tracks only the coarse
-// states that matter to its own job as a ledger, keyed off the served Pod's
-// phase/reason: Provisioning (allocating — instance does not exist yet),
-// Initializing (instance exists and is booting but not yet reachable), Bound
-// (instance running — the guard the teardown backstop trusts), and Terminated
-// (instance gone). Finer states (e.g. Preempted) are deliberately absent:
-// preemption cannot be detected — the provider contract's InstanceState has no
-// Preempted value, and an absent instance only tells us it is gone, not why.
-// Reintroduce a phase only when something actually sets it.
+// track finer workload runtime status (CPU/logs/restarts/readiness) — the Pod is
+// the source of truth for that (see pkg/vnode/status.go). It tracks only the
+// coarse states that matter to its own job as a ledger, keyed off the served
+// Pod's phase/reason: Provisioning (instance does not exist yet), Bound (an
+// instance EXISTS at the provider — the guard the teardown backstop trusts), and
+// Terminated (instance gone). Finer states (e.g. Preempted) are deliberately
+// absent: preemption cannot be detected — the provider contract's InstanceState
+// has no Preempted value, and an absent instance only tells us it is gone, not
+// why. Reintroduce a phase only when something actually sets it.
 //
-// Only Bound is a teardown guard: neither Provisioning nor Initializing earns the
-// "trust a later disappearance" trust, because until the instance is confirmed up
-// an absent Pod may be cache lag rather than a real teardown.
+// The ledger's question is EXISTENCE, not readiness: what the backstop must know
+// is whether there is an instance out there to reclaim. A booting instance and a
+// serving one are equally real — equally billable, equally in need of teardown —
+// so both are Bound, and readiness is left entirely to the Pod. (This is why
+// there is no Initializing phase: it would be a readiness distinction on an
+// object that does not track readiness.)
 type NodeClaimPhase string
 
 const (
 	// NodeClaimProvisioning: the served Pod has been observed but the external
 	// instance does not yet exist — provisioning is still allocating it. The claim
 	// does NOT earn the Bound teardown guard here: a Pod that vanishes while still
 	// provisioning is treated as possible cache lag (grace window), not a real
-	// teardown, because we never confirmed the instance was actually up.
+	// teardown, because we never confirmed an instance was actually created.
 	NodeClaimProvisioning NodeClaimPhase = "Provisioning"
-	// NodeClaimInitializing: the external instance EXISTS at the provider but is not
-	// yet reachable (e.g. EC2 is "pending", or "running" but its 2/2 status checks
-	// have not passed). The served Pod is Pending with reason Initializing (see
-	// pkg/vnode/status.go). Like Provisioning it does NOT earn the Bound guard — the
-	// instance is not yet confirmed up — but it is surfaced as a distinct phase so
-	// "allocating" and "booting" are distinguishable on the ledger.
-	NodeClaimInitializing NodeClaimPhase = "Initializing"
-	// NodeClaimBound: the served Pod has been observed running (present and not in
-	// a terminal phase). This is the durable guard the backstop trusts — a Bound
-	// claim whose Pod later disappears is a real teardown, not cache lag. The claim
-	// does not track finer workload status; the Pod is the source of truth for that.
+	// NOTE: there is deliberately no "Initializing" phase. It used to mean "the
+	// instance exists but is not reachable yet" and did NOT earn the teardown guard,
+	// which stranded a real, billable instance behind the grace window whenever its
+	// Pod vanished mid-boot. Existence is what the ledger tracks, so that state is
+	// now Bound; readiness lives on the Pod alone.
+	//
+	// NodeClaimBound: an external instance EXISTS at the provider for this claim.
+	// This is the durable guard the backstop trusts — a Bound claim whose Pod later
+	// disappears is a real teardown, not cache lag, so it is reclaimed immediately
+	// rather than after the grace window.
+	//
+	// Existence, NOT readiness: an instance that is booting (EC2 "pending", or
+	// running with its 2/2 status checks still pending; a Modal sandbox whose
+	// readiness probe has not passed) is Bound, because it is just as real and just
+	// as billable as one that is serving. Whether the workload is actually usable is
+	// the Pod's Ready condition, not this phase.
 	NodeClaimBound NodeClaimPhase = "Bound"
 	// NodeClaimTerminating: the served Pod is being deleted (its DeletionTimestamp
 	// is set) but the external instance may not be reclaimed yet — teardown is in

api/v1alpha1/nodepool_types.go

@@ -167,7 +167,8 @@ const (
 
 // NodePoolStatus surfaces the current placement picture for observability.
 type NodePoolStatus struct {
-	// Placed counts running instances per provider, for at-a-glance balance.
+	// Placed counts existing instances per provider (booting included), for
+	// at-a-glance balance.
 	// +optional
 	Placed map[string]int32 `json:"placed,omitempty"`
 

config/crd/bases/nebula.inftyai.com_nodepools.yaml

@@ -246,8 +246,9 @@ spec:
                 additionalProperties:
                   format: int32
                   type: integer
-                description: Placed counts running instances per provider, for at-a-glance
-                  balance.
+                description: |-
+                  Placed counts existing instances per provider (booting included), for
+                  at-a-glance balance.
                 type: object
             type: object
         type: object

config/samples/deployment.yaml

@@ -28,7 +28,7 @@ metadata:
   labels:
     app.kubernetes.io/managed-by: nebula
 spec:
-  replicas: 2
+  replicas: 8
   selector:
     matchLabels:
       app: gpu-workload-sample
@@ -38,7 +38,7 @@ spec:
         app: gpu-workload-sample
         nebula.inftyai.com/enabled: "true"
         nebula.inftyai.com/nodepool: sample
-        nebula.inftyai.com/accelerator-type: t4
+        nebula.inftyai.com/accelerator-type: a100-80gb
     spec:
       # Do NOT set nodeName or a provider nodeSelector yourself — the placement
       # controller fills the nodeSelector in when it ungates the Pod. Setting
@@ -57,6 +57,25 @@ spec:
         # this output, read it on the PROVIDER side (the Modal dashboard or
         # `modal app logs`), not via kubectl.
         command: ["sh", "-c", "nvidia-smi --query-gpu=index,name,memory.total --format=csv || echo 'no nvidia-smi'; sleep 3600"]
+        # The readiness bar for the EXTERNAL instance. On Modal this is the only
+        # thing that lets Nebula tell "still coming up" (queued, pulling the image,
+        # attaching the GPU) from "up and serving": the cheap poll signal answers
+        # only "has the process exited?", so WITHOUT a probe the Pod — and its
+        # Deployment's ready count — goes Running the moment the sandbox is created.
+        # With one, it stays Pending/Initializing until the probe passes.
+        #
+        # An EXEC probe, because this sample serves nothing and declares no ports:
+        # tcpSocket/httpGet need a numeric port, and a named or missing one omits
+        # the probe entirely. `nvidia-smi -L` is a real bar here — it passes only
+        # once the GPU is actually attached and the driver answers.
+        #
+        # Only exec and tcpSocket reach Modal (httpGet degrades to tcpSocket on its
+        # port); periodSeconds maps to the probe interval, and the other timing
+        # fields are the kubelet's, which never runs here.
+        readinessProbe:
+          exec:
+            command: ["nvidia-smi", "-L"]
+          periodSeconds: 5
         resources:
           requests:
             cpu: "1"
@@ -65,4 +84,4 @@ spec:
             # GPU count. Standard extended resource, so the scheduler's fit check
             # and provisioning read the same number. 8 => 8x the accelerator-type
             # above.
-            nvidia.com/gpu: "1"
+            nvidia.com/gpu: "8"

docs/architecture.md

@@ -11,7 +11,9 @@ seam is meant to support more backends, but this document describes the
 implementation that is in the repository today. Planned or partial work is
 called out in
 [Current implementation status](#current-implementation-status). For deployment
-and credential setup, see [docs/deploy.md](deploy.md).
+and credential setup, see [docs/deploy.md](deploy.md); for how an instance's
+lifecycle becomes Pod and NodeClaim status (including each provider's own status
+mapping), see [docs/status.md](status.md).
 
 - [Goals and Non-Goals](#goals-and-non-goals)
 - [System Overview](#system-overview)
@@ -236,6 +238,10 @@ Important details:
 - NodeClaim does not mirror logs, restarts, container state, or fine-grained
   runtime health. Those belong on the Pod.
 
+The full mapping — Pod phase/reason and the claim phase each produces, plus each
+provider's own status vocabulary and the limits of what is observable — lives in
+[docs/status.md](status.md).
+
 ---
 
 ## Components
@@ -330,8 +336,8 @@ Reconcile behavior:
 
 - add the terminate finalizer before doing anything else;
 - fetch the served Pod by namespace/name and UID;
-- set coarse phase from the served Pod: `Provisioning`, `Initializing`, `Bound`,
-  `Terminating`, or `Terminated`;
+- set coarse phase from the served Pod: `Provisioning`, `Bound`, `Terminating`, or
+  `Terminated`;
 - best-effort record `status.instanceID` by matching the provider instance by
   claim name;
 - when a previously observed Pod disappears, delete the claim so the finalizer
@@ -492,9 +498,9 @@ status:
   instanceID: i-0123456789abcdef0
 ```
 
-Valid phases are `Provisioning`, `Initializing`, `Bound`, `Terminating`, and
-`Terminated`. The claim deliberately does not duplicate PodSpec and does not
-mirror fine-grained runtime status.
+Valid phases are `Provisioning`, `Bound`, `Terminating`, and `Terminated`. The
+claim deliberately does not duplicate PodSpec and does not mirror fine-grained
+runtime status — `Bound` answers existence, not readiness.
 
 ### Sandbox and SandboxSet