feat: support regions in modal

README.md

@@ -53,11 +53,11 @@ metadata:
   name: gpu
 spec:
   providers:
-  - name: modal            # NeoCloud; region-simple, no regions needed
-  - name: aws              # hyperscaler; region-aware, at least one required
+  - name: modal            # NeoCloud; regions omitted = place anywhere (cheapest)
+  - name: aws              # hyperscaler; "us" expands to every US region
     regions:
-    - us-east-1
-    - us-west-1
+    - us
+    - eu-west-1            # or name one region exactly
   capacityTypes:           # prefer cheap Spot, fall back to OnDemand
   - Spot
   - OnDemand

api/v1alpha1/nodeclaim_types.go

@@ -31,13 +31,20 @@ type NodeClaimSpec struct {
 	// +optional
 	CapacityType CapacityType `json:"capacityType,omitempty"`
 
-	// Region is the provider region the placement optimizer selected, in the
+	// Region is the region candidate the placement optimizer selected, in the
 	// provider's own vocabulary (e.g. AWS "us-east-1"). Stored durably alongside
 	// Provider/CapacityType because it is a provisioning input that cannot be read
 	// off the Pod, and Provision needs it to re-issue the request in the same
 	// region after a controller restart. Immutable, like Provider. Empty means
-	// "the provider's configured default region" — region-simple providers (Modal,
-	// RunPod) leave it empty.
+	// "the provider's configured default region" — a provider with no region
+	// constraint declared on the pool leaves it empty.
+	//
+	// It is not always a single region NAME: a provider whose create cannot fail
+	// over collapses every region the pool declared into ONE candidate, and stores
+	// them joined by a provider-private separator (Modal uses "|", so a pool
+	// declaring us-east and us-west records "us-east|us-west"). Only that provider
+	// can split the value back, which it does at the API boundary. Treat the field
+	// as an opaque provider-scoped token rather than parsing it.
 	// +optional
 	Region string `json:"region,omitempty"`
 

api/v1alpha1/nodepool_types.go

@@ -34,7 +34,10 @@ import (
 // static property of the spec, so it is enforced at admission by the CEL rule
 // below rather than surfaced as a status condition after the fact.
 // +kubebuilder:validation:XValidation:rule="self.strategy != 'Weighted' || self.providers.all(p, has(p.weight))",message="strategy Weighted requires a weight on every provider"
-// +kubebuilder:validation:XValidation:rule="self.providers.all(p, p.name != 'aws' || (has(p.regions) && size(p.regions) > 0))",message="provider aws requires at least one region"
+// (AWS once required at least one region here, because an omitted list meant "the
+// client's default region" and its client has none. Omitted now means "every region
+// the provider serves", which is a valid — if broad — AWS policy, so the rule is gone.
+// See ProviderSpec.Regions.)
 type NodePoolSpec struct {
 	// Providers is the ordered set of NeoClouds this pool is allowed to use.
 	// A Pod bound to this pool can only ever be placed on a provider in this
@@ -79,26 +82,37 @@ type ProviderSpec struct {
 	// +optional
 	Weight *int32 `json:"weight,omitempty"`
 
-	// Regions constrains this provider to a subset of its regions, in the
-	// provider's OWN vocabulary (e.g. ["us-east-1","eu-west-2"] for AWS). Region
-	// is provider-namespaced — there is no cross-provider region vocabulary — so
-	// it lives here per provider, not on the pool. Two cases:
-	//   - omitted/empty => the provider's configured default region (the region
-	//     its client resolved from env/config/instance metadata at startup). This
-	//     is the no-surprise default for region-simple providers (Modal, RunPod),
-	//     which have a single region and ignore this field.
-	//   - explicit list => exactly those regions.
-	// AWS is the exception: it is region-aware with no meaningful single default,
-	// so a `- name: aws` entry MUST list at least one region. That is enforced at
-	// admission by the CEL rule on NodePoolSpec (a per-provider requirement, so it
-	// belongs on the spec where all provider entries are visible, not as a blanket
-	// MinItems that would burden region-simple providers). An "all regions"
-	// wildcard is intentionally NOT supported yet: it only makes sense once the
-	// price-ranking optimizer can expand it against the provider's catalog and
-	// choose among the results, so it is reserved for then. At most 8 regions;
-	// maxLength bounds each entry.
+	// Regions constrains where this provider may place, in the provider's OWN
+	// vocabulary. Region is provider-namespaced — there is no cross-provider region
+	// vocabulary — so it lives here per provider, not on the pool. It is a
+	// CONSTRAINT, not a list of regions to use, and it has three levels:
+	//   - omitted/empty => unconstrained: every region the provider serves. For a
+	//     region-simple provider (Modal) this means "send no region and let the
+	//     provider place freely", which is also its widest and cheapest mode.
+	//   - a geography GROUP token ("us", "eu", "ap", ...) => that geography's
+	//     regions. This is the recommended way to ask for breadth with a data
+	//     residency boundary.
+	//   - a literal region name ("us-east-1" on AWS, "us-east" on Modal) => exactly
+	//     that region.
+	// The provider resolves which level a value is, since only it knows its own
+	// geography (see provider.Provider's ExpandRegions). Group tokens are shared
+	// across providers but the regions behind them are not: "eu" is eu-west-1 and
+	// friends on AWS, while London is eu-west-2 there and there is no "uk" group.
+	//
+	// A value that is not a group token is passed to the provider UNVALIDATED.
+	// Region names change faster than Nebula ships, so an unrecognized one is
+	// forwarded rather than rejected: a genuinely bad name fails at provision time
+	// with the provider's own error, which is better than refusing a region that
+	// launched last week. It is also the escape hatch for AWS opt-in regions, which
+	// no group contains.
+	//
+	// Unconstrained on a region-aware provider is the widest setting and costs
+	// something: every region becomes a placement candidate to walk on failover, and
+	// every region is swept by the observability poll loop. Prefer a group unless the
+	// workload genuinely needs global reach. There is no cap on the number of entries
+	// (a group already expands to many, so capping the declaration would be
+	// arbitrary); maxLength bounds each entry.
 	// +optional
-	// +kubebuilder:validation:MaxItems=8
 	// +kubebuilder:validation:items:MaxLength=32
 	Regions []string `json:"regions,omitempty"`
 }

cmd/main.go

@@ -485,7 +485,14 @@ func awsRegionSource(c client.Client) awsprovider.RegionSource {
 		for i := range pools.Items {
 			for _, ps := range pools.Items[i].Spec.Providers {
 				if ps.Name == provider.ProviderAWS {
-					regions = append(regions, ps.Regions...)
+					// Expand PER POOL, before unioning. ProviderSpec.Regions is a
+					// constraint, not a list: an omitted one means "every region", and
+					// unioning the raw lists first would collapse that to "nothing" —
+					// the swept set would miss regions placement provisions into, and
+					// List's absence is reported as Terminated on live instances.
+					// This is the same expansion regionsFor applies on the placement
+					// side; both must agree, so both call this one function.
+					regions = append(regions, awsprovider.ExpandRegions(ps.Regions)...)
 				}
 			}
 		}

config/crd/bases/nebula.inftyai.com_nodeclaims.yaml

@@ -128,13 +128,20 @@ spec:
                 type: string
               region:
                 description: |-
-                  Region is the provider region the placement optimizer selected, in the
+                  Region is the region candidate the placement optimizer selected, in the
                   provider's own vocabulary (e.g. AWS "us-east-1"). Stored durably alongside
                   Provider/CapacityType because it is a provisioning input that cannot be read
                   off the Pod, and Provision needs it to re-issue the request in the same
                   region after a controller restart. Immutable, like Provider. Empty means
-                  "the provider's configured default region" — region-simple providers (Modal,
-                  RunPod) leave it empty.
+                  "the provider's configured default region" — a provider with no region
+                  constraint declared on the pool leaves it empty.
+
+                  It is not always a single region NAME: a provider whose create cannot fail
+                  over collapses every region the pool declared into ONE candidate, and stores
+                  them joined by a provider-private separator (Modal uses "|", so a pool
+                  declaring us-east and us-west records "us-east|us-west"). Only that provider
+                  can split the value back, which it does at the API boundary. Treat the field
+                  as an opaque provider-scoped token rather than parsing it.
                 type: string
             required:
             - podRef

config/crd/bases/nebula.inftyai.com_nodepools.yaml

@@ -71,7 +71,11 @@ spec:
               capacity tier; it never crosses tiers.\n\nThe Weighted strategy requires
               a weight on every provider ref. This is a\nstatic property of the spec,
               so it is enforced at admission by the CEL rule\nbelow rather than surfaced
-              as a status condition after the fact."
+              as a status condition after the fact.\n(AWS once required at least one
+              region here, because an omitted list meant \"the\nclient's default region\"
+              and its client has none. Omitted now means \"every region\nthe provider
+              serves\", which is a valid — if broad — AWS policy, so the rule is gone.\nSee
+              ProviderSpec.Regions.)"
             properties:
               capacityTypes:
                 default:
@@ -127,28 +131,39 @@ spec:
                       type: string
                     regions:
                       description: |-
-                        Regions constrains this provider to a subset of its regions, in the
-                        provider's OWN vocabulary (e.g. ["us-east-1","eu-west-2"] for AWS). Region
-                        is provider-namespaced — there is no cross-provider region vocabulary — so
-                        it lives here per provider, not on the pool. Two cases:
-                          - omitted/empty => the provider's configured default region (the region
-                            its client resolved from env/config/instance metadata at startup). This
-                            is the no-surprise default for region-simple providers (Modal, RunPod),
-                            which have a single region and ignore this field.
-                          - explicit list => exactly those regions.
-                        AWS is the exception: it is region-aware with no meaningful single default,
-                        so a `- name: aws` entry MUST list at least one region. That is enforced at
-                        admission by the CEL rule on NodePoolSpec (a per-provider requirement, so it
-                        belongs on the spec where all provider entries are visible, not as a blanket
-                        MinItems that would burden region-simple providers). An "all regions"
-                        wildcard is intentionally NOT supported yet: it only makes sense once the
-                        price-ranking optimizer can expand it against the provider's catalog and
-                        choose among the results, so it is reserved for then. At most 8 regions;
-                        maxLength bounds each entry.
+                        Regions constrains where this provider may place, in the provider's OWN
+                        vocabulary. Region is provider-namespaced — there is no cross-provider region
+                        vocabulary — so it lives here per provider, not on the pool. It is a
+                        CONSTRAINT, not a list of regions to use, and it has three levels:
+                          - omitted/empty => unconstrained: every region the provider serves. For a
+                            region-simple provider (Modal) this means "send no region and let the
+                            provider place freely", which is also its widest and cheapest mode.
+                          - a geography GROUP token ("us", "eu", "ap", ...) => that geography's
+                            regions. This is the recommended way to ask for breadth with a data
+                            residency boundary.
+                          - a literal region name ("us-east-1" on AWS, "us-east" on Modal) => exactly
+                            that region.
+                        The provider resolves which level a value is, since only it knows its own
+                        geography (see provider.Provider's ExpandRegions). Group tokens are shared
+                        across providers but the regions behind them are not: "eu" is eu-west-1 and
+                        friends on AWS, while London is eu-west-2 there and there is no "uk" group.
+
+                        A value that is not a group token is passed to the provider UNVALIDATED.
+                        Region names change faster than Nebula ships, so an unrecognized one is
+                        forwarded rather than rejected: a genuinely bad name fails at provision time
+                        with the provider's own error, which is better than refusing a region that
+                        launched last week. It is also the escape hatch for AWS opt-in regions, which
+                        no group contains.
+
+                        Unconstrained on a region-aware provider is the widest setting and costs
+                        something: every region becomes a placement candidate to walk on failover, and
+                        every region is swept by the observability poll loop. Prefer a group unless the
+                        workload genuinely needs global reach. There is no cap on the number of entries
+                        (a group already expands to many, so capping the declaration would be
+                        arbitrary); maxLength bounds each entry.
                       items:
                         maxLength: 32
                         type: string
-                      maxItems: 8
                       type: array
                     weight:
                       description: |-
@@ -179,9 +194,6 @@ spec:
             x-kubernetes-validations:
             - message: strategy Weighted requires a weight on every provider
               rule: self.strategy != 'Weighted' || self.providers.all(p, has(p.weight))
-            - message: provider aws requires at least one region
-              rule: self.providers.all(p, p.name != 'aws' || (has(p.regions) && size(p.regions)
-                > 0))
           status:
             description: NodePoolStatus surfaces the current placement picture for
               observability.

config/samples/nodepool.yaml

@@ -9,15 +9,16 @@ spec:
   providers:
   - name: aws
     regions:
-    - us-east-1
-    - us-west-1
-    - ap-south-1
-    - ap-northeast-1
-    - eu-central-1
-    - eu-west-1
+    - us
+    - eu
+    - ap
     - ca-central-1
     - sa-east-1
-  # - name: modal
+  - name: modal
+    regions:
+    - us
+    - eu
+    - ap-melbourne
   # - name: runpod
   # Outer axis: try OnDemand on every provider first, fall back to Spot.
   capacityTypes:

docs/add-a-provider.md

@@ -6,8 +6,9 @@ drives everything through the `provider.Provider` interface and a price/availabi
 catalog, so a new provider is an adapter package plus a little wiring — no changes
 to the placement controller, virtual kubelet, or NodeClaim controller.
 
-Use `pkg/provider/modal` (region-simple NeoCloud) and `pkg/provider/aws`
-(region-aware hyperscaler) as references.
+Use `pkg/provider/modal` (NeoCloud, coarse regions, all optional) and
+`pkg/provider/aws` (hyperscaler, a region is mandatory on every call) as
+references.
 
 ## 1. Implement the adapter
 
@@ -24,7 +25,8 @@ Create `pkg/provider/<name>/` and implement `provider.Provider`
 | `List(ctx)` | Every Nebula-owned instance, in as few API calls as possible. This drives the poll loop — preemption is detected by an instance disappearing here. |
 | `Offerings(ctx)` | Price/availability rows for the optimizer (see the catalog below). |
 | `MapAccelerator(canonical, count)` | Translate a canonical accelerator (type + count) to the provider's own id; `ok=false` if unsupported. |
-| `ClassifyProvisionError(err, accel, region)` | Map a Provision failure to the `BlockScope` failover should blocklist (a capacity error → that {accel, tier, region}; an auth/quota error → the whole provider). |
+| `ClassifyProvisionError(err, accel, region)` | Map a Provision failure to the `BlockScope` failover should blocklist. Only an **auth** error widens to the whole provider (`DenyAll`); capacity, quota, and unrecognized errors are all scoped to that {accel, tier, region} so failover can route around one candidate instead of fencing off the provider. Delegate to `provider.ClassifyError` for the shared part and decorate only what is provider-specific (e.g. the region axis). |
+| `ExpandRegions(declared)` | Turn a pool's `regions` into the region candidates placement will walk. `catalog.Base` passes them through unchanged — one candidate per declared region, token used verbatim. Override for **either** of two independent reasons: the tokens are not callable (`pkg/provider/aws` expands the group `us` into every US EC2 region via a static table, since `us` is not a region you can call), or the provider's create **cannot fail over**, in which case splitting shrinks the capacity pool instead of widening it (`pkg/provider/modal` collapses every declared region into ONE candidate). Note Modal's own names already include the group tokens, so it overrides for the *second* reason alone — the two axes are orthogonal. |
 
 The Pod is the single source of truth for the workload shape; `ProvisionRequest`
 carries only what the Pod cannot express (the optimizer's capacity tier and the