design: expected final ateapi.proto for decoupled substrate resources (#368)

docs/design/ateapi-crd-migration.md

@@ -0,0 +1,150 @@
+# Migrating substrate-plane resources from CRDs to ateapi (issue #368)
+
+Companion to [ateapi-resource-api.md](ateapi-resource-api.md), which defines
+the target API. This doc covers how to move the current Kubernetes-backed
+implementation to the Worker-centric model.
+
+## Target shape
+
+Substrate core should manage scheduling intent and observed worker capacity:
+
+- `ActorTemplate` is a global control-plane resource in ateapi.
+- `Actor` is an atespace-scoped tenant/control-plane resource in ateapi.
+- `Worker` is the substrate scheduling primitive and a managed Admin resource.
+- Kubernetes may still have a WorkerPool CRD, Helm value, Deployment wrapper,
+  or autoscaler input, but that belongs to the Kubernetes worker provider.
+
+The split mirrors Kubernetes Nodes and cloud-provider NodePools: core
+substrate schedules against Workers; providers decide how to create and manage
+many Workers.
+
+## Current CRD responsibilities
+
+The existing Kubernetes CRDs combine several responsibilities that need to be
+split apart:
+
+| Current concept | Current responsibility | Target owner |
+|---|---|---|
+| `ActorTemplate` CRD | workload template and scheduling constraints | ateapi `ActorTemplate` |
+| `WorkerPool` CRD | desired worker capacity, Kubernetes placement, common labels | Kubernetes worker provider |
+| Worker pod | actual schedulable capacity | ateapi `Worker` projection/resource |
+
+The key change is that substrate scheduling no longer follows
+`ActorTemplate -> WorkerPool -> Pod`. It follows
+`ActorTemplate/Actor -> Worker`.
+
+## Kubernetes provider flow
+
+The Kubernetes implementation can start with a simple pod watcher:
+
+1. Some Kubernetes mechanism creates worker Pods. This can be a Deployment,
+   Helm-managed workload, an existing WorkerPool CRD, or a future autoscaler.
+2. Worker Pods opt into substrate by setting
+   `pod.ate.dev/is-worker: "true"` from
+   [label-registry.md](label-registry.md).
+3. A worker sync goroutine watches those Pods.
+4. For each matching Pod, the syncer creates or updates an ateapi `Worker`.
+5. When a matching Pod disappears, the syncer deletes or marks the Worker not
+   ready.
+6. The scheduler uses the existing worker cache, but the cache is populated
+   from `WatchWorkers` / `ListWorkers` rather than Kubernetes-specific pod
+   state.
+
+The syncer can initially run inside `ateapi` as a standalone goroutine. It
+should be factored so it can later move into a separate worker-provider
+process without changing the core API.
+
+## Worker data copied from Pods
+
+The first Kubernetes syncer should populate provider-neutral Worker fields:
+
+- `meta.name`: stable substrate Worker name.
+- `meta.labels`: scheduling labels copied under a documented policy.
+- `spec.provider`: `"kubernetes"`.
+- `spec.provider_id`: Kubernetes Pod UID or namespace/name for correlation.
+- `spec.address`: routable ateom endpoint.
+- `spec.sandbox_class`: runtime class advertised by this Worker.
+- `status.state`: ready/not-ready/draining.
+- `status.slots_total` and `status.slots_available`: reported capacity.
+- `status.last_heartbeat_time`: last observed provider update.
+
+Open label policy: copy all Pod labels, only substrate-owned labels, or a
+configured allowlist. Because selectors depend on Worker labels, copied labels
+become API contract and should be documented before broad use.
+
+## Kubernetes WorkerPool
+
+WorkerPool remains useful as a Kubernetes/provider abstraction, just not as a
+portable substrate API resource.
+
+Provider-level WorkerPool can own Kubernetes-specific concerns:
+
+- desired replica count or autoscaling policy
+- pod template
+- node selectors, tolerations, affinity, resources, priority class
+- common labels copied onto produced Workers
+- future desired distributions of worker sizes, e.g. `4x2Gi`, `8x4Gi`,
+  `12x8Gi`
+
+Substrate core should not depend on that shape. If the Kubernetes provider
+uses WorkerPool internally, it reconciles WorkerPool to Pods; the pod syncer
+then reconciles Pods to ateapi Workers.
+
+## Behavior deltas
+
+1. **Scheduling uses Workers directly.** Selectors match
+   `Worker.meta.labels`, and sandbox compatibility matches
+   `ActorTemplate.spec.sandbox_class` to `Worker.spec.sandbox_class`.
+2. **Worker is no longer debug-only.** It becomes the managed Admin resource
+   that worker providers create/update/delete.
+3. **Kubernetes fields leave core ateapi.** Pod namespace/name/UID, node name,
+   tolerations, affinity, and resource requirements are provider details.
+   Correlation can live in `Worker.spec.provider_id`; placement stays in the
+   provider.
+4. **Process snapshots, including goldens, are deferred.** P0 should work
+   without golden snapshots. Re-enabling them requires a compatibility model
+   across runtime version, architecture, kernel/hardware shape, and possibly N
+   goldens per template.
+5. **Delete returns the resource.** Standard methods return the deleted
+   resource instead of `Empty`.
+6. **Required update_mask.** Update without a mask returns
+   `INVALID_ARGUMENT`; accepted paths are defined by the API.
+7. **Watch SYNCED event.** Watch streams deliver initial state, `SYNCED`, then
+   incremental events.
+
+## Validation migration
+
+Validation currently expressed as CEL/OpenAPI on CRDs moves into ateapi
+handlers or provider controllers, depending on ownership:
+
+| Validation | Target owner |
+|---|---|
+| DNS-1123 names for ateapi resources | ateapi handlers |
+| ActorTemplate spec immutability | no UpdateActorTemplate method |
+| container images pinned by digest | ateapi handlers |
+| volume mount-path rules | ateapi handlers |
+| `snapshots_config.on_commit` subset of `on_pause` | ateapi handlers |
+| Worker labels and selector syntax | ateapi handlers |
+| Worker status/capacity invariants | ateapi handlers |
+| Kubernetes namespace, pod template, tolerations, affinity, resources | Kubernetes provider |
+
+## Implementation migration path
+
+1. Update the draft proto to expose Worker CRUD/watch.
+2. Update the store to key Workers by provider-neutral resource identity
+   instead of Kubernetes namespace/pool/pod.
+3. Convert `cmd/ateapi/internal/workercache` to consume `ListWorkers` and
+   `WatchWorkers` over managed Worker records.
+4. Add a Kubernetes worker sync goroutine that watches Pods labeled
+   `pod.ate.dev/is-worker=true` and writes Worker records through the same
+   store/API path as any future provider.
+5. Change scheduling to select Workers directly using Worker labels,
+   `sandbox_class`, readiness, and capacity.
+6. Remove Actor references to WorkerPool; record the assigned Worker instead.
+7. Move Kubernetes WorkerPool reconciliation out of substrate core. It may
+   stay as a Kubernetes provider controller or be replaced by plain
+   Deployments for the first cut.
+8. Cut clients and demos to create ActorTemplates/Actors through ateapi and to
+   inspect Workers through Admin APIs.
+9. Delete the old CRD-watching/projector path from core once the Kubernetes
+   provider path owns pod-to-Worker reconciliation.

docs/design/ateapi-resource-api.md

@@ -0,0 +1,174 @@
+# ateapi resource API (issue #368)
+
+Proto: [`pkg/proto/ateapipb/ateapi.proto`](../../pkg/proto/ateapipb/ateapi.proto)
+
+This is the target API direction for moving substrate scheduling resources out
+of Kubernetes CRDs into the substrate API
+([#368](https://github.com/agent-substrate/substrate/issues/368)). Substrate
+core understands Workers, while WorkerPool is a provider-level capacity
+abstraction, similar to Kubernetes understanding Nodes while cloud providers
+manage NodePools. The transition from the Kubernetes-backed implementation is
+covered separately in
+[ateapi-crd-migration.md](ateapi-crd-migration.md).
+
+The API follows the draft
+[API style guide](https://github.com/agent-substrate/substrate/pull/351);
+deliberate divergences are listed at the [end](#style-guide-conformance).
+
+## Resource model
+
+| Resource | Scope | Service | Mutable? | Notes |
+|---|---|---|---|---|
+| Atespace | global | Control | no | isolation boundary |
+| ActorTemplate | global | Control | no (immutable spec) | + Watch |
+| Actor | atespace | Control | `worker_selector` only | + Suspend/Pause/Resume |
+| Worker | global | Admin | yes | provider-owned capacity; + Watch |
+
+All managed resources carry `ResourceMetadata meta = 1` (atespace, name,
+uid, version, create_time, update_time, labels) and use the standard method
+shapes from the style guide: Get/Create/Update return the bare resource,
+Delete returns the deleted resource, List paginates with
+`page_size`/`page_token`, identity travels as `ObjectRef{atespace, name}`,
+and optimistic concurrency uses `meta.version` (int64, `ABORTED` on
+mismatch, 0 skips).
+
+WorkerPool is not part of the core ateapi resource model in this direction. A
+Kubernetes provider may still have a WorkerPool CRD or config object, but that
+object belongs to the provider implementation. Its job is to create
+Pods/processes that become Workers in substrate.
+
+## Revised direction
+
+1. **Worker is the substrate scheduling primitive.** The scheduler matches
+   Actor/ActorTemplate requirements against Worker labels, runtime class, and
+   readiness/capacity. It does not schedule against WorkerPool.
+2. **WorkerPool is provider-level.** Kubernetes can keep a WorkerPool CRD,
+   Deployment wrapper, Helm value, or autoscaler input, but substrate core
+   only sees the Workers produced by that provider. WorkerPool may later grow
+   useful provider semantics such as desired distributions of worker sizes.
+3. **Worker registration is one-directional.** Worker owners create, update,
+   heartbeat, and delete Workers. The scheduler already talks to ateom for
+   placement, so the worker-owner API does not need assignment watches.
+4. **Kubernetes worker discovery starts with Pods.** A standalone sync
+   goroutine can watch Pods labeled `pod.ate.dev/is-worker=true`, copy the
+   approved metadata onto Worker records, and later move into a distinct
+   process.
+5. **Owned label keys must be documented.** Scheduling labels become API
+   contract, so substrate-owned labels live in a registry:
+   [label-registry.md](label-registry.md).
+6. **Process snapshots and golden snapshots are deferred for P0.** They are
+   valuable, but they imply compatibility constraints across gVisor/runtime
+   version, kernel/hardware shape, architecture, and possibly one golden per
+   template/runtime/hardware version. The first API should work without
+   depending on process snapshots.
+
+## Design decisions
+
+**D1. Versioned package `ateapi.v1alpha1`.** gRPC method paths embed the
+package, so the package name is the API-versioning mechanism.
+
+**D2. Two services with distinct audiences.** `Control` is the tenant
+surface (atespaces, templates, actors); `Admin` is the platform surface
+(workers and debug). Watch RPCs live next to their resource. The split gives
+a future authz story a natural boundary.
+
+**D3. ActorTemplate spec is immutable — no UpdateActorTemplate.** Template
+changes = create a new template and migrate actors. ActorTemplates start
+global-scoped because it is easier to add atespace scoping later than to
+relax it after clients rely on cross-atespace reuse. Delete returns
+`FAILED_PRECONDITION` while any actor references the template.
+
+**D4. Worker is a managed Admin resource.** Worker owners create/update/delete
+Workers through ateapi; substrate stores them, watches them, and schedules
+Actors onto them. The existing internal worker cache remains useful, but its
+input should become Worker CRUD/watch rather than Kubernetes-specific pod
+state.
+
+**D5. WorkerPool policy is provider-owned.** Once WorkerPool is not a core
+resource, tenant access to provider capacity is not modeled as a core ateapi
+object. If a provider-level WorkerPool needs multi-tenant policy, that policy
+belongs to the provider until substrate has a concrete cross-provider access
+model.
+
+**D6. Referential integrity on delete.** Deletes that would orphan
+dependents return `FAILED_PRECONDITION`: atespace↔actors, template↔actors,
+worker↔assigned actors.
+
+**D7. Status is a closed enum, not phase strings or conditions.** Actors,
+templates, and workers use closed state enums plus a human-readable error or
+message field where needed.
+
+**D8. Equality-only `Selector{match_labels}`** for template and per-actor
+worker selectors, matched against Worker labels. The scheduler
+implements equality matching only; set-based expressions can be added as a
+new field if a consumer appears.
+
+**D9. Env vars are a `oneof {value | secret}`** with a plane-neutral
+`SecretRef{name, key, optional}` — value-vs-secret is exclusive by
+construction, and no k8s types leak into the container spec.
+
+**D10. `sandbox_class` is an open string** on templates and workers. It is a
+scheduling compatibility label. Runtime setup remains provider/runtime
+configuration. Workers may technically support multiple sandbox classes; P0
+keeps this scalar and can extend it to a repeated field once scheduling
+semantics for multi-runtime workers are needed.
+
+**D11. Update masks accept top-level paths only.** `update_mask` is
+required (per the style guide); paths one level below the resource root
+(`spec.address`, `status`, `meta.labels`) are accepted, and
+naming a message or map field replaces it wholesale. Unknown, immutable, or
+deeper paths → `INVALID_ARGUMENT`. Servers may widen to deeper paths later
+without breaking clients.
+
+**D12. Watch = snapshot, SYNCED, then incremental events.** Streams
+(templates, workers) deliver an initial snapshot, a `SYNCED` marker, then
+at-least-once CREATED/UPDATED/DELETED events carrying full resource state.
+No resume tokens: consumers are in-cluster caches and re-syncing is cheap;
+`meta.version` lets them discard stale events.
+
+**D13. Worker identity is provider-neutral.** Core identity is
+`ResourceMetadata.name`. Provider details such as Kubernetes namespace, Pod
+name, Pod UID, VM ID, or runtime endpoint are fields on Worker spec/status for
+debugging and connection, not part of the resource key.
+
+**D14. Actor is a managed resource like the rest.** `meta` carries its
+identity; `actor_template` and assigned `worker` are `ObjectRef`s; lifecycle
+verbs (Suspend/Pause/Resume) are custom methods returning the Actor.
+
+**D15. Atespace and SessionIdentity are unchanged** beyond conforming to
+`meta` and standard method shapes. Redesigning them is a non-goal of #368.
+
+## Style-guide conformance
+
+The surface follows PR #351 (identity model, `ResourceMetadata`,
+`ObjectRef`, standard method shapes, required `update_mask`, `version`
+concurrency). Four deliberate divergences remain in this draft:
+
+1. **`labels` in `ResourceMetadata`** — the guide's meta has no labels
+   field, but scheduling requires Worker labels as a selector match target.
+2. **Immutable resources may omit Update** (D3) — the guide implies all
+   five standard methods everywhere.
+3. **Top-level-only mask paths** (D11) — the guide requires the mask but
+   leaves path granularity undefined.
+4. **Empty `atespace` = list across all atespaces** — the guide reads as if
+   `atespace` were required on List; admin/debug flows need cross-atespace
+   listing.
+
+## Open questions
+
+- **Label copy policy**: for Kubernetes-discovered Workers, copy all labels,
+  only `*.ate.dev/*`, or a configured allowlist? This is part of the API
+  contract once selectors depend on it.
+- **Worker heartbeat semantics**: define whether Worker owners use Update,
+  an explicit Heartbeat RPC, or both. CRUD is enough on paper; Heartbeat may
+  be operationally clearer.
+- **Golden snapshots**: process snapshots, including goldens, are deferred.
+  If re-enabled, define the compatibility key (runtime version, architecture,
+  kernel/hardware shape) and whether templates maintain N goldens.
+- **Secrets plane**: `SecretRef` resolves against the worker's environment
+  (a k8s Secret today). Freeze that rule, or design secret distribution
+  before v1beta1?
+- **Pagination defaults**: pick a server default page size (proposal: 500,
+  max 1000) and document it once, centrally.
+- **Watch scope**: watches are global today (atelet caches everything).
+  Add an optional `atespace` filter now, or when a consumer needs it?

docs/design/label-registry.md

@@ -0,0 +1,16 @@
+# Agent Substrate label registry
+
+This document lists label keys Agent Substrate claims as API or integration
+contract. Provider implementations may use other labels internally, but only
+labels listed here should be treated as portable substrate behavior.
+
+## Pod labels
+
+### `pod.ate.dev/is-worker`
+
+Value: `"true"`
+
+Marks a Kubernetes Pod as an Agent Substrate Worker. A Kubernetes worker syncer
+may watch Pods with this label and create/update/delete corresponding Worker
+records in ateapi.
+