diff --git a/docs/design/ateapi-crd-migration.md b/docs/design/ateapi-crd-migration.md new file mode 100644 index 000000000..f5597634f --- /dev/null +++ b/docs/design/ateapi-crd-migration.md @@ -0,0 +1,150 @@ +# Migrating substrate-plane resources from CRDs to ateapi (issue #368) + +Companion to [ateapi-resource-api.md](ateapi-resource-api.md), which defines +the target API. This doc covers how to move the current Kubernetes-backed +implementation to the Worker-centric model. + +## Target shape + +Substrate core should manage scheduling intent and observed worker capacity: + +- `ActorTemplate` is a global control-plane resource in ateapi. +- `Actor` is an atespace-scoped tenant/control-plane resource in ateapi. +- `Worker` is the substrate scheduling primitive and a managed Admin resource. +- Kubernetes may still have a WorkerPool CRD, Helm value, Deployment wrapper, + or autoscaler input, but that belongs to the Kubernetes worker provider. + +The split mirrors Kubernetes Nodes and cloud-provider NodePools: core +substrate schedules against Workers; providers decide how to create and manage +many Workers. + +## Current CRD responsibilities + +The existing Kubernetes CRDs combine several responsibilities that need to be +split apart: + +| Current concept | Current responsibility | Target owner | +|---|---|---| +| `ActorTemplate` CRD | workload template and scheduling constraints | ateapi `ActorTemplate` | +| `WorkerPool` CRD | desired worker capacity, Kubernetes placement, common labels | Kubernetes worker provider | +| Worker pod | actual schedulable capacity | ateapi `Worker` projection/resource | + +The key change is that substrate scheduling no longer follows +`ActorTemplate -> WorkerPool -> Pod`. It follows +`ActorTemplate/Actor -> Worker`. + +## Kubernetes provider flow + +The Kubernetes implementation can start with a simple pod watcher: + +1. Some Kubernetes mechanism creates worker Pods. This can be a Deployment, + Helm-managed workload, an existing WorkerPool CRD, or a future autoscaler. +2. Worker Pods opt into substrate by setting + `pod.ate.dev/is-worker: "true"` from + [label-registry.md](label-registry.md). +3. A worker sync goroutine watches those Pods. +4. For each matching Pod, the syncer creates or updates an ateapi `Worker`. +5. When a matching Pod disappears, the syncer deletes or marks the Worker not + ready. +6. The scheduler uses the existing worker cache, but the cache is populated + from `WatchWorkers` / `ListWorkers` rather than Kubernetes-specific pod + state. + +The syncer can initially run inside `ateapi` as a standalone goroutine. It +should be factored so it can later move into a separate worker-provider +process without changing the core API. + +## Worker data copied from Pods + +The first Kubernetes syncer should populate provider-neutral Worker fields: + +- `meta.name`: stable substrate Worker name. +- `meta.labels`: scheduling labels copied under a documented policy. +- `spec.provider`: `"kubernetes"`. +- `spec.provider_id`: Kubernetes Pod UID or namespace/name for correlation. +- `spec.address`: routable ateom endpoint. +- `spec.sandbox_class`: runtime class advertised by this Worker. +- `status.state`: ready/not-ready/draining. +- `status.slots_total` and `status.slots_available`: reported capacity. +- `status.last_heartbeat_time`: last observed provider update. + +Open label policy: copy all Pod labels, only substrate-owned labels, or a +configured allowlist. Because selectors depend on Worker labels, copied labels +become API contract and should be documented before broad use. + +## Kubernetes WorkerPool + +WorkerPool remains useful as a Kubernetes/provider abstraction, just not as a +portable substrate API resource. + +Provider-level WorkerPool can own Kubernetes-specific concerns: + +- desired replica count or autoscaling policy +- pod template +- node selectors, tolerations, affinity, resources, priority class +- common labels copied onto produced Workers +- future desired distributions of worker sizes, e.g. `4x2Gi`, `8x4Gi`, + `12x8Gi` + +Substrate core should not depend on that shape. If the Kubernetes provider +uses WorkerPool internally, it reconciles WorkerPool to Pods; the pod syncer +then reconciles Pods to ateapi Workers. + +## Behavior deltas + +1. **Scheduling uses Workers directly.** Selectors match + `Worker.meta.labels`, and sandbox compatibility matches + `ActorTemplate.spec.sandbox_class` to `Worker.spec.sandbox_class`. +2. **Worker is no longer debug-only.** It becomes the managed Admin resource + that worker providers create/update/delete. +3. **Kubernetes fields leave core ateapi.** Pod namespace/name/UID, node name, + tolerations, affinity, and resource requirements are provider details. + Correlation can live in `Worker.spec.provider_id`; placement stays in the + provider. +4. **Process snapshots, including goldens, are deferred.** P0 should work + without golden snapshots. Re-enabling them requires a compatibility model + across runtime version, architecture, kernel/hardware shape, and possibly N + goldens per template. +5. **Delete returns the resource.** Standard methods return the deleted + resource instead of `Empty`. +6. **Required update_mask.** Update without a mask returns + `INVALID_ARGUMENT`; accepted paths are defined by the API. +7. **Watch SYNCED event.** Watch streams deliver initial state, `SYNCED`, then + incremental events. + +## Validation migration + +Validation currently expressed as CEL/OpenAPI on CRDs moves into ateapi +handlers or provider controllers, depending on ownership: + +| Validation | Target owner | +|---|---| +| DNS-1123 names for ateapi resources | ateapi handlers | +| ActorTemplate spec immutability | no UpdateActorTemplate method | +| container images pinned by digest | ateapi handlers | +| volume mount-path rules | ateapi handlers | +| `snapshots_config.on_commit` subset of `on_pause` | ateapi handlers | +| Worker labels and selector syntax | ateapi handlers | +| Worker status/capacity invariants | ateapi handlers | +| Kubernetes namespace, pod template, tolerations, affinity, resources | Kubernetes provider | + +## Implementation migration path + +1. Update the draft proto to expose Worker CRUD/watch. +2. Update the store to key Workers by provider-neutral resource identity + instead of Kubernetes namespace/pool/pod. +3. Convert `cmd/ateapi/internal/workercache` to consume `ListWorkers` and + `WatchWorkers` over managed Worker records. +4. Add a Kubernetes worker sync goroutine that watches Pods labeled + `pod.ate.dev/is-worker=true` and writes Worker records through the same + store/API path as any future provider. +5. Change scheduling to select Workers directly using Worker labels, + `sandbox_class`, readiness, and capacity. +6. Remove Actor references to WorkerPool; record the assigned Worker instead. +7. Move Kubernetes WorkerPool reconciliation out of substrate core. It may + stay as a Kubernetes provider controller or be replaced by plain + Deployments for the first cut. +8. Cut clients and demos to create ActorTemplates/Actors through ateapi and to + inspect Workers through Admin APIs. +9. Delete the old CRD-watching/projector path from core once the Kubernetes + provider path owns pod-to-Worker reconciliation. diff --git a/docs/design/ateapi-resource-api.md b/docs/design/ateapi-resource-api.md new file mode 100644 index 000000000..efa30b864 --- /dev/null +++ b/docs/design/ateapi-resource-api.md @@ -0,0 +1,174 @@ +# ateapi resource API (issue #368) + +Proto: [`pkg/proto/ateapipb/ateapi.proto`](../../pkg/proto/ateapipb/ateapi.proto) + +This is the target API direction for moving substrate scheduling resources out +of Kubernetes CRDs into the substrate API +([#368](https://github.com/agent-substrate/substrate/issues/368)). Substrate +core understands Workers, while WorkerPool is a provider-level capacity +abstraction, similar to Kubernetes understanding Nodes while cloud providers +manage NodePools. The transition from the Kubernetes-backed implementation is +covered separately in +[ateapi-crd-migration.md](ateapi-crd-migration.md). + +The API follows the draft +[API style guide](https://github.com/agent-substrate/substrate/pull/351); +deliberate divergences are listed at the [end](#style-guide-conformance). + +## Resource model + +| Resource | Scope | Service | Mutable? | Notes | +|---|---|---|---|---| +| Atespace | global | Control | no | isolation boundary | +| ActorTemplate | global | Control | no (immutable spec) | + Watch | +| Actor | atespace | Control | `worker_selector` only | + Suspend/Pause/Resume | +| Worker | global | Admin | yes | provider-owned capacity; + Watch | + +All managed resources carry `ResourceMetadata meta = 1` (atespace, name, +uid, version, create_time, update_time, labels) and use the standard method +shapes from the style guide: Get/Create/Update return the bare resource, +Delete returns the deleted resource, List paginates with +`page_size`/`page_token`, identity travels as `ObjectRef{atespace, name}`, +and optimistic concurrency uses `meta.version` (int64, `ABORTED` on +mismatch, 0 skips). + +WorkerPool is not part of the core ateapi resource model in this direction. A +Kubernetes provider may still have a WorkerPool CRD or config object, but that +object belongs to the provider implementation. Its job is to create +Pods/processes that become Workers in substrate. + +## Revised direction + +1. **Worker is the substrate scheduling primitive.** The scheduler matches + Actor/ActorTemplate requirements against Worker labels, runtime class, and + readiness/capacity. It does not schedule against WorkerPool. +2. **WorkerPool is provider-level.** Kubernetes can keep a WorkerPool CRD, + Deployment wrapper, Helm value, or autoscaler input, but substrate core + only sees the Workers produced by that provider. WorkerPool may later grow + useful provider semantics such as desired distributions of worker sizes. +3. **Worker registration is one-directional.** Worker owners create, update, + heartbeat, and delete Workers. The scheduler already talks to ateom for + placement, so the worker-owner API does not need assignment watches. +4. **Kubernetes worker discovery starts with Pods.** A standalone sync + goroutine can watch Pods labeled `pod.ate.dev/is-worker=true`, copy the + approved metadata onto Worker records, and later move into a distinct + process. +5. **Owned label keys must be documented.** Scheduling labels become API + contract, so substrate-owned labels live in a registry: + [label-registry.md](label-registry.md). +6. **Process snapshots and golden snapshots are deferred for P0.** They are + valuable, but they imply compatibility constraints across gVisor/runtime + version, kernel/hardware shape, architecture, and possibly one golden per + template/runtime/hardware version. The first API should work without + depending on process snapshots. + +## Design decisions + +**D1. Versioned package `ateapi.v1alpha1`.** gRPC method paths embed the +package, so the package name is the API-versioning mechanism. + +**D2. Two services with distinct audiences.** `Control` is the tenant +surface (atespaces, templates, actors); `Admin` is the platform surface +(workers and debug). Watch RPCs live next to their resource. The split gives +a future authz story a natural boundary. + +**D3. ActorTemplate spec is immutable — no UpdateActorTemplate.** Template +changes = create a new template and migrate actors. ActorTemplates start +global-scoped because it is easier to add atespace scoping later than to +relax it after clients rely on cross-atespace reuse. Delete returns +`FAILED_PRECONDITION` while any actor references the template. + +**D4. Worker is a managed Admin resource.** Worker owners create/update/delete +Workers through ateapi; substrate stores them, watches them, and schedules +Actors onto them. The existing internal worker cache remains useful, but its +input should become Worker CRUD/watch rather than Kubernetes-specific pod +state. + +**D5. WorkerPool policy is provider-owned.** Once WorkerPool is not a core +resource, tenant access to provider capacity is not modeled as a core ateapi +object. If a provider-level WorkerPool needs multi-tenant policy, that policy +belongs to the provider until substrate has a concrete cross-provider access +model. + +**D6. Referential integrity on delete.** Deletes that would orphan +dependents return `FAILED_PRECONDITION`: atespace↔actors, template↔actors, +worker↔assigned actors. + +**D7. Status is a closed enum, not phase strings or conditions.** Actors, +templates, and workers use closed state enums plus a human-readable error or +message field where needed. + +**D8. Equality-only `Selector{match_labels}`** for template and per-actor +worker selectors, matched against Worker labels. The scheduler +implements equality matching only; set-based expressions can be added as a +new field if a consumer appears. + +**D9. Env vars are a `oneof {value | secret}`** with a plane-neutral +`SecretRef{name, key, optional}` — value-vs-secret is exclusive by +construction, and no k8s types leak into the container spec. + +**D10. `sandbox_class` is an open string** on templates and workers. It is a +scheduling compatibility label. Runtime setup remains provider/runtime +configuration. Workers may technically support multiple sandbox classes; P0 +keeps this scalar and can extend it to a repeated field once scheduling +semantics for multi-runtime workers are needed. + +**D11. Update masks accept top-level paths only.** `update_mask` is +required (per the style guide); paths one level below the resource root +(`spec.address`, `status`, `meta.labels`) are accepted, and +naming a message or map field replaces it wholesale. Unknown, immutable, or +deeper paths → `INVALID_ARGUMENT`. Servers may widen to deeper paths later +without breaking clients. + +**D12. Watch = snapshot, SYNCED, then incremental events.** Streams +(templates, workers) deliver an initial snapshot, a `SYNCED` marker, then +at-least-once CREATED/UPDATED/DELETED events carrying full resource state. +No resume tokens: consumers are in-cluster caches and re-syncing is cheap; +`meta.version` lets them discard stale events. + +**D13. Worker identity is provider-neutral.** Core identity is +`ResourceMetadata.name`. Provider details such as Kubernetes namespace, Pod +name, Pod UID, VM ID, or runtime endpoint are fields on Worker spec/status for +debugging and connection, not part of the resource key. + +**D14. Actor is a managed resource like the rest.** `meta` carries its +identity; `actor_template` and assigned `worker` are `ObjectRef`s; lifecycle +verbs (Suspend/Pause/Resume) are custom methods returning the Actor. + +**D15. Atespace and SessionIdentity are unchanged** beyond conforming to +`meta` and standard method shapes. Redesigning them is a non-goal of #368. + +## Style-guide conformance + +The surface follows PR #351 (identity model, `ResourceMetadata`, +`ObjectRef`, standard method shapes, required `update_mask`, `version` +concurrency). Four deliberate divergences remain in this draft: + +1. **`labels` in `ResourceMetadata`** — the guide's meta has no labels + field, but scheduling requires Worker labels as a selector match target. +2. **Immutable resources may omit Update** (D3) — the guide implies all + five standard methods everywhere. +3. **Top-level-only mask paths** (D11) — the guide requires the mask but + leaves path granularity undefined. +4. **Empty `atespace` = list across all atespaces** — the guide reads as if + `atespace` were required on List; admin/debug flows need cross-atespace + listing. + +## Open questions + +- **Label copy policy**: for Kubernetes-discovered Workers, copy all labels, + only `*.ate.dev/*`, or a configured allowlist? This is part of the API + contract once selectors depend on it. +- **Worker heartbeat semantics**: define whether Worker owners use Update, + an explicit Heartbeat RPC, or both. CRUD is enough on paper; Heartbeat may + be operationally clearer. +- **Golden snapshots**: process snapshots, including goldens, are deferred. + If re-enabled, define the compatibility key (runtime version, architecture, + kernel/hardware shape) and whether templates maintain N goldens. +- **Secrets plane**: `SecretRef` resolves against the worker's environment + (a k8s Secret today). Freeze that rule, or design secret distribution + before v1beta1? +- **Pagination defaults**: pick a server default page size (proposal: 500, + max 1000) and document it once, centrally. +- **Watch scope**: watches are global today (atelet caches everything). + Add an optional `atespace` filter now, or when a consumer needs it? diff --git a/docs/design/label-registry.md b/docs/design/label-registry.md new file mode 100644 index 000000000..5055ad3a8 --- /dev/null +++ b/docs/design/label-registry.md @@ -0,0 +1,16 @@ +# Agent Substrate label registry + +This document lists label keys Agent Substrate claims as API or integration +contract. Provider implementations may use other labels internally, but only +labels listed here should be treated as portable substrate behavior. + +## Pod labels + +### `pod.ate.dev/is-worker` + +Value: `"true"` + +Marks a Kubernetes Pod as an Agent Substrate Worker. A Kubernetes worker syncer +may watch Pods with this label and create/update/delete corresponding Worker +records in ateapi. + diff --git a/pkg/proto/ateapipb/ateapi.proto b/pkg/proto/ateapipb/ateapi.proto index 249d13c7d..7ed6419d1 100644 --- a/pkg/proto/ateapipb/ateapi.proto +++ b/pkg/proto/ateapipb/ateapi.proto @@ -12,64 +12,258 @@ // See the License for the specific language governing permissions and // limitations under the License. -// TODO: Here and everywhere, s/Session/Actor to align with the glossary. +// DESIGN DRAFT (issue #368) — expected final shape of the ateapi surface +// after decoupling substrate-plane resources from Kubernetes CRDs. +// For review only: generated code and server implementation intentionally +// lag this file until the shape is agreed. +// +// The API follows docs/api-style-guide.md (PR #351); deliberate +// divergences are listed in docs/design/ateapi-resource-api.md +// ("Style-guide conformance"). +// +// Because the package renames from `ateapi` to `ateapi.v1alpha1`, every +// message here is a new proto type: field numbers are assigned fresh and +// carry no wire-compatibility obligation to the previous proto. Migration +// is a coordinated client cutover, not in-place field evolution. +// +// Error model (canonical gRPC codes): +// - INVALID_ARGUMENT: malformed names, missing required fields, +// empty/absent update_mask, unknown mask paths, +// negative page_size, scope violations +// (e.g. atespace set on a global resource). +// - NOT_FOUND: resource does not exist. +// - ALREADY_EXISTS: Create with an existing (atespace, name). +// - ABORTED: version guard mismatch (client echoed a non-zero +// meta.version / request version that is stale). +// - FAILED_PRECONDITION: referential-integrity or state-machine +// violations (delete a referenced template/worker, +// delete a non-empty atespace, resume a +// non-suspended actor, ...). syntax = "proto3"; -package ateapi; +package ateapi.v1alpha1; +import "google/protobuf/field_mask.proto"; +import "google/protobuf/timestamp.proto"; + +// Go package layout (ateapipb vs ateapipb/v1alpha1) is decided at +// implementation time; out of scope for this design. option go_package = "github.com/agent-substrate/substrate/pkg/proto/ateapipb"; +// --------------------------------------------------------------------------- +// Services +// --------------------------------------------------------------------------- -// Control is the primary RPC interface for Agentic Substrate. +// Control is the user-plane API: everything a tenant needs to define +// templates and run actors inside an atespace. service Control { + // --- Atespace (global-scoped, immutable: no Update) --- + + // Create an Atespace. + rpc CreateAtespace(CreateAtespaceRequest) returns (Atespace) {} + + // Get an Atespace. + rpc GetAtespace(GetAtespaceRequest) returns (Atespace) {} + + // List Atespaces. + rpc ListAtespaces(ListAtespacesRequest) returns (ListAtespacesResponse) {} + + // Delete an empty Atespace. + // Returns FAILED_PRECONDITION if any actors remain. + rpc DeleteAtespace(DeleteAtespaceRequest) returns (Atespace) {} + + // --- ActorTemplate (global-scoped, immutable spec: no Update) --- + + // Create an ActorTemplate. The spec is immutable after creation: to change + // a template, create a new one and migrate actors. + rpc CreateActorTemplate(CreateActorTemplateRequest) returns (ActorTemplate) {} + + // Get an ActorTemplate. + rpc GetActorTemplate(GetActorTemplateRequest) returns (ActorTemplate) {} + + // List ActorTemplates. + rpc ListActorTemplates(ListActorTemplatesRequest) returns (ListActorTemplatesResponse) {} + + // Delete an ActorTemplate. + // Returns FAILED_PRECONDITION while any actor references it. + rpc DeleteActorTemplate(DeleteActorTemplateRequest) returns (ActorTemplate) {} + + // Watch ActorTemplate changes after an initial snapshot of current state. + // Delivery is at-least-once; events carry full resource state. A SYNCED + // event marks the end of the initial snapshot. + rpc WatchActorTemplates(WatchActorTemplatesRequest) returns (stream WatchActorTemplatesResponse) {} + + // --- Actor (atespace-scoped) --- + + // Create a new Actor deriving from an ActorTemplate. + rpc CreateActor(CreateActorRequest) returns (Actor) {} + // Get an Actor. - rpc GetActor(GetActorRequest) returns (GetActorResponse) {} + rpc GetActor(GetActorRequest) returns (Actor) {} + + // List Actors. + rpc ListActors(ListActorsRequest) returns (ListActorsResponse) {} - // Create a new Actor deriving from a given ActorTemplate. - rpc CreateActor(CreateActorRequest) returns (CreateActorResponse) {} + // Update mutable Actor fields (currently only worker_selector). + // Changes take effect on the next ResumeActor call. + rpc UpdateActor(UpdateActorRequest) returns (Actor) {} - // Update mutable fields on an existing Actor. - rpc UpdateActor(UpdateActorRequest) returns (UpdateActorResponse) {} + // Delete an Actor. Returns FAILED_PRECONDITION unless the actor is + // suspended. + rpc DeleteActor(DeleteActorRequest) returns (Actor) {} - // Suspend a given actor to a new snapshot. - rpc SuspendActor(SuspendActorRequest) returns (SuspendActorResponse) {} + // Suspend an actor to a new snapshot. + rpc SuspendActor(SuspendActorRequest) returns (Actor) {} - // Pause a given actor and keep its snapshots on node VM. - rpc PauseActor(PauseActorRequest) returns (PauseActorResponse) {} + // Pause an actor, keeping its snapshots on the node VM. + rpc PauseActor(PauseActorRequest) returns (Actor) {} // Resume an actor from its latest snapshot. - rpc ResumeActor(ResumeActorRequest) returns (ResumeActorResponse) {} + rpc ResumeActor(ResumeActorRequest) returns (Actor) {} +} - // Delete an actor. Only suspended actors can be deleted. - rpc DeleteActor(DeleteActorRequest) returns (DeleteActorResponse) {} +// Admin is the platform-plane API: worker capacity and debug surfaces. +// Intended for platform operators and worker providers, not tenants. +service Admin { + // --- Worker (global-scoped) --- - // List all workers currently reflected in redis. - rpc ListWorkers(ListWorkersRequest) returns (ListWorkersResponse) {} + // Create a Worker. Worker providers call this when capacity becomes + // available to substrate. + rpc CreateWorker(CreateWorkerRequest) returns (Worker) {} - // List all actors currently reflected in redis. - rpc ListActors(ListActorsRequest) returns (ListActorsResponse) {} + // Get a Worker. + rpc GetWorker(GetWorkerRequest) returns (Worker) {} - // Create a new Atespace. Substrate-native, stored in Redis. - rpc CreateAtespace(CreateAtespaceRequest) returns (CreateAtespaceResponse) {} + // List Workers. + rpc ListWorkers(ListWorkersRequest) returns (ListWorkersResponse) {} - // Get an Atespace by name. - rpc GetAtespace(GetAtespaceRequest) returns (GetAtespaceResponse) {} + // Update mutable Worker fields. Worker providers may use this for + // heartbeat-style status updates. + rpc UpdateWorker(UpdateWorkerRequest) returns (Worker) {} - // List all Atespaces. - rpc ListAtespaces(ListAtespacesRequest) returns (ListAtespacesResponse) {} + // Delete a Worker. Returns FAILED_PRECONDITION while any actor is assigned + // to it. + rpc DeleteWorker(DeleteWorkerRequest) returns (Worker) {} - // Delete an empty Atespace. Rejects (FailedPrecondition) if any actors remain. - rpc DeleteAtespace(DeleteAtespaceRequest) returns (DeleteAtespaceResponse) {} + // Watch Worker changes after an initial snapshot of current state. + // Same delivery semantics as WatchActorTemplates. + rpc WatchWorkers(WatchWorkersRequest) returns (stream WatchWorkersResponse) {} // Debugging: drop all data from the ate database. rpc DebugClear(DebugClearRequest) returns (DebugClearResponse) {} } +// --------------------------------------------------------------------------- +// Shared types +// --------------------------------------------------------------------------- + +// ResourceMetadata holds the standard fields common to all managed +// resources. Always field 1 of the resource message, named `meta`. +// See docs/api-style-guide.md section 6. +message ResourceMetadata { + // The atespace this resource belongs to. Must be empty for global-scoped + // resource types (Atespace, ActorTemplate, Worker) and set for + // atespace-scoped types (Actor). + string atespace = 1; + + // The name of this resource, unique within its scope. Must be a valid + // RFC-1123 DNS label. Caller-specified on Create. + string name = 2; + + // Server-assigned UUID4, immutable for the life of the resource. + string uid = 3; + + // Monotonically increasing version: starts at 1 on creation and + // increments by 1 on every mutation, including system-internal ones. + // Output-only; echo a non-zero value back on Update/Delete to guard + // against concurrent modification (mismatch returns ABORTED). + int64 version = 4; + + // When the resource was created. Output-only. + google.protobuf.Timestamp create_time = 5; + + // When the resource was last updated. Output-only. + google.protobuf.Timestamp update_time = 6; + + // labels are user-defined key/value pairs used for organization and + // selection. Worker labels are matched by template and actor + // worker_selectors at scheduling time. Mutable on resources that support + // Update. (Not yet in the draft style guide's ResourceMetadata.) + map labels = 7; +} + +// ObjectRef is a pointer to a resource: request identity and +// cross-resource references. For references to global-scoped resources, +// atespace must be empty. +message ObjectRef { + string atespace = 1; + string name = 2; +} + +// Selector matches workers by their meta.labels. +// Only equality-based matching is supported (all pairs must match). +// An empty selector matches every worker. +message Selector { + map match_labels = 1; +} + +// ResourceEventType classifies watch stream events. +enum ResourceEventType { + RESOURCE_EVENT_TYPE_UNSPECIFIED = 0; + RESOURCE_EVENT_TYPE_CREATED = 1; + RESOURCE_EVENT_TYPE_UPDATED = 2; + RESOURCE_EVENT_TYPE_DELETED = 3; + // SYNCED is sent exactly once, after the initial snapshot has been fully + // delivered. It carries no resource. Consumers can use it to know their + // local cache is complete before acting. + RESOURCE_EVENT_TYPE_SYNCED = 4; +} + +// --------------------------------------------------------------------------- +// Atespace +// --------------------------------------------------------------------------- + +// Atespace is the isolation boundary actors are created into. +// Global-scoped; immutable (no Update method). +message Atespace { + ResourceMetadata meta = 1; +} + +message CreateAtespaceRequest { + // atespace.meta.name is required; atespace.meta.atespace must be empty. + Atespace atespace = 1; +} + +message GetAtespaceRequest { + ObjectRef atespace = 1; +} + +message ListAtespacesRequest { + // Maximum number of results to return; see style guide 3.2 for paging + // semantics (applies to all List methods in this file). + int32 page_size = 1; + string page_token = 2; +} +message ListAtespacesResponse { + repeated Atespace atespaces = 1; + string next_page_token = 2; +} + +message DeleteAtespaceRequest { + ObjectRef atespace = 1; + // Optional freshness guard; 0 = skip check. + int64 version = 2; +} + +// --------------------------------------------------------------------------- +// Actor +// --------------------------------------------------------------------------- + enum SnapshotType { SNAPSHOT_TYPE_UNSPECIFIED = 0; SNAPSHOT_TYPE_LOCAL = 1; - SNAPSHOT_TYPE_EXTERNAL = 2; + SNAPSHOT_TYPE_EXTERNAL = 2; } message ExternalSnapshotInfo { @@ -78,7 +272,7 @@ message ExternalSnapshotInfo { message LocalSnapshotInfo { string snapshot_prefix = 1; - // Node VMs that have local snapshots for this actor, while it's PAUSED. + // Node VMs that have local snapshots for this actor while it is PAUSED. repeated string node_vms_with_local_snapshots = 2; } @@ -90,18 +284,15 @@ message SnapshotInfo { } } -// Selector matches worker pools by label. -// Only equality-based matching is supported. -message Selector { - map match_labels = 1; -} - +// Actor is a running (or suspended) instance of an ActorTemplate. +// Atespace-scoped. message Actor { - string actor_id = 1; - int64 version = 2; + ResourceMetadata meta = 1; - string actor_template_namespace = 3; - string actor_template_name = 4; + // The ActorTemplate this actor derives from. Set at creation; immutable. + // ActorTemplates are global resources, so actor_template.atespace must be + // empty. + ObjectRef actor_template = 2; enum Status { STATUS_UNSPECIFIED = 0; @@ -112,207 +303,348 @@ message Actor { STATUS_PAUSING = 5; STATUS_PAUSED = 6; } - Status status = 5; - - string ateom_pod_namespace = 6; - string ateom_pod_name = 7; - string ateom_pod_ip = 8; - - // last_snapshot - reserved 9; - string in_progress_snapshot = 10; - string ateom_pod_uid = 11; - SnapshotInfo latest_snapshot_info = 12; + // Output-only lifecycle state. + Status status = 3; // worker_selector is the per-actor placement constraint. The scheduler - // evaluates the AND of this selector and the template's workerSelector to - // find eligible pools. Set at CreateActor; may be updated at any time via - // UpdateActor. Changes take effect on the next ResumeActor call. - Selector worker_selector = 13; + // evaluates the AND of this selector and the template's worker_selector + // to find eligible workers. Set at CreateActor; mutable via UpdateActor. + // Changes take effect on the next ResumeActor call. + Selector worker_selector = 4; - // worker_pool_name is the name of the WorkerPool that owns the currently - // assigned worker, in the same namespace as ateom_pod_namespace (pool and - // pod namespaces always match). Set by the scheduler at assignment time - // and cleared when the worker is freed. Needed to release the worker on - // suspend/pause since eligibility is no longer a single fixed pool - // reference on the ActorTemplate. - string worker_pool_name = 14; + // The currently assigned Worker. Output-only: set by the scheduler at + // assignment time, cleared when the worker is freed on suspend/pause. + ObjectRef worker = 5; - // The atespace this actor belongs to. Part of the actor's - // resource identity; folded into the Redis key as actor::. - string atespace = 15; + // Output-only snapshot bookkeeping. + string in_progress_snapshot = 6; + SnapshotInfo latest_snapshot_info = 7; } -// Atespace is the isolation boundary an Actor is created into. -message Atespace { - string name = 1; +message CreateActorRequest { + // The actor to create. Caller sets meta.atespace, meta.name (a valid + // DNS-1123 label), actor_template, and optionally worker_selector. + // All other fields are output-only and must be unset. + Actor actor = 1; } -// ActorRef identifies an actor: a name is only unique within its atespace. -message ActorRef { - string atespace = 1; - string name = 2; +message GetActorRequest { + ObjectRef actor = 1; } -message CreateAtespaceRequest { - string name = 1; +// List actors with pagination. This operation provides "soft" guarantees: +// actors may be missed or duplicated if the system state changes during +// pagination. +message ListActorsRequest { + // The atespace to list actors from. Empty lists across all atespaces. + string atespace = 1; + int32 page_size = 2; + string page_token = 3; } -message CreateAtespaceResponse { - Atespace atespace = 1; +message ListActorsResponse { + repeated Actor actors = 1; + string next_page_token = 2; } -message GetAtespaceRequest { - string name = 1; +message UpdateActorRequest { + // actor.meta.atespace and actor.meta.name identify the resource. + // A non-zero actor.meta.version acts as a freshness guard. + Actor actor = 1; + + // Required. The only updatable path today is "worker_selector"; any + // other path returns INVALID_ARGUMENT. + google.protobuf.FieldMask update_mask = 2; } -message GetAtespaceResponse { - Atespace atespace = 1; + +message DeleteActorRequest { + ObjectRef actor = 1; + // Optional freshness guard; 0 = skip check. + int64 version = 2; } -message ListAtespacesRequest {} -message ListAtespacesResponse { - repeated Atespace atespaces = 1; +message SuspendActorRequest { + ObjectRef actor = 1; } -message DeleteAtespaceRequest { - string name = 1; +message PauseActorRequest { + ObjectRef actor = 1; } -message DeleteAtespaceResponse {} -message GetActorRequest { - ActorRef actor_ref = 1; +message ResumeActorRequest { + ObjectRef actor = 1; + + // If true, boot the workload from scratch. Process snapshots, including + // golden snapshots, are deferred from the P0 API. + bool boot = 2; } -message GetActorResponse { - Actor actor = 1; +// --------------------------------------------------------------------------- +// ActorTemplate +// --------------------------------------------------------------------------- + +// ActorTemplate defines the workload actors are stamped from. +// Atespace-scoped. The spec is immutable after creation, so there is no +// UpdateActorTemplate method: create a new template instead. +message ActorTemplate { + ResourceMetadata meta = 1; + ActorTemplateSpec spec = 2; + // Output-only. + ActorTemplateStatus status = 3; } -// Request to create a new Actor. -message CreateActorRequest { - // The new actor's atespace and actor_id (actor_id must be a valid DNS-1123 label). - ActorRef actor_ref = 1; +message ActorTemplateSpec { + // Image used to hold the sandbox while no actor is resumed into it. + string pause_image = 1; - // The namespace of the ActorTemplate to derive from. - string actor_template_namespace = 2; + // The workload containers. Images must be pinned by digest. + repeated Container containers = 2; - // The name of the ActorTemplate to derive from. - string actor_template_name = 3; + SnapshotsConfig snapshots_config = 3; - // worker_selector sets the actor's placement constraint at creation time. - // If empty, the actor matches any pool admitted by the template's selector. - Selector worker_selector = 4; + // The sandbox class this workload requires. Matched against + // WorkerSpec.sandbox_class at scheduling time. Open string: providers are + // responsible for creating workers that advertise accurate classes. + string sandbox_class = 4; + + // Restricts which workers actors from this template may be placed on. ANDed + // with the per-actor worker_selector. + Selector worker_selector = 5; + + repeated Volume volumes = 6; } -message CreateActorResponse { - Actor actor = 1; +message ActorTemplateStatus { + enum State { + STATE_UNSPECIFIED = 0; + // Waiting for validation or required provider state. + STATE_PENDING = 1; + // Actors can be created from this template. + STATE_READY = 2; + // Template validation failed; see error_message. + STATE_FAILED = 3; + } + State state = 1; + + // Human-readable failure detail when state is FAILED. + string error_message = 2; } -// Request to update mutable fields on an existing Actor. -// May be called regardless of the actor's current status. -// Changes take effect on the next ResumeActor call. -message UpdateActorRequest { - ActorRef actor_ref = 1; +message Container { + string name = 1; + // Must be pinned by digest (image@sha256:...) so actor starts are + // reproducible. + string image = 2; + repeated string command = 3; + repeated EnvVar env = 4; + ContainerReadyz readyz = 5; + repeated VolumeMount volume_mounts = 6; +} - // worker_selector replaces the actor's current placement constraint. - // Takes effect on the next ResumeActor call. - Selector worker_selector = 2; +message ContainerReadyz { + HTTPGetAction http_get = 1; } -message UpdateActorResponse { - Actor actor = 1; +message HTTPGetAction { + string path = 1; + int32 port = 2; } -message SuspendActorRequest { - ActorRef actor_ref = 1; +message EnvVar { + string name = 1; + oneof source { + // Literal value. + string value = 2; + // Value read from a named secret at actor start. + SecretRef secret = 3; + } } -message SuspendActorResponse { - Actor actor = 1; +// SecretRef names a secret managed by the platform the workers run on +// (a Kubernetes Secret today). Deliberately plane-neutral: no k8s types. +message SecretRef { + // Name of the secret, resolved in the worker's environment. + string name = 1; + // Key within the secret. + string key = 2; + // If true, a missing secret or key is skipped instead of failing the + // actor start. + bool optional = 3; } -message PauseActorRequest { - ActorRef actor_ref = 1; +message Volume { + string name = 1; + VolumeSource volume_source = 2; } -message PauseActorResponse { - Actor actor = 1; +// VolumeSource is an extension point; DurableDir is the only source today. +message VolumeSource { + DurableDirVolumeSource durable_dir = 1; } -message ResumeActorRequest { - ActorRef actor_ref = 1; +message DurableDirVolumeSource {} - // If true, skip golden snapshot and boot the workload from scratch. - bool boot = 2; +message VolumeMount { + string name = 1; + string mount_path = 2; } -message ResumeActorResponse { - Actor actor = 1; +message SnapshotsConfig { + enum Scope { + SCOPE_UNSPECIFIED = 0; + // Deprecated for P0: process snapshots imply runtime/version/hardware + // compatibility constraints that are intentionally deferred. + SCOPE_FULL = 1 [deprecated = true]; + // Snapshot durable data only. + SCOPE_DATA = 2; + } + + // URI prefix under which snapshots are stored. + string location = 1; + + // Scope taken on PauseActor / SuspendActor. + Scope on_pause = 2; + + // Scope taken on commit. Must be a subset of on_pause + // (INVALID_ARGUMENT otherwise). + Scope on_commit = 3; } -message DeleteActorRequest { - ActorRef actor_ref = 1; +message CreateActorTemplateRequest { + ActorTemplate actor_template = 1; } -message DeleteActorResponse {} +message GetActorTemplateRequest { + ObjectRef actor_template = 1; +} -message ListWorkersRequest {} +message ListActorTemplatesRequest { + int32 page_size = 1; + string page_token = 2; +} +message ListActorTemplatesResponse { + repeated ActorTemplate actor_templates = 1; + string next_page_token = 2; +} -message ListWorkersResponse { - repeated Worker workers = 1; +message DeleteActorTemplateRequest { + ObjectRef actor_template = 1; + // Optional freshness guard; 0 = skip check. + int64 version = 2; } -// Request to list actors with pagination. -// This operation provides "soft" guarantees: actors may be missed or duplicated -// if the system state changes during pagination. -message ListActorsRequest { - // The maximum number of actors to return. The server may enforce a hard limit - // (e.g., 1000) regardless of the requested size. - int32 page_size = 1; +message WatchActorTemplatesRequest {} +message WatchActorTemplatesResponse { + ResourceEventType type = 1; + // Unset when type is SYNCED. + ActorTemplate actor_template = 2; +} - // An opaque pagination token obtained from a previous ListActorsResponse. - // Empty for the first request. - string page_token = 2; +// --------------------------------------------------------------------------- +// Worker +// --------------------------------------------------------------------------- + +// Worker is the substrate scheduling primitive. Worker providers create and +// maintain Workers; substrate schedules Actors onto Workers by matching +// Actor/ActorTemplate requirements against Worker labels, sandbox_class, and +// readiness/capacity. +message Worker { + ResourceMetadata meta = 1; + WorkerSpec spec = 2; + // Output/status from the worker provider. + WorkerStatus status = 3; +} + +message WorkerSpec { + // Provider that owns this Worker, e.g. "kubernetes", "vm", or "local". + string provider = 1; + + // Provider-local identity for debug/correlation. For Kubernetes this may be + // a Pod UID or namespace/name. Not part of the substrate resource key. + string provider_id = 2; + + // Routable ateom endpoint for this Worker. + string address = 3; + + // Runtime compatibility class this Worker provides. Matched against + // ActorTemplateSpec.sandbox_class at scheduling time. This is scalar for P0; + // it can be extended to repeated sandbox classes if workers need to + // advertise multiple compatible runtimes. + string sandbox_class = 4; +} + +message WorkerStatus { + enum State { + STATE_UNSPECIFIED = 0; + // Worker can accept actor assignments. + STATE_READY = 1; + // Worker exists but should not receive new assignments. + STATE_NOT_READY = 2; + // Worker is intentionally draining and should not receive new assignments. + STATE_DRAINING = 3; + } + + State state = 1; + + // Capacity reported by the worker provider. + int32 slots_total = 2; + int32 slots_available = 3; - // If set, list only actors in this atespace (scoped SCAN actor::*). - string atespace = 3; + // Human-readable status detail. + string message = 4; + + // Last successful provider heartbeat/update time. Output-only. + google.protobuf.Timestamp last_heartbeat_time = 5; } -// Response for a ListActors operation. -message ListActorsResponse { - // The page of actors. This list may be empty even if there are more results. - repeated Actor actors = 1; +message CreateWorkerRequest { + // worker.meta.name is required; worker.meta.atespace must be empty. + Worker worker = 1; +} - // An opaque token to retrieve the next page of results. - // If empty, there are no more results. Clients must loop until this is empty. - string next_page_token = 2; +message GetWorkerRequest { + ObjectRef worker = 1; } -message Worker { - string worker_namespace = 1; - string worker_pool = 2; - string worker_pod = 3; +message ListWorkersRequest { + int32 page_size = 1; + string page_token = 2; +} +message ListWorkersResponse { + repeated Worker workers = 1; + string next_page_token = 2; +} - Assignment assignment = 4; +message UpdateWorkerRequest { + // worker.meta.name identifies the resource; a non-zero + // worker.meta.version acts as a freshness guard. + Worker worker = 1; - string ip = 5; - int64 version = 6; - string worker_pod_uid = 7; - string node_name = 8; + // Required. Top-level field paths only (e.g. "spec.address", + // "status", "meta.labels"); naming a message or map field replaces it + // wholesale. Unknown or immutable paths return INVALID_ARGUMENT. + google.protobuf.FieldMask update_mask = 2; } -message Assignment { - KubeNamespacedObjectRef actor_template = 1; - ActorRef actor = 2; +message DeleteWorkerRequest { + ObjectRef worker = 1; + // Optional freshness guard; 0 = skip check. + int64 version = 2; } -message KubeNamespacedObjectRef { - string namespace = 1; - string name = 2; +message WatchWorkersRequest {} +message WatchWorkersResponse { + ResourceEventType type = 1; + // Unset when type is SYNCED. + Worker worker = 2; } message DebugClearRequest {} - message DebugClearResponse {} +// --------------------------------------------------------------------------- +// SessionIdentity +// --------------------------------------------------------------------------- + // SessionIdentity allows substrate workloads to exchange their // infrastructure-level credentials (k8s service account token, etc.) for a // substrate session-level credential. A given substrate session might migrate