Add basic per-actor external volume flow by msau42 · Pull Request #405 · agent-substrate/substrate

Michelle Au (msau42) · 2026-07-08T01:01:24Z

Initial part of #232.

Extends the ActorTemplate API to add support for per-actor external volumes and adds control plane and atelet hooks following the Actor lifecycle.

Callouts to volume operations are abstracted with a Volume interface. Right now, only a mock volume plugin for testing has been implemented, but we will add CSI support next.

Tests pass
Appropriate changes to documentation are included in the PR - Not going to update documentation until we add CSI support.

Michelle Au (msau42) · 2026-07-08T15:46:21Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


open for discussion: can we reference k8s objects or should we create our own api?

I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Michelle Au (msau42) · 2026-07-08T15:48:09Z

  map<string, string> match_labels = 1;
 }

+message ExternalVolume {


discuss: should ExternalVolume be its own standalone object rather than embedded fields into the Actor?

Right now ExternalVolume is tied to an Actor but we are also thinking about the scenario where you want to share a volume across multiple actors.

Michelle Au (msau42) · 2026-07-08T15:56:14Z

 	}
 	err = s.persistence.CreateActor(ctx, actor)
 	if err != nil {
+		// Cleanup created volumes if DB write fails


Discuss: In general, how should we approach errors when an operation only partially succeeds? Should we rollback all the partially completed operations? What happens if the rollback fails?

Or we leave the system in the partial state and require users to cleanup by calling Delete? Even then, if deletion fails and Create is called again, we could still have partially leftover resources from a previous invocation.

My current thoughts:

If there is a pair of "Create" and "Delete" operations, then we can leave the partial state and clean it up on the Delete. This does imply we will have to add some "Creating" and possibly "Deleting" state to Actor before we start creating resources for it. I already have marked that as a todo.

For operations like "Resume", there isn't a corresponding delete operation if the resume failed. It's possible resume could be called on a different node. That means we could have partial state left on the first node. In those cases we can try to rollback the state as much as possible but that would still be a best effort operation. This has a few implications:

We probably need to have some process on the node that monitors oprhaned resources and tries to clean them up and/or alert

We need to make sure these orphaned resources don't get reused if a new actor with the same name gets created. This potentially implies that we should have some sort of actor uuid and be using that in our actor directories.

This adds a new externalVolumeTemplate where users specify the capacity, storage driver and storage properties they would like via StorageClass. When an actor is created, we will also provision a volume for the actor according to the template. The volume is deleted when the actor is deleted.

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Haven Xia (HavenXia) · 2026-07-09T00:03:24Z

+	for _, vol := range template.Spec.Volumes {
+		if vol.ExternalVolumeTemplate != nil {
+			// Use a unique name for the volume to ensure idempotency
+			uniqueVolName := fmt.Sprintf("%s-%s", actorID, vol.Name)


This name isn't unique cluster-wide: actor names are only unique within an atespace. We already have atespace in all ateom path, I think we should also make the external volume atespaced, e.g. <atespace>-<actorID>-<volName>, WDYT?

Tim Hockin (thockin)

I didn't get too far into the reading of this yet, but wanted to send a couple questions before EOD

Tim Hockin (thockin) · 2026-07-09T00:19:39Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Tim Hockin (thockin) · 2026-07-09T00:27:07Z

+
+// VolumePlugin abstracts storage operations.
+type VolumePlugin interface {
+	CreateVolume(ctx context.Context, name string, capacity string, storageClass string) (volumeID string, err error)


Do you see this as "kubernetes PVC is the plugin", or "CSI is the plugin", or "NFS is the plugin" ?

What I really want to understand is what credentials are granted to the API server to create and destroy volumes, and what additional layers are in between. E.g. if I need to create a million k8s PVCs and PVs, that's trouble, but if I need to give ateapi binaries the ability to delete EBS volumes in the same context as k8s, that's also trouble.

My thought is that substrate will directly call CSI grpc APIs (so ateapi needs to be authorized to communicate with CSI drivers). We will not be creating PVC objects, although we will need to have a substrate API reference to the volume IDs (which I have added in this PR).

Michelle Au (msau42) · 2026-07-09T01:09:45Z

 			Type:        "bind",
-			Source:      ateompath.DurableDirVolumeMountPoint(atespace, actorID, vm.GetName()),
+			Source:      srcPath,
+			Options:     []string{"bind", "rw"},


double check this. durabledir didn't set any options.

Michelle Au (msau42) commented Jul 8, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 3a4a327 to ccc7a2d Compare July 8, 2026 17:05

Michelle Au (msau42) changed the title ~~Add basic external volume flow~~ Add basic per-actor external volume flow Jul 8, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch from ccc7a2d to c2286ea Compare July 8, 2026 21:28

Michelle Au (msau42) added 6 commits July 8, 2026 22:37

ateapi handling of external volumes.

479b718

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

atelet handling of external volumes.

aadc3a5

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

Allow ateom to see host mounts

c737f32

Volume plugin interface and mock volume implementation.

e2c56d1

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

Update counter demo with external volume using mock volume plugin.

96fa815

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Michelle Au (msau42) force-pushed the mock-mount-dev branch from c2286ea to 96fa815 Compare July 8, 2026 22:50

Haven Xia (HavenXia) reviewed Jul 9, 2026

View reviewed changes

Tim Hockin (thockin) reviewed Jul 9, 2026

View reviewed changes

Michelle Au (msau42) commented Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add basic per-actor external volume flow#405

Add basic per-actor external volume flow#405
Michelle Au (msau42) wants to merge 6 commits into
agent-substrate:mainfrom
msau42:mock-mount-dev

Michelle Au (msau42) commented Jul 8, 2026

Uh oh!

Michelle Au (msau42) Jul 8, 2026

Uh oh!

Tim Hockin (thockin) Jul 9, 2026

Uh oh!

Michelle Au (msau42) Jul 8, 2026

Uh oh!

Michelle Au (msau42) Jul 8, 2026 •

edited

Loading

Uh oh!

Michelle Au (msau42) Jul 8, 2026

Uh oh!

Haven Xia (HavenXia) Jul 9, 2026

Uh oh!

Tim Hockin (thockin) left a comment

Uh oh!

Tim Hockin (thockin) Jul 9, 2026

Uh oh!

Tim Hockin (thockin) Jul 9, 2026

Uh oh!

Michelle Au (msau42) Jul 9, 2026

Uh oh!

Michelle Au (msau42) Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Michelle Au (msau42) commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Michelle Au (msau42) Jul 8, 2026 •

edited

Loading