Skip to content

feat(ateapi): add actor egress policy API - #856

Open
Eitan Yarmush (EItanya) wants to merge 14 commits into
agent-substrate:mainfrom
kagent-dev:egress-policy-api
Open

feat(ateapi): add actor egress policy API#856
Eitan Yarmush (EItanya) wants to merge 14 commits into
agent-substrate:mainfrom
kagent-dev:egress-policy-api

Conversation

@EItanya

@EItanya Eitan Yarmush (EItanya) commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Part of #823.

Design: https://docs.google.com/document/d/1s2klDhbF2mB5sslwUyZBdyXD7JqRt8FWONwJ7wgsomE/edit?tab=t.0

Summary

  • add egress policy documents as resources nested under an Actor and stored separately with Actor deletion cascading to them
  • expose GetActorEgressPolicy, SetActorEgressPolicy, and DeleteActorEgressPolicy; v0 supports zero or one implicitly named document
  • define all documents under an Actor as one logical policy with the same semantics as their combined rules
  • make SetActorEgressPolicy a full create-or-replace operation with version-based collision detection
  • model authorization as repeated atomic allow predicates with separate effects; missing or empty policies deny all traffic
  • support exact/wildcard hostname and CIDR predicates, static-header credential-provider URIs, Actor JWT injection, and optional RFC 8693 exchange parameters
  • add PostgreSQL persistence with Actor deletion cascading to its policy
  • add the internal on-demand resolver with Actor UID incarnation checks and egress-gateway mTLS authorization

Policy distribution is intentionally on demand for v0; this does not add policy xDS. Secret material is retrieved directly by the egress gateway from the referenced credential provider and never passes through ate-api. Ports, document names and metadata, rule IDs, expiry, policy sharing, Atespace-level policy, and generic extensions are intentionally excluded from v0.

Testing

  • go test ./cmd/ateapi ./cmd/ateapi/internal/controlapi ./cmd/ateapi/internal/store/atepg
  • go vet ./cmd/ateapi ./cmd/ateapi/internal/controlapi ./cmd/ateapi/internal/store/atepg
  • repository protobuf generation, proto formatting, and Go formatting verifiers

@EItanya
Eitan Yarmush (EItanya) marked this pull request as draft August 11, 2026 14:47

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the Actor Egress Policy proposal, introducing egress policy bindings and egress credentials along with their CRUD APIs, Redis persistence, validation, and an internal Resolver service. The review feedback highlights critical safety issues in egress_policy.go, specifically potential runtime panics from nil pointer dereferences when cloning policies or accessing Kubernetes secret selectors, as well as a recommendation to improve error handling by not masking internal database errors as permission denied errors.

if err != nil {
return nil, fmt.Errorf("while resolving egress policy binding: %w", err)
}
response := &egresspolicypb.EffectiveEgressPolicy{Policy: proto.Clone(binding.GetPolicy()).(*ateapipb.EgressPolicySpec)}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

If binding.GetPolicy() is nil, proto.Clone(nil) will return a nil interface. Attempting to type-assert a nil interface to *ateapipb.EgressPolicySpec will cause a runtime panic. We should check if binding.GetPolicy() is nil before cloning, or handle it safely.

	var policy *ateapipb.EgressPolicySpec
	if binding.GetPolicy() != nil {
		policy = proto.Clone(binding.GetPolicy()).(*ateapipb.EgressPolicySpec)
	}
	response := &egresspolicypb.EffectiveEgressPolicy{Policy: policy}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obsolete after the API flattening: EgressPolicySpec was removed, so there is no nullable nested policy message to dereference.

Comment on lines +221 to +222
selector := credential.GetKubernetesSecret()
secret, err := s.kubeClient.CoreV1().Secrets(selector.GetNamespace()).Get(ctx, selector.GetName(), metav1.GetOptions{})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If selector is nil (for example, if the credential source is not a Kubernetes secret or is unset), calling selector.GetNamespace() will cause a nil pointer dereference panic. A defensive check should be added to ensure selector is non-nil before accessing its fields.

Suggested change
selector := credential.GetKubernetesSecret()
secret, err := s.kubeClient.CoreV1().Secrets(selector.GetNamespace()).Get(ctx, selector.GetName(), metav1.GetOptions{})
selector := credential.GetKubernetesSecret()
if selector == nil {
return nil, status.Errorf(codes.FailedPrecondition, "egress credential \"%q\" has no Kubernetes secret selector", name)
}
secret, err := s.kubeClient.CoreV1().Secrets(selector.GetNamespace()).Get(ctx, selector.GetName(), metav1.GetOptions{})

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated getters are nil-safe, but I added an explicit selector check so malformed (and future unsupported) credential sources clearly fail closed with FailedPrecondition.

Comment on lines +201 to +204
actor, err := s.persistence.GetActor(ctx, actorRef)
if err != nil || actor.GetMetadata().GetUid() != req.GetActorUid() || actor.GetStatus() != ateapipb.Actor_STATUS_RUNNING {
return nil, status.Error(codes.PermissionDenied, "actor is not authorized for egress")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If s.persistence.GetActor fails due to an internal database error (rather than the actor simply not existing), returning codes.PermissionDenied masks the system failure as a client authorization issue. This can prevent the egress gateway from retrying transient errors. It is better to distinguish between store.ErrNotFound (which should return PermissionDenied or NotFound) and other database errors (which should return codes.Internal or codes.Unavailable).

	actor, err := s.persistence.GetActor(ctx, actorRef)
	if err != nil {
		if errors.Is(err, store.ErrNotFound) {
			return nil, status.Error(codes.PermissionDenied, "actor is not authorized for egress")
		}
		return nil, status.Errorf(codes.Internal, "failed to resolve actor: %v", err)
	}
	if actor.GetMetadata().GetUid() != req.GetActorUid() || actor.GetStatus() != ateapipb.Actor_STATUS_RUNNING {
		return nil, status.Error(codes.PermissionDenied, "actor is not authorized for egress")
	}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: operational actor lookup failures now return Unavailable; a missing actor and actor authorization mismatches remain PermissionDenied.

@EItanya
Eitan Yarmush (EItanya) force-pushed the egress-policy-api branch 4 times, most recently from 42fe4a0 to 4c4d5fc Compare August 11, 2026 15:24
@EItanya
Eitan Yarmush (EItanya) marked this pull request as ready for review August 11, 2026 16:25
Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
ResourceMetadata metadata = 1;
}

// EgressPolicy grants one Actor access to destinations. Rules are ORed;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is Egress atespaced?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. EgressPolicy is Atespace-scoped: its target Actor must be in the same Atespace, and Credential references resolve within that Atespace. I clarified the proto comment and added a regression assertion for cross-Atespace targets.

@EItanya
Eitan Yarmush (EItanya) force-pushed the egress-policy-api branch 2 times, most recently from 1225585 to 279aa81 Compare August 11, 2026 20:30
Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
repeated EgressRule rules = 4;
// Every extension is required. An enforcement point that does not
// understand one must fail closed.
repeated google.protobuf.Any extensions = 5;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should use google.protobuf.Any in our public API. An any field is just a bytes blob and cannot be interpreted by any system that doesn't link against the proto. This means that, for example, a substrate client won't be able to even deserialize / print it unless it links against every single extension proto. I think this will also complicate updates.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning can be foudn here: https://docs.google.com/document/d/1s2klDhbF2mB5sslwUyZBdyXD7JqRt8FWONwJ7wgsomE/edit?tab=t.0#heading=h.7hkev5flhg13

TLDR is that policy computation and delivery is left to Substrate because of scale, so in order to have custom policy it has to be colocated on the object. This is the best way to do that in protobuf. Also, this will be an implementation detail in those instances.

I'm of course open to suggestions on better ways.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this -- API extensions should probably be solved/reasoned about generically before we get too far in this. I know I said this could be ok, but it's also the case that leaving it out for now until we understand the more general case may be easier to deal with...

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d like to use this field to extend policy fucntions ,that are not supported by the public API.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
oneof target {
ObjectRef actor = 2;
}
google.protobuf.Empty allow_all = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this an Empty message?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a signifier, we can also make it a bool if we want.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a boolean is more meaningful?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Louis Ryan (@louiscryan) will agree with me that we should avoid boolean fields and prefer a two-value enum to defend against future breaking changes.

const egressGatewayPrincipal = "spiffe://cluster.local/ns/ate-system/sa/atenet-egress"

var (
egressPolicyMutableFields = mutableFields[*ateapipb.EgressPolicy]{

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: You can remove this after you rebase on main. See #862

@EItanya
Eitan Yarmush (EItanya) force-pushed the egress-policy-api branch 2 times, most recently from 99bc40e to 817f5f4 Compare August 12, 2026 11:57
Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message IPBlockMatch {
repeated string cidrs = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do credential injection for IPBlackMatch?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change seems unrelated to this PR.

Comment thread cmd/atelet/main_test.go
if err != nil {
t.Fatalf("ParseSnapshotURI: %v", err)
}
fullRec := func(class string) sandboxAssetsRecord {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change to this file seems unrelated to this PR.

@LiorLieberman

Copy link
Copy Markdown
Collaborator

FWIW, left some comments on the doc, I think its much easier to align on the API in google doc than GH comments.

Eitan Yarmush (@EItanya) we should also bring this to tomorrows meeting and/or set up additional targeted time to talk about it after that call to prioritize closing it.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
// matching rules from applying their effects. With neither, traffic is denied.
message EgressPolicy {
ResourceMetadata metadata = 1;
oneof target {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Target implies you can have N policies pointing to the same Actor modulo checking for this as part of API validation.

Adding a N-to-1 relationship creates a join-dependency that impacts data partitioning wrt to the these resources, which is a deeper impact to the system. i.e. references are ideally contained in a single partition to do this efficiently.

Maybe we can do "same name, same atespace, same target"?

  • Maintains 1-1 relationship
  • Still has a data partitioning impact, but it's more limited.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're gonna do that then we should just put it on the Actor itself

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the 1:1 approach. Rather than having N policies pointing to the same Actor, it would be cleaner to let the Actor explicitly reference a single EgressPolicy — or even embed it directly on the Actor template. This avoids the N-to-1 join dependency and keeps the data partitioning story simple: each Actor and its policy live in the same partition.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big +1 - the composition design I put together took this approach as well. There are several benefits outside of just partitioning though; for instance, the RBAC becomes very nicely aligned. If you can edit an actor, you can edit its policies, without additional cost to the system.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
repeated EgressRule rules = 4;
// Every extension is required. An enforcement point that does not
// understand one must fail closed.
repeated google.protobuf.Any extensions = 5;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this -- API extensions should probably be solved/reasoned about generically before we get too far in this. I know I said this could be ok, but it's also the case that leaving it out for now until we understand the more general case may be easier to deal with...

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
oneof target {
ObjectRef actor = 2;
}
google.protobuf.Empty allow_all = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would need to be a oneof with EgressRule?

Protobuf type system has a modicum semantic intent -- do we want to use it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Protobuf type system has a modicum semantic intent

What do you mean by this?

This would need to be a oneof with EgressRule?

You can't do a oneof with a repeated type :(

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message HostnameMatch {
string pattern = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to have more documentation on what is a valid / invalid value and interpretation. (This is general comment)

pattern is a ?

  • just DNS name
  • not IP address
  • could it include wildcard (s) and what format

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
message HostnameMatch {
string pattern = 1;
// Credential injection requires an exact hostname match.
HeaderCredentialInjection credential_injection = 2;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to separate actions like inject from decl policy like block.

Ideally, we would want decl policy to be semantically free of ordering side effects. This will make it easier to evolve later, not mention give multiple ways to implement efficiently and optimize.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about this, and I was torn. The only issue is that I really wanted the API to be intentional, and simple to the use case. Can you give an example of what shape you're thinking about here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplest thing that comes to mind is keep the predicate away from the actions.

So you would have "allow rules" and then "mod rules".

Yes, in theory this has some duplication if you want to keep expanding this out
to a full HTTP attribute match, but we know we don't want to go there in the
basic API.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message IPBlockMatch {
repeated string cidrs = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs precise statement of what is valid syntax

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}
}

message KubernetesSecretKeySelector {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was discussed but we need to understand how the permissions work here as I assume you don't want to be able to pull arbitrary secrets from the cluster.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need an object that controls access and refer to that object instead of direct secret reference.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I responded in the doc, but my idea here was that an admin would create these credential object in a given atespace, and then users who have access to that atespace could reference the credential object. So creating the credential object at all would require elevated permissions.

It's a bit hard to chat about these sorts of permissions when there's no RBAC system backing it

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message HeaderCredentialInjection {
string header = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need reference to exact allowed values for header

also basic HTTP stuff:

  • is it case insensitive etc

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

case insensitivity is an interesting point because even thought headers are insensitive there is a 100% chance that some enterprise customer is going to ask for header case sensitivity, but maybe we can leave this to vedors.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserving case sensitivity during injection/replacement is possibly a different concern than internal mechanics of lookup/mapping (especially if the header "replacement value" case as described in Alan Grosskurth (@grosskur)'s comment at https://docs.google.com/document/d/1s2klDhbF2mB5sslwUyZBdyXD7JqRt8FWONwJ7wgsomE/edit?disco=AAACFgOHjCc is more prevalant than the header "addition" case)

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

# Conflicts:
#	pkg/proto/ateapipb/ateapi.pb.go
#	pkg/proto/ateapipb/ateapi.proto
@EItanya
Eitan Yarmush (EItanya) marked this pull request as ready for review August 24, 2026 20:39
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

# Conflicts:
#	cmd/ateapi/internal/store/atepg/atepg.go
#	cmd/ateapi/internal/store/ateredis/ateredis.go
#	cmd/ateapi/main.go
#	internal/proto/grpcechopb/egress_policy.go
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

# Conflicts:
#	pkg/proto/ateapipb/ateapi.pb.go
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

# Conflicts:
#	cmd/ateapi/internal/controlapi/service.go
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

message EgressMatch {
oneof predicate {
google.protobuf.Empty all = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this is not a bool?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bool would mean nothing here, specifically false. This is an item in a oneof, not a true/false in that sense.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message EgressMatch {
oneof predicate {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let's not use oneof as we cannot validate them using DV. Just add one field for each plus a validation to enforce the union type.

@EItanya Eitan Yarmush (EItanya) Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5a95201. EgressMatch now uses separate optional fields with +k8s:unionMember; generated declarative validation enforces that exactly one predicate is set.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message CreateActorEgressPolicyRequest {
// Parent Actor. V0 assigns the policy document's identity implicitly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "V0 assigns the policy document's identity implicitly." mean?

@EItanya Eitan Yarmush (EItanya) Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified in 5a95201. V0 supports one policy document named default under each Actor, so the parent Actor identifies its scope and these RPCs do not take a separate policy name.

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

message GetActorEgressPolicyResponse {
// Empty when the Actor has no policy. V0 returns at most one document.
repeated EgressPolicy egress_policies = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a repeated list? Is this just in case we ever support multiple resources? Wouldn't that be a separate ListActorEgressPolicies request?

If we want to derisk and make sure we can support multiple policies under an actor in the future, something we could do is what a separate PR which tries the exercise of adding that and see if there are unknown unknowns.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53a2950. GetActorEgressPolicy now returns the single EgressPolicy directly and returns NOT_FOUND when it is absent. A future multiple-resource API can add ListActorEgressPolicies.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth discussion: is "Get" going to get a single policy or all of the policies for the actor?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetActorEgressPolicy gets the single policy resource supported today. If multiple policy resources are added later, they will use a separate ListActorEgressPolicies RPC.

}

message StaticHeaderInjection {
string header = 1;

@juli4n Julian Gutierrez Oschmann (juli4n) Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you document all the fields that are added? We have apitool now that generates vioaltions for fields which are not documented.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every field should be optional xor required and define the rules needed to validate it.

This could be another customValidation to parse and validate it (unless there's a defined format for header names). We could make that a +k8s:format=... with an upstream patch

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53a2950. All newly added egress policy fields now have API documentation.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
// named "default" per Actor.
message EgressPolicy {
// Standard resource metadata. Atespace is inherited from the parent Actor
// and, in v0, name is always "default". These identity fields, UID, version,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like we should either let this field empty or require the client to pass "default" and validate it. That way we keep the rule that names are client-provided at create time and immutable.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than making it opaqueType, which disables all validation, we should make the default name be literally "default" (client-specified). Or maybe "the same name as the actor", that way, if we need to back out of this model the schema doesn't change much. It's a little tedious but if we keep this model we could possibly relax it to allow "" somehow (there's no DV way to change a subfield to optional yet, but we can ask for features).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add a DV rule (not blocking) like +k8s:const="default"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to keep name and atespace server-managed for now. The default name is an implementation detail of the current one-policy-per-Actor model, and exposing it as caller input would make that detail part of the API contract. Both fields remain immutable on update; we can expose policy naming later if multiple resources are added.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up after aligning with the lead: changed in f1248a0. Create now requires the caller to provide metadata.name as default and the Actor atespace; both are validated and immutable.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message CreateActorEgressPolicyRequest {
// Parent Actor. V0 creates the policy document named "default" under this

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove the references to "v0". Sounds like an agent artifact? There is no v0 in substrate.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53a2950. Removed the v0 terminology and described the current behavior directly.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
ActorStatus status = 7;
}

// EgressPolicy is a policy document nested under an Actor. All documents for

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Le's say "resource" instead of "document" so we use consistent terminology.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53a2950. The API comments now consistently use resource terminology.

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
// named "default" per Actor.
message EgressPolicy {
// Standard resource metadata. Atespace is inherited from the parent Actor
// and, in v0, name is always "default". These identity fields, UID, version,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than making it opaqueType, which disables all validation, we should make the default name be literally "default" (client-specified). Or maybe "the same name as the actor", that way, if we need to back out of this model the schema doesn't change much. It's a little tedious but if we keep this model we could possibly relax it to allow "" somehow (there's no DV way to change a subfield to optional yet, but we can ask for features).

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
// named "default" per Actor.
message EgressPolicy {
// Standard resource metadata. Atespace is inherited from the parent Actor
// and, in v0, name is always "default". These identity fields, UID, version,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add a DV rule (not blocking) like +k8s:const="default"

// matching rules are then applied once per rule. Rule order has no meaning.
// An empty rule list denies all traffic.
//
// +k8s:optional

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it allowed to have a policy with 0 rules? Just checking.

message EgressRule {
// Entries are ORed. The rule matches when any entry matches.
//
// +k8s:required

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this required means we can never evolve this in a way that has something other than "allow" - are we OK with that? Why not just make it optional - a rule which matches nothing would be valid, even if it has no impact.

repeated EgressMatch allow = 1;

// Effects do not authorize traffic. They are applied only after at least one
// rule authorizes the request.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional?


// Source-agnostic reference interpreted by a registered credential provider:
// substrate-secret://<provider-class>/<provider-name>/<provider-specific-tail>
string credential_uri = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

customValidation

optional or required?

}

message ActorTokenInjection {
string header = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't repeat for all fields, but optional/required and formatting matters

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

message GetActorEgressPolicyRequest {
// +k8s:required
// +k8s:opaqueType

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for opaqueType - I put it on all the existing ones so we can convert them 1 by 1. Net new fields are OK - just add good validation tests! See actor_test for my "please follow this pattern" pattern

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

message GetActorEgressPolicyResponse {
// Empty when the Actor has no policy. V0 returns at most one document.
repeated EgressPolicy egress_policies = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth discussion: is "Get" going to get a single policy or all of the policies for the actor?

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
// Actor, so no policy name is supplied.
//
// +k8s:required
// +k8s:opaqueType

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will discuss with you in realtime - opaqueType is wrong :)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f1248a0. The egress request references and policy resource are no longer opaque, so generated validation descends through ObjectRef, metadata, rules, matches, and effects.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
// and update_time are server-managed.
//
// +k8s:required
// +k8s:opaqueType

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not opaque, use custom to force == "default" ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f1248a0. The caller must provide metadata.name as default, and a custom declarative validator enforces that value. metadata.atespace must also match the parent Actor.


// Effects do not authorize traffic. They are applied only after at least one
// rule authorizes the request.
EgressRuleEffects effects = 2;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried this API is too easy to misuse. IIUC, you can specify allow all and credential injection, which will basically leak credentials by attaching them to all outbound requests? Same with CIDR block based matching.

Have you considered separating allowlisting rules and credential injection rules so they are disjoint and have their own matching rules? That way the API doesn't even allow you to express the bad combinations (i.e. L4 CIDR range or allow all + cred injection)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the current API is actually a good middle ground of matches separated from the effects. It started more coupled, but we wound up here to give some of the separation you’re suggesting. I could see allow_all useful in conjunction with attaching the actor JWT.

At the end of the day ALLOW_ALL is always dangerous, so my preference if anything would be to remove that, but given that these policies only ever apply to a single actor I think it’s ok.

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated
}

message GetActorEgressPolicyResponse {
// Empty when the Actor has no policy. V0 returns at most one document.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does V0 mean? Is it a field?

//
// +k8s:required
// +k8s:opaqueType
ResourceMetadata metadata = 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there validation (maybe in the future) on metadata contents being non-empty? Do we get into trouble here?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating as per Tim’s suggestions


// A request is authorized when at least one rule matches. Effects from all
// matching rules are then applied once per rule. Rule order has no meaning.
// An empty rule list denies all traffic.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Channeling our experience from netpol, the issue with implict empty => meaning is that is hard to add a new field in a way that is backwards compatible.

The issue is

{
  rules: [],
  newField: xxx, <<-- after upgrade, older software will not be able to see this.
}

We might want to add an omnipresent sigil to give ourselves a way out:

{
  rules: [],
  defaultAction: DENY_ALL, <<-- explicit backstop.
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ll remove the comment, it’s not accurate. The stance is always DENY_ALL. So a rule has to be present to change that stance

}

message HostnameMatch {
// An ASCII DNS name, or a wildcard in the complete leftmost label.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ASCII is redundant, DNS names have a set format from RFC anyway.

// An ASCII DNS name, or a wildcard in the complete leftmost label.
//
// The pattern and candidate hostname are normalized to lowercase. One
// trailing dot is permitted and removed. An optional port is removed from an

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"An optional port..." is a bit weird, probably remove this text.


message HostnameMatch {
// An ASCII DNS name, or a wildcard in the complete leftmost label.
//

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest

// A DNS name or a DNS name with a wildcard "*" in the leftmost label.
//
// DNS name should be a valid DNS name (see RFC xxx) normalized to lowercase.
// One trailing dot is ...
//
// A wildcard is a DNS name (in lowercase) with the leftmost label replaced
// with the "*" character. ...describe matching semantics
//
// Example invalid values:
//
// - IP address
// - Include port (e.g. hostname.com:port).
// - Malformed DNS (foo..bar.com)

RFC8693ExchangeParameters rfc_8693_exchange = 3;
}

message RFC8693ExchangeParameters {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh interesting. I thought we may be doing this as a follow-up instead of in the first drop?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. This token exchange flow sounds like a sufficiently complex feature to deserve a separate PR? Do we need it in this first API?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m fine with that as long as we can inject the actor JWT

EgressPolicy egress_policy = 2;
}

message UpdateActorEgressPolicyRequest {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have etag (metadata.version), do we need both Create and Update?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that’s the API convention

message StaticHeaderInjection {
string header = 1;

// For example, "Bearer " for the Authorization header.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should be specific whether or not the space character is needed


// When present, exchanges the Actor JWT at an RFC 8693 endpoint and injects
// the result. A returned expires_in value controls caching of the token.
RFC8693ExchangeParameters rfc_8693_exchange = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a oneof or we think this is a config parameter that will be shared?

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants