In RFC-0010 we implemented object-level workload identity for cloud providers leveraging ServiceAccount tokens. However, Flux can still improve in terms of workload identity/short-lived-credentials forms of authentication that do not depend on any cloud providers, but rather on open standards (e.g. OIDC, SPIFFE).
A couple of examples:
- The Kustomization and HelmRelease APIs support vendor-agnostic workload identity through the
generic provider inside .spec.kubeConfig.configMapRef -> .data.provider. This generic provider issues a ServiceAccount token and uses it for authentication with the remote cluster. The remote cluster must be configured with external OIDC authentication for this to work.
- This discussion was recently started asking for SPIFFE integration with the Zot container registry. While the specific form with which the request was made does not align well with the Flux multi-tenancy model, it does inspire the idea I'd like to discuss here.
I've been working with the Zot container registry to introduce OIDC authentication for workloads (Zot already supports OIDC for humans going through the OAuth2 authorization flow in a browser, which is an entirely different thing). With this in mind, I thought about introducing a new authentication pattern for short-lived credentials across Flux, starting with the OCIRepository and ImageRepository APIs.
Proposal
I'd like to introduce the spec field .spec.credential. Valid values would be:
ServiceAccountToken. When the field is set to this value, we do the exactly the same we did for Kustomization and HelmRelease. Users can optionally specify also .spec.serviceAccountName in order not to use the ServiceAccount of the respective controller. The open standard here is obviously OIDC. Again, we leverage Kubernetes for this.SpiffeJWT. When the field is set to this value, we issue a short-lived SPIFFE SVID JWT. Here, (@stealthybox will love it) the identity will not be based on a ServiceAccount! Kubernetes can only issue OIDC JWTs for ServiceAccounts, but we have freedom to issue a SPIFFE JWT where the identity is the Flux object itself, including kind, name and namespace. The cluster will be identified by the helper flag --spiffe-issuer, which will be the iss claim of the JWT.SpiffeCertificate. When the field is set to this value, we issue a short-lived SPIFFE SVID x509 certificate for client authentication via mTLS. Both OCIRepository and ImageRepository already support client authentication via mTLS, so this should be easy to implement.
For the JWT credentials, i.e. ServiceAccountToken and SpiffeJWT, we can also introduce the field .spec.audiences, which will be required for these credentials.
For both Spiffe* values of the field, we can add the binary flag --spiffe-secret-name for watching a secret containing the private key required for issuing the short-lived credentials, and also the CA in case of certificates.
Note that this proposal introduces a pattern that is significantly different from the configuration API we used for Kustomization and HelmRelease. However, these two APIs have their own very specific circumstances:
- They are the only ones that use a ConfigMap reference.
- They target only Kubernetes clusters, which is a third-party service that already supports the OIDC standard and therefore will not require support for SPIFFE.
Why a new field and not .spec.provider
- The word provider calls the idea of a cloud provider, which is not what we are doing here. Here we are implementing authentication standards, and not targeting cloud vendors. Credentials is a better word for this.
- I want the values for the new field to follow PascalCase, like most enums in Kubernetes (especially because of the
ServiceAccountToken value). The .spec.provider field uses lowercase initials, e.g. aws, gcp.
- Say we wanted to use
.spec.provider: generic like in the Kustomization and HelmRelease APIs. This is a strong no-go because this already has a very specific behavior in the OCI APIs.
How trust should be established for SPIFFE
For JWTs, users must host the issuer and JWKS documents somewhere reachable from the third-party services they are integrating with (e.g. from Zot in this first use case).
For certificates, users must configure the third-party service (e.g. Zot) with the CA provided via --spiffe-secret-name.
In RFC-0010 we implemented object-level workload identity for cloud providers leveraging ServiceAccount tokens. However, Flux can still improve in terms of workload identity/short-lived-credentials forms of authentication that do not depend on any cloud providers, but rather on open standards (e.g. OIDC, SPIFFE).
A couple of examples:
genericprovider inside.spec.kubeConfig.configMapRef->.data.provider. This generic provider issues a ServiceAccount token and uses it for authentication with the remote cluster. The remote cluster must be configured with external OIDC authentication for this to work.I've been working with the Zot container registry to introduce OIDC authentication for workloads (Zot already supports OIDC for humans going through the OAuth2 authorization flow in a browser, which is an entirely different thing). With this in mind, I thought about introducing a new authentication pattern for short-lived credentials across Flux, starting with the OCIRepository and ImageRepository APIs.
Proposal
I'd like to introduce the spec field
.spec.credential. Valid values would be:ServiceAccountToken. When the field is set to this value, we do the exactly the same we did for Kustomization and HelmRelease. Users can optionally specify also.spec.serviceAccountNamein order not to use the ServiceAccount of the respective controller. The open standard here is obviously OIDC. Again, we leverage Kubernetes for this.SpiffeJWT. When the field is set to this value, we issue a short-lived SPIFFE SVID JWT. Here, (@stealthybox will love it) the identity will not be based on a ServiceAccount! Kubernetes can only issue OIDC JWTs for ServiceAccounts, but we have freedom to issue a SPIFFE JWT where the identity is the Flux object itself, including kind, name and namespace. The cluster will be identified by the helper flag--spiffe-issuer, which will be theissclaim of the JWT.SpiffeCertificate. When the field is set to this value, we issue a short-lived SPIFFE SVID x509 certificate for client authentication via mTLS. Both OCIRepository and ImageRepository already support client authentication via mTLS, so this should be easy to implement.For the JWT credentials, i.e.
ServiceAccountTokenandSpiffeJWT, we can also introduce the field.spec.audiences, which will be required for these credentials.For both
Spiffe*values of the field, we can add the binary flag--spiffe-secret-namefor watching a secret containing the private key required for issuing the short-lived credentials, and also the CA in case of certificates.Note that this proposal introduces a pattern that is significantly different from the configuration API we used for Kustomization and HelmRelease. However, these two APIs have their own very specific circumstances:
Why a new field and not
.spec.providerServiceAccountTokenvalue). The.spec.providerfield uses lowercase initials, e.g.aws,gcp..spec.provider: genericlike in the Kustomization and HelmRelease APIs. This is a strong no-go because this already has a very specific behavior in the OCI APIs.How trust should be established for SPIFFE
For JWTs, users must host the issuer and JWKS documents somewhere reachable from the third-party services they are integrating with (e.g. from Zot in this first use case).
For certificates, users must configure the third-party service (e.g. Zot) with the CA provided via
--spiffe-secret-name.