Skip to content

Allow using kubectl --context for cluster credentials #43

@msiebuhr

Description

@msiebuhr

Unless one explicitly sets user in k8s_deploy(), it is assumed a username matching the cluster can be found:

user_arg = """$(kubectl config view -o jsonpath='{.users[?(@.name == '"\\"${CLUSTER}\\")].name}")"""

(My jsonpath-foo isn't strong, but I read this as "return the .name of the user who's .name == $CLUSTER - I feel I'm missing something obvious here?)

On our setup this fails as we sometimes have multiple users per cluster. We don't like running with admin-powers on by default, so everyone has <username>-<clustername>, and some has admin-<clustername> as backups. The query above returns an empty string on our setups, so things only work when the user happens to be in the right kubernetes context, so the blank user-name doesn't make a difference:

kubectl config use-context $USER-$OTHER_CLUSTER
bazel run :deploy-to-kubernetes
...
error: error validating "STDIN": error validating data: failed to download openapi: the server has asked for the client to provide credentials; if you choose to ignore these errors, turn validation off with --validate=false

kubectl config use-context $USER-$CORRECT_CLUSTER
bazel run :deploy-to-kubernetes
...
service/xxxx unchanged
deployment.apps/xxxx configured
sealedsecret.bitnami.com/xxxx unchanged
ingress.networking.k8s.io/xxxx unchanged

Kubernetes config does have contexts tying users, clusters and - optionally - namespaces together (which is also what rules_k8s relies on):

k8s_deploy(
    # ...
    context = "morten-siebuhr-some-cluster-name"
)

Which is then passed to kubectl --context $CONTEXT?

Docs: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions