Is it possible to use AWS ECR as repository for K3S running on AWS EC2

[urupaud]

We have a k3s cluster which is running on AWS ec2, we want to deploy containers into this cluster using images in our AWS ECR, is this possible ?

[HuJake]

We also want to ask this question.

[brandond]

It appears that containerd does not support external credential helpers like docker does. See: https://github.com/containerd/cri/issues/1131. This would prevent you from being able to authenticate to ECR when using the default containerd backend.

[HuJake]

@brandond thanks your reply my question.

We have changed it to docker but after setting the following flag

k3s server --docker --kubelet-arg = "cloud-provider = external" --kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/ availability-zone) / $ (curl -s http://169.254.169.254/latest/meta-data/instance-id) "

Seems unable to solve..
It seems there is no information to refer to

[bigbohne]

not nice but you can always generate the /etc/rancher/k3s/registries.yaml using crontab ...

[brandond]

@HuJake did you actually install the out-of-tree aws cloud provider, set up the aws credential helper, and configure docker registry auth to use the helper?

@bigbohne will it actually re-read it, or is it just loaded on startup? I couldn't tell.

[HuJake]

@brandond

I simply set the relevant flag,

k3s server 
--docker
--kubelet-arg = "cloud-provider = external" \
--kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)/$(curl -s http: / /169.254.169.254/latest/meta-data/instance-id) "

I'm want to use [aws cloud provider] but not sure how to apply related yaml?
Are there references to successful cases?
thanks

[brandond]

@HuJake All that does is set the ID. If you want a full cloud provider you can find deployment documentation at https://github.com/kubernetes/cloud-provider-aws. For authenticating Docker to ECR for image pulls you would want https://github.com/awslabs/amazon-ecr-credential-helper.

[bigbohne]

With docker as container engine this is easily possible using the ecr-credential-helper. But what about containerd? Or is it a design decision to not include plugable authenticators?

[brandond]

The containerd folks seem to have philosophical issues that preclude them from adding support for pluggable authentication.

[bigbohne]

Than it is what it is ...

[brandond]

@bigbohne see https://github.com/containerd/cri/issues/1131#issuecomment-631633816

[canadiannomad]

will it actually re-read it, or is it just loaded on startup? I couldn't tell.

I'm wondering the same... I can easily write a cronjob to grab new creds, but restarting k3s every time....?

We are running k3s with docker as container engine here. I installed and configured the amazon-ecr-credential-helper.
I can successfully docker pull images from our private ECR repo as a regular user, but k3s itself keeps complaining with ErrImagePull.

A kubectl describe pod shows: Failed to pull image "***": rpc error: code = Unknown desc = Error response from daemon: Get https://***: no basic auth credentials

So my understanding of this is that the kubelet service (running inside a docker container) cannot pull the image because it does not know about the credential helper. Any pointers on how to overcome this?

[brandond]

Probably related: awslabs/amazon-ecr-credential-helper#210