I want to develop a workflow when developer makes changes to the code and this code continuously deployed to the test Kubernetes AWS EKS cluster.
Images are building in the EKS cluster by Kaniko because we are using ARM flavors.
Skaffold Looks like a good solution for such workflow, but I could not manage to start it to work in my environment.
I'm using the following skaffold.yaml:
---
apiVersion: skaffold/v2beta26
kind: Config
metadata:
name: releases
build:
artifacts:
- image: <account id>.dkr.ecr.<region>.amazonaws.com/releases
kaniko:
cache:
repo: <account id>.dkr.ecr.<region>.amazonaws.com/cache
cluster:
dockerConfig:
secretName: skaffold-docker-config
skaffold build
fails to run:
> skaffold build
Generating tags...
- <account id>.dkr.ecr.<region>.amazonaws.com/releases -> <account id>.dkr.ecr.<region>.amazonaws.com/releases:1ef2c39-dirty
Checking cache...
- <account id>.dkr.ecr.<region>.amazonaws.com/releases: Not found. Building
Starting build...
Creating docker config secret [skaffold-docker-config]...
checking for existing kaniko secret: Unauthorized
From the trace logs I see the 401 error for my registry:
TRAC[0000] Checking base image <account id>.dkr.ecr.<region>.amazonaws.com/releases:arm64-v0.1.5 for ONBUILD triggers. subtask=-1 task=DevLoop
TRAC[0000] --> GET https://<account id>.dkr.ecr.<region>.amazonaws.com/v2/
TRAC[0000] GET /v2/ HTTP/1.1
TRAC[0000] Host: <account id>.dkr.ecr.<region>.amazonaws.com
TRAC[0000] User-Agent: Go-http-client/1.1
TRAC[0000] Accept-Encoding: gzip
TRAC[0000]
TRAC[0000]
TRAC[0000] <-- 401 https://<account id>.dkr.ecr.<region>.amazonaws.com/v2/ (189.277611ms)
TRAC[0000] HTTP/1.1 401 Unauthorized
TRAC[0000] Content-Length: 15
TRAC[0000] Content-Type: text/plain; charset=utf-8
TRAC[0000] Date: Mon, 31 Jan 2022 08:00:51 GMT
TRAC[0000] Docker-Distribution-Api-Version: registry/2.0
TRAC[0000] Sizes:
TRAC[0000] Www-Authenticate: Basic realm="https://<account id>.dkr.ecr.<region>.amazonaws.com/",service="ecr.amazonaws.com"
TRAC[0000]
TRAC[0000] Not Authorized
But I can pull images on my laptop well.
There are other GET requests in the trace which is successful:
TRAC[0000] --> GET https://<account id>.dkr.ecr.<region>.amazonaws.com/v2/releases/manifests/arm64-v0.1.5
TRAC[0000] GET /v2/releases/manifests/arm64-v0.1.5 HTTP/1.1
TRAC[0000] Host: <account id>.dkr.ecr.<region>.amazonaws.com
TRAC[0000] User-Agent: go-containerregistry/v0.7.0
TRAC[0000] Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
TRAC[0000] Authorization: <redacted>
TRAC[0000] Accept-Encoding: gzip
TRAC[0000]
TRAC[0000]
TRAC[0000] <-- 200 https://<account id>.dkr.ecr.<region>.amazonaws.com/v2/releases/manifests/arm64-v0.1.5 (101.445121ms)
TRAC[0000] HTTP/1.1 200 OK
TRAC[0000] Content-Length: 1723
TRAC[0000] Content-Type: application/vnd.docker.distribution.manifest.v2+json
TRAC[0000] Date: Mon, 31 Jan 2022 08:00:51 GMT
TRAC[0000] Docker-Distribution-Api-Version: registry/2.0
On the AWS EKS Cluster side I'm using the instance principal for the ECR auth and such job for kaniko build works well:
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
controller-uid: 27ec2141-e691-4217-940d-02dc87b894cc
job-name: releases-job
name: releases-job
spec:
selector:
matchLabels:
controller-uid: 27ec2141-e691-4217-940d-02dc87b894cc
template:
metadata:
labels:
controller-uid: 27ec2141-e691-4217-940d-02dc87b894cc
job-name: releases-job
spec:
containers:
- args:
- --context=$(CONTEXT)
- --dockerfile=$(DOCKERFILE_LOCATION)
- --destination=<account id>.dkr.ecr.<region>.amazonaws.com/$(REPO):$(TAG)
- --cache-repo=<account id>.dkr.ecr.<region>.amazonaws.com/cache
- --cache=true
env:
- name: CONTEXT
value: git://github.com/account/releases.git
- name: DOCKERFILE_LOCATION
value: docker/Dockerfile
- name: REPO
value: releases
- name: TAG
value: arm64-v0.1.5
image: gcr.io/kaniko-project/executor:latest
name: kaniko
volumeMounts:
- mountPath: /kaniko/.docker/
name: docker-config
volumes:
- configMap:
defaultMode: 420
name: releases-docker-config-c28k6bh5tm
name: docker-config
---
apiVersion: v1
data:
config.json: '{ "credsStore": "ecr-login" }'
kind: ConfigMap
metadata:
name: releases-docker-config-c28k6bh5tm