K
Q

How do I store Kubernetes configuration in same repository as code?

8/1/2021

I currently have a GitHub repository that contains my application code and a Kubernetes deployment configuration file (called deployment.yml). I.e., my repository has the following structure:

repository
  +- ...application code...
  +- Dockerfile
  \- deployment.yml

When a change is pushed to this GitHub repository, a series of GitHub Actions are executed that containerize my application into a Docker image and publish that image to Docker Hub.

On a development machine, I have a Kubernetes cluster running. I pull the deployment.yml file from the repository and either apply that configuration using kubectl apply -f deployment.yml or perform a rolling-update using kubectl rollout restart deployment/<name>.

My deployment.yml is configured as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  replicas: 1
  ...
  template:
    ...
    spec:
      containers:
      - name: <name>
        image: foo/bar:v1.0.0
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: <port>
      imagePullSecrets:
      - name: <creds-id>

The version of the image (i.e., v1.0.0 in foo/bar:v1.0.0) originates from the tags I use in Git. I.e., if I tag a commit as v1.0.0, a new build is run for that tag and a new Docker image is published to Docker Hub with the tag v1.0.0.

My problem is that I am storing my Kubernetes configuration (deployment.yml) in the same repository that is being tagged. This means that I am tagging a commit (i.e., v2.0.0) that contains an image of foo/bar:v1.0.0 in the deployment.yml. I.e., I make changes to my code and then decide once those changes are sufficient, that a specific commit will be tagged. Since I want the cluster on my development machine to use the latest, approved (i.e., tagged) code, I then go an update the deployment.yml and commit, but that new commit is after the tagged commit.

To resolve this, I would have to change the deployment.yml file for the commit I know will be tagged. I.e., knowing that the next commit I make will be tagged as v2.0.0, I will have to change the deployment.yml to use the image foo/bar:v2.0.0 and add that change to the commit (i.e., the one to be tagged). In that case, the commit tagged as v2.0.0 will have an image of foo/bar:v2.0.0 in its deployment.yml.

Is there a technique or best practice (such as templating or another practice) that can solve this issue?

Thank you.

-- Justin Albano
containers
docker
git
kubernetes

Similar Questions

Set up multiple sub domain in aws route 53 to expose Kubernetes
Kubernetes Jenkins connectivity Issue
Using a single static IP address as LoadBalancerIP across multiple services with different ports
How do I set the imagePullPolicy with Minikube
Traefik Ingress Controller on Kubernetes, receiving permissions error

1 Answer

8/1/2021

Helm templating is also a good option however if your project is basic and won't be much requirement there use this basic method keeping the deployment.yaml in repo simple way.

Ideally, you can keep your deployment file dynamic as much as possible rather than fix values to the add-in. 

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: test-image
  labels:
    app: test-image
spec:
  selector:
    matchLabels:
      app: test-image
      tier: frontend
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: test-image
        tier: frontend
    spec:
      containers:
      - image: TEST_IMAGE_NAME
        name: test-image
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 443
          name: https

and in the YAML CI config we change the IMAGE URL in deployment.yaml file as per need.

Google could build CI file example, however you can write or update the logic in your YAML CI

steps:
- id: 'set test core image in yamls'
  name: 'ubuntu'
  args: ['bash','-c','sed -i "s,TEST_IMAGE_NAME,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," deployment.yaml']

In your process, the YAML file and run time replace the string with URL you want to add.

Git will give you necessary variables like COMMIT HASH or TAGGED VERSION.

Example CI file YAML file,

steps:
- id: 'build test core image'
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA', '.']
- id: 'push test image'
  name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
- id: 'set test core image in yamls'
  name: 'ubuntu'
  args: ['bash','-c','sed -i "s,TEST_IMAGE_NAME,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," deployment.yaml']
- name: 'gcr.io/cloud-builders/kubectl'
  args: ['apply', '-f', 'deployment.yaml']

inside your deployment.yaml there will be one string TEST_IMAGE_NAME which will get replaced during the CI operation

using simple ubuntu command sed : sed -i "s,TEST_IMAGE_NAME,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," deployment.yaml

variables like $SHORT_SHA will get auto injected or added by Github so now on your CI server you have one deployment.yaml with Image-URL that you have pushed to docker hub.

We have replaced our TEST_IMAGE_NAME string in deployment.yaml dynamically during CI.

Now you can apply this deployment.yaml from CI server, & it will update or push the deployment to K8s cluster wherever it's running.

Note :

if you want to store the config in repo also, you can commit back file from CI server repo, you have to store 3 deployment file, dev-deploymentl.yaml, stag-deployment.yaml if any changes in any branch you can commit that file from CI server to repo again and your YAML config will be saved in repo also.

-- Harsh Manvar
Source: StackOverflow