How can I reuse common configuration across different kubernetes manifests?

2/6/2020

Assume I have this manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: initialize-assets-fixtures
spec:
  template:
    spec:
      initContainers:
      - name: wait-for-minio
        image: bitnami/minio-client
        env:
          - name: MINIO_SERVER_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio
                key: access-key
          - name: MINIO_SERVER_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio
                key: secret-key
          - name: MINIO_SERVER_HOST
            value: minio
          - name: MINIO_SERVER_PORT_NUMBER
            value: "9000"
          - name: MINIO_ALIAS
            value: minio
        command: 
          - /bin/sh
          - -c 
          - |
            mc config host add ${MINIO_ALIAS} http://${MINIO_SERVER_HOST}:${MINIO_SERVER_PORT_NUMBER} ${MINIO_SERVER_ACCESS_KEY} ${MINIO_SERVER_SECRET_KEY}
      containers:
      - name: initialize-assets-fixtures
        image: bitnami/minio
        env:
          - name: MINIO_SERVER_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio
                key: access-key
          - name: MINIO_SERVER_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio
                key: secret-key
          - name: MINIO_SERVER_HOST
            value: minio
          - name: MINIO_SERVER_PORT_NUMBER
            value: "9000"
          - name: MINIO_ALIAS
            value: minio
        command:
          - /bin/sh
          - -c
          - |
            mc config host add ${MINIO_ALIAS} http://${MINIO_SERVER_HOST}:${MINIO_SERVER_PORT_NUMBER} ${MINIO_SERVER_ACCESS_KEY} ${MINIO_SERVER_SECRET_KEY}

            for category in `ls`; do
              for f in `ls $category/*` ; do
                mc cp $f ${MINIO_ALIAS}/$category/$(basename $f)
              done
            done
      restartPolicy: Never

You see I have here one initContainer and one container. In both containers, I have the same configuration, i.e. the same env section.

Assume I have yet another Job manifest where I use the very same env section again.

It's a lot of duplicated configuration that I bet I can simplify drastically, but I don't know how to do it. Any hint? Any link to some documentation? After some googling, I was not able to come up with anything useful. Maybe with kustomize, but I'm not sure. Or maybe I'm doing it the wrong way with all those environment variables, but I don't think I have a choice, depending on the service I'm using (here it's minio, but I want to do the same kind of stuff with other services which might not be as flexible as minio).

-- Laurent Michel
kubernetes

2 Answers

2/8/2020

So, long story short: to solve my problem, I first created a new chart for my service and transformed the k8s manifests I had into helm templates. Then, I completed the _helpers.tpl with the following code:

{{/*
Common minio environment variables setup
*/}}
{{- define "minio.envvarsblock" -}}
- name: MINIO_SERVER_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: {{ .Values.minio.fullname }}
      key: access-key
- name: MINIO_SERVER_SECRET_KEY
  valueFrom:
    secretKeyRef:
      name: {{ .Values.minio.fullname }}
      key: secret-key
- name: MINIO_SERVER_HOST
  value: {{ .Values.minio.fullname }}
- name: MINIO_SERVER_PORT_NUMBER
  value: {{ .Values.minio.server.port | quote }}
- name: MINIO_ALIAS
  value: {{ .Values.minio.client.alias }}
{{- end -}}

{{/*
Wait for minio init container definition
*/}}
{{- define "wait-for-minio" -}}
- name: wait-for-minio
  image: {{ .Values.minio.client.image }}
  env: {{- include "minio.envvarsblock" . | nindent 4 }}
  command:
    - /bin/sh
    - -c
    - |
      mc config host add ${MINIO_ALIAS} http://${MINIO_SERVER_HOST}:${MINIO_SERVER_PORT_NUMBER} ${MINIO_SERVER_ACCESS_KEY} ${MINIO_SERVER_SECRET_KEY}
{{- end -}}

The first section above allows to reuse the env section throughout all my templates and the second allows to reuse an initContainer that I use all over the place too. I was then able to inject those partial templates into my helm templates like so (to take the example I put in my original post):

{{- if .Values.fixtures.enabled -}}
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "chart.fullname" . }}-init-fixtures
  labels:
{{ include "chart.labels" . | indent 4 }}
spec:
  template:
    spec:
      initContainers:
      {{- include "wait-for-minio" . | nindent 6 }}
      containers:
      - name: {{ .Chart.Name }}-init-fixtures
        image: {{ .Values.image }}
        env: {{- include "minio.envvarsblock" . | nindent 10 }}
        command:
          - /bin/sh
          - -c
          - |
            mc config host add ${MINIO_ALIAS} http://${MINIO_SERVER_HOST}:${MINIO_SERVER_PORT_NUMBER} ${MINIO_SERVER_ACCESS_KEY} ${MINIO_SERVER_SECRET_KEY}

            for category in `ls`; do
              for f in `ls $category/*` ; do
                mc cp $f ${MINIO_ALIAS}/$category/$(basename $f)
              done
            done
      restartPolicy: OnFailure
{{- end -}}
-- Laurent Michel
Source: StackOverflow

2/6/2020

Based on my knowledge you have those 3 options


ConfigMap


You can use either kubectl create configmap or a ConfigMap generator in kustomization.yaml to create a ConfigMap.

The data source corresponds to a key-value pair in the ConfigMap, where

key = the file name or the key you provided on the command line

value = the file contents or the literal value you provided on the command line.

More about how to use it in pod here


Helm


As @Matt mentionted in comments you can use helm

helm lets you template the yaml with values. Also once you get into it there are ways to create and include partial templates – Matt

By the way, helm has it's own created minio chart, you might take a look how it is created there.


Kustomize


It's well described here and here how could you do that in kustomize.

Let me know if you have any more questions.

-- jt97
Source: StackOverflow