K
Q

Question

Kubernetes: move from deployment to statefulset - set env from secrets

3/17/2020

Kubernetes documentation says that for mysql pods we need to use Stateful sets in order to avoid "split brain" situations when one pod dies, in other words, to declare one "master" node to which data will be written to, and if that pod dies, elect new master, that's why i want this deployment and service to transfer to stateful set:

  ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: mysql-container
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: mysql-container
      template:
        metadata:
          labels:
            app: mysql-container
        spec:
          containers:
          - name: mysql-container
            image: mysql:dev
            imagePullPolicy: "IfNotPresent"
            envFrom:
              - secretRef:
                 name: prod-secrets
            ports:
            - containerPort: 3306
             # container (pod) path
            volumeMounts:
              - name: mysql-persistent-storage
                mountPath: /data/db

          # minikube path
          volumes:
            - name: mysql-persistent-storage
              persistentVolumeClaim:
               claimName: mysql-pvc
            #resources:
            #  requests:
            #    memory: 300Mi
            #    cpu: 400m
            #  limits:
            #    memory: 400Mi
            #    cpu: 500m  
          restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: mysql

spec:
  # Open port 3306 only to pods in cluster
  selector:
    app: mysql-container

  ports:
    - name: mysql
      port: 3306
      protocol: TCP
      targetPort: 3306
  type: ClusterIP

i created stateful set following: this guide

Under containers section i specified environment variables from file, ie. removed

 env:
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "1"

Statefulset:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: mysql
  replicas: 3
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:5.7
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Generate mysql server-id from pod ordinal index.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          echo [mysqld] > /mnt/conf.d/server-id.cnf
          # Add an offset to avoid reserved server-id=0 value.
          echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
          # Copy appropriate conf.d files from config-map to emptyDir.
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/config-map/master.cnf /mnt/conf.d/
          else
            cp /mnt/config-map/slave.cnf /mnt/conf.d/
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config-map
      - name: clone-mysql
        image: gcr.io/google-samples/xtrabackup:1.0
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Skip the clone if data already exists.
          [[ -d /var/lib/mysql/mysql ]] && exit 0
          # Skip the clone on master (ordinal index 0).
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          [[ $ordinal -eq 0 ]] && exit 0
          # Clone data from previous peer.
          ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
          # Prepare the backup.
          xtrabackup --prepare --target-dir=/var/lib/mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      containers:
      - name: mysql
        image: mysql:dev
        imagePullPolicy: "IfNotPresent"
        envFrom:
          - secretRef:
             name: prod-secrets
        ports:
        - name: mysql
          containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          #requests:
           # cpu: 300m
           # memory: 1Gi
        livenessProbe:
          exec:
            command: ["mysqladmin", "ping"]
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            # Check we can execute queries over TCP (skip-networking is off).
            command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
      - name: xtrabackup
        image: gcr.io/google-samples/xtrabackup:1.0
        ports:
        - name: xtrabackup
          containerPort: 3307
        command:
        - bash
        - "-c"
        - |
          set -ex
          cd /var/lib/mysql

          # Determine binlog position of cloned data, if any.
          if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
            # XtraBackup already generated a partial "CHANGE MASTER TO" query
            # because we're cloning from an existing slave. (Need to remove the tailing semicolon!)
            cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
            # Ignore xtrabackup_binlog_info in this case (it's useless).
            rm -f xtrabackup_slave_info xtrabackup_binlog_info
          elif [[ -f xtrabackup_binlog_info ]]; then
            # We're cloning directly from master. Parse binlog position.
            [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
            rm -f xtrabackup_binlog_info xtrabackup_slave_info
            echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                  MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
          fi

          # Check if we need to complete a clone by starting replication.
          if [[ -f change_master_to.sql.in ]]; then
            echo "Waiting for mysqld to be ready (accepting connections)"
            until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done

            echo "Initializing replication from clone position"
            mysql -h 127.0.0.1 \
                  -e "$(<change_master_to.sql.in), \
                          MASTER_HOST='mysql-0.mysql', \
                          MASTER_USER='root', \
                          MASTER_PASSWORD='', \
                          MASTER_CONNECT_RETRY=10; \
                        START SLAVE;" || exit 1
            # In case of container restart, attempt this at-most-once.
            mv change_master_to.sql.in change_master_to.sql.orig
          fi

          # Start a server to send backups when requested by peers.
          exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
            "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
      volumes:
      - name: conf
        emptyDir: {}
      - name: config-map
        configMap:
          name: mysql
      - name: data
        persistentVolumeClaim:
            claimName: mysql-pvc

Services:

# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
    protocol: TCP
    targetPort: 3306
  type: ClusterIP
  selector:
    app: mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the master: mysql-0.mysql.
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
    protocol: TCP
    targetPort: 3306
  type: ClusterIP  
  selector:
    app: mysql

I have env file from which i created secret:

kubectl create secret prod-secrets \
       --from-env-file=env.example

Problem is that i can't access mysql (Access denied), pods using credentials specified in secret, without Stateful set, all works fine. All pods are running, no errors in logs

How to specify values in secrets into Statefulset ?

I presume that i need somehow to pass those secrets to command section but have no idea how, example from Kuberenets page assumes credentials are not used

If there is less complicated way to use stateful set for mysql,please let me know, thanks.

-- overflowed

kubernetes

kubernetes-pod

mysql

2 Answers

3/20/2020

At the end i managed to escape above complications by creating volume templates, created PV for each pod, both volumes are synchronized, no duplicate entries in database, and if one node fails,data are preserved

storage.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:

  name: localstorage

provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
reclaimPolicy: Delete
allowVolumeExpansion: True

---

kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-01
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql01"

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-02
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql02"

Statefulset:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-container
spec:
  serviceName: mysql
  replicas: 2
  selector:
    matchLabels:
      app: mysql-container
  template:
    metadata:
      labels:
        app: mysql-container
    spec:
      containers:
      - name: mysql-container
        image: mysql:dev
        imagePullPolicy: "IfNotPresent"
        envFrom:
          - secretRef:
             name: prod-secrets
        ports:
        - containerPort: 3306
        # container (pod) path
        volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql

        resources:
          requests:
            memory: 300Mi
            cpu: 400m
          limits:
            memory: 400Mi
            cpu: 500m
      restartPolicy: Always

  volumeClaimTemplates:
    - metadata:
        name: mysql-persistent-storage
      spec:
        storageClassName: localstorage
        accessModes: ["ReadWriteOnce"]
        resources:
         requests:
          storage: 5Gi
        selector:
         matchLabels:
          type: local

-- overflowed

Source: StackOverflow

3/17/2020

Let's start creating a secret:

$ kubectl create secret generic test-secret --from-literal=username='my-app' --from-literal=password='39528$vdg7Jb'

$ kubectl get secrets test-secret -o yaml
apiVersion: v1
data:
  password: Mzk1MjgkdmRnN0pi
  username: bXktYXBw
kind: Secret
metadata:
  name: test-secret
  namespace: default

Using envFrom:

Use envFrom to define all of the Secret’s data as container environment variables. The key from the Secret becomes the environment variable name in the Pod. ref

apiVersion: v1
kind: Pod
metadata:
  name: envfrom-secret
  namespace: default
spec:
  containers:
  - name: envars-test-container
    image: nginx
    envFrom:
    - secretRef:
        name: test-secret

Check env:

$ kubectl exec -it envfrom-secret printenv
.
.
password=39528$vdg7Jb
username=my-app
.
.

Using env:

When you want some specific key-value pairs to be on the env list or you want to set env with variable name other than the key name, you can set env from k8s secret like below:

apiVersion: v1
kind: Pod
metadata:
  name: envfrom-secret
  namespace: default
spec:
  containers:
  - name: envars-test-container
    image: nginx
    env:
    - name: USERNAME
      valueFrom:
        secretKeyRef:
          name: test-secret
          key: username
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: test-secret
          key: password

Check env:

$  kubectl exec -it envfrom-secret printenv
.
.
USERNAME=my-app
PASSWORD=39528$vdg7Jb
.
.

Same process goes for deployments, statefulsets, daemonsets...

-- Kamol Hasan

Source: StackOverflow

KQ

Kubernetes: move from deployment to statefulset - set env from secrets

Similar Questions

2 Answers

K
Q