Combining multiple Local-SSD on a node in Kubernetes (GKE)

2/18/2019

The data required by my container is too large to fit on one local SSD. I also need to access the SSD's as one filesystem from my container. So I would need to attach multiple ones. How do I combine them (single partition, RAID0, etc) and make them accessible as one volume mount in my container?

This link shares how to mount an SSD https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd to a mount path. I am not sure how you would merge multiple.

edit

The question asks how one would "combine" multiple SSD devices, individually mounted, on a single node in GKE.

-- VBK
google-kubernetes-engine
kubernetes

2 Answers

5/18/2020

WARNING this is experimental and not intended for production use without knowing what you are doing and only tested on gke version 1.16.x.

The approach includes a daemonset using a configmap to use nsenter (with wait tricks) for host namespace and privileged access so you can manage the devices. Specifically for GKE Local SSDs, we can unmount those devices and then raid0 them. InitContainer for the dirty work as this type of task seems most apparent for something you'd need to mark complete, and to then kill privileged container access (or even the Pod). Here is how it is done.

The example assumes 16 SSDs, however, you'll want to adjust the hardcoded values as necessary. Also, ensure your OS image reqs, I use Ubuntu. Also make sure the version of GKE you use starts local-ssd's at sd[b]

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: local-ssds-setup
  namespace: search
data:
    setup.sh: |
        #!/bin/bash
        # returns exit codes: 0 = found, 1 = not found
        isMounted() { findmnt -rno SOURCE,TARGET "$1" >/dev/null;} #path or device

        # existing disks & mounts
        SSDS=(/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq)

        # install mdadm utility
        apt-get -y update && apt-get -y install mdadm --no-install-recommends
        apt-get autoremove

        # OPTIONAL: determine what to do with existing, I wipe it here
        if [ -b "/dev/md0" ]
            then
            echo "raid array already created"

            if isMounted "/dev/md0"; then
                echo "already mounted - unmounting"
                umount /dev/md0 &> /dev/null || echo "soft error - assumed device was mounted"
            fi

            mdadm --stop /dev/md0
            mdadm --zero-superblock "${SSDS[@]}"                
        fi

        # unmount disks from host filesystem
        for i in {0..15}
        do 
            umount "${SSDS[i]}" &> /dev/null || echo "${SSDS[i]} already unmounted"
        done

        if isMounted "/dev/sdb";
            then 
            echo ""
            echo "unmount failure - prevent raid0" 1>&2
            exit 1
        fi

        # raid0 array
        yes | mdadm --create /dev/md0 --force --level=0 --raid-devices=16 "${SSDS[@]}"

        echo "raid array created"

        # format 
        mkfs.ext4 -F /dev/md0

        # mount, change /mnt/ssd-array to whatever
        mkdir -p /mnt/ssd-array
        mount /dev/md0 /mnt/ssd-array
        chmod a+w /mnt/ssd-array

    wait.sh: |
        #!/bin/bash
        while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do sleep 1; done

DeamonSet pod spec

spec:
      hostPID: true
      nodeSelector:
        cloud.google.com/gke-local-ssd: "true"
      volumes:
      - name: setup-script
        configMap:
          name: local-ssds-setup
      - name: host-mount
        hostPath:
          path: /tmp/setup
      initContainers:
      - name: local-ssds-init
        image: marketplace.gcr.io/google/ubuntu1804
        securityContext:
          privileged: true
        volumeMounts:
        - name: setup-script
          mountPath: /tmp
        - name: host-mount
          mountPath: /host
        command:
          - /bin/bash
          - -c
          - |
            set -e
            set -x

            # Copy setup script to the host
            cp /tmp/setup.sh /host

            # Copy wait script to the host 
            cp /tmp/wait.sh /host

            # Wait for updates to complete
            /usr/bin/nsenter -m/proc/1/ns/mnt -- chmod u+x /tmp/setup/wait.sh

            # Give execute priv to script
            /usr/bin/nsenter -m/proc/1/ns/mnt -- chmod u+x /tmp/setup/setup.sh

            # Wait for Node updates to complete
            /usr/bin/nsenter -m/proc/1/ns/mnt /tmp/setup/wait.sh

            # If the /tmp folder is mounted on the host then it can run the script
            /usr/bin/nsenter -m/proc/1/ns/mnt /tmp/setup/setup.sh
      containers:
      - image: "gcr.io/google-containers/pause:2.0"
        name: pause
-- rambossa
Source: StackOverflow

2/28/2019

You can use DaemonSet yaml file to deploy the pod will run on startup, assuming already created a cluster with 2 local-SSD (this pod will be in charge of creating the Raid0 disk)

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: ssd-startup-script
  labels:
    app: ssd-startup-script
spec:
  template:
    metadata:
      labels:
        app: ssd-startup-script
   spec:
     hostPID: true
     containers:
     - name: ssd-startup-script
    image: gcr.io/google-containers/startup-script:v1
    imagePullPolicy: Always
    securityContext:
      privileged: true
    env:
    - name: STARTUP_SCRIPT
      value: |
        #!/bin/bash
        sudo curl -s https://get.docker.com/ | sh
        echo Done

The pod that will have access to the disk array in the above example is “/mnt/disks/ssd-array”

apiVersion: v1
kind: Pod
metadata:
 name: test-pod
spec:
 containers:
- name: test-container
  image: ubuntu
  volumeMounts:
  - mountPath: /mnt/disks/ssd-array
   name: ssd-array
  args:
  - sleep
  - "1000"
nodeSelector:
 cloud.google.com/gke-local-ssd: "true"
tolerations:
- key: "local-ssd"
 operator: "Exists"
 effect: "NoSchedule"
volumes:
- name: ssd-array
  hostPath:
   path: /mnt/disks/ssd-array

After deploying the test-pod, SSH to the pod from your cloud-shell or any instance.

Then run :

  kubectl exec -it test-pod -- /bin/bash

After that you should be able to see the created file in the ssd-array disk.

cat test-file.txt
-- Milad
Source: StackOverflow