How to avoid creating pod on node that doesn't have enough storage?

4/25/2019

I'm setting up my application with Kubernetes. I have 2 Docker images (Oracle and Weblogic). I have 2 kubernetes nodes, Node1 (20 GB storage) and Node2 (60 GB) storage.

When I run kubectl apply -f oracle.yaml it tries to create oracle pod on Node1 and after few minutes it fails due to lack of storage. How can I force Kubernetes to check the free storage of that node before creating the pod there?

Thanks

-- Hassan Senobar
kubernetes

2 Answers

4/26/2019

You may try to specify ephemeral storage requirement for pod:

resources:
  requests:
    ephemeral-storage: "40Gi"
  limits:
    ephemeral-storage: "40Gi"

Then it would be scheduled only on nodes with sufficient ephemeral storage available.

You can verify the amount of ephemeral storage available on each node in the output of "kubectl describe node".

$ kubectl describe node somenode | grep -A 6 Allocatable
Allocatable:
 attachable-volumes-gce-pd:  64
 cpu:                        3920m
 ephemeral-storage:          26807024751
 hugepages-2Mi:              0
 memory:                     12700032Ki
 pods:                       110
-- Vasily Angapov
Source: StackOverflow

4/25/2019

First of all, you probably want to give Node1 more storage.

But if you don't want the pod to start at all you can probably run a check with an initContainer where you check how much space you are using with something like du or df. It could be a script that checks for a threshold that exits unsuccessfully if there is not enough space. Something like this:

#!/bin/bash 

# Check if there are less than 10000 bytes in the <dir> directory
if [ `du <dir> | tail -1 | awk '{print $1}'` -gt "10000" ]; then exit 1; fi

Another alternative is to use a persistent volume (PV) with a persistent volume claim (PVC) that has enough space together with the default StorageClass Admission Controller, and you do allocate the appropriate space in your volume definition.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 40Gi
  storageClassName: mytype

Then on your Pod:

kind: Pod
apiVersion: v1
metadata:
  name: mypod
spec:
  containers:
    - name: mycontainer
      image: nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: mypd
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim

The Pod will not start if your claim cannot be allocated (There isn't enough space)

-- Rico
Source: StackOverflow