Pods not getting scheduled to node with matching labels

8/6/2020

I'm getting this error when exec'ing into my pod. Error from server (BadRequest): pod es-master-5cb49c68cc-w6dxv does not have a host assigned

It seemed to be related to my nodeAffinity but I don't see anything immediately wrong with it. I can't seem to get my deployment to attach its pod to any of my nodes. I don't have any taints or tolerations setup on the node or pod. I've tried switching to labels that are automatically generated that are on every node, but nothing seems to work. I've even tried removing my affinity section entirely, and also tried adding nodeSelector to spec by itself.

Here is my deployment config and output from kubectl describe pod -n elasticsearch

<!-- begin snippet: js hide: false console: true babel: false --><!-- language: lang-html -->
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: elasticsearch
    role: master
  name: es-master
  namespace: elasticsearch
spec:
  replicas: 3
  selector:
    matchLabels:
      component: elasticsearch
      role: master
  template:
    metadata:
      labels:
        component: elasticsearch
        role: master
      annotations:
        iam.amazonaws.com/role: {REDACTED}
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: "kubernetes.io/hostname"
            labelSelector:
              matchLabels:
                component: elasticsearch
                role: master
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/region
                operator: In
                values:
                - us-east-2
<!-- end snippet --><!-- begin snippet: js hide: false console: true babel: false --><!-- language: lang-html -->
Name:           es-master-866f7fb558-298ht
Namespace:      elasticsearch
Priority:       0
Node:           <none>
Labels:         component=elasticsearch
                pod-template-hash=866f7fb558
                role=master
Annotations:    iam.amazonaws.com/role: {REDACTED}
                kubernetes.io/psp: eks.privileged
Status:         Pending
IP:             
Controlled By:  ReplicaSet/es-master-866f7fb558
Init Containers:
  init-sysctl:
    Image:      busybox:1.27.2
    Port:       <none>
    Host Port:  <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xflv6 (ro)
Containers:
  elasticsearch:
    Image:       amazon/opendistro-for-elasticsearch:0.9.0
    Ports:       9300/TCP, 9200/TCP, 9600/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Limits:
      cpu:     2
      memory:  12Gi
    Requests:
      cpu:     2
      memory:  12Gi
    Liveness:  tcp-socket :transport delay=60s timeout=1s period=10s #success=1 #failure=3
    Environment:
      CLUSTER_NAME:            logs
      NUMBER_OF_MASTERS:       3
      NODE_MASTER:             true
      NODE_INGEST:             false
      NODE_DATA:               false
      NETWORK_HOST:            0.0.0.0
      TRANSPORT_TLS_PEM_PASS:  
      HTTP_TLS_PEM_PASS:       
      NODE_NAME:               es-master-866f7fb558-298ht (v1:metadata.name)
      DISCOVERY_SERVICE:       elasticsearch-discovery
      KUBERNETES_NAMESPACE:    elasticsearch (v1:metadata.namespace)
      PROCESSORS:              2 (limits.cpu)
      ES_JAVA_OPTS:            -Xms6g -Xmx6g
    Mounts:
      /usr/share/elasticsearch/config/admin-crt.pem from certs (ro,path="admin-crt.pem")
      /usr/share/elasticsearch/config/admin-key.pem from certs (ro,path="admin-key.pem")
      /usr/share/elasticsearch/config/admin-root-ca.pem from certs (ro,path="admin-root-ca.pem")
      /usr/share/elasticsearch/config/elasticsearch.yml from config (rw,path="elasticsearch.yml")
      /usr/share/elasticsearch/config/elk-crt.pem from certs (ro,path="elk-crt.pem")
      /usr/share/elasticsearch/config/elk-key.pem from certs (ro,path="elk-key.pem")
      /usr/share/elasticsearch/config/elk-root-ca.pem from certs (ro,path="elk-root-ca.pem")
      /usr/share/elasticsearch/config/logging.yml from config (rw,path="logging.yml")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xflv6 (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      elasticsearch
    Optional:  false
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-tls-data
    Optional:    false
  default-token-xflv6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xflv6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  59s (x3 over 3m44s)  default-scheduler  0/8 nodes are available: 8 Insufficient cpu.
<!-- end snippet -->

All nodes are m5a.large ec2 instances.

-- Anthony Harley
amazon-eks
kubernetes

1 Answer

8/6/2020

The error is pretty clear 0/8 nodes are available: 8 Insufficient cpu which means nodes don't have 2 cpu cores free as specified in requests. Solution is to either provision nodes with more cpu or reduce the cpu requests in pod spec.

-- Arghya Sadhu
Source: StackOverflow