eureka pod turn to pending state when running a period of time in kubernetes cluster

3/24/2020

I am deploy a eureka pod in kubernetes cluster(v1.15.2),today the pod turn to be pending state and the actual state is running.Other service could not access to eureka, the eureka icon to indicate pod status shows:this pod is in a pending state.This is my stateful deploy yaml:

{
  "kind": "StatefulSet",
  "apiVersion": "apps/v1beta2",
  "metadata": {
    "name": "eureka",
    "namespace": "dabai-fat",
    "selfLink": "/apis/apps/v1beta2/namespaces/dabai-fat/statefulsets/eureka",
    "uid": "92eefc3d-4601-4ebc-9414-8437f9934461",
    "resourceVersion": "20195760",
    "generation": 21,
    "creationTimestamp": "2020-02-01T16:55:54Z",
    "labels": {
      "app": "eureka"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "eureka"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "eureka"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "eureka",
            "image": "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0",
            "ports": [
              {
                "name": "server",
                "containerPort": 8761,
                "protocol": "TCP"
              },
              {
                "name": "management",
                "containerPort": 8081,
                "protocol": "TCP"
              }
            ],
            "env": [
              {
                "name": "APP_NAME",
                "value": "eureka"
              },
              {
                "name": "POD_NAME",
                "valueFrom": {
                  "fieldRef": {
                    "apiVersion": "v1",
                    "fieldPath": "metadata.name"
                  }
                }
              },
              {
                "name": "APP_OPTS",
                "value": " --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/"
              },
              {
                "name": "APOLLO_META",
                "valueFrom": {
                  "configMapKeyRef": {
                    "name": "fat-config",
                    "key": "apollo.meta"
                  }
                }
              },
              {
                "name": "ENV",
                "valueFrom": {
                  "configMapKeyRef": {
                    "name": "fat-config",
                    "key": "env"
                  }
                }
              }
            ],
            "resources": {
              "limits": {
                "cpu": "2",
                "memory": "1Gi"
              },
              "requests": {
                "cpu": "2",
                "memory": "1Gi"
              }
            },
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "IfNotPresent"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 10,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "imagePullSecrets": [
          {
            "name": "regcred"
          }
        ],
        "schedulerName": "default-scheduler"
      }
    },
    "serviceName": "eureka-service",
    "podManagementPolicy": "Parallel",
    "updateStrategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "partition": 0
      }
    },
    "revisionHistoryLimit": 10
  },
  "status": {
    "observedGeneration": 21,
    "replicas": 1,
    "readyReplicas": 1,
    "currentReplicas": 1,
    "updatedReplicas": 1,
    "currentRevision": "eureka-5976977b7d",
    "updateRevision": "eureka-5976977b7d",
    "collisionCount": 0
  }
}

this is the describe output of the pending state pod:

$ kubectl describe pod eureka-0
Name:           eureka-0
Namespace:      dabai-fat
Priority:       0
Node:           uat-k8s-01/172.19.104.233
Start Time:     Mon, 23 Mar 2020 18:40:11 +0800
Labels:         app=eureka
                controller-revision-hash=eureka-5976977b7d
                statefulset.kubernetes.io/pod-name=eureka-0
Annotations:    <none>
Status:         Running
IP:             172.30.248.8
IPs:            <none>
Controlled By:  StatefulSet/eureka
Containers:
  eureka:
    Container ID:   docker://5e5eea624e1facc9437fef739669ffeaaa5a7ab655a1297c4acb1e4fd00701ea
    Image:          registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0
    Image ID:       docker-pullable://registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka@sha256:7cd4878ae8efec32984a2b9eec623484c66ae11b9449f8306017cadefbf626ca
    Ports:          8761/TCP, 8081/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 23 Mar 2020 18:40:18 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     2
      memory:  1Gi
    Environment:
      APP_NAME:     eureka
      POD_NAME:     eureka-0 (v1:metadata.name)
      APP_OPTS:      --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/
      APOLLO_META:  <set to the key 'apollo.meta' of config map 'fat-config'>  Optional: false
      ENV:          <set to the key 'env' of config map 'fat-config'>          Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xnrwt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-xnrwt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xnrwt
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 360s
                 node.kubernetes.io/unreachable:NoExecute for 360s
Events:
  Type    Reason     Age   From                 Message
  ----    ------     ----  ----                 -------
  Normal  Scheduled  16h   default-scheduler    Successfully assigned dabai-fat/eureka-0 to uat-k8s-01
  Normal  Pulling    16h   kubelet, uat-k8s-01  Pulling image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
  Normal  Pulled     16h   kubelet, uat-k8s-01  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
  Normal  Created    16h   kubelet, uat-k8s-01  Created container eureka
  Normal  Started    16h   kubelet, uat-k8s-01  Started container eureka

how could this happen? what should I do to avoid this situation? After I restart the eureka pod,this problem disappeared,but I still want to know the reason cause this problem.

-- Dolphin
kubernetes

1 Answer

3/24/2020

Sounds like a Kubernetes bug? Try to reproduce it on the current version of Kubernetes. You can also dive into the kubelet logs to see if there is anything useful on those.

-- coderanger
Source: StackOverflow