Pod crash CrashLoopBackOff

8/26/2019

I am getting after run "CrashLoopBackOff"

kubectl get pods

This is my yml file.

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: cloudtrail-pipe
spec:
  template:
    metadata:
      labels:
        app: cloudtrail-pipe
    spec:
      hostname: cloudtrail-pipe
      containers:
      - name: cloudtrail-pipe
        ports:
          - containerPort: 5047
            name: filebeat
          - containerPort: 9600
            name: logstash
        image: docker.elastic.co/logstash/logstash:6.5.4
        volumeMounts:
        - name: cloudtrail-pipe-config
          mountPath: /usr/share/logstash/pipeline/
        - name: logstash-jvm-options
          mountPath: /usr/share/logstash/config/
        command:
        - logstash
      volumes:
      - name: cloudtrail-pipe-config
        configMap:
          name: cloudtrail-pipe
          items:
          - key: cloudtrail.conf
            path: cloudtrail.conf
      - name: logstash-output-log
        configMap:
          name: logstash-output-log
          items:
          - key: cloudtrail.log
            path: cloudtrail.log
                  - name: logstash-jvm-options
        configMap:
          name: logstash-jvm-options
          items:
          - key: jvm.options
            path: jvm.options
---
kind: Service
apiVersion: v1
metadata:
  name: cloudtrail-pipe
spec:
  type: NodePort
  selector:
    app: cloudtrail-pipe
  ports:
  - protocol: TCP
    port: 5047
    targetPort: 5047
    nodePort: 30104
    name: filebeat
  - protocol: TCP
    port: 9600
    targetPort: 9600
    name: logstash

And this is the output of

kubectl --v=8 logs cloudtrail-pipe-59bbd75b44-5wcgv --namespace=default -p

I0826 09:17:00.060776   28458 round_trippers.go:416] GET https://xx.xx.xx.xx:6443/api/v1/namespaces/default/pods/cloudtrail-pipe-59bbd75b44-5wcgv
I0826 09:17:00.060800   28458 round_trippers.go:423] Request Headers:
I0826 09:17:00.060811   28458 round_trippers.go:426]     Accept: application/json, */*
I0826 09:17:00.060821   28458 round_trippers.go:426]     User-Agent: kubectl/v1.15.3 (linux/amd64) kubernetes/2d3c76f
I0826 09:17:00.067284   28458 round_trippers.go:441] Response Status: 200 OK in 6 milliseconds
I0826 09:17:00.067300   28458 round_trippers.go:444] Response Headers:
I0826 09:17:00.067307   28458 round_trippers.go:447]     Content-Type: application/json
I0826 09:17:00.067313   28458 round_trippers.go:447]     Content-Length: 3772
I0826 09:17:00.067319   28458 round_trippers.go:447]     Date: Mon, 26 Aug 2019 09:17:00 GMT
I0826 09:17:00.067356   28458 request.go:947] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"cloudtrail-pipe-59bbd75b44-5wcgv","generateName":"cloudtrail-pipe-59bbd75b44-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/cloudtrail-pipe-59bbd75b44-5wcgv","uid":"ebb671b8-0840-4874-9a03-15bf6a01da62","resourceVersion":"97628","creationTimestamp":"2019-08-26T09:12:45Z","labels":{"app":"cloudtrail-pipe","pod-template-hash":"59bbd75b44"},"annotations":{"kubernetes.io/limit-ranger":"LimitRanger plugin set: memory request for container cloudtrail-pipe; memory limit for container cloudtrail-pipe"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"cloudtrail-pipe-59bbd75b44","uid":"697b2314-921b-416f-91ea-0cc295916283","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"cloudtrail-pipe-config","configMap":{"name":"cloudtrail-pipe","items":[{"key":"cloudtrail.conf","path":"cloudtrail.conf"}],"defaultMode":420}},{"name":"logstash-output-log","configMap":{"name":"logstash-output-log","items":[{"key":" [truncated 2748 chars]
I0826 09:17:00.071390   28458 round_trippers.go:416] GET https://xx.xx.xx.xx:6443/api/v1/namespaces/default/pods/cloudtrail-pipe-59bbd75b44-5wcgv/log?previous=true
I0826 09:17:00.071408   28458 round_trippers.go:423] Request Headers:
I0826 09:17:00.071415   28458 round_trippers.go:426]     Accept: application/json, */*
I0826 09:17:00.071422   28458 round_trippers.go:426]     User-Agent: kubectl/v1.15.3 (linux/amd64) kubernetes/2d3c76f
I0826 09:17:30.073747   28458 round_trippers.go:441] Response Status: 500 Internal Server Error in 30002 milliseconds
I0826 09:17:30.073775   28458 round_trippers.go:444] Response Headers:
I0826 09:17:30.073785   28458 round_trippers.go:447]     Content-Type: application/json
I0826 09:17:30.073792   28458 round_trippers.go:447]     Content-Length: 252
I0826 09:17:30.073799   28458 round_trippers.go:447]     Date: Mon, 26 Aug 2019 09:17:30 GMT
I0826 09:17:30.073834   28458 request.go:947] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get https://xx.xx.xx.xx:10250/containerLogs/default/cloudtrail-pipe-59bbd75b44-5wcgv/cloudtrail-pipe?previous=true: dial tcp xx.xx.xx.xx:10250: i/o timeout","code":500}
I0826 09:17:30.074166   28458 helpers.go:199] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Get https://xx.xx.xx.xxx:10250/containerLogs/default/cloudtrail-pipe-59bbd75b44-5wcgv/cloudtrail-pipe?previous=true: dial tcp xx.xx.xx.xx:10250: i/o timeout",
  "code": 500
}]
F0826 09:17:30.074198   28458 helpers.go:114] Error from server: Get https://xx.xx.xx.xx:10250/containerLogs/default/cloudtrail-pipe-59bbd75b44-5wcgv/cloudtrail-pipe?previous=true: dial tcp xx.xx.xx.xx:10250: i/o timeout

Could you help to found the error, please?

EDIT

Below is the config map logstash-jvm-options which maps the file jvm.options. After comment the line with the volume logstash-jvm-options of the yml the deployment works ok.

-Xms2g
-Xmx2g
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djruby.compile.invokedynamic=true
-Djruby.jit.threshold=0
-Djava.security.egd=file:/dev/urandom
-XX:+HeapDumpOnOutOfMemoryError
-- Juan
kubernetes

1 Answer

4/30/2020

I solved this problem I increased memory resource

 resources:
      limits:
        cpu: 1
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 250Mi 

memory

-- Yosra ADDALI
Source: StackOverflow