pod not restarting automatically when crashing after reaching memory limit

12/17/2018

I am running my elixir app on GKE

here is my deployment configuration:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: myapp
        tier: backend
    spec:
      securityContext:
        runAsUser: 0
        runAsNonRoot: false
      containers:
      - name: myapp
        image: myimage
        resources:
          limits:
            cpu: 3000m
            memory: 2000Mi
          requests:
            cpu: 2500m
            memory: 1000Mi
        ports:
        - containerPort: 80
        args:
          - foreground

as you can see in the image, the pod reached its memory limit and crashed pod

these are my last logs:

erl_child_setup closed

Crash dump is being written to: erl_crash.dump...done

Shutting down..

Node is not running!

and then my app is frozen, I get 502 when trying to request the app,

In order to restart I restart the pod (kubectl delete pod), and then it runs again,

my question is: why doesnt the pod restart automatically when reaches memory limit?

-- dina
google-kubernetes-engine
kubernetes

2 Answers

12/17/2018

You'll need to add probes that will check if your application is healthy.

Since you mentioned a 502, I'm assuming this is a Phoenix application and you can add a health-check endpoint:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 3

When this request stops receiving a 200, then the Kubernetes Controller will restart your pod.

-- Rawkode
Source: StackOverflow