I am running my elixir app on GKE
here is my deployment configuration:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp
namespace: production
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: myapp
tier: backend
spec:
securityContext:
runAsUser: 0
runAsNonRoot: false
containers:
- name: myapp
image: myimage
resources:
limits:
cpu: 3000m
memory: 2000Mi
requests:
cpu: 2500m
memory: 1000Mi
ports:
- containerPort: 80
args:
- foreground
as you can see in the image, the pod reached its memory limit and crashed
these are my last logs:
erl_child_setup closed
Crash dump is being written to: erl_crash.dump...done
Shutting down..
Node is not running!
and then my app is frozen, I get 502 when trying to request the app,
In order to restart I restart the pod (kubectl delete pod), and then it runs again,
my question is: why doesnt the pod restart automatically when reaches memory limit?
You'll need to add probes that will check if your application is healthy.
Since you mentioned a 502
, I'm assuming this is a Phoenix application and you can add a health-check endpoint:
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 3
periodSeconds: 3
When this request stops receiving a 200
, then the Kubernetes Controller will restart your pod.
You can also use a memory or CPU trigger, as described in detail here: https://blog.powerupcloud.com/autoscaling-based-on-cpu-memory-in-kubernetes-part-ii-fe2e495bddd4