I have a AWS EKS K8s setup with autoscaling group that uses spot instances.
Once a day there is a scheduled K8s cronjob which scales down a bunch of deployments, runs a script in a pod for 15min , then scales the deployments back up.
Issue is after deployments are scaled down, the script starts running and cluster autoscaling triggers to remove some nodes around 10min later which kills the script pod and moves it to different node.
This can't happen, how do I configure the cronjob script pod so that it never gets killed by autoscaler?
Thanks!
This can't happen, how do I configure the cronjob script pod so that it never gets killed by autoscaler?
You can use the Node or POD affinity for scheduling the POD on specific Node so cronjob will not runt he PODs on Spot instances only you stateless workload will run on spot instance about which you don't care much.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- web-store
topologyKey: "kubernetes.io/hostname"
containers:
- name: hello
image: bash
command: ["echo", "Hello world"]
restartPolicy: OnFailureYou can use the Node selector also to schedule the POD on specific type of Node group which is not scaling up & down.
Node affinity example
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
Using the affinity you can plan on which node to schedule your POds.
Read more at : https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/