I have created two node pools. A small one for all the google system jobs and a bigger one for my tasks. The bigger one should reduce its size to 0 after the job is done.
The problem is: Even if there are no cron jobs, the node pool do not reduce his size to 0.
Creating cluster:
gcloud beta container --project "projectXY" clusters create "cluster" --zone "europe-west3-a" --username "admin" --cluster-version "1.9.6-gke.0" --machine-type "n1-standard-1" --image-type "COS" --disk-size "100" --scopes "https://www.googleapis.com/auth/cloud-platform" --num-nodes "1" --network "default" --enable-cloud-logging --enable-cloud-monitoring --subnetwork "default" --enable-autoscaling --enable-autoupgrade --min-nodes "1" --max-nodes "1"
Creating node pool:
The node pool should reduce its size to 0 after all tasks are done.
gcloud container node-pools create workerpool --cluster=cluster --machine-type="n1-highmem-8", -m "n1-highmem-8" --zone=europe-west3-a, -z europe-west3-a --disk-size=100 --enable-autoupgrade --num-nodes=0 --enable-autoscaling --max-nodes=2 --min-nodes=0
Create cron job:
kubectl create -f cronjob.yaml
I ran into the same issue and tested a number of different scenarios. I finally got it to work by doing the following:
Create your node pool with an initial size of 1 instead of 0:
gcloud container node-pools create ${NODE_POOL_NAME} \
--cluster ${CLUSTER_NAME} \
--num-nodes 1 \
--enable-autoscaling --min-nodes 0 --max-nodes 1 \
--zone ${ZONE} \
--machine-type ${MACHINE_TYPE}
Configure your CronJob in a similar fashion to this:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: cronjob800m
spec:
schedule: "7 * * * *"
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 0
successfulJobsHistoryLimit: 0
jobTemplate:
spec:
template:
spec:
containers:
- name: cronjob800m
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
resources:
requests:
cpu: "800m"
restartPolicy: Never
Note that the resources are set in a way that the job is only able to be run on the large node pool but not on the small one. Also note that we set both failedJobsHistoryLimit
and successfulJobsHistoryLimit
to 0 in order for the job to be automatically cleaned from the node pool after success/failure.
That should be it.
Quoting from Google Documentation:
"Note: Beginning with Kubernetes version 1.7, you can specify a minimum size of zero for your node pool. This allows your node pool to scale down completely if the instances within aren't required to run your workloads. However, while a node pool can scale to a zero size, the overall cluster size does not scale down to zero nodes (as at least one node is always required to run system Pods)."
Notice also that:
"Cluster autoscaler also measures the usage of each node against the node pool's total demand for capacity. If a node has had no new Pods scheduled on it for a set period of time, and [this option does not work for you since it is the last node] all Pods running on that node can be scheduled onto other nodes in the pool , the autoscaler moves the Pods and deletes the node.
Note that cluster autoscaler works based on Pod resource requests, that is, how many resources your Pods have requested. Cluster autoscaler does not take into account the resources your Pods are actively using. Essentially, cluster autoscaler trusts that the Pod resource requests you've provided are accurate and schedules Pods on nodes based on that assumption."
Therefore I would check:
If still everything likely it is an issue with the autoscaler and you can either open a private issue specifying your project ID with Google since there is not much the community can do.
If you are interested place in the comments the link of the issue tracker and I will take a look in your project (I work for Google Cloud Platform Support)