I have GKE and I need to use customised Ubuntu image for GKE nodes. I am planning to enable autoscaling. So I require to install TLS certificates to trust the private docker registry in each nodes. It's possible for existing nodes manually. But when I enable auto scale the cluster, it will spin up the nodes. Then docker image pull request will fail, because of the docker cannot trust the private docker registry which hosted in my on premise.
I have created a customised Ubuntu image and uploaded to image in GCP. I was trying to create a GKE and tried to set the node's OS image as that image which I created.
Do you know how to create a GKE cluster with customised Ubuntu Image? Has anyone experienced with incidents like this?
Node pools in GKE are based off GCE instance templates and can't be modified. That means that you aren't allowed to set metadata such as startup-scripts or make them based on custom images.
However, an alternative approach might be deploying a privileged DaemonSet that manipulates the underlying OS settings and resources.
Is important to mention that giving privileges to resources in Kubernetes must be done carefully.
You can add a custom pool where the image is Ubuntu and be sure to add the special GCE instance metadata startup-script and then you can put your customization on it.
But my advice is to put the URL of a shell script stored in a bucket of the same project, GCE will download every time a new node is created and will execute it on the startup as a root.
https://cloud.google.com/compute/docs/startupscript#cloud-storage