How is the preemption notice handled?

4/19/2018

I'm currently running on AWS and use kube-aws/kube-spot-termination-notice-handler to intercept an AWS spot termination notice and gracefully evict the pods.

I'm reading this GKE documentation page and I see:

Preemptible instances terminate after 30 seconds upon receiving a preemption notice.

Going into the Compute Engine documentation, I see that a ACPI G2 Soft Off is sent 30 seconds before the termination happens but this issue suggests that the kubelet itself doesn't handle it.

So, how does GKE handle preemption? Will the node do a drain/cordon operation or does it just do a hard shutdown?

-- rhefner1
google-cloud-platform
google-kubernetes-engine
kubernetes

2 Answers

2/25/2019

More recent and relevant answer

There's a GitHub project (not mine) that catches this ACPI handler and has the node cordon and drain itself, and then restart itself which in our tests results in a much cleaner preemptible experience, it's almost not noticeable with a highly available deployments on your cluster.

See: https://github.com/GoogleCloudPlatform/k8s-node-termination-handler

-- Farley
Source: StackOverflow

4/19/2018

Yes you are right, so far there is no built in way to handle ACPI G2 Soft Off.

Notice that if normal preemptible instance supports shutdown scripts (where you could introduce some kind of logic to perform drain/cordon), this is not the case if they are Kubernetes nodes:

Currently, preemptible VMs do not support shutdown scripts.

You can perform some test but quoting again from documentation:

You can simulate an instance preemption by stopping the instance.

And so far if you stop the instance, even if it is a Kubernetes node no action is taken to cordon/drain and gratefully remove the node from the cluster.

However this feature is still in beta therefore it is at its early stage of life and in this moment it is a matter of discussion if and how introduce this feature.

Disclaimer: I work For Google Cloud Platform Support

-- GalloCedrone
Source: StackOverflow