I have a Kubernetes cluster running on Google Container Engine (GKE). In there, I have a preemtible node pool I use to run batch Jobs.

Google Compute Engine makes no guarantees on instance availability and they can go offline any time. On the other side, Kubernetes guarantees that the Job will be completed. I'm interested in learning how K8s handles this situation and what's it's behavior when nodes go offline, maybe half way trough the job execution...

• Does the job get rescheduled somewhere else automatically or marked as a failure?
• Do the containers have any way to know they are being cast out?
• Jobs normally have to be manually deleted to free resources, does that apply also on this situation?
• Are logs still available after the node is offline?

Thanks!