Welcome to the community @gil-shelef!

Based on AWS documentation, there should be used additional handlers to add both resilience and cost-savings.

Let's start with understanding how this works: There is a specific node termination handler DaemonSet which adds pods to each spot instances and listens to spot instance interruption notification. This provides a possibility to gracefully terminate any running pods on that node, drain the node from loadbalancer and for Kubernetes scheduler to reschedule removed pods on different instances.

Workflow looks like following (taken from aws documentation - Spot Instance Interruption Handling. This link also has an example):

The workflow can be summarized as:

• Identify that a Spot Instance is about to be interrupted in two minutes.
• Use the two-minute notification window to gracefully prepare the node for termination.
• Taint the node and cordon it off to prevent new pods from being placed on it.
• Drain connections on the running pods.

Once pods are removed from endpoints, kube-proxy will trigger an update in iptables. It takes a little bit of time. To make this smoother for end-users, you should consider adding pre-stop pause about 5-10 seconds. More information about how this happens and how you can mitigate it, you can find in my answer here.

Also here are links for these handlers:

• Node termination handler
• Cluster autoscaler on AWS

For your last question, please check this AWS KB article on how to troubleshoot EKS and 504 errors