Performed an upgrade of K8S AWS cluster from 1.8.4 to 1.9.8 using kops.
All "development" pods and some pods from kube-system namespace (cluster-autoscaler, kube-dns, kube-dns-autoscaler, kubernetes-dashboard, heapster) stuck in "ContainerCreating" state.
Pods describe command shows message: "Failed create pod sandbox" and "Pod sandbox changed, it will be killed and re-created."
Any ideas what could be the reason for such behavior or how to fix it?
This is becoming a FAQ
But Pod sandbox errors are almost always CNI failures, which goes double for your situation since I don't know that 1.8 even used CNI