How do I replace a downed master node? In particular, how do you replace a node on AWS if you are using the kubernetes tools?
If I restart the node, it doesn't start correctly, if I clone the node, it definitely does not.
If replacing it comes up with a new IP, you'll have to update all your nodes, which know how to reach it by IP or by internal DNS (in flags).
You'll have to connect the new master to the current etcd cluster, or create a new etcd cluster from a snapshot of the old one, in order to preserve state of the cluster. Aside from that, you have to ensure that the nodes point to the new master's IP address. I also suggest looking at HA masters if you are running a version greater than 1.0.X http://kubernetes.io/v1.1/docs/admin/high-availability.html.