What happens with kubernetes's HostPath's volume in case of pod dying and rising again on another node?
I guess a new pod on a new rised node will not see it? Will the volume live forever and leak diskspace?
What is the right solution to prevent that?
Host-path volumes are kind of an "escape hatch". There's no particular persistence, life cycle, or anything else associated with them. If a pod gets rescheduled on a different node, then
It will see the contents of the same host directory on the other node, which may or may not be the same, and
If you were storing application data in that host directory on the first node it will in fact become "orphaned".
This makes host-path volumes not a good match for normal application data; choose pretty much any other storage type.
The place where I've seen them used effectively is where you do actually need to manage some data that normally lives on the host system. The standard fluentd Kubernetes deployment, for example, runs fluentd in a daemon set to collect the logs from /var/lib/docker/containers on every node in the cluster; that's data managed by the host Docker daemon and not by any particular container. Because it's managed by a daemon set, it runs on every node, so if a node goes away its logs and its log forwarder will go away with it, and that's fairly expected.
A slightly better solution than HostPath is to use local PersistentVolumes
Local volumes can only be used as a statically created PersistentVolume. Dynamic provisioning is not supported yet. Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.
However, local volumes are still subject to the availability of the underlying node and are not suitable for all applications. If a node becomes unhealthy, then the local volume will also become inaccessible, and a Pod using it will not be able to run. Applications using local volumes must be able to tolerate this reduced availability, as well as potential data loss, depending on the durability characteristics of the underlying disk.
An external static provisioner can be run separately for improved management of the local volume lifecycle. Note that this provisioner does not support dynamic provisioning yet. For an example on how to run an external local provisioner, see the local volume provisioner user guide.
The local PersistentVolume requires manual cleanup and deletion by the user if the external static provisioner is not used to manage the volume lifecycle.