Do you know how to integrate Kubernetes with existing Yarn cluster?
Below page explains the features that Kubernetes Yarn integration offers.
https://hortonworks.com/blog/docker-kubernetes-apache-hadoop-yarn/
However, the tutorial it refers only gives instructions to install a local cluster.
Alternative or a second question would be:
Is that advisable / rational to run Kubernetes cluster in the same physical cluster that Hadoop runs?
Thanks in advance...
The hortonworks solution seems odd to me. Kubernetes manages docker containers on physical machines, Yarn schedule jobs a cluster (built by a set of linux boxes). The nature way of integration should be letting yarn running on docker containers managed by Kubernetes. Why the opposite?
IMO, the right solution is not integrate kubernete to your existing yarn cluster. Instead, you should rebuild your cluster with micro-service oriented architecture, which means every service (yarn is one of them) is running in docker containers managed by kubernetes. Long term, this will be a more manageable, maintenance-save approach for a big data cluster.