The situation is:
I am trying to deploy HDFS on Kubernetes. I am using DaemonSet to start one datanode on one node and the network mode for the pod is hostNetWork. In HDFS, datanodes and namenodes communicate by hostname. But now, if I add a new node to k8s cluster, existed nodes cannot get the internal IP of the new node by the hostname of the new node.
That is, I want to get node's Internal IP by hostname. Could kube-dns do this now ? Or using some other method to scale nodes for HDFS cluster on k8s.
Thanks a lot!
Expose Pod Information to Containers Through Environment Variables
You can get the node name, node IP where a pod is deployed as below.
spec:
containers:
- name: hdfs-pod
env:
- name: MY_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountNameAs in DNS for Services and Pods, you can then use the pod IP or name ,and namespace to query the K8S internal DNS.
When enabled, pods are assigned a DNS A record in the form of “pod-ip-address.my-namespace.pod.cluster.local”
How about considering using StatefulSet? Each pod gets a specific name and based on the name, able to assign a specific role e.g. name node.
Just quickly search and found several articles on HDFS with K8S/StaefulSet.