K
Q

Connect ResourceManager for Flink FLIP-6 start new "session job"

4/25/2019

I want to create a new k8s deployment with a session job; and have one taskmanager deployed with a configuration like this in flink-conf.yaml:

jobmanager.rpc.address: analytics-job
jobmanager.rpc.port: 6123

However, it would seem that my TaskManager refuses to use the port 6123 and always picks a high ports? The analytics job's k8s service looks like this:

apiVersion: v1
kind: Service
metadata:
  name: analytics-job
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob
    port: 6124
  - name: query
    port: 6125
    # nodePort: 30025
  - name: ui
    port: 8081
    # nodePort: 30081
  selector:
    app: analytics
    stack: flink
    component: job-cluster

and as you can see I've tried both ClusterIP and NodePort service types. I'd rather have a ClusterIP type since that creates an internal load balancer in front of my k8s Job/standalone-job.sh Flink process.

-- Henrik
apache-flink
flink-streaming
kubernetes

Similar Questions

GKE with Aws worker nodes
How to avoid creating pod on node that doesn't have enough storage?
Proxy K8S app delegating authentication of requests from other pods
How to get the current namespace of current context using kubectl?

2 Answers

4/29/2019

In flink-conf.yaml, set

high-availability.jobmanager.port: 6123

That will bring the resource manager connection back down to the static port you'd like it to be using.

-- David Anderson
Source: StackOverflow
4/25/2019

The reason you don't get nodePorts like 6123, 6124, and 6125 allocated by the Kubernetes master is that there's a port range for nodePorts identified by the --service-node-port-range option on the kube-apiserver. The default for that range is 30000-32767.

You can just add that flag --service-node-port-range to your kubeapi-server configuration typically under /etc/kubernetes/manifests/kube-apiserver.yaml with some value like:

--service-node-port-range=6000-32767

Keep in mind that there could be port overlap with some other services you are running on the node.

As a side node FLIP-6 is a work in progress in K8s (as of this writing) and you can see it's pretty barebones in Kubernetes in that the jobmanager is not really dynamically creating your taskmanager and allocating resources.

-- Rico
Source: StackOverflow