How do I install Istio with fixed/static NodePort assignments?

12/12/2019

I've installed Istio on my kubeadm-installed Kubernetes cluster using:

  1. istioctl manifest generate > out.yaml

    This YAML file does not contain any specific NodePort.

  2. kubectl apply -f out.yaml

  3. kubectl -n istio-system get service istio-ingressgateway -o yaml

    Now I see random assigned NodePort numbers in the ports section, e.g.:

    [...]
    - name: https
      nodePort: 31680      # <- this is random/dynamically assigned
      port: 443
      protocol: TCP
      targetPort: 443
    [...]
    - name: prometheus
      nodePort: 32646      # <- also this one
      port: 15030
      protocol: TCP
      targetPort: 15030
    [...]

Who/what assigns these port numbers? It seems so magic and I don't like Istio to open up random ports on my nodes; this is a security concern to me!

My questions:

  • How can I tell Istio to pick a predefined NodePort port number for each port on installation time and NOT open up random ones?
  • How can I tell Istio to disable use of NodePorts on clusters where I have a cloud native LoadBalancer? I really dislike the idea of routing traffic on every single interface from every single node to a service. Some services are supposed to be hidden/firewalled and this Istio behaviour undermines my security policies.

I would like to have a way to do this on installation time already rather than patching the Istio-dynamically managed service/istio-ingressgateway.

I've found:

-- gertvdijk
istio
kubernetes-ingress

1 Answer

12/12/2019

To address Your first question:

This is because the LoadBalancer service type uses NodePort. For example i created the following LoadBalancer service:

apiVersion: v1
kind: Service
metadata:
  name: examplelb
spec:
  type: LoadBalancer
  selector:
    app: asd
  ports:
    -
      name: koala
      port: 22223
      targetPort: 22225
    -
      name: grisly
      port: 22224
      targetPort: 22226
    -
      name: polar
      port: 22225
      targetPort: 22227

And here is how it looked after deploying:

$ kubectl apply -f loadbalancer.yaml 
service/examplelb created

$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                           AGE
examplelb    LoadBalancer   10.111.8.204    <pending>     22223:31776/TCP,22224:32400/TCP,22225:32539/TCP   6s
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP                                           6d2h
sleep        ClusterIP      10.108.213.84   <none>        80/TCP                                            25h

$ kubectl describe svc examplelb
Name:                     examplelb
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"examplelb","namespace":"default"},"spec":{"ports":[{"name":"koala...
Selector:                 app=asd
Type:                     LoadBalancer
IP:                       10.111.8.204
Port:                     koala  22223/TCP
TargetPort:               22225/TCP
NodePort:                 koala  31776/TCP
Endpoints:                <none>
Port:                     grisly  22224/TCP
TargetPort:               22226/TCP
NodePort:                 grisly  32400/TCP
Endpoints:                <none>
Port:                     polar  22225/TCP
TargetPort:               22227/TCP
NodePort:                 polar  32539/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
$

As You can see Kubernetes API automatically allocated each NodePort port dynamically. This is explained in kubernetes documentation:

Type NodePort

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767). Each node proxies that port (the same port number on every Node) into your Service. Your Service reports the allocated port in its .spec.ports[*].nodePort field.


This is why in the default configuration of istio like in the out.yaml manifest you generated You can find the following Service definition:

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  annotations:
  labels:
    app: istio-ingressgateway
    release: istio
    istio: ingressgateway
spec:
  type: LoadBalancer
  selector:
    app: istio-ingressgateway
  ports:
    -
      name: status-port
      port: 15020
      targetPort: 15020
    -
      name: http2
      port: 80
      targetPort: 80
    -
      name: https
      port: 443
    -
      name: kiali
      port: 15029
      targetPort: 15029
    -
      name: prometheus
      port: 15030
      targetPort: 15030
    -
      name: grafana
      port: 15031
      targetPort: 15031
    -
      name: tracing
      port: 15032
      targetPort: 15032
    -
      name: tls
      port: 15443
      targetPort: 15443

This results in NodePort configuration you mentioned.

According to documentation You can pick specific port number for NodePort service.

If you want a specific port number, you can specify a value in the nodePort field. The control plane will either allocate you that port or report that the API transaction failed. This means that you need to take care about possible port collisions yourself. You also have to use a valid port number, one that’s inside the range configured for NodePort use.

Using a NodePort gives you the freedom to set up your own load balancing solution, to configure environments that are not fully supported by Kubernetes, or even to just expose one or more nodes’ IPs directly.

So by specifying nodePort:<PORT_NUMBER> i was able to chose a port:

apiVersion: v1
kind: Service
metadata:
  name: examplelb
spec:
  type: LoadBalancer
  selector:
    app: asd
  ports:
    -
      name: koala
      port: 22223
      targetPort: 22225
      nodePort: 31913
    -
      name: grisly
      port: 22224
      targetPort: 22226
      nodePort: 31914
    -
      name: polar
      port: 22225
      targetPort: 22227
      nodePort: 31915

Which resulted with:

$ kubectl describe svc examplelb
Name:                     examplelb
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"examplelb","namespace":"default"},"spec":{"ports":[{"name":"koala...
Selector:                 app=asd
Type:                     LoadBalancer
IP:                       10.111.8.204
Port:                     koala  22223/TCP
TargetPort:               22225/TCP
NodePort:                 koala  31913/TCP
Endpoints:                <none>
Port:                     grisly  22224/TCP
TargetPort:               22226/TCP
NodePort:                 grisly  31914/TCP
Endpoints:                <none>
Port:                     polar  22225/TCP
TargetPort:               22227/TCP
NodePort:                 polar  31915/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Solution:

So by modifying Your out.yaml manifest by adding nodePort: annotations you can pre-define ports you wish to use.

Hope this helps.

-- Piotr Malec
Source: StackOverflow