kubernetes (kubectl) port forwarding not working on mac for IBM MQ/Docker deployment

7/11/2018

I am trying to deploy IBM MQ to my local MAC machine using an image hosted on docker hub repository. I am using docker edge version with Kubernetes support on it.

I am able to deploy the image successfully using kubernetes and also have the Queue Manager running fine inside the container. I am also able to ssh into the container and make sure all the MQ processes are running as expected.

But when I use port forwarding using the following kubectl command, it opens the port, but does not let me telnet to it using "IP or hostname" (even from the local machine). But when I use "localhost" to telnet it works fine.

While troubleshooting, I deployed the same image using docker commands instead of kubernetes and with docker deployment, the port forwarding works as expected. It lets me telnet using IP, hostname and localhost.

So, definitely its some issue with Kubernetes port forwarding. Can some one please let me know if I am missing anything here? Let me know if there is some additional information needed from my end.

I am new to kubernetes and docker, but pretty familiar with IBM MQ.

Commands being used:

To create port forwarding rule using kubectl, checking netstat and connecting with telnet:


HOSTNAME:Test2 an0s5v4$ sudo kubectl port-forward private-reg 1414:1414 &

[1] 3001


HOSTNAME:Test2 an0s5v4$ Forwarding from 127.0.0.1:1414 -> 1414
Forwarding from [::1]:1414 -> 1414

HOSTNAME:Test2 an0s5v4$ netstat -an |grep 1414
tcp6       0      0  ::1.1414               *.*                    LISTEN     
tcp4       0      0  127.0.0.1.1414         *.*                    LISTEN  

HOSTNAME:Test2 an0s5v4$ ps -ef|grep 1414
    0  3001   920   0 10:27AM ttys006    0:00.03 sudo kubectl port-forward private-reg 1414:1414

    0  3002  3001   0 10:27AM ttys006    0:00.18 kubectl port-forward private-reg 1414:1414
  502  3007   920   0 10:28AM ttys006    0:00.00 grep 1414

HOSTNAME:Test2 an0s5v4$ telnet IP 1414

Trying IP...

telnet: Unable to connect to remote host: Connection refused

HOSTNAME:Test2 an0s5v4$ telnet localhost 1414

Trying ::1...

Connected to localhost.

Escape character is '^]'.

Handling connection for 1414

L-RCC9048942:Test2 an0s5v4$ telnet HOSTNAME 1414

Trying IP ...

telnet: Unable to connect to remote host: Connection refused

HOSTNAME:Test2 an0s5v4$ nslookup HOSTNAME

;; Truncated, retrying in TCP mode.

Name:   HOSTNAME
Address: IP

Kubernetes pod YAML file contents

apiVersion: v1

kind: Pod

metadata: 

  name: private-reg

  labels:

     app: ibmmq

spec: 

  containers: 

    - 

      env: 

        - 

          name: LICENSE

          value: accept

        - 

          name: MQ_QMGR_NAME

          value: QM4

      image: "image path in docker hub"

      name: private-reg-container

      ports: 

        - 

          containerPort: 1414

          hostPort: 1414

EDIT: ADDED K8S Service to the post

Kubernetes service YAML file contents

apiVersion: v1
kind: Service
metadata:
  name: myservice-nodeport
  labels:
    app: ibmmq
spec:
  ports:
  - port: 3000 
    targetPort: 1414
    nodePort: 31414
  selector:
    app: ibmmq
  type: NodePort
-- Anurag
docker
ibm-mq
kubectl
kubernetes
telnet

1 Answer

9/13/2018

This is an known issue with the kubectl port-forward command. It only connects to the localhost interface. See this PR: https://github.com/kubernetes/kubernetes/pull/46517. I still am looking onto this same issue too. I see two work-arounds:

  1. socat option: (need to install with brew install socat) https://github.com/kubernetes/kubernetes/issues/43962#issuecomment-418305960, which I don't really like as I need another port to listen to than to forward
  2. pf solution: https://superuser.com/questions/473039/pfctl-port-forwarding-in-mac-osx#521803, which I still can't get working.

I added to the /etc/pf.conf file rules like: rdr pass on en0 inet proto tcp from any to any port 10001 -> 127.0.0.1 port 10001, where en0 is my wifi interface:

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether 6c:96:cf:dd:98:e9 
    inet6 fe80::b0:4b17:388d:8a1b%en0 prefixlen 64 secured scopeid 0x6 
    inet 192.168.12.82 netmask 0xfffffc00 broadcast 192.168.15.255
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

I have a terminal with the kubectl port-forward: kubectl port-forward mypod 10001:10001, but there is not listener on the *:10001 port (you can check with the command: sudo lsof -PiTCP -sTCP:LISTEN)

If I fix it I will edit this answer.

-- Sven
Source: StackOverflow