I have installed a simple two member (master, node) k8s cluster on Centos7.4 Followed https://www.linuxtechi.com/install-kubernetes-1-7-centos7-rhel7/

 kubectl.exe get nodes
NAME               STATUS    ROLES     AGE       VERSION
x042.x.int.com     Ready     <none>    36m       v1.10.1
x051.x.int.com     Ready     master    40m       v1.10.1

I have installed:

• tiller
• helm
• weave (networking)
• helmfile installation of prometheus and grafana

Everything looks perfect and I have set up NodePort for grafana service

 kubectl.exe get svc,pods
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
svc/graf-grafana                         NodePort    10.103.172.192   <none>        80:30598/TCP   33m
svc/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP        42m
svc/prom-prometheus-alertmanager         ClusterIP   10.98.26.164     <none>        80/TCP         33m
svc/prom-prometheus-kube-state-metrics   ClusterIP   None             <none>        80/TCP         33m
svc/prom-prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP       33m
svc/prom-prometheus-pushgateway          ClusterIP   10.103.22.73     <none>        9091/TCP       33m
svc/prom-prometheus-server               ClusterIP   10.103.210.21    <none>        80/TCP         33m

NAME                                                     READY     STATUS    RESTARTS   AGE
po/graf-grafana-8748b7bf-2pc8g                           1/1       Running   1          33m
po/prom-prometheus-alertmanager-688fb9dbbf-ptdpr         2/2       Running   2          33m
po/prom-prometheus-kube-state-metrics-5ccdb7cb7c-qf8xq   1/1       Running   1          33m
po/prom-prometheus-node-exporter-r56zh                   1/1       Running   1          33m
po/prom-prometheus-pushgateway-5b6f76698f-jhqhh          1/1       Running   1          33m
po/prom-prometheus-server-7d9564d579-lj4jj               2/2       Running   2          33m

So far so good but there is a problem.

When I am trying to access

$ curl x042.x.int.com:30598 -v
* STATE: INIT => CONNECT handle 0x600057990; line 1423 (connection #-5000)
* Rebuilt URL to: N.N.N.N:30598/
* Added connection 0. The cache now contains 1 members
*   Trying N.N.N.N...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x600057990; line 1475 (connection #0)

but when I ssh to actual node

ssh x042.x.int.com or ssh x051.x.int.com

I can easily access

curl x042.x.int.com:30598 -v

and I will get response I am expecting

<a href="/login">Found</a>.

HINT

lsof -i :30598
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kube-prox 3094 root    9u  IPv6  29431      0t0  TCP *:30598 (LISTEN)