Kubernetes nginx Ingress configuration not working for Grafana

9/25/2018

I am new to configuring Ingress rules for my Kubernetes cluster.

My Kubernetes cluster is deployed on Bare Metal. No cloud.

I followed this link to set up my nginx-controller with RBAC in my cluster.

This is what I have deployed :

# kubectl get all -n ingress-nginx
NAME                                           READY     STATUS    RESTARTS   AGE
pod/default-http-backend-7c5bc89cc9-ks6kd      1/1       Running   0          2h
pod/nginx-ingress-controller-5b6864749-8xbhf   1/1       Running   0          2h

NAME                           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/default-http-backend   ClusterIP   10.233.15.56   <none>        80/TCP                       2h
service/ingress-nginx          NodePort    10.233.38.84   <none>        80:31118/TCP,443:32003/TCP   2h

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/default-http-backend       1         1         1            1           2h
deployment.apps/nginx-ingress-controller   1         1         1            1           2h

NAME                                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/default-http-backend-7c5bc89cc9      1         1         1         2h
replicaset.apps/nginx-ingress-controller-5b6864749   1         1         1         2h

Given that I have my setup, I want to access my grafana dashboard using a URL.

My grafana setup is working perfectly fine.

# kubectl get all -n default
NAME                           READY     STATUS    RESTARTS   AGE
pod/grafana-67c6585fbd-4jl7p   1/1       Running   0          2h

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/grafana      NodePort    10.233.5.111   <none>        3000:32093/TCP   2h


NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1         1         1            1           2h

NAME                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/grafana-67c6585fbd   1         1         1         2h

I can access the dashboard using http://10.27.239.145:32093 which is the IP of one of my K8S worker nodes.

Now rather than accessing via IP:NodePort, I want to access via URL e.g. grafana.test.mydomain.com

So the ingress rule that I configured in my default namespace is :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  creationTimestamp: 2018-09-25T20:32:24Z
  generation: 5
  name: grafana
  namespace: default
  resourceVersion: "28485"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/jenkins-tls
  uid: 1c51cece-c102-11e8-bf0f-02000a1bef39
spec:
  rules:
  - host: grafana.test.mydomain.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
        path: /

On my local laptop from where I am testing, Ive added to my /etc/hosts the following entry :

10.27.239.145 grafana.test.mydomain.com

And in my browser, I am trying to access http://grafana.test.mydomain.com but I only get This site can’t be reached grafana.test.mydomain.com refused to connect.

I have a strong feeling that I am missing out on something but can't figure it out.

I changed the NodePort to ClusterIP but no luck.

I know that my ingress controller is working since everytime I make a change to my ingress rules, I get logs from my ingress controller.

I0925 21:00:19.041440       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"grafana", UID:"1c51cece-c102-11e8-bf0f-02000a1bef39", APIVersion:"extensions/v1beta1", ResourceVersion:"28485", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/grafana
I0925 21:00:19.041732       9 controller.go:171] Configuration changes detected, backend reload required.
I0925 21:00:19.216044       9 controller.go:187] Backend successfully reloaded.
I0925 21:00:19.217645       9 controller.go:204] Dynamic reconfiguration succeeded.

Any help will strongly be appreciated regarding what might I have missed.

--
kubernetes
kubernetes-ingress
nginx-ingress

2 Answers

9/26/2018

From what I see, you need to set grafana.test.mydomain.com to point to 10.233.38.84.

Basically, your nginx controller service is directing the traffic to your ingress and then your ingress forwards it to the backend on the nodePort (this is implicit in the ingress). It works for me, but I'm using an AWS ELB, I basically set grafana.test.mydomain.com to point to aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-0000000000.us-west-2.elb.amazonaws.com

$ kubectl get all -n ingress-nginx
NAME                                            READY     STATUS    RESTARTS   AGE
pod/default-http-backend-6586bc58b6-snxbv       1/1       Running   0          1h
pod/grafana-5b969bb7f9-tsv5k                    1/1       Running   0          52m
pod/nginx-ingress-controller-6bd7c597cb-lfwcf   1/1       Running   0          1h
pod/prometheus-server-5dbf9f4fc9-mnwn4          1/1       Running   0          53m

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                      AGE
service/default-http-backend   ClusterIP      10.x.x.x         <none>                                                                    80/TCP                       1h
service/grafana                NodePort       10.x.x.x         <none>                                                                    3000:30073/TCP               52m
service/ingress-nginx          LoadBalancer   10.x.x.x         aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-0000000000.us-west-2.elb.amazonaws.com   80:30276/TCP,443:32011/TCP   1h
service/prometheus-server      NodePort       10.x.x.x         <none>                                                                    9090:32419/TCP               53m

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/default-http-backend       1         1         1            1           1h
deployment.apps/grafana                    1         1         1            1           52m
deployment.apps/nginx-ingress-controller   1         1         1            1           1h
deployment.apps/prometheus-server          1         1         1            1           53m

NAME                                                  DESIRED   CURRENT   READY     AGE
replicaset.apps/default-http-backend-6586bc58b6       1         1         1         1h
replicaset.apps/grafana-5b969bb7f9                    1         1         1         52m
replicaset.apps/nginx-ingress-controller-6bd7c597cb   1         1         1         1h
replicaset.apps/prometheus-server-5dbf9f4fc9          1         1         1         53m

$ kubectl describe ingress grafana-ingress -n ingress-nginx
Name:             grafana-ingress
Namespace:        ingress-nginx
Address:          aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-0000000000.us-west-2.elb.amazonaws.com
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  grafana.test.mydomain.com
                             /   grafana:3000 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"grafana-ingress","namespace":"ingress-nginx"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000},"path":"/"}]}}]}}

  nginx.ingress.kubernetes.io/rewrite-target:  /
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  CREATE  40m                nginx-ingress-controller  Ingress ingress-nginx/grafana-ingress
  Normal  UPDATE  22m (x2 over 40m)  nginx-ingress-controller  Ingress ingress-nginx/grafana-ingress
-- Rico
Source: StackOverflow

9/25/2018

As far as I can see, you only have a NodePort Service on port 32093.

Your NodePort publishes the port 3000 to 32093 to any external node address as you have already proven, but you configured Ingress to contact port 3000 on grafana service.

Either add the targetPort, port and nodePort to the service for your Grafana instance and point targetPort and port to 3000 and leave nodePort empty/set it to 32092. Then the ingress should work as you posted. Snippet:

nodePort: 32093
port: 3000
protocol: TCP
targetPort: 3000

Or try to set servicePort: 3000 in your ingress configuration to 32093. Warning: I never tested this. I do not know if Ingress supports that. According to the documentation it should as NodePort is a superset of ClusterIP:

NodePort: Exposes the service on each Node’s IP at a static port (the NodePort). A ClusterIP service, to which the NodePort service will route, is automatically created. You’ll be able to contact the NodePort service, from outside the cluster, by requesting :.

Edit Btw: http://grafana.test.mydomain.com:32093 should then already work with your configuration (NodePort)

-- christoph
Source: StackOverflow