Kubernetes nginx ingress-controller on Azure not reachable

9/11/2017

I am very new to Azure, Kubernetes, even Docker itself, and playing with the system to learn and evaluate for a possible deployment later. I have so far dockerized my services and successfully deployed them and made the web frontend publicly visible using a service with type: LoadBalancer.

Now I would like to add TLS termination and have learned that for that I am supposed to configure an ingress controller with the most commonly mentioned one being nginx-ingress-controller.

Strictly monkeying examples and then afterwards trying to read up on the docs I have arrived at a setup that looks interesting but does not work. Maybe some kind soul can point out my mistakes and/or give me pointers on how to debug this and where to read more about it.

I have kubectl apply'd the following file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend-deployment
  namespace: kube-system
spec:
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: default-http-backend
          image: gcr.io/google_containers/defaultbackend:1.0
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend-service
  namespace: kube-system
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80  
  selector:
    app: default-http-backend
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-controller-conf
  namespace: kube-system
data:
  # enable-vts-status: 'true'
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller-deployment
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-ingress-controller
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13
          name: nginx-ingress-controller
          ports:
            - containerPort: 80
              hostPort: 80
            - containerPort: 443
              hostPort: 443
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-ingress-controller-conf
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-controller-service
  namespace: kube-system
spec:
  ports:
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx-ingress-controller
  sessionAffinity: None
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: 
      http:
        paths:
          - path: /
            backend:
              serviceName: default-http-backend-service
              servicePort: 80

This gave me two pods:

c:\Projects\Release-Management\Azure>kubectl get pods --all-namespaces

NAMESPACE     NAME                                                   READY     STATUS    RESTARTS   AGE
<some lines removed>
kube-system   default-http-backend-deployment-3108185104-68xnk       1/1       Running   0          39m
<some lines removed>
kube-system   nginx-ingress-controller-deployment-4106313651-v7p03   1/1       Running   0          24s

Also two new services. Note that I have also configured the default-http-backend-service with type: LoadBalancer, this is for debugging only. I have included my web-frontend which is called webcms:

c:\Projects\Release-Management\Azure>kubectl get services --all-namespaces

NAMESPACE     NAME                               CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
<some lines removed>
default       webcms                             10.0.105.59    13.94.250.173   80:31400/TCP                 23h
<some lines removed>
kube-system   default-http-backend-service       10.0.106.233   13.80.68.38     80:31639/TCP                 41m
kube-system   nginx-ingress-controller-service   10.0.33.80     13.95.30.39     443:31444/TCP,80:31452/TCP   37m

And finally an ingress:

c:\Projects\Release-Management\Azure>kubectl get ingress --all-namespaces

NAMESPACE     NAME            HOSTS     ADDRESS      PORTS     AGE
kube-system   nginx-ingress   *         10.240.0.5   80        39m

No errors that I can immediately detect. I then went to the Azure Dashboard and looked at the loadbalancer and its rules and that looks good to my (seriously untrained) eye. I did not touch these, the loadbalancer and the rules were created by the system. There is a screenshot here:

https://qvwx.de/tmp/azure-loadbalancer.png

But unfortunately it does not work. I can curl my webcms-service:

c:\Projects\Release-Management\Azure>curl -v http://13.94.250.173
* Rebuilt URL to: http://13.94.250.173/
*   Trying 13.94.250.173...
* TCP_NODELAY set
* Connected to 13.94.250.173 (13.94.250.173) port 80 (#0)
<more lines removed, success>

But neither default-http-backend nor the ingress work:

c:\Projects\Release-Management\Azure>curl -v http://13.80.68.38
* Rebuilt URL to: http://13.80.68.38/
*   Trying 13.80.68.38...
* TCP_NODELAY set
* connect to 13.80.68.38 port 80 failed: Timed out
* Failed to connect to 13.80.68.38 port 80: Timed out
* Closing connection 0
curl: (7) Failed to connect to 13.80.68.38 port 80: Timed out

(ingress gives the same with a different IP)

If you read this far: Thank you for your time and I would appreciate any hints.

Marian

-- Marian Aldenhövel
azure
azure-container-service
kubernetes

1 Answer

9/17/2017

Kind of a trivial thing, but it'll save you some $$: the default-http-backend is not designed to be outside facing, and thus should not have type: LoadBalancer -- it is merely designed to 404 so the Ingress controller can universally /dev/null traffic for Pod-less Services.


Moving slightly up the triviality ladder, and for extreme clarity: I do not think what you have is wrong but I did want to offer you something to decide if you want to change. Typically the contract for a Pod's container is to give an ideally natural-language name to the port ("http", "https", "prometheus", whatever) that maps into the port of underlying image. And then, set targetPort: in the Service to that name and not the number which offers the container the ability to move the port number without breaking the Service-to-Pod contract. The nginx-ingress Deployment's container:ports: agrees with me on this one.


Now, let's get into the parts that may be contributing to your system not behaving as you wish.

I can't prove it right now, but the presence of containers:hostPort: is suspicious without hostNetwork: true. I'm genuinely surprised kubectl didn't whine, because those config combinations are a little weird.

I guess the troubleshooting step would be to get on the Node (that is, something within the cluster which is not a Pod -- you could do it with a separate VM within the same subnet as your Node, too, if you wish) and then curl to port 31452 of the Node upon which the nginx-ingress-controller Pod is running.

kubectl get nodes will list all available Nodes, and

kubectl get -o json pod nginx-ingress-controller-deployment-4106313651-v7p03 | jq -r '.items[0].status.hostIP' should produce the specific VM's IP address, if you don't already know it. Err, I just realized from your prompt you likely don't have jq -- but I don't know PowerShell well enough to know its JSON-querying syntax.

Then, from any Node: curl -v http://${that_host_IP_value}:31452 and see what materializes. It may be something, or it may be the same "wha?!" that the LoadBalancer is giving you.


As for the Ingress resource specifically, again default-http-backend is not supposed to have an Ingress resource -- I don't know if it hurts anything because I've never tried it, but I'd also bet $1 it is not helping your situation, either.

Since you already have a known working Service with default:webcms, I would recommend creating an Ingress resource in the default namespace with pretty much exactly what your current Ingress resource is, but pointed at webcms instead of default-http-backend. That way your Ingress controller will actually have something to target which isn't the default backend.

If you haven't already seen it, adding --v=2 will cause the Pod to emit the actual diff of its nginx config change, which can be unbelievably helpful in tracking down config misunderstandings


I'm so sorry that you're having to do battle with Azure, Ingress controllers, and rough-around-the-edges documentation for your first contact with Kubernetes. It really is amazing when you get all the things set up correctly, but it is a pretty complex piece of machinery, for sure.

-- mdaniel
Source: StackOverflow