Nginx and Ingress with Kubernetes not routing my request

9/14/2017

I have Docker, Kubernetes(1.7) and Nginx all running on my RHEL7 server with my own services being inside a docker container and being picked up by Kubernetes. I know Kubernetes is working right with docker because I can call a get request of the Kubernete pod using its own IP:PORT addresses and it works. I set up Nginx with a default backend and have all of this working. I know this by calling get pods and get svc commands and everything is running as it should. When I create ingress, I know Nginx is picking it up because when I use the command kubectl describe pods {NGNIX-CONTROLLER} I see it updates its ingress and even logs what I named it. Now I get the IP address of Kubernetes master using kubectl clusterinfo and I use this ip address to attempt to call my services, something along the lines of http://KUBEIPADDRESS/PATH/TO/MY/SERVICE, with no port number but it doesn't work. I have no idea what is going on. Can someone help me why Ingress and/or Nnginx isn't routing properly to my services? I'll give my ingress and nginx file down below.

(Note, for the nginx yaml file, the deployment of the nginx controller is all the way in the bottom.)

Ingress yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/rewrite-target: /
spec:
  backend:
    serviceName: default-http-backend
    servicePort: 80
  rules:
  - host: testhost
    http:
      paths:
      - path: /customer
        backend:
          serviceName: customer
          servicePort: 9001

nginx controller yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ingress
rules:
- apiGroups:
  - ""
  - "extensions"
  resources:
  - configmaps
  - secrets
  - services
  - endpoints
  - ingresses
  - nodes
  - pods
  verbs:
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - ingresses
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - events
  - services
  verbs:
  - create
  - list
  - update
  - get
- apiGroups:
  - "extensions"
  resources:
  - ingresses/status
  - ingresses
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: ingress-ns
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
  - create
  - update  
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: ingress-ns-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-ns
subjects:
  - kind: ServiceAccount
    name: ingress
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingress-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress
subjects:
  - kind: ServiceAccount
    name: ingress
    namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      # like with kubeadm
      hostNetwork: true
      terminationGracePeriodSeconds: 60
      serviceAccountName: ingress
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend

Also when I do kubectl describe ing I get

Name:                   gateway-ingress
Namespace:              default
Address:
Default backend:        default-http-backend:80 (<none>)
Rules:
  Host          Path    Backends
  ----          ----    --------
  testhost

                /customer    customer:9001 ({IP}:9001,{IP}:9001)
Annotations:
  rewrite-target:       /
Events:                 <none>

Here are my deployment and service of the customer in case anyone needs that

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: customer
  labels:
    run: customer
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: customer
    spec:
      containers:
      - name: customer
        image: customer
        imagePullPolicy: Always
        ports:
        - containerPort: 9001
          protocol: TCP
---
kind: Service
apiVersion: v1
metadata:
  name: customer
spec:
  selector:
    run: customer
  type: NodePort
  ports:
  - name: port1
    protocol: TCP
    port: 9001
    targetPort: 9001
-- anonuser1234
docker
kubernetes
nginx
rhel7

1 Answer

9/16/2017

There are some issues with your setup as far as I can see:

  • KUBEIPADDRESS in the URL you call: an IP address won't work because you configured your Ingress to listen on testhost. So you need to call http://testhost/customer, and configure your network to resolve testhost to the correct IP address

  • but what is the correct IP address? You are trying to use k8s master on port 80. That won't work without further configuration. For that you need to use a NodePort service for the Ingress Controller, which exposes it on port 80 (and probably 433). In order to use that low ports, you need to allow it with an option of kube-apiserver, see --service-node-port-range on https://kubernetes.io/docs/admin/kube-apiserver/. Once that works, you can use any IP address of any node of your k8s cluster for testhost. Note: be sure that no other application uses these ports on any node!

-- slintes
Source: StackOverflow