Nginx Ingress giving 404 not found for any ingress resource

8/9/2019

Nginx controller is throwing 404 error for any ingress resource setup in local.

nginx-ingress controller setup was made according to steps detailed in the below document and ingress controller was exposed as NodePort Service in local setup

https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md#5-access-the-live-activity-monitoring-dashboard--stub_status-page

The following sample ingress resources were used to test ingress banana.yaml

kind: Pod
apiVersion: v1
metadata:
  name: banana-app
  namespace: nginx-ingress
  labels:
    app: banana
spec:
  containers:
    - name: banana-app
      image: hashicorp/http-echo
      args:
        - "-text=banana"

---

kind: Service
apiVersion: v1
metadata:
  name: banana-service
  namespace: nginx-ingress
spec:
  selector:
    app: banana
  ports:
    - port: 5678 

apple.yaml

kind: Pod
apiVersion: v1
metadata:
  name: apple-app
  namespace: nginx-ingress
  labels:
    app: apple
spec:
  containers:
    - name: apple-app
      image: hashicorp/http-echo
      args:
        - "-text=apple"

---

kind: Service
apiVersion: v1
metadata:
  name: apple-service
  namespace: nginx-ingress
spec:
  selector:
    app: apple
  ports:
    - port: 5678 # 

ingressfile.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  namespace: nginx-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - http:
      paths:
        - path: /apple
          backend:
            serviceName: apple-service
            servicePort: 5678
        - path: /banana
          backend:
            serviceName: banana-service
            servicePort: 5678

Following is the list of services

NAMESPACE       NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
nginx-ingress   apple-service          ClusterIP   10.111.230.109   <none>        5678/TCP                     95m
nginx-ingress   banana-service         ClusterIP   10.102.139.127   <none>        5678/TCP                     95m
nginx-ingress   nginx-ingress          NodePort    10.97.65.187     <none>        80:30207/TCP,443:31031/TCP   6d23h
[root@kube01 ingress]# kubectl get ing
NAME              HOSTS              ADDRESS   PORTS     AGE
example-ingress   *                            80        131m
[root@kube01 ingress]# kubectl describe ing example-ingress
Name:             example-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /apple    apple-service:5678 (10.244.2.7:5678)
        /banana   banana-service:5678 (10.244.1.11:5678)
Annotations:
  ingress.kubernetes.io/rewrite-target:  /
Events:                                  <none>

If I directly curl the pods as follows, it's working as expected

[root@kube01 ingress]# curl http://10.244.2.7:5678/apple
apple
[root@kube01 ingress]# curl http://10.244.1.11:5678/banana
banana
[root@kube01 ingress]#

However if I try to access it through Nodeport ingress controller I'm always getting not found as below

[root@kube01 ingress]# curl http://xx.xx.xx.193:30207/apple
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.17.2</center>
</body>
</html>
[root@kube01 ingress]# curl http://xx.xx.xx.193:30207/banana
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.17.2</center>
</body>
</html>
[root@kube01 ingress]#

curl http://xx.xx.xx.193:30207/apple curl http://xx.xx.xx.193:30207/banana The above statements should display apple and banana respectively when accessed from outside or from a browser

Update: I went into the pod where the ingress controller is running and checked its config file and found that ingress resources are not applied to the controller. I tried relaunching the container with no luck. I also tried reapplying the ingress resource but the config file in the pod is not changing. Everything that goes to controller is throwing 404 as in location tag in the config file. Any help is appreciated on why the resource is not getting applied to the controller

root@nginx-ingress-685d7964cd-djvgf:/etc/nginx# cat nginx.conf

user  nginx;
worker_processes  auto;
daemon off;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';


    access_log  /var/log/nginx/access.log  main;


    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 65s;
    keepalive_requests 100;

    #gzip  on;

    server_names_hash_max_size 512;


    variables_hash_bucket_size 256;
    variables_hash_max_size 1024;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }





    server {
        listen 80 default_server;
        listen 443 ssl default_server;

        ssl_certificate /etc/nginx/secrets/default;
        ssl_certificate_key /etc/nginx/secrets/default;

        server_name _;
        server_tokens "on";
        access_log off;



        location / {
           return 404;
        }
    }
    # stub_status
    server {
        listen 8080;

        allow 127.0.0.1;
        deny all;
        location /stub_status {
            stub_status;
        }
    }

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;

    server {
        listen unix:/var/run/nginx-502-server.sock;
        access_log off;

        location / {
            return 502;
        }
    }
}

stream {
    log_format  stream-main  '$remote_addr [$time_local] '
                      '$protocol $status $bytes_sent $bytes_received '
                      '$session_time';

    access_log  /var/log/nginx/stream-access.log  stream-main;


}
root@nginx-ingress-685d7964cd-djvgf:/etc/nginx#
-- Abhinav
kubernetes-ingress
nginx
nginx-ingress

1 Answer

8/28/2019

I have few comments (what i have noticed):

1, Your ingress is working in default namespace:

  Namespace:        default
  instead of nginx-ingress namespace

2, In your kubectl describe ing example-ingress there is no annotation like:

kubernetes.io/ingress.class:  nginx

3, I got the same issue and I couldn't reach the service but it will work when you change configmap for ingress controller with ssl-redirect: "false" option. Then it will work without spec.rules.host parameter inside your ingress resources.

Hope this help

-- Hanx
Source: StackOverflow