Failed to get response from nginx ingress controller

9/8/2021

I deploy an Nginx Ingress Controller with 1 replica, exposed by Nodeport.

[root@test-120 test]# kubectl get pod -n ingress -owide
NAME                                           READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
nginx-ingress-nginx-ingress-5b84865587-srtvt   1/1     Running   0          46m   10.96.154.5   test-120   <none>           <none>

And deploy an EchoServer on each node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  replicas: 3
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - echoserver
              topologyKey: "kubernetes.io/hostname"
      containers:
        - image: ealen/echo-server
          imagePullPolicy: IfNotPresent
          name: echoserver
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: echoserver
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: "*.amazonaws.com.cn"
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: echoserver
                port:
                  number: 80

When try to access the ingress on my PC

curl http://test.amazonaws.com.cn

I can't get the response, and here is the Nginx Ingress log

kubectl logs -f nginx-ingress-nginx-ingress-5b84865587-srtvt -ningress

2021/09/08 14:00:04 [warn] 134#134: *21 upstream server temporarily disabled while reading response header from upstream, client: 10.88.0.1, server: *.amazonaws.com.cn, request: "GET / HTTP/1.1", upstream: "http://10.96.175.3:80/", host: "test.amazonaws.com.cn"
2021/09/08 14:00:04 [error] 134#134: *21 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.88.0.1, server: *.amazonaws.com.cn, request: "GET / HTTP/1.1", upstream: "http://10.96.175.3:80/", host: "test.amazonaws.com.cn"

With the logs, I try to enter the Ingress Pod and access the Upstream Server, it works fine.

kubectl exec -it nginx-ingress-nginx-ingress-5b84865587-srtvt -ningress -- curl http://10.96.175.3:80

output:

{"host":{"hostname":"10.96.113.66","ip":"::ffff:10.96.154.5","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/","protocol":"http"},"request":{"params":{"0":"/"},"query":{},"cookies":{},"body":{},"headers":{"host":"10.96.113.66","user-agent":"curl/7.64.0","accept":"*/*"}},"environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME":"echoserver-7d74b86448-4zjsf","ECHOSERVER_PORT":"tcp://10.111.205.38:80","ECHOSERVER_PORT_80_TCP_PORT":"80","KUBERNETES_SERVICE_PORT_HTTPS":"443","KUBERNETES_PORT":"tcp://10.96.0.1:443","KUBERNETES_PORT_443_TCP":"tcp://10.96.0.1:443","KUBERNETES_PORT_443_TCP_ADDR":"10.96.0.1","ECHOSERVER_SERVICE_PORT":"80","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_PORT_443_TCP_PORT":"443","ECHOSERVER_PORT_80_TCP_PROTO":"tcp","ECHOSERVER_PORT_80_TCP_ADDR":"10.111.205.38","KUBERNETES_SERVICE_HOST":"10.96.0.1","ECHOSERVER_SERVICE_HOST":"10.111.205.38","ECHOSERVER_PORT_80_TCP":"tcp://10.111.205.38:80","NODE_VERSION":"14.17.1","YARN_VERSION":"1.22.5","HOME":"/root"}}

So I want to find out why the nginx ingress controller can't get response from upstream server http://10.96.175.3:80 when I curl it on PC, but it can get response by curl from itself.

-- Janson
amazon-web-services
curl
kubernetes
nginx-ingress

0 Answers