K8s Ingress service returning 503, nothing in Pod logs

7/1/2018

I am running an Ingress, which supposed to connect to images inside my Pods. When I do describe it looks fine e.g.

$ kubectl describe svc solar-demo
Name:              solar-demo
Namespace:         default
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"solar-demo","namespace":"default"},"spec":{"ports":[{"name":"city","port":3000...
Selector:          app=solardemo
Type:              ClusterIP
IP:                10.97.245.248
Port:              city  3000/TCP
TargetPort:        3000/TCP
Endpoints:         172.17.0.4:3000,172.17.0.6:3000
Port:              solar  3001/TCP
TargetPort:        3001/TCP
Endpoints:         172.17.0.4:3001,172.17.0.6:3001
Session Affinity:  None
Events:            <none>

It even correctly lists IP addresses of my images. However, when I try to reach services, I get 404 when I just ask for a root, which is fine since I do not map map / to anything and 503 error when I try to reach routes /solar and/or /city.

When I check the logs, it returns:

$ kubectl logs solar-demo-5845984b94-xp82l solar-svc
npm info it worked if it ends with ok
npm info using npm@5.6.0
npm info using node@v9.11.2
npm info lifecycle nf-images-test@1.0.0~prestart-solar-svc: nf-images-test@1.0.0
npm info lifecycle nf-images-test@1.0.0~start-solar-svc: nf-images-test@1.0.0

> nf-images-test@1.0.0 start-solar-svc /opt/app-root/src
> node solar-svc.js

{"level":30,"time":1530271233676,"msg":"Server listening at http://0.0.0.0:3001","pid":26,"hostname":"solar-demo-5845984b94-xp82l","v":1}
server listening on 3001

and the same thing for the other service:

$ kubectl logs solar-demo-5845984b94-xp82l api
npm info it worked if it ends with ok
npm info using npm@5.6.0
npm info using node@v9.11.2
npm info lifecycle nf-images-test@1.0.0~prestart-api: nf-images-test@1.0.0
npm info lifecycle nf-images-test@1.0.0~start-api: nf-images-test@1.0.0

> nf-images-test@1.0.0 start-api /opt/app-root/src
> node api.js

{"level":30,"time":1530271244205,"msg":"Server listening at http://0.0.0.0:3000","pid":21,"hostname":"solar-demo-5845984b94-xp82l","v":1}
server listening on 3000

I get 503s and images never get any requests, as if Ingress "thought" that every Pod is down or something. What could I check?

$ curl -v http://shmukler.example.com/solar
*   Trying 192.168.99.101...
* Connected to shmukler.example.com (192.168.99.101) port 80 (#0)
> GET /solar HTTP/1.1
> Host: shmukler.example.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 503 Service Temporarily Unavailable
< Server: nginx/1.13.7
< Date: Sun, 01 Jul 2018 13:49:38 GMT
< Content-Type: text/html
< Content-Length: 213
< Connection: keep-alive
< 
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.13.7</center>
</body>
</html>
* Connection #0 to host shmukler.example.com left intact

Please advise.

-- Moshe Shmukler
kubernetes
kubernetes-ingress
minikube
nginx

1 Answer

7/1/2018

In annotations, the config was missing:

nginx.org/server-snippet: "proxy_ssl_verify off;"
-- Moshe Shmukler
Source: StackOverflow