After giving nginx-controller an externalIPs list, I am able to hit the nginx load balancer but it gives me 503s. I've inspected the genereated nginx file and I cannot see any mention of the upstream servers, this part is not being populated in the nginx configuration file, I get placeholders for the upstream parts.
my deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80My ingress controller YAML:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: nginx-service
servicePort: 80My service:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
name: nginx
spec:
type: NodePort
ports:
- port: 80
name: http
selector:
name: nginxMy logs from nginx:
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.34.1
Build: v20200715-ingress-nginx-2.11.0-8-gda5fa45e2
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.1
-------------------------------------------------------------------------------
I0808 10:03:07.530453 6 flags.go:205] Watching for Ingress class: nginx
W0808 10:03:07.530755 6 flags.go:250] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0808 10:03:07.530805 6 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0808 10:03:07.531018 6 main.go:231] Creating API client for https://10.96.0.1:443
I0808 10:03:07.535965 6 main.go:275] Running in Kubernetes cluster version v1.18 (v1.18.6) - git (clean) commit dff82dc0de47299ab66c83c626e08b245ab19037 - platform linux/amd64
I0808 10:03:07.661731 6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0808 10:03:07.662262 6 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
W0808 10:03:07.663809 6 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
I0808 10:03:07.667982 6 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I0808 10:03:07.695581 6 nginx.go:263] Starting NGINX Ingress controller
I0808 10:03:07.699045 6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"7f8efb11-d326-4a79-96e1-87e9e610d189", APIVersion:"v1", ResourceVersion:"11100", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0808 10:03:08.799119 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-ingress", UID:"5b3b5c37-d81e-4144-9c8e-d4ff424401ea", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"31926", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/test-ingress
I0808 10:03:08.895939 6 nginx.go:307] Starting NGINX process
I0808 10:03:08.896060 6 leaderelection.go:242] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0808 10:03:08.896177 6 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
I0808 10:03:08.896317 6 controller.go:141] Configuration changes detected, backend reload required.
I0808 10:03:08.897732 6 status.go:86] new leader elected: ingress-nginx-controller-7fd7d8df56-7kls4
I0808 10:03:08.943105 6 controller.go:157] Backend successfully reloaded.
I0808 10:03:08.943137 6 controller.go:166] Initial sync, sleeping for 1 second.
I0808 10:03:48.343214 6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0808 10:03:48.343262 6 status.go:86] new leader elected: ingress-nginx-controller-7fd7d8df56-54v6b
I0808 10:03:48.353140 6 status.go:275] updating Ingress default/test-ingress status from [] to [{10.0.2.15 } {49.12.86.195 }]
I0808 10:03:48.355684 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-ingress", UID:"5b3b5c37-d81e-4144-9c8e-d4ff424401ea", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"33002", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/test-ingress
10.32.0.1 - - [08/Aug/2020:18:37:39 +0000] "GET / HTTP/1.1" 503 197 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0" 424 0.000 [default-nginx-1337] [] - - - - 68e58595b011156ca288edba56a53a99
10.32.0.1 - - [08/Aug/2020:18:37:42 +0000] "GET / HTTP/1.1" 503 197 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0" 367 0.000 [default-nginx-1337] [] - - - - 77659ecd8f16cbc8a1d9f9e603dcba90
W0808 18:38:37.982021 6 controller.go:916] Service "default/nginx" does not have any active Endpoint.
I0808 18:38:38.010466 6 main.go:115] successfully validated configuration, accepting ingress test-ingress in namespace default
I0808 18:38:38.012769 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-ingress", UID:"5b3b5c37-d81e-4144-9c8e-d4ff424401ea", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"106029", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/test-ingress
W0808 18:38:38.012885 6 controller.go:916] Service "default/nginx" does not have any active Endpoint.
I0808 18:38:38.012926 6 controller.go:141] Configuration changes detected, backend reload required.
I0808 18:38:38.060293 6 controller.go:157] Backend successfully reloaded.
10.32.0.1 - - [08/Aug/2020:18:38:39 +0000] "GET / HTTP/1.1" 503 197 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0" 367 0.000 [default-nginx-80] [] - - - - 49267e557ee39cfed6c41b1592d2ceafI'm not sure what kind of service the nginx service is meant to be, I suspected NodePort.
The part that is suspicious to me is
Service "default/nginx" does not have any active Endpoint.But I cannot work out what it means.
EDIT: Output of kubectl describe svc ngin
Name: nginx
Namespace: default
Labels: name=nginx
Annotations: Selector: name=nginx
Type: NodePort
IP: 10.105.68.13
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32250/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>kubectl describe pod nginx-deployment
Name: nginx-deployment-68bd55b8f6-5gcpn
Namespace: default
Priority: 0
Node: ubuntu/10.0.2.15
Start Time: Sat, 08 Aug 2020 08:33:11 +0100
Labels: app=nginx
pod-template-hash=68bd55b8f6
Annotations: <none>
Status: Terminating (lasts 3d5h)
Termination Grace Period: 30s
IP: 10.44.0.2
IPs:
IP: 10.44.0.2
Controlled By: ReplicaSet/nginx-deployment-68bd55b8f6
Containers:
nginx:
Container ID: docker://d6553f9fe848c4b535cbc05a3009accb06f7825f742f39db9e15d6511b0dba00
Image: nginx:1.14.2
Image ID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Port: 1337/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 08 Aug 2020 08:33:22 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8hfzj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
default-token-8hfzj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8hfzj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: nginx-deployment-68bd55b8f6-zpzj5
Namespace: default
Priority: 0
Node: ubuntu/10.0.2.15
Start Time: Sat, 08 Aug 2020 08:33:11 +0100
Labels: app=nginx
pod-template-hash=68bd55b8f6
Annotations: <none>
Status: Terminating (lasts 3d5h)
Termination Grace Period: 30s
IP: 10.44.0.1
IPs:
IP: 10.44.0.1
Controlled By: ReplicaSet/nginx-deployment-68bd55b8f6
Containers:
nginx:
Container ID: docker://4ed8b3e6896cc3923a4e446c102a4822e118eb66cb531df1e865ee4ad78e3fdb
Image: nginx:1.14.2
Image ID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Port: 1337/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 08 Aug 2020 08:33:21 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8hfzj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
default-token-8hfzj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8hfzj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: nginx-deployment-6b474476c4-9zsbx
Namespace: default
Priority: 0
Node: ubuntu-2004-focal-64-minimal/49.12.86.195
Start Time: Tue, 11 Aug 2020 16:14:55 +0100
Labels: app=nginx
pod-template-hash=6b474476c4
Annotations: <none>
Status: Running
IP: 10.32.0.7
IPs:
IP: 10.32.0.7
Controlled By: ReplicaSet/nginx-deployment-6b474476c4
Containers:
nginx:
Container ID: docker://c1610bd533ea78339aca54a252f4058e08805940dc2e55ab803728d00bda9274
Image: nginx:1.14.2
Image ID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 11 Aug 2020 16:14:56 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8hfzj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-8hfzj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8hfzj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m51s default-scheduler Successfully assigned default/nginx-deployment-6b474476c4-9zsbx to ubuntu-2004-focal-64-minimal
Normal Pulled 9m51s kubelet, ubuntu-2004-focal-64-minimal Container image "nginx:1.14.2" already present on machine
Normal Created 9m50s kubelet, ubuntu-2004-focal-64-minimal Created container nginx
Normal Started 9m50s kubelet, ubuntu-2004-focal-64-minimal Started container nginx
Name: nginx-deployment-6b474476c4-xlj4m
Namespace: default
Priority: 0
Node: ubuntu-2004-focal-64-minimal/49.12.86.195
Start Time: Tue, 11 Aug 2020 16:14:56 +0100
Labels: app=nginx
pod-template-hash=6b474476c4
Annotations: <none>
Status: Running
IP: 10.32.0.4
IPs:
IP: 10.32.0.4
Controlled By: ReplicaSet/nginx-deployment-6b474476c4
Containers:
nginx:
Container ID: docker://05d30b9bcbb92bccfaec96dd8298a2ed1690394910db4703a3b7bb6624920071
Image: nginx:1.14.2
Image ID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 11 Aug 2020 16:14:57 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8hfzj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-8hfzj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8hfzj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m50s default-scheduler Successfully assigned default/nginx-deployment-6b474476c4-xlj4m to ubuntu-2004-focal-64-minimal
Normal Pulled 9m50s kubelet, ubuntu-2004-focal-64-minimal Container image "nginx:1.14.2" already present on machine
Normal Created 9m50s kubelet, ubuntu-2004-focal-64-minimal Created container nginx
Normal Started 9m49s kubelet, ubuntu-2004-focal-64-minimal Started container nginxIn this case you wanted to use service with selectors. There is also option to create service without selectors.
In your Deployment you have used app: nginx, however in service you used name: nginx. More details you can find in Service Documentation.
Second thing is that in Ingress you pointed to serviceName called nginx-service where your service name is nginx.
Another thing is that you have only specified port in your service manifest. In this case is no matter as Nginx is listening on port 80, however if you wouldn't specify it, it will assign targetPort with the same value as port.
A Service can map any incoming
portto atargetPort. By default and for convenience, thetargetPortis set to the same value as theportfield.
This YAMLs should work in your case:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
labels:
app: nginx
spec:
type: NodePort
selector:
app: nginx
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: nginx-service
servicePort: 80
As last thing, please keep in mind that pathType: Prefix was introduced in Kubernetes 1.18, so it will give error if you will use it in older versions.