K
Q

Question

deployment, service, ingress implemented. Got default backend - 404

3/27/2019

I am practicing k8s deployment, service, and ingress implementation from here on GKE. Clicking through the simulator help me quickly understand concept, but when I do hands on. I stuck default backend - 404. Here are my manifest files and bash description.

deployment.yaml

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: tunnel-deployment
  labels:
    app: tunnel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tunnel
  template:
    metadata:
      labels:
        app: tunnel
    spec:
      containers:
      - name: tunnel
        image: gcr.io/k8s-v1-235608/tunnel:latest
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: MONGODB_HOST
            value: moon-mongodb-replicaset-client
          - name: RABBIT_HOST
            value: rodent-rabbitmq-headless
          - name: RABBIT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: rodent-rabbitmq
                key: rabbitmq-password
          - name: REDIS_HOST
            value: ninja-redis-ha
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

See deployments in bash

$ kubectl get deployments -o wide
NAME                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                               SELECTOR
tunnel-deployment   1         1         1            1           11m   tunnel       gcr.io/k8s-v1-235608/tunnel:latest   app=tunnel

See po in bash

$ kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP          NODE                                  NOMINATED NODE
bbox                                1/1     Running   0          1d    10.4.0.6    gke-doom-default-pool-4b763b09-pxnz   <none>
moon-mongodb-replicaset-0           1/1     Running   0          1d    10.4.2.9    gke-doom-default-pool-4b763b09-lcs6   <none>
moon-mongodb-replicaset-1           1/1     Running   0          1d    10.4.0.8    gke-doom-default-pool-4b763b09-pxnz   <none>
moon-mongodb-replicaset-2           1/1     Running   0          1d    10.4.2.10   gke-doom-default-pool-4b763b09-lcs6   <none>
ninja-redis-ha-server-0             2/2     Running   0          1d    10.4.0.9    gke-doom-default-pool-4b763b09-pxnz   <none>
ninja-redis-ha-server-1             2/2     Running   0          1d    10.4.1.11   gke-doom-default-pool-4b763b09-85ch   <none>
ninja-redis-ha-server-2             2/2     Running   0          1d    10.4.2.11   gke-doom-default-pool-4b763b09-lcs6   <none>
rodent-rabbitmq-0                   1/1     Running   0          1d    10.4.2.12   gke-doom-default-pool-4b763b09-lcs6   <none>
tunnel-deployment-fddf78dcc-lpq8l   1/1     Running   0          11m   10.4.1.37   gke-doom-default-pool-4b763b09-85ch   <none>

service.yaml. I use NodePort with my tunnel-service. With my selection.app

apiVersion: v1
kind: Service
metadata:
  name: tunnel-service
  labels:
    app: tunnel
spec:
  type: NodePort
  ports:
  - name: tunnel-port
    port: 80
    targetPort: 8080
  selector:
    app: tunnel

Then I have single IP address to represent my service.

$ kubectl describe svc tunnel-service
Name:                     tunnel-service
Namespace:                default
Labels:                   app=tunnel
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"tunnel"},"name":"tunnel-service","namespace":"default"},...
Selector:                 app=tunnel
Type:                     NodePort
IP:                       10.7.242.110
Port:                     tunnel-port  80/TCP
TargetPort:               8080/TCP
NodePort:                 tunnel-port  32713/TCP
Endpoints:                10.4.1.37:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

ingress.yaml. I route the / to my tunnel-service

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tunnel-ingress
spec:
  rules:
  - host: abc.hbot.io
    http:
      paths:
      - path: /
        backend:
          serviceName: tunnel-service
          servicePort: 80

$ kubectl describe ingress
Name:             tunnel-ingress
Namespace:        default
Address:          35.244.186.216
Default backend:  default-http-backend:80 (10.4.1.9:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc.hbot.io
               /   tunnel-service:80 (<none>)
Annotations:
  ingress.kubernetes.io/backends:                    {"k8s-be-31768--d2232907436d0807":"HEALTHY","k8s-be-32713--d2232907436d0807":"UNHEALTHY"}
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/url-map:                     k8s-um-default-tunnel-ingress--d2232907436d0807
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"tunnel-ingress","namespace":"default"},"spec":{"rules":[{"host":"abc.hbot.io","http":{"paths":[{"backend":{"serviceName":"tunnel-service","servicePort":80},"path":"/"}]}}]}}

Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     16m   loadbalancer-controller  default/tunnel-ingress
  Normal  CREATE  15m   loadbalancer-controller  ip: 35.244.186.216

In my browser. I had routed my hostname to my correct DNS and got default backend - 404

I confirm my pod in deployment works. My endpoint does response on my request.

bash-4.3# curl http://localhost:8080/api
Hi I'm API Interface

Update on Attempt #1:
Add * after / and apply

$ kubectl apply -f ingress.yaml
ingress.extensions/tunnel-ingress configured

$ kubectl describe ingress
Name:             tunnel-ingress
Namespace:        default
Address:          35.244.186.216
Default backend:  default-http-backend:80 (10.4.1.9:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc.hbot.io
               /*   tunnel-service:80 (<none>)
Annotations:
  ingress.kubernetes.io/backends:                    {"k8s-be-31768--d2232907436d0807":"HEALTHY","k8s-be-32713--d2232907436d0807":"UNHEALTHY"}
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/url-map:                     k8s-um-default-tunnel-ingress--d2232907436d0807
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"tunnel-ingress","namespace":"default"},"spec":{"rules":[{"host":"abc.hbot.io","http":{"paths":[{"backend":{"serviceName":"tunnel-service","servicePort":80},"path":"/*"}]}}]}}

Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     42m   loadbalancer-controller  default/tunnel-ingress
  Normal  CREATE  40m   loadbalancer-controller  ip: 35.244.186.216

Not work.

Attempt #2
Add annotations

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tunnel-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: abc.hbot.io
    http:
      paths:
      - path: /
        backend:
          serviceName: tunnel-service
          servicePort: 80

Check the ingress

$ kubectl describe ingress
Name:             tunnel-ingress
Namespace:        default
Address:          35.244.186.216
Default backend:  default-http-backend:80 (10.4.1.9:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc.hbot.io
               /   tunnel-service:80 (<none>)
Annotations:
  ingress.kubernetes.io/backends:                    {"k8s-be-31768--d2232907436d0807":"HEALTHY","k8s-be-32713--d2232907436d0807":"UNHEALTHY"}
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/url-map:                     k8s-um-default-tunnel-ingress--d2232907436d0807
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"tunnel-ingress","namespace":"default"},"spec":{"rules":[{"host":"abc.hbot.io","http":{"paths":[{"backend":{"serviceName":"tunnel-service","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class:  nginx
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     54m   loadbalancer-controller  default/tunnel-ingress
  Normal  CREATE  53m   loadbalancer-controller  ip: 35.244.186.216

Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

Attemp #3
Install ingress-nginx to cluster. I follow the cert-manager doc. Because I used it once with former cluster.

Follow the same name.

$ helm install stable/nginx-ingress --name quickstart

NAME:   quickstart
LAST DEPLOYED: Wed Mar 27 16:09:15 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                                 DATA  AGE
quickstart-nginx-ingress-controller  1     0s

==> v1/Pod(related)
NAME                                                       READY  STATUS             RESTARTS  AGE
quickstart-nginx-ingress-controller-5b8d54d964-rnvw6       0/1    ContainerCreating  0         0s
quickstart-nginx-ingress-default-backend-57bdfdcd46-vtf4h  0/1    Pending            0         0s

==> v1/Service
NAME                                      TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
quickstart-nginx-ingress-controller       LoadBalancer  10.7.241.190  <pending>    80:32341/TCP,443:32762/TCP  0s
quickstart-nginx-ingress-default-backend  ClusterIP     10.7.254.207  <none>       80/TCP                      0s

==> v1/ServiceAccount
NAME                      SECRETS  AGE
quickstart-nginx-ingress  1        0s

==> v1beta1/ClusterRole
NAME                      AGE
quickstart-nginx-ingress  0s

==> v1beta1/ClusterRoleBinding
NAME                      AGE
quickstart-nginx-ingress  0s

==> v1beta1/Deployment
NAME                                      READY  UP-TO-DATE  AVAILABLE  AGE
quickstart-nginx-ingress-controller       0/1    1           0          0s
quickstart-nginx-ingress-default-backend  0/1    1           0          0s

==> v1beta1/Role
NAME                      AGE
quickstart-nginx-ingress  0s

==> v1beta1/RoleBinding
NAME                      AGE
quickstart-nginx-ingress  0s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w quickstart-nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

describe ingress

$ kubectl describe ingress
Name:             tunnel-ingress
Namespace:        default
Address:          35.244.186.216
Default backend:  default-http-backend:80 (10.4.1.9:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc.hbot.io
               /   tunnel-service:80 (<none>)
Annotations:
  ingress.kubernetes.io/backends:                    {"k8s-be-31768--d2232907436d0807":"HEALTHY","k8s-be-32713--d2232907436d0807":"UNHEALTHY"}
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-tunnel-ingress--d2232907436d0807
  ingress.kubernetes.io/url-map:                     k8s-um-default-tunnel-ingress--d2232907436d0807
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"tunnel-ingress","namespace":"default"},"spec":{"rules":[{"host":"abc.hbot.io","http":{"paths":[{"backend":{"serviceName":"tunnel-service","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class:  nginx
Events:
  Type    Reason  Age                  From                      Message
  ----    ------  ----                 ----                      -------
  Normal  CREATE  5m12s                nginx-ingress-controller  Ingress default/tunnel-ingress
  Normal  CREATE  7s (x7 over 92m)     loadbalancer-controller   ip: 35.244.186.216
  Normal  UPDATE  7s (x13 over 5m12s)  nginx-ingress-controller  Ingress default/tunnel-ingress

Test with curl on my laptop

$ curl -H 'Host: abc.hbot.io' 35.244.186.216/api

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

Attempt 4th:
Solve insufficient cpu otherwise my new pods will not run values.yaml

controller:
  resources:
    limits:
      cpu: 100m
      memory: 64Mi
    requests:
      cpu: 100m
      memory: 64Mi

defaultBackend:
  resources:
   limits:
     cpu: 10m
     memory: 20Mi
   requests:
     cpu: 10m
     memory: 20Mi

Install ingress-nginx helm install --values values.yaml stable/nginx-ingress --name quickstart

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tunnel-ingress
  annotations:
    kubernetes.io/ingress.class: nginx

spec:
  rules:
    - host: abc.hbot.io
      http:
        paths:
          - backend:
              serviceName: tunnel-service
              servicePort: 80
            path: /

describe ingress

$ kubectl describe ingress
Name:             tunnel-ingress
Namespace:        default
Address:          35.240.162.185
Default backend:  default-http-backend:80 (10.4.1.9:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc.hbot.io
               /   tunnel-service:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"tunnel-ingress","namespace":"default"},"spec":{"rules":[{"host":"abc.hbot.io","http":{"paths":[{"backend":{"serviceName":"tunnel-service","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class:  nginx
Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  14m    nginx-ingress-controller  Ingress default/tunnel-ingress
  Normal  CREATE  14m    nginx-ingress-controller  Ingress default/tunnel-ingress
  Normal  UPDATE  14m    nginx-ingress-controller  Ingress default/tunnel-ingress
  Normal  CREATE  9m27s  nginx-ingress-controller  Ingress default/tunnel-ingress

I delete and apply ingress again. IP address is changed

$ kubectl get ingress -o wide
NAME             HOSTS         ADDRESS          PORTS   AGE
tunnel-ingress   abc.hbot.io   35.240.162.185   80      15m

My situation is even worse.

$ curl -H 'Host: abc.hbot.io' http://35.240.162.185/api
curl: (7) Failed to connect to 35.240.162.185 port 80: Connection refused

Final Attempt #5:
I gave up using plain http setup. And start over with https approach. By following cert-manager and then replace kuard service by my service. Then it is done!

Question:
Where am I wrong?

-- Sarit

docker

kubernetes

1 Answer

3/27/2019

I never used GKE, but with AWS and ALB, the path needs to include the wildcard. It seems that it's the same on GKE. https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#multiple_backend_services https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#step_6_optional_serving_multiple_applications_on_a_load_balancer

Try to configure your ingress as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tunnel-ingress
spec:
  rules:
  - host: abc.hbot.io
    http:
      paths:
      - path: /*
        backend:
          serviceName: tunnel-service
          servicePort: 80

-- MiLk

Source: StackOverflow

KQ

deployment, service, ingress implemented. Got default backend - 404

Similar Questions

1 Answer

K
Q