K
Q

gke: ingress is not forwarding to the NodePort service: <code>/healthz</code> was not found on this server

11/18/2019

I have too many LoadBalancer services consuming too many external IPs and I'd like to switch to using an Ingress controller.

I did the tutorial and everything worked fine with the google provided pods.

However, with my pod I am able to hit the NodePort service ...

    >curl http://35.223.89.81:32607/healthz
OK    >

... but calls to the Ingress Controller are consistently failing ...

    >curl http://35.241.21.71:80/healthz
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/healthz</code> was not found on this server.  <ins>That’s all we know.</ins>

This is the version of k8s I am using:

    >gcloud container clusters list
NAME              LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
monza-predictors  us-central1-a  1.13.11-gke.14  35.193.247.210  n1-standard-1  1.13.11-gke.9 *  2          RUNNING

YAML for the ingress

    >cat fanout-ingress-v2.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
spec:
  rules:
  - http:
      paths:
      - path: /healthz
        backend:
          serviceName: predictor-classification-seatbelt-driver-service-node-port
          servicePort: 4444
      - path: /seatbelt-driver
        backend:
          serviceName: predictor-classification-seatbelt-driver-service-node-port
          servicePort: 4444

describe of the ingress

    >kubectl describe ing fanout-ingress 
Name:             fanout-ingress
Namespace:        default
Address:          35.241.21.71
Default backend:  default-http-backend:80 (10.40.2.10:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     
        /healthz           predictor-classification-seatbelt-driver-service-node-port:4444 (<none>)
        /seatbelt-driver   predictor-classification-seatbelt-driver-service-node-port:4444 (<none>)
Annotations:
  ingress.kubernetes.io/url-map:                     k8s-um-default-fanout-ingress--62f4c45447b62142
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"fanout-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"predictor-classification-seatbelt-driver-service-node-port","servicePort":4444},"path":"/healthz"},{"backend":{"serviceName":"predictor-classification-seatbelt-driver-service-node-port","servicePort":4444},"path":"/seatbelt-driver"}]}}]}}

  ingress.kubernetes.io/backends:         {"k8s-be-31413--62f4c45447b62142":"HEALTHY","k8s-be-32607--62f4c45447b62142":"UNHEALTHY"}
  ingress.kubernetes.io/forwarding-rule:  k8s-fw-default-fanout-ingress--62f4c45447b62142
  ingress.kubernetes.io/target-proxy:     k8s-tp-default-fanout-ingress--62f4c45447b62142
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     21m   loadbalancer-controller  default/fanout-ingress
  Normal  CREATE  19m   loadbalancer-controller  ip: 35.241.21.71

I noticed that 1 of the 2 backends are UNHEALTHY.

YAML for the NodePort service:

    >cat service-node-port-classification-predictor.yaml
apiVersion: v1
kind: Service
metadata:
  name: predictor-classification-seatbelt-driver-service-node-port
  namespace: default
spec:
  ports:
  - port: 4444
    protocol: TCP
    targetPort: 4444
  selector:
    app: predictor-classification-seatbelt-driver
  type: NodePort

describe of the NodePort service

    >kubectl describe svc predictor-classification-seatbelt-driver-service-node-port
Name:                     predictor-classification-seatbelt-driver-service-node-port
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"predictor-classification-seatbelt-driver-service-node-port","name...
Selector:                 app=predictor-classification-seatbelt-driver
Type:                     NodePort
IP:                       10.43.243.69
Port:                     <unset>  4444/TCP
TargetPort:               4444/TCP
NodePort:                 <unset>  32607/TCP
Endpoints:                10.40.2.16:4444
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

YAML for the deployment

    >cat deployment-classification-predictor-v2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: predictor-classification-seatbelt-driver
  labels:
    app: predictor-classification-seatbelt-driver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: predictor-classification-seatbelt-driver
  template:
    metadata:
      labels:
        app: predictor-classification-seatbelt-driver
    spec:
      containers:
      - name: predictor-classification-seatbelt-driver
        image: gcr.io/annotator-1286/classification-predictor
        command: ["/app/server.sh"]
        args: ["4444", "https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/mobile.pb", "https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/labels.csv"]
        ports:
        - containerPort: 4444
        livenessProbe:
          httpGet:
            path: /healthz
            port: 4444
          initialDelaySeconds: 120

describe of the deployment

    >kubectl describe deploy predictor-classification-seatbelt-driver
Name:                   predictor-classification-seatbelt-driver
Namespace:              default
CreationTimestamp:      Mon, 18 Nov 2019 12:17:13 -0800
Labels:                 app=predictor-classification-seatbelt-driver
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"predictor-classification-seatbelt-driver"},"name...
Selector:               app=predictor-classification-seatbelt-driver
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=predictor-classification-seatbelt-driver
  Containers:
   predictor-classification-seatbelt-driver:
    Image:      gcr.io/annotator-1286/classification-predictor
    Port:       4444/TCP
    Host Port:  0/TCP
    Command:
      /app/server.sh
    Args:
      4444
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/mobile.pb
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/labels.csv
    Liveness:     http-get http://:4444/healthz delay=120s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   predictor-classification-seatbelt-driver-85bc679444 (1/1 replicas created)
Events:          <none>

describe of the pod

    >kubectl describe po predictor-classification-seatbelt-driver-85bc679444-lcb7v
Name:               predictor-classification-seatbelt-driver-85bc679444-lcb7v
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-monza-predictors-default-pool-268f57e3-1bs6/10.128.0.65
Start Time:         Mon, 18 Nov 2019 12:17:13 -0800
Labels:             app=predictor-classification-seatbelt-driver
                    pod-template-hash=85bc679444
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container predictor-classification-seatbelt-driver
Status:             Running
IP:                 10.40.2.16
Controlled By:      ReplicaSet/predictor-classification-seatbelt-driver-85bc679444
Containers:
  predictor-classification-seatbelt-driver:
    Container ID:  docker://90ce1466b852760db92bc66698295a2ae2963f19d26111e5be03d588dc83a712
    Image:         gcr.io/annotator-1286/classification-predictor
    Image ID:      docker-pullable://gcr.io/annotator-1286/classification-predictor@sha256:63690593d710182110e51fbd620d6944241c36dd79bce7b08b2823677ec7b929
    Port:          4444/TCP
    Host Port:     0/TCP
    Command:
      /app/server.sh
    Args:
      4444
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/mobile.pb
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/labels.csv
    State:          Running
      Started:      Mon, 18 Nov 2019 12:17:15 -0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Liveness:     http-get http://:4444/healthz delay=120s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8q95m (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-8q95m:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8q95m
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

UPDATE: Using a Single Service Ingress did not fix the problem

    >cat fanout-ingress-v3.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
spec:
  backend:
    serviceName: predictor-classification-seatbelt-driver-service-node-port
    servicePort: 4444
    >kubectl apply -f fanout-ingress-v3.yaml 
ingress.extensions/fanout-ingress created


    >kubectl describe ing fanout-ingress 
Name:             fanout-ingress
Namespace:        default
Address:          35.244.250.224
Default backend:  predictor-classification-seatbelt-driver-service-node-port:4444 (10.40.2.16:4444)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     predictor-classification-seatbelt-driver-service-node-port:4444 (10.40.2.16:4444)
Annotations:
  ingress.kubernetes.io/url-map:                     k8s-um-default-fanout-ingress--62f4c45447b62142
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"fanout-ingress","namespace":"default"},"spec":{"backend":{"serviceName":"predictor-classification-seatbelt-driver-service-node-port","servicePort":4444}}}

  ingress.kubernetes.io/backends:         {"k8s-be-32607--62f4c45447b62142":"Unknown"}
  ingress.kubernetes.io/forwarding-rule:  k8s-fw-default-fanout-ingress--62f4c45447b62142
  ingress.kubernetes.io/target-proxy:     k8s-tp-default-fanout-ingress--62f4c45447b62142
Events:
  Type    Reason  Age    From                     Message
  ----    ------  ----   ----                     -------
  Normal  ADD     3m31s  loadbalancer-controller  default/fanout-ingress
  Normal  CREATE  2m56s  loadbalancer-controller  ip: 35.244.250.224



   >curl 35.244.250.224/healthz
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/healthz</code> was not found on this server.  <ins>That’s all we know.</ins>
-- Alex Ryan
google-kubernetes-engine
kubernetes
kubernetes-ingress

Similar Questions

Cloud Endpoints, Kubernetes and SSL
terraform keeps overwriting token for kubernetes provider
Artemis Replication In Kubernetes Issues
Kubeadm join failing and hangs at Setting node annotation to enable volume controller attach/detach
Kubernetes control pod name

1 Answer

11/19/2019

Add a readinessProbe to your Deployment object.

    readinessProbe:
      httpGet:
        path: /healthz
        port: 4444
      initialDelaySeconds: 120

An IngressController may wait with routing traffic to the service until the pods in the service become ready to handle requests from the Ingress proxy.

-- Jonas
Source: StackOverflow