K
Q

Kubelet still sending requests even after springboot readiness probes is in REFUSING_TRAFFIC stage

5/24/2021

I was trying readiness probe in kubernetes with a springboot app. After the app starts, lets say after 60 seconds I fire ReadinessState.REFUSING_TRAFFIC app event.

I use port-forward for kubernetes service(Cluster-Ip) and checked /actuator/health/readiness and see "status":"OUT_OF_SERVICE" after 60 seconds.

I, then fire some GET/POST requests to service.

Expected: Service unavailable message

Actual: GET/POST endpoints return data as usual

Is the behavior expected. Please comment.

Sample liveness/readiness probe yaml

        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Authorization
              value: Basic xxxxxxxxxxxxxx
            path: /actuator/health/liveness
            port: http
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 10
        name: sample-app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Authorization
              value: Basic xxxxxxxxxxxxxx
            path: /actuator/health/readiness
            port: http
            scheme: HTTP
          initialDelaySeconds: 140
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 10
-- v47
kubernetes
spring-boot

Similar Questions

How can I access external MongoDB server on ec2 instance from an app running inside Kubernetes cluster created with kops?
LDAP based user authentication for Kubernetes
Calling docker stack deploy on a docker host from within a Jenkins container
Persistent Volume and OpenShift 3.11
How to access Jenkins Dashboard on Kubernetes on private cloud?

1 Answer

6/7/2021

This is expected behavior as:

  • $ kubectl port-forward service/SERVICE_NAME LOCAL_PORT:TARGET_PORT

is not considering the state of the Pod when doing a port forwarding (directly connects to a Pod).

Explanation

There is already a great answer which pointed me on further investigation here:

Let's assume that you have a Deployment with a readinessProbe (in this example probe will never succeed):

  • $ kubectl get pods, svc (redacted not needed part)
NAME                                     READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-64ff4d8749-7khht    0/1     Running   0          97m
pod/nginx-deployment-64ff4d8749-bklnf    0/1     Running   0          97m
pod/nginx-deployment-64ff4d8749-gsmml    0/1     Running   0          97m

NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/nginx                ClusterIP   10.32.31.105   <none>        80/TCP      97m
  • $ kubectl describe endpoints nginx
Name:         nginx
Namespace:    default
Labels:       <none>
Annotations:  <none>
Subsets:
  Addresses:          <none>
  NotReadyAddresses:  10.36.0.62,10.36.0.63,10.36.0.64 # <-- IMPORTANT
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  80    TCP

Events:  <none>

As you can see all of the Pods are not in Ready state and the Service will not send the traffic to it. This can be seen in a following scenario ( create a test Pod that will try to curl the Service):

  • $ kubectl run -it --rm nginx-check --image=nginx -- /bin/bash
  • $ curl nginx.default.svc.cluster.local
curl: (7) Failed to connect to nginx.default.svc.cluster.local port 80: Connection refused

Using kubectl port-forward:

  • $ kubectl port-forward service/nginx 8080:80
  • $ curl localhost:8080
<-- REDACTED --> 
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<-- REDACTED -->

More light can be shed on why it happened by getting more verbose output from the command:

  • kubectl port-forward service/nginx 8080:80 -v=6 (the number can be higher)
I0606 21:29:24.986382    7556 loader.go:375] Config loaded from file:  /SOME_PATH/.kube/config
I0606 21:29:25.041784    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/services/nginx 200 OK in 51 milliseconds
I0606 21:29:25.061334    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/pods?labelSelector=app%3Dnginx 200 OK in 18 milliseconds
I0606 21:29:25.098363    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/pods/nginx-deployment-64ff4d8749-7khht 200 OK in 18 milliseconds
I0606 21:29:25.164402    7556 round_trippers.go:444] POST https://API_IP/api/v1/namespaces/default/pods/nginx-deployment-64ff4d8749-7khht/portforward 101 Switching Protocols in 62 milliseconds
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

What happened:

  • kubectl requested the information about the Service: nginx
  • kubectl used the selector associated with the Service and looked for Pods with the same selector (nginx)
  • kubectl chose a single Pod and port-forwarded to it.

The Nginx welcome page showed as the port-forward connected directly to a Pod and not to a Service.

Additional reference:

# Listen on port 8443 locally, forwarding to the targetPort of the service's port named "https" in a pod selected by the service

kubectl port-forward service/myservice 8443:https

-- Dawid Kruk
Source: StackOverflow