K
Q

Question

Kubernetes TCP Health Check

12/17/2019

I am building .NET core Console microservice and it has been suggested by Architect to use TCP Health Check instead of HTTP Health Check. Hence in order to implement TCP health Check, please find the below configuration that I have added in the OCP file(deploymentconfig section). Jenkins build was successful and also the deployment config roll out was successful.

Query:

How to ensure that probes are working properly.Is there a way to verify readiness and liveliness probes is being done in regular interval with the TCP Health Check ?

Is there any syntax by which I can explicitly check the Container Health Status using TCP Health Check.

    readinessProbe:
      tcpSocket:
          port: 8080
      initialDelaySeconds: "${{READINESS_DELAY}}"
      periodSeconds: "${{READINESS_TIMEOUT}}"
      timeoutSeconds: "${{READINESS_TIMEOUT}}"
    livenessProbe:
      tcpSocket:
          port: 8080
      initialDelaySeconds: "${{LIVENESS_DELAY}}"
      periodSeconds: "${{LIVENESS_TIMEOUT}}"
      timeoutSeconds: "${{LIVENESS_TIMEOUT}}"

-- Naveen Bhogi

kubernetes-health-check

tcp

1 Answer

12/31/2019

Output of the probe is saved to kubelet component on each node.

As default in Kubernetes you can check Probes if you will describe pod. For example for pod where everything workings correctly you will not find any information about this. It shows only relevant events like Unhealthy or Killing, etc.

To check if this container is using any LivenessProbe or ReadinessProbe you need to describe pod and find: Containers.<containerName>.Liveness and Containers.<containerName>.Readiness.

Below example based on docs, but with additional changes which will guarantee it will fail.

Added:

livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy

Output:

$ kubectl describe pod goproxy-fail
...
Events:
  Type     Reason     Age                  From                 Message
  ----     ------     ----                 ----                 -------
  Normal   Scheduled  3m38s                default-scheduler    Successfully assigned default/goproxy-fail to kubeadm-16
  Normal   Pulled     26s (x4 over 3m37s)  kubelet, kubeadm-16  Container image "k8s.gcr.io/goproxy:0.1" already present on machine
  Normal   Created    26s (x4 over 3m37s)  kubelet, kubeadm-16  Created container goproxy
  Normal   Killing    26s (x3 over 2m26s)  kubelet, kubeadm-16  Container goproxy failed liveness probe, will be restarted
  Normal   Started    25s (x4 over 3m36s)  kubelet, kubeadm-16  Started container goproxy
  Warning  Unhealthy  6s (x10 over 3m6s)   kubelet, kubeadm-16  Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"cat\": executable file not found in $PATH": unknown

It means over 3m6s there was 10 checks which all failed and status is Unhealthy.

Another default option is to use events using $ kubectl get events. Output will be smilar but it will gather all events from cluster. You can specify namespace etc.

How check succeed probes

Output of Successful probes is not recorded anywhere in default settings. You will need change Kubelet logging level, --verbosity to at least debugging mode (4).

To do it, you have to:

ssh to master node
edit file /var/lib/kubelet/kubeadm-flags.env (on ubuntu you need sudo rights to do it $ sudo su). Default output looks like KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf" you have to add --v=4 at the end.
Depends on your future need you can take higher log level. More information can be found here.
Desired value in kubeadm-flags.env would look like below:
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf --v=4"
After that you need to restart kubelet to apply this new logging level. You can do it using sudo systemctl restart kubelet.

Next step is to check kubelet logs using journactl.

$ journalctl -u kubelet you can also grep it as you will get huge amount of logs as --v is set to 4. I have created another pod with similar config but with pod name tetest and container name goproxy, easier to find.

$ journalctl -u kubelet | grep tetest
...
Dec 31 10:29:46 kubeadm-16 kubelet[17767]: I1231 10:29:46.303112   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:29:55 kubeadm-16 kubelet[17767]: I1231 10:29:55.289330   17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:29:56 kubeadm-16 kubelet[17767]: I1231 10:29:56.303326   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:06 kubeadm-16 kubelet[17767]: I1231 10:30:06.302931   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:15 kubeadm-16 kubelet[17767]: I1231 10:30:15.289462   17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:16 kubeadm-16 kubelet[17767]: I1231 10:30:16.303267   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:26 kubeadm-16 kubelet[17767]: I1231 10:30:26.303248   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:35 kubeadm-16 kubelet[17767]: I1231 10:30:35.289164   17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:36 kubeadm-16 kubelet[17767]: I1231 10:30:36.303071   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:46 kubeadm-16 kubelet[17767]: I1231 10:30:46.303751   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:49 kubeadm-16 kubelet[17767]: I1231 10:30:49.237565   17767 kubelet.go:1965] SyncLoop (SYNC): 1 pods; tetest_default(a518f558-9b08-4ce8-86a2-81875f205826)
...

Tested on Kubernetes 1.16.3, OS Ubuntu 18.04.

Hope it will help

-- PjoterS

Source: StackOverflow

KQ

Kubernetes TCP Health Check

Similar Questions

1 Answer

K
Q