livenessProbe seems not to be executed

12/16/2021

A container defined inside a deployment has a livenessProbe set up: by definition, it calls a remote endpoint and checks, whether response contains useful information or an empty response (which should trigger the pod's restart).

The whole definition is as follows (I removed the further checks for better clarity of the markup):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fc-backend-deployment
  labels:
    name: fc-backend-deployment
    app: fc-test
spec:
  replicas: 1
  selector:
    matchLabels:
      name: fc-backend-pod
      app: fc-test
  template:
    metadata:
      name: fc-backend-pod
      labels:
        name: fc-backend-pod
        app: fc-test
    spec:
      containers:
      - name: fc-backend
        image: localhost:5000/backend:1.3
        ports:
        - containerPort: 4044
        env:
        - name: NODE_ENV
          value: "dev"
        - name: REDIS_HOST
          value: "redis"
        livenessProbe:
          exec:
            command:
            - curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats | head -c 30 > /app/out.log
          initialDelaySeconds: 20
          failureThreshold: 12
          periodSeconds: 10

I also tried putting the command into an array:

command: ["sh", "-c", "curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats", "|", "head", "-c", "30", ">", "/app/out.log"]

and splitting into separate lines:

- /bin/bash
- -c
- curl
- -X
- GET
- $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats
- |
- head
- -c
- "30"
- >
- /app/out.log

and even like this:

command:
  - |
      curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats | head -c 30 > /app/out.log

All attempts were made with and without (/bin/ba)sh -c - with the same result.

But, as you're reading this, you already know that none of these worked.

I know it by exec'ing into running container and trying to find the /app/out.log file - it wasn't present any time I watched the directory contents. It looks like the probe gets never executed.

The command run inside running container works just fine: data gets fetched and written to the specified file.

What might be causing the probe not to get executed?

-- AbreQueVoy
kubernetes
yaml

1 Answer

12/17/2021

When using the exec type of probes, Kubernetes will not run a shell to process the command, it will just run the command directly. This means that you can only use a single command and that the | character is considered just another parameter of your curl.

To solve the problem, you need to use sh -c to exec shell code, something like the following:

    livenessProbe:
      exec:
        command:
          - sh
          - -c
          - >-
              curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats |
              head -c 30 > /app/out.log
-- AndD
Source: StackOverflow