Keep getting error status on Kubernetes Cron Job with connection refused?

4/22/2021

I am trying to write a cron job which hits a rest endpoint of the application it is pulling image of. Below is the sample code:

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: {{ .Chart.Name }}-cronjob
  labels:
    app: {{ .Release.Name }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version }}
    release: {{ .Release.Name }}
spec:
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  startingDeadlineSeconds: 1800
  jobTemplate:
    spec:
      template:
        metadata:
          name: {{ .Chart.Name }}-cronjob
          labels:
            app: {{ .Chart.Name }}
        spec:
          restartPolicy: OnFailure
          containers:
            - name: demo
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
              command: ["/bin/sh", "-c", "curl http://localhost:8080/hello"]
              readinessProbe:
                httpGet:
                  path: "/healthcheck"
                  port: 8081
                initialDelaySeconds: 300
                periodSeconds: 60
                timeoutSeconds: 30
                failureThreshold: 3
              livenessProbe:
                httpGet:
                  path: "/healthcheck"
                  port: 8081
                initialDelaySeconds: 300
                periodSeconds: 60
                timeoutSeconds: 30
                failureThreshold: 3
              resources:
                requests:
                  cpu: 200m
                  memory: 2Gi
                limits:
                  cpu: 1
                  memory: 6Gi
  schedule: "*/5 * * * *"

But i keep running into *curl: (7) Failed to connect to localhost port 8080: Connection refused*. I can see from the events that it creates the container and immediately throws: Back-off restarting failed container. I already have pods running of demo app and it works fine, it is just when i am trying to point to this existing app and hit a rest endpoint i start running into connection refused errors.

Exact output when seeing the logs:

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8080: Connection refused

Event Logs:

Container image "wayfair/demo:728ac13-as_test_cron_job" already present on machine
9m49s       Normal    Created                pod/demo-cronjob-1619108100-ndrnx   Created container demo
6m17s       Warning   BackOff                pod/demo-cronjob-1619108100-ndrnx   Back-off restarting failed container
5m38s       Normal    SuccessfulDelete       job/demo-cronjob-1619108100         Deleted pod: demo-cronjob-1619108100-ndrnx
5m38s       Warning   BackoffLimitExceeded   job/demo-cronjob-1619108100         Job has reached the specified backoff limit

Being new to K8, Any pointers are helpful!

-- anzie001
kubernetes
kubernetes-cronjob

1 Answer

4/22/2021

You are trying to connect to localhost:8080 with your curl which doesn't make sense from what I understand of your CronJob definition.

From the docs (at https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#define-a-command-and-arguments-when-you-create-a-pod )

The command and arguments that you define in the configuration file override the default command and arguments provided by the container image. If you define args, but do not define a command, the default command is used with your new arguments.

Note: The command field corresponds to entrypoint in some container runtimes. Refer to the Notes below.

If you define a command for the image, even if the image would start a rest application on port 8080 on localhost with its default entrypoint (or command, depends on the container type you are using), the command overrides the entrypoint and no application is start.

If you have the necessity of both starting the application and then performing other operations, like curls and so on, I suggest to use a .sh script or something like that, depending on what is the Job objective.

-- AndD
Source: StackOverflow