Kubernetes CronJob Issue

8/10/2021

I'm having strange issue in cronjob. Few empty pods are created at random time.

PFA details related to cronjob: 1. kubectl --kubeconfig ~/.kube/deploy.kubeconfig -n *** get cronjob

NAME                  SCHEDULE     SUSPEND   ACTIVE   LAST SCHEDULE   AGE
srp-sitemap-cronjob   5 12 * * *   False     0        7h53m           4d7h
  1. kubectl --kubeconfig ~/.kube/cluster.kubeconfig get pods -n *** -o wide | grep -i srp-site
srp-sitemap-cronjob-1628577300-tfl9g                      0/1     Completed   0          7h53m   10.241.28.194    npl9app17.ieil.net   <none>           <none>
  1. kubectl --kubeconfig ~/.kube/deploy.kubeconfig -n ** get jobs srp-sitemap-cronjob-1628577300 -o yaml
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: "2021-08-10T06:35:00Z"
  labels:
    controller-uid: 3d80b485-6a9a-46a2-8835-82d77ac147b2
    job-name: srp-sitemap-cronjob-1628577300
  name: srp-sitemap-cronjob-1628577300
  namespace: ******
  ownerReferences:
  - apiVersion: batch/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CronJob
    name: srp-sitemap-cronjob
    uid: d1a0fe6b-f573-4996-a54c-897f9bf333a2
  resourceVersion: "131531653"
  selfLink: /apis/batch/v1/namespaces/********/jobs/srp-sitemap-cronjob-1628577300
  uid: 3d80b485-6a9a-46a2-8835-82d77ac147b2
spec:
  backoffLimit: 0
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: 3d80b485-6a9a-46a2-8835-82d77ac147b2
  template:
    metadata:
      creationTimestamp: null
      labels:
        controller-uid: 3d80b485-6a9a-46a2-8835-82d77ac147b2
        job-name: srp-sitemap-cronjob-1628577300
    spec:
      containers:
      - env:
        - name: JVM_OPTS
          value: -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=200
            -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=8 -XX:GCLogFileSize=60M
            -DpasswordFile=/etc/password
        - name: SERVICE_NAME
          value: srp-sitemap-cronjob
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_SERVICE_ACCOUNT
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.serviceAccountName
        envFrom:
        - configMapRef:
            name: active-profile
        image: *******/srp-sitemap-cronjob-prod:xxxx
        imagePullPolicy: IfNotPresent
        name: srp-sitemaps-cronjob
        resources:
          limits:
            cpu: "4"
            memory: 6096Mi
          requests:
            cpu: 500m
            memory: 2Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data/ws-logs/sitemaps-cron/
          name: ws-volume
          subPathExpr: $(POD_NAME)
        - mountPath: /etc/password
          name: passwords
          subPath: password
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: xxxxxxxxx
      nodeSelector:
        99-cron-job: "true"
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /data/ws-logs/sitemaps-cron/
          type: Directory
        name: ws-volume
      - configMap:
          defaultMode: 420
          name: srp-sitemap-password-prod
        name: passwords
status:
  completionTime: "2021-08-10T09:35:47Z"
  conditions:
  - lastProbeTime: "2021-08-10T09:35:47Z"
    lastTransitionTime: "2021-08-10T09:35:47Z"
    status: "True"
    type: Complete
  startTime: "2021-08-10T06:35:00Z"
  succeeded: 1

#####################################################################

drwxr-xr-x 4 root root  41 Aug  8 23:30 srp-sitemap-cronjob-1628164800-p7k7q 
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627476600-kxrjg
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627476000-gtt29
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627475400-47rqz
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627474800-ggt65
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627486200-7gxrf
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627485000-mgnd2
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627483800-nqcs7
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627481400-ltpqf
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627479900-s2frh
drwxr-xr-x 2 root root   6 Aug  9 05:30 srp-sitemap-cronjob-1627479000-4ltxl
drwxr-xr-x 4 root root 114 Aug  9 13:05 srp-sitemap-cronjob-1628494500-69zph  ( Desired pod at scheduled time )
drwxr-xr-x 4 root root  41 Aug  9 17:30 srp-sitemap-cronjob-1628232600-tkskg
drwxr-xr-x 4 root root  41 Aug  9 17:30 srp-sitemap-cronjob-1628235300-vzgjm
drwxr-xr-x 2 root root   6 Aug  9 17:30 srp-sitemap-cronjob-1627543200-brhts
drwxr-xr-x 2 root root   6 Aug  9 17:30 srp-sitemap-cronjob-1627542600-kfwzb
drwxr-xr-x 2 root root   6 Aug  9 17:30 srp-sitemap-cronjob-1627541400-7psc9
drwxr-xr-x 2 root root   6 Aug  9 17:30 srp-sitemap-cronjob-1627540800-5zmmx
drwxr-xr-x 2 root root   6 Aug  9 17:30 srp-sitemap-cronjob-1627540200-mx7zz
drwxr-xr-x 2 root root   6 Aug  9 17:30 srp-sitemap-cronjob-1627539600-5xs84
drwxr-xr-x 2 root root   6 Aug  9 23:30 srp-sitemap-cronjob-1627546200-l4xth
drwxr-xr-x 4 root root 114 Aug 10 12:05 srp-sitemap-cronjob-1628577300-tfl9g  ( Desired pod at scheduled time )
drwxr-xr-x 4 root root  41 Aug 10 17:30 srp-sitemap-cronjob-1628321700-ddbqz
drwxr-xr-x 3 root root  21 Aug 10 17:30 srp-sitemap-cronjob-1627632600-nqtfj

Any help would be highly appreciated. Thanks in advance.

-- Aman Gupta
k8s-cronjobber
kubernetes
kubernetes-cronjob
kubernetes-jobs

0 Answers