K
Q

Kubernetes deployment not auto-terminating after successful run of command

10/11/2018

I have a Kubernetes cluster in which I've created a deployment to run a pod. Unfortunately after running it the pod does not want to self-terminate, instead it enters a continuous state of restart/CrashLoopBackOff cycle.

The command (on the entry point) runs correctly when first deployed, and I want it to run only one time.

I am programatically deploying the docker image with the entrypoint configured, using the Python K8s API. Here is my deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kio
  namespace: kmlflow
  labels:
    app: kio
    name: kio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kio
      name: kio
  template:
    metadata:
      labels:
        app: kio
        name: kio
    spec:
      containers:
      - name: kio-ingester
        image: MY_IMAGE
        command: ["/opt/bin/kio"]
        args: ["some", "args"]
        imagePullPolicy: Always
      restart: Never
  backofflimit: 0

Thanks for any help

Output from kubectl pod is:

Name:               ingest-160-779874b676-8pgv5
Namespace:          kmlflow
Priority:           0
PriorityClassName:  <none>
Node:               02-w540-02.glebe.kinetica.com/172.30.255.205
Start Time:         Thu, 11 Oct 2018 13:31:20 -0400
Labels:             app=kio
                    name=kio
                    pod-template-hash=3354306232
Annotations:        <none>
Status:             Running
IP:                 10.244.0.228
Controlled By:      ReplicaSet/ingest-160-779874b676
Containers:
  kio-ingester:
    Container ID:  docker://b67a682d04e69c2dc5c1be7e02bf2e4cf7a12a7557dfbe642dfb531ca4b03f07
    Image:         kinetica/kinetica-intel
    Image ID:      docker-pullable://docker.io/kinetica/kinetica-intel@sha256:eefbb6595eb71822300ef97d5cbcdac7ec58f2041f8190d3a2ba9cffd6a0d87c
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/gpudb/bin/kio
    Args:
      --source
      kinetica://172.30.50.161:9191::dataset_iris
      --destination
      kinetica://172.30.50.161:9191::iris5000
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 11 Oct 2018 13:33:27 -0400
      Finished:     Thu, 11 Oct 2018 13:33:32 -0400
    Ready:          False
    Restart Count:  4
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-69wkn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-69wkn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-69wkn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                                    Message
  ----     ------     ----                 ----                                    -------
  Normal   Scheduled  2m39s                default-scheduler                       Successfully assigned kmlflow/ingest-160-779874b676-8pgv5 to 02-w540-02.glebe.kinetica.com
  Normal   Created    89s (x4 over 2m28s)  kubelet, 02-w540-02.glebe.kinetica.com  Created container
  Normal   Started    89s (x4 over 2m28s)  kubelet, 02-w540-02.glebe.kinetica.com  Started container
  Warning  BackOff    44s (x7 over 2m15s)  kubelet, 02-w540-02.glebe.kinetica.com  Back-off restarting failed container
  Normal   Pulling    33s (x5 over 2m28s)  kubelet, 02-w540-02.glebe.kinetica.com  pulling image "kinetica/kinetica-intel"
  Normal   Pulled     33s (x5 over 2m28s)  kubelet, 02-w540-02.glebe.kinetica.com  Successfully pulled image "kinetica/kinetica-intel"

There is no output from Kubectl logs <crashing-pod> because a successful run of the command KIO with the injected parameters does not print anything to standard output.

-- jpjenk
kubernetes

Similar Questions

pending spark pod on google kubernetes cluster: insufficient cpu
Get current resource usage of a pod in Kubernetes with Go client
Discovery / Registration only works when K8s pod already registered.
How to use existing Docker image in Kubernetes?

1 Answer

10/12/2018

If you'd like to run your task one time and finish after a successful completion you should consider using Kubernetes Jobs or CronJobs

Something like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: kio
  namespace: kmlflow
  labels:
    app: kio
    name: kio
spec:
  template:
    metadata:
      labels:
        app: kio
        name: kio
    spec:
      containers:
      - name: kio-ingester
        image: MY_IMAGE
        command: ["/opt/bin/kio"]
        args: ["some", "args"]
        imagePullPolicy: Always
      restart: Never
  backoffLimit: 4

To delete the jobs automatically if you have Kubernetes 1.12 or later you can use ttlSecondsAfterFinished. Unfortunately, you if you are using Kuberbetes 1.11 or earlier you will have to delete them manually or you can set up a CronJob to do it.

-- Rico
Source: StackOverflow