K
Q

Question

K8s init container restart itself

5/31/2020

I have pod that includes one init container and one app container, between them there is a volume with shared folder.

My issue is that once a day or even more, the init container run itself and therefore it delete the node modules from the volume, then the main app crash because of missing modules. The app container is not making any restarts, only the init container.

is anyone familiar with this issue in k8s? why those restarts happens only in the init container?

Thanks :)

edit: the deployment yaml file -

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "25"
  creationTimestamp: "2020-05-19T06:48:18Z"
  generation: 25
  labels:
    apps: ******
    commit: ******
  name: *******
  namespace: fleet
  resourceVersion: "24059934"
  selfLink: *******
  uid: *******
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: *******
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: *******
        commit: *******
        revision: *******
    spec:
      containers:
        image: XXXXXXXXXXXX
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: http
            scheme: HTTP
          initialDelaySeconds: 120
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 30
        name: *******
        ports:
        - containerPort: 1880
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /ping
            port: http
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 30
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/breeze
          name: workdir
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: flowregistrykey
      initContainers:
        image: XXXXXXXXXXXX
        imagePullPolicy: IfNotPresent
        name: get-flow-json
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /work-dir
          name: workdir
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: workdir
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-06-01T12:30:10Z"
    lastUpdateTime: "2020-06-01T12:30:10Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-05-19T06:48:18Z"
    lastUpdateTime: "2020-06-01T12:45:05Z"
    message: ReplicaSet "collection-associator.sandbox.services.collection-8784dcb9d"
      has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 25
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

and this image explain the problem- the pod was created 7 days ago, but the files inside were created today, and there is no node_modules folder - because only the init container run again and not the app container so there was no mpm install

-- Sariel

kubernetes

1 Answer

6/1/2020

the init container run itself and therefore it delete the node modules from the volume

initContainer is only run when a pod restart. Don't treat it as service or application. It should be a script, a job only for setup before your application.

then the main app crash because of missing modules.

node_modules is not dynamic loading. It's loaded when you npm start

You might want to try livenessProbe.

Many applications running for long periods of time eventually transition to broken states, and cannot recover except by being restarted. Kubernetes provides liveness probes to detect and remedy such situations.

initContainers:
  - name: set-delete-time
    command: ["/bin/sh","-c"]
    args: 
      - |
        # Let's set 3600
        echo 3600 > deleteTime

...
containers:
  - name: node-app
    livenessProbe:
      exec:
        command:
        # If deleteTime > 0 exit 0
        # Else /tmp/deleteTime - $periodSeconds > /tmp/deleteTime
        - cat
        - /tmp/deleteTime
        - ...
      initialDelaySeconds: 5
      periodSeconds: 5

-- RammusXu

Source: StackOverflow

KQ

K8s init container restart itself

Similar Questions

1 Answer

You might want to try livenessProbe.

K
Q