How to wait for the other container completed execution in k8s?

7/25/2019

Assume I have an app A container and another container called resources-preparation which will try to create DB tables and etc in order to bootstrap app A.

App A container and resources-preparation container are living in different pods. How can I bring up App A container after resources-preparation container completes.

PS: resources-preparation container is not a service at all. So I may not be able to use the waitfor image to detect the resources-preparation container completes.

-- injoy
docker
google-kubernetes-engine
kubernetes

4 Answers

7/26/2019

did you consider to look at the initContainers?

In your case, A container and resources-preparation container are living in different "PODS" so never tried that but take a look at the https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/

-- Salman Memon
Source: StackOverflow

7/26/2019

you need to use initContainer. refer the below code. in the belwo example The container first waits for myservice, and second waits for mydb. Once both init containers complete, the Pod runs myapp-container

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
  - name: init-mydb
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']
-- P Ekambaram
Source: StackOverflow

7/26/2019

App A container and resources-preparation container are living in different pods.

There is dependency among pods with conditions in your use-case.

I would try to use wait command till a specific condition is true

kubectl wait --for=condition=complete job/myJob

After than you can rollout your deployment

for further detail kubectl-wait

-- Suresh Vishnoi
Source: StackOverflow

7/30/2019

It seems there is a kind of architectural inconsistency: the existing application architecture does not fit Kubernetes paradigm well:

  • The pod A is tied to the pod resources-preparation, so has to wait for its successful completion, whereas k8s assumes independent or loosely coupled microservices.
  • Being tightly dependent, containers A and resources-preparation are put into different pods whereas assisting applications should be placed into the same container with the primary one. See the Discussion Communicate Between Containers in the Same Pod.
  • The pod A is dependent on an external database whereas in k8s microservices should work with their own database or replica to keep independency.
  • The pods A and resources-preparation should communicate via k8s API. That means the pod A should fetch information about the resources-preparation completion from the kube-apiserver.

The listed principles cause extra overhead but this is the price you pay for the redundancy Kubernetes relies on.

Possible approaches to the problem:

  1. Redesign or modify the application and backend database accordingly with the k8s principles, decompose them into a set of loosely coupled microservices. As a supposal:

    • a) let the app A start with its DB replica independently;
    • b) in parallel let the resources-preparation to start and create tables in its own replica;
    • c) then add the new tables to the existing Replication or create a new Replication. In this approach the pod A does not have to wait for the pod resources-preparation. The DB replication will be waiting instead. That way the dependency will be moved off the k8s level to the upper layer.

    Unfortunately, adaptation of existing applications to k8s could be challenging and often requires re-developing the application from scratch. It is the time- and resource-consuming task.

    A good whitepaper is available here: Principles of container-based application design.

  2. Since the resources-preparation is an assisting container for the A, put both containers into the same pod. That way the sample code from the Init Containers concept will do exactly the container A needs. What's important for the container A awaiting for the resources-preparation completion is that:

    • Init containers always run to completion.
    • Each init container must complete successfully before the next one starts.
  3. If you can not join both containers into the same pod for some reason, as a workaround the application components could be put into a "wrapper" that helps them to pretend behaving as loosely coupled microservices. This wrapper should be implemented below the pod level to be transparent for Kubernetes: around the container or application. In a simple case you might launch the application A from within a shell script with the until loop. The script should fetch the status of the resources-preparation pod running in a StatefulSet via the kube-apiserver to make decision if the application A may start or not.

    A REST API request could be used for that (see the answer Kubernetes API server, serving pod logs).

    A way to authenticate on the kube-apiserver should be provided for the API request to work. The theory and practical examples are here:

    Access Clusters Using the Kubernetes API

    cURLing the Kubernetes API server

-- mebius99
Source: StackOverflow