We are using Google kubernetes to deploy our microservices, circlci for deployment integration, we define our k8s files inside the githup repos.
The problem we are facing is that some services are taking time on startup by loading database schema and other pre data, but google kubernetes shutdown old pods before the new pods fully started.
can we tell kubernetes somehow to wait till the new pods are fully loaded or at least to wait 10 seconds before shutdown the old pods.
Yes, this is possible. Based on the description it sounds like you are using a single deployment that gets updated. The new pods are created and the old ones are removed before the new ones become ready.
To address this, you need to have a proper readinessProbe configured or readinessGates on the pods so that the pod status only becomes ready once it actually is ready. If you are not sure what to put as the probe, you can also define initialDelaySeconds with a guess at how much time you think the pod needs to start up.
You should also look into using the deployment spec field for minReadySeconds as well as defining a proper deployment strategy. You can ensure that the rolling update creates new pods (by defining the maxSurge field) and ensure that the old pod is not removed until the new one is ready and receiving traffic (using the maxUnavailable field = 0).
an example would be:
spec: replicas: 3 strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0
This will maintain 3 working replicas at any given time. When a new version is pushed, 1 new pod will be created with the new image. No pods will be taken offline until the new one is in ready state. Once it is, one of the old pods will be terminated and the cycle goes again. Feel free to change the maxSurge value to a higher number if you want the rollout to happen in one go.