The problem is that after deploying new deployment like
kubectl apply -f deployment.yml(let's say deployment with one replica)
Kubernetes will create second pod and shutdown the previous one - ok so far.
But immediately after kubectl apply I would like to detect in CI/CD if deployment was successful and in any case (no matter if rollout succeeded or failed) fetch log from one of newly deployed pod in order to provide as much as possible information in CI/CD log to determine what went wrong.
So I'm using
kubectl rollout status deployment deployment-namewhich is waiting for deployment to rollout. Immediately after though you will end up with two pods, one in status "Running" and another "Terminating".
Now the problematic part: Normally I would use method like
kubectl get pods --selector=app=deployment-name --output=jsonpath='{.items[*].metadata.name}' --field-selector=status.phase=Runningbut unfortunately it will return names of both pods ("Running" and "Terminating") separated with space.
Now I've tried also
kubectl get pods --selector=app=deployment-name --output=jsonpath='{.items[*].metadata.name}' --field-selector=status.phase=Running,status.phase!=Terminatingaccording to documentation:
but for some reason this will return exactly the same result, both pods running and terminating.
The quesiton is:
How to properly exclude TERMINATING pods from the result?
from the command line you can use commands like
grepto catch what you like or you can do what I do and just look at all the pods with
watch command to see the changes and look at the errors from a separate terminal.
What I use for these kind of situations:
watch -n0.5 " kubectl get pods | grep -v 'Running' "to catch pods with non Running status (Completed, OOMKilled, Pending, etc.) then from a separate terminal
kubectl logs podName && kubectl describe pods podNameI hope I was a little help.
Even for getting the latest revision number I use
kubectl rollout history deployment.v1.apps/deploymentName | tail -2 | head -1 | awk '{print $1}'But maybe you can use
kubectl get pods --sort-by=.metadata.creationTimestamp and then take the last one so finally maybe, you can use
kubectl logs -c [containerName] $(kubectl get pods --sort-by=.metadata.creationTimestamp | tail -1 | awk '{print$1}') this can give me the latest created pod and its logs
Known issue since 2018. Havent resolved yet.
Please refer to Kubectl returns pods in various states when only those in Running state are selected via --field-selector, or -o jsonpath for more details.
In short: There is NO normal, SHORT, adequate one-line command that would do what you want.
This looks like not a bug. kubectl get pods output STATUS column not shows status.phase. kubectl get pods displays table format of PodList object and uses status.containerStatuses states to display STATUS column data.
Pod phase valid states are pending, running, succeeded, failed and unknown. From this document 'Running' state depends on 'restartPolicy'.
Jsonpath applies on 'PodList' object, not on kubectl output. status.containerStatuses gives whether pod containers running or not. Field selectors vary by Kubernetes resource types. status.containerStatuses not supported in pods field selectors.
Workarounds:
1. kubectl get pods | grep Running
2. kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].state.running},{.items[*].metadata.name}' --field-selector=status.phase==Running | sed 's/ /\n/' | grep startedAt | awk -F',' '{print $2}'
3. (source) kubectl get pods --field-selector=status.phase=Running --template {{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}