I'm troubleshooting liveness probe failures. I'm able to extract specific entries from k8s event using this approach
k get events --sort-by=.metadata.creationTimestamp | grep Liveness
I'd like to get only the pod(s) that causing the issue.
I'm considering to pipe a cut, but I'm not sure which delimiter should I use to get the specific column.
Where I can find the delimiter related to that specific k8s resource(Events) used to printout the kubectl output?
Any other suggestion is appreciated
UPDATE so far these are the best options (w/o using extra tools) satisfying my specific needs:
k get events -o jsonpath='{range .items[*]}{.involvedObject.name}/{.involvedObject.namespace}: {.message}{"\n"}{end}' | grep Liveness
k get events -o custom-columns=POD:.involvedObject.name,NS:.involvedObject.namespace,MGG:.message | grep Liveness
Did you try to change the type of output? If you use JSON output("-o json" add at the end of the line.), you can handle your need by using small python code.
kubectl get event -n kube-system -o json
If you find the pod in the "FROM" column, you can show the messages.
there is a feature in kubernetes called jsonpath
validate if your jsonpath is correct with this online tool: https://jsonpath.com/
easily go through json keys with this online tool, so you needn't manually type the key names any more): http://jsonpathfinder.com/
so your command will be:
k get events --sort-by=.metadata.creationTimestamp --jsonpath '{ .xxxxxx }'
Jsonpath is a little bit limited to be used with filter and conditions, maybe for your case jq
will be more suitable.
I did a test using jq
to filter the Output of my probe: I've tested using the yaml from this link
The message of probe failure from this pod is:
Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
and the path for this message in json is .items[*].message
Using jq
I can filter only message
that contains "Liveness probe failed": and show the pod name:
k get events --sort-by=.metadata.creationTimestamp -o json | jq -c '.items[] | select(.message|test("Liveness probe failed")) | .metadata.name'
The output is:
"liveness-exec.15e791c17b80a3c1"
You can use jq
to format the message in order to get a more helpful output, with pod details.
Try to look this references links:
I hope it helps!