I'm setting up a service, which use k8s to start jupyter pod for each user. I need to provide a real-time resource usage of the jupyter pod when user query for it. I tried metrics-server and kubectl top. Their results are from cache and have about 1 minute delay. Is there any solution?
> kubectl versionClient Version: version.Info{Major:"1", Minor:"14",GitVersion:"v1.14.0",GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z",GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"8cb561c", GitTreeState:"", BuildDate:"2019-05-23T11:28:17Z", GoVersion:"go1.10.8", Compiler:"gc",Platform:"linux/amd64"}
you could query the kubelet stats endpoint:
curl --insecure https://<node url>:10250/stats/summaryyou can also make the query more specific to pod/container
curl --insecure https://<node url>:10250/{namespace}/{podName}/{uid}/{containerName}where uid is basically any string...
The part of the code for your version is here
Another tip: if your pod run as part of host network you could query localhost also and would need to assign a service account with access to it. The query would look like this:
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl https://localhost:10250/stats/summary --header "Authorization: Bearer $TOKEN" --insecure