How can I tell with kubectl how much ephemeral storage a pod is currently using?
In a Kubernetes pod spec, I can specify resource requests and limits for CPU, memory, and ephemeral storage:
resources:
requests:
memory: "60Mi"
cpu: "70m"
ephemeral-storage: "2Gi"
limits:
memory: "65Mi"
cpu: "75m"
ephemeral-storage: "4Gi"However, to set good requests and limits on ephemeral storage, I need to know what this value actually is for a running pod, which I can't figure out. I can get CPU and memory usage using kubectl top pod, but, from what I can tell, ephemeral storage usage is only actually calculated when making an actual eviction decision.
The pure raw approach for this is to use the disk usage (du) Unix command line.
Shell into your pod:
$ kubectl exec -it <pod-id> shChange dirs to the mount point of your ephemeral-storage (if you are using volume mounts):
$ mount # check mount points if you'd like
$ cd /mnt/of/ephemeral
$ du .If you are not using volume mounts:
$ du .There are other tools that you can use to get metrics:
cAdvisor also embedded into the kubelet code, exposed under the /stats/summary or /metrics endpoint. More info here. An example output:
$ curl -k -H 'Authorization: Bearer <Redacted>' \
https://node-hostname:10250/stats/summary
{
"node": {
"nodeName": "node-hostname",
"systemContainers": [
{
"name": "kubelet",
...
"volume": [
{
"time": "2018-11-08T23:52:03Z",
"availableBytes": 1969168384,
"capacityBytes": 1969180672,
"usedBytes": 12288,
"inodesFree": 480748,
"inodes": 480757,
"inodesUsed": 9,
"name": "kube-proxy-token-pprwb"
}
],
"ephemeral-storage": {
"time": "2018-11-09T00:05:10Z",
"availableBytes": 31057477632,
"capacityBytes": 41567858688,
"inodesFree": 4873887,
"inodes": 5120000
}
...
}
Similarly:
$ curl -k -H 'Authorization: Bearer <Redacted>' \
https://node-hostname:10250/stats/summary
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
...
More info on kubelet authentication/authorization.
More info on K8s metrics here.