I followed this tutorial to create a Kubernetes cluster on Azure to run build agents: http://www.chrisjohnson.io/2018/07/07/using-azure-kubernetes-service-aks-for-your-vsts-build-agents/
To recap what is there: a helm chart to do a deployment with a secret and a config map. For this deployment, I created a kubernetes cluster on Azure with all default settings and it is pulling an image from the docker hub with vsts build agent installed.
All was working fine, but recently pods started to be evicted pretty regularly, the message on them is:
Message: Pod The node was low on resource: [DiskPressure].
How can I fix this issue?
Just an FYI if you don't want to increase your disk space...
You can go to your agent's settings (Organization Settings > Agent Pools > {select you agent} > Settings) and make sure the option 'Delete unused working directory after x days' is checked.
If it's checked you may want to decrease the number of days to clean up unused working directories quicker.
Either/Or:
You upgrade the size of your main node disks with something like this.
Check what pods are taking up space. Is it logs? Is it cached data? is it swap? Every application is different so you will have to go case by case.
Set local ephemeral storage at the pod level for your workloads so that they don't go over. Pods using a lot will get evicted.
Use Persistent Volumes for your workloads, especially some that are not local and just reserved for your applications.