I did look at this, but I don't feel like the answer is covered: kubernetes persistence volume and persistence volume claim exceeded storage
Anyways, I have tried to look in the documentation but could not find out what is going to happen when an PVC Azure disk is full? So, we have a grafana application which monitors some data. We use the PVC to make sure that the data is saved if the pod gets killed. Right now the pod continously fetches data and the disk gets more and more full. What happens when the disk is full? Ideally it would be nice to implement some sort of functionality, such that when it gets like 80% full, it removes the let's say 20% of the data, starting from the oldest for example. Or how do we tackle this problem?
pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graphite-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: managed-premium
resources:
requests:
storage: 256GiThink of PVC as a folder that is mounted to your container running grafana service. It has a fixed size which you provided and as per the question, it is not going to increase.
What happens when the disk is full?
There is nothing different here that happens from a normal service running on a system and runs out of the system. If it were your local machine or cloud VM you would get an alert about storage and if you dint take action, the service will error out saying out of disk space error Now you can use services like Prometheus with Kubernetes plugin to get storage space alerts, but by default, Kubernetes won't provide any alert.
how do we tackle this(disk space) problem?
Again same way you would on a normal system, there is a number of solutions. But if you think of it, the System or VM or Kubernetes is not the right candidate to decide which files should be removed and which should be kept, the reason being kubernetes does not know what data is and for the fact, it does not own the data. The service does. On the other hand, you can use the service or create a custom new archiving service to take the data from your Grafana PVC and place it S3 or any other storage.