I have Prometheus 4.x running on my Kubernetes cluster, All my metrics were unable to retrive since error below:

level=warn ts=2021-11-29T08:43:50.103Z caller=scrape.go:945 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.0.10.4:9100/metrics msg="appending scrape report failed" err="write to WAL: log samples: write /data/wal/00007153: read-only file system"

I am unable to create new file under /data/wal as well which returns "read-only file system"

Any idea of why does it happen?