I currently have an Azure Kubernetes cluster running with Promtheus and Grafana deployments. Prometheus is using the local cluster storage for the database and I want to mount a persistent volume in the Kubernetes cluster that points back to an Azure Storage Account (file share) for the Prometheus database.
I would like to do this because it seems cleaner than setting up a remote-write configuration and solves the issue that remote-writes solve and that is 'scalability and durability'. I've done some testing and proven out this does in fact work for a non-production, low traffic environment.
I would like to know if there are any pitfalls I should be aware of if I do move forward with this plan. Has anybody else done this and encountered any issues?
Create storage class to be used for prometheus data. Update the details in Prometheus manifest file. sample is given below
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: k8s
labels:
prometheus: k8s
spec:
replicas: 2
version: PROMETHEUS_VERSION
externalUrl: PROMETHEUS_EXTERNAL_URL
serviceAccountName: prometheus-k8s
serviceMonitorSelector:
matchExpressions:
- {key: k8s-app, operator: Exists}
ruleSelector:
matchLabels:
role: alert-rules
prometheus: k8s
nodeSelector:
node_label_key: node_label_value
resources:
requests:
memory: PROMETHEUS_MEMORY_REQUEST
retention: PROMETHEUS_STORAGE_RETENTION
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
storage:
class: STORAGE_CLASS_TYPE
selector:
resources:
volumeClaimTemplate:
metadata:
annotations:
annotation1: prometheus
spec:
storageClassName: STORAGE_CLASS_TYPE
resources:
requests:
storage: PROMETHEUS_STORAGE_VOLUME_SIZE