I got a kubernetes cluster running on AWS using kops. I also got prometheus and grafana set up using kube-prometheus.
What I'm trying to do is to store metrics gathered by prometheus on EBS. My persistent volume claim yaml is:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-data
namespace: monitoring
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10GiAnd prometheus.yaml is:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
baseImage: quay.io/prometheus/prometheus
nodeSelector:
beta.kubernetes.io/os: linux
replicas: 2
resources:
requests:
memory: 400Mi
volumeMounts:
- name: prometheus-data
mountPath: "/data"
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
volumes:
- name: prometheus-data
persistentVolumeClaim:
claimName: prometheus-data
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.4.3The 10Gi EBS volume is getting created but it's state remains available. I also tried deleting prometheus pods hoping the data will be retained. Unfortunately that was not the case.
I am able to setup the kube-prometheus with the persistent storage. Please check the following json files:
Promethues-deploy.json
{
"apiVersion": "monitoring.coreos.com/v1",
"kind": "Prometheus",
"metadata": {
"labels": {
"prometheus": "k8s"
},
"name": "k8s-prom",
"namespace": "monitoring"
},
"spec": {
"alerting": {
"alertmanagers": [
{
"name": "alertmanager-main",
"namespace": "monitoring",
"port": "web"
}
]
},
"baseImage": "quay.io/prometheus/prometheus",
"replicas": 1,
"resources": {
"requests": {
"memory": "400Mi"
}
},
"ruleSelector": {
"matchLabels": {
"prometheus": "k8s",
"role": "alert-rules"
}
},
"securityContext": {
"fsGroup": 0,
"runAsNonRoot": false,
"runAsUser": 0
},
"serviceAccountName": "prometheus",
"serviceMonitorSelector": {
"matchExpressions": [
{
"key": "k8s-app",
"operator": "Exists"
}
]
},
"storage": {
"class": "",
"resources": {},
"selector": {},
"volumeClaimTemplate": {
"spec": {
"resources": {
"requests": {
"storage": "10Gi"
}
},
"selector": {
"matchLabels": {
"app": "k8s-vol"
}
},
"storageClassName": "no-provision"
}
}
},
"version": "v2.2.1"
} }
Prometheus-pv.json
{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"labels": {
"app": "k8s-vol"
},
"name": "prometheus-vol",
"namespace": "monitoring"
},
"spec": {
"storageClassName": "no-provision",
"accessModes": [
"ReadWriteOnce"
],
"capacity": {
"storage": "10Gi"
},
"hostPath": {
"path": "/data"
},
"persistentVolumeReclaimPolicy": "Retain"
},
"status": {
"phase": "Bound"
} }
Hope it helps.