I am deploying a new prometheus pod to an existing namespace I prepared the clusterRole config and the configMap and everything works fine until I created the deployment in kubernetes. The pod crashed once its created and the k8s log shows this error
err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 20: mapping values are not allowed in this context"
Does anyone know how to solve this please?
Here is my deployment YAML file
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: default
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: prom/prometheus
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
emptyDir: {}
and here is the error log from k8s
level=info ts=2019-11-08T08:59:43.155Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.13.1, branch=HEAD, revision=6f92ce56053866194ae5937012c1bec40f1dd1d9)"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:333 build_context="(go=go1.13.1, user=root@88e419aa1676, date=20191017-13:15:01)"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:334 host_details="(Linux 4.15.0 #1 SMP Sun Jun 23 23:02:01 PDT 2019 x86_64 prometheus-deployment-6568ff7b6b-jvfsl (none))"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:335 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-11-08T08:59:43.161Z caller=main.go:657 msg="Starting TSDB ..."
level=info ts=2019-11-08T08:59:43.161Z caller=web.go:450 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-11-08T08:59:43.191Z caller=head.go:514 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-11-08T08:59:43.192Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
level=info ts=2019-11-08T08:59:43.192Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:672 fs_type=EXT4_SUPER_MAGIC
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:673 msg="TSDB started"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:526 msg="Stopping scrape discovery manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:540 msg="Stopping notify discovery manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:562 msg="Stopping scrape manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:536 msg="Notify discovery manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:522 msg="Scrape discovery manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:556 msg="Scrape manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."
level=info ts=2019-11-08T08:59:43.194Z caller=main.go:727 msg="Notifier manager stopped"
level=error ts=2019-11-08T08:59:43.194Z caller=main.go:736 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 20: mapping values are not allowed in this context"
The error says the content of /etc/prometheus/prometheus.yml
is malformed. Making sure that file contains valid YAML should fix your issue.
Judging from your deployment, the prometheus-server-conf
ConfigMap is mounted on /etc/prometheus/
so whatever is in under the key prometheus.yml
in that ConfigMap is likely the culprit.