parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 20: mapping values are not allowed in this context"

11/8/2019

I am deploying a new prometheus pod to an existing namespace I prepared the clusterRole config and the configMap and everything works fine until I created the deployment in kubernetes. The pod crashed once its created and the k8s log shows this error

err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 20: mapping values are not allowed in this context"

Does anyone know how to solve this please?

Here is my deployment YAML file

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - "--config.file=/etc/prometheus/prometheus.yml" 
            - "--storage.tsdb.path=/prometheus" 
            - "--web.console.libraries=/usr/share/prometheus/console_libraries" 
            - "--web.console.templates=/usr/share/prometheus/consoles"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf

        - name: prometheus-storage-volume
          emptyDir: {}

and here is the error log from k8s

level=info ts=2019-11-08T08:59:43.155Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.13.1, branch=HEAD, revision=6f92ce56053866194ae5937012c1bec40f1dd1d9)"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:333 build_context="(go=go1.13.1, user=root@88e419aa1676, date=20191017-13:15:01)"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:334 host_details="(Linux 4.15.0 #1 SMP Sun Jun 23 23:02:01 PDT 2019 x86_64 prometheus-deployment-6568ff7b6b-jvfsl (none))"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:335 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-11-08T08:59:43.156Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-11-08T08:59:43.161Z caller=main.go:657 msg="Starting TSDB ..."
level=info ts=2019-11-08T08:59:43.161Z caller=web.go:450 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-11-08T08:59:43.191Z caller=head.go:514 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-11-08T08:59:43.192Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
level=info ts=2019-11-08T08:59:43.192Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:672 fs_type=EXT4_SUPER_MAGIC
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:673 msg="TSDB started"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:526 msg="Stopping scrape discovery manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:540 msg="Stopping notify discovery manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:562 msg="Stopping scrape manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-11-08T08:59:43.193Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:536 msg="Notify discovery manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:522 msg="Scrape discovery manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=main.go:556 msg="Scrape manager stopped"
level=info ts=2019-11-08T08:59:43.193Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."
level=info ts=2019-11-08T08:59:43.194Z caller=main.go:727 msg="Notifier manager stopped"
level=error ts=2019-11-08T08:59:43.194Z caller=main.go:736 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 20: mapping values are not allowed in this context"
-- SOF
kubernetes
prometheus
yaml

1 Answer

11/8/2019

The error says the content of /etc/prometheus/prometheus.yml is malformed. Making sure that file contains valid YAML should fix your issue.

Judging from your deployment, the prometheus-server-conf ConfigMap is mounted on /etc/prometheus/ so whatever is in under the key prometheus.yml in that ConfigMap is likely the culprit.

-- Alassane Ndiaye
Source: StackOverflow