Prometheus operator "Error opening query log file: permission denied"

10/31/2020

I'm trying to run Prometheus operator on Kubernetes, but after I trying to persist data on Rancher-RBD, there is an error:

level=info ts=2020-10-31T12:40:33.171Z caller=main.go:353 msg="Starting Prometheus" version="(version=2.22.0, branch=HEAD, revision=0a7fdd3b76960808c3a91d92267c3d815c1bc354)"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:358 build_context="(go=go1.15.3, user=root@6321101b2c50, date=20201015-12:29:59)"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:359 host_details="(Linux 4.14.35-1902.3.2.el7uek.x86_64 #2 SMP Tue Jul 30 03:59:02 GMT 2019 x86_64 prometheus-prometheus-0 (none))"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:360 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:361 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2020-10-31T12:40:33.173Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff711299c3, 0xb, 0x14, 0x30867c0, 0xc000e6f050, 0x30867c0)
	/app/promql/query_logger.go:117 +0x4cf
main.main()
	/app/cmd/prometheus/main.go:388 +0x536c

this is my operator deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.43.0
  name: prometheus-operator
  namespace: monitorings
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/name: prometheus-operator
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/name: prometheus-operator
        app.kubernetes.io/version: v0.43.0
    spec:
      containers:
      - args:
        - --kubelet-service=kube-system/kubelet
        - --logtostderr=true
        - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.43.0
        image: quay.io/prometheus-operator/prometheus-operator:v0.43.0
        name: prometheus-operator
        ports:
        - containerPort: 8080
          name: http
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
      nodeSelector:
        beta.kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: prometheus-operator

I tried to add an initContainers to change permission but the problem already exusts:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    app: strimzi
spec:
  replicas: 1
  serviceAccountName: prometheus-server
  podMonitorSelector:
    matchLabels:
      app: strimzi
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
  ruleSelector:
    matchLabels:
      role: alert-rules
      app: strimzi
  alerting:
    alertmanagers:
    - namespace: monitorings
      name: alertmanager
      port: alertmanager
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml
  imagePullSecrets:
        - name: nexuspullsecret
  initContainers:
    - name: init
      image: debian:stable
      command:  ["chmod", "-R", "777", "/mnt"]
      volumeMounts:
      - name: prometheus-prometheus-db
        mountPath: /mnt
        subPath: prometheus
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: rancher-rbd
        name: prometheus-prometheus-db
        resources:
          requests:
            storage: 10Gi
-- Amir Soleimani Borujerdi
kubernetes
prometheus-operator

1 Answer

11/1/2020

I changed my initContainers like here and it works:

  initContainers:
      - name: "init-datapath"
        image: debian:stable
        command: ["chown", "-R", "65534:65534", "/data"]
        command: ["/bin/chmod","-R","777","/data"]
        volumeMounts:
        - name: prometheus-prometheus-db
          mountPath: /data
          subPath: ""

  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: rancher-rbd
        name: prometheus-prometheus-db
        resources:
          requests:
            storage: 10Gi
-- Amir Soleimani Borujerdi
Source: StackOverflow