K
Q

Question

Prometheus is crash looping when the pod recreates

2/25/2022

We are Prometheus version 2.26.0 and kubernetes verion 1.21.7 in Azure. We mount the data in Azure storage NFS and it was working fine. From last few days Prometheus container is crash-looping and below are the logs

level=info ts=2022-01-26T08:04:14.375Z caller=main.go:418 msg="Starting Prometheus" version="(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:423 build_context="(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:424 host_details="(Linux 5.4.0-1065-azure #68~18.04.1-Ubuntu SMP Fri Dec 3 14:08:44 UTC 2021 x86_64 prometheus-6b9d9d54f4-nc45x (none))"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:425 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:426 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-01-26T08:04:14.503Z caller=web.go:540 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-01-26T08:04:14.507Z caller=main.go:795 msg="Starting TSDB ..."
level=info ts=2022-01-26T08:04:14.509Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2022-01-26T08:04:14.560Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641478251052 maxt=1641513600000 ulid=01FRSEHC4YHV3N26JY5AMNZFRW
level=info ts=2022-01-26T08:04:14.593Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641513600037 maxt=1641578400000 ulid=01FRVCAP2VJGDF0Z9CS24EXAJJ
level=info ts=2022-01-26T08:04:14.624Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641578400038 maxt=1641643200000 ulid=01FRXA4AQHMHAEYWRKQFGP075M
level=info ts=2022-01-26T08:04:14.651Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641643200422 maxt=1641708000000 ulid=01FRZ7XQQ4RA96DCPPBP22D71N
level=info ts=2022-01-26T08:04:14.679Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641708000020 maxt=1641772800000 ulid=01FS15QDG6BS7H6M6Y09HG3E12
level=info ts=2022-01-26T08:04:14.707Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641772800011 maxt=1641837600000 ulid=01FS33GT38PRSB9VP56YFXT2M0
level=info ts=2022-01-26T08:04:14.736Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641963555381 maxt=1641967200000 ulid=01FS6MRNZEWT1Z6P697K09KHD7
level=info ts=2022-01-26T08:04:14.763Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641837600100 maxt=1641902400000 ulid=01FS6R88C70TCD8CYC4XJ95X23
level=info ts=2022-01-26T08:04:14.810Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641967200019 maxt=1642032000000 ulid=01FS8WXQP3YJ7EXBVNYBQG4DVY
level=info ts=2022-01-26T08:04:14.836Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642032000072 maxt=1642096800000 ulid=01FSATQBR4XBQRDM72ATFS9PQ2
level=info ts=2022-01-26T08:04:14.863Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642096800059 maxt=1642161600000 ulid=01FSCRHE2YBDX7GPRPSH6BNGRX
level=info ts=2022-01-26T08:04:14.895Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642161600091 maxt=1642226400000 ulid=01FSEPB1GPGAANVCQ2VKW9BQ4G
level=info ts=2022-01-26T08:04:14.948Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642226400026 maxt=1642291200000 ulid=01FSGM4J0G1D0A6H1GD3N9C372
level=info ts=2022-01-26T08:04:14.973Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642291200005 maxt=1642356000000 ulid=01FSJHY6W0FRYDHCXBVB5XPFYG
level=info ts=2022-01-26T08:04:15.002Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642356000027 maxt=1642420800000 ulid=01FSMFR96DASV6YPN66W7C86H9
level=info ts=2022-01-26T08:04:15.077Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642420800042 maxt=1642485600000 ulid=01FSPDHGWRT65D8CKWQ2JPRHW3
level=info ts=2022-01-26T08:04:15.105Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642485600006 maxt=1642550400000 ulid=01FSRBAVP2MW71H08F32D6HGB4
level=info ts=2022-01-26T08:04:15.130Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642550400028 maxt=1642615200000 ulid=01FST9482FD0Z3PHXHNW2W616E
level=info ts=2022-01-26T08:04:15.157Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642680000018 maxt=1642687200000 ulid=01FSW00TJKJ7CGCQ7JJS3XQK8G
level=info ts=2022-01-26T08:04:15.187Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642687200018 maxt=1642694400000 ulid=01FSW6WHTSEAXHWV5J7PQP94X7
level=info ts=2022-01-26T08:04:15.213Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642615200021 maxt=1642680000000 ulid=01FSW6XYH2Y429PG5YRM0K45XS
level=info ts=2022-01-26T08:04:15.275Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642694400018 maxt=1642701600000 ulid=01FSWDR92Y7H302NDZRX1V2PX9
level=info ts=2022-01-26T08:04:21.840Z caller=head.go:696 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-01-26T08:04:22.623Z caller=head.go:710 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=782.403397ms
level=info ts=2022-01-26T08:04:22.623Z caller=head.go:716 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2022-01-26T08:04:34.169Z caller=head.go:742 component=tsdb msg="WAL checkpoint loaded"
level=info ts=2022-01-26T08:04:38.895Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=299 maxSegment=7511
level=warn ts=2022-01-26T08:04:46.423Z caller=main.go:645 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:668 msg="Stopping scrape discovery manager..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:682 msg="Stopping notify discovery manager..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:704 msg="Stopping scrape manager..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:678 msg="Notify discovery manager stopped"
level=info ts=2022-01-26T08:04:46.425Z caller=main.go:698 msg="Scrape manager stopped"
level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:934 component="rule manager" msg="Stopping rule manager..."
level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:944 component="rule manager" msg="Rule manager stopped"
level=info ts=2022-01-26T08:04:46.426Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
level=info ts=2022-01-26T08:04:46.426Z caller=main.go:872 msg="Notifier manager stopped"
level=info ts=2022-01-26T08:04:46.426Z caller=main.go:664 msg="Scrape discovery manager stopped"
level=info ts=2022-01-26T08:04:46.792Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=300 maxSegment=7511
level=info ts=2022-01-26T08:04:46.870Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=301 maxSegment=7511
level=info ts=2022-01-26T08:04:46.901Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=302 maxSegment=7511
level=info ts=2022-01-26T08:04:46.946Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=303 maxSegment=7511
level=info ts=2022-01-26T08:04:46.974Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=304 maxSegment=7511
level=info ts=2022-01-26T08:04:47.008Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=305 maxSegment=7511
level=info ts=2022-01-26T08:04:47.034Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=306 maxSegment=7511
level=info ts=2022-01-26T08:04:47.067Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=307 maxSegment=7511
level=info ts=2022-01-26T08:04:47.098Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=308 maxSegment=7511
level=info ts=2022-01-26T08:04:47.124Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=309 maxSegment=7511
level=info ts=2022-01-26T08:04:47.158Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=310 maxSegment=7511
level=info ts=2022-01-26T08:04:47.203Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=311 maxSegment=7511
level=info ts=2022-01-26T08:04:47.254Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=312 maxSegment=7511
level=info ts=2022-01-26T08:04:47.486Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=313 maxSegment=7511
level=info ts=2022-01-26T08:04:47.511Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=314 maxSegment=7511
level=info ts=2022-01-26T08:04:47.539Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=315 maxSegment=7511
level=info ts=2022-01-26T08:04:47.564Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=316 maxSegment=7511

.
.
.
.
.
.
.
.
.

level=info ts=2022-01-26T08:05:15.161Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1401 maxSegment=7511
level=info ts=2022-01-26T08:05:15.182Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1402 maxSegment=7511
level=info ts=2022-01-26T08:05:15.205Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1403 maxSegment=7511
level=info ts=2022-01-26T08:05:15.229Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1404 maxSegment=7511
level=info ts=2022-01-26T08:05:15.251Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1405 maxSegment=7511
level=info ts=2022-01-26T08:05:15.274Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1406 maxSegment=7511
level=info ts=2022-01-26T08:05:15.297Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1407 maxSegment=7511
level=info ts=2022-01-26T08:05:15.323Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1408 maxSegment=7511
level=info ts=2022-01-26T08:05:15.349Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1409 maxSegment=7511
level=info ts=2022-01-26T08:05:15.372Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1410 maxSegment=7511
level=info ts=2022-01-26T08:05:15.426Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1411 maxSegment=7511
level=info ts=2022-01-26T08:05:15.452Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1412 maxSegment=7511
level=info ts=2022-01-26T08:05:15.475Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1413 maxSegment=7511
level=info ts=2022-01-26T08:05:15.498Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1414 maxSegment=7511
rpc error: code = NotFound desc = an error occurred when try to find container "ae14079418f59b04bb80d8413e8fdc34f167bfe762317ef674e05466d34c9e1f": not found

So I deleted the deployment and redeployed to the the same storage account, the I got new error

level=info ts=2022-01-26T11:10:11.530Z caller=main.go:418 msg="Starting Prometheus" version="(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)"
level=info ts=2022-01-26T11:10:11.534Z caller=main.go:423 build_context="(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)"
level=info ts=2022-01-26T11:10:11.535Z caller=main.go:424 host_details="(Linux 5.4.0-1064-azure #67~18.04.1-Ubuntu SMP Wed Nov 10 11:38:21 UTC 2021 x86_64 prometheus-6b9d9d54f4-wnmzh (none))"
level=info ts=2022-01-26T11:10:11.536Z caller=main.go:425 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-01-26T11:10:11.536Z caller=main.go:426 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-01-26T11:10:14.168Z caller=web.go:540 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-01-26T11:10:15.385Z caller=main.go:795 msg="Starting TSDB ..."
level=info ts=2022-01-26T11:10:16.022Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641837600024 maxt=1641902400000 ulid=01FS51ANKBFTVNRPZ68FGQQ5GA
level=info ts=2022-01-26T11:10:16.309Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2022-01-26T11:10:16.494Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641902400005 maxt=1641967200000 ulid=01FS6Z46FGXN932K7D39D9166D
level=info ts=2022-01-26T11:10:16.806Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641967200106 maxt=1642032000000 ulid=01FS8WXRJ7Q80FKD4C8EJNR0AD
level=info ts=2022-01-26T11:10:17.011Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642032000003 maxt=1642096800000 ulid=01FSATQE1VMNR101KRW1X10Q75
level=info ts=2022-01-26T11:10:17.305Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642096800206 maxt=1642161600000 ulid=01FSCRGVT1E7562SF7EQN12JBM
level=info ts=2022-01-26T11:10:18.240Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642161600059 maxt=1642226400000 ulid=01FSEPAFP2CX03ANRB7Q1AG514
level=info ts=2022-01-26T11:10:21.046Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642226400051 maxt=1642291200000 ulid=01FSGM3WT0TKR0XW9BD4QSKPQE
level=info ts=2022-01-26T11:10:21.422Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642291200113 maxt=1642356000000 ulid=01FSJHXKHMANW0E6FXDXVM265G
level=info ts=2022-01-26T11:10:22.822Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642356000032 maxt=1642420800000 ulid=01FSMFQ6XJ97VJFKNCYQBVB4DZ
level=info ts=2022-01-26T11:10:23.536Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642420800021 maxt=1642485600000 ulid=01FSPDGM95FDDV2CDWX93BTDCS
level=info ts=2022-01-26T11:10:23.880Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642485600072 maxt=1642550400000 ulid=01FSRBA555RWY4QNP4HD9YKRBM
level=info ts=2022-01-26T11:10:25.021Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642550400031 maxt=1642615200000 ulid=01FST93N3C82K9VS20MKTMGGYC
level=info ts=2022-01-26T11:10:25.713Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642615200014 maxt=1642680000000 ulid=01FSW6X95FRNSN1XJZ2YK0MXW7
level=info ts=2022-01-26T11:10:26.634Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642680000012 maxt=1642744800000 ulid=01FSY4PXA7V1XQHHA3MC35JSWQ
level=info ts=2022-01-26T11:10:27.776Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642744800174 maxt=1642809600000 ulid=01FT02G9XGHPV8GME53ZPMYXE6
level=info ts=2022-01-26T11:10:28.760Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642809600070 maxt=1642874400000 ulid=01FT209WP8AXXVZB1NCSC55ACE
level=info ts=2022-01-26T11:10:29.618Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642874400253 maxt=1642939200000 ulid=01FT3Y3A4H72FFW318RKHEXXGA
level=info ts=2022-01-26T11:10:30.313Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642939200047 maxt=1643004000000 ulid=01FT5VX3YC838QN5VQFAERV1QX
level=info ts=2022-01-26T11:10:30.483Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643004000040 maxt=1643068800000 ulid=01FT7SPHC5EV0SS1R0WT04H9FR
level=info ts=2022-01-26T11:10:30.696Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643068800035 maxt=1643133600000 ulid=01FT9QFZXBZ7EYY2CTE8WXZTB9
level=info ts=2022-01-26T11:10:31.838Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643133600000 maxt=1643155200000 ulid=01FTA574G4M45WX97Z470DQF73
level=info ts=2022-01-26T11:10:33.686Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643176800008 maxt=1643184000000 ulid=01FTASSZCG8V5N2VGAGFBYJBSR
level=info ts=2022-01-26T11:10:36.078Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643184000000 maxt=1643191200000 ulid=01FTB0NP47JW5JCF808QZZ8WZQ
level=info ts=2022-01-26T11:10:36.442Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643155200065 maxt=1643176800000 ulid=01FTB0P9H3H09B2ADD5X1RXFW6
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:668 msg="Stopping scrape discovery manager..."
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:682 msg="Stopping notify discovery manager..."
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:704 msg="Stopping scrape manager..."
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:678 msg="Notify discovery manager stopped"
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:664 msg="Scrape discovery manager stopped"
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:698 msg="Scrape manager stopped"
level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:934 component="rule manager" msg="Stopping rule manager..."
level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:944 component="rule manager" msg="Rule manager stopped"
level=info ts=2022-01-26T11:10:40.080Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
level=info ts=2022-01-26T11:10:40.080Z caller=main.go:872 msg="Notifier manager stopped"
level=error ts=2022-01-26T11:10:40.080Z caller=main.go:881 err="opening storage failed: lock DB directory: resource temporarily unavailable"

The yaml is provided by the Istio .Below is deployment yaml file.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-14.6.1
    heritage: Helm
  name: prometheus
  namespace: istio-system
spec:
  selector:
    matchLabels:
      component: "server"
      app: prometheus
      release: prometheus
  replicas: 1
  template:
    metadata:
      labels:
        component: "server"
        app: prometheus
        release: prometheus
        chart: prometheus-14.6.1
        heritage: Helm
        
        sidecar.istio.io/inject: "false"
    spec:
      enableServiceLinks: true
      serviceAccountName: prometheus
      containers:
        - name: prometheus-server-configmap-reload
          image: "jimmidyson/configmap-reload:v0.5.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://127.0.0.1:9090/-/reload
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
              readOnly: true

        - name: prometheus-server
          image: "prom/prometheus:v2.26.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --storage.tsdb.retention.time=15d
            - --config.file=/etc/config/prometheus.yml
            - --storage.tsdb.path=/data
            - --web.console.libraries=/etc/prometheus/console_libraries
            - --web.console.templates=/etc/prometheus/consoles
            - --web.enable-lifecycle
          ports:
            - containerPort: 9090
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 4
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
            - name: azurefileshare
              mountPath: /data
              subPath: ""
      hostNetwork: false
      dnsPolicy: ClusterFirst
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      terminationGracePeriodSeconds: 300
      volumes:
        - name: config-volume
          configMap:
            name: prometheus
        - name: azurefileshare
          azureFile:
            secretName: log-storage-secret
            shareName: prometheusfileshare
            readOnly: false

Expected Behavior When I mount the data to new container, It should load the data.

Actual Behavior Not able to load the data or not able bind the data with newly created pod when old pod dies

Help me out, to resolve the issue.

-- kiran

azure

kubernetes

prometheus

1 Answer

3/2/2022

Thank You YwH for your suggestion, Posting this an answer so it can help other community member if they encounter the same issue in future.

As stated in this document Istio provides a basic sample installation to quickly get Prometheus up and running: This is intended for demonstration only, and is not tuned for performance or security.

Note : Isio configuration is well-suited for small clusters and monitoring for short time horizons, it is not suitable for large-scale meshes or monitoring over a period of days or weeks

Solution : Prometheus is a stateful application, better deployed with a StatefulSet, not Deployment.

StatefulSets are valuable for applications that require one or more of the following.

Stable, persistent storage. Ordered, graceful deployment and scaling.

You can use this [Stateful]3 code for deployment of prometheus container.

-- RahulKumarShaw-MT

Source: StackOverflow

KQ

Prometheus is crash looping when the pod recreates

Similar Questions

1 Answer

K
Q