how to integrate prometheus on kubernetes cluster?

9/4/2017

This is my prometheus.yml file and i create configmap using kubectl create configmap prometheus-server-config --from-file=prometheus.yml

global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
- job_name: 'goserver'
  scheme: http
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt         
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - api_servers:
    - 'https://kubernetes.default.svc'
    in_cluster: true
    role: pod        
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    regex: goserver
    action: keep        
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_ready]
    action: replace
    target_label: kubernetes_pod_ready    
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__   

and below is the deployment-prometheus.yaml file

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
      app: prometheus-server
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
          - "-config.file=/etc/prometheus/conf/prometheus.yml"
          # Metrics are stored in an emptyDir volume which
          # exists as long as the Pod is running on that Node.
          # The data in an emptyDir volume is safe across container crashes.
          - "-storage.local.path=/prometheus/"
        ports:
          - containerPort: 9090
        volumeMounts:
          - name: prometheus-server-volume
            mountPath: /etc/prometheus/conf/
          - name: prometheus-storage-volume
            mountPath: /prometheus/
      volumes:
        - name: prometheus-server-volume
          configMap:
            name: prometheus-server-config
        - name: prometheus-storage-volume
          emptyDir: {} # containers in the Pod can all read and write the same files here.

But whenever i am creating the deployment using kubectl create -f deployment-prometheus.yaml the pod status displays CrashLoopBackOff. I went through prometheus-kubernetes example and i'm running my cluster in minikube. What should be the cause of this?

-- Utkarsh Mani Tripathi
kubernetes
prometheus

2 Answers

9/2/2018

Oneliner

kubectl apply --filename https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests-all.yaml

Pre-Requisite

kubectl create namespace monitoring

Outcome

  1. Prometheus Server
  2. Service Endpoint for Prometheus Dashboard
  3. Alert Manager config

Reference

https://github.com/giantswarm/kubernetes-prometheus#quick-start

@Utkarsh Mani Tripathi

-- suryakrupa
Source: StackOverflow

6/6/2018

I am running the cluster on AWS, the following tutorials work for me https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13. Let see if you can adapt it. Basicallly, I used helm to install coreos/kube-prometheus.

# initialize tiller account kubectl create serviceaccount -n kube-system tiller kubectl create clusterrolebinding tiller-binding --clusterrole=cluster-admin --serviceaccount kube-system:tiller helm init --service-account tiller # install Prometheus app sleep 1m helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/ helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring helm install coreos/kube-prometheus --name kube-prometheus --namespace monitoring --set global.rbacEnable=true --set prometheus.resources.requests.memory=300Mi # forward ports kubectl port-forward -n monitoring prometheus-kube-prometheus-0 9090 & kubectl port-forward $(kubectl get pods --selector=app=kube-prometheus-grafana -n monitoring --output=jsonpath="{.items..metadata.name}") -n monitoring 3000 & kubectl port-forward -n monitoring alertmanager-kube-prometheus-0 9093 &

-- beahacker
Source: StackOverflow