K
Q

Thanos on Bare Metal

6/28/2019

I currently have Prometheus installed bare metal and running as docker containers. I use the same to monitor our infrastructure as well as Kubernetes clusters.

In order to make this set up HA, I was trying to deploy a proxy or a querier in front of the 2 Prometheus instances. And my first goal was to try Thanos. But I am not finding much documentation or information about bare-metal usage. The docs are all on Thanos implementation on Kubernetes.

Has anyone tried Thanos on bare metal?

UPDATE:

I used docker-compose to spin up sidecar and query components:

thanos-sidecar:
  image: improbable/thanos:v0.5.0
  restart: always
  volumes:
    - tsdb-vol:/prometheus
  command: ['sidecar', '--tsdb.path="/prometheus"', '--prometheus.url=http://metrics_prometheus_1:9090' ]
  ports:
    - '10902:10902'
    - '10901:10901'
  depends_on:
  - Prometheus
  network:
    - thanos


thanos-querier:
  image: improbable/thanos:v0.5.0
  logging:
  # limit logs retained on host to 25MB
    driver: "json-file"
    options:
      max-size: "500k"
      max-file: "50"
  restart: always
  command: ['query' , '--http-address=0.0.0.0:19192' , '--query.replica-label=replica' , '--store=metrics_thanos-sidecar_1:10901', '--store=172.XX.XX.XXX:10901']
  ports:
    - '19192:19192'
  depends_on:
  - thanos-sidecar
  network:
    - thanos

I have exposed the store API,s gRPC ports at 10901 but the thanos-querier is still not able to reach them. Is there anything else that's missing on sidecar configs?

-- swetad90
kubernetes
prometheus

Similar Questions

expose internal dns kubernetes
Something that I am not understanding about redis in K8s
How can I fix Edit cancelled, no changes made in shell
How do I delete kubernetes resources on a local deploy?
EKS cluster created by kops does not appear in AWS console
ActiveMQ Artemis web console through Kubernetes pod
aws-iam-authenticator returns already expired token - eks

2 Answers

6/29/2019

It shouldn't be that much different from running in in Kubernetes. The are K8s manifest files here, but you should be able to run each one of the components separately, either in a container or outside a containers.

For example, Store API:

thanos sidecar \
    --tsdb.path                 /var/prometheus \
    --objstore.config-file      bucket_config.yaml \       # Bucket config file to send data to
    --prometheus.url            http://localhost:9090 \    # Location of the Prometheus HTTP server
    --http-address              0.0.0.0:19191 \            # HTTP endpoint for collecting metrics on the Sidecar
    --grpc-address              0.0.0.0:19090              # GRPC endpoint for StoreAPI

or Query Gateway

thanos query \
    --http-address 0.0.0.0:19192 \                                # HTTP Endpoint for Query UI
    --store        1.2.3.4:19090 \                                # Static gRPC Store API Address for the query node to query
    --store        1.2.3.5:19090 \                                # Also repeatable
    --store        dnssrv+_grpc._tcp.thanos-store.monitoring.svc  # Supports DNS A & SRV records

or Compactor

thanos compact \
    --data-dir             /var/thanos/compact \  # Temporary workspace for data processing
    --objstore.config-file bucket_config.yaml \   # Bucket where to apply the compacting
    --http-address         0.0.0.0:19191          # HTTP endpoint for collecting metrics on the Compactor)

or Ruler

thanos rule \
    --data-dir             "/path/to/data" \
    --eval-interval        "30s" \
    --rule-file            "/path/to/rules/*.rules.yaml" \
    --alert.query-url      "http://0.0.0.0:9090" \ # This tells what query URL to link to in UI.
    --alertmanagers.url    "alert.thanos.io" \
    --query                "query.example.org" \
    --query                "query2.example.org" \
    --objstore.config-file "bucket.yml" \
    --label                'monitor_cluster="cluster1"'
    --label                'replica="A"

Thanos is a Go binary so it can run on most systems that Go supports as a target.

-- Rico
Source: StackOverflow
7/8/2019

TO BE NOTED:

  1. Docker-compose name resolution did not work with default network. So Had to create a explicit network for all Thanos components

  2. We need not give http in the store url. That was the issue why Thanos query was not able to connect with the remote .store API

Now, all runs fine!!

-- swetad90
Source: StackOverflow