I have a few openshift (4.8)
clusters monitored centrally by Thanos.
I'm trying to configure the Grafana (version install outside the clusters 8.3.1
) Datasource with no success.
When accessing the URL from my desktop works well. However, when accessing from the Datasource configuration panel I am getting a 403 error
My hypothesis is that I have misconfigured something related to the TLS keys.
Any ideas?
This is the content of the logs error:
t=2021-12-31T16:08:05+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="Post \"https://thanos-querier-openshift-monitoring.mycluster/api/v1/query\": x509: certificate signed by unknown authority"
t=2021-12-31T16:08:05+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=33 size=166 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
t=2021-12-31T16:08:13+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="client_error: client error: 403"
t=2021-12-31T16:08:13+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=53 size=65 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
t=2021-12-31T16:08:14+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="client_error: client error: 403"
t=2021-12-31T16:08:14+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=50 size=65 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
t=2021-12-31T16:39:17+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="client_error: client error: 403"
t=2021-12-31T16:39:17+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=81 size=65 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
You have a problem with TLS verification:
x509: certificate signed by unknown authority
The easiest option is to check Skip TLS Verify
in the Prometheus datasource, so Grafana will not verify used TLS certificate used by Prometheus.
More secure (and more complicated) option will be to add used CA cert also to the Grafana container, so Grafana will be able to verify Prometheus cert.