K
Q

Prometheus Operator fails after Istio install

12/10/2019

I'm trying to install the prometheus operator and inject using the sidecar. Mutual TLS is turned on and works okay for Jaeger. For the operator though we get a failure on the oper-admission job (see image). Prometheus Operator Error

I believe Istio is causing this as if I release prometheus-operator prior to istio or without istio it works okay, but then it isn't injected.

I've tried setting the following in the istio operator sidecar settings:

rewriteapphttpprobe:true

I've also tried to extend the readinessInitialDelaySeconds to 10s but still get the error. Does anyone else have any ideas?

-- HandsDown
istio
kubernetes
kubernetes-helm
prometheus-operator

Similar Questions

Package 'python-software-properties' has no installation candidate
Kubernetes: weave picked the public IP on one of the worker nodes
kubernetes nginx ingress Request Header Or Cookie Too Large
kubernetes health check look for string

1 Answer

12/11/2019

Fisrt of all according to istio documentation Prometheus is used as default observation operator in istio mesh by default:

The default Istio metrics are defined by a set of configuration artifacts that ship with Istio and are exported to Prometheus by default. Operators are free to modify the shape and content of these metrics, as well as to change their collection mechanism, to meet their individual monitoring needs.

So by having istio injected prometheus operator You end up with two Prometheus operators in Your istio mesh.

Secondly, when you enforce Mutual TLS in Your istio mesh every connection has to be secure (TLS). And as You mentioned it works when there is no istio injection.

So the most likely cause is that the readiness probe fails because it is using HTTP protocol which is insecure (plain text) and this is one of the reason why You would get 503 error.

If you really need prometheus operator within istio mesh, this could be fixed by creating DestinationRule to Disable tls mode just for the readiness probe.

Example:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: "readiness-probe-dr"
 namespace: "prometheus-namespace"
spec:
 host: "prometheus-prometheus-oper-prometheus.svc.cluster.local"
 trafficPolicy:
   tls:
     mode: DISABLE
EOF

Note: Make sure to modify it so that it matches Your namespaces and hosts. Also there could be some other prometheus collisions within mesh.

The other solution would be not to have prometheus istio injected in the first place. You can disable istio injection in prometheus namespace by using the following commands:

$ kubectl get namespace -L istio-injection
NAME              STATUS   AGE     ISTIO-INJECTION
default           Active   4d22h   enabled
istio-system      Active   4d22h   disabled
kube-node-lease   Active   4d22h   
kube-public       Active   4d22h   
kube-system       Active   4d22h   
prometheus        Active   30s     enabled
$ kubectl label namespace prometheus istio-injection=disabled --overwrite
namespace/prometheus labeled
$ kubectl get namespace -L istio-injection
NAME              STATUS   AGE     ISTIO-INJECTION
default           Active   4d22h   enabled
istio-system      Active   4d22h   disabled
kube-node-lease   Active   4d22h   
kube-public       Active   4d22h   
kube-system       Active   4d22h   
prometheus        Active   73s     disabled
-- Piotr Malec
Source: StackOverflow