K
Q

istio doesnt serve static files for prom helm chart

12/16/2019

My team is using istio (version 1.2.8) on our k8s (v 1.15.6) landscape, and we want to provide Prometheus with external IP, when applying the vs We were able to access Prometheus in the browser but without and css/js files. we got 404 error for the .js (see logs below) files (see envoy logs below) as the /static files are not served.

This is the UI we got (no css and js files are served)

image

instead of the following Prometheus default UI (when using loadbalancer or port forwarding...)

image

This is the minimal steps to see the issue:

Install Prometheus via helm as-is(latest- we didn't change any default config of Prometheus from the chart )

Take the name of the service (with kubectl get svc on the ns which the service deployed) put it on the destination->host section in the VS (update the gw host etc) and apply the VS file

vs.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
    - match:
        - uri:
            prefix: /prometheus
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80

BTW,

If I just change the type of Prometheus to use LoadBalancer it work, I was able to get external-ip and see istio UI as expected.

another info, if I remove the following

      rewrite:
        uri: /graph

I got 404 error in the browser without any data from prom

in the browser without js/css files , the network in the browser is like following:

image

I even try the following which doesn't work either

        - uri:
            prefix: /prometheus
      rewrite:
        uri: /static

or

        - uri:
            prefix: /prometheus/static

our gateway spec looks like following

...
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - lzs.dev10.int.str.cloud.rpn
    port:
      name: https-manager
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/de-tls/tls.key
      serverCertificate: /etc/istio/de-tls/tls.crt

using port forwarding (local) or loadbalancer for Prometheus it works. How can we make it work with istio ?

update

I've tried also to add the static and got the same results:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
    - match:
        - uri:
            prefix: /prometheus
        - uri:
            prefix: /static
        - uri:
            regex: '^.*\.(ico|png|jpg)
#x27;
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80

update 2

after using the yaml which provided as answer, now I see the ui with the css etc however it's not functional , I got error: Error loading available metrics! in the browser debug mode network tab I can see that the following is not working

This is the logs for envoy for the error

[2019-12-17T09:04:18.670Z] "GET /api/v1/query?query=time()&_=1576573457737 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "57592874-27f5-4b57-9dea-1bcf13365f60" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.13:443 100.96.0.1:24664 lzs.dev10.int.str.cloud.rpn
[2019-12-17T09:04:18.670Z] "GET /api/v1/label/__name__/values?_=1576573457738 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "edad441d-58fe-4214-aae0-a0aec9012030" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.13:443 100.96.0.1:24664 lzs.dev10.int.str.cloud.rpn

enter image description here

(we are not talking about Prometheus which comes with istio, we need to install diff Prometheus on diff namespace...)

This is the logs from envoy

2019-12-15T13:57:16.977357Z info Envoy proxy is ready

[2019-12-15 14:29:51.226][14][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, [2019-12-15 15:00:50.980][14][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, [2019-12-15T15:11:02.572Z] "GET /prometheus HTTP/2" 200 - "-" "-" 0 5785 2 1 "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "531e2f39-0c9f-44d3-b11b-e336126ea836" "lzs.dev10.int.str.cloud.rpn" "100.96.0.16:9090" outbound|80||prom-prometheus-server.mon.svc.cluster.local - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn [2019-12-15T15:11:02.705Z] "GET /static/vendor/js/jquery-3.3.1.min.js?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "40119d8d-2103-4453-b589-e1561d44d363" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn [2019-12-15T15:11:02.705Z] "GET /static/vendor/js/popper.min.js?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "dbdf2a2a-cfd3-422a-82f4-e6e466407671" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn [2019-12-15T15:11:02.706Z] "GET /static/vendor/bootstrap-4.3.1/js/bootstrap.min.js?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "efd95571-03e9-492d-98ff-b4910d1646d6" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn [2019-12-15T15:11:02.706Z] "GET /static/vendor/bootstrap-4.3.1/css/bootstrap.min.css?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "8ae04546-06cb-4ba0-8430-f04388811460" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn [2019-12-15T15:11:02.706Z] "GET /static/css/prometheus.css?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "70c88f5f-b582-4dd9-a2e2-47605c812344" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn

-- Nina S
istio
kubernetes
prometheus

Similar Questions

Best Practice For Getting Credentials in to Spring boot app from gitlab
Can a Pod be managed by two different ReplicaSets?
K8s service unreachable via master IP
Why is a pod in my GKE cluster receiving repeated requests to the root path of web server?
Accessing Istio enabled K8S pod from a service outside of the mesh
Error: `pod has unbound immediate PersistentVolumeClaims` while deploying microservices on local machine
How do I use kubernetes volumeMounts: mountPath and hostPath in helm charts?
Kubernetes client (certificate) exception is thrown when spring-cloud-starter-kubernetes-config added as a dependency
run multi-threading processes in aws instances
How to deploy a .NET core console app to Azure Kubernetes Service?
Openstack Magnum kube_master: Went to status ERROR due to "Message: Exceeded maximum number of retries

1 Answer

12/16/2019

For start I found some information about istio 1.2 that tells

Istio 1.2 has been tested with these Kubernetes releases: 1.12, 1.13, 1.14.

So if You use kubernetes 1.15 I would recommend to upgrade your istio to latest version.

About prometheus, I see You want to use prometheus helm chart, why won't You use built-in prometheus? As provided in istio documentation prometheus is enabled in versions default,demo and sds.

Based on istio remotely accessign telemetry addons You can use either secure(https) or insecure(http) option to expose prometheus.

Personally i did an insecure reproduction by following above tutorial and everything is working.

Kubernetes Version: 1.13.11-gke.14

Istio Version: 1.4.2

Steps to follow

1.Install

2.Expose prometheus

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: prometheus-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15030
      name: http-prom
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prometheus-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - prometheus-gateway
  http:
  - match:
    - port: 15030
    route:
    - destination:
        host: prometheus
        port:
          number: 9090
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: prometheus
  namespace: istio-system
spec:
  host: prometheus
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

3.Result

enter image description here

EDIT

Could You try use this yaml?

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
  - match:
        - uri:
            prefix: /prometheus
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80
  - match:
        - uri:
            prefix: /static
        - uri:
            regex: '^.*\.(ico|png|jpg)
#x27;
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80

EDIT2 Please add /api prefix to your second match like below

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
  - match:
        - uri:
            prefix: /prometheus
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80
  - match:
        - uri:
            prefix: /static
        - uri:
            regex: '^.*\.(ico|png|jpg)
#x27;
        - uri:
            prefix: /api
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80

EDIT3

In your answer you separate it to two matches , why?

This link is the answer here I think, You rewrite /prometheus to /graph since it's main Prometheus url, and that's okay. But you can't rewrite /static, /api to /graph because you need those paths to grab files and metrics, if it won't match then error 404 appears.

-- jt97
Source: StackOverflow