nginx-prometheus-exporter deployment failing

5/14/2019

I am trying to deploy Nginx with nginx-prometheus-exporter. Following is my deployment

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: local
  name: nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Always
        ports:
        - name: nginx-port
          containerPort: 80
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /
            port: nginx-port
            httpHeaders:
              - name: Host
                value: KubernetesLivenessProbe
          initialDelaySeconds: 10
          timeoutSeconds: 1
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /
            port: nginx-port
            httpHeaders:
              - name: Host
                value: KubernetesLivenessProbe
          initialDelaySeconds: 3
          timeoutSeconds: 1
          periodSeconds: 8
        volumeMounts:
        - mountPath: /etc/nginx/conf.d/ # mount nginx-conf volume to /etc/nginx
          readOnly: true
          name: frontend-conf
      - name: nginx-exporter
        image: nginx/nginx-prometheus-exporter:0.3.0
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - name: nginx-ex-port
          containerPort: 9113
        args:
          - -nginx.scrape-uri=http://nginx.local.svc.cluster.local:80/stub_status
      volumes:
      - name: frontend-conf
        configMap:
          name: frontend-conf # place ConfigMap `nginx-conf` on /etc/nginx
          items:
            - key: frontend.local.conf
              path: frontend.local.conf 

Single container pod works, but fails when I add nginx-prometheus-exporter sidecar container.

mackbook: xyz$ kubectl logs nginx-6dbbdb64fb-8drmc -c nginx-exporter
2019/05/14 20:17:48 Starting NGINX Prometheus Exporter Version=0.3.0 GitCommit=6570275
2019/05/14 20:17:53 Could not create Nginx Client: Failed to create NginxClient: failed to get http://nginx.local.svc.cluster.local:80/stub_status: Get http://nginx.local.svc.cluster.local:80/stub_status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Also was able to curl http://nginx.local.svc.cluster.local from another container running in different namespace.

Anyone know what's the correct way to specify -nginx.scrape-uri ?

From Nginx container

root@nginx-6fd866c4d7-t8wwf:/# curl localhost/stub_status
<html>
<head><title>404 Not Found</title></head>
<body>
<h1>404 Not Found</h1>
<ul>
<li>Code: NoSuchKey</li>
<li>Message: The specified key does not exist.</li>
<li>Key: beta/stub_status</li>
<li>RequestId: 3319E5A424C7BA31</li>
<li>HostId: zVv5zuAyzqbB2Gw3w1DavEJwgq2sbgI8RPMf7RmPsNQoh/9ftxCmiSwyj/j6q/K3lxEEeUy6aZM=</li>
</ul>
<hr/>
</body>
</html>
root@nginx-6fd866c4d7-t8wwf:/#
-- roy
kubernetes
nginx
prometheus
prometheus-blackbox-exporter
prometheus-node-exporter

1 Answer

5/14/2019

nginx container is listening on port 80, and you configure the sidecar to access to 8080 port. Moreover, you deploy this in "production" namespace, but using "test" to access to nginx.

Try with -nginx.scrape-uri=http://nginx.production.svc.cluster.local:80/stub_status.

BTW you must know that containers within a Pod share an IP address and port space, and can find each other via localhost.

So you can use -nginx.scrape-uri=http://localhost/stub_status.

-- Alexandre Cartapanis
Source: StackOverflow