I am new to Prometheus and relatively new to kubernetes so bear with me, please. I am trying to test Prometheus out and have tried two different approaches.
Run Prometheus as a docker container outside of kubernetes. To accomplish this I have created this Dockerfile:
FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/and this yaml file:
global:
scrape_interval: 15s
external_labels:
monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'kubernetes-apiservers'
scheme: http
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: endpoints
api_server: localhost:443When I run this I get:
Failed to list *v1.Pod: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
Failed to list *v1.Service: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
Failed to list *v1.Endpoints: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"on a loop. Prometheus will load when I go to localhost:9090 but there is no data.
I thought deploying Prometheus as a Kubernetes deployment may help, so I made this yaml and deployed it.
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: prometheus-monitor
spec:
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
containers:
- name: prometheus-monitor
image: prom/prometheus
# args:
# - '-config.file=/etc/prometheus/prometheus.yaml'
imagePullPolicy: IfNotPresent
ports:
- name: webui
containerPort: 9090The deployment was successful, but if I go to localhost:9090 I get 'ERR_SOCKET_NOT_CONNECTED'. (my port is forwarded)
Can anyone tell me the advantage of in vs out of Kubernetes and how to fix at least one of these issues?
Also, my config file is suppressed because it was giving an error, and I will look into that once I am able to get Prometheus loaded.
Kubernetes does not map the port outside it's cluster when you deploy your container.
You also have to create a service (can be inside the same file) to make it available from your workstation (append this to your prometheus yaml):
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-web
labels:
app: prometheus
spec:
type: NodePort
ports:
- port: 9090
protocol: TCP
targetPort: 9090
nodePort: 30090
name: webui
selector:
app: prometheusNodePort opens the given port on all nodes you have. You should be able to see the frontend with http://localhost:30090/
Per default, kubernetes allow ports 30000 to 32767 for NodePort type (https://kubernetes.io/docs/concepts/services-networking/service/#nodeport).
Please consider reading the documentation in general for more information on services in kubernetes: https://kubernetes.io/docs/concepts/services-networking/service/
So 2 different issues. On:
You are trying to connect to localhost:443 where Prometheus is running and it's expecting to talk to a Kubernetes API server. Apparently, nothing is listening on localhost:443. Are you doing port forwarding to your kube-apiserver?
In this case you need to expose your deployment port. With something like:
kubectl expose deployment prmetheus-web --type=LoadBalancer # or
kubectl expose deployment prmetheus-web --type=NodePortdepending on how you want to expose your service. NodePort exposes it in service that maps to a port on your Kubernetes nodes (IPAddress:Port) and LoadBalancer exposes the deployment using an external load balancer that may vary depending on what cloud you are using (AWS, GCP, OpenStack, Azure, etc). More about exposing your Deployments or DaemonSets or StatefulSets here. More about services here
Hope it helps.