K
Q

Metrics endpoint not working when deploying a Ballerina based micro service in Kubernetes

10/24/2019

I have a micro-service implemented in Ballerina. Whenever I deploy the service using following command, in my local machine ballerina run --observe [name_of_the_file].bal, I get metrics endpoint mapped to port 9797. That is curl localhost:9797/metrics gives server metrics as a response.

Even when I deployed the above service in Docker environment I got the same results. But when I deployed it in Kubernetes (On GoogleCloudPlatform), I don't get the metrics mapped to the port 9797.

kubectl logs [pod_name] gives the following output.

Initiating service(s) in 'ballerina-prime.balx'
[ballerina/http] started HTTP/WS endpoint 0.0.0.0:8688

Can someone help me get the metrics mapped to port 9797?

Dockerfile

FROM ballerina/ballerina:0.991.0
COPY ballerina-prime.balx /home/ballerina
EXPOSE 8688 
EXPOSE 9797
CMD ballerina run --observe ballerina-prime.balx

ballerina-prime.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ballerina-prime
  labels:
    app: ballerina-prime
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ballerina-prime
  template:
    metadata:
      labels:
        app: ballerina-prime
    spec:
      containers:
      - name: ballerina-prime
        image: anushiya/ballerina-prime:v3
        resources:
            limits:
                cpu: "100m"
            requests:
                cpu: "100m"
        ports:
        - containerPort: 8688
          name: echo-service
        - containerPort: 9797
          name: metrics
-- anushiya-thevapalan
ballerina
google-kubernetes-engine

Similar Questions

upstream checks crash system
Does ClusterIssuer has cache behavior in multiple namespaces environment?
How kubernetes request and limit works in pressure?
Docker image not pulling latest from dockerhub.com registry
Ingress not showing up in namespace

2 Answers

10/24/2019

As a test, I ran the curl request within a Kubernetes node. First, find the podIP by using the following command:

kubectl get pods -l app=ballerina-prime -o yaml | grep podIP

app= [nameOfThePod]. You can use Cloud shell to get the Pod IP.

-Next steps:

Since you're using GKE (Kubernetes) on Google Cloud Platform, navigate to Compute Engine and SSH to one of the GKE nodes. Once inside, run the curl request using the podIP IP @gke-deploymentbuild-default-pool-f3c9c9aa-fkx5 ~ $ curl -i http://10.60.2.10:9797/metrics

HTTP/1.1 200 OK
content-type: text/plain
content-length: 1060
server: ballerina/0.991.0
date: Thu, 24 Oct 2019 18:19:29 GMT

-External:

Obviously, you wouldn't want to access this from a GKE Node, this is where Load Balancers come into play.

Navigate to GKE, and in workloads, select the ballerina workload. Once in the menu, select “Actions” and “Expose”. Here you can set the port number and select “load balancer” as the serviceType:

Menu>Kubernetes Engine>Workloads>”ballerina”>Actions>Expose>Service type: Load balancer.

Head over to services & ingress, and once the Load balancer is finish being created, you will see the endpoints exposed with the port number of your choice ( 9797 in this case.). The IP with the /metrics endpoint should work.

See links: 1 3

-- Cam
Source: StackOverflow
10/27/2019

Your container in pure docker environment will run the same in kubernetes. In fact, GKE runs on top of docker. If you exec into the pod, you should be able to reach the metrics on the same port:

kubectl exec [pod_name] -- curl localhost:9797/metrics

Considering you are using the same image and your pod.spec had port 9797 exposed, I don't see why you should have avril issues, despite the loss showing a binding on port 8688

-- Patrick W
Source: StackOverflow