We have a Prometheus running in our cluster and we are able to use grafana to watch our cluster / pods metrics, now I want to add some custom metrics , is there a way to do it ? if so how should I connect the code to Prometheus , I mean if i write golang program using Prometheus API , and deploy it as docker to k8s , now does the program know to connect with Prometheus ? e.g. this program is exposing data to the /metrics endpoint but what else should I do to make prom to be able to read this data ?
You probably want Prometheus to discover the targets to scrape from automatically, if so you should configure K8s service discovery in Prometheus to discover pods, nodes and services from your cluster (maybe you already did something like this since you are already monitoring k8s metrics).
To get your go application monitored you can for example add annotations to your pods or to your services to enable scraping from this targets and define where the metrics are available (path, port). However this depends on your scrape configuration and relabeling. Good example can be found here
If you are using Prometheus Operator you need to define a servicemonitor resource instead.