We are running Kubernetes 1.9.1. We are using Kubernetes client python based library script to connect to Kubernetes server and generating some information related to pods like list of pods which r in terminating state currently. We want to send this data to prometheus server as metric and want to raise an alert in prometheus. Do I need to create custom metric of prometheus to achieve this? Alert for prometheus will be created in my python script using prometheus client?
According to prometheus documentation:
Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.
So - you don't send data to prometheus. You have to expose those data in format readable by prometheus and configure prometheus to scrap it. If it's not possible - you can push data to prometheus push gateway. If you use python client you can use this code as an example.
You have to define alert rules as part of prometheus configuration. Please note - alerts are not sent by prometheus, but by the alertmanager which is a separate process.