Since Prometheus scrapes metrics at a regular interval (30 seconds or so), and some kubernetes pods only live a few seconds, can we depend on the metric kube_pod_created to actually show a value for each pod that existed in the system?
My worry is that it only sees pods that existed during the scrape. But I'm not sure of the exporter implementation to be sure.
For such cases, Prometheus offers pushgateway which is a metrics cache. Any short-lived jobs can push their metrics to the pushgateway, from where they can be scraped by Prometheus server at the next scraping interval. Client libraries are available for a few platforms such as java, go, etc. so relevant jobs can easily push their metrics to pushgateway. The Prometheus text protocol however makes it so simple to push metrics that a script can even use a command-line HTTP tool like curl to send metrics.
Take a look: kubernetes-prometheus-monitoring.