I am writing an exporter and am having trouble with metrics for my collector. From what I can tell, metrics in Prometheus need to be defined beforehand. Is there any way to define them dynamically at runtime instead? I won't know how many metrics I'll have or what metrics I'll need until the code is running. (For example, if k8s finds all volumes connected to the cluster, and I need EACH volume to get its own total_capacity/available_capacity metric).
I have only used Python and Java Prometheus clients so far, but isn't it just a recommendation to define metrics beforehand? In Java I can create the metrics at run-time inside methods or wherever I want. I just have to make sure that the same metric is not created again or an exception will be thrown by the library.
You would handle this by using dynamic label values rather than dynamic metric names. That said, you can call prometheus.Register() at runtime just fine, client_golang won't know the difference.