I'm currently exploring more and more of Google Cloud Platform and try to evaluate how I can change my default stack. I like the idea of using all the built-in tooling instead of having to go to various Saas products. One of the more open questions has been how to make use of Stackdriver Monitoring. I understand that the built-in monitoring uses Heapster with the Google Cloud Sink. However, how do I expose my own custom metrics to Stackdriver for use? I don't mean the logs-based metrics, but the real thing?
I know I could always install the agent myself onto the cluster, but it appears to me that as part of Google Container Engine, it is already running, so how could I push metrics to it?
To answer my own question, the best strategy for now seems to be using https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd
The fluentd pod pre-installed in GKE is in kube-system namespace and managed by k8s, so you shouldn't mess with it. Currently it seems only proxying stdout/stderr to Stackdriver. But according to this post, k8s team is working on providing accessible node-wide fluentd port out of the box.
This document has details about setting up custom logging.
I think GCP already give us related docs, see Using Custom Metrics