TL;DR - it's ok

Assuming your context, I suppose, it's ok to shutdown ptometheus component of your cluster. Except cases when reports, alerts and monitoring - are critical parts of your system.

Let dive in the sources of GCP

As per source code at GoogleCloudPlatform:

prometheus-to-sd is a simple component that can scrape metrics stored in prometheus text format from one or multiple components and push them to the Stackdriver. Main requirement: k8s cluster should run on GCE or GKE.

Prometheus

From their Prometheus Github Page:

The Prometheus monitoring system and time series database.

To get a picture what is it for - you can read awesome guide on Prometheus: Prometheus Monitoring : The Definitive Guide in 2019 – devconnected

Also, there are hundreds of videos on their Youtube channel Prometheus Monitoring

Your questions

So, answering to your questions:

Can I delete this daemonset without issue?

It depends. As I said, you can. Except cases when reports, alerts and monitoring - are critical parts of your system.

What is it being used for

It's a TSDB for monitoring

what functionality would I be loosing without it?

• metrics
• → therefore dashboards
• → therefore alerting

TL;DR

Even if you delete it, it will be back.

A little bit more explanation

Citing explanation by user @Yasen what prometheus-to-sd is:

prometheus-to-sd is a simple component that can scrape metrics stored in prometheus text format from one or multiple components and push them to the Stackdriver. Main requirement: k8s cluster should run on GCE or GKE.

Github.com: Prometheus-to-sd

Assuming that the command deleting this daemonset will be:

$ kubectl delete daemonset prometheus-to-sd --namespace=kube-system

Executing this command will indeed delete the daemonset but it will be back after a while.

prometheus-to-sd daemonset is managed by Addon-Manager which will recreate deleted daemonset back to original state.

Below is the part of the prometheus-to-sd daemonset YAML definition which states that this daemonset is managed by addonmanager:

  labels:
    addonmanager.kubernetes.io/mode: Reconcile

You can read more about it by following: Github.com: Kubernetes: addon-manager

Deleting this daemonset is strictly connected to the monitoring/logging solution you are using with your GKE cluster. There are 2 options:

• Stackdriver logging/monitoring
• Legacy logging/monitoring

Stackdriver logging/monitoring

You need to completely disable logging and monitoring of your GKE cluster to delete this daemonset.

You can do it by following a path:

GCP -> Kubernetes Engine -> Cluster -> Edit -> Kubernetes Engine Monitoring -> Set to disabled.

Legacy logging/monitoring

If you are using a legacy solution which is available to GKE version 1.14, you need to disable the option of Legacy Stackdriver Monitoring by following the same path as above.

Let me know if you have any questions in that.