Google Kubernetes Engine Stackdriver logging/monitoring is gone at gke version 1.15

4/7/2020

I'm using GKE for more than year and i never had any problems with stackdriver logging/monitoring. But when i created new cluster with version 1.15.9-gke.26 i don't see any logs in stackdriver (neither metrics). It also didn't work with new cluster with version 1.14. Although it works for older cluster which was updated to version 1.14 from 1.13.

Some settings:
gke version = 1.15.9-gke.26
Stackdriver Kubernetes Engine Monitoring = System and workload logging and monitoring
VPC-native (alias IP) = Enabled
Workload Identity = Disabled

Weird things:
Following daemon sets have 0/0 pods (DaemonSet has no nodes selected):
- metadata-proxy-v0.1
- nvidia-gpu-device-plugin (doesn't sound useful)

I'm not sure how exactly stackdriver works and how to debug it... I will appreciate any tips

Deployments and daemonsets currently running at cluster:

kubectl get daemonsets,deployments --all-namespaces 
NAMESPACE     NAME                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                              AGE
kube-system   daemonset.extensions/fluentd-gcp-v3.1.1         3         3         3       3            3           beta.kubernetes.io/fluentd-ds-ready=true,beta.kubernetes.io/os=linux       16h
kube-system   daemonset.extensions/metadata-proxy-v0.1        0         0         0       0            0           beta.kubernetes.io/metadata-proxy-ready=true,beta.kubernetes.io/os=linux   16h
kube-system   daemonset.extensions/nvidia-gpu-device-plugin   0         0         0       0            0           <none>                                                                     16h
kube-system   daemonset.extensions/prometheus-to-sd           3         3         3       3            3           beta.kubernetes.io/os=linux                                                16h

NAMESPACE     NAME                                                             READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.extensions/event-exporter-v0.3.0                      1/1     1            1           16h
kube-system   deployment.extensions/fluentd-gcp-scaler                         1/1     1            1           16h
kube-system   deployment.extensions/heapster-gke                               1/1     1            1           16h
kube-system   deployment.extensions/kube-dns                                   2/2     2            2           16h
kube-system   deployment.extensions/kube-dns-autoscaler                        1/1     1            1           16h
kube-system   deployment.extensions/l7-default-backend                         1/1     1            1           16h
kube-system   deployment.extensions/metrics-server-v0.3.3                      1/1     1            1           16h
kube-system   deployment.extensions/stackdriver-metadata-agent-cluster-level   1/1     1            1           16h
-- DawPawel
google-cloud-platform
google-kubernetes-engine
kubernetes
stackdriver

1 Answer

4/13/2020

per documentation, and as @Darshan Naik mentioned:

If you are using Legacy Logging and Monitoring, then you must switch to Kubernetes Engine Monitoring before support for Legacy Logging and Monitoring is removed. Legacy Logging and Monitoring will no longer be supported as of GKE 1.15.

https://cloud.google.com/monitoring/kubernetes-engine#select

-- c69
Source: StackOverflow