Kubernetes - Metrics not showing after reboot

10/8/2019

I have an on prem K8s cluster, using 2 pieces of hardware - a master on each.

kubectl get nodes NAME STATUS ROLES AGE VERSION master1 Ready master 52d v1.15.3 master2 Ready master 52d v1.15.3 node2 Ready <none> 52d v1.15.3 node3 Ready <none> 52d v1.15.3 node4 Ready <none> 52d v1.15.3

After a reboot of a master node to upgrade memory - the metrics are no longer showing, they did before the reboot.

kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master2 251m 13% 1548Mi 66% node2 234m 12% 1875Mi 41% node3 137m 7% 1721Mi 38% node4 588m 30% 2661Mi 48% master1 <unknown> <unknown> <unknown> <unknown>

kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-7b9f9648d8-8525s 1/1 Running 0 48d calico-node-66n4f 1/1 Running 2 48d calico-node-cml5f 1/1 Running 2 48d calico-node-hdf7c 1/1 Running 0 48d calico-node-pr2zc 1/1 Running 3 48d calico-node-wbldc 1/1 Running 6 48d coredns-74c9d4d795-8jkcg 1/1 Running 0 48d coredns-74c9d4d795-psnqd 1/1 Running 0 19h dns-autoscaler-7d95989447-mlpxt 1/1 Running 0 19h heapster-55c5f89fbd-xb2n7 2/2 Running 2 4d3h kube-apiserver-master1 1/1 Running 2 48d kube-apiserver-master2 1/1 Running 0 48d kube-controller-manager-master1 1/1 Running 2 48d kube-controller-manager-master2 1/1 Running 0 48d kube-proxy-hngdr 1/1 Running 6 48d kube-proxy-mk9pn 1/1 Running 0 48d kube-proxy-t4954 1/1 Running 2 48d kube-proxy-wjh56 1/1 Running 3 48d kube-proxy-ww56g 1/1 Running 2 48d kube-scheduler-master1 1/1 Running 2 48d kube-scheduler-master2 1/1 Running 0 48d kubernetes-dashboard-7d75c474bb-bg5s4 1/1 Running 0 4d3h metrics-server-5889f5b847-dl6zl 1/1 Running 2 13d nginx-proxy-node2 1/1 Running 9 52d nginx-proxy-node3 1/1 Running 6 52d nginx-proxy-node4 1/1 Running 7 52d nodelocaldns-87vdg 1/1 Running 0 48d nodelocaldns-f2d2c 1/1 Running 6 48d nodelocaldns-p8vmj 1/1 Running 2 48d nodelocaldns-sn8gk 1/1 Running 3 48d nodelocaldns-vd62b 1/1 Running 2 48d tiller-deploy-767d9b9584-7ljjb 1/1 Running 4 4d3h

All services running ok.

Any ideas?

`kubectl describe node master1
`Name:               master1
`Roles:              master
`Labels:             beta.kubernetes.io/arch=amd64
`                    beta.kubernetes.io/os=linux
`                    kubernetes.io/arch=amd64
`                    kubernetes.io/hostname=master1
`                    kubernetes.io/os=linux
`                    node-role.kubernetes.io/master=
`Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: `/var/run/dockershim.sock
`                    node.alpha.kubernetes.io/ttl: 0
`                    volumes.kubernetes.io/controller-managed-attach-detach: `true
`CreationTimestamp:  Fri, 16 Aug 2019 14:26:15 +0000
`Taints:             node-role.kubernetes.io/master:NoSchedule
`Unschedulable:      false
`Conditions:
`  Type                 Status  LastHeartbeatTime                 `LastTransitionTime                Reason                       Message
`  ----                 ------  -----------------                 ------------`------                ------                       -------
`  NetworkUnavailable   False   Tue, 08 Oct 2019 08:24:42 +0000   Tue, 08 Oct `2019 08:24:42 +0000   CalicoIsUp                   Calico is running on this `node
`  MemoryPressure       False   Wed, 09 Oct 2019 14:54:13 +0000   Tue, 08 Oct `2019 08:24:35 +0000   KubeletHasSufficientMemory   kubelet has sufficient `memory available
`  DiskPressure         False   Wed, 09 Oct 2019 14:54:13 +0000   Tue, 08 Oct `2019 08:24:35 +0000   KubeletHasNoDiskPressure     kubelet has no disk `pressure
`  PIDPressure          False   Wed, 09 Oct 2019 14:54:13 +0000   Tue, 08 Oct `2019 08:24:35 +0000   KubeletHasSufficientPID      kubelet has sufficient PID `available
`  Ready                True    Wed, 09 Oct 2019 14:54:13 +0000   Tue, 08 Oct `2019 08:24:35 +0000   KubeletReady                 kubelet is posting ready `status. AppArmor enabled
`Addresses:
`  InternalIP:  10.200.40.23
`  Hostname:    master1
`Capacity:
` cpu:                2
` ephemeral-storage:  41019616Ki
` hugepages-2Mi:      0
` memory:             4948784Ki
` pods:               110
`Allocatable:
` cpu:                1800m
` ephemeral-storage:  37803678044
` hugepages-2Mi:      0
` memory:             4346384Ki
` pods:               110
`System Info:
` Machine ID:                 0994b13409574735a1682c5fd60c9c2c
` System UUID:                0994B134-0957-4735-A168-2C5FD60C9C2C
` Boot ID:                    3d886b98-ba60-4fa2-b1b1-3a603fcb97bb
` Kernel Version:             4.15.0-65-generic
` OS Image:                   Ubuntu 18.04.2 LTS
` Operating System:           linux
` Architecture:               amd64
` Container Runtime Version:  docker://18.9.7
` Kubelet Version:            v1.15.3
` Kube-Proxy Version:         v1.15.3
`PodCIDR:                     10.233.64.0/24
`Non-terminated Pods:         (6 in total)
`  Namespace                  Name                               CPU Requests  `CPU Limits  Memory Requests  Memory Limits  AGE
`  ---------                  ----                               ------------  `----------  ---------------  -------------  ---
`  kube-system                calico-node-cml5f                  150m (8%)     `300m (16%)  64M (1%)         500M (11%)     49d
`  kube-system                kube-apiserver-master1             250m (13%)    `0 (0%)      0 (0%)           0 (0%)         49d
`  kube-system                kube-controller-manager-master1    200m (11%)    `0 (0%)      0 (0%)           0 (0%)         49d
`  kube-system                kube-proxy-t4954                   0 (0%)        `0 (0%)      0 (0%)           0 (0%)         49d
`  kube-system                kube-scheduler-master1             100m (5%)     `0 (0%)      0 (0%)           0 (0%)         49d
`  kube-system                nodelocaldns-p8vmj                 100m (5%)     `0 (0%)      70Mi (1%)        170Mi (4%)     49d
`Allocated resources:
`  (Total limits may be over 100 percent, i.e., overcommitted.)
`  Resource           Requests        Limits
`  --------           --------        ------
`  cpu                800m (44%)      300m (16%)
`  memory             137400320 (3%)  678257920 (15%)
`  ephemeral-storage  0 (0%)          0 (0%)``
-- jimmcl400
kubernetes
metrics

0 Answers