K
Q

CoreDNS logs report unauthorized

3/23/2021

We deployed new Kubernetes cluster, and it has 2 pods for Coredns.

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                      READY   STATUS    RESTARTS   AGE
coredns-74ff55c5b-7v9bd   0/1     Running   0          7h22m
coredns-74ff55c5b-tfpqb   0/1     Running   0          7h23m

There suppose to be 2 replicas, but 0 READY.

When I check the logs to find the reason for not running, I see there are many Unauthorized errors.

$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --tail 20 --namespace=kube-system $p; done
E0323 00:58:04.393710       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:34.184217       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:51.873269       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:00.966217       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:23.151006       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:47.362409       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
E0323 00:59:48.563791       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
E0323 00:59:56.278764       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:07.504557       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:24.948534       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:33.605013       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:56.471477       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:20.436808       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
E0323 00:59:21.200346       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:29.597663       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"

When I try to find some help on net, I found out, it use coredns service user. I check for all that roles and bindings.

SERVICE ACCOUNT

$ kubectl get sa coredns -n kube-system -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2021-03-03T15:17:38Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "297"
  uid: 13633498-2e6b-4ac4-bb34-f2d5c9e4d262
secrets:
- name: coredns-token-sg7p9

TOKEN SECRET

$ kubectl get secret coredns-token-sg7p9 -n kube-system
NAME                  TYPE                                  DATA   AGE
coredns-token-sg7p9   kubernetes.io/service-account-token   3      19d

CLUSTER ROLE

$ kubectl get clusterrole system:coredns -n kube-system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2021-03-03T15:17:38Z"
  managedFields:
  - apiVersion: rbac.authorization.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:rules: {}
    manager: kubeadm
    operation: Update
    time: "2021-03-03T15:17:38Z"
  name: system:coredns
  resourceVersion: "292"
  uid: 35adc9a3-7415-4498-81b2-a4eab50882b1
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get

CLUSTER ROLE BINDINGS

$ kubectl get clusterrolebindings system:coredns -n kube-system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2021-03-03T15:17:38Z"
  managedFields:
  - apiVersion: rbac.authorization.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:roleRef:
        f:apiGroup: {}
        f:kind: {}
        f:name: {}
      f:subjects: {}
    manager: kubeadm
    operation: Update
    time: "2021-03-03T15:17:38Z"
  name: system:coredns
  resourceVersion: "293"
  uid: 2d47c2cb-6641-4a62-b867-8a598ac3923a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system

From unauthorized error, I can predict it might be related to token, like token expired and not renewed. I was trying to find help on net, for how to renew token for Coredns, but didn't find any help.

I might be doing something wrong, but can't find that.

There is help available if Pod is not in Running state, but not for unauthorized after running pod.

-- Nilesh
coredns
kube-apiserver
kubernetes
unauthorized

Similar Questions

MongoDB stopped being accessible when started using Kompose on Minikube
Can I use env in preStop command
Configmaps - Angular
Kubernetes Cluster - Using L3 routing - Cannot ping pods across network
Having trouble with KOPS and Traefik ingress controller installation

0 Answers