K
Q

Question

kubedns can not start with "Failed to list v1.Endpoints: Unauthorized" and "Failed to list v1.Service: Unauthorized"

9/29/2018

I have a problem when I install the kube-dns add on. My OS is CentOS Linux release 7.0.1406 (Core)

Kernel:Linux master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

My api-server config:

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
#KUBE_API_ADDRESS="--insecure-bind-address=177.1.1.40"

# The port on the local server to listen on.
KUBE_API_PORT="--secure-port=443"

# Port minions listen on
KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://master:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=AlwaysAdmit,NamespaceLifecycle,LimitRanger,SecurityContextDeny,ResourceQuota,ServiceAccount"

# Add your own!
KUBE_API_ARGS="--client-ca-file=/etc/kubernetes/k8s-certs/CA/rootCA.crt --tls-private-key-file=/etc/kubernetes/k8s-certs/master/api-server.pem --tls-cert-file=/etc/kubernetes/k8s-certs/master/api-server.crt"

The api-server authorization-mode is set to AlwaysAllow

Sep 29 17:31:22 master kube-apiserver: I0929 17:31:22.952730    1311 flags.go:27] FLAG: --authorization-mode="AlwaysAllow"

My kube-dns config YAML file is :

# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.254.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
---
#apiVersion: rbac.authorization.k8s.io/v1
#kind: RoleBinding
#metadata:
#  name: kube-dns
#  namespace: kube-system
#roleRef:
#  apiGroup: rbac.authorization.k8s.io
#  kind: ClusterRole
#  name: Cluster-admin
#subjects:
#- kind: ServiceAccount
#  name: default
#  namespace: kube-system
#---
#apiVersion: v1
#kind: ServiceAccount
#metadata:
#  name: kube-dns
#  namespace: kube-system
#  labels:
#    kubernetes.io/cluster-service: "true"
#    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubecfg-file
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  kubecfg-file: |
    apiVersion: v1
    kind: Config
    clusters:
      - cluster:
          certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          server: https://177.1.1.40:443
        name: kube-test
    users:
    - name: kube-admin
      user:
        token: /var/run/secrets/kubernetes.io/serviceaccount/token
    contexts:
    - context:
        cluster: kube-test
        namespace: default
        user: kube-admin
      name: test-context
    current-context: test-context
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      - name: kube-kubecfg-file
        configMap:
          name: kubecfg-file
          optional: true
      containers:
      - name: kubedns
        image: 177.1.1.35/library/kube-dns:1.14.8
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=cluster.local.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --kubecfg-file=/kubecfg-file/kubecfg-file
        - --kube-master-url=https://10.254.0.1:443
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
        - name: kube-kubecfg-file
          mountPath: /kubecfg-file
      - name: dnsmasq
        image: 177.1.1.35/library/dnsmasq:1.14.8
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - -v=2
        - -logtostderr
        - -configDir=/etc/k8s/dns/dnsmasq-nanny
        - -restartDnsmasq=true
        - --
        - -k
        - --cache-size=1000
        - --no-negcache
        - --log-facility=-
        - --server=/cluster.local/127.0.0.1#10053
        - --server=/in-addr.arpa/127.0.0.1#10053
        - --server=/ip6.arpa/127.0.0.1#10053
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: 177.1.1.35/library/sidecar:1.14.8
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
      dnsPolicy: Default  # Don't use cluster DNS.
      #serviceAccountName: kube-dns

When I start the kube-dns the kubedns container log:

I0929 09:33:22.666182       1 dns.go:48] version: 1.14.8
I0929 09:33:22.668521       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0929 09:33:22.668586       1 server.go:119] FLAG: --alsologtostderr="false"
I0929 09:33:22.668604       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0929 09:33:22.668613       1 server.go:119] FLAG: --config-map=""
I0929 09:33:22.668619       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0929 09:33:22.668629       1 server.go:119] FLAG: --config-period="10s"
I0929 09:33:22.668637       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0929 09:33:22.668643       1 server.go:119] FLAG: --dns-port="10053"
I0929 09:33:22.668662       1 server.go:119] FLAG: --domain="cluster.local."
I0929 09:33:22.668671       1 server.go:119] FLAG: --federations=""
I0929 09:33:22.668683       1 server.go:119] FLAG: --healthz-port="8081"
I0929 09:33:22.668689       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0929 09:33:22.668695       1 server.go:119] FLAG: --kube-master-url="https://10.254.0.1:443"
I0929 09:33:22.668707       1 server.go:119] FLAG: --kubecfg-file="/kubecfg-file/kubecfg-file"
I0929 09:33:22.668714       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0929 09:33:22.668727       1 server.go:119] FLAG: --log-dir=""
I0929 09:33:22.668733       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0929 09:33:22.668739       1 server.go:119] FLAG: --logtostderr="true"
I0929 09:33:22.668744       1 server.go:119] FLAG: --nameservers=""
I0929 09:33:22.668754       1 server.go:119] FLAG: --stderrthreshold="2"
I0929 09:33:22.668760       1 server.go:119] FLAG: --v="2"
I0929 09:33:22.668765       1 server.go:119] FLAG: --version="false"
I0929 09:33:22.668774       1 server.go:119] FLAG: --vmodule=""
I0929 09:33:22.668831       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0929 09:33:22.669125       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0929 09:33:22.669170       1 dns.go:146] Starting endpointsController
I0929 09:33:22.669181       1 dns.go:149] Starting serviceController
I0929 09:33:22.669508       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0929 09:33:22.669523       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E0929 09:33:22.695489       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Unauthorized
E0929 09:33:22.696267       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Unauthorized
I0929 09:33:23.169540       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0929 09:33:23.670206       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

After few minutes the pod crash.

kubectl describe pod -n kube-system kube-dns-b7d556f59-h8xqp 
Name:           kube-dns-b7d556f59-h8xqp
Namespace:      kube-system
Node:           node3/177.1.1.43
Start Time:     Sat, 29 Sep 2018 17:50:17 +0800
Labels:         k8s-app=kube-dns
                pod-template-hash=638112915
Annotations:    scheduler.alpha.kubernetes.io/critical-pod=
Status:         Running
IP:             172.30.59.3
Controlled By:  ReplicaSet/kube-dns-b7d556f59
Containers:
  kubedns:
    Container ID:  docker://5d62497e0c966c08d4d8c56f7a52e2046fd05b57ec0daf34a7e3cd813e491f09
    Image:         177.1.1.35/library/kube-dns:1.14.8
    Image ID:      docker-pullable://177.1.1.35/library/kube-dns@sha256:6d8e0da4fb46e9ea2034a3f4cab0e095618a2ead78720c12e791342738e5f85d
    Ports:         10053/UDP, 10053/TCP, 10055/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      --domain=cluster.local.
      --dns-port=10053
      --config-dir=/kube-dns-config
      --kubecfg-file=/kubecfg-file/kubecfg-file
      --kube-master-url=https://10.254.0.1:443
      --v=2
    State:          Running
      Started:      Sat, 29 Sep 2018 17:50:20 +0800
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:      100m
      memory:   70Mi
    Liveness:   http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
    Environment:
      PROMETHEUS_PORT:  10055
    Mounts:
      /kube-dns-config from kube-dns-config (rw)
      /kubecfg-file from kube-kubecfg-file (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dsxql (ro)
  dnsmasq:
    Container ID:  docker://17ae73b52eb69c35a027cb5645a3801d649b262a8650862d64e7959a22c8e92e
    Image:         177.1.1.35/library/dnsmasq:1.14.8
    Image ID:      docker-pullable://177.1.1.35/library/dnsmasq@sha256:93c827f018cf3322f1ff2aa80324a0306048b0a69bc274e423071fb0d2d29d8b
    Ports:         53/UDP, 53/TCP
    Host Ports:    0/UDP, 0/TCP
    Args:
      -v=2
      -logtostderr
      -configDir=/etc/k8s/dns/dnsmasq-nanny
      -restartDnsmasq=true
      --
      -k
      --cache-size=1000
      --no-negcache
      --log-facility=-
      --server=/cluster.local/127.0.0.1#10053
      --server=/in-addr.arpa/127.0.0.1#10053
      --server=/ip6.arpa/127.0.0.1#10053
    State:          Running
      Started:      Sat, 29 Sep 2018 17:50:21 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        150m
      memory:     20Mi
    Liveness:     http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dsxql (ro)
  sidecar:
    Container ID:  docker://9449b13ff4e4ba1331d181bd6f309a34a4f3da1ce536c61af7a65664e3ad803a
    Image:         177.1.1.35/library/sidecar:1.14.8
    Image ID:      docker-pullable://177.1.1.35/library/sidecar@sha256:23df717980b4aa08d2da6c4cfa327f1b730d92ec9cf740959d2d5911830d82fb
    Port:          10054/TCP
    Host Port:     0/TCP
    Args:
      --v=2
      --logtostderr
      --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV
      --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
    State:          Running
      Started:      Sat, 29 Sep 2018 17:50:22 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     20Mi
    Liveness:     http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dsxql (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  kube-dns-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-dns
    Optional:  true
  kube-kubecfg-file:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubecfg-file
    Optional:  true
  default-token-dsxql:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dsxql
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age              From               Message
  ----     ------                 ----             ----               -------
  Normal   SuccessfulMountVolume  3m               kubelet, node3     MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume  3m               kubelet, node3     MountVolume.SetUp succeeded for volume "kube-kubecfg-file"
  Normal   SuccessfulMountVolume  3m               kubelet, node3     MountVolume.SetUp succeeded for volume "default-token-dsxql"
  Normal   Pulled                 3m               kubelet, node3     Container image "177.1.1.35/library/kube-dns:1.14.8" already present on machine
  Normal   Created                3m               kubelet, node3     Created container
  Normal   Started                3m               kubelet, node3     Started container
  Normal   Pulled                 3m               kubelet, node3     Container image "177.1.1.35/library/dnsmasq:1.14.8" already present on machine
  Normal   Created                3m               kubelet, node3     Created container
  Normal   Started                3m               kubelet, node3     Started container
  Normal   Pulled                 3m               kubelet, node3     Container image "177.1.1.35/library/sidecar:1.14.8" already present on machine
  Normal   Created                3m               kubelet, node3     Created container
  Normal   Started                3m               kubelet, node3     Started container
  Warning  Unhealthy              3m (x3 over 3m)  kubelet, node3     Readiness probe failed: Get http://172.30.59.3:8081/readiness: dial tcp 172.30.59.3:8081: getsockopt: connection refused
  Normal   Scheduled              43s              default-scheduler  Successfully assigned kube-dns-b7d556f59-h8xqp to node3

My kubernetes version is:

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"archive", BuildDate:"2018-03-29T08:38:42Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"archive", BuildDate:"2018-03-29T08:38:42Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

docker version:

docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64
 Experimental:    false

My kubernetes service config: api-server

/usr/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=http://master:2379 --secure-port=443 --kubelet-port=10250 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --enable-admission-plugins=AlwaysAdmit,NamespaceLifecycle,LimitRanger,SecurityContextDeny,ResourceQuota,ServiceAccount --client-ca-file=/etc/kubernetes/k8s-certs/CA/rootCA.crt --tls-private-key-file=/etc/kubernetes/k8s-certs/master/api-server.pem --tls-cert-file=/etc/kubernetes/k8s-certs/master/api-server.crt

controller-manager:

/usr/bin/kube-controller-manager --logtostderr=true --v=4 --master=https://master:443 --root-ca-file=/etc/kubernetes/k8s-certs/CA/rootCA.crt --service-account-private-key-file=/etc/kubrnetes/k8s-certs/master/api-server.pem --kubeconfig=/etc/kubernetes/cs_kubeconfig

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
#KUBE_API_ADDRESS="--insecure-bind-address=177.1.1.40"

# The port on the local server to listen on.
KUBE_API_PORT="--secure-port=443"

# Port minions listen on
KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://master:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=AlwaysAdmit,NamespaceLifecycle,LimitRanger,SecurityContextDeny,ResourceQuota,ServiceAccount"

# Add your own!
KUBE_API_ARGS="--client-ca-file=/etc/kubernetes/k8s-certs/CA/rootCA.crt --tls-private-key-file=/etc/kubernetes/k8s-certs/master/api-server.pem --tls-cert-file=/etc/kubernetes/k8s-certs/master/api-server.crt"
[root@master ~]# cat /etc/kubernetes/controller-manager 
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate
#--root-ca-file=/var/run/kubernetes/CA/rootCA.crt --service-account-private-key-file=/var/run/kubernetes/controler_scheduler/cs_client.crt 
# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS= "--root-ca-file=/etc/kubernetes/k8s-certs/CA/rootCA.crt --service-account-private-key-file=/etc/kubernetes/k8s-certs/master/api-server.pem --kubeconfig=/etc/kubernetes/cs_kubeconfig"

scheduler:

/usr/bin/kube-scheduler --logtostderr=true --v=4 --master=https://master:443 --kubeconfig=/etc/kubernetes/cs_kubeconfig
###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS="--kubeconfig=/etc/kubernetes/cs_kubeconfig"

common config:

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=4"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=https://master:443"

kubelet:

/usr/bin/kubelet --logtostderr=true --v=4 --cluster-dns=10.254.0.10 --cluster-domain=cluster.local --hostname-override=master --allow-privileged=false --cgroup-driver=systemd --fail-swap-on=false --pod_infra_container_image=177.1.1.35/library/pause:latest --kubeconfig=/etc/kubernetes/kp_kubeconfig

###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
#KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=master"
#KUBELET_API_SERVER="--api-servers=http://master:8080
#KUBECONFIG="--kubeconfig=/root/.kube/config-demo"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kp_kubeconfig"
KUBELET_DNS="--cluster-dns=10.254.0.10"
KUBELET_DOMAIN="--cluster-domain=cluster.local"
# Add your own!
KUBELET_ARGS="--cgroup-driver=systemd --fail-swap-on=false --pod_infra_container_image=177.1.1.35/library/pause:latest"

kube-proxy:

/usr/bin/kube-proxy --logtostderr=true --v=4 --master=https://master:443 --kubeconfig=/etc/kubernetes/kp_kubeconfig
###
# kubernetes proxy config

# default config should be adequate

# Add your own!
KUBE_PROXY_ARGS="--kubeconfig=/etc/kubernetes/kp_kubeconfig"

controller-manager & scheduler kubeconfig file:

apiVersion: v1
kind: Config
users:
- name: controllermanger
  user:
    client-certificate: /etc/kubernetes/k8s-certs/controler_scheduler/cs_client.crt
    client-key: /etc/kubernetes/k8s-certs/controler_scheduler/cs_client.pem
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/k8s-certs/CA/rootCA.crt
    server: https://master:443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: controllermanger
  name: cs-context
current-context: cs-context

kubelet & kube-proxy kubeconfig file:

apiVersion: v1
kind: Config
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/k8s-certs/kubelet_proxy/kp_client.crt
    client-key: /etc/kubernetes/k8s-certs/kubelet_proxy/kp_client.pem
clusters:
- name: kubernetes
  cluster:
    certificate-authority: /etc/kubernetes/k8s-certs/CA/rootCA.crt
    server: https://master:443
contexts:
- context:
    cluster: kubernetes
    user: kubelet
  name: kp-context
current-context: kp-context

APISERVER logs:

authentication.go:63] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, invalid bearer token]]
authentication.go:63] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, Token has been invalidated]]

I tried ues the normal pod to access the apiserver using:

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://10.254.0.1:443

It's successed and return data.

-- frank

kube-dns

kubernetes

3 Answers

10/22/2019

Not directly related to this problem, but with the same error (Failed to list *v1.Endpoints: Unauthorized), but it may be useful for someone while searching:

If in the master, kube-apiserver.log show this error:

E1022 14:13:35.488259       1 reflector.go:125] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.ServiceAccount: Get https://[::1]:443/api/v1/serviceaccounts?limit=500&resourceVersion=0: dial tcp [::1]:443: connect: cannot assign requested address

then the problem is a disabled ipv6 in the kubernetes master when it was enabled during install.

Doing this in the kubernetes masters fix the problem:

sudo sysctl net.ipv6.conf.all.disable_ipv6=0

After this, running if -6 a should show basic ipv6 support, namely the lo interface with ::1

-- higuita

Source: StackOverflow

9/29/2018

Unauthorized errors mean the authentication credentials being used are not valid.

-- Jordan Liggitt

Source: StackOverflow

10/1/2018

Inside your kube-dns config, you have #serviceAccountName: kube-dns commented out, remove the comment. Also make sure you are using system:serviceaccount:kube-system:kube-dns and have correct RBAC.

For more information, you can check this GitHub issue #50799.

-- Crou

Source: StackOverflow

KQ

kubedns can not start with "Failed to list *v1.Endpoints: Unauthorized" and "Failed to list *v1.Service: Unauthorized"

Similar Questions

3 Answers

K
Q

kubedns can not start with "Failed to list v1.Endpoints: Unauthorized" and "Failed to list v1.Service: Unauthorized"