kube-apiserver - invalid bearer token, Token has been invalidated

12/15/2019

When I try to start kube-apiserver I get following logs:

I1215 14:18:23.130968       1 controller.go:83] Starting OpenAPI controller
I1215 14:18:23.131021       1 customresource_discovery_controller.go:208] Starting DiscoveryController
I1215 14:18:23.131047       1 naming_controller.go:288] Starting NamingConditionController
I1215 14:18:23.131067       1 establishing_controller.go:73] Starting EstablishingController
I1215 14:18:23.131084       1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController
I1215 14:18:23.149275       1 controller_utils.go:1036] Caches are synced for crd-autoregister controller
I1215 14:18:23.191491       1 client.go:354] parsed scheme: ""
I1215 14:18:23.191600       1 client.go:354] scheme "" not registered, fallback to default scheme
I1215 14:18:23.191681       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I1215 14:18:23.206235       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
E1215 14:18:23.224439       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1215 14:18:23.232453       1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I1215 14:18:23.234017       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:23.279768       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I1215 14:18:23.289645       1 cache.go:39] Caches are synced for autoregister controller
I1215 14:18:23.292993       1 client.go:354] parsed scheme: ""
I1215 14:18:23.293085       1 client.go:354] scheme "" not registered, fallback to default scheme
I1215 14:18:23.293264       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I1215 14:18:23.294432       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:23.328179       1 cache.go:39] Caches are synced for AvailableConditionController controller
I1215 14:18:23.375123       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:23.834251       1 controller.go:107] OpenAPI AggregationController: Processing item 
I1215 14:18:23.834297       1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I1215 14:18:23.849807       1 storage_scheduling.go:128] all system priority classes are created successfully or already exist.
E1215 14:18:24.259439       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1215 14:18:24.943441       1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
E1215 14:18:25.269310       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:26.277415       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:27.283456       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:28.290149       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:29.297812       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:30.303821       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:31.312197       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:32.320085       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:33.331813       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:34.337389       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:35.345553       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:36.354106       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1215 14:18:36.916194       1 client.go:354] parsed scheme: ""
I1215 14:18:36.916270       1 client.go:354] scheme "" not registered, fallback to default scheme
I1215 14:18:36.916389       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I1215 14:18:36.916756       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:36.933986       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:36.944414       1 client.go:354] parsed scheme: ""
I1215 14:18:36.944474       1 client.go:354] scheme "" not registered, fallback to default scheme
[ more invalid token logs goes from here ]

When I try to run: kubectl get pods:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

My api server config:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=...
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.15.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: ...
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

Kube controller config:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    image: k8s.gcr.io/kube-controller-manager:v1.15.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-controller-manager
    resources:
      requests:
        cpu: 200m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      name: flexvolume-dir
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      type: DirectoryOrCreate
    name: flexvolume-dir
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}  

I use k8s in 1.15.0 version

I wonder what is the root cause - how I can refresh token in order to apiserver function correctly?

-- pixel
kube-apiserver
kubeadm
kubernetes-apiserver

0 Answers