We deploy new Kubernetes cluster, when I was trying to troubleshoot the things, I find the link to test the API is working fine or not. Access Cluster.

When I ran same code snippet, it failed with 401.

[devops@kubemaster01 ~]$ APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
[devops@kubemaster01 ~]$ SECRET_NAME=$(kubectl get secrets | grep ^default | cut -f1 -d ' ')
[devops@kubemaster01 ~]$ TOKEN=$(kubectl describe secret $SECRET_NAME | grep -E '^token' | cut -f2 -d':' | tr -d " ")
[devops@kubemaster01 ~]$
[devops@kubemaster01 ~]$ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

I might have to renew my default token or is there any way to see why token is not working ?

When I check the api status, it says its Unhealthy

$ kubectl describe pods kube-apiserver-kubemaster03 -n kube-system
Name:                 kube-apiserver-kubemaster03
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 kubemaster03/172.17.201.207
Start Time:           Tue, 16 Mar 2021 12:36:36 -0400
Labels:               component=kube-apiserver
                      tier=control-plane
Annotations:          kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.17.201.207:6443
                      kubernetes.io/config.hash: 5e945fc0a5010d1e9de2db70436d943a
                      kubernetes.io/config.mirror: 5e945fc0a5010d1e9de2db70436d943a
                      kubernetes.io/config.seen: 2021-03-16T12:36:31.043020249-04:00
                      kubernetes.io/config.source: file
Status:               Running
IP:                   172.17.201.207
IPs:
  IP:           172.17.201.207
Controlled By:  Node/kubemaster03
Containers:
  kube-apiserver:
    Container ID:  containerd://d7b9c260c7c1c5d7c487151d42a3dfe588ae19bf4258b3e48f4e9e949e61a0b7
    Image:         k8s.gcr.io/kube-apiserver:v1.20.4
    Image ID:      k8s.gcr.io/kube-apiserver@sha256:adef5d31ea2fcf9c523e47bbcc6a955f3c247abbd8a9a97f4a26fdeb18f9b4b8
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --advertise-address=172.17.201.207
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-servers=http://172.17.201.233:2380,http://172.17.201.232:2380,http://172.17.201.234:2380,http://172.17.201.230:2380,http://172.17.201.231:2380
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-issuer=https://kubernetes.default.svc.cluster.local
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Running
      Started:      Tue, 16 Mar 2021 12:36:37 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        250m
    Liveness:     http-get https://172.17.201.207:6443/livez delay=10s timeout=15s period=10s #success=1 #failure=8
    Readiness:    http-get https://172.17.201.207:6443/readyz delay=0s timeout=15s period=1s #success=1 #failure=3
    Startup:      http-get https://172.17.201.207:6443/livez delay=10s timeout=15s period=10s #success=1 #failure=24
    Environment:  <none>
    Mounts:
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/pki from etc-pki (ro)
      /etc/ssl/certs from ca-certs (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  etc-pki:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/pki
    HostPathType:  DirectoryOrCreate
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute op=Exists
Events:
  Type     Reason     Age                        From     Message
  ----     ------     ----                       ----     -------
  Warning  Unhealthy  5m29s (x11090 over 6d23h)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500

When I check the log, it continue showing blockingPicker related message.

I0323 16:06:10.047596       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:10.673251       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
E0323 16:06:10.812587       1 authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive]
I0323 16:06:11.004416       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:11.201781       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:11.629293       1 client.go:360] parsed scheme: "passthrough"
I0323 16:06:11.629334       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{http://172.17.201.234:2380  <nil> 0 <nil>}] <nil> <nil>}
I0323 16:06:11.629349       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0323 16:06:12.246757       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:12.312942       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:12.858684       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:13.899798       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:13.940296       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:14.166107       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
E0323 16:06:14.258808       1 authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive]
I0323 16:06:14.674708       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:14.866273       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:15.086297       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:15.195023       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:15.228564       1 client.go:360] parsed scheme: "passthrough"
I0323 16:06:15.228619       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{http://172.17.201.233:2380  <nil> 0 <nil>}] <nil> <nil>}
I0323 16:06:15.228630       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0323 16:06:15.674298       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:15.692577       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:16.054344       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:16.502775       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:17.092373       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:17.249160       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:17.978726       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.085490       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.382714       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.389807       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.484660       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.547850       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.799944       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:18.941454       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0323 16:06:19.667581       1 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick