Kubernetes HA Setup with External Nginx Loadbalancer or F5 Loadbalancer User "system:anonymous" cannot create nodes at the cluster scope

8/17/2018

I am using external nginx loadbalancer and trying to configure K8s Master but its failing with below error :

error uploading configuration: unable to create configmap: configmaps is forbidden: User "system:anonymous" cannot create configmaps in the namespace "kube-system"**

For me it more looks like cert issue but i am having hard time to find what i am missing , any help is appreciated in our infrastructure we use F5 loadbalancer in front of apiserver and i am seeing the same issue there this is the env i have created for troubleshooting

kubeadm-config:

    apiVersion: kubeadm.k8s.io/v1alpha2
    kind: MasterConfiguration
    kubernetesVersion: v1.11.0
    apiServerCertSANs:
    - "ec2-23-23-244-63.compute-1.amazonaws.com"
    api:
    controlPlaneEndpoint: "ec2-23-23-244-63.compute-1.amazonaws.com:6443"
    etcd:
    external:
        endpoints:
        - https://172.31.32.160:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
     networking:
     # This CIDR is a calico default. Substitute or remove for your CNI provider.
     podSubnet: "10.244.0.0/16"

Env : Kubelet : 1.11.1 kubeadm 1.11.1 kubectl 1.11.1

Output

    [certificates] Using the existing ca certificate and key.
    [certificates] Using the existing apiserver certificate and key.
    [certificates] Using the existing apiserver-kubelet-client certificate and 
    key.
    [certificates] Using the existing sa key.
    [certificates] Using the existing front-proxy-ca certificate and key.
    [certificates] Using the existing front-proxy-client certificate and key.
    [certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
     [endpoint] WARNING: port specified in api.controlPlaneEndpoint overrides 
  api.bindPort in the controlplane address
     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller- 
  manager.conf"




     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
     [controlplane] wrote Static Pod manifest for component kube-apiserver to 
  "/etc/kubernetes/manifests/kube-apiserver.yaml"
     [controlplane] wrote Static Pod manifest for component kube-controller-manager 
  to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
     [controlplane] wrote Static Pod manifest for component kube-scheduler to 
  "/etc/kubernetes/manifests/kube-scheduler.yaml"
     [init] waiting for the kubelet to boot up the control plane as Static Pods 
  from directory "/etc/kubernetes/manifests"
  [init] this might take a minute or longer if the control plane images have to 
  be pulled
     [apiclient] All control plane components are healthy after 41.036802 seconds


     [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" 
     in the "kube-system" Namespace
     error uploading configuration: unable to create configmap: configmaps is 
     forbidden: User "system:anonymous" cannot create configmaps in the 
     namespace "kube-system"

logs:

    Unable to register node "ip-172-31-40-157" with API server: nodes is forbidden: User "system:anonymous" cannot create nodes at the cluster scope
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
on_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "ip-172-31-40-157" not found
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
:172] Unable to update cni config: No networks found in /etc/cni/net.d

Nginx :

   upstream mywebapp1 {
      server 172.31.40.157:6443;

   }
server {
    listen 6443 ssl;
    server_name ec2-23-23-244-63.compute-1.amazonaws.com;

    ssl on;
    ssl_certificate         /opt/certificates/server.crt;
    ssl_certificate_key     /opt/certificates/server.key;
    ssl_trusted_certificate /opt/certificates/ca.crt;

    location / {
        proxy_pass https://mywebapp1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Nginx Server : 172-31-44-203
Master Server : 172-31-40-157

I am using Self Signed Certs and the CA to generate all the certs including the one in nginx are same

I had same issue in our infrastructure when we use f5 loadbalancer

-- mannam14387
kubernetes

1 Answer

8/18/2018

If your nodes speak to the apiserver through a load balancer, and expect to use client certificate credentials to authenticate (which is typical for nodes), the load balancer must not terminate or re-encrypt TLS, or the client certificate information will be lost and the apiserver will see the request as anonymous.

-- Jordan Liggitt
Source: StackOverflow