I am using external nginx loadbalancer and trying to configure K8s Master but its failing with below error :
error uploading configuration: unable to create configmap: configmaps is forbidden: User "system:anonymous" cannot create configmaps in the namespace "kube-system"**
For me it more looks like cert issue but i am having hard time to find what i am missing , any help is appreciated in our infrastructure we use F5 loadbalancer in front of apiserver and i am seeing the same issue there this is the env i have created for troubleshooting
kubeadm-config:
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.0
apiServerCertSANs:
- "ec2-23-23-244-63.compute-1.amazonaws.com"
api:
controlPlaneEndpoint: "ec2-23-23-244-63.compute-1.amazonaws.com:6443"
etcd:
external:
endpoints:
- https://172.31.32.160:2379
caFile: /etc/kubernetes/pki/etcd/ca.crt
certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
# This CIDR is a calico default. Substitute or remove for your CNI provider.
podSubnet: "10.244.0.0/16"
Env : Kubelet : 1.11.1 kubeadm 1.11.1 kubectl 1.11.1
Output
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and
key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[endpoint] WARNING: port specified in api.controlPlaneEndpoint overrides
api.bindPort in the controlplane address
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-
manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to
"/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager
to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to
"/etc/kubernetes/manifests/kube-scheduler.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods
from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to
be pulled
[apiclient] All control plane components are healthy after 41.036802 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config"
in the "kube-system" Namespace
error uploading configuration: unable to create configmap: configmaps is
forbidden: User "system:anonymous" cannot create configmaps in the
namespace "kube-system"
logs:
Unable to register node "ip-172-31-40-157" with API server: nodes is forbidden: User "system:anonymous" cannot create nodes at the cluster scope
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
on_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "ip-172-31-40-157" not found
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nginx :
upstream mywebapp1 {
server 172.31.40.157:6443;
}
server {
listen 6443 ssl;
server_name ec2-23-23-244-63.compute-1.amazonaws.com;
ssl on;
ssl_certificate /opt/certificates/server.crt;
ssl_certificate_key /opt/certificates/server.key;
ssl_trusted_certificate /opt/certificates/ca.crt;
location / {
proxy_pass https://mywebapp1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
Nginx Server : 172-31-44-203
Master Server : 172-31-40-157
I am using Self Signed Certs and the CA to generate all the certs including the one in nginx are same
I had same issue in our infrastructure when we use f5 loadbalancer
If your nodes speak to the apiserver through a load balancer, and expect to use client certificate credentials to authenticate (which is typical for nodes), the load balancer must not terminate or re-encrypt TLS, or the client certificate information will be lost and the apiserver will see the request as anonymous.