Steps

I have a use case in which I want to create a kubernetes cluster from scratch using kubeadm.

$ kubeadm init --config admin.yaml --v=7

admin.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: /run/containerd/containerd.sock
  ignorePreflightErrors:
    - SystemVerification
localAPIEndpoint:
  bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  extraArgs:
    feature-gates: CSIBlockVolume=true,CSIDriverRegistry=true,CSINodeInfo=true,VolumeSnapshotDataSource=true
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  CSIBlockVolume: true
  CSIDriverRegistry: true
  CSINodeInfo: true

All Operations seem to work until the connection to the kublet should be established. This is the final log before the crash. The GET requests are sent approx 100 times before it crashes.

LOG:

I1216 12:31:45.043530   15460 round_trippers.go:463] GET https://<IP>:6443/healthz?timeout=10s
I1216 12:31:45.043550   15460 round_trippers.go:469] Request Headers:
I1216 12:31:45.043555   15460 round_trippers.go:473]     Accept: application/json, */*
I1216 12:31:45.043721   15460 round_trippers.go:574] Response Status:  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.

	Unfortunately, an error has occurred:
		timed out waiting for the condition

	This error is likely caused by:
		- The kubelet is not running
		- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

I read the logs for kubelet with

$ journalctl -xeu kubelet

This is the output I received:

- The job identifier is 49904.
Dec 16 13:40:42 <IP> kubelet[24113]: I1216 13:40:42.883879   24113 server.go:198] "Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in tha>
Dec 16 13:40:42 <IP> kubelet[24113]: E1216 13:40:42.885069   24113 server.go:217] "Failed to set feature gates from initial flags-based config" err="unrecognized feature gate: CSIBlockVolume"
Dec 16 13:40:42 <IP> systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit kubelet.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Dec 16 13:40:42 <IP> systemd[1]: kubelet.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit kubelet.service has entered the 'failed' state with result 'exit-code'.

Setup

Software Versions:

$ kubelet --version
Kubernetes v1.23.0

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:15:11Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes and platform

$ uname -a
Linux 5.11.0-1022-aws #23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The server is deployed in amazon AWS

Container runtime: containerd

Docker installed: No

I also checked the Kubernetes documentaion, which, if I read this correctly, states that all the Feature Gates are now in the GA state, so integrated into Kubernetes not not experimental anymore.