Unable to start Kube-apiserver service

2/24/2020

I am installing kubernetes the hardway by mumshad(https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md). currently stuck at Bootstrapping the Kubernetes Control Plane phase, have followed the instructions in the document carefully but for some reason the kube-apiserver is not running and it is in auto restart state. Could any one of you help me on this. Same issue on both master nodes, however the kube-schedular and kube-control manager are running properly. Errors are provided below.

root@master-1:~# service kube-apiserver status.
● kube-apiserver.service - Kubernetes API Server
 Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
 Active: activating (auto-restart) (Result: exit-code) since Sat 2020-02-22 09:28:07 UTC; 476ms ago
 Docs: https://github.com/kubernetes/kubernetes
 Process: 10656 ExecStart=/usr/local/bin/kube-apiserver --advertise-address=192.168.5.11 --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audi
 Main PID: 10656 (code=exited, status=1/FAILURE)Feb 22 09:28:07 master-1 kube-apiserver[10656]: --log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximu
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --logtostderr log to standard error instead of files (default true)
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --skip-headers If true, avoid header prefixes in the log messages
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --skip-log-headers If true, avoid headers when opening log files
Feb 22 09:28:07 master-1 systemd[1]: kube-apiserver.service: Failed with result 'exit-code'.
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --stderrthreshold severity logs at or above this threshold go to stderr (default 2)
Feb 22 09:28:07 master-1 kube-apiserver[10656]: -v, --v Level number for the log level verbosity (default 0)
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --version version[=true] Print version information and quit
Feb 22 09:28:07 master-1 kube-apiserver[10656]: --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered loggingroot@master-1:~# kubectl get componentstatuses --kubeconfig admin.kubeconfig
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
root@master-1:~#apiserver_service file:
cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
 --advertise-address=${INTERNAL_IP} \\
 --allow-privileged=true \\
 --apiserver-count=3 \\
 --audit-log-maxage=30 \\
 --audit-log-maxbackup=3 \\
 --audit-log-maxsize=100 \\
 --audit-log-path=/var/log/audit.log \\
 --authorization-mode=Node,RBAC \\
 --bind-address=0.0.0.0 \\
 --client-ca-file=/var/lib/kubernetes/ca.crt \\
 --enable-admission-plugins=NodeRestriction,ServiceAccount \\
 --enable-swagger-ui=true \\
 --enable-bootstrap-token-auth=true \\
 --etcd-cafile=/var/lib/kubernetes/ca.crt \\
 --etcd-certfile=/var/lib/kubernetes/etcd-server.crt \\
 --etcd-keyfile=/var/lib/kubernetes/etcd-server.key \\
 --etcd-servers=https://192.168.5.11:2379,https://192.168.5.12:2379 \\
 --event-ttl=1h \\
 --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
 --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \\
 --kubelet-client-certificate=/var/lib/kubernetes/kube-apiserver.crt \\
 --kubelet-client-key=/var/lib/kubernetes/kube-apiserver.key \\
 --kubelet-https=true \\
 --runtime-config=api/all \\
 --service-account-key-file=/var/lib/kubernetes/service-account.crt \\
 --service-cluster-ip-range=10.96.0.0/24 \\
 --service-node-port-range=30000-32767 \\
 --tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt \\
 --tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key \\
 --v=2
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF
-- DevOpsGeek
kubernetes

2 Answers

2/24/2020

Share systemctl status kube-apiserver -l command output, also check /var/log/messages file and post error here.

-- Pawan Kumar
Source: StackOverflow

2/25/2020

I have figured out the issue. The issue is with the parameter --runtime-config=api/all it is not set to any value (true/false). Error:

    Feb 25 05:49:40 master-1 kube-apiserver[1228]: I0225 05:49:40.192274    1228 server.go:639] Initializing cache sizes based on 0MB limit
    Feb 25 05:49:40 master-1 kube-apiserver[1228]: I0225 05:49:40.192416    1228 server.go:150] Version: v1.17.0
    Feb 25 05:49:40 master-1 kube-apiserver[1228]: Error: invalid value api/all=
    Feb 25 05:49:40 master-1 kube-apiserver[1228]: Usage:
    Feb 25 05:49:40 master-1 kube-apiserver[1228]:   kube-apiserver [flags]

Once I have set it to true(--runtime-config=api/all) and restarted the service I was able to make the kube-apiserver running.

Results:

    root@master-1:~# service kube-apiserver status
    ● kube-apiserver.service - Kubernetes API Server
       Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
       Active: active (running) since Tue 2020-02-25 07:17:09 UTC; 2min 31s ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 864 (kube-apiserver)
        Tasks: 12 (limit: 2361)
       CGroup: /system.slice/kube-apiserver.service
               └─864 /usr/local/bin/kube-apiserver --advertise-address=192.168.5.11 --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxs

    Feb 25 07:17:09 master-1 systemd[1]: Started Kubernetes API Server.
    Feb 25 07:17:17 master-1 systemd-journald[412]: Suppressed 1644 messages from kube-apiserver.service
    Feb 25 07:17:17 master-1 kube-apiserver[864]: I0225 07:17:17.017139     864 controller.go:606] quota admission added evaluator for: serviceaccounts

    root@master-1:~# kubectl get componentstatuses --kubeconfig admin.kubeconfig
    NAME                 STATUS    MESSAGE             ERROR
    controller-manager   Healthy   ok                  
    scheduler            Healthy   ok                  
    etcd-0               Healthy   {"health":"true"}   
    etcd-1               Healthy   {"health":"true"}   
    root@master-1:~# 
-- Satish Reddy
Source: StackOverflow