I am trying to build a kubernetes master with kubelet and kube-api server running as a static pod.

My unit for kubelet is:

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/bin/kubelet \
  --cloud-provider=external \
  --config=/var/lib/kubelet/config.yaml \
  --network-plugin=cni \
  --register-node=false \
  --kubeconfig=/var/lib/kubelet/kubeconfig.yaml \
  --v=2

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

When I start the kubelet I see the following errors:

.0.1:6443/api/v1/nodes/master-3-tm?resourceVersion=0&timeout=10s: dial tcp 127.0.0.1:6443: connect: connection refused
Nov 25 15:40:14 master-3-tm kubelet[2584]: E1125 15:40:14.254850    2584 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "master-3-tm": Get https://127.0.0.1:6443/api/v1/nodes/master-3-tm?timeout=10s: dial tcp 127.0.0.1:6443: connect: connection refused
Nov 25 15:40:14 master-3-tm kubelet[2584]: E1125 15:40:14.255466    2584 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "master-3-tm": Get https://127.0.0.1:6443/api/v1/nodes/master-3-tm?timeout=10s: dial tcp 127.0.0.1:6443: connect: connection refused
Nov 25 15:40:14 master-3-tm kubelet[2584]: E1125 15:40:14.255956    2584 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "master-3-tm": Get https://127.0.0.1:6443/api/v1/nodes/master-3-tm?timeout=10s: dial tcp 127.0.0.1:6443: connect: connection refused
Nov 25 15:40:14 master-3-tm kubelet[2584]: E1125 15:40:14.256403    2584 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "master-3-tm": Get https://127.0.0.1:6443/api/v1/nodes/master-3-tm?timeout=10s: dial tcp 127.0.0.1:6443: connect: connection refused
Nov 25 15:40:14 master-3-tm kubelet[2584]: E1125 15:40:14.256696    2584 kubelet_node_status.go:379] Unable to update node status: update node status exceeds retry count
Nov 25 15:40:14 master-3-tm kubelet[2584]: W1125 15:40:14.604686    2584 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 25 15:40:14 master-3-tm kubelet[2584]: E1125 15:40:14.604828    2584 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Which make sense, because the kube-api server is still not running. But the question is how do I get it to running?

I have the following manifests:

root@master-3-tm:/home/ubuntu# cat /etc/kubernetes/manifests/kube-api-server.yaml 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --advertise-address=10.32.192.20
    - --allow-privileged=true
    - --audit-log-maxage=30
    - --audit-log-maxbackup=3
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --bind-address=10.32.192.20
    - --client-ca-file=/var/lib/kubernetes/ca.pem
    - --cloud-config=/etc/kubernetes/cloud.conf
    - --cloud-provider=openstack
    - --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/api-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/api-etcd-client.key
    - --etcd-servers=master-1-tm=https://10.32.192.69:2380,master-3-tm=https://10.32.192.20:2380,master-2-tm=https://10.32.192.76:2380
    - --insecure-port=0
    - --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem
    - --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem
    - --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem
    - --kubelet-https=true
    - --secure-port=6443
    - --service-account-key-file=/var/lib/kubernetes/service-accounts.pem
    - --service-cluster-ip-range=10.32.0.0/16
    - --service-node-port-range=30000-32767
    - --runtime-config=api/all
    - --tls-cert-file=/var/lib/kubernetes/api.cert
    - --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem
    - --token-auth-file=/var/lib/kubernetes/token.csv
    - --v=2
    - --insecure-bind-address=127.0.0.1
    image: k8s.gcr.io/kube-apiserver-amd64:v1.11.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 10.32.192.20
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /var/lib/kubernetes
      readOnly: true
      name: var-lib-kubernetes
    - mountPath: /var/log/kubernetes
      name: var-log-kubernetes
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /var/lib/kuberentes
      type: DirectoryOrCreate
  - hostPath:
      path: /var/log/kuberentes
      type: DirectoryOrCreate
status: {}
root@master-3-tm:/home/ubuntu# cat /etc/kubernetes/manifests/etcd.yml 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    env:
      - name: ETCD_NAME
        value: master-3-tm
      - name: ETCD_DATA_DIR
        value: /var/lib/data
      - name: ETCD_INITIAL_CLUSTER_STATE
        value: new
      - name: ETCD_INITIAL_CLUSTER_TOKEN
        value: k8s-cluster
      - name: ETCD_INITIAL_CLUSTER
        value: master-1-tm=https://10.32.192.69:2380,master-3-tm=https://10.32.192.20:2380,master-2-tm=https://10.32.192.76:2380
      - name: ETCD_ADVERTISE_CLIENT_URLS
        value: https://10.32.192.20:2379
      - name: ETCD_LISTEN_PEER_URLS
        value: https://10.32.192.20:2380
      - name: ETCD_LISTEN_CLIENT_URLS
        value: https://10.32.192.20:2379
      - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
        value: https://10.32.192.20:2380
    image: quay.io/coreos/etcd:v3.3.10
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -ec
        - ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
          --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
          get foo
      failureThreshold: 8
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: etcd
    resources: {}
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
status: {}

Oddly, kubelet will start etcd, but not the api server...

Also worth noting:

 * the kubelet isn't trying to register, at least according to the flag:
   Nov 25 15:50:43 master-3-tm kubelet[3440]: I1125 15:50:43.578457    3440 flags.go:27] FLAG: --register-node="false"
Nov 25 15:50:43 master-3-tm kubelet[3440]: I1125 15:50:43.578464    3440 flags.go:27] FLAG: --register-schedulable="true"
Nov 25 15:50:43 master-3-tm kubelet[3440]: I1125 15:50:43.578471    3440 flags.go:27] FLAG: --register-with-taints=""

How do I start the api pod before the kubelet service? or how do I find why kubelet won't start that specific pod?

update

The kubelet should find the static pods in the correct place:

ubuntu@master-3-tm:~$ grep manifests /var/lib/kubelet/config.yaml 
staticPodPath: /etc/kubernetes/manifests