I have a kubernetes cluster provisioned using kube-aws. I added two additional controllers and clustered etcd across the three controller nodes. The etcd cluster seems (mostly) healthy and works when I set and get values on the different nodes.
I added the new etcd servers to /etc/kubernetes/manifests/kube-apiserver.yaml:
--etcd-servers=http://10.0.0.50:2379,http://10.0.1.50:2379,http://10.0.2.50:2379Now, the kube-apiserver refuses connections on secure port 443.
The kubelet logs contain entries like this...
E0617 02:07:17.488289 30972 pod_workers.go:138] Error syncing pod
b394008e8164, skipping: failed to "StartContainer" for "etcd-container" with
CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd-container
pod=etcd-server-ip-10-0-0-50.eu-west-1.compute.internal_default(b394008e8164)"
and this...
E0617 02:03:00.542951 30972 event.go:193] Server rejected event
'&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""},
ObjectMeta:api.ObjectMeta{Name:"etcd-server-ip-10-0-0-50.eu-
west-1.compute.internal.1458b", GenerateName:"", Namespace:"default",
SelfLink:"", UID:"", ResourceVersion:"1189", Generation:0,
CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0,
loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil),
DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil),
Annotations:map[string]string(nil)},
InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"default",
Name:"etcd-server-ip-10-0-0-50.eu-west-1.compute.internal",
UID:"b394008e8164", APIVersion:"v1", ResourceVersion:"", FieldPath:""},
Reason:"FailedSync", Message:"Error syncing pod, skipping: failed to
\"StartContainer\" for \"etcd-container\" with RunContainerError:
\"runContainer: API error (500): mkdir /usr/local/openssl: read-only file
system\\n\"\n", Source:api.EventSource{Component:"kubelet", Host:"ip-
10-0-0-50.eu-west-1.compute.internal"},
FirstTimestamp:unversioned.Time{Time:time.Time{sec:63601711997, nsec:0,
loc:(*time.Location)(0x3b1a5c0)}},
LastTimestamp:unversioned.Time{Time:time.Time{sec:63601725780,
nsec:522501285, loc:(*time.Location)(0x3b1a5c0)}}, Count:7, Type:"Warning"}':
'events "etcd-server-ip-10-0-0-50.eu-west-1.compute.internal.1458b" not
found' (will not retry!)
Querying the kube-apiserver locally on port 8080 like so...
$ curl http://localhost:8080/api/v1/namespaces... returns a valid response.
But it refuses any attempt on the secure port.
Can anyone point me the right direction?
For anyone that finds this question having trouble with the kube-apiserver...
My problem turned out to be bad TLS assets which reside in /etc/kubernetes/ssl/. (In hindsight, it is obvious since it was responding to HTTP requests but not HTTPS.) The log entries in my question were irrelevant to the problem of the TLS assets. The kubelet logs helped me track down the problem:
$ journalctl -u kubelet -e # jump to end of kubelet logsThe following also came in handy:
$ journalctl -u oem-cloudinit # if using aws will help you debug your cloud-config
$ journalctl -u etcd2 -f # follow etcd logs