I have a Kubernetes 1.5.3 cluster brought up on two nodes using kubeadm (details below). I've been trying to start up prometheus-operator tonight. After I deleted a resource of kind=Prometheus, my cluster got in to a very strange state, where when I attempted to recreate the prometheus resource, no pod was ever created, though the TPR exists. Investigation lead me to:
rrix@hypervisor01:~$ kubectl logs -n kube-system kube-apiserver-hypervisor01 | tail -10 W0226 03:57:34.641244 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:57:34.645073 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.642150 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.647953 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.642118 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.646427 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.642189 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.647978 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.646129 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.666355 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
The workaround in this rancher github issue is to simply restart the apiserver, which would be fine, except this didn't resolve the issue. Otherwise, it points to high load on the etcd, which as you can see below, is hosted as a pod, on the host which currently has a load of less than 1. etcdctl commands run successfully, leading me to believe that the etcd itself is fine. For good measure I bumped the etcd docker container, and it still works fine. From here, however, I am lost and need some assistance getting my cluster back in working order.
Cluster overview:
rrix@hypervisor01:~$ kubectl get no NAME STATUS AGE hypervisor01 Ready,master 2d kubes01.pss9.kickass.systems Ready 2d rrix@hypervisor01:~$ kubectl get -n kube-system po NAME READY STATUS RESTARTS AGE dummy-2088944543-2p3zf 1/1 Running 1 2d etcd-hypervisor01 1/1 Running 2 2d kube-apiserver-hypervisor01 1/1 Running 3 2d kube-controller-manager-hypervisor01 1/1 Running 1 2d kube-discovery-1769846148-v8h50 1/1 Running 1 2d kube-dns-2924299975-3s26d 4/4 Running 4 2d kube-proxy-vpw73 1/1 Running 1 2d kube-proxy-zfh13 1/1 Running 0 2d kube-registry-proxy-6hhk9 1/1 Running 1 2d kube-registry-proxy-nl1s1 1/1 Running 0 2d kube-registry-v0-4d94t 1/1 Running 0 2d kube-scheduler-hypervisor01 1/1 Running 1 2d rrix@hypervisor01:~$ kubectl get po NAME READY STATUS RESTARTS AGE kube-flannel-ds-589fw 2/2 Running 3 2d kube-flannel-ds-7f5sx 2/2 Running 0 2d kube-state-metrics-3229993571-20nbc 1/1 Running 0 1h node-exporter-1cdlj 1/1 Running 0 1h node-exporter-jc54s 1/1 Running 0 1h prometheus-operator-996254120-0wzg6 1/1 Running 0 2h
E0226 16:20:47.861762 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.862793 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:20:47.865352 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.865472 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.854692 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.855043 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.858436 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.858561 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value
User error facepalm, didn't think to look in the manager logs.