Am trying to install opensource version of openshift origin i.e. OKD v3.11 without internet using ansible. During the complete installation process my internet is disabled on the environment. After the successful installation, I observe that the two pods in kube-service-catalog namespace namely apiserver and controller-manager aren't running. After investigating the playbooks, I discover the playbooks generate API Server keys.
Does the generation of API server keys expect a active internet connection? Is there any internet dependency for the apiserver and controller-manager pods to be in running state?
I tried:- Enabling the internet and redeploying the pods of the kube-service-catalog namespace. They were in the running state without any restart as expected.
Expected behaviour:- The two pods in the kube-service-catalog namespace should be stable and be in the Running state with internet disabled.
Actual behaviour:- The two pods in the kube-service-catalog namespace are in CrashLoopBackOff state.
Version:- OKD- 3.11, ansible- 2.9
Logs of apiserver pod:-
I0512 04:53:30.258151 1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true NamespacedServiceBroker:true]
I0512 04:53:30.258177 1 hyperkube.go:192] Service Catalog version v3.11.0-0.1.35+8d4f895-2;Upstream:v0.1.35 (built 2019-01-08T23:12:26Z)
W0512 04:53:31.020172 1 util.go:112] OpenAPI spec will not be served
I0512 04:53:31.021577 1 util.go:182] Admission control plugin names: [NamespaceLifecycle MutatingAdmissionWebhook ValidatingAdmissionWebhook ServicePlanChangeValidator BrokerAuthSarCheck DefaultServicePlan ServiceBindingsLifecycle]
I0512 04:53:31.021949 1 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionWebhook,ServicePlanChangeValidator,BrokerAuthSarCheck,DefaultServicePlan,ServiceBindingsLifecycle.
I0512 04:53:31.021971 1 plugins.go:161] Loaded 1 validating admission controller(s) successfully in the following order: ValidatingAdmissionWebhook.
I0512 04:53:31.023932 1 storage_factory.go:285] storing {servicecatalog.k8s.io clusterservicebrokers} in servicecatalog.k8s.io/v1beta1, reading as servicecatalog.k8s.io/__internal from storagebackend.Config{Type:"", Prefix:"/registry", ServerList:[]string{"https://cic-90-master.novalocal:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000}
I0512 04:53:31.023978 1 storage_factory.go:285] storing {servicecatalog.k8s.io clusterserviceclasses} in servicecatalog.k8s.io/v1beta1, reading as servicecatalog.k8s.io/__internal from storagebackend.Config{Type:"", Prefix:"/registry", ServerList:[]string{"https://cic-90-master.novalocal:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000}
I0512 04:53:31.023998 1 storage_factory.go:285] storing {servicecatalog.k8s.io clusterserviceplans} in servicecatalog.k8s.io/v1beta1, reading as servicecatalog.k8s.io/__internal from storagebackend.Config{Type:"", Prefix:"/registry", ServerList:[]string{"https://cic-90-master.novalocal:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000}
I0512 04:53:31.024031 1 storage_factory.go:285] storing {servicecatalog.k8s.io serviceinstances} in servicecatalog.k8s.io/v1beta1, reading as servicecatalog.k8s.io/__internal from storagebackend.Config{Type:"", Prefix:"/registry", ServerList:[]string{"https://cic-90-master.novalocal:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000}
I0512 04:53:31.024055 1 storage_factory.go:285] storing {servicecatalog.k8s.io servicebindings} in servicecatalog.k8s.io/v1beta1, reading as servicecatalog.k8s.io/__internal from storagebackend.Config{Type:"", Prefix:"/registry", ServerList:[]string{"https://cic-90-master.novalocal:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000}
F0512 04:53:51.025999 1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry [https://cic-90-master.novalocal:2379] /etc/origin/master/master.etcd-client.key /etc/origin/master/master.etcd-client.crt /etc/origin/master/master.etcd-ca.crt true true 0 {0xc420345080 0xc420345100} <nil> 5m0s 1m0s}), err (context deadline exceeded)Logs of controller-manager pods:-
I0512 05:05:01.273888 1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true]
I0512 05:05:01.274109 1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true]
I0512 05:05:01.274128 1 feature_gate.go:194] feature gates: map[NamespacedServiceBroker:true OriginatingIdentity:true AsyncBindingOperations:true]
I0512 05:05:01.274155 1 hyperkube.go:192] Service Catalog version v3.11.0-0.1.35+8d4f895-2;Upstream:v0.1.35 (built 2019-01-08T23:12:26Z)
I0512 05:05:01.276689 1 leaderelection.go:185] attempting to acquire leader lease kube-service-catalog/service-catalog-controller-manager...
I0512 05:05:01.303464 1 leaderelection.go:194] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I0512 05:05:01.303609 1 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-service-catalog", Name:"service-catalog-controller-manager", UID:"724069a9-9362-11ea-b5c1-fa163e86d97a", APIVersion:"v1", ResourceVersion:"126373", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' controller-manager-jvx4f-external-service-catalog-controller became leader
F0512 05:05:01.332950 1 controller_manager.go:237] error running controllers: failed to get api versions from server: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the requestOutput of kubectl get events:-
LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
2h 2h 1 service-catalog-controller-manager.160e29595b5f2ac8 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
1h 1h 1 service-catalog-controller-manager.160e29a1c8d44d5f ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
1h 1h 1 service-catalog-controller-manager.160e29e88bcdabf4 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
1h 1h 1 service-catalog-controller-manager.160e2a2ea2d553cf ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
1h 1h 1 service-catalog-controller-manager.160e2abce844b1a6 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
1h 1h 1 service-catalog-controller-manager.160e2bd884a3fd98 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
1h 17h 183 apiserver-28mjt.160df6e8ab679328 Pod spec.containers{apiserver} Normal Pulled kubelet, cic-90-master.novalocal Container image "docker.io/openshift/origin-service-catalog:v3.11.0" already present on machine
1h 1h 1 service-catalog-controller-manager.160e2c1f807c24b0 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
59m 59m 1 service-catalog-controller-manager.160e2cac5f27eb61 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
48m 48m 1 service-catalog-controller-manager.160e2d3d315161ed ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
43m 43m 1 service-catalog-controller-manager.160e2d84348e29c6 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
38m 38m 1 service-catalog-controller-manager.160e2dcbb5d88e66 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
33m 33m 1 service-catalog-controller-manager.160e2e13307a6011 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
23m 23m 1 service-catalog-controller-manager.160e2ea16c9db85d ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
8m 8m 1 service-catalog-controller-manager.160e2f75c0f6468a ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
4m 17h 4491 apiserver-28mjt.160df6f2fa5c8d45 Pod spec.containers{apiserver} Warning BackOff kubelet, cic-90-master.novalocal Back-off restarting failed container
2m 2m 1 service-catalog-controller-manager.160e2fbf5d9a2418 ConfigMap Normal LeaderElection service-catalog-controller-manager controller-manager-jvx4f-external-service-catalog-controller became leader
2m 20h 5739 controller-manager-jvx4f.160dec6599cd8b00 Pod spec.containers{controller-manager} Warning BackOff kubelet, cic-90-master.novalocal Back-off restarting failed container