Deploying the spring application only works in the default namespace. Deploying the same application to other namespaces fails with the following error:
I have noticed this has something to do with the service (named: Kubernetes) in the default namespace which is having the clusterIP of 10.242.0.1. Deploying the application to other namespaces still routes to this Kubernetes service in the default namespace. Adding service link to this service doesn't help. Creating a replica of this service with endpoints in the other namespaces doesn't solve the issue either.
2021-05-13 11:46:24.514 WARN [some-service,,] 1 --- [ main] s.c.k.f.c.Fabric8ConfigMapPropertySource : Can't read configMap with name: [my-configmap] in namespace:[my-namespace]. Ignoring.
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [ConfigMap] with name: [my-configmap] in namespace: [my-namespace] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:225) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:186) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:84) ~[kubernetes-client-4.13.2.jar:na]
at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.getData(Fabric8ConfigMapPropertySource.java:62) [spring-cloud-kubernetes-fabric8-config-2.0.2.jar:2.0.2]
at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.<init>(Fabric8ConfigMapPropertySource.java:50) [spring-cloud-kubernetes-fabric8-config-2.0.2.jar:2.0.2]
at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySourceLocator.getMapPropertySource(Fabric8ConfigMapPropertySourceLocator.java:51) [spring-cloud-kubernetes-fabric8-config-2.0.2.jar:2.0.2]
at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.getMapPropertySourceForSingleConfigMap(ConfigMapPropertySourceLocator.java:81) [spring-cloud-kubernetes-commons-2.0.2.jar:2.0.2]
at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.lambda$locate$0(ConfigMapPropertySourceLocator.java:67) [spring-cloud-kubernetes-commons-2.0.2.jar:2.0.2]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_282]
at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.locate(ConfigMapPropertySourceLocator.java:67) [spring-cloud-kubernetes-commons-2.0.2.jar:2.0.2]
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:51) ~[spring-cloud-context-3.0.2.jar:3.0.2]
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:47) ~[spring-cloud-context-3.0.2.jar:3.0.2]
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:95) ~[spring-cloud-context-3.0.2.jar:3.0.2]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:650) ~[spring-boot-2.4.5.jar:2.4.5]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:403) ~[spring-boot-2.4.5.jar:2.4.5]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.4.5.jar:2.4.5]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1340) ~[spring-boot-2.4.5.jar:2.4.5]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) ~[spring-boot-2.4.5.jar:2.4.5]
at com.ubs.wma.service.accountgroup.AccountGroupServiceApplication.main(AccountGroupServiceApplication.java:19) ~[classes/:na]
Caused by: java.net.ConnectException: Failed to connect to /10.242.0.1:443
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:133) ~[kubernetes-client-4.13.2.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at io.fabric8.kubernetes.client.utils.OIDCTokenRefreshInterceptor.intercept(OIDCTokenRefreshInterceptor.java:41) ~[kubernetes-client-4.13.2.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[kubernetes-client-4.13.2.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:151) ~[kubernetes-client-4.13.2.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[okhttp-3.14.9.jar:na]
at okhttp3.RealCall.execute(RealCall.java:81) ~[okhttp-3.14.9.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:472) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:402) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:384) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:935) ~[kubernetes-client-4.13.2.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:220) ~[kubernetes-client-4.13.2.jar:na]
... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_282]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_282]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_282]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_282]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_282]
at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_282]
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263) ~[okhttp-3.14.9.jar:na]
... 54 common frames omitted Added Role and RoleBinding resources to my deployment like this:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: namespace-reader
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: namespace-reader-binding
namespace: default
subjects:
- kind: ServiceAccount
name: default
apiGroup: ""
roleRef:
kind: Role
name: namespace-reader
apiGroup: ""My bootstrap.yaml config is looking like this:
spring:
cloud:
kubernetes:
config:
sources:
- name: some-service
namespace: ${MY_POD_NAMESPACE}My service deployment is looking like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: some-service
spec:
selector:
matchLabels:
app: some-service
template:
metadata:
labels:
app: some-service
spec:
containers:
- name: some-service
image: registry/some-service:latest
resources:
requests:
memory: 320Mi
limits:
memory: 512Mi
ports:
- containerPort: 8080
env:
- name: SPRING_PROFILES_ACTIVE
value: "some-profile"
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
initialDelaySeconds: 40
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 40
lifecycle:
preStop:
exec:
command: [ "sh", "-c", "sleep 10" ]Finally, my configmap looks as follows:
apiVersion: v1
data:
application-properties.yaml: 'property: "value"'
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/managed-by: Helm
name: my-configmapI have found a workaround that works for me. Created an admin clusterrolebinding resource for my namespace my-namespace like so:
kubectl create clusterrolebinding admin-my-namespace --clusterrole=cluster-admin --serviceaccount=my-namespace:defaultDeleted the Role and the RoleBinding resources as they are not needed any longer.
This works but would appreciate why Role and Rolebinding to ServiceAccount wasn't enough at the first place.