How to debug Spring Cloud Kubernetes Configuration Watcher when it is not working as expectted

11/22/2021

There are two components in my k8s namespace.

  1. The first component is a simple Spring Boot Web Service with actuator/refresh endpoint exposed. I have mannualy make post request to the endpoint and it will trigger a configuration context refresh succesfully.

  2. The second component is Spring Cloud Kubernetes Configuration Watcher which I pull the image per official document guidance and make it run in my k8s env. Per official spring document, it should detect the changes on configmap with label spring.cloud.kubernetes.config="true" and make post request to the actuator/refresh endpoint of the application whose spring.application.name is equal to the configmap name.

The second component is not working as expectted, and I don't know how to trouble shoot the root cause.

My Springboot application is called spring-boot-demo, the configmap is also named as spring-boot-demo. But I never find any mentions of "spring-boot-demo" in Spring Cloud Kubernetes Configuration Watcher's log and nor can I assure if it has sent post request to related endpoint.

I can only see the logs showed up repeatedly below:

2021-11-22 02:58:53.332  INFO 1 --- [192.168.0.1/...] .w.HttpBasedConfigMapWatchChangeDetector : Added new Kubernetes watch: config-maps-watch-event
2021-11-22 02:58:53.332  INFO 1 --- [192.168.0.1/...] .w.HttpBasedConfigMapWatchChangeDetector : Kubernetes event-based configMap change detector activated
2021-11-22 03:34:06.555  WARN 1 --- [192.168.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : ConfigMaps watch closed

io.fabric8.kubernetes.client.WatcherException: too old resource version: 5491278743 (5554041906)
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onStatus(AbstractWatchManager.java:263) [kubernetes-client-5.5.0.jar:na]
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onMessage(AbstractWatchManager.java:247) [kubernetes-client-5.5.0.jar:na]
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onMessage(WatcherWebSocketListener.java:93) [kubernetes-client-5.5.0.jar:na]
        at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:322) [okhttp-3.14.9.jar:na]
        at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219) [okhttp-3.14.9.jar:na]
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105) [okhttp-3.14.9.jar:na]
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273) [okhttp-3.14.9.jar:na]
        at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209) [okhttp-3.14.9.jar:na]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) [okhttp-3.14.9.jar:na]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [okhttp-3.14.9.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_312]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_312]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 5491278743 (5554041906)
        ... 13 common frames omitted

2021-11-22 03:34:06.605  INFO 1 --- [192.168.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : Added new Kubernetes watch: config-maps-watch-event
2021-11-22 03:34:06.605  INFO 1 --- [192.168.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : Kubernetes event-based configMap change detector activated
2021-11-22 03:34:06.607  INFO 1 --- [192.168.0.1/...] s.c.k.f.c.Fabric8ConfigMapPropertySource : Loading ConfigMap with name 'spring-cloud-kubernetes-configuration-watcher' in namespace 'my-namespace'
2021-11-22 03:34:06.621  WARN 1 --- [192.168.0.1/...] o.s.c.k.f.config.Fabric8ConfigUtils      : config-map with name : 'spring-cloud-kubernetes-configuration-watcher' not present in namespace : 'my-namespace'
2021-11-22 03:34:06.625  WARN 1 --- [192.168.0.1/...] o.s.c.k.f.config.Fabric8ConfigUtils      : config-map with name : 'spring-cloud-kubernetes-configuration-watcher-kubernetes' not present in namespace : 'my-namespace'

The docker image I use is springcloud/spring-cloud-kubernetes-configuration-watcher:2.1.0-RC1

Any hint to debug this issue is appreciated.

-- Json
kubernetes
spring-boot
spring-boot-actuator
spring-cloud-config
spring-cloud-kubernetes

0 Answers