Today I found my kubernetes(v1.15.x
) stateful service showing this error:
2021-06-22 09:51:09.017 ERROR 24 --- [ver-test-beta-3] c.n.e.cluster.ReplicationTaskProcessor : Network level connection to peer statefulset-apollo-config-server-test-beta-1.service-apollo-meta-server-test-beta; retrying after delay
com.sun.jersey.api.client.ClientHandlerException: java.net.UnknownHostException: statefulset-apollo-config-server-test-beta-1.service-apollo-meta-server-test-beta
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48)
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:116)
at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:80)
at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: statefulset-apollo-config-server-test-beta-1.service-apollo-meta-server-test-beta
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:263)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:134)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
... 10 common frames omitted
But the stateful service only start one pod(do not change in recent, always one pod with the stateful service), the only one pod name is statefulset-apollo-config-server-test-beta-0
。 But the service connect to statefulset-apollo-config-server-test-beta-1
, it is wired. I also check my configmap and the connection address like this:
application-github.properties:
spring.datasource.url = jdbc:mysql://apollo-mysql.dabai-fat.svc.cluster.local:3306/ApolloConfigDBTEST?characterEncoding=utf8
spring.datasource.username = root
spring.datasource.password = gl4LucnXwLeLwAd29QqJn4
eureka.service.url = http://statefulset-apollo-config-server-test-alpha-0.service-apollo-meta-server-test-alpha:8080/eureka/
Why did this happen and what should I do to fix it?