Why the pod connect the unknown service address in kubernetes

6/22/2021

Today I found my kubernetes(v1.15.x) stateful service showing this error:

2021-06-22 09:51:09.017 ERROR 24 --- [ver-test-beta-3] c.n.e.cluster.ReplicationTaskProcessor   : Network level connection to peer statefulset-apollo-config-server-test-beta-1.service-apollo-meta-server-test-beta; retrying after delay
com.sun.jersey.api.client.ClientHandlerException: java.net.UnknownHostException: statefulset-apollo-config-server-test-beta-1.service-apollo-meta-server-test-beta
	at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
	at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48)
	at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27)
	at com.sun.jersey.api.client.Client.handle(Client.java:652)
	at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
	at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
	at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
	at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:116)
	at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:80)
	at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: statefulset-apollo-config-server-test-beta-1.service-apollo-meta-server-test-beta
	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:263)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162)
	at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
	at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:134)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
	... 10 common frames omitted

But the stateful service only start one pod(do not change in recent, always one pod with the stateful service), the only one pod name is statefulset-apollo-config-server-test-beta-0。 But the service connect to statefulset-apollo-config-server-test-beta-1, it is wired. I also check my configmap and the connection address like this:

application-github.properties:
spring.datasource.url = jdbc:mysql://apollo-mysql.dabai-fat.svc.cluster.local:3306/ApolloConfigDBTEST?characterEncoding=utf8
spring.datasource.username = root
spring.datasource.password = gl4LucnXwLeLwAd29QqJn4
eureka.service.url = http://statefulset-apollo-config-server-test-alpha-0.service-apollo-meta-server-test-alpha:8080/eureka/

Why did this happen and what should I do to fix it?

-- Dolphin
kubernetes

0 Answers