I've setup a vert.x / hazelcast cluster on a Kubernetes platform:

• 2 Instances / Pods are running
• Configuration via the Hazelcast Discovery Plugin for Kubernetes
• Kubernetes API is used as discovery method

The Hazelcast cluster itself is working without problems. I can see that the cluster consists of two members:

[100.116.0.36]:5701 [dev] [3.12] 

Members {size:2, ver:8} [
    Member [100.122.0.1]:5701 - 4d42a914-a6a5-4d65-897d-835eb5f5da4d
    Member [100.116.0.36]:5701 - be0010b5-1a0e-4359-bc94-70078fe4f2d4 this
]

I'm also using a shared map provided by Hazelcast which is synchronizing fine between the two pods.

Vert.x itself however is logging the following warnings:

Connecting to server localhost:41635 failedConnection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Connecting to server localhost:37251 failedConnection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

These messages may not be linked to vert.x directly, but they disappear when it is started in local mode.

I've also setup monitoring with micrometer and vert.x does not provide any value for vertx_eventbus_bytesWritten as stated here: https://vertx.io/docs/vertx-micrometer-metrics/java/

Hence, i'm fairly sure that vert.x sends all messages to local consumers only because of these warnings.

The way i have setup vert.x in clustered mode is the following:

VertxOptions vertxOptions = new VertxOptions();
vertxOptions.getEventBusOptions().setClustered(true);
vertxOptions.setClusterManager(new HazelcastClusterManager(hazelcastInstance)); // the hazelcast instance itself is setup by spring boot which i'm using for dependency injection

Vertx.rxClusteredVertx(vertxOptions).doOnSuccess((result -> {
            LOGGER.info("Clustered vertx setup complete. Deploying vertices....");
           // ...
        })).doOnError((throwable -> {
            LOGGER.error("Failed to setup clustered verx.", throwable);
        }));

Does not provide any error logs.

Any thoughts on what is going wrong here? Would greatly appreciate it.