Openshift routers update from version 3.7 to version 3.9 caused hundreds of warnings in Openshift logs:

[timestamp] [hz._hzInstance_1_dev.IO.thread-in-2] WARN com.hazelcast.nio.tcp.TcpIpConnection - [x.x.19.150]:5701 [dev] [3.11.4] Connection[id=157132, /x.x.19.150:5701->/x.x.25.1:50370, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=157132, /x.x.19.150:5701->/x.x.25.1:50370, endpoint=null, alive=true, type=NONE], thread=hz._hzInstance_1_dev.IO.thread-in-2 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
[timestamp] [hz._hzInstance_1_dev.IO.thread-in-0] WARN  com.hazelcast.nio.tcp.TcpIpConnection - [x.x.31.153]:5701 [dev] [3.11.4] Connection[id=156553, /x.x.31.153:5701->/x.x.9.1:48700, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=156553, /x.x.31.153:5701->/x.x.9.1:48700, endpoint=null, alive=true, type=NONE], thread=hz._hzInstance_1_dev.IO.thread-in-0\njava.io.IOException: Connection reset by peer\n at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
[timestamp] [hz._hzInstance_1_dev.IO.thread-in-2] WARN  com.hazelcast.nio.tcp.TcpIpConnection - [x.x.3.34]:5701 [dev] [3.11.4] Connection[id=157179, /x.x.3.34:5701->/x.x.25.1:60596, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=157179, /x.x.3.34:5701->/x.x.25.1:60596, endpoint=null, alive=true, type=NONE]
[timestamp] [hz._hzInstance_1_dev.IO.thread-in-1] WARN  com.hazelcast.nio.tcp.TcpIpConnection - [x.x.10.75]:5701 [dev] [3.11.4] Connection[id=157171, /x.x.10.75:5701->/x.x.25.1:33826, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=157171, /x.x.10.75:5701->/x.x.25.1:33826, endpoint=null, alive=true, type=NONE]
[timestamp] [hz._hzInstance_1_dev.IO.thread-in-1] WARN  com.hazelcast.nio.tcp.TcpIpConnection - [x.x.27.206]:5701 [dev] [3.11.4] Connection[id=157157, /x.x.27.206:5701->/x.x.25.1:49578, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=157157, /x.x.27.206:5701->/x.x.25.1:49578, endpoint=null, alive=true, type=NONE]
[timestamp] [hz._hzInstance_1_dev.IO.thread-in-1] WARN  com.hazelcast.nio.tcp.TcpIpConnection - [x.x.31.153]:5701 [dev] [3.11.4] Connection[id=157127, /x.x.31.153:5701->/x.x.25.1:42506, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=157127, /x.x.31.153:5701->/x.x.25.1:42506, endpoint=null, alive=true, type=NONE]

Issue was temporary "solved" by roll back to version 3.7: no warnings in Hazelcast logs anymore.

Current findings:

• All exceptions contain IPs that end to x.x.x.1. That are Openshift routers IPs.
• Target ports are 50370, 48700, 60596, 39840, 35046, 59900, etc.

hazelcast.xml:

...
<properties>
    <property name="hazelcast.discovery.enabled">true</property>
    <property name="hazelcast.logging.type">slf4j</property>
</properties>
<network>
    <port port-count="1" auto-increment="false">5701</port>
    <reuse-address>true</reuse-address>
    <join>
        <multicast enabled="false"/>
        <kubernetes enabled="true">
            <namespace>project-name</namespace>
            <service-name>hazelcast-discovery</service-name>
        </kubernetes>
    </join>
</network>

Versions details:

• Hazelcast version: 3.11.4
• OpenShift Master: v3.9.68 (Kubernetes Master: v1.9.1+a0ce1bc657), routers version 3.9

Misc:

• The issue is not reproduced with another cluster (Openshift 3.11, router 3.9, Hazelcast 3.11.4)
• The issue was reproduced in the same cluster with Hazelcast version 3.10.

Questions:

• What is the root cause of these warnings?
• Can we tune our configuration to avoid such connections?

Would it help?

• remove <property name="hazelcast.discovery.enabled">true</property>
• add <tcp-ip enabled="false"></tcp-ip> (I didn’t find in hazelcast documentation what is default value for tcp-ip. Official example of hazelcast.xml explicitly set tcp-ip to false.)

Edit 24.10.2019. Deployment details:

• Application and Hazelcast are in the same project. Application connects to hazelcast by service name: hazelcast:5701
• We use custom livenessProbe/readinessProbe that start bash script every 10s. See below.
• We have also have a route for Hazelcast. Endpoint /hazelcast/rest/cluster showed correct number of members.
• Here is full hazelcast config: https://gitlab.com/snippets/1907166 Are our service settings correct?

bash script for probes:

#!/bin/bash
URL="http://127.0.0.1:5701/hazelcast/health/node-state"
HTTP_RESPONSE=$(curl -m 5 -sS $URL | head -1)
if [ "_${HTTP_RESPONSE}" != "_ACTIVE" ]; then
  echo "failure on ${URL}, response: ${HTTP_RESPONSE}"
  exit 1
fi
exit 0

Edit 25.10.2019:

$ oc describe svc hazelcast-discovery
Name:              hazelcast-discovery
Namespace:         [project-name]
Labels:            app=hazelcast
                   template=hazelcast-statefulset-template
Annotations:       service.alpha.kubernetes.io/tolerate-unready-endpoints=true

Selector:          name=hazelcast-node-cluster
Type:              ClusterIP
IP:                None
Port:              5701-tcp  5701/TCP
TargetPort:        5701/TCP
Endpoints:         x.x.1.45:5701,x.x.12.144:5701,x.x.13.251:5701 and more...
Session Affinity:  None
Events:            <none>

Pods were restarted after the issue, so IPs might differ from the logs. Could it be connected to tolerate-unready-endpoints=true?