We are running hazelcast in embedded mode and the application is running in kubernetes cluster. We are using Kubernetes API for discovery.
It was all working fine and now we just started using envoy as sidecar for SSL. Now for both inbound and outbound on TCP at hazelcast port 5701 we have enabled TLS in envoy but are yet to do changes for kubernetes API call.
Right now we are getting below Exception :
"class":"com.hazelcast.internal.cluster.impl.DiscoveryJoiner","thread_name":"main","type":"log","data_version":2,"description":"[10.22.69.149]:5701 [dev] [3.9.4] Operation: [get] for kind: [Endpoints] with name: [servicename] in namespace: [namespace] failed.","stack_trace":"j.n.ssl.SSLException: Unrecognized SSL message, plaintext connection?\n\tat s.s.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)\n\tat s.s.ssl.InputRecord.read(InputRecord.java:527)\n\tat s.s.s.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)\n\tat s.s.s.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)\n\tat s.s.s.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)\n\tat s.s.s.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)\n\tat o.i.c.RealConnection.connectTls(RealConnection.java:281)\n\tat o.i.c.RealConnection.establishProtocol(RealConnection.java:251)\n\tat o.i.c.RealConnection.connect(RealConnection.java:151)\n\tat
Can someone help with the overall changes which should be needed for Hazelcast k8s discovery using APIs with envoy as sidecar ?
You can find an example config below for how to deploy Hazelcast with Envoy sidecar and use it with mTLS.
https://github.com/hazelcast/hazelcast-kubernetes/issues/118#issuecomment-553588983
If you want to achieve the same with an embedded architecture, you need to create a headless kubernetes service besides your microservice's kubernetes service. Then you need to give headless service name to hazelcast-kubernetes plugin service-name parameter.
You can find more info on hazelcast-kubernetes plugin README.md file.
EDIT: Hazelcast-Istio-SpringBoot step-by-step guide can be found here.