I am trying to connect to an Apache Ignite (2.8.0) deployed on Kubernetes Cluster. The cluster itself seems fine, as i am able to exec into the pods and access via sqlline tool as shown below
root@ip-172-17-28-68:/opt/apache-ignite-2.8.0-bin# kubectl exec -it ignite-cluster-6d69696b67-8vvmm /bin/bash
bash-4.4# apache-ignite/bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://127.0.0.1:10800/
issuing: !connect jdbc:ignite:thin://127.0.0.1:10800/ '' '' org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://127.0.0.1:10800/
Connected to: Apache Ignite (version 2.8.0#20200226-sha1:341b01df)
Driver: Apache Ignite Thin JDBC Driver (version 2.8.0#20200226-sha1:341b01df)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0
0: jdbc:ignite:thin://127.0.0.1:10800/>
However if i try connecting from the external LB it gives the following error.
root@ip-172-17-28-68:/opt/apache-ignite-2.8.0-bin/bin# ./sqlline.sh --verbose=true -u jdbc:ignite:thin://abc-123.us-east-1.elb.amazonaws.com:10800
issuing: !connect jdbc:ignite:thin://abc-123.us-east-1.elb.amazonaws.com:10800 '' '' org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://abc-123.us-east-1.elb.amazonaws.com:10800
Error: Failed to connect to server [url=jdbc:ignite:thin://weiury734ry34ry34urt.us-east-1.elb.amazonaws.com:10800/PUBLIC] (state=08001,code=0)
java.sql.SQLException: Failed to connect to server [url=jdbc:ignite:thin://abc123.us-east-1.elb.amazonaws.com:10800/PUBLIC]
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.handleConnectExceptions(JdbcThinConnection.java:1529)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.connectInCommonMode(JdbcThinConnection.java:1506)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.ensureConnected(JdbcThinConnection.java:231)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.<init>(JdbcThinConnection.java:210)
at org.apache.ignite.IgniteJdbcThinDriver.connect(IgniteJdbcThinDriver.java:154)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
at sqlline.Commands.connect(Commands.java:1095)
at sqlline.Commands.connect(Commands.java:1001)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:791)
at sqlline.SqlLine.initArgs(SqlLine.java:566)
at sqlline.SqlLine.begin(SqlLine.java:643)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
Suppressed: java.io.IOException: Failed to read incoming message (not enough data).
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:546)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:524)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.handshake(JdbcThinTcpIo.java:266)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.<init>(JdbcThinTcpIo.java:212)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.connectInCommonMode(JdbcThinConnection.java:1477)
... 17 more
Suppressed: java.io.IOException: Failed to read incoming message (not enough data).
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:546)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:524)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.handshake(JdbcThinTcpIo.java:266)
at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.<init>(JdbcThinTcpIo.java:212)
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.connectInCommonMode(JdbcThinConnection.java:1477)
... 17 more
sqlline version 1.3.0
Interestingly telnet to the port via LB says connected as shown below.
root@ip-172-17-28-68:/opt/apache-ignite-2.8.0-bin/bin# telnet abc-123.us-east-1.elb.amazonaws.com 10800
Trying 3.209.73.243...
Connected to abc-123.us-east-1.elb.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.
Why is it not able to connect via jdbc client even though telnet works on port 10800? I also tried to connect via DBeaver, but the error is the same.
Here are the service and deployment yaml used in Kubernetes EKS Deployment.
Ignite-service.yaml
apiVersion: v1
kind: Service
metadata:
name: ignite
namespace: default
labels:
app: ignite
spec:
type: LoadBalancer
ports:
- name: rest
port: 8080
protocol: TCP
targetPort: 8080
- name: sql
port: 10800
protocol: TCP
targetPort: 10800
- name: thinclients
port: 10900
protocol: TCP
targetPort: 10900
selector:
app: ignite
Ignite-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ignite-cluster
namespace: default
spec:
selector:
matchLabels:
app: ignite
replicas: 2
template:
metadata:
labels:
app: ignite
spec:
serviceAccount: ignite
containers:
- name: ignite-node
image: apacheignite/ignite:2.8.0
env:
- name: OPTION_LIBS
value: ignite-kubernetes
- name: CONFIG_URI
value: https://raw.githubusercontent.com/apache/ignite/master/modules/kubernetes/config/example-kube.xml
ports:
- containerPort: 11211 # REST port number.
- containerPort: 47100 # communication SPI port number.
- containerPort: 47500 # discovery SPI port number.
- containerPort: 49112 # JMX port number.
- containerPort: 10800 # SQL port number.
check the service itself: kubectl describe services my-service
make sure an external IP is provided and works.
Restrict K8 to one pod and re-try.
Set the externalTrafficPolicy and the pod selector
externalTrafficPolicy: Local
selector:
statefulset.kubernetes.io/pod-name: pod-name
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-name-label
A thin client needs to communicate to a specific pod/server
Check the server logs to make sure that data is flowing.