Unable to connect to Apache Ignite (jdbc) through LB in AWS EKS

6/1/2020

I am trying to connect to an Apache Ignite (2.8.0) deployed on Kubernetes Cluster. The cluster itself seems fine, as i am able to exec into the pods and access via sqlline tool as shown below

root@ip-172-17-28-68:/opt/apache-ignite-2.8.0-bin# kubectl exec -it ignite-cluster-6d69696b67-8vvmm /bin/bash

bash-4.4# apache-ignite/bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://127.0.0.1:10800/
issuing: !connect jdbc:ignite:thin://127.0.0.1:10800/ '' '' org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://127.0.0.1:10800/
Connected to: Apache Ignite (version 2.8.0#20200226-sha1:341b01df)
Driver: Apache Ignite Thin JDBC Driver (version 2.8.0#20200226-sha1:341b01df)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0
0: jdbc:ignite:thin://127.0.0.1:10800/>

However if i try connecting from the external LB it gives the following error.

root@ip-172-17-28-68:/opt/apache-ignite-2.8.0-bin/bin# ./sqlline.sh --verbose=true -u jdbc:ignite:thin://abc-123.us-east-1.elb.amazonaws.com:10800
issuing: !connect jdbc:ignite:thin://abc-123.us-east-1.elb.amazonaws.com:10800 '' '' org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://abc-123.us-east-1.elb.amazonaws.com:10800
Error: Failed to connect to server [url=jdbc:ignite:thin://weiury734ry34ry34urt.us-east-1.elb.amazonaws.com:10800/PUBLIC] (state=08001,code=0)
java.sql.SQLException: Failed to connect to server [url=jdbc:ignite:thin://abc123.us-east-1.elb.amazonaws.com:10800/PUBLIC]
        at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.handleConnectExceptions(JdbcThinConnection.java:1529)
        at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.connectInCommonMode(JdbcThinConnection.java:1506)
        at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.ensureConnected(JdbcThinConnection.java:231)
        at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.<init>(JdbcThinConnection.java:210)
        at org.apache.ignite.IgniteJdbcThinDriver.connect(IgniteJdbcThinDriver.java:154)
        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
        at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
        at sqlline.Commands.connect(Commands.java:1095)
        at sqlline.Commands.connect(Commands.java:1001)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
        at sqlline.SqlLine.dispatch(SqlLine.java:791)
        at sqlline.SqlLine.initArgs(SqlLine.java:566)
        at sqlline.SqlLine.begin(SqlLine.java:643)
        at sqlline.SqlLine.start(SqlLine.java:373)
        at sqlline.SqlLine.main(SqlLine.java:265)
        Suppressed: java.io.IOException: Failed to read incoming message (not enough data).
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:546)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:524)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.handshake(JdbcThinTcpIo.java:266)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.<init>(JdbcThinTcpIo.java:212)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.connectInCommonMode(JdbcThinConnection.java:1477)
                ... 17 more
        Suppressed: java.io.IOException: Failed to read incoming message (not enough data).
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:546)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read(JdbcThinTcpIo.java:524)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.handshake(JdbcThinTcpIo.java:266)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.<init>(JdbcThinTcpIo.java:212)
                at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.connectInCommonMode(JdbcThinConnection.java:1477)
                ... 17 more
sqlline version 1.3.0

Interestingly telnet to the port via LB says connected as shown below.

root@ip-172-17-28-68:/opt/apache-ignite-2.8.0-bin/bin# telnet abc-123.us-east-1.elb.amazonaws.com 10800
Trying 3.209.73.243...
Connected to abc-123.us-east-1.elb.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.

Why is it not able to connect via jdbc client even though telnet works on port 10800? I also tried to connect via DBeaver, but the error is the same.

Here are the service and deployment yaml used in Kubernetes EKS Deployment.

Ignite-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ignite
  namespace: default
  labels:
    app: ignite
spec:
  type: LoadBalancer
  ports:
    - name: rest
      port: 8080
      protocol: TCP
      targetPort: 8080
    - name: sql
      port: 10800
      protocol: TCP
      targetPort: 10800
    - name: thinclients
      port: 10900
      protocol: TCP
      targetPort: 10900
  selector:
    app: ignite

Ignite-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ignite-cluster
  namespace: default
spec:
  selector:
    matchLabels:
      app: ignite
  replicas: 2
  template:
    metadata:
      labels:
        app: ignite
    spec:
      serviceAccount: ignite
      containers:
      - name: ignite-node
        image: apacheignite/ignite:2.8.0
        env:
        - name: OPTION_LIBS
          value: ignite-kubernetes
        - name: CONFIG_URI
          value: https://raw.githubusercontent.com/apache/ignite/master/modules/kubernetes/config/example-kube.xml
        ports:
        - containerPort: 11211 # REST port number.
        - containerPort: 47100 # communication SPI port number.
        - containerPort: 47500 # discovery SPI port number.
        - containerPort: 49112 # JMX port number.
        - containerPort: 10800 # SQL port number.
-- Sandeep Shanbhag
amazon-eks
ignite
jdbc
kubernetes

1 Answer

6/1/2020

check the service itself: kubectl describe services my-service

https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#finding-your-ip-address

make sure an external IP is provided and works.

Restrict K8 to one pod and re-try.

Set the externalTrafficPolicy and the pod selector

externalTrafficPolicy: Local
  selector:
    statefulset.kubernetes.io/pod-name: pod-name

https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-name-label

A thin client needs to communicate to a specific pod/server

Check the server logs to make sure that data is flowing.

-- Alex K
Source: StackOverflow