I've made a SpringBoot application that authenticate with Gloud Storage and performs action on it. It works locally, but when I deploy it on my GKE as a Pod, it suffers some errors.
I have a VPC environment Where I have a Google Cloud Storage, and a Kubernetes Cluster that will run some Spring Boot applications that performs actions on it through com.google.cloud.storage library.
It has Istio enabled for the Cluster and also a Gateway Resource with Secure HTTPS which targets the Ingress Load Balancer as defined here:
https://istio.io/docs/tasks/traffic-management/secure-ingress/sds/
Then my pods all are being reached through a Virtual Service of this Gateway, and it's working fine since they have the Istio-Car container on it and then I can reach them from outside.
So, I have configured this application in DEV environment to get the Credentials from the ENV values:
ENV GOOGLE_APPLICATION_CREDENTIALS="/app/service-account.json"I know it's not safe, but just wanna make sure it's authenticating. And as I can see through the logs, it is.
As my code manipulates Storages, an Object of this type is needed, I get one by doing so:
this.storage = StorageOptions.getDefaultInstance().getService();It works fine when running locally. But when I try the same on the Api now running inside the Pod container on GKE, whenever I try to make some interaction to the Storage it returns me some errors like:
[2019-04-25T03:17:40.040Z] [org.apache.juli.logging.DirectJDKLog] [http-nio-8080-exec-1] [175] [ERROR] transactionId=d781f21a-b741-42f0-84e2-60d59b4e1f0a Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.google.cloud.storage.StorageException: Remote host closed connection during handshake] with root cause
java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
...
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:142)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1011)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:358)
... 65 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
...
Looks like when I make the call from the Pod, it is expected some extra Https configuration. I don't know right.
So what I'm wondering is:
If this is some kind of Firewall Rule blocking this call from my Pod to "outside" (What is weird since they run on the same network, or at least I thought so).
If it's because of the Gateway I defined that is kind of not enabling this Pod
Or if I need to create the Storage Object using some custom HTTP configurations as can be seen on this reference: https://googleapis.github.io/google-cloud-java/google-cloud-clients/apidocs/com/google/cloud/storage/StorageOptions.html#getDefaultHttpTransportOptions--
My knowledge of HTTPs and Secure conections is not very good, so maybe my lacking on concept on this area is making me not be able to see something obvious.
If some one have any idea on what maybe causing this, I would appreciate very much.
Solved it. It was really Istio.
I didn't know that we need a Service Entry resource to define what inbound and outbound calls OUTSIDE the mesh.
So, even that GCS is in the same project of the GKE, they are threated as completely separated services.
Just had to create it and everything worked fine:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
namespace: {{ cloud_app_namespace }}
name: external-google-api
spec:
hosts:
- "www.googleapis.com"
- "metadata.google.internal"
- "storage.googleapis.com"
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: HTTPS
- number: 80
name: http
protocol: HTTPhttps://istio.io/docs/reference/config/networking/v1alpha3/service-entry/
EDIT
I have disabled the Istio Injection on the namespace I were deploying the applications, by simply using:
kubectl label namespace default istio-injection=disabled --overwriteThen redeployed the application and tried a curl there, and it worked fine.
My doubt now is: I though Istio only intercept on it's gateway layer, and after that the message keeps untouched, but this is not what seems to be working. Apparently he embbed some SSL layer on the request that my application doesn't do/require. So sould I need to change my application just to fit on the service mesh requirements?