I am trying deploy a pod to a kubernetes 11.2 cluster and seeing this error.

Weird part is that this workers fine on another cluster (different envs) and config is same exact. Only thing, I noticed is with this failing cluster, I felt like the nodes were little slow to login. But everything seems to be deploying correctly, except for this error.

There are no changes to the code or the config to deploy, what could be the reason for this to happen only on this particular cluster and not other envs. (It works on dev, test and pre-prod) doesn't work on prod, totally dazed I am and not sure if this is infrastructure issue possibly to the kubernetes config or the application needs to be able to handle this. another thing is that i don't see the nodes going down or any errors related to the memory or lack of resources, like disk pressure and such.

any advise would be highly appreciated.

2018-09-02 18:29:51.048  INFO 29 --- [-nio-443-exec-6] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2018-09-02 18:29:56.493 ERROR 29 --- [-nio-443-exec-6] c.t.s.est.server.ServletDispatcher       : An unexpected error occured while processing a request on the following uri /.well-known/est/App Service/senroll

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
...
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_171]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_171]
...
2018-09-02 18:29:56.499 ERROR 29 --- [-nio-443-exec-6] o.a.c.c.C.[.[.[/].[servletDispatcher]    : Servlet.service() for servlet [servletDispatcher] in context with path [] threw exception

java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
    at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:472) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
    at com.trilliantnetworks.security.est.server.ServletDispatcher.unexpectedError(ServletDispatcher.java:230) ~[est-servlet-1.0.1-SNAPSHOT.jar!/:na]
    at com.trilliantnetworks.security.est.server.ServletDispatcher.doPost(ServletDispatcher.java:211) ~[est-servlet-1.0.1-SNAPSHOT.jar!/:na]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
...
...
2018-09-02 18:29:56.527 ERROR 29 --- [-nio-443-exec-6] o.a.c.c.C.[Tomcat].[localhost]           : Exception Processing ErrorPage[errorCode=0, location=/error]

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:321) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
    at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]

The nodes are fine. They are not going down or anything. I also looked at the events to see if any errors related to memory show up, but unfortunately nothing.

So, I am seeing this error in one of the pods, which installs some things to the database. This is the first step and should not fail

Mainly this is happening, the read timeout. So, I am wondering if there is some kind of timeout that  I can set in the cluster, to wait for the API response, little longer.

2018-09-02 18:33:35.818  INFO 29 --- [           main] c.t.s.c.i.r.impl.ExternalSignerService   : Exception while generatePermanentKeyStore:java.net.SocketTimeoutException: Read timed out