EDIT: added status code received on client

EDIT 2: The issue seems to be random in nature and when it occurs it seems to affect particular networks at a time only. Recently a team member in a different country faced the issue while at the same time it is working without issue from my own network. Earlier I was facing the issue on my network while I was able to run the same call via curl after ssh'ing into a cloud server.

EDIT 3: The issue is happening on requests other than PUT too

I have Spring Boot based REST API implementation that is giving a strange issue when deployed on Google Container Engine. Some key points about the issue:

1. Works locally on embedded tomcat without any issue.
2. On container engine, the issue is happening only on Http PUT request. If I change the method to PATCH the error goes away.
3. I have another application with same stack deployed on container engine for several months, and never faced this issue on that application.
4. At client I receive 504 Gateway Timeout after some delay, which seems to suggest that loadbalancer was waiting for webserver's response while webserver was still waiting for whole request to arrive.

I suspect that I am hitting a bug in load balancing/network layer of GKE.

Following is the exception trace generated by spring:

2016-10-03 11:56:51.067 ERROR 1 --- [nio-8090-exec-3] o.s.d.r.w.RepositoryRestExceptionHandler : Could not read an object of type class com.mycompany.Incident from the request!; nested exception is java.io.EOFException: Unexpected EOF read on the socket

org.springframework.http.converter.HttpMessageNotReadableException: Could not read an object of type class com.mycompany.Incident from the request!; nested exception is java.io.EOFException: Unexpected EOF read on the socket
    at org.springframework.data.rest.webmvc.config.PersistentEntityResourceHandlerMethodArgumentResolver.readPutForUpdate(PersistentEntityResourceHandlerMethodArgumentResolver.java:220) ~[spring-data-rest-webmvc-2.5.2.RELEASE.jar!/:na]
    at org.springframework.data.rest.webmvc.config.PersistentEntityResourceHandlerMethodArgumentResolver.read(PersistentEntityResourceHandlerMethodArgumentResolver.java:186) ~[spring-data-rest-webmvc-2.5.2.RELEASE.jar!/:na]
    at org.springframework.data.rest.webmvc.config.PersistentEntityResourceHandlerMethodArgumentResolver.resolveArgument(PersistentEntityResourceHandlerMethodArgumentResolver.java:138) ~[spring-data-rest-webmvc-2.5.2.RELEASE.jar!/:na]
    at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:121) ~[spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161) ~[spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128) ~[spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:883) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:651) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.1.1.RELEASE.jar!/:4.1.1.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:522) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1110) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.io.EOFException: Unexpected EOF read on the socket
    at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:747) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.http11.Http11InputBuffer.access$400(Http11InputBuffer.java:38) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1073) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:100) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:303) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.coyote.Request.doRead(Request.java:511) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:318) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.tomcat.util.buf.ByteChunk.checkEof(ByteChunk.java:397) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:379) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:338) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:189) ~[tomcat-embed-core-8.5.4.jar!/:8.5.4]
    at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:517) ~[jackson-core-2.8.1.jar!/:2.8.1]
    at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:126) ~[jackson-core-2.8.1.jar!/:2.8.1]
    at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:243) ~[jackson-core-2.8.1.jar!/:2.8.1]
    at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1271) ~[jackson-core-2.8.1.jar!/:2.8.1]
    at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:810) ~[jackson-core-2.8.1.jar!/:2.8.1]
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2312) ~[jackson-databind-2.8.1.jar!/:2.8.1]
    at org.springframework.data.rest.webmvc.config.PersistentEntityResourceHandlerMethodArgumentResolver.readPutForUpdate(PersistentEntityResourceHandlerMethodArgumentResolver.java:215) ~[spring-data-rest-webmvc-2.5.2.RELEASE.jar!/:na]
    ... 88 common frames omitted