We are running spring-boot microservices on k8s on Amazon EC2, using undertow as our embedded web server.

Whenever - for whatever reason - our downstream services are overwhelmed by incoming requests, and the downstream pods' worker queue grows too large (i've seen this issue happen at 400-ish), then spring-boot stops processing queued requests completely and the app goes silent.

Monitoring the queue size via JMX we can see that the queue size continues to grow as more requests are queued by the IO worker - but by this point no queued requests are ever processed by any worker threads.

We can't see any log output or anything to indicate why this might be happening.

This issue cascades upstream, whereby the paralyzed downstream pods cause the traffic in the upstream pods to experience the same issue and they too become unresponsive - even when we turn off all incoming traffic through the API gateway.

To resolve the issue we have to stop incoming traffic upstream, and then kill all of the affected pods, before bringing them back up in greater numbers and turning the traffic back on.

Does anyone have any ideas about this? Is it expected behaviour? If so, how can we make undertow refuse connections before the queue size grows too large and kills the service? If not, whhat is causing this behaviour?

Many thanks. Aaron.