I checked Azure AKS log by kubectl logs <-POD-> which I log the message ID at the beginning when start to handle it. However, there was no log to log down a few messages which went to Deadletter suddenly. I tried to put them to normal queue again but faced similar result.
Why is it happening? How to solve it?
Below is log that I can see sometimes but it should be also resume processing messages after a period of time. (I ran 4 pods in parallel)
Error for serviceBusService.receiveQueueMessage()
Error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
at Function.ServiceClient._normalizeError (/usr/src/app/node_modules/azure-common/lib/services/serviceclient.js:757:12)
at ServiceBusService.ServiceClient._processResponse (/usr/src/app/node_modules/azure-common/lib/services/serviceclient.js:455:41)
at Request.processResponseCallback [as _callback] (/usr/src/app/node_modules/azure-common/lib/services/serviceclient.js:213:35)
at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
at Request.emit (events.js:203:13)
at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1161:10)
at Request.emit (events.js:203:13)
at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1083:12)
at Object.onceWrapper (events.js:291:20)
at IncomingMessage.emit (events.js:208:15)Looks like you may be hitting into a transient exception. On such errors, you could setup an retry with a backoff until it succeeds.
I believe in most cases such errors should resolve in a couple of retries unless there is a major problem with the service at the time.