Celery 'Error: No nodes replied within time constraint' error - how to debug?

6/10/2019

I'm having an issue that's come up multiple times before, but none of the previous answers seem to help me here.

I'm running Celery (via Docker/Kubernetes) with a Redis back-end. I'm using this command:

celery worker --uid 33 -A finimize_django --loglevel=DEBUG -E

(I've just set it to debug now)

I am using celery==4.3.0 and redis==3.2.1.

Whenever I run celery -A app_name status I get:

Error: No nodes replied within time constraint.

What's weird is Celery seems to be working fine. I can see tasks being processed, and even if I monitor Redis stuff seems to be running successfully. This has also been running fine in production for months, only for this to start happening last week.

It is causing a problem because my liveness probe kills the pod because of this error message.

How can I debug the underlying issue? There is nothing in the log output that is erroring.

Thanks!

-- MDalt
celery
kubernetes
python
redis

1 Answer

9/16/2019

I had the same issue or at least very similar. I've manged to fix it in my project by pinning kombu to version 4.6.3. According to this issue on the github for celery it is an issue with 4.6.4. Really insidious problem to debug, but I hope this helps!

-- PirateNinjas
Source: StackOverflow