Django - Google cloud - pubsub - celery - kubernetes problem

3/30/2021

I have microservice for email sending in cloud function on some GCP project. On different GCP project i have Ecommerce application deployed with kubernetes. I want to send mails after some activity on ecommerce via my microservice i use pub/sub for communication. I have problem with publishing messages in celery tasks. When I publish message using managment command everything is ok, but when i trigger celery task from managment command there is a problem. code image This will cause an error like in the picture below (Process "ForkPoolWorker-10" pid:276 exited with "signal 11 (SIGSEGV)") But werid thing - message is being send, and recipient recive this message. error image

With other tasks there are no such problems. If i comment line " future =publisher.publish(topic_path, json_data.encode("utf-8")) " There is no error. I know signal 11 (SIGSEGV) is connected to memory but i already make my kubernetes machines stronger so i don't think this is the issue.

Logs from GCP Weird is also why ForkPoolWorker-7 succeeded with this task, but 'ForkPoolWorker-3' pid:227 exited with 'signal 11'

-- janek_axpo
celery
django-celery
google-cloud-pubsub
kubernetes
redis

1 Answer

3/31/2021

Ok, problem was with that publisher PublisherClient creating multi threads and celery don't work correct with threads.

-- janek_axpo
Source: StackOverflow