Getting Celery tasks repeated multiple times in production while locally not

1

I created an app that connects to a web socket server to receive tasks and schedule them using Celery. My Celery queue display tasks based on the type of the message (text messages first, and then buttons that run the next task if one of them is clicked).
Locally, everything run as expected. But, some tasks are repeated multiple times in production, especially the triggers (buttons). In my production environment, I created a web service for Django and one Celery background worker (to run celery and celerybeat) with a Redis database.

Here are the commands I used to run Django, Celery worker, and beat on Render:

# Start command (Celery background worker)
`celery -A bot.celery worker --beat --scheduler django --loglevel=info --concurrency 4`
# Start command (Django web service)
daphne -b 0.0.0.0 bot.asgi:application

My Django/Celery settings:

CELERY_BROKER_URL = "redis://localhost:6379/0" # localhost replaced with Redis internal URL
CELERY_RESULT_BACKEND = "redis://localhost:6379/1" # localhost replaced with Redis internal URL

TIME_ZONE = 'UTC'

CELERY_ENABLE_UTC = True
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
ASGI_APPLICATION = 'bot.asgi.application'

Some of the worker output:

Nov 1 09:00:15 AM  [2022-11-01 08:00:15,178: INFO/MainProcess] missed heartbeat from celery@srv-cdcl4202i3msb94icl70-5469f7b9d8-gzks7
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,207: INFO/MainProcess] Task schedule_chat_message[574703ea-51d8-4acf-b9e9-202c37d0cbe5] received
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,211: INFO/MainProcess] Task schedule_chat_message[ef2b2dd0-f297-4480-9577-da83a44bc244] received
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,223: INFO/MainProcess] Task schedule_chat_message[a3619feb-9768-4e6b-b8be-966c335eae5f] received
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,182: INFO/ForkPoolWorker-2] Task schedule_chat_message[a3619feb-9768-4e6b-b8be-966c335eae5f] succeeded in 0.9560789109673351s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,272: INFO/ForkPoolWorker-3] Task schedule_chat_message[574703ea-51d8-4acf-b9e9-202c37d0cbe5] succeeded in 1.0607709939940833s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,280: INFO/ForkPoolWorker-4] Task schedule_chat_message[ef2b2dd0-f297-4480-9577-da83a44bc244] succeeded in 1.0533532970002852s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,936: INFO/Beat] DatabaseScheduler: Schedule changed.
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,274: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.211860+00:00 (send_out_chat_message)
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,289: INFO/MainProcess] Task send_out_chat_message[37854a6c-d20a-45b1-99ba-37b22d818af4] received
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,562: INFO/ForkPoolWorker-3] Task send_out_chat_message[37854a6c-d20a-45b1-99ba-37b22d818af4] succeeded in 0.27148923301137984s: None
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,090: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.227478+00:00 (send_out_chat_message)
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,096: INFO/MainProcess] Task send_out_chat_message[80ca730b-0ff4-43df-b3ec-2758b0b493df] received
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,276: INFO/ForkPoolWorker-3] Task send_out_chat_message[80ca730b-0ff4-43df-b3ec-2758b0b493df] succeeded in 0.1784440579940565s: None
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,359: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.226634+00:00 (send_out_chat_message)
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,365: INFO/MainProcess] Task send_out_chat_message[a1bcff89-78af-41b6-8e81-95aa5d3172d3] received
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,561: INFO/ForkPoolWorker-3] Task send_out_chat_message[a1bcff89-78af-41b6-8e81-95aa5d3172d3] succeeded in 0.19544574100291356s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,089: INFO/MainProcess] Task schedule_chat_message[a83a6683-1750-422f-8494-7226152249b4] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,099: INFO/MainProcess] Task schedule_chat_message[d983a9d0-2243-4a87-9844-5f2e986f7626] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,103: INFO/MainProcess] Task schedule_chat_message[f30d05ec-257f-4516-bfcb-5880b85d3e85] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,109: INFO/MainProcess] Task schedule_chat_message[f8e1a7a3-21a9-41f3-a8d9-9ceae4f0c10f] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,864: INFO/ForkPoolWorker-3] Task schedule_chat_message[a83a6683-1750-422f-8494-7226152249b4] succeeded in 0.773912389006s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,869: INFO/ForkPoolWorker-4] Task schedule_chat_message[d983a9d0-2243-4a87-9844-5f2e986f7626] succeeded in 0.7689848589943722s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,968: INFO/ForkPoolWorker-2] Task schedule_chat_message[f30d05ec-257f-4516-bfcb-5880b85d3e85] succeeded in 0.8630019359989092s: None

I’m following Render Celery documentation, and I don’t know if there is something wrong with the way I deployed the app.

Your help is much appreciated!

2

Hi Rochdi,

This sounds like a question that might get more traction in a Celery user group. A quick google search tells me this is a common issue with Celery with various solutions:

Again, I think a Django/Celery community might yield better information.

Regards,

Matt

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.