Critical Worker time out in Celery-Flask server application

1

I have a Flask app server with celery worker for task queue using Redis database.

This is the Render.yaml file for the service

services:
  - type: worker
    name: celery-worker
    region: oregon
    runtime: python
    buildCommand: "pip install -r requirements.txt"
    startCommand: "celery -A celeryconfig.celery_app worker -Q default -l info --pool=solo --concurrency=8"
    autoDeploy: false
    envVars:
      - key: REDIS_URL
        fromService:
          name: celery-redis
          type: redis
          property: connectionString
  - type: web
    name: app
    region: oregon
    runtime: python
    buildCommand: "pip install -r requirements.txt"
    startCommand: "gunicorn app:app -t 60 --keep-alive 60"
    autoDeploy: false
    envVars:
      - key: REDIS_URL
        fromService:
          name: celery-redis
          type: redis
          property: connectionString
  - type: web
    name: flower
    region: oregon
    plan: free
    runtime: python
    buildCommand: "pip install -r requirements.txt"
    startCommand: "celery -A celeryconfig.celery_app flower"
    autoDeploy: false
    envVars:
      - key: REDIS_URL
        fromService:
          type: redis
          name: celery-redis
          property: connectionString
  - type: redis
    name: celery-redis
    region: oregon
    plan: starter # we choose a plan with persistence to ensure tasks are not lost upon restart
    maxmemoryPolicy: noeviction # recommended policy for queues
    ipAllowList: [] # only allow internal connections

and this is a /submit endpoint in the Flask server

@app.route('/submit', methods=['POST'])
def submit():
    json_data = request.get_json()
    task = generate_word_groups_from_input.apply_async(args=[json_data])
    return jsonify({"task_id": task.id}), 202

From the client side, when I access the /submit endpoint by submitting some data as json. I am getting the following error in the Render log

[2024-08-15 03:31:59 +0000] [90] [CRITICAL] WORKER TIMEOUT (pid:109)
[2024-08-15 03:31:59 +0000] [109] [INFO] Worker exiting (pid: 109)
[2024-08-15 03:32:00 +0000] [90] [WARNING] Worker with pid 109 was terminated due to signal 9
[2024-08-15 03:32:00 +0000] [110] [INFO] Booting worker with pid: 110
==> Detected service running on port 10000
==> Docs on specifying a port: https://render.com/docs/web-services#port-binding
[2024-08-15 03:47:28 +0000] [90] [INFO] Handling signal: term
[2024-08-15 03:47:28 +0000] [110] [INFO] Worker exiting (pid: 110)
[2024-08-15 03:47:31 +0000] [90] [INFO] Shutting down: Master

I also get this error on console of my browser

ailed to load resource: the server responded with a status of 502 ()Understand this error
index.js:265 Error: Error: Server responded with an error
    at index.js:250:19
(anonymous) @ index.js:265Understand this error
index.js:269 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'style')
    at index.js:269:26

I never faced any problem in the local deployment.

I read the previous posts in this regard and changed the timeout for gunicorn

startCommand: "gunicorn app:app -t 60 --keep-alive 60"

But still the problem has not been solved.

Can someone please help.

2

Hey,

A WORKER TIMEOUT log message will likely be coming from Gunicorn.

You would either need to:

It looks like you’ve already set a timeout value for Gunicorn, but you might need to increase it or check if something is causing a delay between the endpoint being hit and your code executing. You’ll need to debug the endpoint you’re calling—maybe add some more logging to pinpoint where the process is taking longer than expected.

Every app is unique, with different purposes, styles, and implementations, so we can’t provide specific debugging help. Since you know your app best, you’ll have the best insight into where the issue might be.

Jérémy.
Render Support, UTC+3

3

Thank you Jeremy.

My app has three services the Flask app, the Celery Worker and the Flower service for visualizing Celery worker.

In local development, i used to open three separate terminals for these three and could see the logs separately. But here I am not able to see the logs from Celery and Flower.

Additionally I am not able to access the Flower page. In local dev it was always at port-5555, but here on render I understand that the port is dynamically assigned. So how will I know which is the port for Flower in my application

4

Hey,

Are you using 3 different services on Render for each of your application?

Jérémy.
Render Support, UTC+3

5

No, all the three are in the same webservice.

I followed the instructions in this regard from the Render web page - Deploy a Celery Worker – Render Docs

Here I initialized all there services on the same render.yam file. But i did not create a new Blueprint, but deployed as a web service.
I assume this might be the problem, either deploy the repo as a blueprint or deploy as three separate web services.

What do you suggest for the sake of efficiency ?

6

Hey,

render.yaml files can deploy multiple services, but you’ll need to create a Blueprint to sync and set up those services. The tutorial you linked suggests deploying the different applications in 3 separate services, including a Redis database that can’t be deployed on the same service type as the other two.

So yes, whether you choose to use Blueprints or not, you’ll need to deploy three different service types here to properly segment each application’s purpose.

Jérémy.
Render Support, UTC+3

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.