Background worker

#1

Hi
I am trying to implement celery with my Django application (terradiem).
I have already activated a Redis server (redisgreen.com) and it works.
I would like to activate a worker together with the terradiem web service. Is it something I can do ? What start command should I enter in the Settings of render ?
I tried to create a background worker (terraworker) but I don’t think it can work as it cannot access to my Django views and to the database.
Thanks a lot for your help
Best
Julien

#2

Hi Julien! Have you tried following the steps laid out in our guide to deploying celery on Render? The key step is creating a CELERY_BROKER_URL environment variable that holds the service address for your redis server. You can find your service address in the Render dashboard (it’ll be something like redis:6379).

#3

Hi David ! Thank you for your answer. Yes, I created a REDIS_URL environment variable (pointing to redisgreen) which I use in my Django application. The connection is fine.

I know need to activate a worker. As my worker needs Django views and models, I guess it should be activated together with the Django application and cannot be separated ? This is the part I am struggling to manage…

With Heroku, I had a “Procfile” file to activate both a gunicorn and a worker.

Thanks a lot
Julien

#4

Got it. You might benefit from using Render’s infrastructure as code offering. As you can see from the render.yaml spec, with IaC you can define and activate multiple services in the same file. It’s similar to a Procfile, but much more expressive.

Make sure you click YAML and then New from YAML when you go to apply your render.yaml file in the dashboard. And don’t hesitate to ask any follow-up questions!

#5

Great David.
I was able to deploy an IAC, thank you for that. Unfortunately, the services do not seem to talk to each others. I am actually not very sure how I should do that.

  1. I deployed a web service (terradiem), based on Django, which contains all Celery views
  2. I deployed the same code with a worker and a --beat instruction
  3. I deployed a message broker on redisgreen.
    There is no error, but the worker does not seem to receive the messages.
    Any clue ?
    Please find the content of the render.yalm file below.
    Best,
    Julien

services:

Python web service

  • type: web
    name: terradiem
    env: python
    plan: starter
    repo: https://github.com/jpuaux/terradiem.git
    #healthCheckPath: /health
    region: frankfurt
    branch: master
    numInstances: 3
    buildCommand: pip install -r requirements.txt
    startCommand: gunicorn terradiem.wsgi:application
    domains:
    • www.capelight.com
      envVars:
    • fromGroup: terra-settings
    • key: DATABASE_URL
      fromDatabase:
      name: terradiem-db-new
      property: connectionString
      autoDeploy: true # optional

Background worker that consumes a queue

  • type: worker
    name: terra-queue
    env: python
    buildCommand: pip install -r requirements.txt # contains installation of requirements.txt
    startCommand: celery -A terradiem worker --beat -l info
    envVars:
    • fromGroup: terra-settings
      autoDeploy: true # optional
#6

Hi @jpuaux, sorry I missed your response! Were you able to debug this? If not, I’ll be happy to assist.