I have a django web service and a repo that holds the code and all the tasks I wrote for my website remotetechjob.co.
On another repo, following Render’s docs, I’ve deployed a repo into a celery background worker service with the boilerplate code.
I don’t believe this is documented anywhere, how do I use my seperate deployed celery worker to run tasks from my repo remotetechjob?
I saw a post that says that we should use render.yaml files in the root of each repo and the start command should point to remotetechjob, however:
==> Starting service with 'celery -A remotetechjob worker --beat -l info'
Oct 20 12:27:16 AM Error:
Oct 20 12:27:16 AM Unable to load celery application.
Oct 20 12:27:16 AM The module remotetechjob was not found.
Hi, have you tried deploying with your remotetechjob repo instead of our template repo? The template repo just has one simple example task: celery/tasks.py at master · render-examples/celery · GitHub. We suggest either forking the template repo and adding your own tasks, or using your existing repo and configuring the background worker in a similar manner to what we describe in the doc: Deploy a Celery Worker | Render
Is this really the right way? Then the background worker will have so much unnecessary code since it is connected to the main app repo, have to install unneccesarry packages etc.
Not to mention when I use this setup with my Django app, the background worker is the one performing the tasks, which should happen on the main web service as that one likely has more processing power?
This is the recommended way to do it. Have your web service take requests and then push anything that cannot be handled quickly to a background worker via a queue to be handled asynchronously. This is because you want to avoid your HTTP request handlers being tied up handling long-running tasks, as they cannot then respond to any additional requests your web service receives.
Do you have to run Django and Celery as two separate services? No, this depends on how much traffic and tasks you expect your service to receive. Running Django/Gunicorn and Celery is okay if you start them both in a single web service, provided you don’t consume all the service resources. However, this will mean that the Django and Celery will be tightly coupled, meaning if one of those services crashes, it will impact the other as your service may be restarted. Imagine the situation where a Celery task causes an error that crashes Celery, you don’t want your web service going down if this happens.
This is why it is recommended to run them separately.
We get why we need to offload tasks, that’s why we have built our apps with Celery for time consuming functions or API calls.
The frustration is having to upload a duplicate Django app just to run a simple Celery task. How does this scale? If I want 10 workers, then I have to upload my Django project 10 times? It is a lot of unnecessary bulk and server space.
