Cron vs Job vs Celery?

1

I have a couple of low priority regular tasks that I want my django web service to run.

  • Pulling data from external API (once nightly)
  • Generating alert emails to users (twice daily)
  • Sending queued emails (somewhat frequently, as emails get generated)

Reading through the Render documents, it sounds like there are a few methods I could use to accomplish this, but I’m not sure which is best.

  • cron: seems the simplest, but sounds like (at least a year ago Render cron jobs can not directly interact with web services. I could make a mirror of my web service that the job runs in, but that seems super wasteful. Alternatively, I could make a web endpoint that the cron job is pointed at. Seems better, but hacky. Also, the cost for these would be $1/mo per cron job? (Plus usage, but that should be pretty low for my jobs, as long as I didn’t re-deploy for each job…)
  • jobs: This sounds more like what the right solution would be, as it is directly tied into the already deployed web service, but it does not appear that jobs are schedulable? But this has logging, so really seems like it would be handy. Maybe have cron trigger jobs? Cost seems to be included with the web service, but without the scheduler it needs additional effort.
  • celery: It sounds like Render supports celery, but I’m a little unclear on how this would integrate with my django project. Do I just add the celery and celery-beat libraries to my project, and run /etc/init.d/celerybeat start in my build.sh? But I still need to make a Redis or RabbitMQ instance, which are $10/mo? This I assume would be the most robust system, but also definitely the most expensive.

What is the recommended way to trigger lightweight scheduled tasks? One of these, or something different?

2

Hi there,

Thanks for reaching out.

I think overall the answer will be “it depends”.

Cron Jobs could certainly be used to cover the tasks you describe and sound likes the scheduled nightly/twice daily tasks are a perfect fit, while the sending “realtime” emails from a queue doesn’t feel like the best fit, but if the interval was short enough, emails may not be waiting too long to be sent. This would require 3 Cron services and depending on the time each run of the tasks take, billing will likely vary each month.

However, using a Background Worker and Celery, it may cover all your requirements in a single service. A single Starter Background Worker instance would be consistent $7 per month, keeping all of your tasks in one place. Emails could be queued and sent as required and it also seems Celery supports periodic tasks to cover for your nightly/twice daily tasks.

Jobs are more suited to one-off processes rather than regular tasks/queues.

It feels cost is a concern, if the tasks are as infrequent as you mention, Cron Job might be the lowest cost to start, however, this could vary depending on workload. You may also need to add another cron job service if you wanted another task/queue processed.

The Background Worker and Celery approach feels more consistent in billing and scalable if you needed more tasks/queues in future.

Hope that helps

Kind regards

Alan

1 Like
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.