[Sidekiq] Add description to deploy safely with long running background jobs

1

Hi All,

I think it’s better to add description to this sidekiq page about how to deploy safely with long running jobs, because default behavior just kill running jobs when deployement and killed jobs will not run again.
This is not kind behavior because we could not notice if there were running jobs and killed jobs after deployment.

I think simple solution is using timeout option when running sidekiq like following.

bundle exec sidekiq -t 3

Default sidekiq timeout settings is 25 seconds (for Heroku user) but it seems that Render kill within fewer seconds. So we should specify timeout option.

For every sidekiq users, could you please add above description? Thanks!

2

related: Deploys & Long Running Jobs

3

From the sidekiq documentation the recommendation seems to be to send a TSTP signal, and then the SIGTERM.

Any more info on this? (I found a couple of other posts on the topic, like Shutdown timetous for Sidekiq worker ) If I was running on a virtual server, I’d be able to login and send the signal to the running process. The only other idea I had was to to have an in-app signalling mechanism for the sidekiq worker to shut itself down. That could be triggered by the deploy process, currently triggered a Github Action in my case.

While I don’t have a critical need for it yet (hopefully soon), I’m wondering how other production workloads are handling frequent deploys that might kill running jobs.

4

Hey there,

We follow a fairly well-recognised path here issuing a SIGTERM and then a SIGKILL 30 seconds later as detailed at https://render.com/docs/deploys#:~:text=We%20terminate%20the%20old%20version%20at%20this%20point%20by%20sending%20your%20app%20a%20SIGTERM%20signal.%20Most%20web%20servers%20automatically%20intercept%20SIGTERM%20and%20shut%20down%20gracefully.%20There%20is%20a%20grace%20period%20of%2030%20seconds%20to%20shut%20everything%20down.%20If%20your%20app%20is%20still%20up%20after%2030%20seconds%2C%20it%20is%20shut%20down%20via%20a%20SIGKILL%20signal.

I’ve previously deployed large workloads to Heroku who also have this behaviour and made heavy use of Sidekiq and never ran into an issue with it.

John B

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.