Shutdown timetous for Sidekiq worker

[HonzaSterba]

Hi, we are rapidly developing and because of that we deploy quite often every day. We use sidekiq pro for our background processing and we are seeing a lot of orphaned jobs in our logs. This would mean that sidekiq is being killed quite quickly and does not have time to finish up running jobs (more details here Batches, failed jobs and retries not showing · Issue #4900 · sidekiq/sidekiq · GitHub).

What are the settings for terminating running processes after deploy? Any way these could be tweaked to better suit Sidekiq? Specifically how long does render wait for process to finish before killing it?

I have found other related topics that were left without an (acceptable) answer:

[Sidekiq] Add description to deploy safely with long running background jobs

Hi All, I think it’s better to add description to this sidekiq page about how to deploy safely with long running jobs, because default behavior just kill running jobs when deployement and killed jobs will not run again. This is not kind behavior because we could not notice if there were running jobs and killed jobs after deployment. I think simple solution is using timeout option when running sidekiq like following. bundle exec sidekiq -t 3 Default sidekiq timeout settings is 25 seconds (fo…

Deploys & Long Running Jobs

Hey All, We have a job queue running on render via the “Background Worker” service type. We’re running into issues around interactions between long running jobs and deploys. Specifically, some of our jobs run for several minutes and if a deploy happens at the same time the job is killed mid-run via Render’s SIGTERM/SIGKILL. In our ideal world, the job would run to completion and then the deploy would happen. Is there a way to do that via Render?

[jennifer]

Hi @HonzaSterba , thanks for reaching out!

There aren’t any settings for terminating running processes after deploy right now. We have a related feature request here: Allow zero downtime deployment to be turned off for Background Workers | Feature Requests | Render , and it looks like another user commented with the exact use case as yours.

To answer your other question, we issue a SIGTERM and wait 30 seconds before issuing a SIGKILL. Is that the behavior you’re seeing or does it look like Sidekiq is getting killed more quickly than 30 seconds?

[HonzaSterba]

Strictly speaking the feature request does not cover this behaviour.

I have not done extensive testing so I do not know that what you are describing matches reality. However I have been seeing a lot of orphaned jobs. I have adjusted the sidekiq timeout to be much lower (5s) and hopefully this will help.