Sidekiq jobs getting stuck in queue and not processing on production

[Sunwoo_Yang]

Hello!

I’m using Rails/SIdekiq and I’m following the guide here.

Here are my rails/sidekiq versions
rails (~> 6.1.4, >= 6.1.4.1)
sidekiq (6.5.8)

and I can’t seem to get the job to process and leave the queue.

1. I did not setup a sidekiq.yml and just using the default queue.
2. I’m executing the job through the .perform_async method
3. I copied the secret_key_base from my rails service. I also copied the Rails_master_key variable into my sidekiq worker service
4. The sidekiq service is running fine most of the time, but sometimes errors out with

Redis::CannotConnectError (Error connecting to Redis on red-cdrhvbta49909bk2c5k0:6379 (SocketError)):

5. I deployed the services using the Render.yaml file

databases:
  - name: mysite
    databaseName: mysite
    user: mysite

services:
  - type: redis
    name: cache
    ipAllowList: [] # only allow internal connections
    plan: free # optional (defaults to starter)
    maxmemoryPolicy: allkeys-lfu # optional (defaults to allkeys-lru). Rails recommends allkeys-lfu as a default.
  - type: redis
    name: sidekiq-redis
    region: ohio
    maxmemoryPolicy: noeviction
    ipAllowList: []
    
   # Let's create our background worker
  - type: worker
    name: sidekiq-worker
    env: ruby
    region: ohio
    buildCommand: bundle install
    startCommand: bundle exec sidekiq
    autoDeploy: false
    envVars:
      - key: REDIS_URL
        fromService:
          type: redis
          name: sidekiq-redis
          property: connectionString

  - type: web
    name: mysite
    env: ruby
    buildCommand: "./bin/render-build.sh"
    startCommand: "bundle exec puma -C config/puma.rb"
    envVars:
      - key: DATABASE_URL
        fromDatabase:
          name: mysite
          property: connectionString
      - key: RAILS_MASTER_KEY
        sync: false
      - key: REDIS_URL # this must match the name of the environment variable used in production.rb
        fromService:
          type: redis
          name: cache
          property: connectionString
      - key: REDIS_URL
        fromService:
          type: redis
          name: sidekiq-redis
          property: connectionString

I see the job getting queued when I visit /sidekiq, but it just stays there. This is working in development, though.

Any help would be appreciated!

[John_B]

Hey Sunwoo,

Just looking through the render.yml you’ve posted there I have some observations.

By not specifying a region for your database, ‘mysite’ webservice and your ‘cache’ services will be placed in our Oregon region. Whilst you have then explicitly set Ohio as the region for your sidekiq-redis and sidekiq-worker. You’re then setting duplicate REDIS_URL keys in the web service which won’t work as the last one is trying to access sidekiq-redis in the Ohio region.

We don’t support cross regional connections (if this was intentional) so your best bet here will be to have all the services in the same region. Then, you’ll need to fix the REDIS_URL in the webservice to use a different key name if you want to use a separate redis instance for your sidekiq jobs as well as then configuring sidekiq to use the same env var.

I suspect if you fix all that (as well as making sure sidekiq-worker has the DATABASE_URL set as well (as typically workers use the DB then things will start working as you want,

Regards,

John B

[Sunwoo_Yang]

Hey John,

Thanks for your helpful tips! I made the changes you listed but the job is still getting stuck in queue.

1. I set all my regions to be “oregon” in my render.yaml file
   

   

2. I deleted the services that were in “ohio” and recreated them in “oregon” and confirmed they were properly set in the dashboard

3. I changed the env variable that sidekiq uses to be sidekiq_redis_url and set the env variable in the sidekiq initilizer file

4. I set the DATABASE_URL to be the same one I have set in my web service. The postgres database

Here’s the render.yaml file now

databases:
  - name: mysite
    region: oregon
    databaseName: mysite
    user: mysite

services:
  - type: redis
    name: cache
    region: oregon
    ipAllowList: [] # only allow internal connections
    plan: free # optional (defaults to starter)
    maxmemoryPolicy: allkeys-lfu # optional (defaults to allkeys-lru). Rails recommends allkeys-lfu as a default.
  - type: redis
    name: sidekiq-redis
    region: oregon
    maxmemoryPolicy: noeviction
    ipAllowList: []
    
   # Let's create our background worker
  - type: worker
    name: sidekiq-worker
    env: ruby
    region: oregon
    buildCommand: bundle install
    startCommand: bundle exec sidekiq
    autoDeploy: false
    envVars:
      - key: SIDEKIQ_REDIS_URL
        fromService:
          type: redis
          name: sidekiq-redis
          property: connectionString

  - type: web
    name: mysite
    env: ruby
    region: oregon
    buildCommand: "./bin/render-build.sh"
    startCommand: "bundle exec puma -C config/puma.rb"
    envVars:
      - key: DATABASE_URL
        fromDatabase:
          name: mysite
          property: connectionString
      - key: RAILS_MASTER_KEY
        sync: false
      - key: REDIS_URL # this must match the name of the environment variable used in production.rb
        fromService:
          type: redis
          name: cache
          property: connectionString
      - key: SIDEKIQ_REDIS_URL # try a different key for sidekiq redis
        fromService:
          type: redis
          name: sidekiq-redis
          property: connectionString

The job gets enqueued, but never processes like before

Here is where I am setting the SIDEKIQ_REDIS_URL in my initializer.

require 'sidekiq'
require 'sidekiq-status'

Sidekiq.configure_client do |config|
  # accepts :expiration (optional)
  Sidekiq::Status.configure_client_middleware config, expiration: 30.minutes
end

Sidekiq.configure_server do |config|
  if Rails.env.production?
    config.redis = { url: ENV["SIDEKIQ_REDIS_URL"] }
  end
  
  # accepts :expiration (optional)
  Sidekiq::Status.configure_server_middleware config, expiration: 30.minutes

  # accepts :expiration (optional)
  Sidekiq::Status.configure_client_middleware config, expiration: 30.minutes
end

And in my background worker, I have set that same sidekiq_redis_url

Any other reasons that it might not be working as intended?

Thank you in advance!

Edit: Actually, just ran across this guide. Is this what I need to use in order to execute sidekiq workers? Does it not work by executing MyWorkerClass.perform_async in my web service?

[Sunwoo_Yang]

Actually, just figured it out. The only thing missing was setting up the sidekiq_redis_url in the sidekiq client configs. It started working after that.

[John_B]

Bingo, there we are. Glad we could get you sorted.

John B