The docs and example code suggest running your Rails database migrations (rails db:migrate) as part of your build script.
I’ve been running the same build script for both my ruby web service and my (background) worker.
The idea here is that whichever service gets deployed first will take care of the migrations and the next one will see the database is already up-to-date and skip the migrations.
This has worked fine, but I now have a Rails app with slower database migrations. This leads to scenario where one service has already started migrating, but isn’t finished yet. The second service sees the database is out-of-date and also tries to migrate the database, which causes the ActiveRecord::ConcurrentMigrationError exception.
What’s the recommended approach here?
Running the migrations in the build script for only one service doesn’t cut it, because the other one might get deployed first.
The approach I usually recommend is to have some locking mechanism on the database to prevent double-migration, so it’s great that you already have that in the form of the ActiveRecord::ConcurrentMigrationError exception.
When your second service builds and the migration lock is already held, what behavior do you want? Do you want your second service to wait for the lock to be released? If that’s the case, you can likely detect that exception and do a loop with a sleep that waits for the lock to be released.
Since this seems like a common setup (having a Rails app with a worker), I’d expect Render to handle it out-of-the-box somehow.
I’m not sure what the right architecture would be for this (perhaps a service can be configured to be ‘dependent’ on another service like in GitHub Actions) but having to add a loop seems a bit hacky.
Do you happen to have some example code you can share to implement this work around? Let’s say bundle exec rake db:migrate is part of my bash script and I want to retry every 10 seconds until it succeeds or after 5 failures.
We don’t have any sample code, unfortunately. I asked internally, and a colleague suggested another workaround: If you’re running into this because both services are auto-deploying, you can disable auto-deploy on your background worker, and have your rails service trigger a deploy of the background worker using a deploy hook (Deploy Hooks | Render · Cloud Hosting for Developers). I know this is also a bit hacky, but I wanted to share it in case it’s helpful for you.
I’ve also created a feature request for dependent services on feedback.render.com. If there’s any more detail you think would be helpful to include, I definitely encourage you to add it.
Thanks Dan. I left some more comments on the Feature Request.
The Deploy Hooks are an interesting idea, but I don’t want my web service and worker to potentially run on different versions of the code base. My whole app is designed in such a way that assumes they are always running the same version.
Hi @marc, how did you solve this problem in the meantime?
update: for anyone coming here from google and looking for a solution, here is what works for me:
Disable auto-deploy on your worker, as @dan suggested.
Add a command to your build script to trigger your deploy hook for the worker, e.g.:
#!/usr/bin/env bash
# exit on error
set -o errexit
bundle install
bundle exec rake assets:precompile
bundle exec rake assets:clean
bundle exec rake db:migrate
curl https://api.render.com/deploy/.... <-- change this. you can find your deploy hook URL in your service settings view.
if [ -n "$DB_MIGRATE" ]
then
bundle exec rake db:migrate
fi
And in my render.yaml I build the web process like this:
buildCommand: DB_MIGRATE=true bin/build
Whereas all the other processes call bin/build without that ENV variable.
This way only the web process will migrate the database. It’s not ideal, as the other processes might be deployed before the migration is done but I haven’t run into substantial problems yet.