App-breaking issue during Rails app deployment time

1

Hey friends, I’m running into a serious issue every time I deploy my rails app on render and am losing customers because of it! Specifically I think there’s an edge case with the distribution of the precompiled assets which breaks the app until the user resets their browser cache:

  1. Render finishing deploy, and the new version & old version are running in parallel for some short amount of time
  2. User access app, pointing them to the new asset (or old asset) (I’ve intermittently reproduced this issue with both .js & .css files)
  3. For some reason my app returns a 404 for the requested asset, which is cached by the user’s browser.
  4. User keeps refreshing the app but since it is cached it’s stuck in a bad state, leaving the app in a broken state unless user clean the browser cache

I love Render a lot but my customers are really bothered by this. So I need to move to another provider if I don’t manage to fix it soon… But maybe I’m doing something wrong on my end? Here’s the configurations of my app. Turning config.public_file_server.enabled on helped a lot but did not completely resolve the issue.

config/environments/production.rb

Rails.application.configure do
  #...
  config.enable_reloading = false
  config.eager_load = true
  config.consider_all_requests_local = false
  config.action_controller.perform_caching = true
  config.require_master_key = true

  # Turning this on for render environment in attempt to fix the deploy-time css crash issue, but doesn't seem to help
  # Taken from https://community.render.com/t/rails-static-assets-behaviors/3360
  config.public_file_server.enabled = ENV["RAILS_SERVE_STATIC_FILES"].present? || ENV["RENDER"].present?

  config.assets.compile = false
  config.force_ssl = true
end

applicaiton.html.erb

<!DOCTYPE html>
<html>
  <head>
    <%= csrf_meta_tags %>
    <%= csp_meta_tag %>

    <%= stylesheet_link_tag "application", media: "all", "data-turbo-track": "reload" %>
    <%= javascript_include_tag "application", "data-turbo-track": "reload", defer: true %>
  </head>

  <body>
    <!-- ... -->
  </body>
</html>

render.yaml

services:
  - type: redis
    name: xxx-redis
    region: ohio
    maxmemoryPolicy: noeviction
    ipAllowList: [] # only allow internal connections
    plan: starter

  - type: web
    name: xxx-web
    repo: https://github.com/xxx
    runtime: ruby
    region: ohio
    healthCheckPath: /up
    buildCommand: "./bin/render-build.sh"
    preDeployCommand: "bundle exec rails db:migrate"
    startCommand: bundle exec rails s -e production
    envVars:
      - key: DATABASE_URL
        fromDatabase:
          name: xxx-postgres
          property: connectionString
      - key: REDIS_URL
        fromService:
          type: redis
          name: xxx-redis
          property: connectionString
      - key: RAILS_MASTER_KEY
        sync: false
      - key: MALLOC_ARENA_MAX
        value: 2 # renduce memory usage, https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
      - key: WEB_CONCURRENCY
        value: 2
    plan: standard

  - type: worker
    name: xxx-worker
    runtime: ruby
    region: ohio
    buildCommand: "./bin/render-build.sh"
    startCommand: bundle exec sidekiq -e production
    envVars:
      - key: DATABASE_URL
        fromDatabase:
          name: xxx-postgres
          property: connectionString
      - key: REDIS_URL
        fromService:
          type: redis
          name: xxx-redis
          property: connectionString
      - key: RAILS_MASTER_KEY
        sync: false
      - key: MALLOC_ARENA_MAX
        value: 2 # renduce memory usage, https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
    plan: starter

./bin/render-build.sh

#!/usr/bin/env bash

# exit on error
set -o errexit

bundle install
bundle exec rake assets:precompile
bundle exec rake assets:clean
2

did you try setting this to false and see what happens, if it is a dynamic page caching could interfere.

Good Luck

1 Like
3

Thanks for the suggestion. Could you shed some light on how this would help with the precompiled assets? I thought perform_caching was just for the page caching stuff. Edit: I’m trying this but don’t think it’s the root cause as there’s no caching in my app currently

4

i thought this caching happened between the controller and redis
i read this page to see what it did
https://guides.rubyonrails.org/v2.3/caching_with_rails.html

1 Like
5

There’s something else at play here possibly.

When you deploy to us, you’re correct that we spin up the new deployment on new instances and then terminate the old deployment. So there is a period of time that old and new are running, this is our Zero downtime deployment feature at work that ensures your service is always available.

During this brief period of time, either the old or the instance may respond and the same for subsequent requests. So an initial request could be received by the old instance but then a second request, for styling, CSS etc may be received by the new instance - and that asset may not exist on that instance, and results in a 404. We try to keep the time that both are running to as minimal as possible, a health check path defined on the service helps here as well.

There isn’t a single one solution that fits all here

We’re also just discussing internally to see if there’s something we could be doing here as well to improve the situation,

Regards,

John B
Render Support, UTC :uk:

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.