Creating large zip file on worker service

1

I have a worker service that downloads hundreds of 3-6 mb files from a bucket, compresses them into one zip file, and uploads that back to the bucket.

This worked fine on Heroku, but It seems that no matter how much RAM I have on my worker service, it’ll crash the service. The logs just show the service restarting with no error but it happens at the exact point that the zipping process starts. It doesn’t crash if I compress a smaller number of the files into, but for the big one it’ll crash.

Is there a max memory per thread in a worker process even if the worker has much more available memory? It seems like something with the render system is preventing this action from succeeding.

2

HI there,

Thanks for reaching out.

I think Heroku supports swap memory on its common runtime, whereas Render does not. Maybe that’s the difference you’re seeing here, Heroku may give a little more leeway on high/max RAM use. Does upgrading to a plan with more RAM stop the issue?

Kind regards

Alan

3

I upgraded to the 8gb plan and tried it out again and it still failed. These actions were working fine on heroku 512mb dyno (with swap there it goes up to 1gb before failure).

I know the metrics don’t always have the most accurate representation of what happens, but they seemed to top out at 1gb of memory usage regardless of how high of a plan I chose.

Is there a place to find a better related log regarding the crash since it doesn’t show up in the dashboard > service > logs? It does show up in dashboard > service > events as unhealthy and restarting but no more information is provided.

I tried using ssh to tail/cat some logs in var/logs but neither of those commands worked.

4

I don’t think we would have any further logs around your applications usage of memory, do you have an APM on this service, which might be able to show memory use at an application level? Are you logging/able to log the size(s) of the downloads/produced zips when the crashes occur, or does it occur every time?

If you’re able to share some further details, I’d like to try and reproduce the issue myself, e.g.:

  • Ruby Version
  • Zip gem name and version
  • A typical file type & size along with an average amount you would expect to zip
  • Anything else I might need to consider in making a simple script (e.g. are you using Tempfile, or “real” files, etc.)

Thanks

Alan

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.