Shell unreachable

1

I there,

shell access on our application (hosted in Frankfurt) is very unreliable.

Very often we can not access it, the page seems to auto-refresh before we got any prompt (I can see a websocket error in the console wss://frankfurt-proxy.render.com/srv-c5680upg7hp0afmtbldg/shell)

This problem isn’t new and is very frequent, what do you suggest?

Best,
Christian

2

Hello Christian,

I’m sorry to hear about your issues with the shell- we’ll be looking further into this. To help with our investigation, I have a few questions:

  1. Does this happen consistently? For example, if you go to the shell right now, is it it working? Does it still keep working if you refresh several times?
  2. What browser do you use when you notice these issues?
  3. Do these issues seem to happen more around the time when you’re deploying your service?
  4. Do you notice this on other services, or just one?
3
  1. The error happens randomly. But when it happens, it seems that no page refresh can fix the issue. Only redeploying the app seems to make the shell working again
  2. Safari and Chrome
  3. I don’t know, I have to check this
  4. Only on one service
4

Thanks for the extra information. The next time you run into this issue, could you post here again before redeploying your app so that we can take a closer look at the service and diagnose things? I’m unable to replicate the issue currently, but would like to take a closer look to find what the underlying problem might be.

1 Like
5

Hi @danielle, right now the shell for our application displays-production is not reachable. Could you please have a look?

6

Hi @danielle, right now the shell for our application displays-production is not reachable. Could you please have a look?

[Discourse post]

7

Hi Christian - thanks for reporting this, keeping your service in this state has been very helpful for debugging.

It looks like whenever you get a Server Unhealthy event in your dashboard, your service recovers but is then in a state where the shell can’t operate correctly. It also looks like this might be connected to some CPU spikes.

From digging on our end, it looks like your service is maxing out the /tmp directory. Is your service writing a large amount of ephemeral data to /tmp? Especially during times when you get Server Unhealthy messaging, I suggest taking a look and seeing if there’s an unusual amount of writing to /tmp.

My other suspicion is that your service’s amount of log lines may be what is filling up /tmp.

Hope this helps with debugging - redeploying your service when it’s in this state should restore your shell access, but if we can prevent /tmp from becoming too full this shouldn’t be necessary in the first place.

1 Like
8

Thank you, it really helps!
We indeed have some jobs leaking files in /tmp
We will solve these issues and then see how it goes

9

Hi, do you have any limit for this dir?

  • size?
  • number of fd?

Cheers