Daily web services outages in Frankfurt region

Hi,

We are getting intermittent HTTP errors 521 and 502 on our web services in the Frankfurt region.
Some (or all) of our web services go down for a few minutes, then come back up. It has been happening every now and then since our migration to Render in early May, but now it happens on a daily basis.

This happens on all our web services, regardless of the plan, language or framework we use.

Could you please have a look? We will be forced to move away from Render if this keeps happening, as API uptime is critical to us.

Best regards,
Tom

I see a similar issue in the Ohio region — at usually 1:30AM-1:32AM EST. I’d be interested to know the time you’re seeing this happen.

Can the render team comment on the random 5xx codes? I ran into this issue when testing Digital Oceans App Platform and chose Render as it didn’t have the same issues, I thought. As I am in the process of migrating the companies infrastructure to Render, there is concern about this happening.

Just this week it has happened so many times, for instance:

  • 2022-07-11 : 03:05, 06:15, 07:01, 18:39, 20:22 and 20:41 UTC
  • 2022-07-13 : 05:01, 13:52, 17:48 and 20:04 UTC
  • 2022-07-14 : 07:24 and 22:51 UTC

From our point of view, as long is this issue is not solved, Render web services are just unsuitable for a production infrastructure.

I’m not sure if this is related but since migrating to Render.com we regularly see problems with name resolution of the database service leading to app errors:

could not translate host name “dpg-cat2g…ke81nm0-a” to address: Name or service not known

These are the most recent errors reported by Sentry:

  • Jul 14, 2022 05:16:28 CEST
  • Jul 13, 2022 11:48:04 CEST
  • Jul 13, 2022 08:49:20 CEST
  • Jul 13, 2022 06:21:49 CEST
  • Jul 13, 2022 06:21:48 CEST
  • Jul 12, 2022 20:40:26 CEST
  • Jul 12, 2022 16:41:53 CEST
  • Jul 12, 2022 15:58:32 CEST
  • Jul 12, 2022 15:58:31 CEST
  • Jul 12, 2022 15:58:27 CEST
  • Jul 12, 2022 13:38:55 CEST

Just to let you all know, we take reports like this very seriously and opened an internal investigation about these reports earlier in the week.

John B

2 Likes

Hi! For those of you running into 502 issues, would you be able to message me your service IDs that are seeing 502s so we can investigate them. The times are really helpful alongside the service IDs.

@partomatl @chrisgrande After looking through our logs at the times you have each indicated, we did see a large increase in traffic into the Render network due to a DDOS attack. While we mostly mitigated the DDOS attack, the scale up of our underlying proxy took some time during the response which resulted in 502 errors for each of you.

We’ve currently scaled up our proxy service to quickly mitigate 502s and are working on additional protections to help improve our ability to scale up when under attack.

@jnns For your DNS related issue, we’re currently investigating these and should have some updates for you early next week.

2 Likes

@maher thank you for your answer. Do you think the DDOS attacks also explain the 521 errors?
The 502 errors happened again:

  • 2022-07-15 at 13:53 UTC
  • 2022-07-17 at 15:53 UTC

Another incident reported in Sentry:

  • Jul 25, 2022 02:07:32 CEST

@maher @John_B it keeps happening:

  • 2022-07-20 at 14:51 UTC
  • 2022-07-20 at 21:41 UTC
  • 2022-07-26 at 21:47 UTC
  • 2022-07-29 at 19:11 UTC
  • 2022-07-30 at 01:36 UTC
  • 2022-07-30 at 22:45 UTC

Could we get an update on this incident? The uptime of the web services that we have migrated from Heroku to Render is disastrous…

If you’re experiencing issues I’d encourage you to open a direct support ticket by emailing support@render.com with details of the service that you’re experiencing problems with, the problem you’re seeing and any accompanying logs and we can investigate the problem you’re experiencing here,

Regards

John B

Hi all,

Our engineers think they have found a cause of 502s and will be implementing some changes to try and mitigate the issue.

As a reminder, if you are continuing to experience issues, we’d encourage you to raise a ticket (support@render.com) so that we can discuss your particular case, services and logs in detail.

Thanks for your patience here.

Alan