Receiving intermittent 502 from API deployed in Render

[pavlos.kosmetatos]

I am receiving intermittent 502s in an otherwise healthy deployment of my app.

Example list of 502 errors (timezone is EEST):

Spikes of 502s don’t seem to be connected with times the service is deployed.

Service details:
Name: wealthyhood-app-api-prod
Region: Frankfurt (EU Central)
ID: srv-c7tf0rjvog4j6ok1779g
Host: wealthyhood-app-api-prod.onrender.com

[al_ps]

Hi Pavlos,

Thanks for reaching out.

Are you able to see the response headers of those 502s? Specifically, are you seeing an x-render-routing header? They may help us with diagnosing the issue you’re seeing.

Thanks

Alan

[pavlos.kosmetatos]

Hi Alan,

Here you go:

"x-render-routing":"dynamic-paid-error"

[al_ps]

Hi Pavlos,

Thanks for finding that header. The dynamic-paid-error value indicates the error is coming from the service and not the Render routing layer.

Are you able to correlate any of these failing requests to your own logs?

Thanks

Alan

[pavlos.kosmetatos]

Hi Alan,

Thanks for your reply.

AFAIK there are no 502s returned from our API service (wealthyhood-app-api-prod) I am going to add some more logs and try to see if those requests ever reach the API.

By the way, not sure if it matters, but the requests that are failing are going from one Render service (web application) to another (API).

[pavlos.kosmetatos]

Hello again!

I added some more logs in our service and I can’t see the 502 requests ever reaching it.

The full list of headers is the following, maybe that could help:
{"date":"Wed, 10 Aug 2022 13:47:57 GMT","content-type":"text/html; charset=utf-8","transfer-encoding":"chunked","connection":"close","cf-ray":"73892fab097b9104-FRA","cf-cache-status":"DYNAMIC","expect-ct":"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"","x-render-routing":"dynamic-paid-error","server":"cloudflare","alt-svc":"h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"}

[edgarhabitat]

Just to chime in here. I was receiving a similar issue (random 502 errors, no logs etc) and it turns out it we were hitting our plan’s memory limits. We just had to scale the server one notch and they went away.

[jake]

Hey Pavlos,

I took a look at our routing layer’s logs and saw a small number of 502s that were failing with “connection reset by peer” when connecting to your API service. This suggests that your application is terminating the connection or the process serving the request is restarting. We don’t have much visibility into the traffic between your web app and your API, but it seems plausible that your web app is seeing the same errors.

It’s possible that the connection is failing before the request gets to the logs that you added. I think the next step would be to add additional instrumentation to your application to get more visibility into why those connections are being terminated.