Node: ETIMEDOUT error when fetching within any loop

Hi,

I have a node express application being hosted on Render which has a /test route. In this route i do the following:

  • Retrieve some object from my MongoDB hosted in Mongo Atlas
  • Create a function that uses a for loop to loop through the object and push urls into an array.
  • A second function that takes that url array and fetches data and pushes it into a data array

app.get(“/test”, async (req, res) => {
// START OF GET URLS FROM DB
const dbBrands = await Brand.find({});
const getUrlForLoop = (obj) => {
let data = ;
for (const el of obj) {
let url = el.url;
data.push(url);
}
return data;
};
const urls = getUrlForLoop(dbBrands);
// END OF GET URLS FROM DB

// START OF GET DATA FROM URL FOR LOOP
const getDataForLoop = async (urls) => {
let data = ;
for (const url of urls) {
let response = await fetch(url);
let resData = await response.json();
data.push(resData);
}
return data;
};
const fetchedData = await getDataForLoop(urls);
res.send(fetchedData);
// END OF GET DATA FROM URL FOR LOOP
});

I’m using Postman to test the API calls.
When my express server is running locally on my machine and i hit the route, i get the data as expected.

However when i try to hit the route that is being hosted on Render, it fails with the below error:

I’ve tested with fetching one url on its own and the data gets retrieved from the fetch as expected so it seems like its an issue during the looping through the urls.

Any advice would be greatly appreciated.

Hi there,

What URL is getting the timeout?

Regards,

Keith
Render Support, UTC+10 :australia:

Hi Keith,

The URLs are being stored in a Mongo DB.

This line pulls the URLs from the db.

const dbBrands = await Brand.find({})

I then loop through the urls and fetch the JSON data and store it in a variable. The following are the urls.

image

Full list of urls at the below link:

Doing an individual fetch on the URL returns the JSON data correctly its just when i try and loop through the URLs that i get the error.

Thanks

UPDATE:

It turns out it was one specific url that was causing the timeout:
https://www.tesco.com/fuel_prices/fuel_prices_data.json

Running my express backend on my localhost I have no timeout issue when fetching the data but when running my express backend on render the fetch request times out.

Hi there,

I also have issues accessing this URL locally from my machine using curl:

$ curl https://www.tesco.com/fuel_prices/fuel_prices_data.jsoncurl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

If I force HTTP 1.1 it does not respond:

$ curl -v --http1.1 -m 120 https://www.tesco.com/fuel_prices/fuel_prices_data.json* Host www.tesco.com:443 was resolved.* IPv6: (none)* IPv4: 104.119.101.140* Trying 104.119.101.140:443...* Connected to www.tesco.com (104.119.101.140) port 443* ALPN: curl offers http/1.1* (304) (OUT), TLS handshake, Client hello (1):* CAfile: /etc/ssl/cert.pem* CApath: none* (304) (IN), TLS handshake, Server hello (2):* (304) (IN), TLS handshake, Unknown (8):* (304) (IN), TLS handshake, Certificate (11):* (304) (IN), TLS handshake, CERT verify (15):* (304) (IN), TLS handshake, Finished (20):* (304) (OUT), TLS handshake, Finished (20):* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF* ALPN: server accepted http/1.1* Server certificate:* subject: C=GB; L=Welwyn Garden City; jurisdictionCountryName=GB; O=Tesco PLC; businessCategory=Private Organization; serialNumber=00445790; CN=www.tesco.com* start date: Mar 20 13:08:28 2024 GMT* expire date: Apr 20 13:08:27 2025 GMT* subjectAltName: host "www.tesco.com" matched cert's "www.tesco.com"* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M* SSL certificate verify ok.* using HTTP/1.x> GET /fuel_prices/fuel_prices_data.json HTTP/1.1> Host: www.tesco.com> User-Agent: curl/8.6.0> Accept: */*>* Recv failure: Operation timed outcurl: (56) Recv failure: Operation timed out

I’m not sure why this happens, but it looks like this URL does have some issues responding to requests.

Regards,

Keith
Render Support, UTC+10 :australia: