Grafana and Prometheus can't communicate over internal network but can using the public endpoint

Hi! I setup both Prometheus and Grafana on Render with little trouble; I used their latest Docker images and two almost barebones Blueprint files. Both of the services are up, and I can access their respective web admin UIs.

I was trying to add my Prometheus instance as a data source to my Grafana, I kept getting a timeout error my Prometheus instance’s internal host:post (prometheus-****:9090 for me).

This is the error I got:

Error reading Prometheus: Post "http://prometheus-****:9090/api/v1/query": dial tcp 10.131.75.117:9090: i/o timeout

I tried a few different things (switch over to HTTPS (not that this would have helped), try using GET vs. POST) but nothing seemed to work. Then out of curiosity, I decided to give the public endpoint (https://prometheus-***.onrender.com) a try, and it just worked. Hah!

Is there something I am doing incorrectly? I do have a workaround obviously but would much rather not expose my Prometheus instance to the web, and also not have to go through the load balancer and all that stuff unnecessarily.

Thanks a lot in advance for your help.

Hi there,
As a matter of interest, could you try prometheus-****:10000 instead of port 9090 and let me know the result?

John B

That worked! Thank you so much. But would you mind explaining how and why?

It’s a bug that we don’t detect and set the port you want your service to use - by any chance do you remember if the first deploy of the prometheus service failed?

John B

Yes, it did. What’s the right thing to do for me now?

We’re working on a fix right now - you clarifying it worked on port 10000 was really useful.

John B

Hi. Is there an update on this issue?