Grafana and Prometheus can't communicate over internal network but can using the public endpoint

1

Hi! I setup both Prometheus and Grafana on Render with little trouble; I used their latest Docker images and two almost barebones Blueprint files. Both of the services are up, and I can access their respective web admin UIs.

I was trying to add my Prometheus instance as a data source to my Grafana, I kept getting a timeout error my Prometheus instance’s internal host:post (prometheus-****:9090 for me).

This is the error I got:

Error reading Prometheus: Post "http://prometheus-****:9090/api/v1/query": dial tcp 10.131.75.117:9090: i/o timeout

I tried a few different things (switch over to HTTPS (not that this would have helped), try using GET vs. POST) but nothing seemed to work. Then out of curiosity, I decided to give the public endpoint (https://prometheus-***.onrender.com) a try, and it just worked. Hah!

Is there something I am doing incorrectly? I do have a workaround obviously but would much rather not expose my Prometheus instance to the web, and also not have to go through the load balancer and all that stuff unnecessarily.

Thanks a lot in advance for your help.

2

Hi there,
As a matter of interest, could you try prometheus-****:10000 instead of port 9090 and let me know the result?

John B

4

That worked! Thank you so much. But would you mind explaining how and why?

5

It’s a bug that we don’t detect and set the port you want your service to use - by any chance do you remember if the first deploy of the prometheus service failed?

John B

6

Yes, it did. What’s the right thing to do for me now?

7

We’re working on a fix right now - you clarifying it worked on port 10000 was really useful.

John B

8

Hi. Is there an update on this issue?

9

@cduruk any chance you could share your setup for how you deployed prometheus and grafana on render successfully?

@John_B was this fix finalized?