Unable to deploy Grafana Alloy

1

Good afternoon,

We have been running Grafana Agent OTel for a while, and Grafana just deprecated it in favor of the new Grafana Alloy.

The migration is quite smooth in terms of configuration, but the new agent doesn’t work out of the box with Render.

After configuring the grafana/alloy Docker image and deploying it, we are receiving:

exec /bin/alloy: operation not permitted

I cannot emulate the failure locally, so I’m thinking that maybe it uses somehow a privileged operations that Render forbids.

Could you help me understand what might be happening for this new OTel Agent?

Thank you

2

I’m debugging in production, here some info.

root@<host>:~# strace -f /bin/alloy 
execve("/bin/alloy", ["/bin/alloy"], 0x7ffc63164c68 /* 40 vars */) = -1 EPERM (Operation not permitted)
strace: exec: Operation not permitted
+++ exited with 1 +++
3

I have successfully deployed Alloy by:

  1. Deploy a Go service from source code (did a fork)
  2. Attach a Disk (otherwise fails)

Attaching a Disk to a Docker deployment doesn’t work

4

Hi Aleix,

I’ve followed up with you in the support thread you opened on this topic.

Regards,

Matt

5

Hi , can you elaborate more on how you were able to deploy? i am facing similar issue too. may be step by step process would help me here. thanks in advance!

6

Sure!

What I did was:

  1. Fork GitHub - grafana/alloy: OpenTelemetry Collector distribution with programmable pipelines
  2. Deploy a render service using Go instead of Docker
  3. Attach a volume to the folder you store the data (what you add as part of the parameter --storage.path=/var/lib/alloy/data)

That it implies this approach is that you will need to keep the fork up to date and some manual work to keep it up compared to using the Docker image.

I think the main reason of Alloy not working on the Docker environment is due to Render has a quite limited permissions and Alloy requires several compared to Grafana Agent in Flow mode.

I hope it helps you.

7

Hey Aleix,

I followed in the same footsteps.

  1. Forked the Go source code of Grafana Alloy
  2. Deployed it on render using the following build command make binaries; make generate-ui and start command ./build/alloy run <config-file-path> --storage.path=/var/lib/alloy/data. I have attached a disk to the instance mounted at /var/lib/alloy
  3. All goes well until Render just keeps deploying not recognising that the service has already started
  4. After a while the deployment times out and fails

I also have set the PORT environment variable to 12345 which is where the alloy UI is hosted by default, as suggested by one of the render engineers in the support chat.

Do you think you can provide some help here to resolve why Render is not able to pick up on the service status? Am I missing something in the config.

8

Okay I figured out the issue, so by default render will check for a service to be live at 0.0.0.0:10000 using the PORT env var you can change the port render checks to any arbitrary value in my case I set it to port 12345 since that is where alloy will host the web UI by default.

But I missed that the default bind address was 127.0.0.1:12345 and not 0.0.0.0:12345 so using the following start command did the trick.

./build/alloy run <config-file-path> --storage.path=/var/lib/alloy/data  --server.http.listen-addr=0.0.0.0:12345

Also for anyone else who stumbles upon the same issue, make sure you set the PORT env var to 12345 as well, otherwise render will try to ping the service at port 10000

9

The fix was 0.0.0.0, the port change is incidental. Fixing the service to bind to 0.0.0.0 is the only required step.