Our private Redis service suddenly stopped working with “Server failed”. Metrics looked fine, no spikes, no bad log lines as far as I can tell. We made no changes at all recently. All build attempts fail with “deploy failed”, again with no explanation as far as I can see.
Any idea for what’s wrong or how to debug this? It’s causing significant downtime in our production app. Thanks in advance for any suggestions!
Hi @simon ,
This was due to an internal issue where we were having issues pulling images from Docker, but we have fixed this. I have checked and your Redis service is up and running. We are sorry for the downtime.
Thanks @jennifer. What’s the reason that this wasn’t reflected on the status page? Our Redis service was down for almost exactly 3 hours. This is a very long downtime, and we had 0 control or visibility.
Hi @simon,
I’ve just updated the status page to reflect the incident and impact for deploys. When we detected this issue yesterday, we initially thought this was only an internal-facing issue. We didn’t realize that it was affecting any users, but we should have updated the status page as soon as we realized that. I’m really sorry you experienced downtime yesterday and that it was so frustrating! We’re working on making our processes better, both on catching these types of issues faster and communicating it better as well.
