Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

(page 3) 31 comments

sorted by: hot top controversial new old

[–] tenchiken@lemmy.dbzer0.com 6 points 5 days ago

Sending positive vibes, and passing offer of technical assistance once you get access again.

Hopefully it's an easy fix once you get into things again.

[–] PM_ME_VINTAGE_30S@lemmy.sdf.org 5 points 5 days ago

That sucks. Thank you for your time and running the instance. Gonna miss you all like hell while you're gone. 😞

[–] originalucifer@moist.catsweat.com 5 points 5 days ago (1 children)

people hate the big cloud solutions, but this is the kind of thing their HA infrastructure prevents against... hardware failures

i dont enjoy using (or paying for) aws, but i will never have a firewall or disk failure.

[–] drspod@lemmy.ml 7 points 5 days ago

Their HA infrastructure is all built on open source projects. The thing they have that we don't is teams of SREs on-call 24/7.

[–] viral.vegabond@piefed.social 3 points 5 days ago

You guys have always been excellent. As others have stated, no need to worry.

I'll be giving piefed some love in the meantime 👍

[–] grue@lemmy.world 2 points 5 days ago (14 children)

I also can’t access it physically during the next few weeks and my usual back up that could restart it is not available either.

Sounds like your 'bus factor' is too high and you need a few more backup admins.

load more comments (13 replies)

[–] Brkdncr@lemmy.world 1 points 4 days ago

Opengear cellular backup appliance with serial/console connectivity to hardware.

load more comments