Hey everyone, and happy new year!
Sorry about that super long downtime there. Yesterday (Sunday) morning at 10:03AM PST our server suffered a physical hardware failure, apparently a power supply failure. Unfortunately despite opening a ticket with our hosting vendor (OVH) a few minutes later and them claiming to have 24/7 support, nobody looked at our ticket until this morning when their phone support lines opened and I called them.
They’ve now replaced a defective power supply and we’re back online, after ~26 hours of being offline. Some pretty disappointing response times, to put it nicely.
We’re planning to move away from OVH at the end of this month, onto proper enterprise grade hardware that we own and control. This will give us a HUGE boost in server resources and allow us to scale for the foreseeable future, while also giving us the control to resolve problems like this much quicker. Expect another follow up post about this in the next couple weeks once I’ve put together the migration plan.
Timeline:
- Jan 5th 10:03am PST - We get alerts to the server being non-responsive.
- Jan 5th 10:05am PST - I pull up the console via IPMI and it’s completely non-responsive. Attempting to power off / on the server or do anything, does not work.
- Jan 5th 10:15am PST - Initial support ticket created with OVH. I followed up a couple times over the next few hours, and got no response.
- Jan 6th 6:32am PST - Called OVH, gave them the case number and asked them to investigate
- Jan 6th 7:34am PST - I get notified they’ll start their “intervention” in 15 minutes.
- Jan 6th 11:04am PST - Call them again, the tech is still working on it and they’ll get back to me with an update
- Jan 6th 11:34am PST - “I was informed by our data centre technician that there is an issue with the power supply unit for the rack on which your server resides. Your server will come back online once they have replaced the power supply.”
- Jan 6th 12:17pm PST - We’re back up finally!
Edit on Jan 7th @ 8:40am PST: We just had another outage of about an hour. Investigating with OVH.
Welcome back everyone! I was sad to see our instance down but I’m grateful we had some good follow up by our administration.
Thanks for all your hard work!
Thank you for all your hard work and top notch comms, Lemmy.ca admin peeps. I spent the day at dbzer0, which is always fun, but happy to be back home!
Thanks for getting it back and all the work to keep it running!
We’re back home!
We are finally back!!! Thank you guys!!
Happy New Year and thank you for your hard work.
Welcome back! OVH sucks, really happy to hear you’re moving away from them!
Make sure you figure out what kind of service credit you’re eligible for due to their SLA snafu. Over 24 hours to replace a simple power supply… what a bunch of jokers. OVH has gone downhill quite a bit over the years sadly.
Nice quality hardware they have. Not even redundant power supplies it seems to have, and no monitoring either. I’ll keep my distance from them.
Yep, planning on it. It looks like we should qualify for one free month, which would be nice to make our migration a little easier.
Not the server power supply per the notes above, but some component that supplies power to the rack. Perhaps the PDU, or other distribution components? Depending on the rack, it could take a couple of hours to replace, but certainly the ticket should have been picked up far earlier for service!
Happy new year!
They’ve now replaced a defective power supply and we’re back online, after ~26 hours of being offline.
It was a nice break from the firehose.
wb gl hf
I didn’t know what to do with my hours at work. I was almost at the point of actually…working…gross.
Lemmy.ca goes down and the PM resigns just a few hours later, obviously it’s all a conspiracy to bury the news /s
I think this is causation. Trudeau must be an avid Lemmy.ca user and the stress of this instance being down pushed him over the edge.
Welcome back! I was looking into OVH but after this outage… Naaah.
Same here!
