Unscheduled downtime - Monday 27th July 2020

While performing maintenance on a VM disk layout on an unrelated service that runs on the same hypervisor, one of the disk operations caused (somehow, still not sure how) the hypervisor to immediately reboot. Because of the lack of clean shutdown for the VMs running, the forum DB suffered a table crash that needed to be repaired.

Total downtime was around 30 mins, as I worked out what had happened, ran a full DB scan, identified the crashed table, and repaired it.

Sorry about that, hopefully not that many posts were in flight at the time. I've added a note to my playbook for VM disk image maintenance (which I sometimes run to grow the images when filesystems inside get full) that will help me avoid issuing the bad command in the future. I'll also try and triage the bug and pass it on to the FreeBSD gpart and kernel developers (depending on where the bug actually lands), if I can, to be a good citizen.

As always, deeply love each and every one of you, especially those that come to Site Feedback to see what went wrong when I fat finger during maintenance ❤️

Also, a dear friend of mine has requested that I add a trollface to this downtime message and I must oblige.



By Source (WP:NFCC#4), Fair use, https://en.wikipedia.org/w/index.php?curid=64267016