Hi all,
Looks like the VM that hosts the forum webserver ran out of memory and swap for some reason, and then effectively deadlocked in the swap handler. Not sure why, there's plenty of memory assigned to that particular VM and it's never gotten close to running out before, so probably a runaway process of some kind.
I'll keep a closer eye on things over the next few hours, and then out into the next few days at lower frequency if things look OK, and I'll also put some better monitoring in place to alert me if it happens again. That seems to have completely failed and caused me to miss it for a few hours.
Thanks to @CarstenS for prodding me on Twitter!
Looks like the VM that hosts the forum webserver ran out of memory and swap for some reason, and then effectively deadlocked in the swap handler. Not sure why, there's plenty of memory assigned to that particular VM and it's never gotten close to running out before, so probably a runaway process of some kind.
I'll keep a closer eye on things over the next few hours, and then out into the next few days at lower frequency if things look OK, and I'll also put some better monitoring in place to alert me if it happens again. That seems to have completely failed and caused me to miss it for a few hours.
Thanks to @CarstenS for prodding me on Twitter!
