Late last night my time (GMT), the virtual machine hosting the software that provides the forum search functionality ran out of disk space. For the last year and a half, whenever a snapshot backup of the data in the search system was created, it was never deleted after it stopped being useful. Snapshots are small, but their number caused the system to eventually run out of disk.
To fix it, I've freed up a small amount of disk space to give it some breathing room in the short term (realistically it'd take a couple of months to use it up, but I'd like to fix it all now), and then I'll do the following:
It should all be mostly transparent. I'll go ahead and do the first two steps now, to get it out of the way.
To fix it, I've freed up a small amount of disk space to give it some breathing room in the short term (realistically it'd take a couple of months to use it up, but I'd like to fix it all now), and then I'll do the following:
- Add a new volume (enough to let the system grow at its current rate for another ~10 years) to the VM
- Extend the logical volume onto it
- Add monitoring to the entire cluster to notify me of low disk space events in the future, so I'm forewarned
- Prune old snapshots periodically
It should all be mostly transparent. I'll go ahead and do the first two steps now, to get it out of the way.
