First of all I'd like to apologize for the downtime we experienced on the 20th and 21st of February. The server hardware upgrade which was the cause of the downtime was only supposed to last 6 hours but it took more like 40 due to lots of complications.
Many of the issues during that upgrade could have been avoided had I done things better. For example had I been more careful with our boot disk it wouldn't of failed suddenly. And if I had installed Windows on the Motherboard+CPU in the test server I could have diagnosed the networking issue and the low voltage problems before taking the main server apart.
Also I'd like to apolagize for the lack of a backup C Drive. Last year I stopped the automatic backing up of the C drive on the server because users said it was lagging the server (which it was). But I forgot to ever turn it back on again and was instead doing incremental manual backups. The latest backup of the C Drive in my possession at the time of the disk failure was 70 days old.
It was through sheer luck that I was able to get the failed C Drive working again by turning it upside down. I was eventually able to get a full image of the entire drive. We now have backups of the C Drive turned back on and I am using a system now that doesn't affect performance. This means the entire server is now being backed up completely not just Minecraft or my personal data on the server but the boot disk too which contains the OS, our webserver and other essential information.
Finally I'd like to apologize for the problems as recent as this morning where both Creative and Survival kept booting players every 15 minutes or so. This was caused by mcstats.org (a plugin developer analytical service) being offline which caused one of our new plugins (Prism) to disconnect players. I have now corrected this issue by turning this feature off in Prism.
So I hope all these troubles can be put behind us. The server is healthy and online, the upgrade is 100% complete and I've learned a lot through the mistakes I made. We have not suffered any long term effects or any data loss.
Things I've learned:
- 1. Backup everything, no excuses
- 2. Perform upgrades on new hardware on other systems before dismantling our main server
- 3. Make sure plugins are stable and won't cause performance problems before deploying them.
