Re: Todays downtime
Posted: Thu Dec 05, 2013 2:41 pm
Today the server was offline for about 40 minutes. This was due to a power situation. There was a brief power cut which lasted around half a second. Unfortunately our UPS (Uninteruptable Power Supply) was unable to perform its job properly because our load level was simply too high.
I knew our power use was high but I wasn't sure just how high it was until I inspected the logs today. It would appear over the past few days (The log only goes back up to a week) we reached a peak load on our UPS of 95.5% - This is incredibly high and in the danger zone.
Most of the time our load is about 77-85% which is still pretty high but tolerable.
This all means I'm going to need to replace our UPS with a more powerful unit, 500 Watts is just not cutting it any more. In the interim between me purchasing a new UPS I have taken both our Modem and Router off the UPS and put them on just a normal surge protector. This means if there is a power cut the Modem and Router will turn off but hopefully the Server will not.
This has reduced our power usage to 67.7% - 72% which is much lower and hopefully low enough for the UPS to last. It is currently projecting a runtime of 9 Minutes should the power cut out. Enough for the server to shut-down gracefully. I'm going to look in to getting us a more powerful UPS at the earliest opportunity.
I've verified all our drives and backups, up-to 30 Minutes of building time on our Minecraft servers may have been lost as a result of the sudden crash of the server. This is normal as we only commit things to disk every 30 Minutes. Anyone logged in at the time of the crash (Four people from what I can tell) may have had their inventories rolled back.
Thank you for your understanding and patience.
I knew our power use was high but I wasn't sure just how high it was until I inspected the logs today. It would appear over the past few days (The log only goes back up to a week) we reached a peak load on our UPS of 95.5% - This is incredibly high and in the danger zone.
Most of the time our load is about 77-85% which is still pretty high but tolerable.
This all means I'm going to need to replace our UPS with a more powerful unit, 500 Watts is just not cutting it any more. In the interim between me purchasing a new UPS I have taken both our Modem and Router off the UPS and put them on just a normal surge protector. This means if there is a power cut the Modem and Router will turn off but hopefully the Server will not.
This has reduced our power usage to 67.7% - 72% which is much lower and hopefully low enough for the UPS to last. It is currently projecting a runtime of 9 Minutes should the power cut out. Enough for the server to shut-down gracefully. I'm going to look in to getting us a more powerful UPS at the earliest opportunity.
I've verified all our drives and backups, up-to 30 Minutes of building time on our Minecraft servers may have been lost as a result of the sudden crash of the server. This is normal as we only commit things to disk every 30 Minutes. Anyone logged in at the time of the crash (Four people from what I can tell) may have had their inventories rolled back.
Thank you for your understanding and patience.