The offical report reads as follows:
Between the hours of 1 and 4 in the morning, November 15th, 2004, the hard drive in the server started returning DMA errors, signifying a hardware failure. At 1:08, the server completed it's last known good checkpoint. At 4:08am, the MOO aptempted to do it's scheduled checkpoint. This caused an error, which was thrown up to the MOO proper. The MOO proper tried to do a PANIC checkpoint, which also failed, causing the entire MOO to shut down.
Sometime around 11 or so it was brought to the administrators attention that the MOO was no longer servicing connection requests. By 12 o'clock noon we had discovered that the box was in a state of catastrophic failure, and set about backing up everthing which we considered vital to an offline location. Fortunately the drive remained functional enough to continue reading files, although writing files became out of the question.
After backing up everything we considered vital, I set about trying to restore the box to a working condition, but to no avail. At this point the box stopped responding, and became a very expesive hosted paperweight.
By 3 o'clock Johnny had requested a harddrive replacement, and I believe by 4 we had a working box once again, albeit devoid of anything beyond the basic OS install.
The past 20 hours we're spent reconfiguring the box (a monumental task) and getting the MOO and main website up and working.
Around 11:50am Tuesday, the MOO came back online with the 1:08am DB from Monday, and we have every reason to suspect that it will remain online and uninterrupted from this point forward.
The last checkpoint was from 1:08am Monday morning. Fortunately between 1 and 4 (when the moo went down) there was basically no one on, so the problem of stranded characters should be kept to an absolute minimum.
It's expected that at this point, RP should resume like no IC interruption happened. If this proves impossible for some specific reason, please email [email protected] with the specifics of your situtation, and accomidations will be considered (But are not gaurenteed!!!) on a case by case basis.
We still have a ton of work ahead of us getting things like all our email accounts set back up, but as of right now [email protected] and [email protected] are working just fine, so use thoes for any offline communication you may require.
We do appologize for the unexpected downtime, and assure everyone we are working as fast as we can to bring things back to normal operation.
Thank you for your continued paitence and understanding.
-Kevlar
