Today, we experienced an outage.  Technically, we experienced a few outages.

We initially discovered that our Galadriel server was offline, and furthermore, that the VPS node that it ran upon was also down.  We began working on restoring this machine when we were distracted by an incoming DoS attack.

Working (remotely) to extinguish the attack, we found ourselves locked out of the network and triggered a reload of the router.  Upon reloading, the system brought up the services in an atypical order triggering a bug/caveat/quirk previously unknown to us.  While everything looked good on our side, discussions with upstream carriers indicated that all was not as it appeared.  With this input, we quickly located a solution which we began to apply, until...

A storm knocked out electricity in parts of our neighborhood.  While we retained power, internet connectivity was lost to our offices (note, we have no servers located in these offices).

Once our branch office internet connectivity was restored, were were able to complete the process of restoring internet connectivity to all servers, restoration of the crashed VPS node and its guest accounts, including the galadriel server.

We apologize for the inconvenience that this caused.  This explaination seeks only to provide transparency to our operations so that our customers may know the details of service affecting events.