This summer our DFW facility had power issues, and as a result, we invested significant resources to improve all aspects of our power systems.  Last night, during one of these steps, we encountered issues and had a brief loss in power.  The power disruption was approximately 5 minutes in duration.  Despite this short power disruption, many customers experienced downtime that was significantly longer.  Since the power disruption hit the core of many of our cloud services, recovery of full operations required more effort than simple recovery of power.   The experience you had last night is not acceptable to us.

Here is what we know about the events:

 · The scheduled maintenance was planned to occur from 12:05am - 6:05am CST in our DFW data center.  This maintenance is part of a preventative maintenance schedule for several PDUs in UPS Cluster G at the DFW datacenter.  The PDUs were down for a total of 5 minutes before power was restored.  At approximately 12:29am CST, all PDUs behind UPS Cluster G lost power.

 · Although the power outage was very brief (5 minutes), it forced a hard re-boot to occur on a portion of our cloud infrastructure.  As our engineers worked to bring hardware back online, we experienced several unforeseen hardware failures.  Further complicating our recovery effort, the incident also created internal DNS issues, which caused additional delays.  With that said, the vast majority of cloud customers affected by this outage had service restored within one hour's time (many in as little as five minutes); however, depending upon the service, a few customers experienced service interruptions for up to few hours.

Here is how we plan to deal with it:

 · We have invested massively in the DFW facility to ensure it delivers at a level you expect from Rackspace - despite last night, we feel very good about our plan and have high confidence in the DFW facility - clearly we have to prove it.

 · We are reviewing our maintenance notifications - we typically do not share information on expected non-impacting events, but clearly we need to ensure we calibrate these events and are fully transparent.

 · We are reviewing our procedures and systems for quickly resuming cloud operations when an unexpected event like this occurs - unexpected events will happen, our job is to minimize their impacts.

We live by high standards and clearly have not lived up to them.  We welcome any feedback.  If you would like a call from me, or anyone on our senior team to discuss these issues personally, please reply with a phone number.

We have work to do to earn back your trust.  We will not rest until we have.

Thank you,

Emil Sayegh
General Manager, The Rackspace Cloud