Down came the rain, off went the Cloud. Or more accurately, flash went the lightning and down went Amazon and Microsoft's Dublin-based data centres.
Amazon said that lightning struck a transformer near its data centre, causing an explosion and fire that knocked out utility service and left it unable to start its generators, resulting in a total power outage.
The bad weather also resulted in disruption to part of the phase control system that synchronises the back-up generator plant, prolonging the outage and forcing Amazon to carry out the synchronisations manually.
On its status update dashboard, Amazon confirmed:
“ Normally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators. The transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the backup generator plant, disabling some of them. Power sources must be phase-synchronized before they can be brought online to load. Bringing these generators online required manual synchronization. We’ve now restored power to the Availability Zone and are bringing EC2 instances up. ”
In fact, Amazon started bringing up EC2 instances within three hours and said it had restored 60% of impacted instances within 12 hours. Microsoft seemed to fare better, restoring services within three hours and getting all European BPOS services up and running within four hours. Inevitably the outage will fuel the Cloud naysayers who will argue that these latest incidents indicate the vulnerability of Cloud services provision.
“This latest outage will do little to calm the nerves of BPOS customers who have suffered a series of service failures due to various reasons from late last year onwards. To be fair there are some things that Microsoft and other providers do not have control over. They can do something about failover and mirrored data centres however, and the immediate question is why was there no such backup in place. This latest problem will also fire off wider questions about the viability of the cloud as a delivery platform. The questions are valid, indeed necessary, as they are part of the ongoing process of improving the whole Cloud operation. ”
But Eager added:
“Outages are not a sign that Cloud does not work however..part of the problem is perception. Cloud is perceived as an always on service (like power, gas, water) and when a perceived always on service goes down our reactions are more extreme than when something that is expected to fail does fail. The 99.99% availability SLA’s that are critical to “selling” Cloud services reinforce the always on perception – but no provider promises 100% uptime or ever will do. The other problem is lack of control. Scheduled downtimes are accepted but when the unexpected occurs all customers can do is scan the wires and hope that someone, somewhere, is fixing the problem and that information will flow and service restoration is imminent. ”
She concluded:
“In-house provided services fail too, the difference is that IT departments feel like they have some control because they can actively work on the problem…Outages will not go away. The onus is on service providers to react quickly in terms of providing status updates. There is also a need to address business contingency on behalf of customers through the use of backup and mirrored facilities. ”
Amazon and Microsoft are two of the major US tech firms to have set up data centres in Ireland after the government there set out to turn the country in an attractive Cloud Computing hub for investment. Microsoft has built one of the world’s most efficient data centres and at 550,000 square feet, one of the largest. Amazon set up shop in Dublin in December of 2008 to house the European availablity zones for its EC2 Cloud Computing services and plans to expand its operation using a 240,000 square foot building in Dublin.
tags for Bolt from the blue as Microsoft and Amazon's Irish eyes go out
