Telecom NZ escapes criticism for massive network outage

27.06.2005
Von Juha Saarinen

While overseas media such as the BBC compared last week"s nationwide Telecom New Zealand Ltd. network outage to the "embarrassing" Auckland power cuts, local observers and the Minister of Communications, David Cunliffe, appear less perturbed by the incident, sharing the telco"s opinion that the outage was a freak occurence.

An inquiry into the lack of robustness in the network, an estimate of losses incurred or a re-evaluation of the regulatory environment don"t appear to be on the cards either. Asked if the Government would take any action over the outage, a spokesman for Cunliffe"s office responded: "What would you expect the Government to do?"

Telecom says the chance of both cables failing at the same time is one in a million, despite running advertising campaigns promising "five-nines" (99.999%) reliability, which allows for only five minutes and 26 seconds outage for its core network every year. As the outage lasted roughly four-and-a-half hours, Telecom"s "five-nines" promise means it will have to run the network for just under 50 years without interruptions to achieve its claim of five-nines reliability.

The outage begun on Monday at 10:48am when a post-hole borer working in the south Taranaki severed the single functional optical fibre cable in Telecom"s national network backbone. Prior to that, Telecom"s other backbone optical fibre circuit in the Rimutaka area had failed. Unconfirmed reports say that the Rimutaka circuit had been out of action for two days before before the Taranaki cable was cut.

Customers nationwide experienced severe service disruptions throughout the day as data and voice communications between cities went offline. Trading services such as Eftpos and bank payments were inoperable and the New Zealand Stock Exchange had to close for much of the day. Government departments such as the IRD were also affected, and Auckland police were unable to access their email. Service was restored fully at 3:18pm, but there are currently no estimates of the financial damage caused by the outage.

New Zealand network operators, however, were asking some hard questions about Telecom"s connectivity architecture after learning it wasn"t able to handle a two-cable cut. The lack of route diversity for traffic on Telecom"s network was singled out, with operators suggesting a more distributed system as the way forward.

Adrian Smith, a network specialist at the National Library, told other operators how TelstraClear had saved the day for his organisation during the outage ? ironically, by quickly setting up temporary internet peering so that traffic could be routed around the damaged Telecom network. TelstraClear has copped criticism from network operators for reducing New Zealand internet route diversity after stopping the peering arrangements with other providers it inherited when it bought Clear Communications.

Smith pointed out in a message to the NZ Network Operators Group mailing list that TelstraClear was only able to assist because the National Library connects to the Wellington Internet Exchange.

Although he didn"t want to be drawn on the issue of peering, TelstraClear spokesman Mathew Bolland told Computerworld a number of customers called the telco"s wholesale department, which was happy to do what it could to help out.