Gulf Coast storm and floods challenge IT

02.09.2005
Von Lucas Mearian

At 2 a.m. on Aug. 27, two days before Hurricane Katrina devastated the Gulf Coast, Tim Babco grabbed a red binder containing the latest version of SCP Pool Corp."s disaster recovery plan, put his dog and cat in the car, locked up his house and drove 500 miles from Covington, La., to the company"s emergency operations center in Dallas.

Babco, senior director of IT at Covington-based SCP, a US$1.3 billion wholesale distributor of swimming pool supplies, had gone through the motions of relocating his operations twice before when hurricanes had threatened neighboring New Orleans. Both of those storms turned out to be near misses, but Babco said this week that the practice runs helped him fine-tune his plan for when the real thing finally hit.

"People would be lying to say these things always go perfectly," Babco said. "But has it succeeded in allowing our business operations to continue to buy, sell and distribute products? It certainly has, and that"s what disaster recovery is all about."

However, the kind of disaster recovery planning that Babco did isn"t universal -- especially for a calamity as massive as this week"s storm. Simon Mingay, a business continuity and disaster recovery analyst at Gartner Inc., said that about 40 percent of the Fortune 1,000 companies aren"t prepared for a regional disaster. And small and midsize businesses are even less ready, he said.

"Obviously, we"re looking at a level of devastation here that few would have considered," Mingay said. "But most still believe that these are things that don"t happen to them."

Mingay said companies that have prepared properly for disasters, such as SCP, have extensive emergency communications plans, hot sites from which they can continue business operations for an extended period and some level of IT systems redundancy outside of their headquarters region.

But even companies that are well prepared might not take into account a crisis of the magnitude as the one spawned by Katrina and the flooding that followed.

For example, John Wade, CIO at Saint Luke"s Health System Inc. in Kansas City, Mo., said his contract with a disaster recovery vendor allows the IT department to work out of a hot-site facility for up to six weeks if necessary. But under the devastation caused by Katrina, the six-week limit could "pose a hardship for our company," Wade said.

Joe Hartman, an application development manager at HydroChem Industrial Services Inc. in the Houston suburb of Deer Park, said the company"s disaster recovery plan involves moving its corporate operations to facilities on the north side of the city. "Anything that would completely wipe out Houston would leave us in a bad way," he said.

Babco said SCP"s headquarters was unscathed by Katrina and the flooding, but it lost all data and voice communications links. Fifteen of the company"s 40 IT workers have relocated for now to Dallas, while another 15 were dispersed among offices around the country.

Three years ago, Babco decided to flip-flop SCP"s primary and secondary data centers, placing its critical systems in the Dallas facility, which is run by Houston-based VeriCenter Inc. He also created an IT disaster recovery team consisting of key employees for functions such as coordinating help desk services and relocating hardware. In addition, SCP has set up an internal business-continuity Web site that posts corporate alerts and provides toll-free numbers and an extensive list of employee contact information.

Not everything went smoothly. On Tuesday, Babco made an unplanned trip from Dallas to SCP"s headquarters to retrieve 12 internal application servers, including ones supporting human resources and e-mail.

He also said he has been unable to get a response from Boston-based Iron Mountain Inc., which handles off-site data storage for SCP. The area around Iron Mountain"s Kenner, La., storage facility is flooded and inaccessible, said Babco, who added that he would have liked to have had SCP"s backup tapes sent to Dallas in advance of the storm.

"They"ve not been able to provide us any information about when they will be able to get tapes out of their facility," Babco said. "I think they weren"t proactive in seeing the event looming and getting the tapes out of harm"s way." He vowed to switch to a different storage archiving vendor.

Richard Kerley, chief operating officer at Shutts & Bowen LLP, said three of the Miami-based law firm"s offices in Florida were hit by multiple hurricanes last year. "We could have had a situation where our main [data center] site in Miami was taken out and the cold site in Orlando was taken out, and we were decapitated," Kerley said.

Six months ago, Shutts & Bowan signed up for data center space in a reinforced concrete bunker owned by Terremark Worldwide Inc. in Miami. "You cannot go through a disaster scrambling to set up a headquarters," Kerley said, adding that the law firm"s neighbors in the bunker include the U.S. Department of State, the Department of Defense and the Federal Emergency Management Agency.

Mingay said the most difficult piece of any disaster recovery plan is dealing with how to staff backup sites outside of the region where a company is based. "You"ve got to have people ready to go," he said. But in a disaster, employees inevitably will be more concerned about their family and homes than they are about IT and business operations, he said.

XanGo LLC, a Salt Lake City-based producer and distributor of juices, this week was trying to determine the extent of damage suffered by some of its distributors along the Gulf Coast, said Darren Pulsipher, the company"s director of IT.

Internally, meanwhile, XanGo is setting up new disaster recovery processes. The company plans to open an office in Japan, which will be a failover site in case of a disaster in Utah, Pulsipher said. He"s also looking for a third-party co-location facility that"s geographically far enough away from Utah to provide protection.

In addition, Pulsipher has brought in workers who are experienced in running full-blown disaster recovery scenarios. "We"ll set up some scenarios to test our systems out to make sure we have everything covered," he said. "We have a large call center and are making plans to figure out what to do if no one can get to work because we"re buried in snow here."

-- Computerworld reporters Carol Sliwa and Heather Havenstein contributed to this story.