For example, Dave Muntz, CIO at Texas Health Resources Inc. in Arlington, has put his 335 IT staffers through disaster recovery drills for years. But after seeing the utter devastation caused by Hurricane Katrina, Muntz added a new twist: He made the drills a surprise. Previously, the IT workers were given some warning that a test was coming.
The first unannounced test was conducted in April. Muntz said IT staffers received a message late one day telling them that the health care provider's systems were unavailable. For some, going into disaster recovery mode meant working all night and into a weekend. In addition, a small team was dispatched to SunGard Data Systems Inc.'s hot-site facility in Philadelphia, where they mounted tapes to restore the company's systems.
To add to the realism, Muntz timed the test to take place when the IT department was busy working on projects. "We do it at the most inopportune time to make sure we can test ourselves under great stress," he said. "We don't do it when everything is going smoothly."
Lessons were learned at the SunGard facility. For instance, a configuration problem arose when a highly available application that requires two CPUs was put on a machine with just one processor. "It wanted the other CPU to be available," Muntz said. "Those are the kinds of things that you can't learn except by doing."
Hurricane Katrina prompted Office Depot Inc. to take steps to improve its internal communications capabilities. Critical teams of workers are being equipped with laptops that have nationwide broadband cellular access, said Tom Serio, director of global business continuity at the Delray Beach, Fla.-based retailer.
But what became particularly important after Katrina was ensuring that the company has multiple ways to contact employees. Standard means of contact, such as addresses and phone numbers, weren't enough "after people evacuated to the four corners of the country," Serio said.
He added that Office Depot now also asks employees to provide contact information for their spouses or partners, plus home e-mail addresses, BlackBerry PINs, the name of an out-of-state relative and details on whether their cell phones accept text messages.
F.A. Richard & Associates Inc., an insurance services firm in Mandeville, La., was hit by three hurricanes last year: Katrina, followed by Rita, which affected its Lafayette, La., and Houston offices, and Wilma, which knocked out power at its Boca Raton, Fla., office for two weeks. Consequently, FARA doesn't just have a Plan B for this year -- it also has a Plan C, said David Casse, its IT and telecommunications manager.
For example, the company has documented what each of its workers does and made plans for how the functions could be performed if individual employees weren't available during an emergency, Casse said. Similarly, FARA has audited all of its computers and documented the tasks they handle, whether the work could be re-created and whether the systems needed to be backed up, mirrored or made redundant.
Mike Clark, systems manager at Unibill Inc. in Lake Charles, La., thought the billing services provider was well protected last year. Unibill is headquartered in an old bank building that is located on high ground and capable of withstanding just about anything, Clark said.
But when Hurricane Rita blew through the area, the total loss of local telecommunications services caught Clark by surprise. It took five weeks for the services to be restored, prompting him to switch service providers in preparation for the new hurricane season.
The lesson he learned: Take nothing for granted. "The implications are so catastrophic that we have no choice but to expect that we will be hit this year," Clark said. "To think of it any other way is to invite more headaches and heartaches."