"Katrina and many of the hurricanes that hit Florida last year alone sort of nailed this home for us," said Les Auerbach, director of computer services for the city of Gainesville, Fla. "It was evident that we needed to rethink, for continuity of business, how we were doing this."
Gainesville's upgraded disaster recovery plans include building a hardened data center that's capable of withstanding nature's fury. The facility will be built out of concrete and designed to be less susceptible to wind damage than conventional buildings are, according to Auerbach. "The idea is that if a hurricane does come through, this center is not going to move," he said.
Auerbach has met with architects to begin drawing up specifications for the data center, which is expected to be shared by the municipal government and a city-owned utility that provides water, gas and electrical services to residents. He said he doesn't have a firm schedule yet for the project or an estimated cost.
In addition, Gainesville wants to give its employees the ability to work remotely in case operations are disrupted by a storm or a pandemic, such as a flu outbreak. The IT staff is assessing how to proceed on that plan, Auerbach said, adding that the city will have to replace desktop PCs with laptops and ensure that its network security tools can accommodate remote workers.
Blue Cross and Blue Shield of Florida is also setting up a hardened data center. Chris Gay, manager of disaster recovery at the Jacksonville-based medical insurer, said it is halfway through a move to a so-called Tier 3 data center that's designed to withstand a Category 5 hurricane. The changeover is scheduled to be completed by the end of October, Gay said, adding that he couldn't say how much the new facility will cost.
Gay said Blue Cross also is moving to reduce its reliance on tape backups by using tools from Double-Take Software in Southboro, Mass., to mirror its servers to an unspecified hot-site facility, with backup copies kept in the insurer's own data center. "Shipping tapes concerns us," he said, citing the Health Insurance Portability and Accountability Act's data privacy regulations, as well as the need to access information in the event of a major storm or other disaster.
Stopgap measures
Like Blue Cross, the office of the clerk of the circuit court for Florida's Hillsborough County is working to upgrade its data backup capabilities. Jeanne Harden, project manager for computer operations in the clerk's office, said that after seeing the damage caused by Katrina, she realized that neither a data center in downtown Tampa nor a second facility 11 miles from the coast would be safe if a similar storm struck.
"We don't feel comfortable at all," Harden said. "If this area gets hit by a Katrina, both my data centers are potentially gone, and there goes my data."
The clerk's office had stopped doing tape backups in favor of backing up information disk-to-disk between systems at the two data centers. But last week, it resumed weekly backups to tapes that will be shipped out of state. Harden said that's just a stop-gap measure while she talks with remote hosting vendors about doing electronic backups to an out-of-state facility -- a capability that she hopes to have in place by next month.
Alphastaff Inc., a payroll-processing firm in Fort Lauderdale, Fla., learned from an extended outage in the aftermath of Hurricane Wilma last year that its disaster recovery plan -- which was based on systems being down for two days -- needed to take longer disruptions into account.
"We figured, we're down for a couple of days, no big deal," said Jack Rahner, Alphastaff's director of IT. "When you're down for 12 days, it becomes a big deal. Ancillary services become not so ancillary."
For example, the company hadn't replicated its Great Plains accounting software, and state income and property taxes came due. This year, Alphastaff is adding more capabilities to its collocation facility in Atlanta and is installing additional equipment and applications there, including its accounting software.
Alphastaff also discovered "that not having a mail server was a huge hindrance" to continued operations, Rahner said. Although a mail server at the company's Atlanta facility remained up and running after Wilma, end users had trouble accessing it, he added. In an effort to avoid similar problems this year, Alphastaff has started using MessageOne Inc.'s e-mail management and continuity services.
Although Savannah, Ga., hasn't been hit by a major hurricane in more than a century, Katrina prompted the city's Memorial Health University Medical Center to step up its disaster recovery efforts. The medical center is setting up capabilities for replicating its electronic X-ray and patient records data to a data center in Atlanta, said Chris Leggett, Memorial Health's manager of technical services. She added that the replication process should begin within six weeks.
Other disaster planners are also beefing up the infrastructures that support their data centers: power systems; telephone and Internet connections; pumps; and even spare batteries for cell phones and laptops. "For communication, it was imperative that we stock up on extra batteries," said Jim Desjarlais, IT director for Lee County in southwest Florida.
In addition, IT organizations such as Lee County's are making arrangements to help one another if necessary. Desjarlais belongs to the Florida Local Government Information Systems Association , which has 140 members. Should he lose his data centers, several other counties in Florida have agreed to host his applications, he said.
Harold Schomaker, CIO for the city of Largo, Fla., is working through the same association to develop mutual IT assistance agreements that involve leasing space for backup systems in other data centers. "A lot of the cities that have sustained quite a bit of damage [from storms] are just getting to the point where they can start looking hard at doing this stuff," he said.
However, expanded disaster-recovery planning and outside help can only do so much in the face of an approaching hurricane. "Everybody's getting a little nervous at this time of year," Desjarlais said.