The telephone service in the county seat, Wauchula, is underground, but the storm pulled many of the cables out of the ground, caught in the roots of 200-year-old oaks. By the time it was finished, 75 percent of the buildings in the county were destroyed or severely damaged. "We are still pulling down homes destroyed by the storm two years later," says Don Faulkner, the sole IT staffer for the county's emergency management center. "Weeks later, I would see concrete block walls fall. The storm had broken the mortar, and suddenly they would just sag and collapse. Today the tallest tree in the county is about 35 feet high."
The following are some of the lessons he learned from the experience.
Lesson 1: Be prepared
Charlie caught Hardee County in the midst of moving the server system into a new emergency management building, actually its old jail, with two-foot-thick walls and windows protected by armored screens "that can take a hit from a 2-by-4 at 60 mph without denting," says Faulkner. But on Aug. 13 the move was not complete. The building had emergency power from a diesel generator, and the D-mark of the county's T1 had been moved to it along with the detached storage server.
"Our other servers were still in the old server room in the building next door, which had no emergency power. As an emergency measure, I ran 400 feet of 12 gauge extension cord from the generator to the old server room and shut down everything non-essential. I brought back backup tapes, disks, basically anything I could carry, and covered everything else in plastic in case the rain got in." As it turned out, that extension cord, lying on the ground between the buildings, survived when everything else failed. A good thing, because the county's DNS, Web and e-mail servers were in that room, and for four days after the storm the only communications the county had with the outside world was e-mail.
Lesson 2: Hope for the best, but prepare for the worst
That morning Charlie seemed to be heading somewhere else, and the county was at Level 2 on the emergency management system. "At 11:00 we heard it had changed direction and was coming directly at us," he says. "Then it speeded up. So we only had a few hours to prepare. It was a madhouse."
At 5:15 p.m., wind speeds reached 45 mph with gusts to 60 mph, and the electric utility shut down power to protect its substations. At that wind speed, firetrucks and ambulances cannot operate safely. But the storm was only starting. An hour later, the center of the 10-mile diameter eye passed one mile east of Wauchula.
"We were on the second floor of the emergency building, watching out the armored windows," Faulkner says. "My parents' house, where my family was gathered, is two blocks away. I watched 100-year-old oaks topple over on top of it. Actually, that might have been good. They did some damage, but they held the roof on."
When the wind reached 120 mph, it tore the weather station off the top of the emergency management building, so no one knows just how strong the wind got. They estimate 35 mph gusts above the 145 mph base speed.
"I watched one of the communications towers break off 45 feet up after being hit by flying debris," he says.
Two hours after it started, the wind was back down to 45 mph and dropping. Charlie had passed. But it left utter devastation behind.
Lesson 3: Plan multiple forms of redundant communications
When the storm passed, Hardee County had no power except for emergency generators, no wireline telephone service, no surviving cell towers. "Anything on a pole was gone," Faulkner says. "Even the satellite phones wouldn't work for some reason."
The emergency management director looked around and asked what his people coulddo. "I said, 'The Internet is still working.' So he sent an e-mail out asking, 'Is anybody out there?' His hands were shaking. When he received a response, there was cheering and a sigh of relief. For the next four days, the Internet was the county's only link with FEMA, the state emergency management office and the outside world in general.
Actually, the survival of the Internet connection was almost accidental. The county had an older T1 installation that was powered from its end rather than from the phone company, and it was hooked into the emergency generator. Without that, the county would have been totally cut off.
Today the county has DSL, wireless satellite and a backpack satellite transceiver. And it has a new T1. "The installers wanted to power it from the phone system, but I insisted that it be powered from our end and hooked into the generator." And it has a new propane generator that can go from cold to full power in seconds.
Lesson 4: Batteries don't like generators -- use line conditioners
Faulkner's next challenge came at 11:30 p.m. when the Internet went down. A quick check showed what had happened. Although the old server room was getting power, it had no line conditioners, so the generator was connected to the APC battery packs. But the batteries were not charging, and when they died, power to the Web, e-mail and DNS servers went with them.
Faulkner put a line conditioner on the power supply and hooked it directly to the critical servers, which gave them power, but when he tried to restore using Windows tools, he discovered that the crash had sent the read/write heads wandering across the disk of the DNS/Web server, damaging the partitioning. He needed to restore.
Lesson 5: Backup, backup, backup -- but not to tape
Fortunately that spring, as part of its emergency preparedness, the county had moved from tape backup to the Acronis True Image Server. The old tape system required 72 hours for a bare metal restore. "Fortunately, the servers went down after Acronis finished its automatic 11 p.m. backup. I put the backup CD into the drive, initiated a restore and held my breath. Eight minutes later it said 'Finished" and rebooted the system. And the server came up as if nothing had happened." He has been an Acronis fan ever since.
Lesson 6: Always keep the primary DNS server separate
"Never consolidate the DNS and Web servers on a single machine," Faulkner says. "If you do, you are asking for trouble."
Lesson 7: Protect your emergency generator
The generator at the county hospital took a pipe through its radiator, putting it out of action. At the fire/rescue station the storm ripped the cover off the generator. It continued to function, but with the circuits soaked the power was very dirty. "We never did find that cover," Faulkner says.
Lesson 8: Lightning strikes twice
Three weeks after Charlie, Hurricane Frances hit the county. The center of the eye passed 10 miles northeast of Wauchula. Again, the county had little warning, and while this was a Category 2 storm, it also moved much more slowly, and the county was still digging out. The emergency management center was down to four of its 25 workstations. The rest had fried their power supplies on the dirty power from the old generator. The school system loaned them 15 laptop computers, but they arrived six hours before Frances, stripped to bare metal.
Faulkner spent four hours building the image on the first laptop. But then he simply used Acronis True Image Workstation to clone that image onto the others. In 22 minutes he had all 15 up and running.
So by the time Hurricane Jeanne arrived a few weeks later, the county had one of the most experienced emergency management teams in the state. Jeanne was a Category 1, barely a hurricane in terms of wind, but it was a slow-moving rainmaker. "It was never made official, but we estimated we had a 500-year flood event. We had water where we never had seen water before."
Lesson 9: Think portable
Today the emergency team is all on laptops, which have several advantages. First, they have their own, built-in line conditioners. "A laptop can convert any power that comes at it into the right voltage."
Second, if the power goes out they keep working. Third, you can pack and run if you have to. "If we have to abandon our location, we can grab an access point, pick up our laptops and relocate," Faulkner says. His laptop is configured to function as a Web, DNS, & e-mail server if necessary, so they can set up anywhere and be back online in minutes. And with portable wireless access points and the backpack satellite transceiver, they can carry their connectivity with them.
Lesson 10: Stay loose
Today, one of Faulkner's hobbies is running "what if" disaster scenarios. And since he knows he will never think of everything, he asks people in other county departments and outside the county for their thoughts. Ultimately, flexibility is the key. Things like the Acronis quick restore and cloning abilities and maximum portability give the emergency team the ability to react to multiple contingencies.
"Statistically, we know that an area that is hit by a major hurricane is likely to get hit again within two years," he says. Hardee County was hit by three hurricanes in a row two years ago. This year, Faulkner is determined to be as prepared as possible.