Surviving Rita: A tale of two data centers

26.09.2005
Von Todd R.

For the second hurricane in a row, New Orleans-based Web hosting vendor and Internet domain name registrar DirectNIC managed to stay up and running overnight as Hurricane Rita pounded the Gulf Coast before coming ashore early this morning.

Sigmund Solares, CEO of Intercosmos Media Group Inc., which owns DirectNIC, rode out Hurricane Rita in New Orleans last night and was busy assessing damage this morning.

Solares said his company, located on the 10th floor of a downtown New Orleans office building, was still dealing with broken windows and other damage left from Katrina, which plowed through the area Aug. 29. During that storm -- and despite massive flooding in much of New Orleans after -- DirectNIC"s operations continued uninterrupted because it had an emergency generator and an adequate fuel supply, he said.

DirectNIC has approximately 800 hosted servers.

"We definitely got hurt by Katrina, and we still have a ways to go [to get completely cleaned up], but we remained open the whole time," Solares said.

Even as Rita"s strong winds brushed by last night, ripping tin pieces from neighboring structures and hurling them at DirectNIC"s building, Solares said he and his four team members were much less worried than when Katrina struck. "It"s still windy outside," he said this morning. "It"s still gusty. [But] right now, we are feeling much better than at any time during Katrina."

Solares said he got a good night"s sleep inside an internal office in the building, where he was able to get away from the howling winds outside. The building continued to have electricity until 7:59 a.m. -- more than four hours after the storm struck near the Louisiana-Texas border. The power went out then, and battery backups kicked on instantaneously, followed by a backup emergency generator, he said. The outage lasted only 11 minutes, however, and the company"s servers remained up and running.

Looking outside, Solares said damage to neighboring buildings doesn"t look as bad as when Katrina hit. Floodwaters didn"t reach his building during the last storm, nor did they do so this time. "With Katrina, when the levees broke, I could see the flooding a half a block away," he said. "I"m looking where there was flooding [before], and I don"t see any water there right now."

In other parts of the city, particularly the Ninth Ward, floodwaters cascaded over newly repaired levee walls yesterday, causing more problems in low-lying areas.

During Katrina, and again last night, DirectNIC"s employees had to build a makeshift levee of their own. Rainwater began entering through the ceiling from broken windows on the floor above, Solares said, forcing him and his staff to use mops, a wet/dry vacuum machine and bundles of promotional T-shirts to create what he called the company"s own "levee system" to keep water away from servers and other electronic gear. Those same methods came in handy again last night, he said, since building repairs have not yet been made. Before Rita drew near, Solares and his crew installed plywood over the open windows and used sealant and tarps, hoping to keep out as much rain as possible. The makeshift repairs helped, he said.

Earlier this week, as Rita swirled in the Gulf of Mexico as a dangerous Category 5 storm, Solares called a competing Web hosting vendor in Houston, EV1Servers.net, and offered help in case Rita hit Houston dead-on.

"For us, there"s a lot we can do," Solares said. "It"s not much effort for us to help people get back up and running." DirectNIC has extra space and capacity for servers and can make that space available to others in an emergency, he said.

"We"re lucky that we"ve always saved for a rainy day and that we"re able to help our employees and try to help other people, even though it"s a very tough time for us overall," Solares said. "It"s also good for morale for our employees to see us helping other people in need."

In Houston, Robert Marsh, CEO of EV1Servers.net, said early today that because Hurricane Rita tracked farther toward the east, away from Houston, the city was spared the storm"s worst effects. The highest wind gusts were in the 50-to-60-mph range, far lower than the 120-mph winds expected at one point. By this morning, only an inch of rain had fallen in the area where EV1"s data centers are located, he said.

"In this instance, we were very fortunate not to have received the worst," Marsh said. Overnight, the company switched over to its emergency generators after some nearby power transformers blew up. EV1 plans to return to utility power sometime today after stable power flow is maintained for at least five hours, he said.

About 25 EV1 employees stayed overnight inside the company"s offices and two data centers to maintain services for customers. The company had also sent seven Web technicians to a hotel in Wichita, Kan., to remotely handle customer trouble tickets had its main data centers been knocked out by the storm.

"Our trouble-ticket load is essentially back to normal at this time," Marsh said today. "If the storm continues to clear, we hope to bring our people back from Wichita as early as Sunday night."