The great earthquake that destroyed the turn-of-the-century Gold Rush boom town on April 18, 1906 was not a one-shot disaster; The U.S. Geological Survey predicts a 62 percent chance of another earthquake in the Bay Area with a magnitude of at least 6.7 during the next 30 years. And lest Southern Californians get too sanguine, the probability is the same for a similarly strong quake there in the same time span.
With that, and last year's devastating hurricanes, in mind, many organizations in the state, even small and midsized firms, are looking to replication systems to keep hot sites ready to go in other locations.
HOB Entertainment Inc.'s corporate offices in Hollywood manage 25 remote locations, House of Blues clubs and amphitheaters, said Adrian Black, manager of network operations for the department of information systems and technology. The main offices, which span four floors, run all the important systems for the company, including financial, accounting, legal and marketing, he said.
After setting up a storage area network in 2005, the company is considering using it to improve its disaster recovery options during the next year, planning for which starts in July. "We're on the 18th floor," Black said. "Who knows how durable this building is?"
The company owns an amphitheater complex -- the Gorge, in a remote location in the middle of Washington State -- which is about to get a 100Mbit/sec Internet connection in addition to the T1 line already in place. Consequently, Black and his manager are thinking about setting up a failover system there and having the company's data constantly replicated. "It's a great location for collocation," he said. "That is such a remote location, and we own the buildings."
Comarco Inc., a maker of wireless test products in Irvine, Calif., also has its disaster recovery center in Washington State, but for a different reason. Its IT Manager, Sean Anderson, was burned out of his house by wildfires in southern California, and moved to his second home in Spokane, where he works remotely. The company decided last fall that setting up its backup data center in Spokane would be a wise idea. "Since I'm up here and electricity is cheap in Spokane and rental space is cheap, it made sense," Anderson said.
If the company's Irvine building were destroyed, or its employees unable to get to it, weekly backups at an offsite location wouldn't really be practical for disaster recovery. "It didn't make sense to have tapes that couldn't get back to a building that wasn't functional," he said.
At first, Comarco looked at hosted data centers through companies such as SunGard. But the "relatively small" Comarco, with earnings of US$50 million a year, decided that solution was too expensive. "For the kinds of plans they have for the recovery we're interested in, the economics didn't work," Anderson said.
Instead, the company decided to replicate its critical systems, which include ERP on a Solaris system, a SQL database for CRM, engineering source code and Exchange. "I never thought e-mail would be that critical, but it is these days," Anderson said. Comarco considered an iSCSI system from Itransa Inc., with software for asynchronous replication, but because the company saved money by using VMware for virtual servers, the Itransa solution would have been too expensive. That option would have meant buying asynchronous replication software, as well as annual maintenance, for each virtual server.
The company tested products such as WANSyncHA from XOsoft and Double-Take from NSI Software Inc. -- it's now Double-Take Software -- and settled on Double-Take, which it tested in Irvine before moving it to Spokane. However, the Exchange failover didn't work because the Microsoft automated failover tool expected Exchange binaries to be on the target machine. "The Exchange binaries were not on the base machine -- they were on a virtual machine that wasn't running," Anderson said. Fortunately both NSI and Microsoft changed their pricing models around that time to enable users to run software using a single license on up to four virtual machines at no additional cost. "That was music to our ears," he said.
The system is still a work-in-progress. The Exchange failover was turned on in early March, with the engineering source code following later in the month and the SQL Server failover expected by the end of April. For the Solaris-based ERP system, the software is dumped into a replicating server using NFS and another Solaris box located in Spokane, Anderson said.
The two data centers are connected between a T1 line in Spokane and a fractional DS3 in Irvine; The company also uses Riverbed Technology Inc.'s WAN optimization software to improve performance. The transactions are posted to the backup system almost as quickly as to the data center one, Anderson said. "It might be half a second behind." The test Exchange failover takes 13 minutes -- and that includes a 10-minute delay the company configured to keep WAN glitches from triggering a failover.
Like Comarco, Costello & Sons Insurance Brokers Inc. in San Rafael, Calif. also finds e-mail backup critical and uses NearPoint from Mimosa Systems Inc. NearPoint saves Exchange files at the block level, which makes them easier to replicate.
Costello & Sons, which provides liability insurance for technology firms, has a four-part disaster recovery method, said IT director Steven Perry. First, it has redundant servers in the office. "If any one server goes down, we can recover right away," he said. Second, the company backs up everything to tape, which is stored at an offsite location in a bank vault across the bay. Third, everyone uses removable external hard drives.
"For the price of a couple of tapes in the old days when they were $50 each, you can put in 250 [GB]," Perry said. "Our Exchange store is around 60 [GB], the insurance data with all our CRM, which gives us policy information, is around 30 [GB] and our document management system has about 60 [GB]. All of that information could go on one 150 [GB] hard drive."
The final disaster recovery solution is a 'doomsday book' -- a laptop with a 100GB drive and enough batteries to run the business for about a day. The laptop is taken offsite every day.
Perry admitted that just sending the tapes across the Bay was not an infallible strategy. "Most of the stuff I worry about in terms of real-time recovery are small kinds of disasters," he said. "The big-time disasters, I tend to think we would be offline for more than the amount of time that having instantaneous recovery requires."