Children's Hospital trims backup with disk-to-disk

31.10.2005
Children's Hospital in Boston was dealing with a nightmare of a backup window earlier this year that at times lasted from 4 p.m. to 10 a.m. the next day and often entailed failed tape drives and shutdowns. On top of at least two drive failures each month, the sheer speeds of data transfers were in the 30MB/sec. to 40MB/sec. range, which guaranteed that the backup window for some 300 servers would continue to grow.

"The backup windows just kept growing and growing, and there were so many problems at night that backups were being missed," said Paul Scheib, director of operations and chief information security officer at Children's Hospital. "You start seeing these slower speeds but you really don't know what it is. It becomes like an Easter egg hunt where you're trying to figure out what's limiting the rate to the drives. Is it the drives? Is it the servers? Is it the network?

The hospital launched a pilot program of a virtual tape library (VTL) technology as an intermediary between its Fibre Channel storage-area network (SAN) and its two tape libraries, but that had problems of its down, according to Scheib. "We had a lot of trouble getting it configured correctly," he said.

Instead of sticking with a VTL, which required integration with his Fibre Channel SAN, Scheib said that in June he chose another disk-to-disk backup technology that removed Fibre Channel from the equation and put server backups on the LAN.

By using DD460 disk arrays from Data Domain Inc. in Palo Alto, Calif., Schieb said, he was able to replace one of his tape libraries with three disk arrays that cost about the same amount: US$75,000 each. The disk arrays doubled data transfer rates, removing three hours from the hospital's lengthy daily backup windows, reducing backup failures by half and speeding up restores from hours or days to minutes.

The disk-based backup technology also allows the hospital to keep about three months of backup data online, nearly eliminating off-site tape recalls for data restores. Time to respond to urgent restore requests has been reduced from four hours to a few minutes, Scheib said, adding that he was also able to eliminate one of his Hewlett-Packard Co. tape libraries.

Tony Asaro, an analyst at Enterprise Strategy Group Inc., said he was surprised that the VTL from FalconStor Software Inc. that Scheib had piloted gave him problems, because they're relatively easy to install and run well. But Asaro acknowledged that Fibre Channel networks in general can be harder to configure and that VTL technology can be seen as more difficult than disk-to-disk technologies that run on Ethernet.

"Historically, Data Domain's product has not been purchased for backup problems but for recovery problems. Their compression allows users to keep lots of data on disk inexpensively for a longer time than tape," Asaro said.

The full data backups on the disk backup servers are compressed at a rate of about 20 to 1, with compression of incremental or subsequent backups of databases at 200 to 1. Scheib said he backs up about 60TB a week to his three DD460 disk arrays using Veritas NetBackup 4.5. So far, the devices have worked flawlessly, he said.

But disk-to-disk technology is not without drawbacks, according to analysts and users.

Jerome Wendt, a senior information systems analyst at a large Midwest data-processing firm, said that as a user, he has experienced issues with some traditional backup software not being properly integrated with disk-to-disk backup technology.

"I know there are issues with [IBM's] Tivoli Storage Manager where it doesn't recognize every virtual tape library interface presented to it. And Veritas [now Symantec] wants to license by every virtual tape drives it sees. I know NetBackup is licensed by the tape drive. So, what do you do if you have 256 drives? Do you license all 256 drives?" asked Wendt, who asked that his firm not be identified.

Asaro said another issue is backup software that's not able to push backups out of the backend of virtual tape libraries to tape drives. Asaro also recognized the issues around software licensing. "There are issues, but each one of these [vendors is] working at changing their licensing models," Asaro said.

Scheib said he hasn't run into any issues with his Data Domain technology and he plans on removing more of his dependency on tape.

"Our tape drives are still there, but we're moving toward doing a weekly tape backup or even a monthly backup, but we'll keep all incremental backups on our Data Domains," he said.