"Most organizations have their fail-over boxes sitting next to their main systems in the data center, and then they have to buy a third set of boxes for their disaster recovery site," Oriol says. "We are able to put our fail-over boxes in the disaster recovery site and eliminate the need for that third set of boxes entirely."
Children's has split its storage clusters, with half at the disaster recovery site and the rest at the main data center downtown. The key to making this work is a very fast, responsive network connection using Ciena Corp.'s Adaptive WAN products running over a private 2Gbit/sec. fiber physical layer. It uses EMC Corp.'s Symmetrix Remote Data Facility and MirrorView to support synchronous backup of all data to the disaster site. The result: "Our tests show that it takes less than five minutes to fail-over, validate and move on," Oriol says.
And because of that, the hospital's disaster recovery operation runs on full duplicates of the computers in its data center rather than minimal systems. "When we do have a cutover, we plan to run on the disaster site until we need to move back to prevent a service interruption, rather than having a second delay to move back to the main data center," Oriol says.
This is particularly important as Children's moves to its new electronic medical record (EMR) for inpatients. The EMR is already in use for outpatients, and the first phase of the rollout in the main hospital is in process now, with the goal of moving all medical charts off paper. Obviously, with medical care dependent on computer access, the hospital cannot afford lengthy downtime.
The system also saves on tape handling. The hospital writes to the tape library at the disaster recovery site directly from its storage-area network disks.
Another benefit is the simplification of the main hospital's 2007 move to its new facility, presently under construction at a new biomedical park being developed at the closed Fitsimmons Air Force Base. "This is a huge move for us," Oriol says. "We are putting in 28 new systems, including [voice over IP], cell phone replication, 802.1x for user identification over the network and even some creative things like a patient portal to provide entertainment and educational programs for our long-term patients."
The Ciena core network has already been connected to the new site, and even though plans are still being finalized, the hospital intends to move the disaster site to this new facility to become the main data center, while the present data center stays downtown. "We plan to maintain a presence downtown, including an emergency department and inpatient rooms, to serve downtown residents," Oriol says.
Children's has already created a "hospital within a hospital" concept in two other hospitals in the region to bring specialized pediatric care closer to patient populations, and it plans a similar facility for its downtown presence. The present data center will probably stay in its existing location to support that effort. With the split-cluster architecture, this will provide full IT support for medical systems at both sites, transparent to users. And in the meantime, either site can serve both facilities through the Ciena core network with no noticeable lag times. This ensures full IT service at the new site from Day One.
The high-end core application has other dividends. Because the hospital owns the network down to the fiber, it does not need to encrypt patient data flowing between the two sites. It does encrypt all data going over public networks, for instance, from satellite locations it maintains in several states surrounding its Denver base.
Oriol also notes that the instant service improves patient care, but it is impossible to quantify. "We left it as a quality issue. Obviously, we want to be sure we can deliver the information clinicians need to provide the best care possible."
So why Ciena? "Basically, EMC recommended Ciena, and our engineers played with it and got the sense that it could do the job," says Oriol. "We bought it in December 2004, and it has been working beautifully."