But as virtual machine technology moves out of development labs and into production server environments in large numbers, some administrators are finding that the growth of virtual servers is getting ahead of the tools available to effectively manage them.
Existing server-monitoring tools are increasingly aware of virtual servers, but most aren't yet sophisticated enough to interpret feedback in a virtual machine context -- much less act on it. "They don't take into account the particulars of virtual machines," says Frank Gillett, an analyst at Forrester Research Inc.
For example, a virtual machine may be running at 100 percent utilization but using only a fraction of the underlying server's resources. "Some of the things you monitor no longer mean the same thing," Kuzmack says.
"It would be nice if all of our standard tools worked in the virtual space, but they don't, and it doesn't look like they're going to anytime soon," says Norm Fjeldheim, CIO at Qualcomm Inc. in San Diego. He is evaluating tools targeted at virtual machine management to fill the gap.
For many organizations, identifying the root cause of virtual server problems and rectifying them remains largely a manual process. As the number of virtual machines in the data center increases, solving those problems in an automated way becomes more urgent.
Performance monitoring is just one aspect of virtual machine management. Other tasks include optimizing the mix of virtual machines that should reside on each physical server to achieve the best possible performance; automating virtual machine provisioning, load balancing, patch management, configuration management and fail-over; and enabling policy-based orchestration to automatically trigger the appropriate responses to events.
For some functions, such as patch management, existing tools work fine, says Paul Poppleton, a senior staff engineer at Qualcomm. In other areas, he says, "we're getting the best wins on the tools that take into account the fact that systems are virtualized."
Even organizations just starting virtual server projects can quickly run into management challenges. Once the decision is made to introduce virtual servers, the numbers can increase much more rapidly than expected because it becomes easier to procure new servers, says Poppleton. "Tack on 20 percent or 30 percent to what you planned on for growth, because it can really take off on you," he warns. Qualcomm has 1,280 VMware ESX Server virtual machines companywide that run a mix of Windows and Linux. About 850 virtual machines are running in Qualcomm's data center, with each physical server hosting an average of 10 virtual machines.
Even fine-tuning the performance for as few as 10 virtual machines can be a challenge. "One place where we've had trouble is trying to manage the resources on a single physical host," says Poppleton. VMware Inc.'s VirtualCenter 2 management software, which Qualcomm is beta-testing, should help with that, he says. The software is expected to ship in the first half of this year.
Finding the right tool
Kuzmack is also beta-testing Virtual-Center 2. One component, Data Resource Scheduler, aggregates servers into pools that can be assigned to groups and managed through policies that the administrator creates. "We can take a group of physical servers and carve them up into resource pools where we can set high-level and low-level limits and resource guarantees," he says. Another feature, Distributed Availability Service, automatically moves virtual machines to a new physical server and restarts them after a physical server fails.
Kuzmack is also working on basic monitoring. He wants to integrate VMware's Vi
rtualCenter control software with Gannett's Microsoft Operations Manager (MOM) software. Other than basic performance metrics, "we haven't determined what we want to expose to the MOM console yet," he says.
Jim Ni, senior technical product manager at Microsoft Corp., says the company is working on adding more virtual machine management capabilities to its management tools, but it's not there quite yet. For example, Systems Management Server can manage physical machine image libraries but can't differentiate between an image for a physical machine and a virtual machine image.
Poppleton is using VirtualCenter but says he also needs more cross-platform tools. "Right now, we're a VMware shop, but in the future that may not be true," he says. He's considering using VirtualIQ, a virtual machine management tool suite being developed by ToutVirtual Inc. in Carlsbad, Calif. It supports automated provisioning, capacity management and security. He's also looking at tools for "grid-style management of physical and virtual systems," such as VM Orchestrator from Platform Computing Inc. in Markham, Ontario. VMO optimizes capacity by dynamically allocating and controlling virtual machine resources and utilization levels based on user- defined policies.
Both ToutVirtual and Platform Computing currently support VMware, but both have also announced plans to support Microsoft Virtual Server and the open-source Xen virtual machine monitor. In the interim, Poppleton's staff has had to develop some of its own tools.
"We're in the first phases of automated provisioning," he says, adding that Qualcomm wrote its own utilities because it couldn't find a tool that met all its needs.
Christopher Ware, assistant vice president of technology services at a Wall Street brokerage house, recently deployed BMC Virtualizer, an orchestration and provisioning tool that's part of a suite of virtual machine management tools from Houston-based BMC Software Inc. "It gives us policy-driven responses to resource utilization issues," he says. The tool also can support virtual machines created with Xen, VMware and Virtual Iron, all of which are running at the brokerage.
Like other cross-platform tools, Virtualizer integrates with APIs from VMware and other virtual machine software to automate the execution of those proprietary tools. For example, Virtualizer supports the automated movement of virtual machines between physical servers in VMware environments using VMotion.
Ware, whose company prefers to remain anonymous, says that getting BMC Virtualizer to work with all of his virtual machines wasn't exactly a plug-and-play experience -- especially since no common standards exist. In his environment, "there was a significant amount of coding and customization required to do a lot of the virtual provisioning," he says, and that accounted for the bulk of the deployment costs. A BMC spokesman says the product works out of the box with more than 50 applications.
"Where standards would really help is these interfaces for controlling and manipulating virtual machines and getting feedback on them," says Forrester's Gillett, noting that basic virtualization is being commoditized at the chip and operating system levels. VMware, Xen developers, Intel Corp. and others are discussing the need for higher-level standards but have yet to reach a consensus.
Stewart Hubbard, director of IT engineering at Coldwater Creek Inc., a clothing retailer in Sandpoint, Idaho, has about 60 production servers running in ESX Server virtual machines. He says resource allocation is a big issue.
"When a particular machine starts to get hammered and the resources aren't available, you'll see a noticeable dip, and the end users will notice it," he says. Right now, he uses VirtualCenter to address the issues, one machine at a time. And because not all application vendors support their software running in virtual machines, he uses a tool from PlateSpin Ltd. to migrate virtual machines back onto physical servers for technical support purposes.
While virtualization-specific management tools are a good option today, ultimately, "you need something that manages the virtual and the physical together," says Gillett. BladeLogic Inc., which offers a virtualization-aware configuration life-cycle management tool, contends that a single set of policies should control both worlds. Once the policy and "personality" of a server are defined, "every instance of the server, whether physical or virtual, is treated more as a compliance exercise," says Vick Vaishnavi, director of product marketing at Blade-Logic in Waltham, Mass.
Qualcomm's Fjeldheim says he sees no reason to wait for all the pieces of the management puzzle to come together. Integrated tools would be nice to have, but "we're happy to use the tools that are specific to the virtual environment. It's not that big a deal," he says.
Today, it's the early adopters in industries such as financial services that are encountering management issues with what vendors like to call "virtual machine sprawl," but the pain will spread as more companies ramp up the number of virtual servers in use between now and next year, analysts say.
"Implementing this stuff is the beginning of rethinking how you manage your IT infrastructure," says Gillett. "You won't get far without thinking about data center automation."
Sidebar
Real self-service for virtual provisioning
Virtualization technology may make provisioning of individual servers faster and easier, but the process can still take up increasing amounts of administrators' time as the number of requests increases. The most demanding groups tend to be development teams, which routinely build and tear down virtual machines as testing platforms.
"We were getting a lot of requests from developers to spin up these machines in ESX Server," says Stewart Hubbard, director of IT engineering at clothing retailer Cold- water Creek. The developers didn't like to wait, but giving them access to the production interface, VMware's VirtualCenter, wasn't feasible. So Hubbard is rolling out Slingshot, a self-service provisioning tool from Akimbi in San Mateo, Calif., that lets developers provision and deprovision their own virtual machines.
Akimbi is one of several vendors offering such tools. Others include Surgient Inc., Enigmatec Corp. and Platform Computing.
Developers at Coldwater Creek choose from a library of virtual machine images. Predefined amounts of virtual machine resources are allocated to individuals or groups. Hubbard says the developers can provision the resources they need when they need them and then release them when they're done. "There aren't [idle] resources running all of the time," he says. In addition, Slingshot can provision virtual environments for multitiered applications and group together virtual servers into a single, integrated environment.
The arrangement is a win-win setup for IT and the end users. Administrators retain control over resource use, while developers get the resources they need without going through red tape. "We want to segment the developers from production but still give them the ability to work as quickly as they need to," Hubbard says.