All in all, a good day. This series of events marked the beginning of the InfoWorld blade server shootout. Three of the Big Four blade server vendors -- Dell, HP, and Sun (minus no-show IBM) -- presented their latest and greatest blade server products for our careful inspection. As the smoke cleared at the end of the week, it became clear that the new crop of blade servers is a giant step up from the previous generation.
The test plan was actually quite simple: We conducted performance tests using the SPEChpc benchmarking suite and examined server management tools. Time was short, so each solution got only a single day to strut its stuff, but the vendors had more than a month prior to the test to prepare their wares. This included preparing the blades for the SPEChpc tests and installing any accompanying software.
We chose the SPEChpc tests not only because we're interested in the blades' HPC performance, but also because they would give each solution a thorough workout, extending to CPU, memory, and interconnect performance. We allowed vendors their choice of hardware, including the type of interconnect to be used, with the only requirements being that the SPEChpc tests were limited to 16 sockets and 32GB of RAM. Each socket might hold a dual- or quad-core CPU, and each blade might have two or four CPUs, but otherwise, the goal was to see the best of the best.
Big Blue's big blank
Test week started with a bang -- or maybe a fizzle, depending on your point of view. At the very last minute, and after months of preparation, IBM pulled a no-show.
Whether this was due to internal coordination problems or fear of competing against HP, Sun, and Dell is open for speculation, but we made several attempts to get IBM back in the game. In fact, when first confronted with the news that they weren't going to make it to the lab, I broke the rules and extended IBM's deadline by 10 business days.
Those 10 days passed with nary a whisper from Big Blue. They followed up a week or so later claiming that they could deliver hardware to the lab in another two weeks, but given their track record, I wasn't going to hold my breath. The test had been over for two weeks, anyway. Too little, too late.
HP BladeSystem c-Class
First up on the block was HP's brand-new BladeSystem c-Class. The c-Class substitutes 2.5-inch SAS drives for the 3.5-inch SCSI drives found in the previous crop of HP blades, and it abstracts much of the blade hardware into a modular backplane that boasts 5Tb throughput. These two factors mean HP's blades are half the size of their predecessors, yet offer more connectivity options and processing power.
The chassis is a complete redesign, boasting a nicely trimmed up-front LCD panel display that can be used to configure a surprising number of chassis operating parameters. The panel has a Web UI counterpart that matches the display exactly, easing 'remote hands'-type administration. Up to 16 blades can fit into a single 10U c-Class chassis with a maximum power draw of 3.6kW. The N+N power supply configuration is also nicely handled, with six hot-swap power units laying low at the bottom of the chassis.
One of the more attractive aspects of blade systems is the ability to mix and match different types of blades within a single chassis. The HP c-Class currently offers three different ProLiant processing blades: the BL460c, an Intel EM64T-based blade; the BL465c, the AMD Opteron counterpart; and the BL480c, a 2P Intel EM64T-based blade. In addition to these blades, HP also offers disk-only blades, which can handle as many as six 2.5-inch SAS drives that appear as local disks to the immediately adjacent blade in the chassis -- a very nice touch.
Any of these blades may occupy a single chassis in any density. An interesting and welcome detail is the single internal USB port on each blade ostensibly present to allow use of a USB licensing dongle, because, unfortunately, many applications are licensed in this fashion.
Our c-Class review unit contained preproduction BL460c blades sporting the new Intel quad-core Xeon CPUs. Running at 1.866Ghz with a 4MB L2 cache, a 1066 Mhz FSB, and 4GB of RAM per socket, these BL460c blades proved quite powerful. They turned in respectable SPEChpc scores, due in no small part to the 16-socket limitation of the testing balanced against four cores per socket. However, the lower clock rate per core and limited FSB may have cost HP in the SPEChpc tests, as their scores fell generally in the middle of the three solutions. It's also quite possible that more time needs to be spent on compiler optimizations for these newborn chips.
Like all the other vendors, HP chose InfiniBand as the interconnect for the HPC tests using an external Voltaire switch. But unlike Sun's X8400 blades, much of the c-Class I/O is handled internally with switching modules. This backplane switching architecture provides a closer relationship between the blades and significantly reduces cabling, but proved to be problematic in the lab: The HP engineers struggled with odd issues relating to InfiniBand connectivity and performance throughout the testing. It wasn't until the very end of the day, in fact, that they were able to complete the SPEChpc suite to satisfy the testing requirements.
On the network I/O side, though, HP can run with Cisco switching modules to keep intrachassis communication within the chassis itself. The Cisco modules behave exactly like external Cisco switches, which will please network admins already familiar with Cisco's hardware. External uplinks take the form of eight gigabit Ethernet ports per switch module, resulting in a total of eight ports that can be trunked to datacenter core switches.
Management ups and downs
BladeSystem runs its own internal management console, accessible via the Web, that can stand alone or be integrated into an HP Insight Manager installation. Multiple c-Class chassis may be managed collectively in this manner, regardless of whether Insight Manager is in place, which is quite useful for large data centers. Administrator-driven tools offer a wide array of monitoring options, from current and maximum power utilization and environmental data to blade health and performance information.
Internal chassis management is even more impressive. The chassis has enough smarts to determine the heat and power loads present and advise admins on proper fan population and placement. It will adapt power supplies as needed and where needed, as well as drop power levels to quiescent blades when possible. This results in lower heat production and power consumption, which are hot buttons (pun intended) as far as blade development and deployment go.
Not everything with the HP blade, however, worked as smoothly as its chassis management. The console redirection available in each blade's ILO cards was somewhat lacking, with problematic mouse tracking and display artifacts. This can be very irritating when work must be performed directly to the console of each blade. The only other console redirection method involves using a front-mounted dongle port to connect a keyboard, monitor, and mouse directly to each blade. Not pretty, but it does work.
We also encountered a few oddities with the chassis, including one blade that seemed to have intermittent connectivity problems and another that spontaneously lost contact with its internal SmartArray RAID controller. It's highly likely that the pre-release nature of the blades contributed to these issues.
Nevertheless, HP completed the full SPEChpc benchmark suite runs at the small and medium dataset levels within a single day, complying with the testing parameters. A few lab hiccups aside, the BladeSystem c-Class is an impressive piece of engineering. The wide variety of blade options, including the disk-only blades; up-front display; adaptive power and cooling features; and density show that the c-Class definitely adheres to HP's 'Invent' slogan.
Sun Blade 8000 Modular System
Although the other blade solutions in our test ranged in size from 7U to 10U, Sun's system came in the door at a whopping 19U. Of course, Sun's take on blades is a little different: It was the only blade solution to support four CPUs per blade, and can handle 10 blades per chassis. With dual-core AMD Opteron CPUs, this equates to 160 cores in a single 42U rack.
That rack had better have plenty of power and cooling, though, as the Sun Blade 8000 draws a significant amount of juice, requiring approximately 9kW (actual draw is generally lower). That's quite a lot compared with the HP and Dell blades, which pull roughly 3.6kW each. Luckily, the Sun system's density makes up for its power thirst.
It took Sun's engineers quite some time to get the tests up and running, and the Sun Blade 8000's results in the SPEChpc benchmark weren't the best. They did better in the SPECseis test, and I'm certain that if they were given more time to optimize the other two tests, Sun's overall results would have been much better.
Sun's X8400 Server Modules are large, each with two 2.5-inch SAS or SATA drives and a RAID0/1 controller. Memory expands to 64GB per blade with 4GB DIMMs, granting a fully populated chassis a total of 640GB of RAM across 40 sockets. Those sockets can hold Opteron 870s at 2.0Ghz, 875s at 2.2Ghz, or 885s at 2.6Ghz, all with 1MB L2 cache. And last week, Sun announced availability of AMD Rev F processors in its new X8420 Server Modules.
I/O options are plentiful. Each blade can handle as many as six different external I/O forms, and there are two different methods of delivering the physical I/O ports to the blades themselves. The X8400 is more focused on pass-through ports than using integrated switching; it has wide Network Express Modules that aggregate a single 8x PCI Express lane from each blade, as well as smaller Express Modules also leveraging a single 8x PCI Express lane, but built into the slim PCI-SIG form factor. These smaller modules reside at the top of the chassis, and are designed to provide more granular I/O access to each blade.
The two NEM modules in our test unit delivered four gigabit Ethernet ports to each blade. The InfiniBand interfaces slotted into the Express Modules, delivering two InfiniBand ports to each blade on a single module. This design is quite flexible and its hot-swap capability is certainly attractive.
Although the Sun Blade 8000 is technically a blade system, it fits the image of a modular server system. The raw horsepower available across each blade's four sockets and the impressive array of modular I/O options position the system directly into the HPC and virtualization arena. This is not a system to run simple Web or directory servers -- unless they're virtualized.
Virtualization-ready
Because of its power, the Sun Blade 8000 really doesn't compare directly to the other blade systems in the test. The Dell and HP solutions can go three ways (standard server builds, HPC, and virtualization) but the Sun solution finds its sweet spot in HPC, high-end database, and virtualization tasks.
The Sun Blade 8000's hardware fits a virtualization build-out plan like a glove. Available I/O options are far better than the other blade systems, and the four sockets per blade, the NUMA (Non-Uniform Memory Access) inherent in the AMD Opteron technology, and maximum RAM supported all make virtualization a foregone conclusion. As a VMware engineer speculated during testing the week after the blade server tests, 'Wow ' at standard loads with quad-core CPUs, this thing could support 600 virtual machines all by itself.' Enough said.
The 8000's management framework falls in line with Sun's N1 Network Manager, and the chassis' Web management interface is quite quick and usable. Of all the solutions tested, Sun's Java-based remote console application is the fastest and easiest to use, not to mention that it runs on all workstation platforms.
Sun's ILOM Web interface was not only the fastest, it was also the easiest to navigate of all three solutions. Working from the chassis Web UI, a single click will launch the console application with tabs linking to each blade's local console. Nice.
Backing up the UI is a set of redundant CMMs (Chassis Management Modules). Each module can be separately linked to the network via a single gigabit NIC and all share a common IP address, providing a fast fail-over in the event of hardware problems. The local ILOM card in each blade is also accessed via internal bridging to these Ethernet ports, so these links are very important to normal chassis operation.
The Sun Blade 8000 is a masterpiece of engineering and aesthetically attractive to boot. At US$100,000 as tested, it's definitely not a low-cost solution, but its focus isn't on the low-end market. This is a system that begs for a heavy workload -- and delivers.
Dell PowerEdge 1955 Blade System
Common thought may lead one to believe that Dell is somewhat behind the curve in the blades world. Much more time and ink has been spent on the blade technology available from Dell's competitors, and hence, Dell doesn't enjoy the mindshare of Sun, HP, and IBM. Even we didn't expect Dell to put up too much of a blade showing.
Common thought turned out to be far from the truth. Whereas the other vendors spent six to eight hours of their testing day working to get the SPEChpc benchmarks running properly and with the best results possible, Dell ran the full benchmark suite in their 90-minute preparation period the day before their official testing day -- and those 90 minutes included their initial chassis powerup and system check procedures.
Not only that, but the Dell PowerEdge 1955 produced the best SPEChpc numbers by far of any of the blade systems tested. Color us surprised, and not a little chagrined at our original assumptions.
Dell's high marks on the SPEChpc tests have plenty to do with the hardware, but they're also the result of heavy tweaking and preparation by the Dell engineers. It's clear they're serious about HPC performance.
The PowerEdge 1955 solution isn't quite as physically elegant as the others in this test, and it certainly lacks the panache of the HP BladeCenter's LCD panel. Dell also uses larger blades than the dual-socket HP BladeSystem c-Class, but packs quite a bit of horsepower into the 7U chassis, which is the smallest of the tested solutions.
Ten blades fit into a single Dell chassis, resulting in a single rack density of 60 dual-socket systems. With the soon-to-be-released quad-core Intel chips, this equates to 240 cores per rack, with a maximum power draw of 3.6 kW.
Each blade sports two 2.5-inch SAS or SATA hot-swap drives with hardware RAID0/1, expanding up to 32GB of DDR2 RAM and dual or quad-core CPUs. The external I/O layout is similar to the HP solution, with integrated switching across a passive midplane and either an integrated Cisco 3030 or a Dell PowerConnect 5316M gigabit blade switch module. Straight gigabit Ethernet pass-through modules are available, as well.
On the FC (Fibre Channel) side of the aisle, you have both McData and Brocade 4GB FC switch modules available, as well as a pass-through module. The PowerEdge 1955 handles InfiniBand with a Topspin pass-through module providing a single port per blade.
In the lab, our PowerEdge 1955 chassis sported a PowerConnect 5316M switch, which is accessible at the console level via the chassis management tool's CLI. Of the 16 ports on the switch, 10 are reserved for blades at one port per blade, and the other six are broken out into RJ45 ports on the back of the module.
We successfully trunked this module to a Cisco 4948-10G switch to provide 6Gb of throughput to the main lab network. It would be nice to see 10 Gigabit Ethernet support in this chassis, but then, none of the blade systems we evaluated could do 10 Gig -- yet.
Access made easy
One of the Dell system's unique features is the integrated KVM switch. It's a Dell-branded Avocent switch that has internal connections to each blade, and breaks out into a standard PS/2 and VGA port via a dongle on the back of the chassis. This permits quick and easy direct KVM access to each blade, and can uplink to another KVM switch relatively easily.
Further, this same KVM module doubles as an Avocent digital KVM port, permitting instant integration into another Avocent/Dell KVM switch to make management even easier. Each blade also has a front-mounted dongle connector that can support a directly connected monitor and keyboard. It's the best direct (non-IP) console management of any blade system.
The PowerEdge 1955 Blade System would be quite at home in a standard datacenter running a single server per blade, in an HPC environment serving as a low-footprint collection of compute nodes, or in a virtualization scenario (the Intel VT extensions are available, but disabled by default). In fact, when VMware Virtual Infrastructure Server 3 was evaluated in the lab, VMware engineers chose to use the Dell chassis to run all their tests -- partly because the Sun Blade 8000 system was still in use re-running the SPEChpc tests, but also because they were sure that there were no compatibility issues with the Dell blade system, and it had the performance levels they needed.
Compared with the Sun unit, the Dell PowerConnect's I/O options are relatively limited but those available are enough for most architectures. The small form factor, reasonable power draw, and overall performance reflects well on Dell engineering, and results in a well-priced and well-appointed product.
Blade futures: 10 Gig ahead
After seeing all three of these blade solutions in action (and cleaning up the broken coffee press), we couldn't ignore the results: Blade technology is undergoing a renaissance of sorts.
Vendors are taking advantage of newer, less power-hungry CPUs and branching out into new levels of I/O that directly combat the common complaints about blade systems, such as heating and cooling concerns and management difficulties. As more infrastructures move toward centralized storage and virtualization, it's impossible to miss the impact that blade systems like these will have.
The near future will introduce another key element into the blade server picture: 10 Gigabit Ethernet. The three blade solutions we tested still rely on link aggregation of individual gigabit Ethernet ports or pass-through interfaces to deliver enough bandwidth to a single blade chassis, but all vendors are currently developing 10 Gig modules that will deliver a one-two punch of significantly reduced complexity and cabling. Once these modules are available, an entire chassis can run with only two 10 Gig connections and power cabling -- and costs will decrease even further as 10 Gig ports drop in price.
That doesn't mean the adoption battle is over. The toughest challenge these systems face isn't one of providing the right mix of power and connectivity options, but rather the real-world planning requirements. It's easy to buy one or two 1U servers that may slide beneath the purchasing limits of many IT departments, but it's harder to push through requisitions for the tens of thousands of dollars necessary for a blade system. Without immediate justification for half a dozen or more servers at a time, it may not be possible at all until it's time for a wholesale server refresh.
However, it's easy to justify a blade system when looking at virtualization, as it's cheaper to ramp up virtual servers in a blade-based infrastructure -- not to mention the overt cooling and power cost reductions. The additional savings in cabling, switch ports, and administration overhead is harder to quantify, but certainly present.
The Dell, HP, and Sun blade solutions we tested have a wide price range, but the low-end cost of entry is getting lower just as the products are getting better. Blades aren't suitable for every infrastructure, but as our test results show, their increasing power and flexibility mean it's getting easier to justify them in the enterprise world.
Brian Chee, director of Advanced Network Computing Laboratory at the University of Hawaii's School of Ocean and Earth Science and Technology, contributed to this article.