Using the Spirent equipment, we designed real-life traffic patterns based on high-volume throughput from the University of Hawaii production network. We wanted to know if this new generation of 10-Gig hardware had really matured enough to be viable in a standard enterprise environment.
When our participants were done with these baseline performance tests, we enabled an optional testing period. One challenge looked to bang on self-defending network technologies. Vendors could choose to participate in this test or design other tests in conjunction with our lab staff to show off features about their product suites that they deemed important or unique.
For our baseline tests, we had to find out just how much data the intercore links could handle. For this we created Baseline 0, where we hooked up 20 copper gigabit feeds into each of the edge distribution switches and ramped up the best-effort layer 3 data until we started dropping packets. Because any routed network requires some interswitch communication, we needed to quantify just how much bandwidth we were giving up.
Next, we ran Baseline 1. This test sought to confirm that high-priority VOIP and video traffic were actually being protected in the proper QoS queues. For this test, we fired up the Spirent Abacus ICQ boards to create hundreds of VOIP calls coming in from each edge and crossing the core. We then used the Abacus' PESQ (Perceptual Evaluation of Speech Quality) and 3Tone (three test tones used for stress testing) to measure how well the switches preserved the quality of voice calls.
Next, we used the Abacus VQM (Video Quality Measurement) to monitor the multicast video streaming edge to edge, again using its own metrics to maintain a tight view of video quality. Lastly, we used the Spirent Avalanche to send synthetic application layer traffic through the network targeted at the Reflector unit on the other side of the network. Together, Avalanche and Reflector simulate real application-style network traffic.
Our Baseline 2 combined Baseline 0 and Baseline 1 to guarantee that we were oversubscribing the intercore links while we sat back and watched the smoke. The MOS and PESQ for VOIP quality and the V-Factor for video quality are all measured on a scale of 1 to 5, with 4.5 being a realistic maximum or MOS and for PESQ and 4.9 being the mark for DVD-quality video. R-Factor measures voice quality on a 100 point scale, with 90 to 100 corresponding to MOS scores of 4.3 to 5. The goal was to measure a slice of a midsize corporation's general traffic flow. We wanted to confirm that each vendor really could protect the voice and video flows according to the standards provisions. As a side benefit, this also gave us a historical view because we've actually done variations on this test for the past 10 years.
What we found confirmed our theory that our baselines were reasonable, and for this particular scenario, nearly identical. In our voice tests, both vendors' gear scored a 4.5 for average PESQ and 93.2 for R-Factor. HP's V-Factor average was around 4.95; Extreme's was about 4.91.
The reality is that we had expected no appreciable variances between our test vendors during the baseline performance tests. This is because 10-Gig networking really has matured enough that large variances between switch products in the day-to-day performance department simply don't happen anymore. Even if Cisco had participated, we're confident that all the solutions would differentiate themselves only in the optional portion of our shootout.
SIDEBAR
Spirent melds voice, video, and data into networking test gear
We've been doing network-switch testing for more than 10 years now, and the concept of a triple-play test (voice, video, and data) has been a part of our plans from the early days of ATM OC-12 through Gigabit Ethernet's birth and now to 10 Gig.
In the old days, however, we'd need to generate our own video and voice data, mainly because, although our traffic-generation tools could emulate video and voice, they couldn't quantify traffic quality; we had to measure that ourselves. Fortunately, with the newest generation of testing gear that Spirent brought to this showdown, the rest of you won't need to suffer as we did.
One of our favorite pieces of Spirent test gear is the Abacus, a VOIP traffic emulator and quality quantifier that has served us well in melting VOIP PBXes throughout the years. New to this platform is Spirent's IPTV Video Quality Test System, which is designed to catch MPEG-2 multicast video. Targeted at cable television providers or anyone deploying IPTV, the VQ Test System yields up V-Factor, Spirent's name for its video quality measurement, providing a quantitative benchmark of video quality from the user's perspective. Previously, we'd measure video quality simply by running it against heavy background traffic noise and looking for pixelation. Now, we can generate repeatable and numerically rendered quality results that offer much greater depth for testing analysis.
Spirent also brought the latest generation of its core network testing tool, now called the STC (Spirent TestCenter). Spirent sent us three of the 5RU STR appliances (model 5000A). An advanced testing suite, the TestCenter includes all the routing flavors under the rainbow. Spirent manages this breadth with a slick new Windows-Explorer-like GUI that allows much easier manipulation of network testing. For instance, it can split up ports in the chassis for automated testing across various parts of the enterprise. Backed by a TCL-based scripting language, the STC has more than enough power and customization capability to suit a wide target audience.
Spirent is actually at Version 1.20 of TestCenter, which has a noticeably easier-to-learn user interface. Unfortunately, 1.20 was literally in its last week of beta testing when our 10-Gig test began, so we had to content ourselves with Version 1.10, which required that we get some training from Spirent. Even so, we were highly impressed with TestCenter's new functionality, including not only the power of its new 10-Gig-capable blades and chassis but the overall direction of its testing software.
Whereas Spirent's SmartBits, Abacus, and Avalanche/Reflector product lines have long remained entirely separate products, TestCenter's new direction is obviously to combine these tools, allowing them to communicate and even share test data across devices. Full-feature synergy isn't quite all here yet, but TestCenter is definitely a step in the right direction.
SIDEBAR
Podcasts: Microsoft and Sun groom their OSes for 10 Gig
With 10 Gig making inroads into mainstream datacenters, we had to check and see how server OS vendors were responding to this fast new media. In the past, even when straight Gigabit Ethernet adapters were first introduced, operating systems simply weren't prepared to handle the increased bandwidth.
So we called Microsoft and Sun Microsystems to find out what specific features and capabilities Longhorn and Solaris have to take advantage of 10 Gig. Turns out, 10 Gig has been on both manufacturers' minds for quite some time. We had in-depth discussions with Ian Hameroff, senior product manager at Microsoft, and Darrin Johnson, Solaris engineering manager at Sun, and we recorded those conversations for your listening pleasure.
Hameroff described several 10-Gig-optimized features in Longhorn, including a redesigned TCP/IP stack and a driver program designed to make 10-Gig NICs more compatible with Windows. Microsoft's redesign of its TCP/IP stack had far more to do with fixing existing problems and advancing Longhorn in general than it did with specific support for 10 Gig, but happily the one has clearly benefited from the other. Go to http://www.infoworld.com/mp3/IFW_10gig_msft_07-07-06.mp3 to download the interview with Hameroff.
Meanwhile, Johnson discussed Sun's long-time customer partnership related to optimizing and standardizing support for 10 Gig under Solaris, as well as a driver program of its own for these exceptionally fast NICs. Sun, too, has taken a close look at its TCP/IP implementation but was a little cagier about the specifics. Go to http://www.infoworld.com/mp3/IFW_10gig_sun_07-07-06.mp3 to download the interview with Johnson.
Although we tried to find an adequate Linux operating system representative to discuss the future of 10 Gig on Linux, the Penguin community has been quiet on the topic. "We're expecting to see more support for 10 Gig under Linux for specific applications, especially computing clusters," said Paul Lindo, CIO of FB2, a New York-based IT consultancy. "But so far ... none of the big Linux OS platform distributors are really making noise about 10 Gig yet."