Government lab deploys ethernet network to support HPC facility

09.06.2012
Rutherford Appleton Laboratory (RAL), one of the UK's principal government labs and part of the Science and Technology Facilities Council (STFC), has deployed a large-scale ethernet network to support a new high performance computing (HPC) facility that has 4.5 petabytes of storage online.

The facility benefits from a 10 gigabit network that supports a parallel Panasas storage system and a highly virtualised computational system, which is used by scientists for Earth observation and climate modelling.

It was funded from the 2011 investment of £145 million in e-infrastructure by the UK government Department for Business, Innovation and Skills (BIS).

Dr Peter Oliver at STFC explained how RAL used Gnodal GS-Series switches in the network deployment.

"The kit was on the ground in late February and the Gnodal network was installed pretty soon after that, with the Panasas storage kit arriving in early March. It took us six hours to have 4.5 petabytes of storage online from the time we put the first switch on," said Oliver.

"We are heavily invested in Avaya networks and Force10, but we chose Gnodal because we didn't want an IP-based network, which would have introduced delays and latency. This would have decreased the performance of the parallel storage."

He added: "Gnodal are also a small company and we like dealing with small companies. Sometimes when we have dealt with large international corporations we have had to explain the business problem to them many times over before we get what we need."

Four GS4008 systems are deployed at RAL, comprising a total of 160 10GbE Edge ports and 32 40Gbps Gnodal Fabric ports in a single spanning system.

Jonathan Churchill, also at STFC, explained that the main challenge with deploying the network was the copper cabling that had to run under the floor in a complicated format.

"We have just installed a cold aisle containment system on the 12 racks that hold the HPC infrastructure, which forced all the cabling to go under the floor," said Churchill.

"The 12 racks for this system are distributed across one cold aisle, and the switching isn't all in one rack. So, the cabling had to go from one rack, into the floor, and then back up on the other side. You can imagine that this wasn't the easiest thing to do."

RAL is now also looking to expand the network, but is working with Gnodal to come up with a solution that avoids further cabling going underground.

"We were very close on the budget with this deployment and as a result we may have painted ourselves into a corner with regards to scalability. Now we are thinking about how we can expand this Gnodal network without going through this cable nightmare," said Oliver.

He added: "We are working actively with Gnodal to come up a solution to this. Want to add more compute servers to the infrastructure. At the moment we have lots of data nodes, but the scientists need more compute infrastructure to do thorough analysis."

"We are essentially looking at adding another Gnodal switch, which will give us access to an additional 40 ports, so an additional 20 servers."