Los Alamos shuts down supercomputers as fire advances

29.06.2011
The Los Alamos National Lab has shut down two of its largest supercomputers, as wildfires continue to burn near this sprawling New Mexico facility.

There were no Wednesday, but smoke is prevalent throughout the area.

The National Weather Service has called air quality "potentially unhealthy," and Los Alamos residents are being urged to minimize exposure to it. The lab will be closed through Thursday.

A Los Alamos spokeswoman said the laboratory conducted an "orderly shutdown" of two of its largest supercomputers, 's Roadrunner, the first the in 2008, and now the 10th ranked most powerful supercomputer in the world, and Cielo, a Craig system that is ranked No. 6 on .

The supercomputer shutdowns were conducted "early on," but an exact day or reason for the action wasn't clear. The laboratory was first closed on Monday in response to wildfires that have now burned more than 100 square miles.

But smoke is a potential threat to data centers and IT equipment. It can trigger a shutdown of air handling systems and IT equipment. Particulates from smoke can damage components. And this has been a record year for smoke-producing wildfires.

According to the National Climate Data Center, there were 6,625 fires which burned approximately 1.1 million acres in May -- the most acres burned during the month of May on record. The amount of acreage burned so far this year is 3.45 million acres, "the largest in the 12-year period of record," the climate center said.

The problem with smoke is that it "is so fine that it can get into things like your hard drive, it can get between your processors and your motherboards, it can get in between your ram and motherboards," said Jason Burnett, the director of infrastructure at NeoSpire, a managed hosting provider in Dallas.

Texas has experienced some big wildfires, and it also had a lot of heat, which is why NeoSpire uses internally air-conditioned air for cooling and not external air, in what's called a closed-loop system. "For our specific purposes I wouldn't be affected [by wildfires] at all because I don't take any air from the outside in," Burnett said.

He said his air cooling approach is a common one in the southwest because of hot temperatures.

In more northern and cooler climates, data centers are increasingly using outside air, or air-side economizers. These designs typically include sufficient filtration to address external issues, said Tad Davies, executive vice president of the Bick Group, a company whose work includes design.

In the case of wild fire activity, the data center would resort to a backup closed loop system. If smoke has infiltrated the building, the smoke detectors will shut down the air handling units.

Once the cooling systems are shut down "you are kind of done," said Vince Renaud, managing principal and vice president of the Uptime Institute. Meaning that once cooling systems turn off, the IT equipment will hit their temperature thresholds rather quickly. At that point, systems have to be shut down, he said.

Jeff Pederson, vice president of data recovery operations at Kroll Ontrack, whose work includes recovering data from damaged systems, said the threat of smoke is more of an issue for small businesses that aren't running traditional data centers with monitoring equipment.

Pederson said he sees problems in businesses and in homes with cigarette smokers, where the smoke affects performance and increases the likelihood of a physical crash.

But Burnett does expect to see more wildfires in Texas. He said he grew up in the country, on a cattle ranch out in south Texas, and sees the population getting closer to rural areas and places which don't have same firefighting equipment available in developed areas.

"I think [wildfires] are going to be a lot more prevalent as the urban creep keeps happening," Burnett said.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at or subscribe to . His e-mail address is .

in Computerworld's Mainframes and Supercomputers Topic Center.