This system handily bested their previous system, demonstrated at Supercomputing 2007, which scanned 1 billion files in three hours.

Key to the increased performance was the use of speedy flash memory to store the metadata that the storage system uses to locate requested information. Traditionally, metadata repositories reside on disk, access to which slows operations.

"If we have that data on very fast storage, then we can do those operations much more quickly," said Bruce Hillsberg, director of storage systems at IBM Research Almaden, where the cluster was built. "Being able to use solid-state storage for metadata operations really allows us to do some of these management tasks more quickly than we could ever do if it was all on disk."

IBM foresees that its customers will be grappling with a lot more information in the years to come.

"As customers have to store and process large amounts of data for large periods of time, they will need efficient ways of managing that data," Hillsberg said.