Along with Apache, Hadoop uses an iteration of MapReduce, a programming technique that originated at , for building parallel programs. Running with Hadoop, MapReduce enables it to perform parallel batch processing.

"The next generation of HDFS will be more resilient, available and reliable," Papaioannou said. "We expect to put it all together in a release some time soon. That's an exercise of collaboration with rest of the development community."

Yahoo also just launched a new project called , which is a table metadata management schema for Hadoop.

"That will help drive different use cases," he said. "It just went into Apache version last week."

Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at or subscribe to . His e-mail address is .