Conquering Big Data with stream computing

10.04.2012

Mahmoud's research led to a paper published in late 2011 on the use of stream-computing to analyse the data as it is produced, instead of storing it to be mined later. He explains that stream-computing is much like putting your finger in the air to gauge which way the wind is blowing, it is quick and relatively effective.

"With stream-computing, rather than storing the data we store the queries we want to apply to it. We probe the data with questions using the queries, and are given real-time answers as it comes by," says Mahmoud.

"The idea is you don't need to wait until there is downtime to process the information. You can immediately elicit out of the stream the relevant information without needing to store it.

"When you consider that 99 percent of the data collected is likely to be nothing but noise, this saves a lot of time, and money wasted on storage."

Mahmoud says stream-computing could be valuable for businesses looking to leverage the large amounts of data created from various sources online, to make better business decisions.