Watson teaches 'big analytics'

06.05.2011

For years, big data was considered a critical problem for businesses trying to capture information and then deliver new products or solutions to customers based on that knowledge. Initially, the costs in storage alone could get out of hand quickly and admittedly, the look and sound daunting.

Retailers regularly collect massive amounts of information about customers from online, in-store and even social media sources. Financial institutions gather millions of daily credit card and bank transactions, and rely on multiple terabytes of historical data to create new business insights. A recent IDC report predicts data will grow some 44 times over the course of the next decade!

Too often, the industry focuses its attention primarily on this piece of the data problem. But today, those are simply big numbers. But the second piece, often ignored or pushed aside, is the problem of big analytics, because even 100 terabytes of data is entirely useless if companies haven't solved the big analytics problem.

This of course includes the aforementioned problems of scale. But modern analytic platforms must also be extremely fast in answering creative, often difficult questions drawn from multiple sources in a variety of programming languages. That is, these platforms require velocity, agility and the capacity to deal with complexity.

Velocity, first and foremost, is about brute speed and power. Watson was not only able to come up with answers with a required level of confidence but also physically buzz in before his human competitors. In business, vast stores of data -- customer information, social media feeds, financial records -- have diminishing returns as time goes by. If the information is not acted on immediately, its value plummets. For instance, financial institutions attempt to identify trades just 30 seconds ahead of the competition to maximize returns, or attempt to identify fraudulent patterns as they occur. They can't predict an event, however, if they must wait for big analytics to come back with an answer. Critically, businesses must now get from problem to question to answer in a drastically reduced timeframe.