A cure for storage hardware dependence

05.12.2005
Netizens, like digital drug addicts, keep buying ever-larger fixes of storage for all the high-definition content on the planet. But back at the knowledge ranch, weird science is going to help us kick the storage habit and the hardware economics that keep us addicted.

Consider that corporate data storage is exploding at a 60 percent annual clip, driven by the need to meet regulatory compliance dictates, keep full-motion video online and support ever-increasing amounts of analysis and correlations for that elusive competitive advantage. Small wonder that the incessant demand for storage means storage revenue grows almost twice as fast as that of the lowly server segment.

These halcyon days of hardware-dominated storage can't last. Current storage costs depend on the economics of hardware. Hardware factors such as magnetic and optical densities determine how much intelligence is needed in controllers, how to optimize redundancies and how to engineer expansion. Optimizing hardware will never remove storage bottlenecks for users pushing the limits of system performance defined by more hardware.

The future of storage lies in kicking the hardware habit. The storage equation must be inverted, with smarter software processing the information before it ever sees hardware. The brave new world of storage economics requires very intelligent software with new algorithms that automate data compression before it's sent to the storage farm.

Entrepreneurs without a stake in the existing world order are moving beyond the rules of data communications and storage laid down by Claude Shannon in 1948. Shannon, dubbed the father of information theory, said that data is a collection of bits and bytes that have no intrinsic value and are mere commodities to be moved and stored.

With the economics of storage, we know now that information has intrinsic value. Patterns and knowledge do in fact exist in Shannon's random collection of bits and bytes. A huge marketplace is waiting to form around the notion that information can be ordered, the similarities removed, and only the differences stored. Imagine the savings.