Consider that corporate data storage is exploding at a 60 percent annual clip, driven by the need to meet regulatory compliance dictates, keep full-motion video online and support ever-increasing amounts of analysis and correlations for that elusive competitive advantage. Small wonder that the incessant demand for storage means storage revenue grows almost twice as fast as that of the lowly server segment.
These halcyon days of hardware-dominated storage can't last. Current storage costs depend on the economics of hardware. Hardware factors such as magnetic and optical densities determine how much intelligence is needed in controllers, how to optimize redundancies and how to engineer expansion. Optimizing hardware will never remove storage bottlenecks for users pushing the limits of system performance defined by more hardware.
The future of storage lies in kicking the hardware habit. The storage equation must be inverted, with smarter software processing the information before it ever sees hardware. The brave new world of storage economics requires very intelligent software with new algorithms that automate data compression before it's sent to the storage farm.
Entrepreneurs without a stake in the existing world order are moving beyond the rules of data communications and storage laid down by Claude Shannon in 1948. Shannon, dubbed the father of information theory, said that data is a collection of bits and bytes that have no intrinsic value and are mere commodities to be moved and stored.
With the economics of storage, we know now that information has intrinsic value. Patterns and knowledge do in fact exist in Shannon's random collection of bits and bytes. A huge marketplace is waiting to form around the notion that information can be ordered, the similarities removed, and only the differences stored. Imagine the savings.
Focusing on the changes is a proven methodology today, used in display and presentation algorithms like Flash and AJAX. By sending only the parts of the display that change, we can save huge amounts of communications bandwidth and push system bottlenecks back into the storage quagmire.
A promising theoretical fix to the hardware-based economics of storage is autosophy (www.autosophy.com), invented and marketed by a small San Francisco-based research firm of the same name. The word autosophy comes from the Greek words auto (self) and sophia (knowledge or wisdom), and its inventors call it the new science of "self-assembling structures."
Autosophy is based on algorithms that selectively acquire knowledge from information and data in random environments. This acquired knowledge grows complex data constructs without human intervention. Examples of self-assembling structures in nature include trees, crystals and even complex societies like beehives and ant colonies.
Here's how autosophy is going to cure our storage addiction. It starts with the concept of an "engram" - a unit of knowledge. Knowledge can be learned only once. You can't learn what you already know, and repeating what the receiver already knows adds no knowledge. Efficient communications send only engrams, telling receivers what they don't yet know and thus omitting all the background information already known or acquired. Focusing only on engrams obviously yields a huge economic benefit by eliminating all the transmission and storage of redundant information.
A second key to the autosophy concept is an address token called a tip. Unlike the monetary reward for good service, an autosophy tip is the address for an engram. In video communications and storage, each tip could represent that part of a frame that changed -- the engram of new information that moves the action in the video. The variability of an engram eliminates any relationship between information and data volume -- a tip can represent any amount of data, from a letter to an entire book or video.
The last part of your autosophy primer is the completed whole, or "hyperspace knowledge library." This new type of data construct must be identical in both the transmitter and receiver, just like in parity-controlled traditional data storage and communications. Autosophy communications becomes more efficient as the hyperspace knowledge library grows and assembles. It can be shared prior to transmission or self-assembled from the tip codes during transmission.
Among humans, the hyperspace knowledge library analog is that people without much knowledge communicate less efficiently than experts with a great deal of knowledge, or engrams must be in short supply in Washington.
-- Mark Willoughby, CISSP, is a 20-year IT industry veteran and journalist. He can be reached at markw@messaginggroup.com.